Upgrading Unleash causes Java Out of Memory

[Syuziko]

Describe the bug

When we upgraded Unleash java client from version 10.2.2 to 11.0.0, we started seeing errors in production

We reverted the upgrade and the errors went away.

Steps to reproduce the bug

No response

Expected behavior

No response

Logs, error output, etc.

An unhandled exception occurred for request: 
com.dylibso.chicory.runtime.WasmRuntimeException: out of bounds memory access: attempted to access address: -759324 but limit is: 1192886272 and size: 4
    at com.dylibso.chicory.runtime.ByteBufferMemory.outOfBoundsException(ByteBufferMemory.java:155)
    at com.dylibso.chicory.runtime.ByteBufferMemory.writeI32(ByteBufferMemory.java:207)
    at io.getunleash.wasm.YggdrasilMachine$AotMethods.memoryWriteInt(AotMethods.java:126)
    at io.getunleash.wasm.YggdrasilMachine.func_1173(wasm)
    at io.getunleash.wasm.YggdrasilMachine.func_1132(wasm)
    at io.getunleash.wasm.YggdrasilMachine.func_1149(wasm)
    at io.getunleash.wasm.YggdrasilMachine.func_438(wasm)
    at io.getunleash.wasm.YggdrasilMachine.func_362(wasm)
    at io.getunleash.wasm.YggdrasilMachine$MachineCall.call_362(Unknown Source)
    at io.getunleash.wasm.YggdrasilMachine$MachineCall.call_dispatch_0(Unknown Source)
    at io.getunleash.wasm.YggdrasilMachine$MachineCall.call(Unknown Source)
    at io.getunleash.wasm.YggdrasilMachine.call(wasm)
    at com.dylibso.chicory.runtime.Instance$Exports.lambda$function$0(Instance.java:214)
    at io.getunleash.engine.WasmInterface.checkEnabled(WasmInterface.java:142)
    at io.getunleash.engine.UnleashEngine.isEnabled(UnleashEngine.java:219)
    at io.getunleash.repository.FeatureRepositoryImpl.isEnabled(FeatureRepositoryImpl.java:183)
    at io.getunleash.EngineProxyImpl.isEnabled(EngineProxyImpl.java:48)
    at io.getunleash.DefaultUnleash.isEnabled(DefaultUnleash.java:117)
    at io.getunleash.Unleash.isEnabled(Unleash.java:20)
    at io.getunleash.DefaultUnleash.isEnabled(DefaultUnleash.java:99)
....

Screenshots

No response

Additional context

No response

Unleash version

11.0.0

Subscription type

None

Hosting type

None

SDK information (language and version)

Java 21

[Karamell]

I can confirm we also see java.lang.OutOfMemoryError after upgrading to version 11.0.0. Downgrading fixes the problem.

[sighphyre]

Hey @Syuziko, we're looking into this one but finding it really hard to reproduce. I have some patches on the way out that I think may help but without a validation case, I can't promise anything.

By any chance are you using multiple instances of the Unleash SDK in the same process?

[sighphyre]

Hey @Karamell, just to confirm, are you seeing the same error as Syuziko or an OutOfMemoryError? The original stack trace looks to be an out of bounds write to my eyes

[Karamell]

@sighphyre: This is how it looks:

stackTrace: com.dylibso.chicory.runtime.WasmRuntimeException: out of bounds memory access: attempted to access address: -326316 but limit is: 301989888 and size: 4
	at com.dylibso.chicory.runtime.ByteBufferMemory.outOfBoundsException(ByteBufferMemory.java:155)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoDefer] :

also a stracktrace with only

java.lang.OutOfMemoryError: Java heap space

that happened at the same time

[sighphyre]

Ah thanks for the info here. Okay I might actually know what's causing this and have a fix for it. Are you using multiple instances of the SDK in the same process by any chance? Or tearing recreating the SDK at some point?

[Karamell]

@sighphyre AFAIK it's not multiple instances. We use spring and it's instantiated in a @Bean. Nothing fancy.

[sighphyre]

Hey @Karamell @Syuziko,

So we can't reproduce this. At all. We've pressure tested the SDK in as many ways as we can think of. We also see that the adoption of this version is already fairly wide spread and no else is reporting this so I think it's something very specific to your setups.

We've made some changes to the way we handle memory and locking here but without a reproduction case we can't promise anything. If you feel like helping us out by checking in a staging/testing environment that would be amazing but no pressure.

Rolling back to a previous version is absolutely fine, we'll continue to support 9.x and 10.x for at least a few years but we also really, really want to get to the bottom of this bug. If there's any insight either of you could shed on the lead up to the exception being raised that would be amazing and if either of you are willing to have a conversation with us on our Slack channel to see if we can't work it out over a chat that would also be amazing.

Also a reproducible example would be absolutely golden, if possible

[Syuziko]

Hi @sighphyre ,

I'll try to reproduce and provide more details.

You mentioned that you've made some changes with handling memory and locking - I can try run it on our side and see whether it resolves the issue or not.

[sighphyre]

@Syuziko Thank you so much!