PubSub: `ConcurrentModificationException` when publishing messages with ordering keys

[kberezin-nshl]

Thanks for stopping by to let us know something could be better!

Steps to reproduce

ConcurrentModificationException rarely occurs under a mild load (we send ~3-5 pub sub messages per second through the same Publisher instance), each message has an ordering key. The ordering keys' set is fixed, and there are only 3 of them if that matters (i.e there is approximately 1-2 messages with the same ordering key per second, and there is only 3 different keys that are used).

Unfortunately, there are no steps to reproduce as it seems like this is not really possible, because everything is guarded by a reentrant lock. I spent a couple of hours trying to reproduce and/or find a root cause just by staring at the code.

I have a theory, however, that that happens when the (another?) critical section is entered again in a nested call by the same thread, therefore nested section unlocks the lock when it is left, and another thread can enter, causing them both to modify the hash map. As I said, I couldn't find the code path that could lead to that though.

Another thing that there was a log message right before the failure:

Mar 15, 2025 2:59:46 PM com.google.cloud.pubsub.v1.Publisher publishOutstandingBatch
WARNING: Attempted to publish batch with zero messages.

Stack trace

java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
    at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
    at x.y.z.FailFastExecutors$FailFastScheduledThreadPoolExecutor.afterExecute(FailFastExecutors.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
caused by: java.util.ConcurrentModificationException: null
    at java.base/java.util.HashMap$HashIterator.remove(Unknown Source)
    at com.google.cloud.pubsub.v1.Publisher.publishAllWithoutInflight(Publisher.java:449)
    at com.google.cloud.pubsub.v1.Publisher.access$1700(Publisher.java:95)
    at com.google.cloud.pubsub.v1.Publisher$2.run(Publisher.java:383)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

External references such as API reference guides

• ?

Any additional information below

Following these steps guarantees the quickest resolution possible.

Thanks!

[michaelpri10]

Hello! Could you confirm which version of this client library and which Java version you are using?

[kberezin-nshl]

Yes, sorry, should've done that earlier:

• com.google.cloud:google-cloud-pubsub:1.137.1
• com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.119.1
• Java 21:
  • docker image is gcr.io/distroless/java21-debian12
  • digest is sha256:4cbf04e9ccec9a20e1d061c13624ac6430dcb7df0b87fc63db42239ca6852def
  • arch is amd64

[michaelpri10]

@kberezin-nshl I've recently tried further attempts to reproduce this error without any luck. Did your publisher client configuration define any batching, executor, or any other custom configuration settings? Additionally, were there any publish failures occurring around when this exception was thrown?

[kberezin-nshl]

Sorry for the late reply @michaelpri10
We do use custom ExecutorProvider which just supplies an instance of ScheduledThreadPoolExecutor, nothing super interesting there. Other than that, everything is standard.

I must admit that we haven't observed the issue for a while now, we do bump all the dependencies (listed above) from time to time. Not sure if there were any changes in any of those that somehow affected / fixed that.

I think we can close this for now, and reopen if it occurs again and/or new evidence emerges.