Online Indexer: replace the synchronized runner with a heartbeat #3530

jjezra · 2025-08-04T15:30:30Z

Each indexing session, for each index, will create a key-value heartbeat of the format:
[prefix, xid] -> [indexing-type, genesis time, heartbeat time]
Indexing session that are expected to be exclusive will throw an exception if another, active, session exists.

Motivation:
1. Represent the heartbeat in every index during multi target indexing (currently - only the master index has a sync lock)
2. Keep a heartbeat during mutual indexing, which can allow better automatic decision making
3. Decide about exclusiveness according to the indexing method (currently - user input)

With this change, the equivalent of a sync lock will be determined by the indexing type and cannot be set by the users. The index configuration function setUseSynchronizedSession will have no effect on the indexing process.

During graduate code upgrade on multiple servers, there may be a situation where one server is indexing with a synchronized session lock, while another server builds the same index with an exclusive heartbeat "lock". If that happens:
a) There will be no more than two concurrent active sessions
b) The indexing sessions will conflict each other until one of the indexers will give up. While this may be not optimal, the generated index will be valid.

Resolve #3529

ScottDugas

I didn't get a chance to look at all of the code, we can discuss some of the things offline.

ScottDugas · 2025-08-12T20:15:18Z

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexerBuildIndexTest.java

@@ -180,7 +178,6 @@ <M extends Message> void singleRebuild(
        if (!safeBuild) {


What does safeBuild here mean in this context? Do we need to continue to prevent concurrency?

The things that had left to non-safeBuild, as it seems, are clearing and marking the index as write-only and setting constrains that, IIUIC, should not be happening after the previous clearing.

After so many changes in the code, maybe it is time to re-design (and possibly simplify) the singleRebuild. Should that be a separate PR?

changing and cleaning up safeBuild should not be part of this PR.

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexerBuildIndexTest.java

...ain/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexOperationConfig.java

...ore/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeat.java

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

...yer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingBase.java

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

ScottDugas · 2025-08-12T21:19:37Z

...er-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexer.java

@@ -1237,7 +1267,7 @@ public static class Builder {
            private DesiredAction ifReadable = DesiredAction.CONTINUE;
            private boolean doAllowUniquePendingState = false;
            private Set<TakeoverTypes> allowedTakeoverSet = null;
-            private long checkIndexingStampFrequency = 60_000;
+            private long checkIndexingStampFrequency = 10_000;


Why check the type stamp more often?

To align the default stamp & heartbeat checking with the default lease time.

Wouldn't you want the lease time to be longer than the heartbeat?
If you only update the heartbeat every 10 seconds, and you consider it too-old after 10 seconds, it doesn't take much clock skew to have indexers stepping over each other.

I had changed the default to check heartbeats and typestamp every 5 seconds. I would consider changing the default least time for longer than 10 seconds, but this should probably be done on a separate PR.

ScottDugas

There were also two comments left on my previous review, that I want to followup on. I resolved everything else in that review, or answered in a (I hope) definitive way, so it should be easy to go back through.

...r-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/FDBRecordStore.java

...yer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingBase.java

ScottDugas · 2025-08-18T19:28:41Z

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

+        List<Index> indexes = new ArrayList<>();
+        indexes.add(new Index("indexA", field("num_value_2"), EmptyKeyExpression.EMPTY, IndexTypes.VALUE, IndexOptions.UNIQUE_OPTIONS));
+        indexes.add(new Index("indexB", field("num_value_3_indexed"), IndexTypes.VALUE));
+        FDBRecordStoreTestBase.RecordMetaDataHook hook = allIndexesHook(indexes);


It's a matter of the store creation. When you create an FDBRecordStore it is passed in a SubspaceProvider, which could present something that is not a tuple. Looking briefly at the amount of testing infrastructure that you'd have to rework, I'm not sure how likely it is that something else wouldn't break.
I think it would be valuable to have, but I don't think it's worth the effort, and I think we might be better off focusing on changing FDBRecordStore to take a KeySpacePath only.
I don't think there's any way to do anything after that that is not a Tuple

ScottDugas · 2025-08-18T19:32:45Z

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

+                    startSemaphore.acquire();
+                    Thread.sleep(100);
+                } catch (InterruptedException e) {
+                    throw new RuntimeException(e);


You should always re-interrupt the current thread when catching InterruptedException

ScottDugas · 2025-08-18T19:37:05Z

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

+                try (FDBRecordContext context = openContext()) {
+                    final Map<UUID, IndexBuildProto.IndexBuildHeartbeat> heartbeats = IndexingHeartbeat.getIndexingHeartbeats(recordStore, indexes.get(0), 0).join();
+                    heartbeatsQueries.add(heartbeats);
+                    context.commit();


You don't need to commit here, you're just doing a read.

Getting warnings. I don't think that creating a context without commit passes a PRB now.

ScottDugas · 2025-08-18T19:55:49Z

...ore/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeat.java

+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class IndexingHeartbeat {


We discussed offline, and decided to take a balanced approach of changing the method to info, so that this is, on disk, future-proofed for being upgraded to be more generic if we want to use it elsewhere. There would still need to be some substantial changes to the code, but it would need to take a subspace instead of a store and an index.

ScottDugas · 2025-08-18T20:04:53Z

.../java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeatLowLevelTest.java

+
+            // clear, partial
+            int countDeleted =
+                    IndexingHeartbeat.clearIndexingHeartbeats(recordStore, index, 0, 7).join();


This is clearing 7 entries, regardless of their last heartbeat, right?

Yes. Max age to delete is 0.

Each indexing session, for each index, will create a key-value heartbeat of the format: [prefix, xid] -> [indexing-type, genesis time, heartbeat time] Indexing session that are expected to be exclusive will throw an exception if another, active, session exists. Motivation: 1. Represent the heartbeat in every index during multi target indexing (currently - only the master index has a sync lock) 2. Keep a heartbeat during mutual indexing, which can allow better automatic decision making 3. Decide about exclusiveness according to the indexing method (currently - user input) Resolve FoundationDB#3529

jjezra added the breaking change Changes that are not backwards compatible label Aug 4, 2025

jjezra force-pushed the indexing_heartbeat branch from 89d0263 to f318f65 Compare August 7, 2025 15:55

jjezra requested a review from ScottDugas August 8, 2025 17:18

jjezra force-pushed the indexing_heartbeat branch from a680857 to 83830c6 Compare August 11, 2025 12:01

jjezra marked this pull request as ready for review August 11, 2025 14:13

jjezra marked this pull request as draft August 11, 2025 15:06

jjezra marked this pull request as ready for review August 11, 2025 20:23

ScottDugas requested changes Aug 12, 2025

View reviewed changes

jjezra force-pushed the indexing_heartbeat branch from 4cf86f4 to 97d02b6 Compare August 15, 2025 13:22

jjezra requested a review from ScottDugas August 15, 2025 13:52

ScottDugas requested changes Aug 18, 2025

View reviewed changes

jjezra added 17 commits August 21, 2025 16:53

Set/clear heartbeats at start/end

27a9095

Optimize heartbeat clear. Deprecate use-sync-indexing API

38833cf

false AvoidUsingHardCodedIP warning

bcbdc8f

Clear heartbeats after rebuild (whting the same transaction)

79c3622

Itarate targets explicitly after rebuild

69638b9

thenApply -> whenComplete

c99bacc

Add API get/clear heartbeats

77a0fb2

Add 1st test - low level

9db0abe

Add two more tests

6e8bdeb

Add a comment..

94c81c4

Deprecated sync lock API functions

5d8db10

Some cleanup

819c232

At genesis, first check indexing type, then heartbeat

a8e853d

Use subspace(...).unpack(..)

933c921

Verify no leftover heartbeats after any singleRebuild

063f506

Apply Scott's requested changes

001b5e6

jjezra force-pushed the indexing_heartbeat branch from 97d02b6 to 149b070 Compare August 22, 2025 20:24

Apply (more of) Scott's requested changes

2a9f7e5

jjezra force-pushed the indexing_heartbeat branch from 86ad032 to 2a9f7e5 Compare August 25, 2025 15:35

jjezra requested a review from ScottDugas August 25, 2025 15:36

		@@ -180,7 +178,6 @@ <M extends Message> void singleRebuild(
		if (!safeBuild) {

Online Indexer: replace the synchronized runner with a heartbeat #3530

Are you sure you want to change the base?

Online Indexer: replace the synchronized runner with a heartbeat #3530

Conversation

jjezra commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ScottDugas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ScottDugas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jjezra commented Aug 4, 2025 •

edited

Loading