Skip to content

Support replication with arbitrary shard counts #65

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

pglass
Copy link
Collaborator

@pglass pglass commented Mar 11, 2025

What was changed

Support replication with arbitrary numbers of history shards.

⚠️ This requires very many replication streams (equal to LCM of the two cluster shard counts). This is pending testing to understand the performance implications.

Why?

Checklist

  1. Closes

  2. How was this tested:

  1. Any docs updates needed?

// LCM calcuates the least common multiple of a and b.
//
// https://en.wikipedia.org/wiki/Least_common_multiple#Using_the_greatest_common_divisor
func LCM(a, b int32) int32 {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are surprisingly few small libraries for doing this, and I wasn't satisfied with their unit testing (just few cases). I figured these functions are so small that I'd implement them myself to avoid the dependency.

@@ -213,21 +225,56 @@ func (s *adminServiceProxyServer) StreamWorkflowReplicationMessages(
if !ok {
return serviceerror.NewInvalidArgument("missing cluster & shard ID metadata")
}
targetClusterShardID, sourceClusterShardID, err := history.DecodeClusterShardMD(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this was my fault when I bumped the temporal version, but the target/source ids here were flipped.

I changed them to match "client/server" terminology of the history.DecodeClusterShardMD function?

I'm fine either way.

@pglass pglass requested review from hehaifengcn March 11, 2025 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant