[RayCluster] Toggle usage of deterministic/non-deterministic head pod name with feature flag #3873
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
|
Tested the feature flag locally
|
Signed-off-by: machichima <[email protected]>
|
@rueian PTAL, thank you! |
|
I added the test in |
rueian
reviewed
Jul 16, 2025
Comment on lines
718
to
722
| enableDeterministicHeadPodName := false | ||
| if s := os.Getenv(ENABLE_DETERMINISTIC_HEAD_POD_NAME); strings.ToLower(s) == "true" { | ||
| enableDeterministicHeadPodName = true | ||
| } | ||
| return enableDeterministicHeadPodName |
There was a problem hiding this comment.
Suggested change
| enableDeterministicHeadPodName := false | |
| if s := os.Getenv(ENABLE_DETERMINISTIC_HEAD_POD_NAME); strings.ToLower(s) == "true" { | |
| enableDeterministicHeadPodName = true | |
| } | |
| return enableDeterministicHeadPodName | |
| return strings.ToLower(os.Getenv(ENABLE_DETERMINISTIC_HEAD_POD_NAME)) == "true" |
|
We can have an e2e test for the autoscaler restart with GCS FT in another PR, since it has nothing to do with the feature flag. |
Signed-off-by: machichima <[email protected]>
Signed-off-by: machichima <[email protected]>
|
@rueian I've resolved the conflicts. PTAL, thank you! |
rueian
approved these changes
Jul 18, 2025
win5923
reviewed
Jul 19, 2025
Comment on lines
+179
to
+181
| # If set to true, we will use deterministic name for head pod. Otherwise, the non-deterministic name is used. | ||
| # - name: ENABLE_DETERMINISTIC_HEAD_POD_NAME | ||
| # value: "false" |
win5923
approved these changes
Jul 19, 2025
laurafitzgerald
pushed a commit
to laurafitzgerald/kuberay
that referenced
this pull request
Jul 25, 2025
… name with feature flag (ray-project#3873) * feat: use gen name for ray version > 2.48 Signed-off-by: machichima <[email protected]> * feat: head pod name to non deterministic Signed-off-by: machichima <[email protected]> * test: compare head pod name between old and new Signed-off-by: machichima <[email protected]> * feat: add feature flag for deterministic head pod name Signed-off-by: machichima <[email protected]> * feat: use deterministic if feature flag set to true Signed-off-by: machichima <[email protected]> * test: head pod name with/without feature flag == true Signed-off-by: machichima <[email protected]> * test: sep test for head/worker pod name Signed-off-by: machichima <[email protected]> * docs: add feature flag desc to helm chart Signed-off-by: machichima <[email protected]> * fix: feature flag name Signed-off-by: machichima <[email protected]> * refactor: simplify check if the feature flag is set Signed-off-by: machichima <[email protected]> --------- Signed-off-by: machichima <[email protected]>
DW-Han
pushed a commit
to DW-Han/kuberay
that referenced
this pull request
Jul 30, 2025
… name with feature flag (ray-project#3873) * feat: use gen name for ray version > 2.48 Signed-off-by: machichima <[email protected]> * feat: head pod name to non deterministic Signed-off-by: machichima <[email protected]> * test: compare head pod name between old and new Signed-off-by: machichima <[email protected]> * feat: add feature flag for deterministic head pod name Signed-off-by: machichima <[email protected]> * feat: use deterministic if feature flag set to true Signed-off-by: machichima <[email protected]> * test: head pod name with/without feature flag == true Signed-off-by: machichima <[email protected]> * test: sep test for head/worker pod name Signed-off-by: machichima <[email protected]> * docs: add feature flag desc to helm chart Signed-off-by: machichima <[email protected]> * fix: feature flag name Signed-off-by: machichima <[email protected]> * refactor: simplify check if the feature flag is set Signed-off-by: machichima <[email protected]> --------- Signed-off-by: machichima <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
KubeRay 1.4.0 uses a deterministic name for the head pod. This, unfortunately, breaks the Autoscaler v2 for Ray < 2.48 after a head node restart under GCS FT configuration.
Based on #3868 (comment), we shouldn't change the behavior in the operator based on the Ray version. Therefor, we decided to add a feature flag that can toggle between using deterministic or non-deterministic head pod name (default to use non-deterministic head pod name)
ENABLE_DETERMINISTIC_HEAD_POD_NAMEto set whether we want to use deterministic name for head podRelated issue number
Closes #3868
Checks