Skip to content

Use timesync status to decide when to upgrade boundary NTP zones #8616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Use timesync status to decide when to upgrade boundary NTP zones #8616

wants to merge 2 commits into from
+1,162 −51

Conversation

smklein
Copy link
Collaborator

@smklein smklein commented Jul 16, 2025 •
edited
Loading

Depends on #8603

Prevent the planner from tearing down boundary NTP zones when we are not at expected "BOUNDARY_NTP_REDUNDANCY" synchronized zones.

Adds a test which demonstrates that we'll only drop "one zone" below redundancy during an upgrade, and never
further than that.

Fixes #8547

@smklein smklein force-pushed the inventory-collect-timesync branch from 75b39c7 to 30a86a1 Compare July 17, 2025 19:49
@smklein smklein force-pushed the planner-ntp-upgrade branch from 6bbc962 to 0873e92 Compare July 17, 2025 19:56
@plotnick plotnick mentioned this pull request Jul 17, 2025
Draft
5 tasks
@smklein smklein force-pushed the inventory-collect-timesync branch from 30a86a1 to 2ded09d Compare July 17, 2025 22:27
@smklein smklein force-pushed the planner-ntp-upgrade branch from 0873e92 to 322598b Compare July 17, 2025 22:28
@smklein smklein force-pushed the inventory-collect-timesync branch from 2ded09d to e784b52 Compare July 17, 2025 23:05
@smklein smklein force-pushed the planner-ntp-upgrade branch from 322598b to bfe4c7a Compare July 17, 2025 23:05
@smklein smklein force-pushed the inventory-collect-timesync branch from e784b52 to 568a4db Compare July 18, 2025 00:08
@smklein smklein force-pushed the planner-ntp-upgrade branch from bfe4c7a to 84fbbe8 Compare July 18, 2025 00:08
@smklein smklein marked this pull request as ready for review July 18, 2025 15:47
@smklein smklein force-pushed the inventory-collect-timesync branch from 568a4db to 71e2738 Compare July 18, 2025 15:49
@smklein smklein mentioned this pull request Jul 18, 2025
Merged
@smklein smklein force-pushed the planner-ntp-upgrade branch from 84fbbe8 to e176df8 Compare July 18, 2025 15:53
@smklein smklein force-pushed the inventory-collect-timesync branch from 71e2738 to 91d9377 Compare July 18, 2025 20:05
@smklein smklein force-pushed the planner-ntp-upgrade branch from e176df8 to d4d98cd Compare July 18, 2025 20:12
andrewjstone
andrewjstone approved these changes Jul 19, 2025
View reviewed changes
Copy link
Contributor

@andrewjstone andrewjstone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great testing. Easy to read :)

nexus/reconfigurator/planning/src/planner.rs
hash: fake_hash,
};
let artifacts = vec![
// Omit `BoundaryNtp` because it has the same artifact name as
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an issue or anything describing this? Is it something that we need to fix?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#8510 is closely related but doesn't talk about this specifically. I think the issue here is that BoundaryNtp and InternalNtp are critically distinct from each other in some ways, but all of those ways are exclusively runtime config things: the actual artifact for both types is identical. I don't think it's something to fix? Although maybe we should have a helper somewhere for "build me a complete fake set of artifacts", so we could stop copy-pasting this.

nexus/reconfigurator/planning/src/planner.rs
@@ -1595,6 +1596,48 @@ impl<'a> Planner<'a> {
}
true
}
ZoneKind::BoundaryNtp => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love how straightforward this logic is.

jgallagher
jgallagher reviewed Jul 21, 2025
View reviewed changes
nexus/reconfigurator/planning/src/planner.rs

// Find all boundary NTP zones
let mut boundary_ntp_zones = std::collections::HashSet::new();
for sled_agent in self.inventory.sled_agents.iter() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be looking at the current blueprint instead of inventory? If we have an out-of-date inventory, this might show us a boundary NTP zone that has since been expunged (and if it's also still in inventory.ntp_timesync, we'd count it as a sync'd zone).

nexus/reconfigurator/planning/src/planner.rs
hash: fake_hash,
};
let artifacts = vec![
// Omit `BoundaryNtp` because it has the same artifact name as
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#8510 is closely related but doesn't talk about this specifically. I think the issue here is that BoundaryNtp and InternalNtp are critically distinct from each other in some ways, but all of those ways are exclusively runtime config things: the actual artifact for both types is identical. I don't think it's something to fix? Although maybe we should have a helper somewhere for "build me a complete fake set of artifacts", so we could stop copy-pasting this.

@smklein smklein force-pushed the inventory-collect-timesync branch 4 times, most recently from bea6e02 to 5f2c4dd Compare July 21, 2025 21:39
Base automatically changed from inventory-collect-timesync to main July 22, 2025 02:36
smklein added a commit that referenced this pull request Jul 22, 2025
@smklein
Collect timesync status in inventory (#8603)
cc1aba6 
Depends on #8597

Queries the NTP admin server for both internal and boundary NTP zones.
Stores a simplified
representation of "who was queried, and is time synced for this
service".

This information will be used in
#8616

Fixes #8546
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgallagher jgallagher jgallagher left review comments

@andrewjstone andrewjstone andrewjstone approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@smklein @andrewjstone @jgallagher