feature: replace discovery service with DHT discovery #629

noot · 2025-07-16T20:54:42Z

changes:

add bootnode service; this is a very basic p2p node which doesn't do anything but helps other nodes join the p2p network. alternative is to have a worker/validator/orchestrator be a bootnode but that would be unideal as bootnodes should be specialized (high uptime and can handle many incoming connections)
add kademlia to p2p service and expose KademliaAction handler to allow services to make dht queries
update worker to advertise itself as a worker in the DHT upon startup
update validator to fetch workers from the DHT during validation loop
update orchestrator DiscoveryMonitor to fetch workers from the DHT for updating its node store

testing:

local setup done with make up, make watch-worker and ensured the worker is discovered by the validator and orchestrator and invited to the compute pool.

… dht on startup

Copilot

Pull Request Overview

This PR replaces the centralized discovery service with a peer-to-peer Distributed Hash Table (DHT) discovery mechanism. This architectural change improves decentralization by removing the single point of failure and dependency on centralized discovery services.

Key changes include:

Introduction of bootnode service for network bootstrapping
Integration of Kademlia DHT for worker discovery and advertising
Replacement of HTTP-based discovery with P2P DHT queries

Reviewed Changes

Copilot reviewed 111 out of 114 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
docs/development-setup.md	Updates documentation to reflect bootnode service replacing discovery service
docker-compose.yml	Replaces discovery service with bootnode service configuration
deployment/k8s/	Complete migration from discovery-chart to bootnode-chart with template updates
crates/worker/src/	Removes discovery service code and integrates DHT advertising for workers
crates/validator/src/	Refactors to use DHT-based node discovery instead of HTTP discovery endpoints
crates/orchestrator/src/	Updates to fetch worker nodes from DHT for node management
crates/shared/src/	Adds DHT discovery utilities and updates node models
crates/p2p/src/	Implements Kademlia DHT functionality with discovery capabilities

Comments suppressed due to low confidence (1)

crates/p2p/src/lib.rs:448

The test is sleeping for 2 seconds instead of properly waiting for peer connection, which could lead to flaky tests. Consider implementing a proper mechanism to verify peer connectivity before proceeding with the test.

        // TODO: implement a way to get peer count (https://github.com/PrimeIntellect-ai/protocol/issues/628)

crates/worker/src/cli/command.rs

Copilot · 2025-07-17T16:41:03Z

crates/worker/src/cli/command.rs

                provider_address: String::new(),
                compute_pool_id: 0,
-                worker_p2p_id: None,
+                worker_p2p_id: "empty".to_string(), // TODO: this should be a different type, as peer id is not needed for this code path


Using a placeholder value like "empty" for worker_p2p_id indicates a design issue. Consider using an Option or creating a separate type for cases where peer ID is not needed.

agreed, there is a todo

crates/validator/src/validator.rs

crates/shared/src/models/node.rs

Copilot · 2025-07-17T16:41:04Z

docker-compose.yml

-      - REDIS_URL=redis://redis:6379
-      - PLATFORM_API_KEY=prime
-      - MAX_NODES_PER_IP=3
+      - LIBP2P_PRIVATE_KEY="d0884c9823a0a2c846dbf5e71853bc5f80b2ec5d2de46532cdbe8ab46f020836"


Hardcoded private key in docker-compose.yml poses a security risk. Use environment variables or secure key management instead.

agreed but this is a known key (in .env.example)

Copilot · 2025-07-17T16:41:04Z

deployment/k8s/validator-chart/values.example.yaml

 env:
  RPC_URL: "http://anvil.example-namespace:8545"
-  DISCOVERY_URLS: "http://discovery.example-namespace:8089"
+  BOOTNODES: "/ip4/127.0.0.1/tcp/4005/p2p/12D3KooWJj3haDEzxGSbGSAvXCiE9pDYC9xHDdtQe8B2donhfwXL"


Using localhost (127.0.0.1) as a bootnode address in Kubernetes deployment configuration will not work in a distributed environment. This should be updated to use proper cluster addresses.

Suggested change

BOOTNODES: "/ip4/127.0.0.1/tcp/4005/p2p/12D3KooWJj3haDEzxGSbGSAvXCiE9pDYC9xHDdtQe8B2donhfwXL"

BOOTNODES: "/dns4/bootnode-service/tcp/4005/p2p/12D3KooWJj3haDEzxGSbGSAvXCiE9pDYC9xHDdtQe8B2donhfwXL"

agreed, i have not tested this with kubernetes

deployment/k8s/orchestrator-chart/values.example.yaml

noot · 2025-07-17T16:49:17Z

crates/bootnode/src/main.rs

+}
+
+#[tokio::main]
+async fn main() {


the bootnode binary is new, but it's very simple, it only runs a p2p node.

noot · 2025-07-17T16:51:22Z

crates/orchestrator/src/discovery/monitor.rs

@@ -1,59 +1,90 @@
+use crate::discovery::location_service::LocationService;


the DiscoveryMonitor was refactored to have 2 internal components: NodeFetcher and Updater. NodeFetcher fetches the nodes from the DHT and returns them, while Updater performs the same "syncing" logic and updating the node store as previously.

noot · 2025-07-17T16:52:11Z

crates/orchestrator/src/discovery/monitor.rs

-        // Check if there's any healthy node with the same IP and port
-        let count_healthy_nodes_with_same_endpoint = self
-            .count_healthy_nodes_with_same_endpoint(
+        // Check if there's any healthy node with the same peer ID


this was changed from ip:port collisions to p2p peer ID collisions under the assumption that distinct nodes will have distinct peer IDs, if two nodes have the same peer ID they're actually the same node.

noot · 2025-07-17T16:52:27Z

crates/orchestrator/src/discovery/monitor.rs

-            .collect())
+use crate::store::NodeStore;
+
+async fn enrich_nodes_without_location(


this was moved from the discovery service

noot · 2025-07-17T16:54:21Z

crates/shared/src/models/node.rs

    pub port: u16,
    pub compute_pool_id: u32,
    pub compute_specs: Option<ComputeSpecs>,
+    pub worker_p2p_id: String, // TODO: change to p2p::PeerId


this was changed to not be Option, i would also like to change worker_p2p_addresses to not be Option also and change the types from String but diff was already huge

noot · 2025-07-17T16:55:13Z

crates/shared/src/p2p/discovery.rs

+/// Given a kademlia action channel backed by a `p2p::Service`,
+/// perform a DHT lookup for all nodes which claim to be a worker using `GetProviders` for `p2p::WORKER_DHT_KEY`.
+/// then, for each of these nodes, query the DHT for their record (which stores their node information) using `GetRecord`.
+pub async fn get_worker_nodes_from_dht(


this is new logic which fetches the workers from the DHT

noot · 2025-07-17T16:55:52Z

crates/validator/src/main.rs

-// Track the last loop duration in milliseconds
-static LAST_LOOP_DURATION_MS: AtomicI64 = AtomicI64::new(0);
-
-async fn get_rejections(


didn't delete this, just moved to end of file

noot · 2025-07-17T16:56:32Z

crates/validator/src/main.rs

        }
    });

-    loop {


i moved this loop's logic to the validator module

noot · 2025-07-17T16:57:10Z

crates/validator/src/validator.rs

+}
+
+#[allow(clippy::too_many_arguments)]
+async fn perform_validation(


same logic as validation loop which was in main before

noot · 2025-07-17T16:57:59Z

crates/worker/src/cli/command.rs

+            // TODO: should update p2p service to expose this better (https://github.com/PrimeIntellect-ai/protocol/issues/628)
+            tokio::time::sleep(Duration::from_secs(1)).await;
+
+            let record_key = p2p::worker_dht_key_with_peer_id(&peer_id);


instead of uploading to the discovery service we now advertise on the DHT

noot added 10 commits July 14, 2025 17:05

add bootnodes to cmd; impl kad actions and handlers; worker upload to…

e9f7914

… dht on startup

implement validator which fetches nodes from dht for validation

07b4533

clean up validation loop

7e0493d

wip put record

1f2374c

impl basic bootnode; set dht to server, provider discovery working

990d70c

update orchestrator to use dht for discovery

3aa5e23

start discovery monitor in orchestrator again

1ae81e6

cleanup, remove discovery crate

7ce274a

update dockerfile, charts, workflows for bootnode

3869b17

cleanup, add dockerfile to bootnode

7a8b834

noot requested a review from Copilot July 16, 2025 20:54

This comment was marked as outdated.

Sign in to view

noot added 2 commits July 16, 2025 22:35

add location enrichment to orchestrator

4facf43

fmt

aa798aa

noot requested a review from Copilot July 17, 2025 02:48

This comment was marked as outdated.

Sign in to view

noot requested a review from Copilot July 17, 2025 16:39

Copilot AI reviewed Jul 17, 2025

View reviewed changes

noot commented Jul 17, 2025

View reviewed changes

crates/validator/src/main.rs

}

});

loop {

Copy link

Contributor Author

noot Jul 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i moved this loop's logic to the validator module

noot commented Jul 17, 2025

View reviewed changes

cleanup

6fe2516

noot marked this pull request as ready for review July 17, 2025 16:58

fix tokio features

3dfc290

itzlambda mentioned this pull request Jul 20, 2025

refactor: task startup and shutdown across binaries #630

Draft

	BOOTNODES: "/ip4/127.0.0.1/tcp/4005/p2p/12D3KooWJj3haDEzxGSbGSAvXCiE9pDYC9xHDdtQe8B2donhfwXL"
	BOOTNODES: "/dns4/bootnode-service/tcp/4005/p2p/12D3KooWJj3haDEzxGSbGSAvXCiE9pDYC9xHDdtQe8B2donhfwXL"

		@@ -1,59 +1,90 @@
		use crate::discovery::location_service::LocationService;

                       }
                   });
-                  loop {

feature: replace discovery service with DHT discovery #629

Are you sure you want to change the base?

feature: replace discovery service with DHT discovery #629

Uh oh!

Conversation

noot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noot Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

noot commented Jul 16, 2025 •

edited

Loading

noot Jul 17, 2025 •

edited

Loading