simln-lib/refactor: fully deterministic produce events #277

f3r10 · 2025-06-03T20:21:20Z

Description

The goal of this PR is to achieve fully deterministic runs to get reproducible simulations

Changes

nodes: HashMap<PublicKey, Arc<Mutex<dyn LightningNode>>> Update HashMap for a BTreeMap. A HashMap does not maintain an order, which has an impact when the simulation is running, making the results unpredictable. Using a BTreeMap, the order of the nodes is always the same.
dispatch_producers acts as a master task, generating all the payments of the nodes, getting the random destination, and only then spawning a threat for producing the events (produce_events)

Addresses #243

f3r10 · 2025-06-04T14:20:06Z

Opening in draft still need to fix some issues.
But I think it is ready for a first review pass @carlaKC

carlaKC · 2025-06-09T17:45:50Z

btw don't worry about fixups until this is out of draft - when review hasn't properly started it's okay to just squash em!

carlaKC

Direction looking good here! Main comment is that I think we need to have a step where we replenish our heap by calling generate_payments again?

If payment_count().is_none() we return a single payment from generate_payments
For RandomPaymentActivity, this means we'll do one payment per node and then shut down?

Related to this is that we possibly don't want to queue up tons of events for when payment_count is defined (say, we want a million payments, we'll queue up a million items which is a bit of a memory waste). This probably isn't much of a big deal, because I'd imagine this use case is primarily for smaller numbers but something to keep in mind as we address the above requirement.

Also would be good to rebase this early on to get to a more current base 🙏

f3r10 · 2025-06-10T20:10:14Z

Direction looking good here! Main comment is that I think we need to have a step where we replenish our heap by calling generate_payments again?

The idea would be to generate all the payments at once, so the master task would dispatch the events.

If payment_count().is_none() we return a single payment from generate_payments

Yes, in this case, only one payment is generated

For RandomPaymentActivity, this means we'll do one payment per node and then shut down?

Yes, right now it is working in this mode 🤔

Related to this is that we possibly don't want to queue up tons of events for when payment_count is defined (say, we want a million payments, we'll queue up a million items which is a bit of a memory waste). This probably isn't much of a big deal, because I'd imagine this use case is primarily for smaller numbers but something to keep in mind as we address the above requirement.

Yes, you are right, maybe it would be better to create some batches of payments. I am going to try to come up with an alternative to reduce the memory waste. 🤔

f3r10 · 2025-06-13T16:06:27Z

Hi @carlaKC , I've developed a new approach for the event generation system. The core idea is to centralize the random number generation to ensure deterministic outcomes for our simulations.

Here's a breakdown of the design:

Central Manager Task: A dedicated thread runs a central manager. This manager is the sole source for generating both random wait times and random destinations. By centralizing this, we ensure that the sequence of random numbers generated for these critical values is entirely reproducible, given a fixed seed.
Executor Event Listeners: For each executor, a separate thread is spawned. These threads act as listeners for payment events, forwarding them to the designated consumers once received.
Payment Event Generators: Concurrently, for each executor, another thread is spawned. These threads are responsible for generating payment events in a continuous loop (e.g., for RandomActivity). Each generator thread communicates with the central manager via a dedicated channel to request a wait time. After awaiting the specified duration, it sends another event to the manager to trigger the calculation of a random destination. Once the destination is determined, the manager dispatches a final event to the respective event listener thread (as described in the previous point).

This design ensures that the wait times and final destinations are entirely deterministic across simulation runs. However, there is a new challenge with the non-deterministic order of thread execution.

The Determinism Challenge

While the values generated (wait times, destinations) are fixed if the random number generator is seeded, the order in which the executor threads request these values is not guaranteed. For example, if we have ex1 and ex2 executors:

Execution 1:
    ex1 gets wait_time 0 → destination node_3
    ex2 gets wait_time 1 → destination node_4

Execution 2 (possible non-deterministic order):
    ex2 gets wait_time 0 → destination node_3
    ex1 gets wait_time 1 → destination node_4

This means that even though the sequence of random numbers from the central manager is the same, which executor consumes which number from that sequence is left to the operating system's scheduler, leading to variations in the overall simulation flow.

Proposed Solution for Execution Order

To achieve full simulation determinism, including the order of execution, I'm considering adding a tiny, randomized initial sleep time before each executor thread begins its main loop. While seemingly counter-intuitive, this jitter can effectively "break ties" in thread scheduling in a controlled, reproducible way when combined with a seeded random number generator. This would allow us to deterministically influence which thread acquires the next available random number from the central manager.

WDYT?

carlaKC · 2025-06-17T15:32:18Z

Deleted previous comment - it had some misunderstandings.

Why can't we keep the current approach of generating a queue of events and then replenish the queue when we run out of events? By generating all of our payment data in one place, we don't need to worry about thread execution order.

I think that this can be as simple as pushing a new event to the queue every time we pop one? We might need to track some state for payment count (because we'll need to remember how many we've had), but for random activity it should be reasonable.

carlaKC · 2025-06-17T15:47:11Z

Rough sketch of what I was picturing:

Queue up initial set of events:

for each executor
- Get wait time and destination
- Push wait time, destination and ExecutorKit onto head

Read from heap:

Pop event off heap
Sleep until wait time is reached
Send SimulationEvent::SendPayment into the channel for the executor
Generate a new wait time and destination from the ExecutorKit
Read from heap until shutdown

Instinct about this is:

Always generating payment destination in one places fixes our determinism issue
Re-queueing on pop + sorting by time means we'll never run out of events for each executor
The nasty thing will be payment counts, we're probably going to have to store those in the heap and know when we don't need to queue anything else

f3r10 · 2025-06-24T14:56:01Z

hi @carlaKC I think that now it is working as expected 💪

carlaKC

Some relative timestamp issues that we need to address - didn't occur to me earlier in design phase.

I noticed this when testing out against a toy network, would be good to have some unit tests to assert that we get the payment sequence that we're expecting (should be reasonable to do with defined activities with set wait times / counts).

carlaKC · 2025-06-25T20:09:28Z

simln-lib/src/lib.rs

+                                    log::info!(
+                                        "Payment count has been met for {}: {c} payments. Stopping the activity."
+                                    , executor.source_info);
+                                    return Ok(());


Shouldn't we continue here? If one activity hits its payment_count, it doesn't mean that we're finished with all other payments?

carlaKC · 2025-06-25T20:12:15Z

simln-lib/src/lib.rs

    payment_generator: Box<dyn PaymentGenerator>,
 }

+struct PaymentEvent {
+    wait_time: Duration,


We're going to have to use an absolute time here - my bad, should have realized that earlier in the design process.

Since we process and sleep all in one queue, a relative value doesn't work because we're always starting from zero when we pop an item off.

For example: say we have two activities, one executes every 5 seconds, the other every 10.

We push two items to the heap, one in 5 seconds one in 10 seconds

We sleep 5 seconds, then pop the 5 second event and fire it

We re-generate another event which should sleep 5 seconds

We'll push that onto the heap, and it's the next soonest event

We then sleep another 5 seconds, then pop the next 5 second event

We'll continue like this forever, never hitting the 10 second event. If we have absolute times, that won't happen because the 10 second event will eventually bubble up.

Oh!! right good catch

Looking at the latest version - isn't relative time still an issue here?

No, really. In the latest version, absolute_time is used for heap ordering, whereas wait_time is only for sleeping.

I added two unit tests, one for the defined activities with a defined payment count and the other for testing the random Rng, and the behaviour is as expected 🤔

No, really. In the latest version, absolute_time is used for heap ordering, whereas wait_time is only for sleeping.

Seems like unnecessary complexity to duplicate these values when we could just track absolute_time and sleep for abs time - now?

But it would also be necessary to track the now time when the next payment was generated, otherwise, in each tick of the loop, the now time changes, and the difference between the abs time and now is not the same as the expected wait_time 🤔

carlaKC · 2025-06-25T20:13:14Z

simln-lib/src/lib.rs

@@ -591,7 +592,7 @@ pub struct Simulation<C: Clock + 'static> {
    /// Config for the simulation itself.
    cfg: SimulationCfg,
    /// The lightning node that is being simulated.
-    nodes: HashMap<PublicKey, Arc<Mutex<dyn LightningNode>>>,
+    nodes: BTreeMap<PublicKey, Arc<Mutex<dyn LightningNode>>>,


Why do we need to change this to a BTreeMap?

Because the iteration of the values of a HashMap is not deterministic.
For example, if there are 5 nodes N1, N2, N3, N4, N5
In one run, the iteration could be:

N5 -> N1 / 1sat

N3 -> N2 / 1sat

N1 -> N5 / 10 sat

On a second run, could it be
N1 -> N1 / 1 sat - this is not valid, and it is necessary to choose another destination
N1 -> N2 / 1 sat
N5 -> N5 / 10 sat - also not valid ....
and so on

Right of course - missed that we're iterating through these to create the initial set of executors, which will determine the order that we're creating that initial set of events.

WDYT about leaving this as a hashmap and doing a once-off sort of the executors by pubkey in dispatch_producers? It feels like a real gotcha that the way we store our nodes could break our determinism in dispatch_producers, and this would make it more contained.

I think that will also be cheaper than a btree because we get O(1) lookups in the hashmap, but I haven't formally checked this.

Ok, I am going to leave as a hashmap 👍

you are right, hashmap lookups are O(1)

carlaKC · 2025-06-25T20:13:49Z

simln-lib/src/lib.rs

+                                    } else {
+                                        generate_payments(&mut heap, executor, current_count + 1)?;
+                                    }


nit: don't need the else branch if we're continuing

carlaKC · 2025-06-25T20:15:08Z

simln-lib/src/lib.rs

@@ -1561,6 +1610,31 @@ async fn track_payment_result(
    Ok(())
 }

+fn generate_payments(


nit: generate_payment? we're only adding one

carlaKC · 2025-06-25T20:18:37Z

simln-lib/src/lib.rs

+                            // Wait until our time to next payment has elapsed then execute a random amount payment to a random
+                            // destination.
+                            pe_clock.sleep(wait_time).await;
+                            t.spawn(async move {


Updated opinion: I think that we can kill produce_events completely and do the following here:

Spawn a task that directly passes SimulationEvent to consume_events

Remove the loop from consume_events and just handle dispatch of the LN event.

This cuts a lot of steps out, and puts all our event handling solidly in one place. If the LN nodes take long, at least they're spawned in a task (nothing we can do about that ordering).

I'm on the fence about whether we need to spawn this in a task. Technically consume_events should pull the event pretty quickly (at least, in the amount of time that it takes to make a RPC call to the LN node).

The channel is buffered - so the question is:
Will we queue two events for a single lightning node faster than we can dispatch a single payment? If yes, then we'll block, if no then we don't need the task.

Without the loop, the consumer channel is closed, and it is not possible to send more consume_events 🤔

Can't we also kill the channel on consume_events completely? And just directly pass it a SimulationEvent that it dispatches to the node? If we do that in a task, it's okay if the event takes a while to dispatch.

yes, it was possible to remove consume_event completely 🚀

carlaKC · 2025-06-26T16:37:02Z

simln-lib/src/lib.rs

-
-            tasks.spawn(async move {
-                let source = executor.source_info.clone();
+            generate_payments(&mut heap, executor, 0)?;


Looking at this again - I think we can actually just store the payment event in the heap and keep a hashmap of executors / payment counts. When we pop an event off, we just look up the stuff we need in the hashmap and increment the count there, rather than clagging up the hashmap with large structs.

f3r10 · 2025-07-03T20:19:50Z

Still thinking about how to add tests to assert that we get the payment sequence that we're expecting 🤔
Right now, there is only one test checking the number of payments and the expected elapsed time

carlaKC · 2025-07-07T19:47:43Z

simln-lib/src/lib.rs

+                                if c == payload.current_count {
+                                    log::info!(
+                                        "Payment count has been met for {}: {c} payments. Stopping the activity."
+                                    , executor.source_info);


nit: just to be aware of, cargo fmt breaks in selects for some ungodly reason, so it's worth trying to do a little manual formatting

carlaKC · 2025-07-07T19:53:40Z

simln-lib/src/lib.rs

+
+                            // Wait until our time to next payment has elapsed then execute a random amount payment to a random
+                            // destination.
+                            pe_clock.sleep(payload.wait_time).await;


Still need to select on shutdown?

yes, otherwise the task that is processing the payment's heap does not stop.

Indeed - my comment is to say that we should be select + listening on a shutdown trigger here?

yes you are right, it is necessary to add a select in this part

simln-lib/refactor: fully deterministic produce events

carlaKC

I'm finding the design decisions in this PR quite difficult to follow. To me the obvious approach is to queue cheap-to-copy payment events and track node stats/executors in one big hashmap. Happy to hear arguments for this way, but currently not convinced.

carlaKC · 2025-07-10T18:40:14Z

simln-lib/src/lib.rs

+                            let payment_amount = executor.payment_generator.payment_amount(payload.capacity);
+                            let amount = match payment_amount {
+                                Ok(amt) => {
+                                    if amt == 0 {
+                                        log::debug!(
+                                            "Skipping zero amount payment for {source} -> {}.", payload.destination
+                                        );
+                                        generate_payment(&mut heap, executor, payload.current_count, payment_event_payloads.clone()).await?;
+                                        continue;
                                    }
-                                }
+                                    amt
+                                },
+                                Err(e) => {
+                                    return Err(SimulationError::PaymentGenerationError(e));
+                                },


let payment_amount = executor .payment_generator .payment_amount(payload.capacity) .map_err(SimulationError::PaymentGenerationError)?; generate_payment(&mut heap, executor, payload.current_count + 1, payment_event_payloads.clone()).await?; if payment_amount == 0 { log::debug!( "Skipping zero amount payment for {source} -> {}.", payload.destination, ); continue; }

Somehow with this change, rust is not able to infer the Err type and force me to add return Ok::<(), SimulationError>(()) 🤔

carlaKC · 2025-07-10T18:53:13Z

simln-lib/src/lib.rs

+        let payment_event_payloads: Arc<Mutex<HashMap<PublicKey, PaymentEventPayload>>> =
+            Arc::new(Mutex::new(HashMap::new()));


This does not need Arc/Mutex - we can just pass a mutable reference to generate payment.

carlaKC · 2025-07-10T18:55:15Z

simln-lib/src/lib.rs

-
-            tasks.spawn(async move {
-                let source = executor.source_info.clone();
+        let mut heap: BinaryHeap<Reverse<PaymentEvent>> = BinaryHeap::new();


This seems backwards to me?

The heap should contain all of the details for the payment (destination/amount/wait time) and the hashmap should contain the executor and count that we track once per source node?

It really doesn't make sense to me to keep pushing executors to the heap and tracking a map of next payment details (especially the way that it's done here, where we have to set values and then overwrite them later).

I was picturing we would have SimulationEvent::SendPayment + a source node pubkey on the heap, and then we use the pubkey to look up in a map of pubkey -> {executor / count } to grab what we need / mutate count.

carlaKC · 2025-07-10T18:56:03Z

simln-lib/src/lib.rs

+
+                            // Wait until our time to next payment has elapsed then execute a random amount payment to a random
+                            // destination.
+                            pe_clock.sleep(payload.wait_time).await;


Indeed - my comment is to say that we should be select + listening on a shutdown trigger here?

carlaKC · 2025-07-10T18:58:52Z

simln-lib/src/lib.rs

+/// event being executed is piped into a channel to handle the result of the event.
+async fn send_payment(
+    node: Arc<Mutex<dyn LightningNode>>,
+    // mut receiver: Receiver<SimulationEvent>,


Commented out receiver can be removed? Likewise for the loop comment below?

carlaKC · 2025-07-10T19:05:39Z

simln-lib/src/lib.rs

    payment_generator: Box<dyn PaymentGenerator>,
 }

+struct PaymentEvent {
+    wait_time: Duration,


No, really. In the latest version, absolute_time is used for heap ordering, whereas wait_time is only for sleeping.

Seems like unnecessary complexity to duplicate these values when we could just track absolute_time and sleep for abs time - now?

carlaKC · 2025-07-10T19:10:05Z

simln-lib/src/lib.rs

+        for executor in executors {
+            let payload = PaymentEventPayload {
+                wait_time: Duration::from_secs(0),
+                destination: executor.source_info.clone(),


Setting a field called destination to the source value and then overwriting it later is some serious code smell to me.

f3r10 · 2025-07-14T11:26:47Z

I'm finding the design decisions in this PR quite difficult to follow. To me the obvious approach is to queue cheap-to-copy payment events and track node stats/executors in one big hashmap. Happy to hear arguments for this way, but currently not convinced.

Thanks for the feedback, using one big hashmap for the executors is a much better design.

carlaKC reviewed Jun 9, 2025

View reviewed changes

f3r10 force-pushed the refactor_fully_deterministic_produce_events branch from b06a289 to 1b3a21f Compare June 10, 2025 20:31

f3r10 force-pushed the refactor_fully_deterministic_produce_events branch from a99bbff to 2beccfa Compare June 23, 2025 20:47

f3r10 marked this pull request as ready for review June 24, 2025 14:56

f3r10 requested a review from carlaKC June 24, 2025 14:56

carlaKC reviewed Jun 25, 2025

View reviewed changes

carlaKC reviewed Jun 26, 2025

View reviewed changes

carlaKC mentioned this pull request Jun 26, 2025

Refactoring the shutdown process to fix a payment count bug #235

Open

f3r10 force-pushed the refactor_fully_deterministic_produce_events branch from 6d7aa41 to aea34ab Compare July 3, 2025 16:49

f3r10 requested a review from carlaKC July 7, 2025 14:48

carlaKC reviewed Jul 7, 2025

View reviewed changes

f3r10 requested a review from carlaKC July 8, 2025 17:42

simln-lib/refactor: fully deterministic produce events

19581f3

simln-lib/refactor: fully deterministic produce events

f3r10 force-pushed the refactor_fully_deterministic_produce_events branch from 72fb38a to 3e21411 Compare July 8, 2025 17:51

f3r10 added 7 commits July 8, 2025 13:06

fixup! 19581f3

8935101

fixup! 19581f3

f0d368a

fixup! 19581f3

98f8358

fixup! 19581f3

3a1dc27

fixup! 19581f3

676ec57

fixup! 19581f3

8f8c775

fixup! 19581f3

fce8c64

f3r10 added 2 commits July 8, 2025 13:06

fixup! 19581f3

60296cb

fixup! 19581f3

3f23420

f3r10 force-pushed the refactor_fully_deterministic_produce_events branch from 4b07890 to 3f23420 Compare July 8, 2025 18:07

carlaKC reviewed Jul 10, 2025

View reviewed changes

f3r10 added 2 commits July 14, 2025 05:50

fixup! 19581f3

0e4b5c7

fixup! 19581f3

2300b87

f3r10 requested a review from carlaKC July 14, 2025 12:13

fixup! 19581f3

b3cb21d

		let payment_event_payloads: Arc<Mutex<HashMap<PublicKey, PaymentEventPayload>>> =
		Arc::new(Mutex::new(HashMap::new()));

simln-lib/refactor: fully deterministic produce events #277

Are you sure you want to change the base?

simln-lib/refactor: fully deterministic produce events #277

Conversation

f3r10 commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Uh oh!

f3r10 commented Jun 4, 2025

Uh oh!

carlaKC commented Jun 9, 2025

Uh oh!

carlaKC left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

f3r10 commented Jun 10, 2025

Uh oh!

f3r10 commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The Determinism Challenge

Proposed Solution for Execution Order

Uh oh!

carlaKC commented Jun 17, 2025

Uh oh!

carlaKC commented Jun 17, 2025

Uh oh!

f3r10 commented Jun 24, 2025

Uh oh!

carlaKC left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

f3r10 Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlaKC Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

f3r10 commented Jul 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

f3r10 commented Jun 3, 2025 •

edited

Loading

carlaKC left a comment •

edited

Loading

f3r10 commented Jun 13, 2025 •

edited

Loading

f3r10 Jul 7, 2025 •

edited

Loading

carlaKC Jun 25, 2025 •

edited

Loading

f3r10 Jul 8, 2025 •

edited

Loading