-
Notifications
You must be signed in to change notification settings - Fork 6
Description
In long-running experiments we often require keeping track of data or variables across runs of the same workflow. In the past this was done by using StateRecoverySubject operators which implicitly save their current state on subject disposal.
This has created confusion in the past leading to proposals for making updates more explicitly as in #213. The implicit operation also extends to file names which further restricts the flexibility of these subjects as raised in #225.
Perhaps a preferred approach is to implement a persistent data store with an explicit file name, and where values in the store are updated explicitly by input data elements arriving to the operator. On subscription, the current value in the store is sent as the first item. Approximate sequence would thus be:
- On subscribe, if there is an existing file store, read it and emit the element; otherwise, emit nothing.
- On each element of the source sequence, create or update the JSON file store, and propagate the element forward.
The store should be written "atomically" (open, write, close) on each update. Consider using File.Replace for this, as it can be considered an atomic operation on Windows NTFS.