snapshot replication system #45
Description
Possibly makes #8 and #9 unneeded.
A more efficient replication system that only tracks page indexes rather than page contents in the log. The insight is that clients only need to fast forward to the latest snapshot. This can be computed by comparing the client and server versions and unioning all page changes in-between them. Then just serve pages from the current snapshot.
Note, that if concurrent writes are desired in this model, this does require some kind of actual snapshot functionality on the send side. To this extent, I've been researching copy-on-write systems to understand various ways to build this in a performant way.
Some rambling thoughts:
- use the sqlite wal or build a similar wal of my own to buffer writes
- consider that multiple clients can be requesting changes starting from different server versions, thus each needs their own changeset
- although, one observation is that if you ignore truncation, changesets always either get smaller or stay the same size, they never grow
- perhaps this means you could just build one snapshot from the oldest known client version and then send it out to all the clients with a tiny bit of filtering for more up to date clients
- although, it's also desirable to stream out snapshots rather than building them in memory
- currently it's nice that both the mutation timeline and the storage log share the same journal and replication abstractions, but this may be preventing me from doing things like easily truncating the timeline
- need to be careful to maintain recovery after failure, currently since all writes go into a pending SparsePages object, any failure will result in the database safely rolling back to the last saved snapshot