feat(storage): caios-backed local-ssd directories solution #16
Conversation
| - This sample is only backing up a `/workdir` directory now, but | ||
| similar techniques can be applied to back up other directories as needed. | ||
| - The initial copy is configured assuming a large numbers of small files | ||
| (--transfers 256). |
There was a problem hiding this comment.
I'm curious if it would be faster to tar/archive everything before starting the upload. Have you tried benchmarking it as well?
There was a problem hiding this comment.
Great point. I wasn't compressing things because the sync upstream job (backup container) runs every 60 seconds and it's better to do incremental in that case. That said, a listed incremental TAR should work on changes only, so it's a good idea. I should test that next (especially for speeding up the restoring process). I'll try to include that before merging here or if that doesn't make it in time, I'll add it in an update later (thank you!).
I haven't benchmarked this thoroughly yet, but when initial testing showed that restoring from CAIOS was faster than git cloning, I decided to create the draft.
New solution added to storage folder: emptyDir directories (backed by local SSD) that are backed asynchronously to a CAIOS bucket. See README.md for details.