Skip to content

log backup: update metafile format #20455

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions br/br-log-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ PITR 的流程如下:
├── v1
│   ├── backupmeta
│   │   ├── ...
│   │   └── {resolved_ts}-{uuid}.meta
│   │   └── {flushTs}-{minDefaultTs}-{minTs}-{maxTs}.meta
│   ├── global_checkpoint
│   │   └── {store_id}.ts
│   └── {date}
Expand All @@ -112,7 +112,14 @@ PITR 的流程如下:

备份文件目录结构的说明如下:

- `backupmeta`:备份的元数据。文件名中的 `resolved_ts` 指备份的进度,表示该 TSO 之前的数据已被完整备份。但是需注意,该 TSO 仅反映部分分片的进度。
- `backupmeta` 目录:存储备份的元数据文件。从 v8.5.3 和 v9.0.0 起,该文件命名格式从 `{resolved_ts}-{uuid}.meta` 修改为 `{flushTs}-{minDefaultTs}-{minTs}-{maxTs}.meta`。文件名中包含以下时间戳字段:

- `flushTs`:该备份文件被定期上传至外部存储的时间戳。该值从 PD 获取,具有全局唯一性。
- `minDefaultTs`:仅针对 Write CF 文件,表示该备份所覆盖的最早事务起始时间。
- `minTs` 和 `maxTs`:该备份文件内包含的所有 key-value 数据的最小时间戳和最大时间戳。

这些时间戳均采用 16 位固定长度的十六进制字符串编码,左侧补零以保证长度一致。这样的编码设计可以确保文件名按照字典序自然排序,方便在外部存储系统中高效执行批量列举和范围过滤操作。

- `global_checkpoint`:备份的全局进度。它记录了可以被 `br restore point` 恢复到的最晚时间点。
- `{date}/{hour}`:对应日期和小时的备份数据。注意在清理存储的时候,需使用 `br log truncate`,不能手动删除数据。这是因为 metadata 会指向这里的数据,手动删除它们会导致恢复失败或恢复后数据不一致等问题。

Expand All @@ -122,9 +129,9 @@ PITR 的流程如下:
├── v1
│   ├── backupmeta
│   │   ├── ...
│   │   ├── 435213818858112001-e2569bda-a75a-4411-88de-f469b49d6256.meta
│   │   ├── 435214043785779202-1780f291-3b8a-455e-a31d-8a1302c43ead.meta
│   │   └── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta
│   │   ├── 060c4bc7b0cdd582-06097a780d1ba138-060ab960016d2f00-060c0b9e47d4787b.meta
│   │   ├── 06123bc6a0cdd591-060c3d24585be000-060c4453954a4000-060c4bc7b0cdcfa4.meta
│   │   └── 063c2ac1c0cdd5c3-0609d2e6b3bcb064-060ab960016d2f84-060c0b9e47d47a77.meta
│   ├── global_checkpoint
│   │   ├── 1.ts
│   │   ├── 2.ts
Expand Down