From 5053b8ab714d56478321b0a7884a3a7cdfefdce2 Mon Sep 17 00:00:00 2001 From: Wenqi Mou Date: Thu, 24 Apr 2025 14:30:48 -0400 Subject: [PATCH 1/7] br: disable restore checksum Signed-off-by: Wenqi Mou --- br/br-pitr-manual.md | 3 ++- br/br-snapshot-manual.md | 47 ++++++++++++++++++++++++++++++++-- br/use-br-command-line-tool.md | 1 + 3 files changed, 48 insertions(+), 3 deletions(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index b3f007c8976a1..20cdef6bee246 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -409,7 +409,8 @@ Expected output: > **Note:** > -> If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. +> - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. +> - For checksum configuration, please refer to [Checksum](/br/br-snapshot-manual.md#checksum). You can run the `tiup br restore point` command to perform a PITR on a new cluster or just restore the log backup data. diff --git a/br/br-snapshot-manual.md b/br/br-snapshot-manual.md index aeccb16e55a6c..89dd981c93098 100644 --- a/br/br-snapshot-manual.md +++ b/br/br-snapshot-manual.md @@ -21,6 +21,7 @@ This document describes the commands of TiDB snapshot backup and restore accordi - [Restore multiple tables with table filter](#restore-multiple-tables-with-table-filter) - [Restore execution plan bindings from the `mysql` schema](#restore-execution-plan-bindings-from-the-mysql-schema) - [Restore encrypted snapshots](#restore-encrypted-snapshots) +- [Checksum](#checksum) For more information about snapshot backup and restore, refer to: @@ -48,7 +49,6 @@ In the preceding command: > **Note:** > -> - Starting from v8.5.0, the BR tool disables the table-level checksum calculation during full backups by default (`--checksum=false`) to improve backup performance. > - The BR tool already supports self-adapting to GC. It automatically registers `backupTS` (the latest PD timestamp by default) to PD's `safePoint` to ensure that TiDB's GC Safe Point does not move forward during the backup, thus avoiding manually setting GC configurations. During backup, a progress bar is displayed in the terminal, as shown below. When the progress bar advances to 100%, the backup is complete. @@ -178,7 +178,7 @@ In the preceding command: - `--ratelimit`: The maximum speed **per TiKV** performing restore tasks. The unit is in MiB/s. - `--log-file`: The target file where the `br` log is written. -During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. Then `br` will verify the restored data to ensure data security. +During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool will perform table data verification to ensure the logical integrity of the data. File-level checksums are always performed, ensuring the basic integrity of the restored files. ```shell Split&Scatter Region <--------------------------------------------------------------------> 100.00% @@ -287,3 +287,46 @@ tiup br restore full\ --crypter.method aes128-ctr \ --crypter.key 0123456789abcdef0123456789abcdef ``` + +## Checksum + +Checksum is a method used by the BR tool to verify the integrity of backup and restore data. BR supports two levels of checksums: + +1. **File-level checksum**: Verifies the backup files themselves to ensure integrity during storage and transmission. This checksum is always enabled and cannot be disabled. +2. **Table-level checksum**: Verifies the integrity of table data content and confirms the business logic consistency of the data. This checksum is disabled by default but can be enabled through parameters. + +Balancing performance and security considerations, BR handles table-level checksums as follows: + +### Backup Checksum + +Starting from v8.5.0, when performing full backups, the BR tool does not calculate table-level checksums by default (`--checksum=false`) to improve backup performance. If you need to calculate table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums will always be calculated to ensure the integrity of backup files. + +Calculating table-level checksums can verify data integrity during backup but increases backup time. In most cases, it's safe to use the default setting (no table-level checksum) to improve backup speed. + +### Restore Checksum + +Starting from v9.5.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. + +After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. + +### Checksum Configuration Examples + +Enable table-level checksums during backup: + +```shell +tiup br backup full \ + --pd "${PD_IP}:2379" \ + --storage "s3://${backup_collection_addr}/snapshot-${date}?access-key=${access-key}&secret-access-key=${secret-access-key}" \ + --checksum=true \ + --log-file backupfull.log +``` + +Enable table-level checksums during restore: + +```shell +tiup br restore full \ + --pd "${PD_IP}:2379" \ + --storage "s3://${backup_collection_addr}/snapshot-${date}?access-key=${access-key}&secret-access-key=${secret-access-key}" \ + --checksum=true \ + --log-file restorefull.log +``` diff --git a/br/use-br-command-line-tool.md b/br/use-br-command-line-tool.md index a92f39e135a2a..e8efbe4b0a8ef 100644 --- a/br/use-br-command-line-tool.md +++ b/br/use-br-command-line-tool.md @@ -62,6 +62,7 @@ A `tiup br` command consists of multiple layers of sub-commands. Currently, br c * `--tikv-max-restore-concurrency`: the maximum number of concurrent tasks per TiKV node during snapshot restore. * `--compression`: determines the compression algorithm used for generating backup files. It supports `lz4`, `snappy`, and `zstd`, with the default being `zstd` (usually no need to modify). For guidance on choosing different compression algorithms, refer to [this document](https://github.com/EighteenZi/rocksdb_wiki/blob/master/Compression.md). * `--compression-level`: sets the compression level corresponding to the chosen compression algorithm for backup. The default compression level for `zstd` is 3. In most cases there is no need to set this option. +* `--checksum`: controls whether to perform table-level checksum verification during backup and restore. Default is `false`. For more details, refer to [Checksum](/br/br-snapshot-manual.md#checksum). ## Commands of full backup From 83a66b49cb377256f3140074ffffceb336a833b3 Mon Sep 17 00:00:00 2001 From: Wenqi Mou Date: Thu, 24 Apr 2025 14:33:57 -0400 Subject: [PATCH 2/7] Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- br/br-pitr-manual.md | 2 +- br/br-snapshot-manual.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index 20cdef6bee246..164bb60ebf695 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -410,7 +410,7 @@ Expected output: > **Note:** > > - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. -> - For checksum configuration, please refer to [Checksum](/br/br-snapshot-manual.md#checksum). +> - For checksum configuration, refer to [Checksum](#checksum) for details on verifying data integrity during backup and restore. You can run the `tiup br restore point` command to perform a PITR on a new cluster or just restore the log backup data. diff --git a/br/br-snapshot-manual.md b/br/br-snapshot-manual.md index 89dd981c93098..38e48c078b54c 100644 --- a/br/br-snapshot-manual.md +++ b/br/br-snapshot-manual.md @@ -178,7 +178,7 @@ In the preceding command: - `--ratelimit`: The maximum speed **per TiKV** performing restore tasks. The unit is in MiB/s. - `--log-file`: The target file where the `br` log is written. -During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool will perform table data verification to ensure the logical integrity of the data. File-level checksums are always performed, ensuring the basic integrity of the restored files. +During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool performs table data verification to ensure the logical integrity of the data. File-level checksums are always performed to ensure the basic integrity of the restored files. ```shell Split&Scatter Region <--------------------------------------------------------------------> 100.00% From 2de57b5409ea09d5bc044f009b9375db1ad6821f Mon Sep 17 00:00:00 2001 From: BornChanger <97348524+BornChanger@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:26:56 +0800 Subject: [PATCH 3/7] Update br/br-snapshot-manual.md --- br/br-snapshot-manual.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-snapshot-manual.md b/br/br-snapshot-manual.md index 38e48c078b54c..03d4bae60838f 100644 --- a/br/br-snapshot-manual.md +++ b/br/br-snapshot-manual.md @@ -305,7 +305,7 @@ Calculating table-level checksums can verify data integrity during backup but in ### Restore Checksum -Starting from v9.5.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. +Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. From fe65735fb424cd2fc4caff965f9471b26f8fc200 Mon Sep 17 00:00:00 2001 From: Wenqi Mou Date: Mon, 2 Jun 2025 17:51:52 -0400 Subject: [PATCH 4/7] Update br-pitr-manual.md --- br/br-pitr-manual.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index 164bb60ebf695..9e1090b31541e 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -410,7 +410,7 @@ Expected output: > **Note:** > > - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. -> - For checksum configuration, refer to [Checksum](#checksum) for details on verifying data integrity during backup and restore. +> - For checksum configuration, refer to [Checksum](/br/br-snapshot-manual.md##checksum) for details on verifying data integrity during backup and restore. You can run the `tiup br restore point` command to perform a PITR on a new cluster or just restore the log backup data. @@ -517,4 +517,4 @@ If any of the above conditions are not met, or if you need to perform a point-in > **Note:** > -> When restoring a log backup that contains records of snapshot (full) restore data, you must use BR v9.0.0 or later. Otherwise, restoring the recorded full restore data might fail. \ No newline at end of file +> When restoring a log backup that contains records of snapshot (full) restore data, you must use BR v9.0.0 or later. Otherwise, restoring the recorded full restore data might fail. From 0b09c36f1d05f40aa0c7e21530ce6277b1b51768 Mon Sep 17 00:00:00 2001 From: Wenqi Mou Date: Mon, 2 Jun 2025 17:52:45 -0400 Subject: [PATCH 5/7] Update br-pitr-manual.md --- br/br-pitr-manual.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index 9e1090b31541e..9140e107047bb 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -410,7 +410,7 @@ Expected output: > **Note:** > > - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. -> - For checksum configuration, refer to [Checksum](/br/br-snapshot-manual.md##checksum) for details on verifying data integrity during backup and restore. +> - For checksum configuration, refer to [Checksum](/br/br-snapshot-manual.md#checksum) for details on verifying data integrity during backup and restore. You can run the `tiup br restore point` command to perform a PITR on a new cluster or just restore the log backup data. From 36193c235d4bab1e3af598bf129399c4118e59ed Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 1 Jul 2025 15:19:05 +0800 Subject: [PATCH 6/7] Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- br/br-snapshot-manual.md | 8 ++++---- br/use-br-command-line-tool.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/br/br-snapshot-manual.md b/br/br-snapshot-manual.md index 03d4bae60838f..ede6318a2bc11 100644 --- a/br/br-snapshot-manual.md +++ b/br/br-snapshot-manual.md @@ -49,7 +49,7 @@ In the preceding command: > **Note:** > -> - The BR tool already supports self-adapting to GC. It automatically registers `backupTS` (the latest PD timestamp by default) to PD's `safePoint` to ensure that TiDB's GC Safe Point does not move forward during the backup, thus avoiding manually setting GC configurations. +> The BR tool already supports self-adapting to GC. It automatically registers `backupTS` (the latest PD timestamp by default) to PD's `safePoint` to ensure that TiDB's GC Safe Point does not move forward during the backup, thus avoiding manually setting GC configurations. During backup, a progress bar is displayed in the terminal, as shown below. When the progress bar advances to 100%, the backup is complete. @@ -297,19 +297,19 @@ Checksum is a method used by the BR tool to verify the integrity of backup and r Balancing performance and security considerations, BR handles table-level checksums as follows: -### Backup Checksum +### Backup checksum Starting from v8.5.0, when performing full backups, the BR tool does not calculate table-level checksums by default (`--checksum=false`) to improve backup performance. If you need to calculate table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums will always be calculated to ensure the integrity of backup files. Calculating table-level checksums can verify data integrity during backup but increases backup time. In most cases, it's safe to use the default setting (no table-level checksum) to improve backup speed. -### Restore Checksum +### Restore checksum Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. -### Checksum Configuration Examples +### Checksum configuration examples Enable table-level checksums during backup: diff --git a/br/use-br-command-line-tool.md b/br/use-br-command-line-tool.md index e8efbe4b0a8ef..72fcd0d20a63c 100644 --- a/br/use-br-command-line-tool.md +++ b/br/use-br-command-line-tool.md @@ -62,7 +62,7 @@ A `tiup br` command consists of multiple layers of sub-commands. Currently, br c * `--tikv-max-restore-concurrency`: the maximum number of concurrent tasks per TiKV node during snapshot restore. * `--compression`: determines the compression algorithm used for generating backup files. It supports `lz4`, `snappy`, and `zstd`, with the default being `zstd` (usually no need to modify). For guidance on choosing different compression algorithms, refer to [this document](https://github.com/EighteenZi/rocksdb_wiki/blob/master/Compression.md). * `--compression-level`: sets the compression level corresponding to the chosen compression algorithm for backup. The default compression level for `zstd` is 3. In most cases there is no need to set this option. -* `--checksum`: controls whether to perform table-level checksum verification during backup and restore. Default is `false`. For more details, refer to [Checksum](/br/br-snapshot-manual.md#checksum). +* `--checksum`: controls whether to perform table-level checksum verification during backup and restore. The default value is `false`, meaning that checksum verification is not performed unless explicitly enabled. For more details, refer to [Checksum](/br/br-snapshot-manual.md#checksum). ## Commands of full backup From dc8f201fb2fe9063938fc04ad9d15e8927ad8c31 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 3 Jul 2025 09:31:46 +0800 Subject: [PATCH 7/7] Apply suggestions from code review --- br/br-pitr-manual.md | 2 +- br/br-snapshot-manual.md | 8 ++++---- br/use-br-command-line-tool.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/br/br-pitr-manual.md b/br/br-pitr-manual.md index 9140e107047bb..f0d504f303d53 100644 --- a/br/br-pitr-manual.md +++ b/br/br-pitr-manual.md @@ -410,7 +410,7 @@ Expected output: > **Note:** > > - If you specify `--full-backup-storage` as the incremental backup address for `restore point`, for restores of this backup and any previous incremental backups, you need to set the parameter `--allow-pitr-from-incremental` to `true` to make the incremental backups compatible with the subsequent log backups. -> - For checksum configuration, refer to [Checksum](/br/br-snapshot-manual.md#checksum) for details on verifying data integrity during backup and restore. +> - For information about checksum configuration, see [Checksum](/br/br-snapshot-manual.md#checksum). You can run the `tiup br restore point` command to perform a PITR on a new cluster or just restore the log backup data. diff --git a/br/br-snapshot-manual.md b/br/br-snapshot-manual.md index ede6318a2bc11..c24dd06dadb0d 100644 --- a/br/br-snapshot-manual.md +++ b/br/br-snapshot-manual.md @@ -178,7 +178,7 @@ In the preceding command: - `--ratelimit`: The maximum speed **per TiKV** performing restore tasks. The unit is in MiB/s. - `--log-file`: The target file where the `br` log is written. -During restore, a progress bar is displayed in the terminal as shown below. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level checksum is enabled (see [Checksum](#checksum)), the BR tool performs table data verification to ensure the logical integrity of the data. File-level checksums are always performed to ensure the basic integrity of the restored files. +During restore, a progress bar is displayed in the terminal as shown in the following example. When the progress bar advances to 100%, the restore task is completed. After the restoration is complete, if table-level [checksum](#checksum) is enabled, the BR tool performs table data verification to ensure the logical integrity of the data. Note that file-level checksums are always performed to ensure the basic integrity of the restored files. ```shell Split&Scatter Region <--------------------------------------------------------------------> 100.00% @@ -299,15 +299,15 @@ Balancing performance and security considerations, BR handles table-level checks ### Backup checksum -Starting from v8.5.0, when performing full backups, the BR tool does not calculate table-level checksums by default (`--checksum=false`) to improve backup performance. If you need to calculate table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums will always be calculated to ensure the integrity of backup files. +Starting from v8.5.0, when performing full backups, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default to improve backup performance. If you need to perform table-level checksums during backup, you can explicitly specify `--checksum=true`. File-level checksums are always performed to ensure the integrity of backup files. -Calculating table-level checksums can verify data integrity during backup but increases backup time. In most cases, it's safe to use the default setting (no table-level checksum) to improve backup speed. +Performing table-level checksums can verify data integrity during backup but increases backup time. In most cases, it is safe to use the default setting (that is, table-level checksum is disabled) to improve backup speed. ### Restore checksum Starting from v9.0.0, the BR tool does not perform table-level checksum verification (`--checksum=false`) by default during restore operations to improve restore performance. If you need to perform table-level checksum verification, you can explicitly specify `--checksum=true`. File-level checksum verification is always performed to ensure the basic integrity of restored data. -After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you may choose to enable table-level checksums. +After restoration, data validation is usually performed to ensure data security. When table-level checksums are disabled, the comprehensive validation step for table data is skipped, thereby accelerating the restore process. For scenarios with strict data integrity requirements, you can enable table-level checksums. ### Checksum configuration examples diff --git a/br/use-br-command-line-tool.md b/br/use-br-command-line-tool.md index 72fcd0d20a63c..a39d587d3feaf 100644 --- a/br/use-br-command-line-tool.md +++ b/br/use-br-command-line-tool.md @@ -62,7 +62,7 @@ A `tiup br` command consists of multiple layers of sub-commands. Currently, br c * `--tikv-max-restore-concurrency`: the maximum number of concurrent tasks per TiKV node during snapshot restore. * `--compression`: determines the compression algorithm used for generating backup files. It supports `lz4`, `snappy`, and `zstd`, with the default being `zstd` (usually no need to modify). For guidance on choosing different compression algorithms, refer to [this document](https://github.com/EighteenZi/rocksdb_wiki/blob/master/Compression.md). * `--compression-level`: sets the compression level corresponding to the chosen compression algorithm for backup. The default compression level for `zstd` is 3. In most cases there is no need to set this option. -* `--checksum`: controls whether to perform table-level checksum verification during backup and restore. The default value is `false`, meaning that checksum verification is not performed unless explicitly enabled. For more details, refer to [Checksum](/br/br-snapshot-manual.md#checksum). +* `--checksum`: controls whether to perform table-level checksum verification during backup and restore. The default value is `false`, meaning that checksum verification is not performed. For more details, see [Checksum](/br/br-snapshot-manual.md#checksum). ## Commands of full backup