GitHub/school-of-sre
1
0
Fork 0
mirror of https://github.com/linkedin/school-of-sre synced 2026-01-19 23:18:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs (level 101): fix typos, punctuation, formatting (#160)

Browse Source 
* docs: formatted for readability

* docs: rephrased and added punctuation

* docs: fix typos, punctuation, formatting

* docs: fix typo and format

* docs: fix caps and formatting

* docs: fix punctuation and formatting

* docs: capitalized SQL commands, fixed puntuation, formatting

* docs: fix punctuation

* docs: fix punctuation and formatting

* docs: fix caps,punctuation and formatting

* docs: fix links, punctuation, formatting

* docs: fix code block formatting

* docs: fix punctuation, indentation and formatting
This commit is contained in:
Jana R
2024-07-28 17:38:19 +05:30
committed by GitHub
parent bdcc6856ed
commit 4239ecf473
58 changed files with 1522 additions and 1367 deletions
Download Patch File Download Diff File Expand all files Collapse all files

200
courses/level101/databases_sql/backup_recovery.md
View File

@@ -1,5 +1,5 @@
### Backup and Recovery
Backups are a very crucial part of any database setup. They are generally a copy of the data that can be used to reconstruct the data in case of any major or minor crisis with the database. In general terms backups can be of two types:-
Backups are a very crucial part of any database setup. They are generally a copy of the data that can be used to reconstruct the data in case of any major or minor crisis with the database. In general terms, backups can be of two types:
- **Physical Backup** - the data directory as it is on the disk
- **Logical Backup** - the table structure and records in it
@@ -7,65 +7,87 @@ Backups are a very crucial part of any database setup. They are generally a copy
Both the above kinds of backups are supported by MySQL with different tools. It is the job of the SRE to identify which should be used when.
#### Mysqldump
This utility is available with MySQL installation. It helps in getting the logical backup of the database. It outputs a set of SQL statements to reconstruct the data. It is not recommended to use mysqldump for large tables as it might take a lot of time and the file size will be huge. However, for small tables it is the best and the quickest option.
This utility is available with MySQL installation. It helps in getting the logical backup of the database. It outputs a set of SQL statements to reconstruct the data. It is not recommended to use `mysqldump` for large tables as it might take a lot of time and the file size will be huge. However, for small tables it is the best and the quickest option.
`mysqldump [options] > dump_output.sql`
```shell
mysqldump [options] > dump_output.sql
```
There are certain options that can be used with mysqldump to get an appropriate dump of the database.
There are certain options that can be used with `mysqldump` to get an appropriate dump of the database.
To dump all the databases
To dump all the databases:
`mysqldump -u<user> -p<pwd> --all-databases > all_dbs.sql`
```shell
mysqldump -u<user> -p<pwd> --all-databases > all_dbs.sql
```
To dump specific databases
To dump specific databases:
`mysqldump -u<user> -p<pwd> --databases db1 db2 db3 > dbs.sql`
```shell
mysqldump -u<user> -p<pwd> --databases db1 db2 db3 > dbs.sql
```
To dump a single database
`mysqldump -u<user> -p<pwd> --databases db1 > db1.sql`
To dump a single database:
```shell
mysqldump -u<user> -p<pwd> --databases db1 > db1.sql
```
OR
```shell
mysqldump -u<user> -p<pwd> db1 > db1.sql
```
`mysqldump -u<user> -p<pwd> db1 > db1.sql`
The difference between the above two commands is that the latter one does not contain the `CREATE DATABASE` command in the backup output.
The difference between the above two commands is that the latter one does not contain the **CREATE DATABASE** command in the backup output.
To dump specific tables in a database:
To dump specific tables in a database
```shell
mysqldump -u<user> -p<pwd> db1 table1 table2 > db1_tables.sql
```
`mysqldump -u<user> -p<pwd> db1 table1 table2 > db1_tables.sql`
To dump only table structures and no data:
To dump only table structures and no data
```shell
mysqldump -u<user> -p<pwd> --no-data db1 > db1_structure.sql
```
`mysqldump -u<user> -p<pwd> --no-data db1 > db1_structure.sql`
To dump only table data and no `CREATE` statements:
To dump only table data and no CREATE statements
```shell
mysqldump -u<user> -p<pwd> --no-create-info db1 > db1_data.sql
```
`mysqldump -u<user> -p<pwd> --no-create-info db1 > db1_data.sql`
To dump only specific records from a table:
To dump only specific records from a table
```shell
mysqldump -u<user> -p<pwd> --no-create-info db1 table1 --where=”salary>80000” > db1_table1_80000.sql
```
`mysqldump -u<user> -p<pwd> --no-create-info db1 table1 --where=”salary>80000” > db1_table1_80000.sql`
`mysqldump` can also provide output in CSV, other delimited text or XML format to support use-cases if any. The backup from `mysqldump` utility is offline, i.e. when the backup finishes it will not have the changes to the database which were made when the backup was going on. For example, if the backup started at 3:00 pm and finished at 4:00 pm, it will not have the changes made to the database between 3:00 and 4:00 pm.
Mysqldump can also provide output in CSV, other delimited text or XML format to support use-cases if any. The backup from mysqldump utility is offline i.e. when the backup finishes it will not have the changes to the database which were made when the backup was going on. For example, if the backup started at 3 PM and finished at 4 PM, it will not have the changes made to the database between 3 and 4 PM.
**Restoring** from mysqldump can be done in the following two ways:-
**Restoring** from `mysqldump` can be done in the following two ways:
From shell
`mysql -u<user> -p<pwd> < all_dbs.sql`
```shell
mysql -u<user> -p<pwd> < all_dbs.sql
```
OR
From shell if the database is already created
From shell, if the database is already created:
`mysql -u<user> -p<pwd> db1 < db1.sql`
```shell
mysql -u<user> -p<pwd> db1 < db1.sql
```
From within MySQL shell
From within MySQL shell:
`mysql> source all_dbs.sql`
```shell
mysql> source all_dbs.sql
```
#### Percona Xtrabackup
This utility is installed separately from the MySQL server and is open source, provided by Percona. It helps in getting the full or partial physical backup of the database. It provides online backup of the database i.e. it will have the changes made to the database when the backup was going on as explained at the end of the previous section.
#### Percona XtraBackup
This utility is installed separately from the MySQL server and is open source, provided by Percona. It helps in getting the full or partial physical backup of the database. It provides online backup of the database, i.e. it will have the changes made to the database when the backup was going on as explained at the end of the previous section.
- **Full Backup** - the complete backup of the database.
- **Partial Backup** - Incremental
@@ -74,74 +96,90 @@ This utility is installed separately from the MySQL server and is open source, p
![partial backups - differential and cummulative](images/partial_backup.png "Differential and Cumulative Backups")
Percona xtrabackup allows us to get both full and incremental backups as we desire. However, incremental backups take less space than a full backup (if taken per day) but the restore time of incremental backups is more than that of full backups.
Percona XtraBackup allows us to get both full and incremental backups as we desire. However, incremental backups take less space than a full backup (if taken per day) but the restore time of incremental backups is more than that of full backups.
**Creating a full backup**
`xtrabackup --defaults-file=<location to my.cnf> --user=<mysql user> --password=<mysql password> --backup --target-dir=<location of target directory>`
```shell
xtrabackup --defaults-file=<location to my.cnf> --user=<mysql user> --password=<mysql password> --backup --target-dir=<location of target directory>
```
Example
Example:
`xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/`
```shell
xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/
```
Some other options
- `--stream` - can be used to stream the backup files to standard output in a specified format. xbstream is the only option for now.
- `--tmp-dir` - set this to a tmp directory to be used for temporary files while taking backups.
- `--stream` - can be used to stream the backup files to standard output in a specified format. `xbstream` is the only option for now.
- `--tmp-dir` - set this to a `tmp` directory to be used for temporary files while taking backups.
- `--parallel` - set this to the number of threads that can be used to parallely copy data files to target directory.
- `--compress` - by default - quicklz is used. Set this to have the backup in compressed format. Each file is a .qp compressed file and can be extracted by qpress file archiver.
- `--decompress` - decompresses all the files which were compressed with the .qp extension. It will not delete the .qp files after decompression. To do that, use `--remove-original` along with this. Please note that the decompress option should be run separately from the xtrabackup command that used the compress option.
- `--compress` - by default - `quicklz` is used. Set this to have the backup in compressed format. Each file is a `.qp` compressed file and can be extracted by `qpress` file archiver.
- `--decompress` - decompresses all the files which were compressed with the `.qp` extension. It will not delete the `.qp` files after decompression. To do that, use `--remove-original` along with this. Please note that the `decompress` option should be run separately from the `xtrabackup` command that used the compress option.
**Preparing a backup**
Once the backup is done with the --backup option, we need to prepare it in order to restore it. This is done to make the datafiles consistent with point-in-time. There might have been some transactions going on while the backup was being executed and those have changed the data files. When we prepare a backup, all those transactions are applied to the data files.
Once the backup is done with the `--backup` option, we need to prepare it in order to restore it. This is done to make the data files consistent with point-in-time. There might have been some transactions going on while the backup was being executed and those have changed the data files. When we prepare a backup, all those transactions are applied to the data files.
`xtrabackup --prepare --target-dir=<where backup is taken>`
```shell
xtrabackup --prepare --target-dir=<where backup is taken>
```
Example
Example:
`xtrabackup --prepare --target-dir=/mnt/data/backup/`
```shell
xtrabackup --prepare --target-dir=/mnt/data/backup/
```
It is not recommended to halt a process which is preparing the backup as that might cause data file corruption and backup cannot be used further. The backup will have to be taken again.
**Restoring a Full Backup**
To restore the backup which is created and prepared from above commands, just copy everything from the backup target-dir to the data-dir of MySQL server, change the ownership of all files to mysql user (the linux user used by MySQL server) and start mysql.
To restore the backup which is created and prepared from above commands, just copy everything from the backup `target-dir` to the `data-dir` of MySQL server, change the ownership of all files to MySQL user (the Linux user used by MySQL server) and start MySQL.
Or the below command can be used as well,
`xtrabackup --defaults-file=/etc/my.cnf --copy-back --target-dir=/mnt/data/backups/`
```shell
xtrabackup --defaults-file=/etc/my.cnf --copy-back --target-dir=/mnt/data/backups/
```
**Note** - the backup has to be prepared in order to restore it.
**Creating Incremental backups**
Percona Xtrabackup helps create incremental backups, i.e only the changes can be backed up since the last backup. Every InnoDB page contains a log sequence number or LSN that is also mentioned as one of the last lines of backup and prepare commands.
```
Percona XtraBackup helps create incremental backups, i.e, only the changes can be backed up since the last backup. Every InnoDB page contains a log sequence number or LSN that is also mentioned as one of the last lines of backup and prepare commands.
```shell
xtrabackup: Transaction log of lsn <LSN> to <LSN> was copied.
```
OR
```
```shell
InnoDB: Shutdown completed; log sequence number <LSN>
<timestamp> completed OK!
```
This indicates that the backup has been taken till the log sequence number mentioned. This is a key information in understanding incremental backups and working towards automating one. Incremental backups do not compare data files for changes, instead, they go through the InnoDB pages and compare their LSN to the last backup’s LSN. So, without one full backup, the incremental backups are useless.
The xtrabackup command creates a xtrabackup_checkpoint file which has the information about the LSN of the backup. Below are the key contents of the file:-
```
The `xtrabackup` command creates a `xtrabackup_checkpoint` file which has the information about the LSN of the backup. Below are the key contents of the file:
```shell
backup_type = full-backuped | incremental
from_lsn = 0 (full backup) | to_lsn of last backup <LSN>
to_lsn = <LSN>
last_lsn = <LSN>
```
There is a difference between **to\_lsn** and **last\_lsn**. When the **last\_lsn** is more than **to\_lsn** that means there are transactions that ran while we took the backup and are yet to be applied. That is what --prepare is used for.
There is a difference between `to_lsn` and `last_lsn`. When the `last_lsn` is more than `to_lsn` that means there are transactions that ran while we took the backup and are yet to be applied. That is what `--prepare` is used for.
To take incremental backups, first, we require one full backup.
`xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/full/`
Let’s assume the contents of the xtrabackup_checkpoint file to be as follows.
```shell
xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/full/
```
Let’s assume the contents of the `xtrabackup_checkpoint` file to be as follows:
```shell
backup_type = full-backuped
from_lsn = 0
to_lsn = 1000
@@ -149,62 +187,80 @@ last_lsn = 1000
```
Now that we have one full backup, we can have an incremental backup that takes the changes. We will go with differential incremental backups.
`xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/incr1/ --incremental-basedir=/mnt/data/backup/full/`
There are delta files created in the incr1 directory like, **ibdata1.delta**, **db1/tbl1.ibd.delta** with the changes from the full directory. The xtrabackup_checkpoint file will thus have the following contents.
```shell
xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/incr1/ --incremental-basedir=/mnt/data/backup/full/
```
There are delta files created in the `incr1` directory like, `ibdata1.delta`, `db1/tbl1.ibd.delta` with the changes from the full directory. The `xtrabackup_checkpoint` file will thus have the following contents.
```shell
backup_type = incremental
from_lsn = 1000
to_lsn = 1500
last_lsn = 1500
```
Hence, the **from\_lsn** here is equal to the **to\_lsn** of the last backup or the basedir provided for the incremental backups. For the next incremental backup we can use this incremental backup as the basedir.
`xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/incr2/ --incremental-basedir=/mnt/data/backup/incr1/`
Hence, the `from_lsn` here is equal to the `to_lsn` of the last backup or the `basedir` provided for the incremental backups. For the next incremental backup, we can use this incremental backup as the `basedir`.
The xtrabackup_checkpoint file will thus have the following contents.
```shell
xtrabackup --defaults-file=/etc/my.cnf --user=some_user --password=XXXX --backup --target-dir=/mnt/data/backup/incr2/ --incremental-basedir=/mnt/data/backup/incr1/
```
The `xtrabackup_checkpoint` file will thus have the following contents:
```shell
backup_type = incremental
from_lsn = 1500
to_lsn = 2000
last_lsn = 2200
```
**Preparing Incremental backups**
Preparing incremental backups is not the same as preparing a full backup. When prepare runs, two operations are performed - *committed transactions are applied on the data files* and *uncommitted transactions are rolled back*. While preparing incremental backups, we have to skip rollback of uncommitted transactions as it is likely that they might get committed in the next incremental backup. If we rollback uncommitted transactions the further incremental backups cannot be applied.
Preparing incremental backups is not the same as preparing a full backup. When prepare runs, two operations are performed - *committed transactions are applied on the data files* and *uncommitted transactions are rolled back*. While preparing incremental backups, we have to skip rollback of uncommitted transactions as it is likely that they might get committed in the next incremental backup. If we rollback uncommitted transactions, the further incremental backups cannot be applied.
We use **--apply-log-only** option along with **--prepare** to avoid the rollback phase.
We use `--apply-log-only` option along with `--prepare` to avoid the rollback phase.
From the last section, we had the following directories with complete backup
```
From the last section, we had the following directories with complete backup:
```shell
/mnt/data/backup/full
/mnt/data/backup/incr1
/mnt/data/backup/incr2
```
First, we prepare the full backup, but only with the --apply-log-only option.
`xtrabackup --prepare --apply-log-only --target-dir=/mnt/data/backup/full`
First, we prepare the full backup, but only with the `--apply-log-only` option.
```shell
xtrabackup --prepare --apply-log-only --target-dir=/mnt/data/backup/full
```
The output of the command will contain the following at the end.
```
```shell
InnoDB: Shutdown complete; log sequence number 1000
<timestamp> Completed OK!
```
Note the LSN mentioned at the end is the same as the to\_lsn from the xtrabackup_checkpoint created for full backup.
Note the LSN mentioned at the end is the same as the `to_lsn` from the `xtrabackup_checkpoint` created for full backup.
Next, we apply the changes from the first incremental backup to the full backup.
`xtrabackup --prepare --apply-log-only --target-dir=/mnt/data/backup/full --incremental-dir=/mnt/data/backup/incr1`
```shell
xtrabackup --prepare --apply-log-only --target-dir=/mnt/data/backup/full --incremental-dir=/mnt/data/backup/incr1
```
This applies the delta files in the incremental directory to the full backup directory. It rolls the data files in the full backup directory forward to the time of incremental backup and applies the redo logs as usual.
Lastly, we apply the last incremental backup same as the previous one with just a small change.
`xtrabackup --prepare --target-dir=/mnt/data/backup/full --incremental-dir=/mnt/data/backup/incr1`
```shell
xtrabackup --prepare --target-dir=/mnt/data/backup/full --incremental-dir=/mnt/data/backup/incr1
```
We do not have to use the **--apply-log-only** option with it. It applies the *incr2 delta files* to the full backup data files taking them forward, applies redo logs on them and finally rollbacks the uncommitted transactions to produce the final result. The data now present in the full backup directory can now be used to restore.
We do not have to use the `--apply-log-only` option with it. It applies the *incr2 delta files* to the full backup data files taking them forward, applies redo logs on them and finally rollbacks the uncommitted transactions to produce the final result. The data now present in the full backup directory can now be used to restore.
**Note** - To create cumulative incremental backups, the incremental-basedir should always be the full backup directory for every incremental backup. While preparing, we can start with the full backup with the --apply-log-only option and use just the last incremental backup for the final --prepare as that has all the changes since the full backup.
**Note**: To create cumulative incremental backups, the `incremental-basedir` should always be the full backup directory for every incremental backup. While preparing, we can start with the full backup with the `--apply-log-only` option and use just the last incremental backup for the final `--prepare` as that has all the changes since the full backup.
**Restoring Incremental backups**