Went to do a test restore of one of my databases and I noticed the dump files over the last few months were all 0kb. Glad I caught it this way and not because I needed to restore. Put it on your calendar, schedule a test restore of your critical stuff a couple times a year. I know y’all are busy but it is worth the time and effort. A backup you can’t actually restore isn’t a backup at all.
Ain’t got time for that
Instead, I let my faith carry me though
Also, if applicable, have a different person perform the restore every time and have them do it just by following the documentation. This way multiple persons have actual experience with the process if the shit ever hits the fan and this also makes sure the documentation is accurate and up-to-date.
We do a restore test once a quarter, but we are SOC2 so we must do it. Doesn’t have to restore an entire VM, just a random used file.
A few months after setting up the backups for my server to two remote locations and patting myself in the back for it, I woke up in the middle of the night realizing I had no idea how to restore.
I worked at a datacenter that wanted to change backup vendors, as we dug into the details we found out that the agent based backups needed an agent running on the machine to restore to and they didn’t have a linux agent. Despite this obvious problem mgmt chose this vendor. It didn’t take long before sysadmins were rebuilding linux boxes from scratch in the wee hours. I left shortly after.
I started out in a big iron shop in the 90s. I was in charge of backups. We had an outage after a power loss, and the generator not kicking in. Our local backup set didn’t restore. The basement set, in a fire/waterproof safe didn’t restore. The off-site set restored.
The hours between the first fail and success were pure terror.
Rules of Tech Support
Rule T9C - If you can’t restore from it, you don’t have a backup.
Rule T9D - If you haven’t tested your backup recently, you don’t have a backup.
3-2-1…
Three copies of your data, in two physical locations, equals one backup. And it’s not real until you do a test restore yes indeed!
My understanding of 3-2-1 is different;
3 copies of your data, on two different storage mediums, with one offsite.
E.g. SSD live copy, hot HDD backup, cold HDD backup offsite.
Is this wrong?
How do you test one
It depends on the type of backup:
For a filesystem backup, restore one or more files to a secondary location. E.g. pick a few files out of the backup and try to restore them to a temporary folder. Then hash the original and restored files to verify integrity.
For a full machine backup (e.g. VM backup), restore a copy of the machine to a test location. Spin up the test machine to verify that it can boot.
For a database backup, restore a copy of the database to a test location (e.g. change the database name as part of the restore process), compare a few tables against the real database to verify integrity.Pretty much, it’s going to be some version of “Restore X to a test location and verify integrity”. You want to both prove that the backup can be restored and that the restored copy is actually intact.
Thanks for explaining this! Very helpful and easy to understand. Do you have preferred programs for the two actions? I currently just rsync my servers.
I use restic to back up files to Wasabi.
It’s all scripted but the steps are:
- Back up
- Prune based on rules
- Perform a repository consistency check and do further checks (takes longer) on 1% of the repo
- Choose 1 random file from the current backup set and get the hash of the current file
- Restore that file to a temporary location and hash it.
- Compare hashes
- Send push alert to me with success/failure and a summary
