Backups you can actually restore
A backup job that reports success has told you almost nothing. It has confirmed that some bytes were written somewhere. It has not confirmed that those bytes are a valid archive, that the destination accepted the whole file, or that the data inside is the data you meant to save.
The failure mode nobody notices
The classic silent failure: the upload tool gets a broken pipe, the archive is half-written, and the exit code is still zero because the pipeline ended on something else. The job goes green for weeks. Nobody looks, because it is green.
# This reports the first entry and exits early.
# tar sees SIGPIPE, dies, and the wrapper records a failure
# that has nothing to do with the archive being fine.
verify=$(tar -tzf "$archive" 2>&1 | head -1)Read the whole listing and check the entry count. It costs seconds and it is the difference between a backup and a rumour.
Restore drills
Once a week, a script pulls the newest snapshot into a scratch directory, opens the database, runs an integrity check, and counts rows in the two tables that matter. If any step fails, it is not a warning — it is a broken backup, and it gets fixed that day.
The only backup that counts is one you have opened and read.
Snapshots of live databases
Copying a live SQLite file while it is being written gives you a file that opens
and lies to you. Use the engine's own mechanism — VACUUM INTO for SQLite,
a dump for anything with one — and verify the result before it leaves the machine.
Retention
Daily for a week, weekly for a month, monthly for a quarter. Beyond that, the cost of retention usually exceeds the value, unless something regulatory says otherwise. Deduplicated repositories make this cheap in space, which is the main argument for using one over a pile of tarballs.