Use Duplicati for daily backups #48

New Issue

lukas · 2024-07-10T00:26:07+02:00

lukas commented

2024-07-10 00:26:07 +02:00

A 1TB Hetzner Storage Box costs 3.81€ per month and can be used by all servers and also includes the ability to make snapshots.
Backblaze B2 is a great service but due to storage constraints on the server itself, I mainly use rclone rcat in pipes to export data directly to B2, which unfortunately can fail if B2 dispatched a server with not enough storage for the file - which can happen as rcat doesn't know what size the file is going to be as it's generated during uploading in the pipe. In the case of a failure, the script will simply not do anything.
Also, some services hold large amounts of data (>1GB), which means tarred backup to B2 multiplies the cost by 7 for 7 days retention, even if nothing changed. I do this to be able to an atomic roll back to an earlier version. For some services I decided that was too much storage and therefore just sync the folders directly 1:1, meaning deleted/changed data from the source is kept through Blackblaze's versioning, but previous versions need to be rolled back manually - easy if you know what you are looking for with Szurubooru or Nextcloud, but not really with Gitea :).

Syncing the local data to a Storage Box will keep storage usage low, enables easy roll back through automatic ZFS snapshots and - if there isn't maintenance being done at that moment - shouldn't fail as there is nothing to dispatch.

Monthly atomic backups are still made to Azure with high redundancy, so Hetzner burning down / disabling my account is accounted for. A simple sync to Backblaze would also be easily possible and stop daily backups from failing due to 503 errors, as they can be uploaded from the storage box - with known, predictable file sizes and possibilities to retry.

This would also allow @leon to back up Matrix onto it.

A 1TB Hetzner Storage Box costs 3.81€ per month and can be used by all servers and also includes the ability to make snapshots. Backblaze B2 is a great service but due to storage constraints on the server itself, I mainly use `rclone rcat` in pipes to export data directly to B2, which unfortunately can fail if B2 dispatched a server with not enough storage for the file - which can happen as `rcat` doesn't know what size the file is going to be as it's generated during uploading in the pipe. In the case of a failure, the script will simply not do anything. Also, some services hold large amounts of data (>1GB), which means tarred backup to B2 multiplies the cost by 7 for 7 days retention, even if nothing changed. I do this to be able to an atomic roll back to an earlier version. For some services I decided that was too much storage and therefore just sync the folders directly 1:1, meaning deleted/changed data from the source is kept through Blackblaze's versioning, but previous versions need to be rolled back manually - easy if you know what you are looking for with Szurubooru or Nextcloud, but not really with Gitea :). Syncing the local data to a Storage Box will keep storage usage low, enables easy roll back through automatic ZFS snapshots and - if there isn't maintenance being done at that moment - shouldn't fail as there is nothing to dispatch. Monthly atomic backups are still made to Azure with high redundancy, so Hetzner burning down / disabling my account is accounted for. A simple sync to Backblaze would also be easily possible and stop daily backups from failing due to 503 errors, as they can be uploaded from the storage box - with known, predictable file sizes and possibilities to retry. This would also allow @leon to back up Matrix onto it.

lukas added the

labels 2024-07-10 00:26:07 +02:00

lukas self-assigned this 2024-07-10 00:26:07 +02:00

lukas added this to the Issue Board project 2024-07-10 00:26:07 +02:00

lukas commented

2024-07-10 00:30:59 +02:00

On the other hand, Backblaze currently costs at most <0,50€, solving a problem that at worst increase it to <1,50€ by purchasing something that always costs 3,81€ is kinda stupid. This needs some more thought.

lukas changed title from ~~Use Storage Box for daily backups~~ to Evaluate Storage Box for daily backups

2024-07-10 00:31:42 +02:00

lukas commented

2024-07-10 00:35:11 +02:00

Using a backup service like Duplicati, that can make incremental backups to object storage and also restore easily (and even through a UI) sounds like a smarter choice.

If these fail, there are always the atomic and simple backups to Azure so...

Using a backup service like Duplicati, that can make incremental backups to object storage and also restore easily (and even through a UI) sounds like a smarter choice. If these fail, there are always the atomic and simple backups to Azure so...

lukas commented

2024-07-11 18:19:25 +02:00

Duplicati is probably the way to go

lukas changed title from ~~Evaluate Storage Box for daily backups~~ to Use Duplicati for daily backups

2024-07-11 18:19:41 +02:00

lukas commented

2024-07-25 21:32:13 +02:00

Closing this as not planned, as I changed the backups for larger volumes to a rclone sync instead of creating an atomic tar-archive.
While not being an optimal solution, it works good for the time being and the services hosted. Might re-open later.

Closing this as not planned, as I changed the backups for larger volumes to a `rclone sync` instead of creating an atomic `tar`-archive. While not being an optimal solution, it works good for the time being and the services hosted. Might re-open later.

lukas closed this issue

2024-07-25 21:32:13 +02:00

lukas removed this from the Issue Board project 2024-07-25 21:32:30 +02:00

Sign in to join this conversation.

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: KomuSolutions/igot99issues#48