Repository: digint/btrbk

Issue #308: extend retention policy to take into account the number of snapshots and/or time of last successful run
Created: 2019-11-19T11:32:16Z
If I understand correctly, setting, for example, 24h as part of a retention policy will preserve at most one hourly snapshot as far back as 24 wall-clock hours (not the 24 most recent hourly snapshots). As far as I'm concerned, that's the right thing to do in principle.
Now consider a source turned off over the weekend. Wouldn't btrbk wipe all the hourly snapshots on its first run after the break? So if I did something stupid on Thursday and notice the on Friday, I get fine-grained rollback, if I botch it on Friday and notice on Monday, all I get is a daily, even though subjectively it's the following workday in both cases. (Same for dailies and a longer holiday, etc.)
I'm not sure I feel comfortable with that, i.e. btrbk treating time passed with regular successful runs and time passed without any the same. I'd prefer to to keep the most recent 24h of actual work done, for some subvolumes at least, to have the option to "stop time" during downtime, maybe taking the timestamp of the most recent snapshot into consideration, or simply something like 24h+ "keep hourly snapshots 24 hours back, but at least the 24 most recent hourlies" and so on.

Duplicate issue #508: raw target: add option to not use the closest parent but the one that is the last to be rotatet away
Created: 2022-11-20T01:55:13Z
Hey.
I noticed this while trying out the branch from #474 ... I'm not sure whether what I describe below could already be done via incremental_prefs, if so I didn’t understand it.
One thing I’ve noticed (when trying https://github.com/digint/btrbk/commits/delete-incremental-raw) is what I have right now:
-       backup1.example.org  /var/local/lcg-backup/data/btrbk/source-host.example.org/system/data.20221120T000012+0100.btrfs.gpg  5h      preserve hourly: first of hour, 2 hours ago
-       backup1.example.org  /var/local/lcg-backup/data/btrbk/source-host.example.org/system/data.20221120T004649+0100.btrfs.gpg  5h      preserve forced: parent of preserved raw target
-       backup1.example.org  /var/local/lcg-backup/data/btrbk/source-host.example.org/system/data.20221120T010012+0100.btrfs.gpg  5h      preserve hourly: first of hour, 1 hours ago
-       backup1.example.org  /var/local/lcg-backup/data/btrbk/source-host.example.org/system/data.20221120T020012+0100.btrfs.gpg  5h      preserve hourly: first of hour, 0 hours ago
-       backup1.example.org  /var/local/lcg-backup/data/btrbk/source-host.example.org/system/data.20221120T023659+0100.btrfs.gpg  5h      preserve min: latest

with the following files:
-rw-r--r-- 1 root root  220 Nov 20 00:04 data.20221120T000012+0100.btrfs.gpg.info
-rw-r--r-- 1 root root  93M Nov 20 00:46 data.20221120T004649+0100.btrfs.gpg
-rw-r--r-- 1 root root  268 Nov 20 00:46 data.20221120T004649+0100.btrfs.gpg.info
-rw-r--r-- 1 root root 8,3M Nov 20 01:00 data.20221120T010012+0100.btrfs.gpg
-rw-r--r-- 1 root root  268 Nov 20 01:00 data.20221120T010012+0100.btrfs.gpg.info
-rw-r--r-- 1 root root  96M Nov 20 02:00 data.20221120T020012+0100.btrfs.gpg
-rw-r--r-- 1 root root  268 Nov 20 02:00 data.20221120T020012+0100.btrfs.gpg.info

i.e. only the oldest one is a full backup and the following sidecars:
-e -n #btrbk-v0.32.5
# Do not edit this file
#t=1668898813
TYPE=raw
FILE=data.20221120T000012+0100.btrfs.gpg
RECEIVED_UUID=6fe09732-fdac-0a47-9087-0b4191679ef9
encrypt=gpg
INCOMPLETE=1

-e -n #t=1668899065
INCOMPLETE=0

#btrbk-v0.32.6-dev
# Do not edit this file
#t=1668901610
TYPE=raw
FILE=data.20221120T004649+0100.btrfs.gpg
RECEIVED_UUID=e63b0251-d34e-484d-939e-388a9dc40b51
RECEIVED_PARENT_UUID=6fe09732-fdac-0a47-9087-0b4191679ef9
encrypt=gpg
INCOMPLETE=1
#t=1668901616
INCOMPLETE=0
#btrbk-v0.32.6-dev
# Do not edit this file
#t=1668902414
TYPE=raw
FILE=data.20221120T010012+0100.btrfs.gpg
RECEIVED_UUID=e51b9ff7-9260-8247-a103-f57ed959ccff
RECEIVED_PARENT_UUID=e63b0251-d34e-484d-939e-388a9dc40b51
encrypt=gpg
INCOMPLETE=1
#t=1668902415
INCOMPLETE=0
#btrbk-v0.32.6-dev
# Do not edit this file
#t=1668906012
TYPE=raw
FILE=data.20221120T020012+0100.btrfs.gpg
RECEIVED_UUID=f99fd7a5-ed28-6541-943c-1def6cc571d5
RECEIVED_PARENT_UUID=e51b9ff7-9260-8247-a103-f57ed959ccff
encrypt=gpg
INCOMPLETE=1
#t=1668906019
INCOMPLETE=0

AFAIU, the one from 20221120T004649 would in principle be deleted, because I have 5h and there’s also the one from 20221120T000012 (which is closer to the full hour at 20221120T0000 and thus the one to be kept), but isn’t because its the parent of 20221120T010012 and the other following incremental dumps.
Right so far?
In principle the rotating is of course correct, but wouldn't it perhaps make sense here to not use the closest parent (20221120T004649) when creating the incremental 20221120T010012, but the closest one that is (with the current retention policy) the last to be rotated away?
The idea here is that one typically has some cron/systemd timer, which would then create cycles of full/incremental backups... while in addition there might be manually created ones (like above 20221120T004649).
And then one wants the normal cycle to be used, rather than such intermediate backups, which should go away after target_preserve_min.
OTOH, any runs without such option (i.e. manual runs) would indeed use the closest parent.
Not sure whether this makes sense. ^^
In any case... it’s not really important to me.
Cheers,
Chris.

Closing comment (2022-11-20T22:26:10Z): I think what you describe here is pretty much a duplicate of #308