Repository: digint/btrbk

Issue #91: btrfs send/receive failures
Created: 2016-06-10T13:58:09Z
This is a possible continuation of Issue#80:
I've been trying to get btrbk to work w/o issue since that time but errors continue to plague me:
I get random(ly) seeming issues each time btrbk runs (currently once a day).
Examples from last night from different subvolumes include:

ERROR: unlink cache/python/eb09d3753c7cccd22a93a5f17f4c7de1 failed. No such file or directory
ERROR: truncate .kodi/userdata/Thumbnails/6/67c8487a.jpg failed: No such file or directory
ERROR: unlink sickbeard.db-journal failed. No such file or directory
ERROR: utimes dockerfiles.git/.git/objects/3f failed: No such file or directory

I asked in #btrfs today about this and had a conversation with @ke.  I've copy-pasted that conversation here just in case it has any leads that may help narrow down this problem.
"9:22:58 AM    zalaare  has anybody else had issues with 'btrfs send snapshots' having unexpected unlink/utime errors during send? http://pastebin.com/raw/X6Fi9ub9
9:25:34 AM Ke  zalaare: ordering errors have happened from time to time
9:26:11 AM zalaare  @Ke: this happens /every/ night on generally the same volumes.
9:27:06 AM Ke  you could resend the parent non-incremetally on a temp image and try to resend to that image
9:29:10 AM zalaare  Actually, I just resend the same snapshot again and it usually goes without error.  But I can't take time every day to resend random snapshots that didn't work during the initial backup.
9:30:07 AM Ke  are you sending the previous snapshots in very rapid succession
9:35:06 AM zalaare  I don't think so.  I've been using btrbk for this for awhile.  AFAIK the only time it sends previous snapshots along and then immediately sends the current, is when there exists multiple snapshots at the source and some/all of them do /not/ exist in the destination.  This can and does happen if the sends fail from one night to the next (like in this case), but hasn't happened with this current example because I manually resent the missing snapshots yesterday.  AKA, it only needed to send 20160610 snapshots this morning.
9:35:53 AM Ke  are you sure btrbk knows about that
9:36:01 AM Ke  I know nothing about that tool
9:36:24 AM Ke  would be nice to figure out what that tool does and then reproduce it manually
9:38:28 AM zalaare  I can/will follow up with a btrbk issue. do you mind if I copy-pasta this conversation for reference?
9:39:13 AM Ke  zalaare: you can, but I don't know, if it has any meaningful info"
As a side-note referring back to the original Issue#80:

Is there anyway in the future to make the warning only appear the first time or something? This string is filling my logs like a maniac: WARNING: Receive target of "<subvolume_name>" exists at unexpected location:

Duplicate issue #371: btrfs receive sometimes can not find some internal files
Created: 2021-02-20T09:37:40Z
I sometimes see an error like:
ERROR: cannot open /mnt/masa-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210220T1134/o223727-162425-0: No such file or directory

During my examinations, I found out that this is caused by issuing wrong parent snapshot to btrfs send:
Failed
sudo btrfs send -p /mnt/heybe-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210214T0015 /mnt/heybe-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210220T1134 | mbuffer -v 1 | sudo btrfs receive /mnt/masa-root/snapshots/erik3/var/lib/lxc/fc4/
At subvol /mnt/heybe-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210220T1134
At snapshot rootfs.20210220T1134
in @ 28.0 MiB/s, out @ 38.0 MiB/s, 48.0 MiB total, buffer  23% full
ERROR: cannot open /mnt/masa-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210220T1134/o223727-162425-0: No such file or directory

summary: 60.0 MiByte in  5.1sec - average of 11.8 MiB/s

Succeeded
sudo btrfs send -p /mnt/heybe-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210213T0356 /mnt/heybe-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210220T1134 | mbuffer -v 1 | sudo btrfs receive /mnt/masa-root/snapshots/erik3/var/lib/lxc/fc4/
At subvol /mnt/heybe-root/snapshots/erik3/var/lib/lxc/fc4/rootfs.20210220T1134
At snapshot rootfs.20210220T1134
in @ 24.0 MiB/s, out @ 24.0 MiB/s,  569 MiB total, buffer   0% full
summary:  571 MiByte in 29.2sec - average of 19.6 MiB/s

Info
In /mnt/heybe-root/:
snapshots/erik3/var/lib/lxc/fc4/rootfs.20210213T0356
	Name: 			rootfs.20210213T0356
	UUID: 			488f5eb5-8ec0-414e-89a6-b28ad85d3005
	Parent UUID: 		7bc2daa5-d6c6-2142-849f-a0c429fa44c2
	Received UUID: 		-
	Creation time: 		2021-02-13 03:56:31 +0300
	Subvolume ID: 		5820
	Generation: 		147362
	Gen at creation: 	147362
	Parent ID: 		5
	Top level ID: 		5
	Flags: 			readonly

snapshots/erik3/var/lib/lxc/fc4/rootfs.20210214T0015
	Name: 			rootfs.20210214T0015
	UUID: 			8df93123-80e1-4149-97d0-e92b5747977b
	Parent UUID: 		7bc2daa5-d6c6-2142-849f-a0c429fa44c2
	Received UUID: 		-
	Creation time: 		2021-02-14 00:15:49 +0300
	Subvolume ID: 		6007
	Generation: 		149486
	Gen at creation: 	149485
	Parent ID: 		5
	Top level ID: 		5
	Flags: 			readonly
	Snapshot(s):

snapshots/erik3/var/lib/lxc/fc4/rootfs.20210220T1134
	Name: 			rootfs.20210220T1134
	UUID: 			0214f146-d691-2b41-87e0-6f7df401e6cc
	Parent UUID: 		229da998-3580-db41-9adb-0d4f47337e5a
	Received UUID: 		-
	Creation time: 		2021-02-20 11:34:39 +0300
	Subvolume ID: 		8115
	Generation: 		164610
	Gen at creation: 	164609
	Parent ID: 		5
	Top level ID: 		5
	Flags: 			readonly

Opinions
I think this is because of a workflow like that:
s1 -> s2 -> s3     s4 -> s5
       \-----------^

Sending above snapshots in the order of s1, s2, s3, s4, s5 creates this kind of conflicts. Maybe not, I don't know.
Workaround
Whenever btrbk fails with that error, I manually try issuing an older parent snapshot manually.
Suggestion
btrbk should automatically try older snapshots in case of such an error.

Closing comment (2021-02-26T16:34:41Z): Duplicate of #91, #223, #225, #251 : upstream (btrfs) bug.