Repository: digint/btrbk

Issue #91: btrfs send/receive failures
Created: 2016-06-10T13:58:09Z
This is a possible continuation of Issue#80:
I've been trying to get btrbk to work w/o issue since that time but errors continue to plague me:
I get random(ly) seeming issues each time btrbk runs (currently once a day).
Examples from last night from different subvolumes include:

ERROR: unlink cache/python/eb09d3753c7cccd22a93a5f17f4c7de1 failed. No such file or directory
ERROR: truncate .kodi/userdata/Thumbnails/6/67c8487a.jpg failed: No such file or directory
ERROR: unlink sickbeard.db-journal failed. No such file or directory
ERROR: utimes dockerfiles.git/.git/objects/3f failed: No such file or directory

I asked in #btrfs today about this and had a conversation with @ke.  I've copy-pasted that conversation here just in case it has any leads that may help narrow down this problem.
"9:22:58 AM    zalaare  has anybody else had issues with 'btrfs send snapshots' having unexpected unlink/utime errors during send? http://pastebin.com/raw/X6Fi9ub9
9:25:34 AM Ke  zalaare: ordering errors have happened from time to time
9:26:11 AM zalaare  @Ke: this happens /every/ night on generally the same volumes.
9:27:06 AM Ke  you could resend the parent non-incremetally on a temp image and try to resend to that image
9:29:10 AM zalaare  Actually, I just resend the same snapshot again and it usually goes without error.  But I can't take time every day to resend random snapshots that didn't work during the initial backup.
9:30:07 AM Ke  are you sending the previous snapshots in very rapid succession
9:35:06 AM zalaare  I don't think so.  I've been using btrbk for this for awhile.  AFAIK the only time it sends previous snapshots along and then immediately sends the current, is when there exists multiple snapshots at the source and some/all of them do /not/ exist in the destination.  This can and does happen if the sends fail from one night to the next (like in this case), but hasn't happened with this current example because I manually resent the missing snapshots yesterday.  AKA, it only needed to send 20160610 snapshots this morning.
9:35:53 AM Ke  are you sure btrbk knows about that
9:36:01 AM Ke  I know nothing about that tool
9:36:24 AM Ke  would be nice to figure out what that tool does and then reproduce it manually
9:38:28 AM zalaare  I can/will follow up with a btrbk issue. do you mind if I copy-pasta this conversation for reference?
9:39:13 AM Ke  zalaare: you can, but I don't know, if it has any meaningful info"
As a side-note referring back to the original Issue#80:

Is there anyway in the future to make the warning only appear the first time or something? This string is filling my logs like a maniac: WARNING: Receive target of "<subvolume_name>" exists at unexpected location:

Duplicate issue #251: btrbk produced an invalid snapshot
Created: 2018-09-07T04:56:52Z
I keep snapshots obtained from my laptop and received over an ssh tunnelled btrfs send / receive at /media/btrfs-top/_btrbk_backups/leonard-xps13. I tried btrbk archive /media/btrfs-top/_btrbk_backups/ /mnt/_btrbk_backups/ to copy all snapshots to an external disk mounted at /mnt.
This fails due some inconsistency in a snapshot:
WARNING: [send/receive] (send=/media/btrfs-top/_btrbk_backups/leonard-xps13/@home.20171001T0001, receive=/mnt/_btrbk_backups/le
onard-xps13) At subvol /media/btrfs-top/_btrbk_backups/leonard-xps13/@home.20171001T0001
ERROR: chown o629958-4676371-0 failed: No such file or directory
WARNING: [send/receive] (send=/media/btrfs-top/_btrbk_backups/leonard-xps13/@home.20171001T0001, receive=/mnt/_btrbk_backups/le
onard-xps13) At snapshot @home.20171001T0001
ERROR: Failed to send/receive btrfs subvolume: /media/btrfs-top/_btrbk_backups/leonard-xps13/@home.20171001T0001 [/media/btrfs-
top/_btrbk_backups/leonard-xps13/@home.20170903T0538] -> /mnt/_btrbk_backups/leonard-xps13
[delete] options: commit-after
[delete] target: /mnt/_btrbk_backups/leonard-xps13/@home.20171001T0001
WARNING: Deleted partially received (garbled) subvolume: /mnt/_btrbk_backups/leonard-xps13/@home.20171001T0001
ERROR: Error while cloning, aborting
Archived 1/15 subvolumes
WARNING: Skipping archiving of "/media/btrfs-top/_btrbk_backups/leonard-xps13/": At least one target aborted earlier

I am not sure what is causing this issue and how to debug it.
Ideally btrbk should have detected a potential issue when transferring the snapshot in the first place, but this may also be problem with btrfs itself.
I run kernel 4.18.6 and believe the problem may be related to the changes between 4.18.5 and 4.18.6. This believe is due to the btrbk archive process also failing on 4.18.5 but with several error messages in dmesg related to certain things hanging for more than 120 seconds. Unfortunately I did not save the output, but upgraded to 4.18.6 instead after running a btrfs scrub on both the internal and external disks (ie. the source and target of the btrbk archive). No errors were found. There is no output in dmesg on 4.18.6.
Please let me know if you have any interest in debugging this further. I will keep the faulty snapshots around for a while, but start afresh with a new "root" snapshot obtained from the laptop.
I currently have the following snapshots:
@home.20170903T0538
@home.20171001T0001
@home.20171105T1050
@home.20171204T0006
@home.20180107T0000
@home.20180204T0025
@home.20180304T0000
@home.20180401T0000
@home.20180603T2121
@home.20180701T0000
@home.20180805T0936
@home.20180812T0000
@home.20180819T0000
@home.20180826T0555
@home.20180902T0045

Closing comment (2018-10-09T13:43:21Z): (sorry for the late answer, I kind of missed this one...)

ERROR: chown o629958-4676371-0 failed: No such file or directory

It's related/duplicate of #91, #223, #225 : upstream bug, nothing that can be fixed in btrbk.
Sadly these incremental send-stream bugs still remain in newer kernels...