Repository: digint/btrbk

Issue #91: btrfs send/receive failures
Created: 2016-06-10T13:58:09Z
This is a possible continuation of Issue#80:
I've been trying to get btrbk to work w/o issue since that time but errors continue to plague me:
I get random(ly) seeming issues each time btrbk runs (currently once a day).
Examples from last night from different subvolumes include:

ERROR: unlink cache/python/eb09d3753c7cccd22a93a5f17f4c7de1 failed. No such file or directory
ERROR: truncate .kodi/userdata/Thumbnails/6/67c8487a.jpg failed: No such file or directory
ERROR: unlink sickbeard.db-journal failed. No such file or directory
ERROR: utimes dockerfiles.git/.git/objects/3f failed: No such file or directory

I asked in #btrfs today about this and had a conversation with @ke.  I've copy-pasted that conversation here just in case it has any leads that may help narrow down this problem.
"9:22:58 AM    zalaare  has anybody else had issues with 'btrfs send snapshots' having unexpected unlink/utime errors during send? http://pastebin.com/raw/X6Fi9ub9
9:25:34 AM Ke  zalaare: ordering errors have happened from time to time
9:26:11 AM zalaare  @Ke: this happens /every/ night on generally the same volumes.
9:27:06 AM Ke  you could resend the parent non-incremetally on a temp image and try to resend to that image
9:29:10 AM zalaare  Actually, I just resend the same snapshot again and it usually goes without error.  But I can't take time every day to resend random snapshots that didn't work during the initial backup.
9:30:07 AM Ke  are you sending the previous snapshots in very rapid succession
9:35:06 AM zalaare  I don't think so.  I've been using btrbk for this for awhile.  AFAIK the only time it sends previous snapshots along and then immediately sends the current, is when there exists multiple snapshots at the source and some/all of them do /not/ exist in the destination.  This can and does happen if the sends fail from one night to the next (like in this case), but hasn't happened with this current example because I manually resent the missing snapshots yesterday.  AKA, it only needed to send 20160610 snapshots this morning.
9:35:53 AM Ke  are you sure btrbk knows about that
9:36:01 AM Ke  I know nothing about that tool
9:36:24 AM Ke  would be nice to figure out what that tool does and then reproduce it manually
9:38:28 AM zalaare  I can/will follow up with a btrbk issue. do you mind if I copy-pasta this conversation for reference?
9:39:13 AM Ke  zalaare: you can, but I don't know, if it has any meaningful info"
As a side-note referring back to the original Issue#80:

Is there anyway in the future to make the warning only appear the first time or something? This string is filling my logs like a maniac: WARNING: Receive target of "<subvolume_name>" exists at unexpected location:

Duplicate issue #292: btrbk fails during incremental send/receive with ERROR referring to "setxattr"
Created: 2019-09-12T10:09:49Z
Hello,
I enjoy btrbk for years!
But now I hit on an unusual ERROR message. Here is the context:
Archiving subvolumes: /storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.*
Creating incremental backup...
[send/receive] target: /storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190908T0645
[send/receive] source: /storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190908T0645
[send/receive] parent: /storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190907T0502
[send/receive] checking target metadata: /storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190908T0645
Creating incremental backup...
[send/receive] target: /storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648
[send/receive] source: /storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648
[send/receive] parent: /storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190908T0645
WARNING: [send/receive] (send=/storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648, receive=/storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps) At subvol /storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648
WARNING: [send/receive] (send=/storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648, receive=/storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps) ERROR: lsetxattr o257-128243-0 system.posix_acl_default= failed: No such file or directory
WARNING: [send/receive] (send=/storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648, receive=/storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps) At snapshot @VMs.20190909T0648
[send/receive] checking target metadata: /storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648
ERROR: Failed to send/receive btrfs subvolume: /storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648 [/storage/T3ButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190908T0645] -> /storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps
[delete] options: commit-after
[delete] target: /storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648
WARNING: Deleted partially received (garbled) subvolume: /storage/4TButterBAK/@backups/_shared/btrbk_VMs_snaps/@VMs.20190909T0648
ERROR: Error while archiving subvolumes, aborting
Archived 1/5 subvolumes

I tried to archive the failing subvolume @VMs.20190909T0648 onto another disk but got the same results.
Could that mean that the source subvolume is corrupted or might it indicate a btrbk or btrfs bug? If it is the former, how would I check its integrity. If it is the latter, how shall we proceed?
Thank you, indeed.

Closing comment (2019-09-14T13:53:39Z): ERROR: lsetxattr o257-128243-0 system.posix_acl_default= failed: No such file or directory

It's related/duplicate of #91, #223, #225, #251 : upstream (btrfs) bug, nothing that can be fixed in btrbk.
Some times deleting the parent subvolume helps, some times deleting the source subvolume helps. I'm afraid I can't give you much more advice on this...
For the record, what kernel / btrfs-progs version do you have?