Ask HN: Is there a data loss bug lurking in MS365 backup solutions?
It sounds crazy, and maybe I'm just doing something dumb, but I've seen a similar issue in two different MS365 backup products this year. I can't reproduce it reliably, but feel like there could be a serious issue, even though I can't prove it.

My issue is specific to OneDrive. When I reconcile a backup set against live data, files are missing. I've had it happen with both Veeam Backup for MS365 and Synology Active Backup for MS365. Neither system reports any issues when backups run. I don't know the cause and I can't reproduce it, but it happens consistently for at least one of my tenants and seems to get worse over time. I've seen the issue on more than one tenant, so I don't think it's anything tenant specific and the only (untenable) solution I've come up with is to restart the backup from scratch.

The tenant that has the most issues is a business with about 200k files. The business owner owns the files and everyone else has access to a few shared folders near the top of the hierarchy. They have about 350GB of data which ends up being about 1TB of quota after versioning.

I originally ran into the issue with Veeam by randomly spot checking 1-2 files every once in a while and running into missing data by random chance. That made me realize I needed to do some kind of bulk reconciliation on a regular basis. I gave up on Veeam because they append the OneDrive file version to all restored files and it makes it difficult to reconcile. For example, locally restored files end up with ' (ver 2)' or similar appended to the file name.

I switched to the Synology system because it's ideal to reconcile since an up-to-date backup set can be shared via SMB. That makes it possible to have an up-to-date OneDrive sync and a mapped drive to an up-to-date backup set on the same machine. After that, it's a matter of comparing two folders as long as care is taken to get a consistent point-in-time for both sets of data.

The only noteworthy thing that I think plays a part is how frequently the tenant reorganizes their data. They're always renaming and moving files and folders to keep things organized. I'd frame it as being frequent, but not unreasonable. The reason I think this is noteworthy is that in cases where I'm able to track the life-cycle of missing files, they seem to "disappear" after being impacted by a directory rename or move operation.

I can't engage with support for this particular tenant because the data is supporting documentation for government work. I can't even share examples or screenshots with the file names AFAIK.

I've seen people complaining about similar issues, but the complaints are from years ago [1]. This [2] caught my eye.

> The root cause for the missing data was due to incorrect representation of the changes from the SharePoint API side.
Veeam RnD team performed an investigation and found that sometimes the SharePoint API mechanism of tracking changes did not track the changes inside the Child files or Folders inside the Site`s list.

Does anyone here reconcile their OneDrive backups well enough to say you're confident you can reliably restore your data with 100% consistency?

Is it possible there's a change tracking bug lurking in the SharePoint API? I don't know anything about how it works, so any insight would be useful. For example, would the OneDrive client and backup clients use the same change tracking? That seems unlikely based on what I'm seeing, but, again, I have no idea how it actually works.

1. https://forums.veeam.com/veeam-backup-for-microsoft-365-f47/...

2. https://forums.veeam.com/veeam-backup-for-microsoft-365-f47/...