Really nice. I have more respect for GitLab now. That's a great write-up, and it led me to read some of their other nice reports too.

It's not exactly new for NFS to have cache coherency "surprises". But it should have "close-to-open" coherency at least, and the bug found by GitLab fails even that.

Here's an anecdote.

A Mac client talking to Samba on Linux. The client deletes random files that the client isn't even looking at, but which happen to be changed on the server around the time the client looks at the directory containing those files.

I am not joking. Randomly deleting files it's not even reading.

It delayed a product rollout for about 8 months. I was sure there must be a flaw in some file-updating code, somewhere in application code running on Linux. What else would make update-by-rename-over files disappear once every few weeks? Surely the usual tmpfile-fsync-rename dance was durable on Linux, on ext4? It must have been a silly, embarrasing error in the application code right? Calling unlink() with the wrong string or something.

But no, application was fine. Libraries were fine. And the awful bugs in VMware Fusion's file sharing were not to blame this time. (Ahem, another anecdote...)

It only happened every few weeks. A random file would disappear and be noticed. A web application would be told to update a file, and it'd spontaneously complain that the file was gone. It wasn't reproducible until we went all-out on trying to make it happen more often. But they kept disappearing.

Things like invoices data files and edited documents. Once every few weeks for no obvious reason. Not happy. And not safe to deploy.

Eventually, we found a very old bug in Emacs which deletes the file that's being saved in rare circumstances that only manifest when file attributes change at the wrong moment, which does happen with the weird and wonderful Mac SMB client's way of caching attributes. We thought we'd found the cause with great relief, and could proceed to rollout. Until after a few weeks, another file disappeared. No!

It took weeks of tracing, reproducing, and learning new debugging tools (like auditd running permanently) to rule out faults in (1) the application code and libraries, (2) Linux itself, (3) Samba, (4) tools used on the Mac when viewing a directory, and viewing and editing files.

Nope, it wasn't a bug in application code after all. There weren't any faulty calls or wrong strings. Logging would have caught them. Linux rename() was fine, not to blame. It wasn't a durability problem on power loss (the reason you need fsync with rename). Nor VMware disk image snapshots, even though other bugs were spotted with those. Nor was it the Emacs bug although that was a surprise to find.

The reproducer turned out to be "run cat a lot on the Mac, on a file which isn't being changed at all, while repeatedly updating another file on Linux in the same directory, using rename to update. Watch the updated file disappear eventually".

auditd showed Samba was doing the deletes, so I suspected a crazy bug in Samba and had to work quite hard to convince myself Samba was only doing what it was told by the client. I hoped it was Samba, because that's open source and I can fix that.

No, it was an astonishingly crappy bug called "delete random files once in a blue moon, hahaha!" in the Mac SMB client, which happened to occasionally be used to look in the same directory, which happened to be shared over Samba for convenience to look at it.

The confirmation of cause was from watching the SMB protocol, looking at Samba logs set to maximum verbosity, and lots of reading.

atq2119 says: "Imagine this same scenario if GitLab was using a closed source operating system. Would they have been able to track this down?"

I think I've had an experience like that - the above bug in the Mac SMB client. (Seriously, deleting random files.)

Googling reveals similar-sounding bugs at least two versions of OSX later. Yuck. I have no idea how to meaningfully get these things fixed or usefully reported. And I've had enough to stop caring anyway. The workaround is "force it to use SMB v1" (ye olde anciente). I can imagine the cause is something trivial in directory caching; it's probably just a few lines to fix.

I'm certain if the Linux client had a bug like that, it would be fixed very quickly, and probably backported by the big distros. I'm certain a Linux SMBFS developer would have been very helpful. And, there's a fairly good chance I could have fixed it myself and submitted the patch - probably less work than finding the cause, in this instance.

As it is, I don't think I could have found the culprit if I couldn't look at the Samba source to understand in detail what was going on in the SMB network protocol, or if I didn't have excellent tracing tools in Linux to find which process was responsible for stray deletions (i.e. not my application code, but Samba, which was doing as requested).