I was a remote sysadmin for a metal fabrication shop that backed up their CAD/machine files to a NAS daily. They had a recurring issue with their backups randomly failing maybe half a dozen times a month. I was told it was nothing to worry about, we've looked into it, tried everything, etc. Client says it's not a big deal. Until it was.

They had lost some file, of course, and needed to pull it from yesterday's backup. "Ok, sure I'll just restore it for you." Nope, backups failed the previous night. I don't know exactly what it was, but it was a huge deal and I was tasked with figuring it out, that was now my job.

The first thing I did was look at the log history. The only pattern I saw was that backups never failed on Monday nights. Huh.

I have no idea what that could mean, so I move on and write a script to ping the NAS from their server using task scheduler every 5 minutes and write a failure to a log. Maybe it's just offline, I have no idea what the cause of the failure is at this point.

A couple weeks later, the backup fails and I check the log. Sure enough, the NAS dropped off the network overnight, and came back online in the morning. So I call my contact (he was their CAD guy, technical enough to be able to help me check things out) and ask if anything happened overnight, power outage, anything. He isn't aware of anything. The NAS is online, uptime is good, hasn't been power cycled in months.

I have him look at it and there's a MAC address sticker on it so I'm able to trace it back to the switch. Check the switch, sure enough, disconnected during the time shown by my ping log. I have him plug it into a different port, replace both the patch cable and cable connected to the NAS, and disable the previous port. And wait.

The next time it happens, I was able to talk them into buying a new NAS thinking it has to be the NIC on it. It's about 3 years old so it's an easy sell, should probably replace it anyway if it's that important. We ship it out, they pop it in, we transfer the data, and we wait.

Happens again.

So now it this point we are talking about replacing switches, routers, and firewalls. I get 3 different vendors involved. No one is seeing anything out of order, and all their hardware is out of warranty.

At this point, the network has been wiresharked to death and everything looks great, absolute dead end. Customer is not happy about having to potentially spend $10k on network gear we can't even prove is bad so I get routed the on call for this backup failure.

It happens and I drive over there at 3AM.

I arrive, find who is in charge, and they direct me to the network closet. I find out that the NAS is not here. I ask about the NAS. He says oh yeah, that's in a different room.

He beings me there and it's in a closet down a flight stairs from the owner's office with the network cable running under the door. The cable is laying on the floor in the office, end completely stripped.

Turns out, that was on the route to some supply closet that only third shift used. Third shift was 4 10's, Tues-Sat. They were tripping over the cable and the owner was plugging it in when he arrived in the morning. He was out of the loop for the whole thing so had no idea what was going on. He said it didn't seem to affect his "internet" so he never mentioned it.

So what was the hack? I threw a rug on the cable and drove home.

(Yes we did move it later, but there were some network changes out of my control that needed to happen first.)