Recently on a trip to Germany which I went on, I took a flight from Heathrow Airport in London to Stuttgart International Airport. The flight was ok I suppose, although I’ve never been on a German airline before, and seeing the exit and gas mask instructions in German just made me feel like I was in some kind of Rammstein music video. Or Das Boot. It was strange enough having the cabin crew showing the exits and correct procedure for gas masks, but in German. I was half expecting the food trolleys to be filled with sausages and cold meats.
Anyways, this is probably my fault for having a British upbringing, but I swear the concept of queueing in an orderly fashion is completely lost as soon as you go past the UK border, and I’m very much used to patiently waiting in line for my turn. Like in Alton Towers or Thorpe Park, you wait patiently in line for your turn, for me this is a fundamental concept.
So here I was at passport control having touched down on a horrendously rainy Sunday at Stuttgart airport having touched down at about 5 o’clock German time. By the time I got to the arrivals entrance, it was about 6:30. Why so long? Because everybody barges in. Now it may be naivety on my part (or mein teil in German, learnt that from Rammstein), but surely when you get to the passport control you need to form an orderly queue. Obviously not in Germany. At first I didn’t notice that it was a bum rush for the kiosk like a free autograph signing for Pamela Anderson, but I only seemed to realise how much farther the kiosk was getting. It was only when I noticed that some guy who looked like a fat Oliver Kahn ,who was definitely behind be when entering the terminal, somehow was like 5 people ahead of me, and that the large crowd behind me had mysteriously dissipated. It’s not the fact that everyone in Europe barges in, it’s the fact that because they don’t queue and that’s it accepted, that they’re so subtle about it.
Is it a manners thing? That here in the UK we are taught decorum? Probably, but never have I seen it so vehemently applied. But what was I expected to do? Climb over and barge aside elderly folk? No I don’t think so. I’ve been in plenty of mosh pits, but I didn’t think applying those skills and experience would be appropriate.
Below are audio snippets for Darren Rolling Keys. You can download the files by right clicking on the song name and selecting "save link as"
We are developing an approach called in-situ model checking to thoroughly check general systems software in a lightweight manner. We’ve made our approach so easy that we have applied it to more than 20 widely used, well tested systems and found nearly a hundred serious errors. Some of these errors can cause unrecoverable loss of an entire file system. Currently we’re collaborating with researchers at Microsoft Research to apply the eXplode approach to distributed systems.
Our goal is to effectively detect these bugs. The main technique we use is model checking. This formal verification technique systematically enumerates the possible execution paths of a distributed system by starting from an initial state and repeatedly performing all possible actions to this state and its successors. This state-space exploration makes rare actions such as machine crashes and network failures appear as often as common ones, thereby quickly driving a system into corner cases where subtle bugs surface.
However,naive application of model checking to large systems is prohibitive because of the huge cost required to write an abstract specification of the checked system. Recent work has developed implementation-level model checkers that check code directly, but these checkers still require an invasive, heavyweight port of the checked system to run inside these model checkers.
Instead of the heavyweight way of creating a simulated environment to run a system, we created an in-situ checking architecture that interlaces the mechanism for comprehensive checking within the checked system. This architecture enables users to check live systems, drastically reducing the overhead and the invasive infrastructure needed. For example, in eXplode (our storage system checker), the checking infrastructure is reduced down to a single device driver, dynamically loadable to a running OS kernel. These drivers are fairly small and easy to write; both the Linux and FreeBSD drivers of eXplode are less than two thousand lines of code. Once such a driver is loaded, eXplode can readily check any storage system that runs inside or above the OS kernel. Compared to the cost of weeks or months to check a system using the old approaches, eXplode only requires minutes, several orders of magnitude reduction.
Using this approach, we have found numerous serious errors in 20 widely used, well-tested systems. For example, we found data-loss errors that can vaporize all user data in 17 storage systems, including three version control software, a database, Linux NFS, ten local file systems, a software RAID, and a popular commercial virtual machine.
MoDist is the first model checker designed for transparently checking unmodified distributed systems running on unmodified operating systems. It achieves this transparency via a novel architecture: a thin interposition layer exposes all actions in a distributed system and a centralized, OS-independent model checking engine explores these actions systematically. We made MoDist practical through three techniques: an execution engine to simulate consistent, deterministic executions and failures; a virtual clock mechanism to avoid false positives and false negatives; and a state exploration framework to incorporate heuristics for efficient error detection.
We implemented MoDist on Windows and applied it to three well-tested distributed systems: Berkeley DB, a widely used open source database; MPS, a deployed Paxos implementation; and PacificA, a primary-backup replication protocol implementation. MoDist found 35 bugs in total. Most importantly, it found protocol-level bugs (i.e., flaws in the core distributed protocols) in every system checked: 10 in total, including 2 in Berkeley DB, 2 in MPS, and 6 in PacificA.
Storage systems such as file systems, databases, and RAID systems have a simple, basic contract: you give them data, they do not lose or corrupt it. Often they store the only copy, making its irrevocable loss almost arbitrarily bad. Unfortunately, their code is exceptionally hard to get right, since it must correctly recover from any crash at any program point, no matter how their state was smeared across volatile and persistent memory.
This paper describes eXplode , a system that makes it easy to systematically check real storage systems for errors. It takes user-written, potentially system-specific checkers and uses them to drive a storage system into tricky corner cases, including crash recovery errors. eXplode uses a novel adaptation of ideas from model checking, a comprehensive, heavy-weight formal verification technique, that makes its checking more systematic (and hopefully more effective) than a pure testing approach while being just as lightweight.
eXplode is effective. It found serious bugs in a broad range of real storage systems (without requiring source code): three version control systems, Berkeley DB, an NFS implementation, ten file systems, a RAID system, and the popular VMware GSX virtual machine. We found bugs in every system we checked, 36 bugs in total, typically with little effort.
This paper shows how to use model checking to find serious errors in file systems. Model checking is a formal verification technique tuned for finding corner-case errors by comprehensively exploring the state spaces defined by a system. File systems have two dynamics that make them attractive for such an approach. First, their errors are some of the most serious, since they can destroy persistent data and lead to unrecoverable corruption. Second, traditional testing needs an impractical, exponential number of test cases to check that the system will recover if it crashes at any point during execution. Model checking employs a variety of state-reducing techniques that allow it to explore such vast state spaces efficiently.
We built a system, FiSC, for model checking file systems. We applied it to four widely-used, heavily-tested file systems: ext3, JFS, ReiserFS and XFS. We found serious bugs in all of them, 33 in total. Most have led to patches within a day of diagnosis. For each file system, FiSC found demonstrable events leading to the unrecoverable destruction of metadata and entire directories, including the file system root directory “/”.
We built a system, FiSC, for model checking file systems. We applied it to three widely-used, heavily-tested file systems: ext3, JFS, and ReiserFS. We found serious bugs in all of them, 32 in total. Most have led to patches within a day of diagnosis. For each file system, FiSC found demonstrable events leading to the unrecoverable destruction of metadata and entire directories, including the file system root directory “/”.
For instructions to compile, build, and use eXplode, Please refer to the README file in the top-level directory after you uncompress the explode tar ball.
The eXplode distribution contains a generic model checker that can be applied to other systems. For details, please refer to the README file under directory mcl.
Real systems are difficult to get right because they must correctly handle a practically infinite number of rare events. For example, file systems must correctly recover from all possible crashes; distributed systems must ensure consistency and liveness despite of a large variety of rare events, such as machine crashes, network partition, message delays, and message loss. The complexity to handle these rare events often leads to corner-case errors that are difficult to test, and once detected in the field, impossible to reproduce.