  HyCoR is a fully-operational fault tolerance mechanism for multiprocessor
workloads, based on container replication, using a hybrid of checkpointing and
replay. HyCoR derives from two insights regarding replication mechanisms: 1)
deterministic replay can overcome a key disadvantage of checkpointing alone --
unacceptably long delays of outputs to clients, and 2) checkpointing can
overcome a key disadvantage of active replication with deterministic replay
alone -- vulnerability to even rare replay failures due to an untracked
nondeterministic events. With HyCoR, the primary sends periodic checkpoints to
the backup and logs the outcomes of sources of nondeterminism. Outputs to
clients are delayed only by the short time it takes to send the corresponding
log to the backup. Upon primary failure, the backup replays only the short
interval since the last checkpoint, thus minimizing the window of
vulnerability. HyCoR includes a "best effort" mechanism that results in a high
recovery rate even in the presence of data races, as long as their rate is low.
The evaluation includes measurement of the recovery rate and recovery latency
based on fault injection. On average, HyCoR delays responses to clients by less
than 1ms and recovers in less than 1s. For a set of eight real-world
benchmarks, if data races are eliminated, the performance overhead of HyCoR is
under 59%.
