A recent paper on benchmarking eventual consistency showed that when a constant workload is applied against Cassandra, the staleness of values returned by read operations exhibits interesting but unexplained variations when plotted against time. In this paper we reproduce this phenomenon and investigate in greater depth the low-level mechanisms that give rise to stale reads. We show that the staleness spikes exhibited by Cassandra are strongly correlated with garbage collection, particularly the "stop-the-world" phase which pauses all application threads in a Java virtual machine. We show experimentally that the staleness spikes can be virtually eliminated by delaying read operations artificially at servers immediately after a garbage collection pause. In our experiments this yields more than a 98% reduction in the number of consistency anomalies that exceed 5ms, and has negligible impact on throughput and latency. In this paper, we study the staleness of values returned by read operations applied to Cassandra-an open-source distributed storage system that supports eventual consistency using quorum-based replication. More precisely, we investigate the causes of sharp spikes that occur when a constant workload is applied to Cassandra and the staleness metric is plotted as a function of time. Similar anomalies have been observed by Wada et al., Bermbach et al., and Rahman et al., leading to speculation regarding possible causes including side effects of caching and DDoS countermeasures, as well as network jitter [2, 5, 7] . Following the methodology of [5] we run experiments in a single private data center, which isolates the storage system from the effects of caching layers, DDoS countermeasures, and ambient network traffic. None of the mechanisms suggested in prior work account adequately for staleness spikes in such a controlled environment. Our specific contributions with respect to finding the cause of consistency anomalies in Cassandra are as follows: • First, we reproduce the staleness time series experiment of Rahman et al. (see Figure 3 in [5] ) in our data center using a more recent version of Cassandra. Our experiments use an improved consistency metric that is more resilient against clock skew, and hence demonstrate that the staleness spikes are not artifacts caused by the use of one specific metric. • Next, we formulate the hypothesis that the observed spikes are caused by processing delays, specifically delays due to the Java virtual machine's garbage collection (GC) "stop-the-world" (STW) pause. • We test our hypothesis experimentally by plotting both the GC pauses and consistency spikes against time. Our results exhibit strong correlations between the times of occurrence of the GC pauses and consistency spikes, as well as between the durations of the GC pauses and the heights of the spikes. • Guided by insights gained from our experiments we propose a method of removing the staleness spikes by delaying read operations artificially at servers in the time period immediately following a GC pause. This technique yields more than a 98% reduction in the number of consistency anomalies that exceed 5ms, and has negligible impact on throughput and latency. EXPERIMENTS The experiments presented herein use the Γ (gamma) metric for consistency [3] , which is similar theoretically to the metric used in [5] but in practice exhibits less noise in environments where clock skew across hosts exceeds operation latencies. The Γ metric quantifies how badly the consistency observed by clients deviates from linearizability [4] , which states (informally speaking) that each operation appears to take effect instantaneously at some point between its start and finish times as measured from the perspective of the client who applied the operation. Our experiments measure a fine-grained form of the Γ metric called the per-value Γ score, which captures deviations from linearizability associated with a collection of operations that access the same 810
The use of server-side artificial delays is explored in REF as a technique for reducing the severity of consistency anomalies in Cassandra when client-side consistency level one is used.
Understanding the Causes of Consistency Anomalies in Apache Cassandra
PVLDB
PVLDB
Computer Science