We study sketching schemes for the cardinality estimation problem in data streams, and advocate for measuring the efficiency of such a scheme in terms of its MVP: Memory-Variance Product, i. e., the product of its space, in bits , and the relative variance of its estimates. Under this natural metric, the celebrated HyperLogLog sketch of Flajolet et al. (2007 ) has an MVP approaching 6(3\ln 2-1)\approx 6.48 for estimating cardinalities up to 2^{64. Applying the Cohen /Ting (2014) martingale transformation results in a sketch Martingale HyperLogLog with MVP \approx 4.16, though it is not composable. Recently Pettie and Wang (2020) proved that it is possible to achieve MVP approaching \approx 1.98 with a composable sketchcalled Fishmonger, though the time required to update this sketch is not constant. Our aim in this paper is to strike a nice balance between extreme simplicity (exemplified by (Martingale) (Hyper)LogLog) and extreme information-theoretic efficiency exemplified by Fishmonger). We develop a new class of "curtain" sketchesthat are a bit more complex than Martingale LogLog but with substantially better }%DIFDELCMD < \MVPs%%% , e. g., Martingale Curtain has MVP \approx 2.31. We also prove that Martingale Fishmonger has   an MVP of around 1.63 , and conjecture this to be an information-theoretic lower bound on the problem, independent of update time.\\%DIF > standard error, we cannot do better than a 2 kilobyte sketch. \Martingale{} \fishmonger{} is neither simple nor practical. We develop a new mergeable sketch called \Curtain{} that strikes a nice balance between simplicity and efficiency, and prove that \Martingale{} \Curtain{} has limiting \MVP\approx 2.31. It can be updated with O(1) memory accesses and it has lower empirical variance than \Martingale{} \LogLog, a practical non-mergeable version of HyperLogLog.
Cardinality estimation is perhaps the simplest non-trivial statistical problem that can be solved via sketching. Industrially-deployed sketches like HyperLogLog, MinHash, and PCSA are mergeable, which means that large data sets can be sketched in a distributed environment, and then merged into a single sketch of the whole data set. In the last decade a variety of sketches have been developed that are non-mergeable, but attractive for other reasons. They are simpler, their cardinality estimates are strictly unbiased, and they have substantially lower variance. We evaluate sketching schemes on a reasonably level playing field, in terms of their memory-variance product (MVP). E.g., a sketch that occupies 5m bits and whose relative variance is 2/m (standard error \sqrt{2/m ) has an MVP . Applying the Cohen /Ting (2014) martingale transformation results in a sketch Martingale HyperLogLog with MVP \approx 4.16, though it is not composable. Recently Pettie and Wang (2020) proved that it is possible to achieve MVP approaching \approx 1.98 with a composable sketchcalled Fishmonger, though the time required to update this sketch is not constant. Our aim in this paper is to strike a nice balance between extreme simplicity (exemplified by (Martingale) (Hyper)LogLog) and extreme information-theoretic efficiency exemplified by Fishmonger). We develop a new class of "curtain" sketchesthat are a bit more complex than Martingale LogLog but with substantially better }%DIFDELCMD < \MVPs%%% of 10. Our contributions are as follows. Cohen and Ting independently discovered what we call the Martingale transform for converting a mergeable sketch into a non-mergeable sketch. We present a simpler way to analyze the limiting MVP of Martingale-type sketches. We prove that the transform is optimal in the non-mergeable world, and that in particular is optimal among linearizable sketches, with an MVP of H_0/2 \approx 1.63 . E.g., this is circumstantial evidence that to achieve 1\\%DIF > standard error, we cannot do better than a 2 kilobyte sketch. \Martingale{} \fishmonger{} is neither simple nor practical. We develop a new mergeable sketch called \Curtain{} that strikes a nice balance between simplicity and efficiency, and prove that \Martingale{} \Curtain{} has limiting \MVP\approx 2.31. It can be updated with O(1) memory accesses and it has lower empirical variance than \Martingale{} \LogLog, a practical non-mergeable version of HyperLogLog.
R
We study sketching schemes for the cardinality estimation problem in data streams, and advocate for measuring the efficiency of such a scheme
Cardinality estimation is perhaps the simplest non-trivial statistical problem that can be solved via sketching. Industrially-deployed sketches like HyperLogLog, MinHash, and PCSA are mergeable, which means that large data sets can be sketched in a distributed environment, and then merged into a single sketch of the whole data set. In the last decade a variety of sketches have been developed that are non-mergeable, but attractive for other reasons. They are simpler, their cardinality estimates are strictly unbiased, and they have substantially lower variance. We evaluate sketching schemes on a reasonably level playing field,
R
its MVP: Memory-Variance Product, i. e., the product of its space, in bits , and the relative variance of its estimates. Under this natural metric, the celebrated HyperLogLog sketch of Flajolet et al. (2007
their memory-variance product (MVP). E.g., a sketch that occupies 5m bits and whose relative variance is 2/m (standard error \sqrt{2/m
D
approaching 6(3\ln 2-1)\approx 6.48 for estimating cardinalities up to 2^{64
R
, e. g., Martingale Curtain has MVP \approx 2.31. We also prove that Martingale Fishmonger has
of 10. Our contributions are as follows. Cohen and Ting independently discovered what we call the Martingale transform for converting a mergeable sketch into a non-mergeable sketch. We present a simpler way to analyze the limiting MVP of Martingale-type sketches. We prove that the
A
transform is optimal in the non-mergeable world, and that
A
in particular is optimal among linearizable sketches, with
R
around
H_0/2 \approx
R
, and conjecture this to be an information-theoretic lower bound on the problem, independent of update time.
. E.g., this is circumstantial evidence that to achieve 1