The distinct elements problem is one of the fundamental problems in streaming algorithms -given a stream of integers in the range {1, . . . , n}, we wish to provide a (1 + ε) approximation to the number of distinct elements in the input. After a long line of research an optimal solution for this problem with constant probability of success, using O( 1 ε 2 + lg n) bits of space, was given by Kane, Nelson and Woodruff in 2010. The standard approach used in order to achieve low failure probability δ is to take the median of lg δ −1 parallel repetitions of the original algorithm. We show that such a multiplicative space blow-up is unnecessary: we provide an optimal algorithm using O( lg δ −1 ε 2 + lg n) bits of space -matching known lower bounds for this problem. That is, the lg δ −1 factor does not multiply the lg n term. This settles completely the space complexity of the distinct elements problem with respect to all standard parameters. We consider also the strong tracking (or continuous monitoring) variant of the distinct elements problem, where we want an algorithm which provides an approximation of the number of distinct elements seen so far, at all times of the stream. We show that this variant can be solved using O( lg lg n+lg δ −1 ε 2 +lg n) bits of space, which we show to be optimal.
Blasiok REF shows that to boost this probability up to 1 − δ for a given 0 < δ < 1, the standard approach of running O log 1 δ independent instances is actually sub-optimal and gives an optimal algorithm that uses O log δ −1 ǫ 2 + log n bits of space.
Optimal streaming and tracking distinct elements with high probability
ArXiv
ArXiv
Computer Science Mathematics