L EARNING -A UGMENTED D ATA S TREAM A LGORITHMS
A BSTRACT
The data stream model is a fundamental model for processing massive data sets with limited memory and fast processing time.
Recently Hsu et al.
incorporated machine learning techniques into the data stream model in order to learn relevant patterns in the input data.
Such techniques led to the training of an oracle to predict item frequencies in the streaming model.
In this paper we explore the full power of such an oracle, showing that it can be applied to a wide array of problems in data streams, sometimes resulting in the ﬁrst optimal bounds for such problems, and sometimes bypassing known lower bounds without such an oracle.
We apply the oracle to counting distinct elements on the difference of streams, estimating frequency moments, estimating cascaded aggregates, and estimating moments of geometric data streams.
For the distinct elements problem, we bypass the known lower bound and obtain a new space-optimal algorithm.
For estimating the p -th frequency moment for 0 < p <
we obtain the ﬁrst algorithms with optimal space and update time.
For estimating the p -th frequency moment for p > 2 we obtain a quadratic savings in memory, bypassing known lower bounds without an oracle.
We empirically validate our results, demonstrating also our improvements in practice.
I NTRODUCTION
Processing data streams has been an active research ﬁeld in the past two decades.
This is motivated by the increasingly common scenario where the size of the data far exceeds the size of the available storage, and the only feasible access to the data is to make a single or a few passes over the data.
This situation occurs, for example, in network applications with high-speed packets being sent through routers with limited resources.
Other examples include processing internet search logs, sensor networks, and scientiﬁc data streams.
For an introduction to data streams, see, e.g., Muthukrishnan (2005).
Formally, in the data stream model, we assume there is an underlying frequency vector x ∈ Z n , initialized to 0 n , which evolves throughout the course of a stream.
The stream consists of updates of the form ( i, w ) , meaning x i ← x i + w .
Suppose M is such that (cid:107) x (cid:107) ∞ =
max i ∈{ 1 ,...,n } | x i | ≤ M throughout the stream.
At the end of the stream, we are asked to approximate f ( x ) for a function f : Z n → R .
Because of the sheer size of the stream, most algorithms are approximate and randomized.
Typically, such algorithms output an estimate Z for which Pr { (1 − (cid:15) )
f ( x ) ≤ Z ≤ (1 + (cid:15) )
f ( x ) } ≥ 2 / 3 , where the probability is over the randomness used by the algorithm, and not of the input stream, which may be a worst-case stream.
The success probability 2 / 3 can be replaced with any constant greater than 1 / 2 by independently repeating the algorithm a constant number of times and taking the median estimate.
The space complexity of the algorithm is measured in bits and the goal is to use much less than the trivial n bits of space required to store x .
A common difﬁculty in designing algorithms in the data stream model is that the coordinates i for which | x i | is large, i.e., the heavy hitters , are not known in advance.
Such coordinates usually require very accurate estimates in order to achieve an accurate ﬁnal estimate of the function value.
A number of techniques have emerged for identifying such coordinates, such as the Count-Min and CountSketch.
These techniques are often used at multiple scales by subsampling the input coordinates and ﬁnding heavy items in the subsample, and have found applications to estimating frequency moments, cascaded aggregates, earthmover distance, and empirical entropy, among other statistics (Indyk & Woodruff, 2005; Chakrabarti et al., 2006; Jayram & Woodruff, 2009;
Andoni et al., 2009; McGregor et al., 2016).
In many applications, however, the worst-case input is rarely seen; in fact, there is usually a pattern inside the underlying vector.
For instance, in network trafﬁc monitoring, the large or “elephant” ﬂows ﬂuctuate on a daily basis, but the actual source and destination servers of those ﬂows do not vary much.
In our formulation, this implies the value x i ﬂuctuates while whether or not x i is a heavy coordinate remains stable.
Motivated by this as well as related work on learning oracles for algorithm design (Gupta & Roughgarden, 2016; Dick et al., 2017; Balcan et al., 2018a;b), Hsu et al.
(2019) introduced the notion of a heavy hitter oracle into the data stream model.
The heavy hitter oracle, trained on past data, receives a coordinate index i and returns a prediction of whether x i will be heavy or not.
They showed that such an oracle simpliﬁes classical heavy hitter algorithms such as C OUNT -M IN and C OUNT -S KETCH .
We note that similar oracles were also studied in previous work, e.g., membership oracles for Bloom ﬁlters by Kraska et al.
These works demonstrate the feasibility of creating such an oracle using machine learning techniques.
Recently, learned oracles for low rank approximation, which can be viewed as an analogue of heavy hitters for heavy directions, was explored by Indyk et al.
The work of Hsu et al.
(2019) is the starting point of our work.
While Hsu et al.
(2019) showed applications of the heavy hitter oracle to frequency estimation, the full power of such an oracle remained largely unexplored.
Our main goal is to understand if such an oracle is more broadly useful, and whether it can give memory and time improvements to the large number of data stream problems mentioned above.
Accurately estimating the heavy hitters in a vector is the bottleneck of existing algorithms for numerous data stream problems, and one cannot help but ask:
Can a heavy hitter oracle be applied to a large number of problems in the data stream model to obtain signiﬁcantly improved bounds for these problems?
We consider estimating the following common functions f ( x ) in the data stream model:
• Distinct Elements: f ( x ) = (cid:107) x (cid:107) 0 , where (cid:107) x (cid:107) 0 is the number of nonzero coordinates of x , deﬁned as (cid:107) x (cid:107)
= |{ i : x i (cid:54) = 0 }| .
This quantity is useful for detecing denial of service attacks and database query optimization (Kane et al., 2010b).
• the F p -Moment: f ( x ) = (cid:107) x (cid:107) pp , where (cid:107) x (cid:107) p is the usual (cid:96) p -norm of x , deﬁned as (cid:107) x (cid:107) p
=
( (cid:80) i | x i | p ) 1 /p .
For 0 < p < 2 , these are often more robust than the Euclidean norm (Indyk, 2000).
For p > 2 these are used to estimate skewness and kurtosis (Indyk & Woodruff, 2005).
• the ( k, p ) -Cascaded
Norm: in this problem x is an n × d matrix that receives updates to its individual entries, and f ( x ) = (cid:107) x (cid:107) k,p :=
( (cid:80) i ( (cid:80) j | x ij | p ) k/p ) 1 /k .
These are useful for executing multiple databse queries (Cormode & Muthukrishnan, 2005; Jayram & Woodruff, 2009).
These problems have also been studied for non-numerical data, such as geometric data.
For example, the F p -moment problem was studied on a stream of rectangles in Tirthapura & Woodruff (2012).
One can think of an underlying ∆ d -dimensional vector x , identiﬁed with a d -dimensional rectangle { 1 , . . . , ∆ } d .
Without loss of generality, we assume that ∆ is a power of 2 .
Each item in the stream is a pair of the form ( R, w ) , where R is a d -dimensional rectangle and w is an integer, meaning
Problem
Result Type
Previous Result
This work
[Table]
S
O ( 1 (cid:15) 2 log n (log 1 (cid:15) + log log n ))
O ( 1 (cid:15) 2 log n log 1 (cid:15) )
Kane et al.
(2010b)
Theorem
F p Moment ( p > 2)
S (cid:101) O ( n 1 − 2 /p ) (cid:101) O
n 1 / 2 − 1 /p )
e.g., Andoni et al.
Theorem
F p Moment
Fast Update ( p < 2)
Update
[Table]
) expected amortized O (1)
Reporting T
O ( 1 (cid:15) 2 log
[Table]
[Table]
log log
[Table]
O ( 1 (cid:15) 2 )
Kane et al.
Theorem
Cascaded Norm
S (cid:101) O ( n 1 − 2 /k d 1 − 2 /p ) (cid:101) O
n 1 − 1 /k − p/ (2 k ) d 1 / 2 − 1 /p )
( k ≥ p ≥ 2)
Jayram & Woodruff (2009)
Corollary
Rectangle
[Table]
S
[Table] [Table]
poly( d(cid:15)
[Table]
(cid:101) O (∆
[Table] [Table]
poly
[Table] [Table]
Moment ( p > 2)
Tirthapura & Woodruff (2012)
Theorem
[Table]
x i ← x i + w for all points i ∈ R .
The goal is to estimate f ( x ) = (cid:107) x (cid:107) pp .
This is called the Rectangle
F p -Moment problem, which occurs in spatial data streams and constraint databases.
Our Results We show that a heavy hitter oracle can greatly improve the complexity of a wide array of commonly studied problems in the data stream model, leading to the ﬁrst optimal bounds for several important problems, and shattering lower bounds that have stood in the way of making further progress on important problems.
We note that not only do we give new algorithms, we also give several lower bounds for algorithms equipped with the oracle, that show optimality of our algorithms even with an oracle.
Our algorithms not only give practically better, theoreticallygrounded improvements to these problems with an oracle, they also shed light on what exactly the difﬁculties are of making further progress on data stream problems without an oracle.
We consider both perfect oracles and oracles that may sometimes make mistakes.
We summarize in Table 1 our improvements to existing algorithms using appropriate heavy hitter oracles, whose conditions are summarized in Table 2.
Throughout, we make the conventional assumption that m, M = poly( n ) .
Distinct Elements: For the problem of estimating (cid:107) x (cid:107) 0 , we improve the prior memory bound of
O ( 1 (cid:15) 2 log n (log 1 (cid:15) + log log n )) to O ( 1 (cid:15) 2 log n log 1 (cid:15) ) .
For constant (cid:15) , this gives an optimal O (log n ) bits of memory, breaking a recent Ω(log n log log n ) bit lower bound shown in Woodruff & Yang (2019).
Examining the lower bound argument in Woodruff & Yang (2019), the hard instance is precisely when there are many items of large frequency, namely, the product of the ﬁrst small number of primes.
This prevents hashing-based algorithms, such as the one in Kane et al.
(2010b), from working with O (log n ) bits of space, since hash values are typically taken modulo a small prime and such heavy items necessitate the use of a large enough prime.
Surprisingly, we show that with a heavy hitter oracle, we can separately handle such items and thus bypass this lower bound.
We note that our O (log n ) -bit algorithm is optimal even given a heavy hitter oracle; see the Ω(log n ) lower bound in Alon et al.
(1999), which holds even if all item frequencies are in the set { 0 , 1 , 2 } .
F p -Moment Estimation, 0 < p < 2 : There is a long line of work on this problem with the best known bounds given in Kane et al.
(2011), which achieve an optimal O ( (cid:15)
− 2 log n ) bits of space, and O (log 2 (1 /(cid:15) ) log log 1 /(cid:15) ) time to process each element.
The existing algorithm is based on separately handling those elements i for which | x i | p ≥ (cid:15) 2 (cid:107) x (cid:107) pp , i.e., the heavy hitters, from the remaining “light items”.
Although we can simply store the heavy hitters given a heavy hitter oracle, unfortunately, the large O (log 2 (1 /(cid:15) ) log log 1 /(cid:15) ) processing time in the algorithm of Kane et al.
(2011) also comes from running a fast multipoint evaluation on the light elements.
We observe that if we instead train two oracles , one for detecting heavy hitters, and one for detecting items x
i with | x i | p ≥
[Table]
x
[Table]
pp , then we can also separate out “medium items”, i.e., items that are detected with this oracle but not our heavy hitter oracle.
There are only
O ( (cid:15) − 2 log 2 (1 /(cid:15) ) log log(1 /(cid:15) )) medium items and so assuming log n ≥ poly(log(1 /(cid:15) )) , one could perfectly hash all such items and maintain their hashed identities in the optimal O ( (cid:15) − 2 log n ) bit of space.
However, one cannot maintain their counts, since each count is O (log n ) bits, and approximate counts do not work if the stream can increment or decrement the coordinate values x i .
This is too much memory, as it would lead to O ( (cid:15) − 2 (log n ) log 2 (1 /(cid:15) ) log log(1 /(cid:15) )) bits of space, even worse than without the oracle.
We could also maintain a so-called p -stable random variable for each medium item, and use it to estimate the F p -moment of the medium items, but this would again be
O ( (cid:15) − 2 (log n ) log 2 (1 /(cid:15) )
log log(1 /(cid:15) ))
bits of space to store one random variable per medium item.
We instead observe that most of the p -stable random variables are small, and so can be stored with many fewer than log n bits, and therefore globally we can get by with only O ( (cid:15) − 2 log n ) bits in total for estimating the medium items.
Finally, for the light items, the critical observation is that we can sub-sample a
[Equation]
fraction of them, and still accurately estimate their contribution.
But this means we can run the multipoint evaluation algorithm of Kane et al.
(2011) on such items, since while that has update time O
(log 2 (1 /(cid:15) ) log log(1 /(cid:15) )) , on average it is only executed on one out of every log 2 (1 /(cid:15) ) log log(1 /(cid:15) ) items.
In summary, with the aid of our oracles and by separately handling these three types of items, we are able to achieve the optimal O (1) amount of update time per item, while retaining the optimal
O ( (cid:15) − 2 log n ) bits of memory.
F p -Moment Estimation for p > 2 : It is well-known (Bar-Yossef et al., 2004) that a hard input distribution for this problem is when a random coordinate of x has absolute value n 1 /p , and all other coordinates have value 0 or 1 .
In this case, one needs to certify that there is indeed a coordinate of value n 1 /p , as it contributes a constant fraction of (cid:107) x (cid:107) pp .
There is a known Ω( n 1 − 2 /p ) bits of space lower bound for this.
However, a heavy hitter oracle exactly allows us to overcome this, as it can be used to separately identify such items.
Our idea is to separate out every item i for which | x i | p ≥ 1 n 1 / 2 (cid:107) x (cid:107) pp with the oracle, and run an existing F p -estimation algorithm on such items.
Since there are at most n 1 / 2 items, existing algorithms give ( n 1 / 2 ) 1 − 2 /p poly(log n ) = n 1 / 2 − 1 /p poly(log n ) bits of memory for this problem.
For the remaining light elements, one can show that if one subsamples only a 1 /n 1 / 2 fraction of them and then runs an existing F p -estimation algorithm on them, one can use this to accurately estimate their contribution.
This again takes n 1 / 2 − 1 /p poly(log n ) bits of memory, balancing the memory with the other part of our algorithm and giving a quadratic improvement in memory in the case without a heavy hitter oracle.
Overall our algorithm uses O ( n 1 / 2 − 1 /p poly(log n )) bits with an oracle and we can in fact show that this is tight up to logarithmic factors as there exists a lower bound of Ω( n 1 / 2 − 1 /p ) bits provided the same oracle.
We describe our algorithms for cascaded norms and rectangle F p -moments in the appendix.
We conduct experiments for the distinct elements and the F p moment ( p > 2 ) problems, on both real-world and synthetic data, which demonstrate signiﬁcant practical beneﬁts.
P RELIMINARIES
Notation We let [ n ] denote the set { 1 , . . . , n } .
The function lsb( x ) denotes the index of the least signiﬁcant bit of a non-negative integer x .
Space and Time Complexity For streaming algorithms, the efﬁciency is characterized by the space (memory) complexity and the time complexity.
The time is further divided into update time and reporting time.
A data stream algorithm maintains a data structure D while processing the data stream and at the end of the stream, it outputs an approximation Z to f ( x ) .
The space complexity refers to the size, in bits, of D .
The update time refers to the time spent to update the data structure D upon receiving a single stream update.
The reporting time refers to the time to output the approximation
Z at the end of the stream.
Heavy Hitter Oracle We assume access to a heavy hitter oracle, which receives an input i ∈ [ n ] , and outputs whether x i will be a heavy hitter at the end of the stream.
The deﬁnition of heavy hitter varies for different data stream problems.
There are two kinds of oracles.
The ﬁrst kind indicates whether or not | x i | ≥ T and the second kind indicates whether or not | x i | p ≥ 1 T (cid:107) x (cid:107) pp , where T is a pre-determined threshold associated with the problem.
We shall use an oracle of the ﬁrst kind for the distinct elements problem and oracles of the second kind for all other problems.
In the experiments, the heavy hitter oracle receives an i ∈ [ n ] and outputs the predicted value of x i .
The implementation of the ﬁrst kind of oracle is thus straightforward.
For the second kind of oracle, there are at most T coordinates x i satisfying | x i | p ≥ 1 T (cid:107) x (cid:107) pp .
We shall regard the largest T coordinates in the predicted vector x
as the heavy hitters in our implementation (instead of calculating the (cid:107) x (cid:107) pp of the predicted vector x ), which is in conformity with the implementation in the prior experiments of Hsu et al.
D ISTINCT E LEMENTS
The current best algorithm for estimating L 0 = (cid:107) x (cid:107) 0 , in the presence of additions and deletions to coordinates of x , is due to Kane et al.
(2010b).
A key component of their algorithm is a constantfactor approximation algorithm, called R OUGH
L0E STIMATOR , which returns a value (cid:101)
L 0 satisfying 1110 L 0 ≤ (cid:101) L 0 ≤ L 0 with probability at least 2 / 3 .
The space is O (log n log log( mM )) bits and the update and reporting times are O (1) .
In this section, we improve the space of R OUGH
L0E
STIMATOR to O (log n ) bits with the help of a trained oracle.
All proofs are postponed to Section A.
The algorithm ﬁrst picks a pairwise hash function h : [ n ] → [ n ] and subsamples n coordinates at log n scales, where the j -th scale consists of the coordinates i for which lsb( h ( i ))
=
j .
In each scale, it uses a small-space counting algorithm E XACT C OUNT to obtain, with probability at least 1 − η , the exact value of (cid:107) x (cid:107)
at that scale.
Then it ﬁnds the deepest scale, i.e., the largest j , with (cid:107) x (cid:107) 0 above some ﬁxed constant threshold, and returns (cid:101) L 0 =
2 j as the estimate.
If such a j does not exist, it returns (cid:101) L 0 = 1 .
The E XACT C OUNT procedure in Kane et al.
(2010b) uses a hash function h (cid:48) :
[ n ] → [ B ] , where
B = Θ( c 2 ) for some constant c for which h (cid:48) is a perfect hashing of a set of size at most c .
Each bucket (cid:96) ∈
[ B ] is a counter which keeps track of ( (cid:80) h ( i )=
(cid:96) x i ) mod p for some large prime p .
The estimate is the number of nonzero buckets, and E XACT C OUNT returns the maximum estimate of Θ(log(1 /η )) parallel repetitions.
Now, with the trained oracle, we can pick p to be a smaller poly(1 /(cid:15) ) value in E XACT C OUNT .
In the higher-level R OUGH
L0E
STIMATOR algorithm, whenever an update is made to a heavy coordinate i identiﬁed by the oracle, the corresponding bucket inside E XACT C OUNT is marked as nonempty regardless of the counter value, since we know the heavy item will never be entirely deleted, since by deﬁnition it is heavy at the end of the stream (note this is not true of non-heavy elements).
In the end, R OUGH L0E
STIMATOR ﬁnds the deepest scale with at least one nonempty bucket.
The following is our new guarantee of E XACT C OUNT .
Lemma 1.
Assume that L 0 ≤ c and suppose that B = Θ( c 2 ) .
The subroutine E XACT C OUNT returns the exact value of L 0 with probability at least 1 − η using O ( B log(1 /(cid:15) ))
bits of space, in addition to storing O (log(1 /η )) independently chosen pairwise-independent hash functions from [ n ] to [ B ] .
The update and reporting times are O (1) .
Now we specify some details of the algorithm R OUGHT
L0E STIMATOR .
It chooses c = 141 and η = 1 / 16 for E XACT C OUNT , and the E XACT C OUNT algorithms for different scales all share the same O (log(1 /η )) hash functions.
The following is our new guarantee.
Theorem 2.
The algorithm R OUGH L0E
STIMATOR outputs a value (cid:101) L 0 , which satisﬁes 1110 L 0 ≤ (cid:101)
L 0 ≤ L 0 with probability at least 13 / 16 .
The space complexity is O (log( n ) log(1 /(cid:15) )) bits, and the update and reporting times are O (1) .
With a constant-factor estimate R to L 0 , the high-level idea is to subsample each index in [ n ] with probability K/R .
Then the number of distinct surviving items is Θ( K ) .
To estimate the number of distinct surviving items efﬁciently, one hashes them into Θ( K ) buckets and counts the number
T of non-empty buckets, which is simulated by updating a bit vector of length Θ( K ) .
The ﬁnal estimate can be deduced from the number of distinct surviving items, which can be shown to be a (1 + eps ) -approximation to F 0 when K = Θ(1 /(cid:15) 2 ) .
Since we do not know L 0 in advance, we have to guess its value using log n geometrically increasing values, and so our memory can be thought of as a bucket matrix A of dimension (log n ) × K .
In parallel we run the R OUGH L0E
STIMATOR above, so at the end of the stream, we will obtain a constant-factor estimate R , allowing us to look into the appropriate row of A (the appropriate buckets).
A problem is how to efﬁciently maintain a bucket.
In Kane et al.
(2010b), each bucket stores the dot product of frequencies hashing to it with a random vector over a large ﬁnite ﬁeld of order poly( K log( nM )) , using O (log K +log log( nM ))
=
O (log(1 /(cid:15) )
+ log log( nM ))
bits.
Now, using a similar argument to the proof of Lemma 1, with the help of our heavy hitter oracle, we can mark the buckets containing a heavy hitter as nonempty and reduce the order of the ﬁnite ﬁeld to O (poly(1 /(cid:15) )) .
This improves the space complexity of each bucket to O (log(1 /(cid:15) ))
bits.
Combined with the complexity of R OUGH
L0E
STIMATOR , we conclude with the following theorem.
Theorem 3.
There is an algorithm for (1 ± (cid:15) )
-approximating
L 0 using space O ( (cid:15) − 2 (log n ) log(1 /(cid:15) )) with success probability at least 2 / 3 , and with O (1) update and reporting times.
F p ESTIMATION , p >
We start with a well-known algorithm for estimating (cid:107) x (cid:107) pp in a stream.
Lemma 4 (Precision Sampling, Andoni et al.
There is a randomized algorithm, called the Precision Sampling algorithm, which returns an estimate X to (cid:107) x
(cid:107) pp such that it holds with probability at least 0 .
9 that (1 − (cid:15) )
(cid:107) x (cid:107) pp ≤ X ≤ (1 + (cid:15) )
(cid:107) x (cid:107) pp .
The algorithm uses B ( n ) =
O ( p 2 (cid:15) − 2 − 4 /p n 1 − 2 /p log( n ) log( M )) bits of space.
In the rest of the section, we assume that there is an oracle which can tell us if | x i | p ≥ 1 s (cid:107) x (cid:107) pp is a heavy hitter.
The main idea is that we separately estimate the F p -moment of the heavy hitters, and for the remaining light elements, we use sub-sampling to estimate their contribution with sampling rate 1 /ρ .
Let I H , I L ⊆
[ n ] denote the subsets of indices of the heavy hitters and the light elements, respectively.
Since | I H | ≤ s , we can use an F p estimator to (1 + (cid:15) )
-approximate (cid:107) x
I H (cid:107) pp with space
B ( s ) .
For the light elements, we use a sub-sampling algorithm: we set a sampling rate 1 /ρ , and for each item i ∈ I L , with probability 1 /ρ
we choose it.
For each item i we choose, we calculate
F s = (cid:80) | x i | p , and use ρF s to estimation (cid:107) x I L (cid:107) pp .
By a Chernoff bound, with high probability, we sample at most 3 n/ρ elements.
To analyze it, we let Y = ρF S , and we have the following lemmas.
All proofs in this section are postponed to Section B.
Lemma 5.
E [ Y ] = (cid:107) x I L (cid:107) pp , Var [ Y ] ≤ ρs (cid:107) x (cid:107) 2 pp .
Lemma 6.
If we repeat the sub-sampling kρ/s times independently with sub-sampling rate ρ(cid:15) 2 , we can get a (1 ± (cid:15) ) -approximation of F p with probability at least 1 − 1 /k .
If we use the estimator in Lemma 4 in both parts (heavy part and light part), The total space complexity is B ( s ) + (cid:15) − 2 ρs B ( nρ(cid:15) 2 ) .
Letting ρ = s = √ n , we obtain an F p estimation algorithm with space O
(cid:15) − 4 n 1 / 2 − 1 /p log( n ) log( M )) .
We remark that this bound is tight in n up to logarithmic factors as there is a lower bound of Ω( n 1 / 2 − 1 /p ) bits even in the presence of the oracle.
Theorem 7.
Under the assumption of a heavy hitter oracle, we can estimate (cid:107) x (cid:107) pp within a factor 1 ± 2 (cid:15) in O ( (cid:15) − 4 n 1 / 2 − 1 /p log( n ) log( M )) bits with success probability at least 3 / 5 .
The above analysis is based on the assumption that the oracle is perfect.
In practice, sometimes the oracle makes mistakes.
Here we assume that for each item i , the probability the oracle gives us an incorrect prediction is δ .
We have the following theorem for noisy oracles.
Theorem 8.
With a noisy heavy hitter oracle with error probability δ , we can estimate (cid:107) x (cid:107) pp within a factor 1 ± 2 (cid:15) in O ( (cid:15) − 4 n 1 / 2 − 1 /p log( n ) log( M )) bits of space when δ = O (1 / √ n ) , or in O ( (cid:15) − 4 ( nδ ) 1 − 2 /p log( n ) log( M )) bits of space otherwise.
Both guarantees hold with success probability at least 3 / 5 .
E XPERIMENTS 5. D ISTINCT E LEMENTS 5.1. I NTERNET T RAFFIC D ATA
For this experiment, the goal is to estimate the number of distinct packets for each network ﬂow.
A ﬂow is a sequence of packets between two machines on the Internet.
It is identiﬁed by the IP addresses of its source and destination and the application ports.
Dataset : The trafﬁc data is collected at a backbone link of a Tier1 ISP between Chicago and Seattle in 2016 (CAIDA).
Each recording session is around one hour.
Within each minute, there are around 30 million packets (meaning 30 million updates) and 1 million unique ﬂows.
The distribution of packet counts over Internet ﬂows is heavy tailed, see Hsu et al.
Model : We use the prediction results in Hsu et al.
(2019), which predict the logarithm of the packet counts for each ﬂow.
In Hsu et al.
(2019), the authors use two RNNs to encode the source and destination IP addresses separately and additionally use two-layer fully-connected networks to encode the source and destination ports.
Then they concatenate the encoded IP vectors, encoded port vectors, and the protocol type as the ﬁnal features to predict the packet counts.
They use the ﬁrstminutes for training, the following minute for validation, and estimate the packet counts in subsequent minutes.
Results : We plot the estimation error vs. total number of buckets for the 20 th minute in the CAIDA data.
We run the estimation algorithm 5 times and calculate the average estimation error.
The result is shown in Figure 2.
The total number of unique ﬂows appearing in a single test minute is about 1 .
1 × 10 6 .
For every light coordinate, we make an extra update that makes this coordinate 0 with probability 0 .
There are two parts to our algorithm.
For R OUGH
L0E
STIMATOR , we set c = 10 and η = 1 / 4 .
We use the heavy hitter oracle to predict whether the coordinate will be larger than 2 10 .
We randomly select a prime from [11 , 31] for the hash buckets.
The total number of buckets in this part is about 4 × 10 3 (each bucket can store a number in { 0 , 1 , . .
. , 31 } and has an extra bit to indicate whether it has been marked).
For the other part of our algorithm, we create a bucket matrix A of dimension (log n ) × K with K ranging from 2 6 to 2 9 .
Each entry in A can also store a number betweenand 31 and with an additional bit to indicate whether it has been marked.
From the plot we can see that the algorithm achieves an approximation error of about 2 .
5% using at most 1 . 5 × 10 4 buckets, approximately 1 .
5% of the space needed if we were to keep an individual counter for each ﬂow.
5.1. S YNTHETIC D ATA
To further demonstrate the advantage of having a heavy hitter oracle, we run the previous algorithm due to Kane et al.
(2010b) (where we randomly choose a prime p ∈ [11 , 31] as the divisor for each bucket) and our modiﬁed algorithm on the synthetic data designed as follows:
ﬁrst, we generate an input vector x of dimension n = 10 6 with i.i.d entries uniform on { 0 , 1 , . .
Then for each coordinate i and each prime p ∈ [11 , 31] , we multiply x i by p with probability 0 .
We set the threshold to be a heavy hitter to be 2 10 , that is, the oracle reports x
i as a heavy hitter if | x i | > 2 10 .
Results : We use the optimal parameters in the last experiment ( c = 10 , η = 1 / 4 and K = 2 9 ) and vary the total number of buckets from 7000 to 15000 .
The results are shown in Figure 1.
We see that having an oracle signiﬁcantly reduces the estimation error.
[Figure]
F p E STIMATION 5.2. S EARCH Q UERY D ATA
For this experiment, the goal is to estimate the frequency moments of the vector indicating the number of occurrences each search query appears.
Dataset : We use the AOL query log dataset, which consists of 21 million search queries collected from 650 thousand users over 90 days.
There are 3.8 million unique queries.
Each query is a search phrase with multiple words.
The distribution of the search query frequencies is Zipﬁan, see Hsu et al.
Model : We use the prediction result in Hsu et al.
(2019), which predicts the number of times each search phrase appears.
In Hsu et al.
(2019), the authors train an RNN with LSTM cells that takes characters of a search phrase as input.
The ﬁnal states encoded by the RNN are fed to a fullyconnected layer to predict the query frequency.
They use the ﬁrst 5 days for training, the following day for validation, and estimate the number of times different search queries appear in subsequent days.
We use this prediction result as an indicator of which search phrases are the heavy hitters.
Results : We plot the estimation error versus the total number of buckets for the 50 th and 80 th day ( p = 3 or p = 4 ).
For the same data, we run the estimator 10 times and calculate the average estimation error.
The results are shown in Figure 3.
For our algorithm, we notice that the total number of search phrases that appear in a single day is about n = 1 . 5 × 10 5 and so √ n < 400 .
We can store the heavy coordinates identiﬁed by the heavy hitter oracle directly.
Given the budget of the number of buckets, we divide them into two parts.
Half of them were used to store the heavy items, and the other half are used by the sub-sampling algorithm with the precision sampling estimators to estimate the frequency moment of the light elements.
In comparison, we also run the classical precision sampling estimators.
Our results show a larger estimation error owing to the constraint of relatively few buckets.
Note that for a ﬁxed bucket budget B tot , there is a trade-off between the number k of repetitions and the number B
= B tot /k of buckets allotted for the hash function in each repetition.
We plot the results for different values of k = 10 , 20 , 30 .
It is clear from the plots that the estimation error of the oracle-aided algorithm is about 1 % even when the total number of buckets is small, demonstrating a strong advantage over classical precision sampling estimators.
5.2. S YNTHETIC D ATA
A number of real-world datasets have the property that the heavy coordinates contribute a large fraction of the F p -moment, as p increases, and so one needs only to obtain a good approximation of the F p -moment of the heavy coordinates.
This is indeed one of the reasons why we achieved low error for search query estimation.
In order to show that our subsampling algorithm also achieves a good approximation for the light coordinates, we considered the following data set.
We generated an input vector x of dimension n = 10 8 with the ﬁrst √ n coordinates being heavy and the remaining coordinates being light.
Each coordinate is a uniform random variable in (0 , 1) , later renormalized such that the heavy and the light parts contribute equally to the total F p -moment, i.e., (cid:107)
x
I H (cid:107) pp = (cid:107) x I
L (cid:107) pp .
Hence, a good approximation of (cid:107) x (cid:107) pp requires a good approximation of both parts.
Results : We plot the estimation error vs. total number of buckets for p = 3 in Figure 4.
For our algorithm and classical precision sampling, we ﬁxed the number k of repetitions to 100 and varied the total number of buckets from 10 6 to 2 × 10 6 ( 1% to 2% of the length of x ).
We repeated the sub-sampling algorithm 9 times.
In each of the 9 repetitions, we also used the precision sampling estimators, repeating 100 times.
From the plots we see that our algorithm achieves signiﬁcantly smaller relative approximation error than the classical precision sampling algorithm for every number of buckets.
In particular, using 2 · 10 6 buckets, i.e., 2% space of what is needed to store the entire vector x , our algorithm achieves a 15% relative estimation error while the classical precision sampling obtains only a 27% relative error.
Another observation is that since subsampling reduces the space consumption, our algorithm has a faster update time when n is large.
A CKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for helpful comments.
Y. Li was supported in part by Singapore Ministry of Education (AcRF) Tier 2 grant MOE2018-T2-1-013. D. Woodruff would like to thank partial support from the National Science Foundation under Grant No. CCF-1815840 and the Ofﬁce of
Naval Research (ONR) under grant N00014-181-2562.
A O MITTED P ROOFS IN S ECTION
Proof of Lemma 1.
If a bucket contains a heavy hitter, it is marked nonempty
and it is indeed nonempty.
It sufﬁces to show that a bucket containing only light coordinates is nonempty modulo p .
Since the light coordinates are at most 2 poly(1 /(cid:15) ) , there are at most c of them by assumption, their sum S is thus at most 2 poly(1 /(cid:15) ) .
This implies that S has at most poly(1 /(cid:15) ) prime factors.
If we choose a random prime p of size poly(1 /(cid:15) ) , then S mod p (cid:54) = 0 with probability at least 1 − poly( (cid:15) ) if S (cid:54) = 0 .
Taking a union bound over Θ( c 2 ) buckets proves the correctness.
The space and time complexity are immediate from the description of the algorithm.
Proof of Theorem 2.
The proof is almost identical to that in Kane et al.
(2010b).
The space and time complexities are follow from the description of the algorithm.
Next we show correctness.
Let L 0 ( j ) denote the true number of distinct elements in the j -th scale.
Then E L 0 ( j ) =
L 0 / 2 j .
Let j ∗ =
max { j : E L 0 ( j ) ≥ 1 } and j ∗∗ = max { j < j ∗ : E L 0 ( j ) ≥ 55 } .
As shown in Kane et al. (2010b), if j ∗∗ exists, it holds that 55 ≤ E L 0 ( j ∗∗ ) <
110 and Pr { 32 <
L 0 ( j ∗∗ ) < 142 } ≥ 8 / 9 .
With our choices of c = 141 and η = 1 / 16 , E XACT C OUNT returns a nonzero value for the j ∗∗ -th scale with probability 8 / 9 − 1 / 16 > 13 / 16 by Lemma 1.
It follows that the deepest scale j we ﬁnd satisﬁes j ∗∗ ≤ j ≤ j ∗ .
Hence, L 0 =
j ∗∗ E L 0 ( j ∗∗ ) ≤ 110 · 2 j and L 0 =
j ∗ E L 0 ( j ∗ ) ≥ 2 j as desired.
If j ∗∗ does not exist, then L 0 < 55 , and (cid:101) L 0 = 1 is a 55 -approximation in this case.
B O MITTED P ROOFS IN S ECTION
Proof of Lemma 5.
We let Y =
ρF s = ρ (cid:80)
i f i | x i | p , where f i =
if i was chosen, and f i =otherwise.
So we have that E [ f i ] = 1 ρ .
It thus follows that E [ Y ] = (cid:107) x I L (cid:107) pp , that is, Y is an unbiased estimate.
Then,
[Equation]
To bound this, we notice that we have the fact that x i < (cid:107) x (cid:107) pp s and p > 2 , so ρ (cid:88) i ∈ I
L
[Equation]
=
[Equation]
So ﬁnally we have that Var [ Y ] ≤ ρs (cid:107)
x
(cid:107) 2 pp .
Proof of Lemma 6.
By Chebyshev’s inequality, we have that
[Equation]
Proof of Theorem 7.
For the heavy hitters, note that | I H | ≤ n/s = √ n .
For this part, we can use 9 B ( √ n ) space to get an H for which | H − (cid:107) x
I H (cid:107) pp | ≤ (cid:15) (cid:107) x
I H (cid:107) pp with probability at least 78 by independently running 9 estimators and then taking the median.
For the light elements, for every sub-sample round i , we independently run O (1) estimators with rate √ n(cid:15) 2 and use their median Z i to be the estimate of this level of sub-sampling.
Let Y i be the true F p -moment of this sub-sample.
Then with probability at least 99100 it holds that | Y i − Z i | ≤ (cid:15)Y i .
Taking a union bound, we get that if we do the sub-sampling 10 times, with probability at least 910 , | Y i − Z i | ≤ (cid:15) (cid:107) x I L (cid:107) pp holds for every i .
Using Lemma 6 we can get that with probability at least 910 , | Z − (cid:107) x I L (cid:107) pp | ≤ (cid:15) (cid:107) x I L (cid:107) pp + (cid:15) (cid:107) x (cid:107) pp .
The overall success probability of this part is at least 45 .
So we get that | H + Z −(cid:107) x (cid:107) pp | ≤ (cid:15) (cid:107) x I L (cid:107) pp
+ (cid:15) (cid:107) x (cid:107) pp
+ (cid:15) (cid:107) x
I H (cid:107) pp = 2 (cid:15) (cid:107) x (cid:107) pp with success probability at least 3 / 5 .
The total space complexity is B ( √ n ) + O (cid:0) (cid:15) − 2 ) B ( √ n(cid:15) 2 ) =
O ( (cid:15) − 4 n 12 − 1 p log( n ) log( M )) .
Proof of Theorem 8.
For an element x i which the oracle indicates as light, we know that with probability 1 − δ , it is truly light.
Combined with Theorem 5, we obtain that
[Equation]
Hence, when δ = O ( 1 √ n ) , Theorem 7 continues to hold.
Otherwise we can let s = nδ and ρ =
(cid:15) 2 δ − 1 , and the space complexity becomes O ( (cid:15) − 4 ( nδ ) 1 − 2 p log( n ) log( M )) bits.
Lower Bound
The upper bound in Theorem 7 is asymptotically tight in n up to logarithmic factors, since we have the following matching lower bound even in the presence of an oracle which indicates whether | x i | > 1 √ n (cid:107) x (cid:107) pp .
Theorem 9.
Suppose that 1 / √ n < (cid:15) < 1 / 4 and that a heavy hitter oracle indicates whether | x i | >√ n (cid:107) x (cid:107) pp for the whole underlying vector x .
Any randomized streaming algorithm that estimates (cid:107) x (cid:107) pp with a factor (1 ± (cid:15) ) with probability ≥ 9 / 10 requires Ω( (cid:15) − 2 /p n 1 / 2 − 1 /p ) bits of space.
Proof of Theorem 9.
We assume √ n to be an integer and deﬁne our hard instance as follows: for every 1 ≤ i ≤ √ n , we let x i = 1 with probability 1 − 1 / √ n , and x i = (4 (cid:15) )
1 /p n 1 / 2 p with probability 1 / √ n .
And for i > √ n , x i =
x i mod √ n .
In this case, (cid:107) x (cid:107) pp ≥ n , so any coordinate will not be a heavy hitter.
We note that if our algorithm can outputs an (1 ± (cid:15) ) approximation of (cid:107) x (cid:107) pp , it can also output an (1 ± (cid:15) ) approximation of (cid:107) x [ √ n ] (cid:107) .
To prove our lower bound, we will use the (cid:96) k ∞ communication problem in Bar-Yossef et al.
(2004): there are two parties, Alice and Bob, holding vectors a, b ∈
Z m respectively, and their goal is to decide if (cid:107) a − b (cid:107) ∞ ≤ 1 or (cid:107) a − b (cid:107) ∞ ≥ k .
From Bar-Yossef et al.
, we can get an Ω( m/k 2 ) bits lower bound.
Now we claim that: if we set m = √ n , k = (4 (cid:15) )
1 /p n 1 / 2 p and make a − b = x [ √ n ] , then any streaming algorithm outputs a (1 ± (cid:15) ) approximation Y of (cid:107) x [ √ n ] (cid:107) pp can be used to build a communication protocol for solving this (cid:96) k ∞ problem with communication proportional to the algorithm’s space complexity.
In this case, if (cid:107) a − b (cid:107) ∞
= 1 , then we have Y ≤ (1 + (cid:15) )
(cid:107) x [ √ n ] (cid:107) p p ≤
(1 + (cid:15) ) √ n .
Otherwise, Y ≥ (1 − (cid:15) )
( √ n − 1 + 4 (cid:15) √ n ) > (1 + (cid:15) ) √ n .
By this reduction, we can ﬁnally get an Ω( m/k 2 )
=
Ω( (cid:15) − 2 /p n 1 / 2 − 1 /p ) bits space lower bound under the oracle assumption.
C F AST M OMENT E STIMATION , 0 < p <
When 0 < p < 2 , the F p estimation problem can be done in an optimal O ( (cid:15) − 2 log n ) bits of space.
Here we show how to improve the update time while maintaining this amount of space.
In the rest of the section, we assume there are two oracles: one can tell us whether | x i | p ≥ (cid:15) 2 (cid:107) x (cid:107) p p , and the other can tell us whether | x i | p ≥ (cid:15) 2 log 2 (1 /(cid:15) )
loglog(1 /(cid:15) )
(cid:107) x (cid:107) pp .
We call these 3 categories of elements heavy elements, medium elements, and light elements.
Let I H , I M , I L ⊆
[ n ] denote the subset of indices of the heavy elements, medium elements, and the light elements, respectively.
For the heavy elements, we can use O ( (cid:15) − 2 log( M )) space to store them.
We now turn to the other two parts.
For the light elements, we will use the LightEstimator in Section 3 in Kane et al.
Lemma 10 (Kane et al.
Suppose we are given 0 < (cid:15) < 1 , and given a list L ⊆ [ n ] such that i ∈ L
if and only if | x i | p ≥ (cid:15) 2 (cid:107) x (cid:107) pp .
There is an algorithm LightEstimator which returns an estimate X to (cid:107) x n \ L (cid:107) pp such that | X − (cid:107) x n \ L (cid:107) pp | ≤ (cid:15) (cid:107) x (cid:107) pp with probability at least 0 .
The space usage is O ( (cid:15) − 2 log( nmM )) , the amortized update time is O (log 2 (1 /(cid:15) ) log log(1 /(cid:15) )) , and the reporting time is O (1 /(cid:15) 2 ) .
For the light elements x i , note we have the condition | x i | p ≤ (cid:15) 2 log 2 (1 /(cid:15) )
loglog(1 /(cid:15) )
(cid:107) x (cid:107) pp .
Using
Lemma 5 Lemma 6 and the argument in Theorem 7, if we use the LightEstimator to do the subsampling algorithm with rate ρ = log 2 (1 /(cid:15) ) log log(1 /(cid:15) ) for the light elements, we just need to sub-sample O (1) times to get an approximation X of (cid:107) x I L (cid:107) pp such that | X − (cid:107) x I L (cid:107) pp | ≤ (cid:15) (cid:107) x (cid:107) pp with probability at least 0 .
Note that for each update ( i, v ) in the sub-sampling process, if x i is not selected by the sub-sampling algorithm, the update time is O (1) (we do not need to do anything), otherwise, by Kane et al.
we know that the amortized update time is O (log 2 (1 /(cid:15) )
log log(1 /(cid:15) )) .
Notice that in Lemma 5 we just need a pairwise independent hash function for the sub-sampling.
So during the process, the expected amortized update time is O ( log 2 (1 /(cid:15) )
loglog(1 /(cid:15) )
ρ )
=
O (1) .
Theorem 11.
Suppose we are given 0 < (cid:15) < 1 , and given the lists I L .
There is an algorithm which returns an estimate X to (cid:107) x I L (cid:107) pp such that | X − (cid:107) x I L (cid:107) pp | ≤ (cid:15) (cid:107) x (cid:107) p p with probability at least 0 .
The space usage is O ( (cid:15) − 2 log( nmM )) , the expected amortized update time is O (1) , and the reporting time is O (1 /(cid:15) 2 ) .
It remains to handle the medium elements.
Here we will still use the LightEstimator , but do some adjustments.
Before describing our LightEstimator data structure, we ﬁrst deﬁne the p-stable distribution .
Deﬁnition 1 (Zolotarev Zolotarev (1986)) .
For 0 < p < 2 , there exists a probability distribution
D p called the p -stable distribution satisfying the following property.
For any positive integer n and vector x ∈ R n , if Z 1 , ..., Z n ∼ D p are independent, then (cid:80) nj =1 Z j x j ∼ (cid:107) x (cid:107) p Z for Z ∼ D p .
Lemma 12 (Pagh and Pagh, Ostlin & Pagh (2003) Theorem 1.1) .
Let S ⊆
U =
[ u ] be a set of z > 1 elements, and let V =
[ v ] , with 1 < v ≤ u .
Suppose the machine word size is Ω( log ( u )) .
For any constant c > 0 there is a word RAM algorithm that, using time log ( z ) log O (1) ( v ) and
O (log( z ) + log log ( u )) bits of space, selects a family H of functions from U to V (independent of
S ) such that:
With probability 1 − O (1 /z c ) , H is z -wise independent when restricted to S .
Any h ∈ H can be represented by a RAM data structure using O ( z log( v )) bits of space, and h can be evaluated in constant time after an initialization step taking O ( z ) time.
In Kane et al.
(2011), LightEstimator was deﬁned by creating R = 4 /(cid:15) 2 independent instantiations of the estimator D 1 , which we label D 1 1 , ..., D R 1 , and picking a hash function h : [ n ] → [ R ] using
Lemma 12.
Upon receiving an update to x i in the stream, the update was fed to D h ( i ).
The estimator
D i 1 is a slightly modiﬁed version of the geometric mean estimator of Li (2008), which takes a matrix
A ∈ R t × n , where each row contains Ω(1 /(cid:15) p ) -wise p -stable entries and the rows being independent from each other, and maintains y
=
Ax in the stream.
Furthermore, in parallel we run the algorithm of Kane et al.
(2010a) with constant error parameter to obtain a value (cid:101) F p in [ (cid:107) x (cid:107) pp / 2 , 3 (cid:107) x (cid:107) pp / 2] .
The estimator Est p is min { C t,p ( (cid:81) tj =1 | y j | p/t ) , (cid:101) F p /(cid:15) } .
Hence in Kane et al.
(2011), the update time is the time to evaluate a Θ(1 /(cid:15) p ) -wise independent hash function over a ﬁeld of size poly ( nmM ) , which is O (log 2 (1 /(cid:15) ) log log(1 /(cid:15) )) .
Therefore, if we can store the hash values of the p -stable entries for the medium elements, then we can improve the update time to O (1) for this part (note that the parameter t is a constant).
We need the following lemma.
Lemma 13.
Let k
= O ( 1 (cid:15) 2 log 2 1 (cid:15) log log 1 (cid:15) ) and X 1 , X 2 , ...,
X k be p -stable random variables.
With probability at least 0 .
9 , the following holds.
There exists T ⊆
[ k ] such that | T | =
O (1 /(cid:15) 2 )
for which | X i | ≤ poly log(1 /(cid:15) )
for all i (cid:54)∈ T , and consequently, each such X i can be approximated by some (cid:101)
X i satisfying |
X
i − (cid:101)
X i | ≤ poly( (cid:15) ) and each such (cid:101)
X i can be recovered from O (log(1 /(cid:15) ))
bits.
To prove this, we need the following result.
Lemma 14 (Nolan (2003), Theorem 1.12) .
For ﬁxed 0 < p < 2 , the probability density function φ p of the p -stable distribution satisﬁes
φ p ( x ) =
O (1 / (1 + | x | p +1 )) and is an even function.
The cumulative distribution function satisﬁes Φ p ( x ) =
O ( | x | − p ) .
Proof of Lemma 13.
Let I =
[ − log 3 /p
(1 /(cid:15) ) , log 3 /p (1 /(cid:15) )] .
For a p -stable random variable X i , by
Lemma 14, Pr { X i / ∈ I } =
O (1 / log 3 (1 /(cid:15) )) .
So the expected number of X i for which X i / ∈
I is O ( k/ log 3 (1 /(cid:15) ))
=
O (1 /(cid:15) 2 ) .
By Markov’s inequality, with probability at least 0 .
9 , the number of these X i are O
(1 /(cid:15) 2 ) .
For X i contained in I , we can uniformly partition I into subintervals of length poly( (cid:15) ) , and there will be O (poly(1 /(cid:15) ))
many subintervals.
Let (cid:101)
X i be the nearest partition point.
It is clear that X i can be recovered from O (log(1 /(cid:15) ))
bits to store the index of the partition point.
The following lemma tells us that we can use the approximation value given by the p -stable random variables in the Est i .
Lemma 15.
Let k
= O ( 1 (cid:15) 2 log 2 1 (cid:15) log log 1 (cid:15) ) .
Suppose we are given a, b, x ∈ R k and | a i − b i | ≤ (cid:15) q for all i .
Then we have |(cid:104) a, x (cid:105) − (cid:104) b, x (cid:105)| ≤ (cid:15)
q − 2 (cid:107) x (cid:107) p for sufﬁciently small (cid:15) .
Proof.
By the Cauchy-Schwarz inequality, |(cid:104) a − b, x (cid:105)| ≤ (cid:107) a − b (cid:107) 2 (cid:107) x
(cid:107) 2 ≤ √ k(cid:15) 2 q (cid:107) x (cid:107) 2 ≤ (cid:15) q − 2 (cid:107) x (cid:107) 2 ≤ (cid:15) q − 2 (cid:107) x (cid:107) p (recall that p < 2 ).
Finally, we have the following theorem.
Theorem 16.
Suppose we are given 0 < (cid:15) < 1 and the lists I M .
There is an algorithm which returns an estimate X to (cid:107) x
I M (cid:107) pp such that | X − (cid:107) x
I M (cid:107) pp | ≤ 2 (cid:15) (cid:107) x (cid:107) pp with probability at least 0 .
The space usage is O ( (cid:15) − 2 max { log( nmM ) , log 3 (1 /(cid:15) ) log log(1 /(cid:15) ) } ) , the amortized update time is
O (1) , and the reporting time is O (1 /(cid:15) 2 ) .
Proof.
Let k = 1 (cid:15) 2 log 2 1 (cid:15) log log 1 (cid:15) .
Note that there are at most k medium elements.
Using
Lemma 12, we pick two hash functions h 1 : [ n ] → [ k ] , h 2 :
[ n ] → [1 /(cid:15) 2 ] , and we can assume with high probability
, they perfectly hash our input set of items.
Lemma 13 tells us with probability at least 0 .
9 , we can use these two hash functions and use O (1) time to read the p -stable random variables for the medium elements (if X i ∈
I , we store X i in the cell which h 1 ( i ) refers to, using
O (log(1 /(cid:15) )) bits.
Otherwise we store X i in the cell which h 2 ( i ) refers to, using O (log( M )) bits).
Suppose that φ ( x ) is the pdf of the p -stable distribution.
For a p -stable Y , we have that Pr {| Y | < (cid:15) q } = (cid:82) (cid:15) q − (cid:15) q φ ( x ) dx ≤ 2 φ (0) (cid:15) q = poly( (cid:15) ) when q is sufﬁciently large.
Hence, in Lemma 13 and
Lemma 15, we can pick a sufﬁciently large q such that with probability at least 1 − poly( (cid:15) ) , all of the true values y i output by the Est i are at least (cid:15) q (cid:107) x
(cid:107) p , and the (cid:101)
y i output by the Est i using the stored p -stable random variables satisfy |
y i − (cid:101)
y i | ≤ (cid:15) q + 1 p (cid:107) x
(cid:107) p .
So the approximation from the p -stable random variables will affect our estimate by at most (cid:15) (cid:107) x (cid:107) pp .
This completes the proof.
Using Theorem 11 and 16 we will have our result:
Theorem 17.
Let 0 < p < 2 and 0 < (cid:15) < 1 / 2 .
There exists a randomized algorithm which outputs (1 ± 3 (cid:15) )
(cid:107) x (cid:107) pp
with probability at least 0 .
7 using O ( (cid:15) − 2 max { log( nmM ) , log 3 (1 /(cid:15) )
log log(1 /(cid:15) ) } ) bits of space.
The expected amortized update time is O (1) .
The reporting time is O (1 /(cid:15) 2 ) .
In the case of a noisy oracle, we assume that for a ﬁxed s , our oracle will not identify more than O ( s ) items x
i as heavy hitters (meaning that | x i | > 1 s (cid:107) x (cid:107) p p ).
Then we can adapt the preceding theorem to the noisy oracle version.
Theorem 18.
Let 0 < p < 2 and 0 < (cid:15) < 1 / 2 .
Suppose that the heavy hitter oracle errs with probability δ
= O ( (cid:15) 2 log 2 (1 /(cid:15) )
loglog(1 /(cid:15) ) ) .
There exists an algorithm which outputs (1 ± 3 (cid:15) )
(cid:107) x (cid:107) pp
with probability at least 0 .
7 using O ( (cid:15) − 2 max { log( nmM ) , log 3 (1 /(cid:15) )
log log(1 /(cid:15) ) } ) bits of space.
The expected amortized update time is O (1) .
The reporting time is O (1 /(cid:15) 2 ) .
Proof.
We only need to consider the difference in the estimation of (cid:107) x I L (cid:107) .
When δ =
O ( (cid:15) 2 log 2 (1 /(cid:15) )
loglog(1 /(cid:15) ) ) , note that our sub-sampling rate is log 2 (1 /(cid:15) ) log log(1 /(cid:15) ) .
It follows from
Theorem 8 that Theorem 17 continues to hold.
D C ASCADED N ORMS
For notational simplicity, we consider approximating F k ( F p ( x ))
=
(cid:80) ni
=1 ( (cid:80) dj
=1 | x ij | p ) k with k ≥ 1 and p ≥ 2 .
The cascaded norm (cid:107) x (cid:107) k,p then satisﬁes (cid:107) x (cid:107) kk,p = F k/p ( F p ( x ) for k ≥ p ≥ 2 .
We follow the algorithm in Jayram & Woodruff (2009), which ﬁrst downsamples the rows in log n levels, and then samples in each level Q = O ( n 1 − 1 /k )
entries from the row-sampled submatrix proportional to | x i | p , and at last estimates the F k ( F p ) cascaded norm from those samples.
The ﬁrst step of (cid:96) p -sampling is further decomposed into two steps: (i) dividing the entries into layers by their magnitudes and uniformly sampling a sufﬁciently large set of entries in each layer, and (ii) performing “ (cid:96) p -sampling” from those samples.
We shall improve the space complexity of Step (i) with a heavy hitter oracle.
We ﬁrst elaborate how Step (i) was done.
It splits the entries in the row-sampled matrix X into layers, where the t -th layer S t ( X ) consists of entries in [ ζη t − 1 , ζη t ] .
A layer t is said to be contributing if | S t ( X ) | ( ζη t ) p ≥ F p
( X ) / ( Bθ ) .
It needs to return β t = θQ
| S t ( X ) | ( ζη t ) p /F p ( X ) samples from each contributing layer t .
Suppose that | S t ( X ) | / 2 j ≤ β t < | S t ( X ) | / 2 j − 1 for some j , and the algorithm subsamples each entry with probability 1 / 2 j , obtaining a subset Y j of entries.
It can be shown that with high probability, for a contributing layer t , it holds that ( ζη t ) 2 ≥ (cid:107) Y j (cid:107)
( Q 2 /p θ 4 | X | 1 − 2 /p log | X | ) , and therefore the standard C OUNT -S
KETCH algorithm can recover all those entries with (cid:101)
O
( Q 2 /p
| X | 1 − 2 /p ) space.
Consider the top level without subsampling of the rows.
Suppose that the heavy hitter oracle determines whether | x i | p ≥ (cid:107)
x
(cid:107)
pp /T for each entry
x i of the matrix x with some threshold T .
In the algorithm of Step (i), without loss of generality, we can assume each layer contains either heavy entries only or light entries only, and we call the layer a heavy layer or a light layer accordingly.
If a light layer is contributing, since each entry is at most F p /T , there must exist at least N t =
T | S t ( X ) | ( ζη t ) p /F p entries in that level.
We would require that N t ≥ β t , for which it sufﬁces to have T = Ω( θQ ) .
We can downsample the layer at rate θQ/T , and then apply the preceding algorithm and thus the number of buckets can be reduced by a factor of T/ ( θQ ) .
Note that the heavy layers come from the at most T heavy hitters.
This leads to a space complexity of (cid:101) O
( Q 2 /p T 1 − 2 /p
+ Q 2 /p
| X | 1 − 2 /p / ( T/ ( θQ )) .
Let T
=
Θ(( | X | Q ) 1 / 2 ) .
In this case, T = Ω( θQ ) is satisﬁed and the space complexity is (cid:101)
O ( Q 1 /
+1 /p
| X | 1 / 2 − 1 /p ) .
For the j -th level of row sampling, note that the same (global) heavy hitter oracle, when applied to this row-subsampled matrix, effectively has T replaced with T/ 2 j .
However, | X | is also replaced with | X | / 2 j , and thus we would require T = Ω(2 j θQ ) and we can downsample the layer at rate 2 j θQ/T .
This means that the space complexity is a (2 − j ) 1 − 2 /p fraction of the space of the top level.
The constraint of T = Ω(2 j θQ ) allows for j ≤ 12 log | X | Q .
For j > 12 log | X | Q , we can just run the old algorithm with space (cid:101) O
( Q 2 /p ( | X | / 2 j ) 1 − 2 /p )
= (cid:101) O
( Q 1 / 2+1 /p
| X | 1 / 2 − 1 /p ) .
Plugging in Q = Θ( n 1 − 1 /k ) and | X | = nd and noting that the space of the subsequent steps are dominated by this very ﬁrst sampling step, we have:
Theorem 19.
Let (cid:15) > 0 and k ≥ 1 , p ≥ 2 be constants.
There exists a randomized algorithm which receives an n × d matrix x and outputs a (1 + (cid:15) ) -approximation to F k ( F p ( x )) with probability at least 2 / 3 using (cid:101) O ( n 1 − 1 kp − 12 k d 12 − 1 p ) space.
Replacing k with k/p , we obtain our ﬁnal result for cascaded norms.
Corollary 20.
Let (cid:15) > 0 and k ≥ p ≥ 2 be constants.
There exists a randomized algorithm which receives an n × d matrix x in a stream of additions and deletions to its coordinates, and outputs a (1 + (cid:15) ) -approximation to (cid:107) x (cid:107) k,p with probability at least 2 / 3 using (cid:101) O
( n 1 − 1 k − p 2 k d 1 2 − 1 p ) space.
E R ECTANGULAR F p
The rectangle-efﬁcient algorithm in Tirthapura & Woodruff (2012) is based on the F p moment estimation algorithm in Braverman & Ostrovsky (2010).
In Braverman & Ostrovsky (2010), they ﬁrst subsample to obtain the substream D j ( j = 0 , 1 , 2 , ... log n ), where D 0 is the full stream, and if i ∈ D j , then i ∈ D
j +1 with probability 1 / 2 .
For every substream D j , they run the C OUNT -S KETCH algorithm to recover all ( α 2 p /n 1 − 2 p ) -heavy hitters, whence they can obtain a good ﬁnal estimate.
The algorithm in Tirthapura & Woodruff (2012) is similar.
Instead of updating the counter in each coordinate inside a rectangle, they developed a rectangle-efﬁcient data structure called R ECTANGLE C OUNT S KETCH .
We follow their notation that O ∗ ( f ) denotes a function of the form f · poly(1 /(cid:15), d, log( m ∆ /δ )) .
Lemma 21 (R ECTANGLE C OUNT S KETCH , Tirthapura & Woodruff (2012)) .
The data structure
R ECTANGLE C OUNT S KETCH ( γ ) can be updated rectangle-efﬁciently in time O ∗ ( γ − 2 ) .
The total space is O ∗ ( γ − 2 ) words.
The data structure can be used to answer any query i ∈ GF (∆) d , returning a number Φ( i ) with | Φ( i ) − x i | ≤ γ (cid:107) x
(cid:107) 2 .
The algorithm succeeds on all queries simultaneously with probability ≥ 1 − δ .
Under an arbitrary mapping g : GF (∆)
d → { 1 , 2 , ..., ∆ } , they subsample the items into a logarithmic number φ
= d log(∆) of levels.
In particular, for j ∈ [ φ ] , i ∈ D
j if g ( i ) ≤ ∆ d / 2 j − 1 .
In each level, they give an algorithm which invokes R ECTANGLE C OUNT S KETCH ( γ ) to recover a list of the
F p heavy hitters.
Then they use the heavy hitters from each layer to calculate the ﬁnal estimate.
Lemma 22 (Tirthapura & Woodruff (2012)) .
For p > 2 , there is an algorithm which uses
R ECTANGLE C OUNT S KETCH ( γ ) with γ = Θ( (cid:15) 1+ 1 p α 1 p / ∆ d/ 2 − d/p ) and with probability at least 1 − δ outputs a set P of O (1 /α ) pairs ( i, x (cid:48) i ) such that (cid:80) ( i,x (cid:48) i )
∈ P || x (cid:48)
i | p − | x i | p | ≤ (cid:15) (cid:107) x (cid:107) pp
and all elements i with | x i | p ≥ α (cid:107) x (cid:107) pp appear as the ﬁrst element of some pair in P .
The algorithm uses
O ∗ ( γ − 2 ) words of space.
Since the algorithm relies on a subroutine that ﬁnds the (cid:96)
p -heavy hitters, the improvement with a heavy hitter oracle is similar to that for the F p -moment problem.
Speciﬁcally, suppose the heavy hitter oracle is able to indicate whether | x i | p > 1∆ d/ 2 (cid:107) x (cid:107) pp .
We know from Theorem 7 that we can use a rectangle-efﬁcient estimator for the substream which contains all the heavy items, while for the light items, we can use the subsampling algorithm with rate ∆ d/ 2 .
Note that for each substream, we have (cid:107) x (cid:107) 0 ≤ ∆ d/ 2 (for those i ’s that are not in the substream, we see x i = 0 ), so by H ¨ older’s inequality, it holds that (cid:107) x (cid:107) 2 ≤ ∆ d/ 4 − d/ (2 p ) (cid:107) x (cid:107) p .
Following the proof of Lemma 22, we can set γ = Θ( (cid:15)
+ 1 p α 1 p / ∆ d/ 4 − d/ (2 p ) ) to ﬁnd those heavy hitters.
We thus have the following theorem.
Theorem 23.
Under the assumption of a heavy hitter oracle, there is a rectangle-efﬁcient singlepass streaming algorithm which outputs a (1 ± (cid:15) )
-approximation to (cid:107) x (cid:107) pp with probability 1 − δ for p > 2 .
It uses O ∗ (∆ d (1 / 2 − 1 /p ) ) bits of space and O ∗ (∆ d (1 / 2 − 1 /p ) ) time to process each rectangle in the stream.