T RIANGLE AND F OUR C YCLE C OUNTING WITH P REDICTIONS IN G RAPH S TREAMS
A BSTRACT
We propose data-driven one-pass streaming algorithms for estimating the number of triangles and four cycles, two fundamental problems in graph analytics that are widely studied in the graph data stream literature.
Recently, Hsu et al. (2019a) and Jiang et al.
(2020) applied machine learning techniques in other data stream problems, using a trained oracle that can predict certain properties of the stream elements to improve on prior “classical” algorithms that did not use oracles.
In this paper, we explore the power of a “heavy edge” oracle in multiple graph edge streaming models.
In the adjacency list model, we present a one-pass triangle counting algorithm improving upon the previous space upper bounds without such an oracle.
In the arbitrary order model, we present algorithms for both triangle and four cycle estimation with fewer passes and the same space complexity as in previous algorithms, and we show several of these bounds are optimal.
We analyze our algorithms under several noise models, showing that the algorithms perform well even when the oracle errs.
Our methodology expands upon prior work on “classical” streaming algorithms, as previous multi-pass and random order streaming algorithms can be seen as special cases of our algorithms, where the ﬁrst pass or random order was used to implement the heavy edge oracle.
Lastly, our experiments demonstrate advantages of the proposed method compared to state-of-theart streaming algorithms.
I NTRODUCTION
Counting the number of cycles in a graph is a fundamental problem in the graph stream model (e.g., Atserias et al.
(2008); Bera & Chakrabarti (2017);
Seshadhri et al.
; Kolountzakis et al.
(2010); Bar-Yossef et al.
; Kallaugher et al.
The special case of counting triangles is widely studied, as it has a vast range of applications.
In particular, it provides important insights into the structural properties of networks (Prat-P´erez et al., 2012; Farkas et al., 2011), and is used to discover motifs in protein interaction networks (Milo et al., 2002), understand social networks (Foucault Welles et al., 2010), and evaluate large graph models (Leskovec et al., 2008).
See Al Hasan & Dave (2018) for a survey of these and other applications.
Because of its importance, a large body of research has been devoted to space-efﬁcient streaming algorithms for (1 + (cid:15) ) -approximate triangle counting.
Such algorithms perform computation in one or few passes over the data using only a sub-linear amount of space.
A common difﬁculty which arises in all previous works is the existence of heavy edges , i.e., edges that are incident to many triangles (four cycles).
As sublinear space algorithms often rely on sampling of edges, and since a single heavy edge can greatly affect the number of triangles (four cycles) in a graph, sampling and storing these edges are often the key to an accurate estimation.
Therefore, multiple techniques have been developed to determine whether a given edge is heavy or not.
Recently, based on the observation that many underlying patterns in real-world data sets do not change quickly over time, machine learning techniques have been incorporated into the data stream model via the training of heavy-hitter oracles.
Given access to such a learning-based oracle, a wide range of signiﬁcant problems in data stream processing — including frequency estimation, estimating the number of distinct elements, F p -Moments or ( k, p ) -Cascaded Norms — can all achieve space bounds that are better than those provided by “classical” algorithms, see, e.g., Hsu et al.
(2019a)
; Cohen et al.
; Jiang et al.
Eden et al.
; Du et al.
More generally, learning-based approaches have had wide success in other algorithmic tasks, such as data structures (Kraska et al., 2018;
Ferragina et al., 2020; Mitzenmacher, 2018; Rae et al., 2019; Vaidya et al., 2021), online algorithms (Lykouris & Vassilvtiskii, 2018; Purohit et al., 2018; Gollapudi &
Panigrahi, 2019; Rohatgi, 2020; Wei, 2020; Mitzenmacher, 2020; Lattanzi et al., 2020; Bamas et al., 2020), similarity search (Wang et al., 2016;
Dong et al., 2020) and combinatorial optimization (Dai et al., 2017; Balcan et al., 2017; 2018a;b; 2019).
See the survey and references therein for additional works (Mitzenmacher & Vassilvitskii, 2020).
Inspired by these recent advancements, we ask: is it possible to utilize a learned heavy edge oracle to improve the space complexity of subgraph counting in the graph stream model?
Our results demonstrate that the answer is yes.
1. O UR R ESULTS AND C OMPARISON TO P REVIOUS T HEORETICAL W ORKS
We present theoretical and empirical results in several graph streaming models, and with several notions of prediction oracles.
Conceptually, it is useful to begin by studying perfect oracles that provide exact predictions.
While instructive theoretically, such oracles are typically not available in practice.
We then extend our theoretical results to noisy oracles that can provide inaccurate or wrong predictions.
We validate the practicality of such oracles in two ways: by directly showing they can be constructed for multiple real datasets, and by showing that on those datasets, our algorithms attain signiﬁcant empirical improvements over baselines, when given access to these oracles.
We proceed to a precise account of our results.
Let G =
( V, E ) denote the input graph, and let n , m , and T denote the number of vertices, edges, and triangles (or four-cycles) in G , respectively.
There are two major graph edge streaming models: the adjacency list model and the arbitrary order model.
We show that training heavy edge oracles is possible in practice in both models, and that such oracles make it possible to design new algorithms that signiﬁcantly improve the space complexity of triangle and four-cycle counting, both in theory and in practice.
Furthermore, our formalization of a heavy edge prediction framework makes it possible to show provable lower bounds as well.
Our results are summarized in Table 1.
In our algorithms, we assume that we know a large-constant approximation of T for the purposes of setting various parameters.
This is standard practice in the subgraph counting streaming literature (e.g., see (Braverman et al., 2013, Section 1), (McGregor et al., 2016, Section 1.2) for an extensive discussions on this assumption).
Moreover, when this assumption cannot be directly carried over in practice, in Subsection F.3 we discuss how to adapt our algorithms to overcome this issue.
1.1. P ERFECT O RACLE
Our ﬁrst results apply for the case that the algorithms are given access to a perfect heavy edge oracle.
That is, for some threshold ρ , the oracle perfectly predicts whether or not a given edge is incident to at least ρ triangles (four cycles).
We describe how to relax this assumption in Section 1.1.2.
Adjacency List Model All edges incident to the same node arrive together.
We show:
[Table]
Theorem 1.1.
There exists a one-pass algorithm, Algorithm 1, with space complexity(cid:101)
O (min( (cid:15) − 2 m 2 / 3 /T 1 / 3 , (cid:15) − 1 m 1 / 2 )) in the adjacency list model that, using a learning-based oracle, returns a (1 ± (cid:15) )
-approximation to the number T of triangles with probability at least 2 7 / 10 .
An overview of Algorithm 1 is given in Section 2, and the full analysis is provided in Appendix B.
Arbitrary Order Model
In this model, the edges arrive in the stream in an arbitrary order.
We present a one-pass algorithm for triangle counting and another one-pass algorithm for four cycle counting in this model, both reducing the number of passes compared to the currently best known space complexity algorithms.
Our next result is as follows:
Theorem 1.2.
There exists a one-pass algorithm, Algorithm 4, with space complexity (cid:101) O ( (cid:15) − 1 ( m/ √ T + √ m )) in the arbitrary order model that, using a learning-based oracle, returns a (1 ± (cid:15) )
-approximation to the number T of triangles with probability at least 7 / 10 .
An overview of Algorithm 4 is given in Section 3, and full details are provided in Appendix C.
We also show non-trivial space lower bounds that hold even if appropriate predictors are available.
In Theorem C.2 in Appendix C.3, we provide a lower bound for this setting by giving a construction that requires Ω(min( m/ √ T, m 3 / 2 /T )) space even with the help of an oracle, proving that our result is nearly tight in some regimes.
Therefore, the triangle counting problem remains non-trivial even when extra information is available.
Four Cycle Counting.
For four cycle counting in the arbitrary order model, we give Theorem 1.
which is proven in Appendix D.
Theorem 1.3.
There exists a one-pass algorithm, Algorithm 5, with space complexity (cid:101)
O ( T 1 / 3 + (cid:15) − 2 m/T 1 / 3 ) in the arbitrary order model that, using a learning-based oracle, returns a (1 ± (cid:15) ) - approximation to the number T of four cycles with probability at least 7 / 10 .
To summarize our theoretical contributions, for the ﬁrst set of results of counting triangles in the adjacency list model, our bounds always improve on the previous state of the art due to McGregor et al.
(2016) for all values of m and T .
For a concrete example, consider the case
that T = Θ( √ m ) .
In this setting previous bounds result in an (cid:101) O ( m 3 / 4 )
-space algorithm, while our algorithm only requires (cid:101)
O ( √ m ) space (for constant (cid:15) ).
For the other two problems of counting triangles and 4-cycles in the arbitrary arrival model, our space bounds have an additional additive term compared to McGregor et al.
(2016) (for triangles) and Vorotnikova (2020) (for 4-cycles)
but importantly run in a single pass rather than multiple passes.
In the case where the input graph has high triangles density, T = Ω( m/(cid:15) 2 ) , our space bound is worse due to the additive factor.
When T =
O ( m/(cid:15) 2 ) , our results achieve the same dependence on m and T as that of the previous algorithms with an improved dependency in (cid:15) .
Moreover, the case T ≤ m/(cid:15) 2 is natural for many real world datasets: for (cid:15) = 0 .
05 , this condition holds for all of the datasets in our experimental results (see Table 2).
Regardless of the triangle density, a key beneﬁt of our results is that they are achieved in a single pass rather than multiple passes.
Finally, our results are for general graphs, and make no assumptions on the input graph (unlike Pagh & Tsourakakis (2012); Kallaugher & Price (2017); Bera & Sheshadhri (2020)).
Most of our algorithms are relatively simple and easy to implement and deploy.
At the same time, some of our results require the use of novel techniques in this context, such as the use of exponential random variables (see Section 2).
1.1. N OISY O RACLES
The aforementioned triangle counting results are stated under the assumption that the algorithms are given access to perfect heavy edge oracles.
In practice, this assumption is sometimes unrealistic.
Hence, we consider several types of noisy oracles.
The ﬁrst such oracle, which we refer to as a
K -noisy oracle, is deﬁned below (see Figure 3 in the Supplementary Section C.2).
Deﬁnition 1.1.
For an edge e = xy in the stream, deﬁne N e as the number of triangles that contain both x and y .
For a ﬁxed constant K ≥ 1 and for a threshold ρ we say that an oracle O ρ is a
K -noisy oracle if for every edge e , 1 − K · ρN e ≤ Pr [ O ρ ( e ) = HEAVY ] ≤ K · N e ρ .
This oracle ensures that if an edge is extremely heavy or extremely light, it is classiﬁed correctly with high probability, but if the edge is close to the threshold, the oracle may be inaccurate.
We further discuss the properties of this oracle in Section G.
For this oracle, we prove the following two theorems.
First, in the adjacency list model, we prove:
Theorem 1.4.
Suppose that the oracle given to Algorithm 1 is a K -noisy oracle as deﬁned in
Deﬁnition 1.1.
Then with probability 2 / 3 , Algorithm 1 returns a value in (1 ± √ K · (cid:15) ) T , and uses space at most (cid:101) O (min( (cid:15) − 2 m 2 / 3 /T 1 / 3 , K · (cid:15) − 1 m 1 / 2 )) .
Hence, even if our oracle is inaccurate for edges near the threshold, our algorithm still obtains an effective approximation with low space in the adjacency list model.
Likewise, for the arbitrary order model, we prove in Theorem C.1 that the O ( (cid:15) − 1 ( m/ √ T + √ m )) 1-pass algorithm of Theorem 1. also works when Algorithm 4 is only given access to a K -noisy oracle.
The proof of Theorem 1.4 is provided in Appendix B.1, and the proof of Theorem C.1 is provided in Appendix C.2.
We remark that Theorems 1.4 and C.1 automatically imply Theorems 1.1 and 1.2, since the perfect oracle is automatically a K -noisy oracle.
(Noisy) Value Oracles
In the adjacency list model, when we see an edge xy , we also have access to all the neighbors of either x or y , which makes it possible for the oracle to give a more accurate prediction.
For an edge xy , let R xy denote the number of triangles { x, z, y } so that x precedes z and z precedes y in the stream arrival order.
Formally, R xy =
| z : { x, y, z } ∈ ∆ and x < s z <
s y | where x < s y denotes that the adjacency list of x arrives before that of y in the stream.
Motivated by our empirical results in Section F.7, it is reasonable in some settings to assume we have access to oracles that can predict a good approximation to R xy .
We refer to such oracles as value oracles .
In the ﬁrst version of this oracle, we assume that the probability of approximation error decays linearly with the error from above but exponentially with the error from below.
Deﬁnition 1.2.
Given an edge e , an ( α, β ) value-prediction oracle outputs a random value p ( e ) where E [ p ( e )] ≤ αR e + β , and Pr [ p ( e ) <
R e
λ − β ] ≤ Ke − λ for some constant K and any λ ≥ 1 .
For this variant, we prove the following theorem.
Theorem 1.5.
Given an oracle with parameters ( α, β ) , there exists a one-pass algorithm, Algorithm 2, with space complexity O ( (cid:15) − 2 log 2 ( K/(cid:15) )
( α + mβ/T )) in the adjacency list model that returns a (1 ± (cid:15) ) -approximation to the number of triangles T with probability at least 7 / 10 .
In the second version of this noisy oracle, we assume that the probability of approximation error decays linearly with the error from both above and below.
For this variant, we prove that we can achieve the same guarantees as Theorem 1.5 up to logarithmic factors (see Theorem B.1).
The algorithms and proofs for both Theorem 1.5 and Theorem B.1 appear in Appendix B.2.
Experiments We conduct experiments to verify our results for triangle counting on a variety of real world networks (see Table 2) in both the arbitrary and adjacency list models.
Our algorithms use additional information through predictors to improve empirical performance.
The predictors are data dependent and include: memorizing heavy edges in a small portion of the ﬁrst graph in a sequence of graphs, linear regression, and graph neural networks (GNNs).
Our experimental results show that we can achieve up to 5x decrease in estimation error while keeping the same amount of edges as other state of the art empirical algorithms.
For more details, see Section 4.
In Section F.7, we show that our noisy oracle models are realistic for real datasets.
Related Empirical Works On the empirical side, most of the focus has been on triangle counting in the arbitrary order model for which there are several algorithms that work well in practice.
We primarily focus on two state-of-the-art baselines, ThinkD (Shin et al., 2018) and WRS (Shin, 2017).
In these works, the authors compare to previous empirical benchmarks such as the ones given in
Stefani et al.
; Han & Sethu (2017); Lim & Kang (2015) and demonstrate that their algorithms achieve superior estimates over these benchmarks.
There are also other empirical works such as
Ahmed et al.
and Ahmed & Dufﬁeld (2020) studying this model
but they do not compare to either ThinkD or WRS.
While these empirical papers demonstrate that their algorithm returns unbiased estimates, their theoretical guarantees on space is incomparable to the previously stated space bounds for theoretical algorithms in Table 1.
Nevertheless, we use ThinkD and WRS as part of our benchmarks due to their strong practical performance and code accessibility.
Implicit Predictors in Prior Works
The idea of using a predictor is implicit in many prior works.
The optimal two pass triangle counting algorithm of McGregor et al.
(2016) can be viewed as an implementation of a heavy edge oracle after the ﬁrst pass.
This oracle is even stronger than the Knoisy oracle as it is equivalent to an oracle that is always correct on an edge e if N e either exceeds or is under the threshold ρ by a constant multiplicative factor.
This further supports our choice of oracles in our theoretical results, as a stronger version of our oracle can be implemented using one additional pass through the data stream (see Section G).
Similarly, the optimal triangle counting streaming algorithm
(assuming a random order) given in McGregor & Vorotnikova (2020) also implicitly deﬁnes a heavy edge oracle using a small initial portion of the random stream (see Lemma 2 . 2 in McGregor & Vorotnikova (2020)).
The random order assumption allows for the creation of such an oracle since heavy edges are likely to have many of their incident triangle edges appearing in an initial portion of the stream.
We view these two prior works as theoretical justiﬁcation for our oracle deﬁnitions.
Lastly, the WRS algorithm also shares the feature of deﬁning an implicit oracle: some space is reserved for keeping the most recent edges while the rest is used to keep a random sample of edges.
This can be viewed as a speciﬁc variant of our model, where the oracle predicts recent edges as heavy.
Preliminaries.
G =
( V, E ) denotes the input graph, and n , m and T denote the number of vertices, edges and triangles (or four-cycles) in G , respectively.
We use N ( v ) to denote the set of neighbors of a node v , and ∆ to denote the set of triangles.
In triangle counting, for each xy ∈ E ( G ) , we recall that N xy =
| z : { x, y, z } ∈ ∆ | is the number of triangles incident to edge xy , and
R xy =
| z : { x, y, z } ∈ ∆ , x < s z < s y | is the number of triangles adjacent to xy with the third vertex z of the triangle between x and y in the adjacency list order.
Table A summarizes the notation.
T RIANGLE C OUNTING IN THE A DJACENCY L IST M ODEL
We describe an algorithm with a heavy edge oracle, and one with a value oracle.
Heavy Edge Oracle.
We present an overview of our one-pass algorithm, Algorithm 1, with a space complexity of (cid:101)
O
(min( (cid:15) − 2 m 2 / 3 /T 1 / 3 , (cid:15) − 1 m 1 / 2 )) , given in Theorem 1.1.
We defer the pseudocode and proof of the theorem to Appendix B.
The adaptations and proofs for Theorems 1.4, 1.5 and B. for the various noisy oracles appear in Appendix B.1 and Appendix B.2.
Our algorithm works differently depending on the value of T .
We ﬁrst consider the case that
T ≥ ( m/(cid:15) ) 1 / 2 . Assume that for each sampled edge xy in the stream, we can exactly know the number of triangles R xy this edge contributes to T .
Then the rate at which we would need to sample each edge would be proportional to p naive ≈ (cid:15) − 2 ∆ E /T , where ∆ E is the maximum number of triangles incident to any edge.
Hence, our ﬁrst idea is to separately consider light and non-light edges using the heavy edge oracle.
This allows us to sample edges that are deemed light by the oracle at a lower rate, p 1 , and compute their contribution by keeping track of R xy for each such sampled edge.
Intuitively, light edges offer us more ﬂexibility and thus we can sample them with a lower probability while ensuring the estimator’s error does not drastically increase.
In order to estimate the contribution due to non-light edges, we again partition them into two types: medium and heavy, according to some threshold ρ .
We then use an observation from McGregor et al.
that since for heavy edges R xy >
ρ
, it is sufﬁcient to sample from the entire stream at rate p 3 ≈ (cid:15) − 2 /ρ , in order to both detect if some edge xy is heavy and if so to estimate R xy .
Therefore, it remains to estimate the contribution to T due to medium edges (these are the edges that are deemed non-light by the oracle, and also non-heavy according to the sub-sampling above).
Since the number of triangles incident to medium edges is higher than that of light ones, we have to sample them at some higher rate p 2 > p 1 .
However, since their number is bounded, this is still space efﬁcient.
We get the desired bounds by correctly setting the thresholds between light, medium and heavy edges.
When T < ( m/(cid:15) ) 1 / 2 our algorithm becomes much simpler.
We only consider two types of edges, light and heavy, according to some threshold T/ρ .
To estimate the contribution due to heavy edges we simply store them and keep track of their R xy values.
To estimate the contribution due to light edges we sub-sample them with rate (cid:15) − 2 · ∆ E /T
= (cid:15) − 2 /ρ .
The total space we use is (cid:101) O
( (cid:15) − 1 √ m ) , which is optimal in this case.
See Algorithm 1 for more details of the implementation.
Value-Based Oracle.
We also consider the setting where the predictor returns an estimate p ( e ) of
R e , and we assume R e ≤ p ( e ) ≤ α · R e , where α ≥ 1 is an approximation factor.
We relax this assumption to also handle additive error as well as noise, but for intuition we focus on this case.
The value-based oracle setting requires the use of novel techniques, such as the use of exponential random variables (ERVs).
Given this oracle, for an edge e , we compute p ( e ) /u
e , were u e is a standard ERV.
We then store the O ( α log(1 /(cid:15) )) edges e for which p ( e )
/u e is largest.
Since we are in the adjacency list model, once we start tracking edge e = xy , we can also compute the true value R e of triangles that the edge e participates in (since for each future vertex
z we see, we can check if x and y are both neighbors of z ).
Note that we track this quantity only for the
O ( α log(1 /(cid:15) )) edges that we store.
Using the max-stability property of ERVs, max e R e /u e is equal in distribution to T/u , where u is another ERV.
Importantly, using the density function of an ERV, one can show that the edge e for which R e /u e is largest is, with probability 1 − O ( (cid:15) 3 ) , in our list of the O ( α log(1 /(cid:15) ))
largest p ( e ) /u
e values that we store.
Repeating this scheme r
= O (1 /(cid:15) 2 ) times
, we obtain independent estimates T/u 1 , . .
, T/u r , where u 1 , . . .
, u r are independent ERVs.
Taking the median of these then gives a (1 ± (cid:15) )
-approximation to the total number T of triangles.
We note that ERVs are often used in data stream applications (see, e.g., Andoni (2017)), though to the best of our knowledge they have not previously been used in the context of triangle estimation.
We also give an alternative algorithm, based on subsampling at O (log n ) scales, which has worse logarithmic factors in theory but performs well empirically.
T RIANGLE C OUNTING IN THE A RBITRARY O RDER M ODEL
In this section we discuss Algorithm 4 for estimating the number of triangles in an arbitrary order stream of edges.
The pseudo-code of the algorithm as well as omitted proofs for the different oracles are given in Supplementary Section C.
Here we give the intuition behind the algorithm.
Our approach relies on sampling the edges of the stream as they arrive and checking if every new edge forms a triangle with the previously sampled edges.
However, as previously discussed, this approach alone fails if some edges have a large number of triangles incident to them as “overlooking” such edges might lead to an underestimation of the number of triangles.
Therefore, we utilize a heavy edge oracle, and refer to edges that are not heavy as light .
Whenever a new edge arrives, we ﬁrst query the oracle to determine if the edge is heavy.
If the edge is heavy we keep it, and otherwise we sample it with some probability.
As in the adjacency list arrival model case, this strategy allows us to reduce the variance of our estimator.
By balancing the sampling rate and our threshold for heaviness, we ensure that the space requirement is not too high, while simultaneously guaranteeing that our estimate is accurate.
In more detail, our algorithm works as follows.
First, we set a heaviness threshold ρ , so that if we predict an edge e to be part of ρ or more triangles, we label it as heavy.
We also set a sampling parameter p .
We let H be the set of edges predicted to be heavy and let S L be a random sample of edges predicted to be light.
Then, we count three types of triangles.
The ﬁrst counter, (cid:96) 1 , counts the triangles ∆ =
( e 1 , e 2 , e ) , where the ﬁrst two edges seen in this triangle by the algorithm, represented by e 1 and e 2 , are both in S L .
Note that we only count the triangle if e 1 and e 2 ,were both in S L at the time e arrives in the stream.
Similarly, (cid:96) 2 counts triangles whose ﬁrst two edges are in S L and
H (in either order), and (cid:96) 3 counts triangles whose ﬁrst two edges are in H .
Finally, we return the estimate (cid:96) =
(cid:96) 1 /p 2 + (cid:96) 2 /p
+ (cid:96) 3 .
Note that if the ﬁrst two edges in any triangle are light, they will both be in S L with probability p 2 , and if exactly one of the ﬁrst two edges is light, it will be in S L with probability p .
Therefore, we divide (cid:96) 1 by p 2 and (cid:96) 2 by p so that (cid:96) is an unbiased estimator.
E XPERIMENTS
We now evaluate our algorithm on real and synthetic data whose properties are summarized in
Table 2 (see Appendix F for more details).
graphs over time (the length of the sequence is given in parentheses) and temporal graphs are formed by edges appearing over time.
The listed values for n (number of vertices), m (number of edges), and T (number of triangles) for Oregon and CAIDA are approximated across all graphs.
The Oregon and CAIDA datasets come from Leskovec & Krevl (2014); Leskovec et al.
(2005), the Wikibooks dataset comes from Rossi & Ahmed (2015), the Reddit dataset comes from Leskovec & Krevl (2014); Kumar et al.
(2018), the Twitch dataset comes from Rozemberczki et al.
(2019), the Wikipedia dataset comes from Rossi & Ahmed (2015), and the Powerlaw graphs are sampled from the Chung-Lu-Vu random graph model with expected degree of the i -th vertex proportional to 1 /i 2 (Chung et al., 2003).
[Table]
We now describe the edge heaviness predictors that we use (see also Table 2).
Our
predictors adapt to the type of dataset and information available for each dataset.
Some datasets we use contain only the graph structure (nodes and edges) without semantic features, thus not enabling us to train a classical machine learning predictor for edge heaviness.
In those cases we use the true counts on a small preﬁx of the data (either 10% of the ﬁrst graph in a sequence of graphs, or a preﬁx of edges in a temporal graph) as predicted counts for subsequent data.
However, we are able to create more sophisticated predictors on three of our datasets, using feature vectors in linear regression or a Graph Neural Network.
Precise details of the predictors follow.
• Snapshot : For Oregon / CAIDA graph datasets, which contain a sequence of graphs, we use exact counting on a small fraction of the ﬁrst graph as the predictor for all the subsequent graphs .
Speciﬁcally, we count the number of triangles per edge, N e , on the ﬁrst graph for each snapshot dataset.
We then only store 10% of the top heaviest edges and use these values as estimates for edge heaviness in all later graphs.
If a queried edge is not stored, its predicted N e value is 0 .
• Preﬁx : In the WikiBooks temporal graph, we use the exact N e counts on the ﬁrst half of the graph edges (when sorted by their timestamps) as the predicted values for the second half.
• Linear Regression :
In the Reddit Hyperlinks temporal graph, we use a separate dataset (Kumar et al., 2019) that contains 300-dimensional feature embeddings of subreddits (graph nodes).
Two embeddings are close in the feature space if their associated subreddits have similar sub-communities.
To produce an edge f ( e ) embedding for an edge e = uv from the node embedding of its endpoints, f ( u ) and f ( v ) , we use the 602-dimensional embedding ( f ( u ) , f ( v ) , (cid:107) ( f ( u ) − f ( v ) (cid:107) 1 , (cid:107) ( f ( u ) − f ( v ) (cid:107) 2 ) .
We then train a linear regressor to predict N e given the edge embedding f ( e ) .
Training is done on a preﬁx of the ﬁrst half of the edges.
• Link Prediction (GNN) :
For each of the two networks, we start with a graph that has twice as many edges as listed in Table 2, ( ∼ 1 . 1 · 10 5 edges for Twitch and 9 . 2 · 10 4 edges for Wikipedia).
We then randomly remove 50% of the total edges to be the training data set, and use the remaining edges as the graph we test on.
We use the method proposed in Zhang & Chen (2018) to train a link prediction oracle using a Graph Neural Network (GNN) that will be used to predict the heaviness of the testing edges.
For each edge that arrives in the stream of the test edges, we use the predicted likelihood of forming an edge given by the the neural network to the other vertices as our estimate for N uv , the number of triangles on edge uv .
See Section F.2 for details of training methodology.
• Expected Value (EV) :
In the Powerlaw graph, the predicted number of triangles incident to each N e is its expected value, which can be computed
analytically in the CLV random graph model.
Baselines.
We compare our algorithms with the following baselines
• ThinkD and WRS (Arbitrary Order):
These are the state of the art empirical one-pass algorithms from Shin et al.
and Shin (2017) respectively.
The ThinkD paper presents two versions of the algorithm, called ‘fast’ and ‘accurate’.
We use the ‘accurate’ version
since it
provides better estimates.
We use the authors’ code
for
our experiments
Shin
et
al.
Shin, 2020)
• MVV (Arbitrary Order and Adjacency List):
We use the one pass streaming algorithms given in McGregor et al.
(2016) for the arbitrary order model and the adjacency list model.
Error
measurement
We measure
accuracy using the relative error | 1 − (cid:101) T/T | , where T is the true triangle count and (cid:101) T is the estimate returned by an algorithm.
Our plots show the space used by an algorithm (in terms of the number of edges) versus the relative error.
We report median errors over 50 independent executions of each experiment, ± one standard deviation.
4. R ESULTS FOR A RBITRARY O RDER T RIANGLE C OUNTING E XPERIMENTS
In this section, we give experimental results for Algorithm 4 which approximates the triangle count in arbitrary order streams.
Note that we need to set two parameters for Algorithm 4: p , which is the edge sampling probability, and ρ , which is the heaviness threshold.
In our theoretical analysis, we assume knowledge of a lower bound on T in order to set p and ρ , as is standard in the theoretical streaming literature.
However, in practice, such an estimate may not be available; in most cases, the only parameter we are given is a space bound for the number of edges that can be stored.
To remedy this discrepancy, we modify our algorithm slightly by setting a ﬁxed fraction of space to use for heavy edges ( 10% of space for all of our experiments) and setting p correspondingly to use up the rest of the space bound given as input.
See details in Supplementary Section F.3.
Oregon and CAIDA In Figures 1(a) and 1(b), we display the relative error as a function of increasing space for graph #4 in the dataset for Oregon and graph #30 for CAIDA 2006 .
These ﬁgures show that our algorithm outperforms the other baselines by as much as a factor of 5 .
We do not display the error bars for MVV and WRS for the sake of visual clarity, but they are comparable to or larger than both ThinkD and our algorithm.
A similar remark applies to all ﬁgures in Figure 1.
As shown in Figure 2, these speciﬁc examples are reﬂective of the performance of our algorithm across the whole sequence of graphs for Oregon and CAIDA.
We also show qualitatively similar results for CAIDA 2007 in Figure 4(a) in Supplementary Section F.4.
We also present accuracy results over the various graphs of the Oregon and CAIDA datasets.
We ﬁx the space to be 10% of the number of edges (which varies across graphs).
Our results for the
Oregon dataset are plotted in Figure 2(a) and the results for CAIDA 2006 and 2007 are plotted in
[Figure]
[Figure]
Figures 2(b) and 2(c), respectively.
These ﬁgures illustrate that the quality of the predictor remains consistent over time even after a year has elapsed between when the ﬁrst and the last graphs were created as our algorithm outperforms the baselines on average by up to a factor of 2 .
Reddit Our results are displayed in Figure 1(c).
All four algorithms are comparable as we vary space.
While we do not improve over baselines in this case, this dataset serves to highlight the fact that predictors can be trained using node or edge semantic information (i
[Figure]
e., features).
Wikibooks and Powerlaw For Wikibooks, we see in Figure 1(d) that our algorithm is outperforming ThinkD and WRS by at least a factor of 2 , and these algorithms heavily outperform the MVV algorithm.
Nonetheless, it is important to note that our algorithm uses the exact counts on the ﬁrst half of the edges (encoded in the predictor), which encode a lot information not available to the baselines.
Thus the takeaway is that in temporal graphs, where edges arrive continuously over time (e.g., citation networks, road networks, etc.), using a preﬁx of the edges to form noisy predictors can lead to a signiﬁcant advantage in handling future data on the same graph.
Our results for Powerlaw are presented in Figure 1(f).
Here too we see that as the space allocation increases, our algorithm outperforms ThinkD, MVV, and WRS.
Twitch and Wikipedia
In Figure 1(e), we see that three algorithms, ours, MVV, and WRS, are all comparable as we vary space.
The qualitatively similar result for Twitch is given in Figure 4(b).
These datasets serve to highlight that predictors can be trained using modern ML techniques such as
Graph Neural Networks.
Nevertheless, these predictors help our algorithms improve over baselines for experiments in the adjacency list model (see Section below).
Results for Adjacency List Experiments
Our experimental results for adjacency list experiments, which are qualitatively similar to the arbitrary order experiments, are given in full detail in Sections
F.5 and F.6.
A CKNOWLEDGMENTS
Justin is supported by the NSF Graduate Research Fellowship under Grant No. 1745302 and MathWorks Engineering Fellowship.
Sandeep and Shyam are supported by the NSF Graduate Research Fellowship under Grant No. 1745302.
Ronitt was supported by NSF awards CCF-2006664, DMS and CCF-1740751.
Piotr was supported by the NSF TRIPODS program (awards CCF1740751 and DMS-2022448), NSF award CCF-2006798 and Simons Investigator Award.
Talya is supported in part by the NSF TRIPODS program, award CCF-1740751 and Ben Gurion University
Postdoctoral Scholarship.
Honghao Lin and David Woodruff would like to thank for partial support from the National Science Foundation (NSF) under Grant No.
CCF-1815840.
A N OTATIONS T ABLE
Notation
Deﬁnition
G =
( V, E ) a graph G with vertex set V and edge set E n number of triangles m number of edges
T number of triangles (or 4-cycles) (cid:15) approximation parameter < s total ordering on the vertices according to their arrival in the adjacency list arrival model ∆ set of triangles in G (cid:3) set of 4-cycles in G
N ( v ) set of neighbors of node v
N xy number of triangles (4-cycles) incident to edge xy , i.e., |{ z | ( x, y, z ) ∈ ∆ }| ( |{ w, z | ( x, y, w.z ) ∈ (cid:3) }| )
R xy number of triangles ( x, z, y ) where x precedes z and z precedes y in the adjacency list arrival model, i.e., |{ z | ( x, z, y ) ∈ ∆ , x < s z < s y }| ∆ E maximum number of triangles incident to any edge ∆ V maximum number of triangles incident to any vertex
O ρ heavy edge oracle with threshold ρ
B O MITTED P ROOFS FROM S ECTION
In this section, we prove correctness and bound the space of Algorithm 1, thus proving Theorem 1.1.
In the adjacency list model, recall that we have a total ordering on nodes < s based on the stream ordering, where x < s y
if and only if the adjacency list of x arrives before the adjacency list of y in the stream.
For each xy ∈ E ( G ) , we deﬁne R xy =
| z : { x, y, z } ∈ ∆ and x < s z <
s y | .
Note that if y < s x , then R xy = 0 .
It holds that (cid:80) xy ∈ E R xy = T .
We ﬁrst give a detailed overview of the algorithm.
We ﬁrst consider the case when T ≥ ( m/(cid:15) ) 1 / 2 .
Let ρ = ( mT ) 1 / 3 .
We deﬁne the edge xy to be heavy if R xy ≥ ρ , light if R xy ≤
Tρ , and medium if
T ρ < R xy < ρ .
We train the oracle to predict, upon seeing xy , whether R xy is light or not.
We deﬁne
H, M , and L as the set of heavy, medium, and light edges, respectively.
Deﬁne T L = (cid:80)
xy ∈
L R xy .
We deﬁne T H and T M similarly.
Since (cid:80) xy ∈ E R xy = T =
T L + T M + T H , it sufﬁces to estimate each of these three terms.
For the light edges, as shown in Lemma B.2, if we sample edges with rate p 1 ≈ (cid:15) − 2 /ρ , with high probability we obtain an estimate of T L within error ± (cid:15)T .
For the heavy edges, we recall an observation in McGregor et al.
(2016): for an edge xy where R xy is large, even if we do not sample xy directly, if we sample from the stream (at rate p 3 ≈ (cid:15) − 2 /ρ ), we are likely to sample some edges in the set | xz : { x, y, z } ∈ ∆ and x <
s z <
s y | .
We refer to the sampled set of these edges as S aux .
Further, we will show that not only can we use S aux to recognize whether R xy is large, but also to estimate R xy (Lemma B.3).
What remains to handle is the medium edges.
Since medium edges have higher values of R xy than light edges, we sample with a larger rate, p 2 , to reduce variance.
However, we show there cannot be too many medium edges, so we do not pay too much extra storage space.
The overall space we use is (cid:101) O
( (cid:15) − 2 m 2 / 3 /T 1 / 3 ) .
When T < ( m/(cid:15) ) 1 / 2 our algorithm becomes much simpler.
Let ρ = √ m/(cid:15) .
We deﬁne the edge xy to be heavy if R xy ≥ T/ρ , and light otherwise (so in this case all the medium edges become heavy edges).
We directly store all the heavy edges and sub-sample with rate (cid:15) − 2 /ρ to estimate T L .
The total space we use is (cid:101) O
( (cid:15) − 1 √ m ) , which is optimal in this case.
See Algorithm 1 for more details of the implementation.
ρ p 1 = α(cid:15) − 2 /ρ , p 2 = min( β(cid:15) − 2 ρ/T
1) , p 3 = γ(cid:15) − 2 log n/ρ
6: else 7:
Set do 9: do 10: do 12: do 14: outputs then 16:
} 17: else
{ z : uz ∈ S aux , B uz = 1 , z ∈ N ( v ) }| .
20: if # ≥ p 3 ρ then { uv is HEAVY } 21:
A
A
22: else if uv ∈ S M then 23:
A m =
A m + C uv . {
Finished counting R uv } 24: else 24:
With probability p 2 , S M = { vu } ∪ S M .
25: return: A l /p 1 + A m /p
+ A h /p 3 .
Remark B.1.
In the adjacency list model, we assume the oracle can predict whether R xy ≥
Tc or not, where R xy is dependent on the node order.
This may sometimes be impractical.
However, observe that N xy ≥ R xy and (cid:80) N xy = 3 T .
Hence, our proof still holds if we assume what the oracle can predict is whether N xy ≥ Tc , which could be a more practical assumption.
Recall the following notation from Section 2.
We deﬁne the edge xy to be heavy if R xy ≥ ρ , light if
R xy ≤ Tρ , and medium if Tρ < R xy < ρ .
We deﬁne H, M , and L as the set of heavy, medium, and light edges, respectively.
Deﬁne T L = (cid:80)
xy ∈
L R xy .
We deﬁne T H and T M similarly.
Also, recall that since (cid:80) xy ∈
E R xy =
T =
T L + T M + T H , it sufﬁces to estimate of these three terms.
Lemma B.2.
Let A be an edge set such that for each edge xy ∈ A , R xy ≤ Tc , for some parameter c , and let S be a subset of A such that every element in A is included in S with probability p = 1 τ for τ = (cid:15) 2 · c/α, so that 1 /τ =
α(cid:15) − 2 1 c independently, for a sufﬁcient large constant α .
Then, with probability at least 9 / 10 , we have
[Equation]
(cid:88)
xy
[Equation]
Proof.
We let Y =
τ (cid:80) i ∈ S R
i =
τ (cid:80) i ∈
A f i R i , where f i =
when i ∈ S
and
f i =
otherwise.
It follows that E [ Y ] =
T A .
Then,
[Equation]
To bound this, we notice that R i ≤ Tc and (cid:80) i R i
=
T
A ≤ T , so
[Equation]
By Chebyshev’s inequality, we have that
[Equation]
The following lemma shows that for the heavy edges, we can use the edges we have sampled to recognize and estimate them.
Lemma B.3.
Let xy be an edge in E .
Assume we sample the edges with rate p 3 = β(cid:15)
− 2 log n/ρ in the stream for a sufﬁciently large constant β , and consider the set # = | xz has been sampled : { x, y, z } ∈ ∆ and x < s z < s y | .
Then we have the following: if R xy ≥ ρ 2 , with probability at least 1 − 1 n 5 , we have p − 1 3 · # = (1 ± (cid:15) )
R xy .
If R xy < ρ 2 , with probability at least 1 − 1 n 5 , we have p − 1 3 · # < ρ .
Proof.
We ﬁrst consider the case when R xy ≥ ρ 2 .
For each edge i ∈ | xz : { x, y, z } ∈ ∆ and x < s z < s y | , let X i = 1 if i has been sampled, or
X i =
otherwise.
We can see the X i are i.i.d.
Bernoulli random variables and E
[ X i ] =
p 3 .
By a
Chernoff bound we have
[Equation]
Therefore, we have
[Equation]
Alternatively, when R xy < ρ 2 , we have
[Equation]
which means p − 1 3 · # < ρ with probability at least 1 − 1 n 5 .
We are now ready to prove Theorem 1.1, restated here for the sake of convenience.
Theorem 1.1.
There exists a one-pass algorithm, Algorithm 1, with space complexity(cid:101)
O (min( (cid:15) − 2 m 2 / 3 /T 1 / 3 , (cid:15) − 1 m 1 / 2 )) in the adjacency list model that, using a learning-based oracle, returns a (1 ± (cid:15) )
-approximation to the number T of triangles with probability at least 4 7 / 10 .
Proof.
We ﬁrst consider the case when T ≥ ( m/(cid:15) ) 1 / 2 .
For each edge xy ∈ H , from Lemma B.3 we have that with probability at least 1 − 1 n 5 , # /p 3 =
(1 ± (cid:15) )
R xy in Algorithm 1.
If xy / ∈ H , then with probability 1 − 1 n 5 , xy will not contribute to A h .
Taking a union bound, we have that with probability at least 1 − 1 /n 3 ,
[Equation]
We now turn to deal with the light and medium edges.
For a light edge xy , it satisﬁes R xy ≤ T/ρ and therefore we can invoke Lemma B.2 with the parameter c set to ρ .
For a medium edge xy , it satisﬁes R xy ≤ ρ = T/
( T/ρ ) , and therefore we can invoke Lemma B.2 with the parameter c set to
T/ρ .
Hence, we have that with probability 4 / 5 ,
[Equation]
Putting everything together, we have that with probability 7 / 10 ,
[Equation]
Now we analyze the space complexity.
We note that there are at most ρ medium edges.
Hence, the expected total space we use is O ( mp 3 + ρp 2 + mp 1 ) =
O ( (cid:15) − 2 m 2 / 3 /T 1 / 3 log n ) 5 .
When T < ( m/(cid:15) ) 1 / 2 , as described in Section 2, we only have two classes of edges: heavy and light.
We will save all the heavy edges and use sub-sampling to estimate T L .
The analysis is almost the same.
B. N OISY O RACLES FOR THE A DJACENCY L IST M ODEL
Proof of Theorem 1.4.
We ﬁrst consider the case that T < ( m/(cid:15) ) 1 / 2 .
Recall that at this time, our algorithms becomes that we save all the heavy edges xy such that p ( e ) ≥ (cid:15)T/ √ m and sampling the light edges with rate p l = (cid:15) − 1 / √ m .
We deﬁne L as the set of deemed light edges, H as the set of deemed heavy edges and L = |L| , H = |H| .
Let ρ =
(cid:15)T/ √ m , then we have the condition that for every edge e ,
[Equation]
We will show that, ( i ) E
[ H ] ≤ ( K + 1) √ m(cid:15) , ( ii ) Var
[ A L /p 1 ] ≤ (4 K + 2) (cid:15) 2 T 2 .
Under the above conditions we can get that with probability at least 9 / 10 we can get a (1 ± O ( √ K ) · (cid:15) ) -approximation of T by Chebyshev’s inequality.
For ( i ) , we divide H
=
H h ∪ H l , where the edges in H h are indeed heavy edges, and the edges in H l are indeed light edges.
Then, it is easy to see that | H h | ≤ T/ρ .
For every light edges, the probability that it is included in H l is at most K · R e ρ , hence we have
[Equation]
m
[Equation]
which implies E [ H ] ≤ ( K + 1) √ m(cid:15) .
For ( ii ) , we also divide L =
L l ∪ L h similarly.
Then we have Var [ A L ] ≤ 2( Var [ X ] + Var [ Y ]) , where X = (cid:80) e ∈ L l ,e has been sampled R e , Y = (cid:80)
e ∈
L h ,e has been sampled R e .
Similar to the proof in
Lemma B.2, we have
[Equation]
Now we bound Var [ Y ] .
For every heavy edge we have that Pr [ e ∈ L h ] ≤ K · ρR e .
Then, condition on the oracle O ρ , we have
[Equation]
Also, we have
[Equation]
From Var [ Y ] =
E [ Var [ Y |
O ρ ]] + Var [ E [ Y | O ρ ]] we know Var [ Y ] ≤ 2 p 1 KρT , which means
Var [ A l /p 1 ] ≤ 1 p 1 (4 K +
[Equation] [Equation]
= (
[Equation]
) (cid:15)
[Equation]
When T ≥ ( m/(cid:15) ) 1 / 2 , recall that the oracle O ρ only needs to predict the edges that are medium edges or light edges.
So, we can use the same way to bound the expected space for the deemed medium edges and the variance of the deemed light edges.
B. A V ALUE P REDICTION O RACLE FOR THE A DJACENCY L IST M ODEL
In this section, we consider two types of value-prediction oracles, and variants of Algorithm 1 that take advantage of these oracles.
Recall that given an edge xy , R xy =
| z : { x, y, z } ∈ ∆ and x < s z <
s y | .
Deﬁnition B.4.
A value-prediction oracle with parameter ( α, β ) is an oracle that, for any edge e , outputs a value p ( e ) for which E [ p ( e )] ≤ αR e + β , and Pr [ p ( e ) <
R e
λ − β ] ≤ Ke − λ for some constant K and any λ ≥ 1 .
Our algorithm is based on the following stability property of exponential random variables (see e.g. Andoni (2017))
Lemma B.5.
Let x 1 , x 2 , ..., x n > 0 be real numbers, and let u 1 , u 2 , ...
u n be i.i.d.
standard exponential random variables with parameter λ = 1 .
Then we have that the random variable max( x 1 u 1 , ..., x n u n ) ∼ xu , where x = (cid:80) i x i and u ∼ Exp(1) .
Lemma B.6.
Let r
= O (ln(1 /δ ) /(cid:15) 2 ) and (cid:15) ≤ 1 / 3 .
If we take r independent samples X 1 , X 2 , ..., X r from Exp(1) and let X = median( X 1 , X 2 , ..., X r ) , then with probability 1 − δ , X ∈
[ln 2(1 − (cid:15) ) , ln 2(1 + 5 (cid:15) )] .
Proof.
We ﬁrst prove the following statement: with probability 1 − δ , F ( X ) ∈ [1 / 2 − (cid:15), 1 / 2 + (cid:15) ] , where F is the cumulative density function of a standard exponential random variable.
Consider the case when F ( X ) ≤ 1 / 2 − (cid:15) .
We use the indicator variable Z i where Z i = 1 if
F ( X i ) ≤ 1 / 2 − (cid:15) or Z i = 0 otherwise.
Notice that when F ( X ) ≤ 1 / 2 − (cid:15) , at least half of the Z i are 1 , so Z = (cid:80)
i
Z i ≥ r/ 2 .
On the other hand, we have E [ Z ] = r/ 2 − r(cid:15) , by a Chernoff bound.
We thus have
[Equation]
Therefore, we have with probability at most δ/ 2 , F ( X ) ≤ 1 / 2 − (cid:15) .
Similarly, we have that with probability at most δ/ 2 , F ( X ) ≥ 1 / 2 + (cid:15) .
Taking a union bound, we have that with probability at least 1 − δ , F ( X ) ∈
[1 / 2 − (cid:15), 1 / 2 + (cid:15) ] .
Now, condition on F ( X ) ∈ [1 / 2 − (cid:15), 1 / 2 + (cid:15) ] .
Note that F − 1 ( x ) = − ln(1 − x ) , so if we consider the two endpoints of F ( X ) , we have
[Equation]
and
[Equation]
which indicates that X ∈ [ln 2(1 − (cid:15) ) , ln 2(1 + 5 (cid:15) )] .
The algorithm utilizing this oracle is shown in Algorithm 2.
As described in Section 2, here we runs
O ( 1 (cid:15) 2 ) copies of the subroutine and takes a medium of them.
For each subroutine, we aim to output the maximum value max e R e /u e .
We will show that with high probability for each subroutine, this maximum will be at least T/ log( K/(cid:15) ) .
Hence, we only need to save the edge e , such that ( R e + β ) /u
e is larger than T/ log( K/(cid:15) ) .
However, one issue here is we need the information of T .
We can remove this assumption when β = 0 using the following way: we will show that with high probability, the total number of edges we save is less than H
= O ( 1 (cid:15) 2 log 2 ( K/(cid:15) )
( α + mβ/T )) , so every time when a new edge comes, we can search the minimum value M such that the total number of edges e that in at least one subroutine ( p ( e ) + β ) /u
e > M is less than H (we count an edge multiple times if it occurs in multiple subroutines), then use M as the threshold.
We can see M will increase as the new edge arriving and it will be always less than H .
for each incident edge yx do 10: for i = 0 to c(cid:15) − 2 do 11: if xy ∈ S
i then { Finished counting C xy , update the current maximum } 12:
A i ← max( A i , C ixy /u ixy ) .
13: else { Put yx into the sample sets temporarily } 14:
Let ˆ R yx be the predicted value of R yx .
15: ˆ R yx ← ˆ R yx + β .
Generate u iyx ∼ Exp(1) .
Set Q i ← Q
i ∪ { u iyx } , S i ← S
i ∪ { yx } .
Set M to be the minimal integer such that (cid:80)
i s i ≤ H , where s i = |{ e ∈ E
| ˆ R e /u ie ≥ M }| .
18: for i = 0 to c(cid:15) − 2 do { Adjust the samples to be within the space limit } 19:
Let Q i ← Q i \ { u iab } , S
i ← S i \ { ab } if ˆ R ab /u iab < M for all ab ∈ E .
20: for i = 0 to c(cid:15) − 2 do 21:
X ← X ∪ { A i } 22: return: ln 2 · median( X ) .
We are now ready to prove Theorem 1.5.
We ﬁrst recall the theorem.
Theorem 1.5.
Given an oracle with parameters ( α, β ) , there exists a one-pass algorithm, Algorithm 2, with space complexity O ( (cid:15) − 2 log 2 ( K/(cid:15) )
( α + mβ/T )) in the adjacency list model that returns a (1 ± (cid:15) ) -approximation to the number of triangles T with probability at least 7 / 10 .
Proof.
It is easy to see that for the i -th subroutine, the value f
( i ) =
max e R e /u ie is the sample from the distribution T/u , where u ∼ Exp(1) .
And if the edge e i , for which the true value R e i /u ie i is the maximum value among E , is included in S i , then the output of the i -th subroutine will be a sample from T/u .
Intuitively, the prediction value p ( e ) will help us ﬁnd this edge.
We will ﬁrst show that with probability at least 9 / 10 the following events will happen:
• ( i ) f
( i ) ≥ c 1 T/ log(1 /(cid:15) ) for all i , where c 1 is a constant.
• ( ii ) Let s i be the number of the edge e in the i -th subroutine
such that ( p ( e ) + β ) /u
ie ≥ c 21 T/ log 2 ( K/(cid:15) ) , then (cid:80) i s i ≤ c 2 ( 1 (cid:15) 2 log 2 ( K/(cid:15) )
( α + mβ/T )) for some constant c 2 .
• ( iii ) p ( e i ) + β ≥ c 1 f ( i ) / log( K/(cid:15) ) for all i .
For ( i ) , recall that f ( i ) ∼ T/u, where u ∼ Exp(1) .
Hence, we have
[Equation]
Taking an union bound we get that with probability at least 99 / 100 , ( i ) will happen.
For each edge e , we have
[Equation]
similarly we can get that with probability at least 99 / 100 , ( iii ) will happen.
For ( ii ) , we ﬁrst bound the expectation of s i .
For each edge e , we have
[Equation]
Hence, we have
[Equation]
Using Markov’s inequality, we get that with probability at least 99 / 100 , ( ii ) will happen.
Finally, taking a union bound, we can get with probability at least 9 / 10 , all the three events will happen.
Now, conditioned on the above three events, we will show that all the subroutine i will output f ( i ) .
For the subroutine i , from ( ii ) we know that M will be always smaller than O ( T/ log 2 ( K/(cid:15) )) , and from ( i ) and ( iii ) we get that ( p ( e i )+ β ) /u ie i ≥ Ω( T/ log 2 ( K/(cid:15) )) .
Hence, the edge e i will be saved in the subroutine i .
From Lemma B.6 we get that with probability at least 7 / 10 , we can ﬁnally get a (1 ± (cid:15) ) -approximation of T .
We note that when β = 0 , the space complexity will become O ( 1 (cid:15) 2 log 2 ( K/(cid:15) ) α ) , and in this case we will not need a lower bound of T .
We will also consider the following noise model.
Deﬁnition B.7.
A value-prediction oracle with parameter ( α, β ) is an oracle that, for any edge e , outputs a value p ( e ) for which Pr [ p ( e ) > λαR e + β ] ≤ Kλ , and Pr [ p ( e ) <
R e
λ − β ] ≤ Kλ for some constant K and any λ ≥ 1 .
The algorithm for this oracle is shown in Algorithm 3.
We now state our theorem.
Theorem B.1.
Given an oracle with parameter ( α, β ) , there exists a one-pass algorithm, Algorithm 3, with space complexity O ( K (cid:15) 2 (cid:0) α (log n ) 3 log log n + mβ/T (cid:1) ) in the adjacency list model that returns a (1 ± √ K · (cid:15) ) -approximation to T with probability at least 7 / 10 .
Proof.
Deﬁne q ( e ) =
p ( e ) + β , it is easy to see that from the condition we can get that
[Equation]
We deﬁne the set E i such that e ∈ E i if and only if q ( e ) ∈
I i .
We can see E
[ A i /p i ]
= (cid:80)
e ∈ E i R e .
Now, we bound Var
[ A i /p i ] when i ≥ 1 .
Let X = (cid:80) e ∈ E i ,R e ≤ 2 i +1 β R e and Y =
(cid:80) e ∈ E
i ,R e > 2 i +1 β R e , then we have Var [ A i ] ≤ 2( Var [ X ]+
Var [ Y ]) .
Similar to the proof in Lemma B.2, we can get that
[Equation]
Let yx do 14:
15: return
For Var [ Y ] , recall that for an edge e such that R e ≥
i +1 β , we have Pr [ e ∈ E i ] ≤ K q ( e ) R e ≤
K
i +1 β R e .
Then, condition on the oracle O θ , we have
E
[Equation]
e
[Equation]
And
Var [ E [ Y | O θ ]]
[Equation]
p
[Equation]
From Var [ Y ] =
E [ Var [ Y | O θ ]]
+ Var [ E [ Y | O θ ]] we get that Var [ Y ] ≤ 2 p
i K
i +1 βT , from which we can get that Var
[ A i /p i ]
= 1 p 2 i Var
[ A i ]
=
O ( K ( (cid:15)/ log n ) 2 T 2 ) .
Using Chebyshev’s inequality and taking a median over O (log log n ) independent trials, we can get X i =
median( A ji ) satisﬁes
| X i − (cid:80) e ∈ E i R e | ≤ √ K ( (cid:15)/ log n ) T with probability 1 − O
(1 / log n ) .
A similar argument also holds for I 0 , which means that after taking a union bound, with probability 9 / 10 , the output of the algorithm 3 is a (1 ± √ K · (cid:15) ) - approximation of T .
Now we analyze the space complexity of Algorithm 3.
For i ≥ 1 , we have
[Equation]
and | E 0 | ≤ m , hence we have that the total expectation of the space is (cid:80) i p i E
[ | E i | ] =
O ( K(cid:15) 2 (cid:0) α (log n ) 3 log log n + mβ/T (cid:1) ) .
C O MITTED P ROOFS FROM S ECTION
We ﬁrst present Algorithm 4, and then continue to prove Theorem 1.2.
In the following algorithm, for two sets A, B of edges, we let w A,B represent the set of pairs of edges ( u, v ) , where u ∈ A, v ∈
B , and u, v share a common vertex.
10: for w do 11: if ( e w ) is a triangle then 12:
Increment (cid:96) 1 = (cid:96)
13: for each w ∈ w S L ,H do 14: if ( e, w ) is a triangle then 15:
Increment (cid:96) 2 = (cid:96)
16: for each w ∈ w H,H do 17: if ( e, w ) is a triangle then 18:
Increment (cid:96) 3 = (cid:96)
19: return: (cid:96)
C. P ROOF OF T HEOREM 1.
Recall that we ﬁrst assume that Algorithm 4 is given access to a perfect oracle O ρ .
That is, for every edge e ∈ E ,
O ρ ( e )
= (cid:26) LIGHT if T e ≤ ρ
HEAVY if T e > ρ, where T e as the number of triangles incident to the edge e .
Theorem 1.2.
First, we assume that T ≥ (cid:15) − 2 .
If not, our algorithm can just store all of the edges, since m ≤ (cid:15) − 1 · m/ √ T.
For each integer i ≥ 1 , let E ( i ) denote the set of edges that are part of exactly i triangles, and let
E i = |E
( i ) | .
Since each triangle has 3 edges, we have that (cid:80) i ≥ 1 i · E ( i ) =
3 T.
Let T 1 denote the set of triangles whose ﬁrst two edges in the stream are light (according to O ρ ).
For every triangle t i in T 1 , let χ i denote the indicator of the event that the ﬁrst two edges of t i are sampled to S L , and let (cid:96)
=
(cid:80) i χ i .
Since each χ i is 1 with probability p 2 , Ex [ χ i ] =
p 2 , and
Var [ χ i ] =
p 2 − p 4 ≤ p 2 .
For any two variables χ i , χ j , they must be uncorrelated unless the triangles t i , t j share a light edge that is among the ﬁrst two edges of t i and among the ﬁrst two edges of t j .
Moreover, in this case, Cov [ χ i , χ j ] ≤ Ex [ χ i χ j ] = P ( χ i = χ j = 1) .
This event only happens if the ﬁrst two edges of both t i and t j are sampled in S L , but due to the shared edge, this comprisesedges in total, so P ( χ i = χ
j = 1) =
p 3 .
Hence, if we deﬁne t 12 i as the ﬁrst two edges of t i in the stream for each triangle t i ,
[Equation]
and
Var
[Equation]
t
= t 2 · E ( t ) ≤ ( p 2 + 3 p 3 · ρ ) · T .
The ﬁrst line follows by adding the covariance terms; the second line follows since each light edge e has at most T 2 e pairs ( t i , t j ) intersecting at e ; the third line follows by summing over t =
T e , ranging from 1 to ρ , instead over e ; and the last line follows since (cid:80) t ≥ 1 t · E ( t ) = 3 T .
Now, let T 2 denote the set of triangles whose ﬁrst two edges in the stream are light and heavy (in either order).
For every triangle t = ( e 1 , e 2 , e 3 ) in T 2 , e 1 , e 2 are sampled to S L ∪ H with probability p .
Also all other triangles have no chance to contribute to (cid:96) 2 .
Hence, Ex [ (cid:96) 2 ] = p · T 2 .
Also, two triangles t i , t j ∈ T 2 that intersect on an edge e have Cov ( χ i , χ j ) (cid:54) = 0 only if e is light and e ∈ t 12 i , t 12 j .
In this case.
[Equation]
since if e is added to S L (which happens with probability p ) then χ i = χ
j = 1 as the other edges in t 12 i , t 12 j are heavy and are added to H with probability 1.
Therefore,
[Equation]
by the same calculations as was done to compute Var [ (cid:96) 1 ] .
Finally, let T 3 denote the set of triangles whose ﬁrst two edges in the stream are heavy.
Then (cid:96) 3 = |T 3 | .
Now, using the well known fact that Var [ X + Y ] ≤ 2( Var [ X ]+ Var [ Y ]) for any (possibly correlated) random variables X, Y, we have that
[Equation]
since (cid:96) 3 has no variance.
Moreover, Ex [ p − 2 (cid:96) 1 + p − 1 (cid:96) 2 + (cid:96) 3 ] = |T 1 | + |T 2 | + |T 3 |
=
T .
So, by
Chebyshev’s inequality, since (cid:96) =
(cid:96) 1 /p 2 + (cid:96) 2 /p
+ (cid:96) 3 ,
Pr[ | (cid:96) − T | > (cid:15)T ] <
Var [ (cid:96) ] (cid:15) 2 · T 2 < 4( p − 2 + 3 p − 1 ρ ) (cid:15) 2 · T .
Therefore, setting p = C · max (cid:110) 1 / ( (cid:15) · √ T ) , ρ/ ( (cid:15) 2 · T ) (cid:111) for some ﬁxed constant C implies that, with probability at least 2 / 3 , (cid:96) is a (1 ± (cid:15) ) -multiplicative estimate of T .
Furthermore, the expected space complexity is O ( mp + H )
=
O ( mp + T/ρ ) .
Setting ρ = max { (cid:15)T/ √ m, 1 } implies that the space complexity is O ( (cid:15) − 1 m/ √ T + (cid:15) − 2 m/T + (cid:15) − 1 √ m · T/T ) =
O ( (cid:15) − 1 m/ √ T + (cid:15) − 1 √ m · T/T ) , since we are assuming that T ≥ (cid:15) − 2 .
Hence, assuming that T is a constant factor approximation of T , the space complexity is O ( (cid:15) − 1 m/ √ T + (cid:15) − 1 √ m ) .
C.
P ROOF OF T HEOREM 1.2 FOR THE K- NOISY ORACLE
In this section, we prove Theorem 1.2 for the case that the given oracle is a K -noisy oracle, as deﬁned in Deﬁnition 1.1.
That is, we prove the following:
Theorem C.1.
Suppose that the oracle in Algorithm 4 is a K -noisy oracle as deﬁned in Deﬁnition 1.1 for a ﬁxed constant K .
Then with probability 2 / 3 , Algorithm 4 returns (cid:96) ∈ (1 ± (cid:15) ) T , and uses space at most O (cid:16) (cid:15) − 1 (cid:16) m/ √ T + √ m (cid:17)(cid:17) .
Recall that in Deﬁnition 1.1, we deﬁned a K -noisy oracle if the following holds.
For every edge e , 1 − K · ρN e ≤ Pr[ O ρ ( e ) = HEAVY ] ≤ K · N e ρ .
We ﬁrst visualize this model.
To visualize this error model, in Figure 3 we have plotted N e versus the range (cid:104) 1 − K · ρN e , K · N e ρ (cid:105) for K = 2 .
We set
N e to vary on a logarithmic scale for clarity.
For example, if N e exceeds the threshold ρ by a factor of 2 , there is no restriction on the oracle output, whereas if N e exceeds ρ by a factor of 4 , then the oracle must classify the edge as heavy with probability at least 0 .
In contrast, the blue piece-wise line shows the probability Pr
[ O ρ ( e ) =
HEAVY ] if the oracle O ρ is a perfect oracle.
Proof of Theorem C.1.
We follow the proof of Theorem 4.1 from the main paper.
Let T 1 be the set of triangles such that their ﬁrst two edges in the stream are light according to the oracle.
Let Tbe the set of triangles such that their ﬁrst two edges in the stream are determined light and heavy according to the oracle.
Finally, let T 3 be the set of triangles for which their ﬁrst two edges are determined heavy by the oracle.
Furthermore, we deﬁne χ i for each triangle t i , t ( e ) for each edge e , and E ( i ) , E i for each i ≥ 1 as done in Theorem 4.1.
Finally, we deﬁne L ( i ) as the set of deemed light edges that are part of exactly i triangles, and L ( i ) =
|L ( i ) | .
First, note that the expectation (over the randomness of O ρ ) of L ( i ) is at most E ( i ) · K · ρi , since our assumption on Pr[ O ρ ( e )
= heavy ] tells us that Pr[ O ρ ( e ) = light ] ≤ K · ρi if t ( e ) = i .
Therefore, by the same computation as in Theorem 4.1, we have that, conditioning on the oracle,
Ex [ (cid:96) 1 | O ρ ] = p 2 |T 1 ] and
[Equation]
But since Ex O ρ [ L ( t )] ≤ K · ρt · E ( t ) for all t , we have that
[Equation]
A similar analysis to the above shows that Ex [ (cid:96) 2 ] = p · |T 2 | and that
[Equation]
Hence, as in Theorem 4.1, we have Ex [ (cid:96) |
O ρ ] = T and Ex O ρ
[ Var [ (cid:96) |
O ρ ]] ≤ 4 T · ( p − 2 + 3 Kp − 1 ρ ) .
Thus, Ex [ (cid:96) ]
=
Ex [ Ex [ (cid:96) | O ρ ]] = T and Var [ (cid:96) ]
=
Ex O ρ [ Var [ (cid:96) |
O ρ ]]+ Var O ρ
[ Ex [ (cid:96) | O ρ ]] ≤ 4 T · ( p − 2 + 3 Kp − 1 ρ ) by the laws of total expectation/variance.
Therefore, since K is a constant, the variance is the same as in Theorem 4.1, up to a constant.
Therefore, we still have that (cid:96) ∈
(1 ± O ( (cid:15) ))
· T with probability at least 2 / 3 .
Finally, the expected space complexity is bounded by
[Equation]
since (cid:80) t ≥ 1 E ( t ) · t = 3 T and K
= O (1) .
Hence, setting ρ and p as in the proof of Theorem 4. implies that the returned value is a (1 ± (cid:15) ) -approximation of T , and the the space complexity is
O ( (cid:15) − 1 ( m/ √ T + √ m )) as before.
C. L OWER B OUND
In this section, we prove a lower bound for algorithms in the arbitrary order model that have access to a perfect heavy edge oracle.
Here heavy means N uv ≥ T/c for a pre-determined threshold c , and we assume c
= o ( m ) and T/c > 1 , as otherwise the threshold will be too small or too close to the average to give an accurate prediction.
The following theorem shows an Ω(min( m/ √ T, m 3 / 2 /T ))
lower bound even with such an oracle.
Theorem C.2.
Suppose that the threshold of the oracle c
= O ( m q ) ,
where 0 ≤ q < 1 .
Then for any T and m , T ≤ m , there exists m (cid:48)
=
Θ( m ) and T (cid:48)
=
Θ( T ) such that any one-pass arbitrary order streaming algorithm that distinguishes between m (cid:48) edge graphs with 0 and T (cid:48) triangles with probability at least 2 / 3 requires Ω( m/ √ T ) space.
For any T and m , T = Ω( m 1+ δ ) where max(0 , q − 12 ) ≤ δ < 12 , there exists m (cid:48)
=
Θ( m ) and T (cid:48)
=
Θ( T ) such that any one-pass arbitrary order streaming algorithm that distinguishes between m (cid:48) edge graphs with 0 and T (cid:48) triangles with probability at least 2 / 3 requires Ω( m 3 / 2 /T ) space.
When T ≤ m , we consider the hubs graph mentioned in Kallaugher & Price (2017).
Deﬁnition C.3 (Hubs Graph, Kallaugher & Price (2017)) .
The hubs graph H r,d consists of a single vertex v with 2 rd incident edges, and d edges connecting disjoint pairs of v ’s neighbors to form triangles.
It is easy to see that in H r,d , each edge has at most one triangle.
Hence, there will not be any heavy edges of this kind in the graph.
In Kallaugher & Price (2017), the authors show an Ω( r √ d ) lower bound for the hubs graph.
Lemma C.4 (Kallaugher & Price (2017)) .
Given r and d , there exist two distributions G 1 , G 2 on subgraphs G 1 and G 2 of H r,d , such that any algorithm which distinguishes them with probability at least 2 / 3 requires Ω( r √ d ) space, where G i has Θ( rd ) edges and T ( G 1 ) =
d, T ( G 2 ) = 0 .
Now, given T and m , where T ≤ m , we let r = Θ( m/T ) and d = T .
We can see H r,d has Θ( m ) edges and T triangles, and we need Ω( r √ d )
= Ω( m/ √ T ) space.
When T > m , we consider the following Ω( m 3 / 2 /T ) lower bound in Bera & Chakrabarti (2017).
Let H be a complete bipartite graph with b vertices on each side (we denote the two sides of vertices by A and B ).
We add an additional N vertex blocks V 1 , V 2 , ...
V N with each | V i | =
d .
Alice has an N -dimensional binary vector x .
Bob has an N -dimensional binary vector y .
Both x and y have exactly N/ 3 coordinates that are equal to 1 .
Then, we deﬁne the edge sets
[Equation]
and
[Equation]
Let the ﬁnal resulting graph be denoted by G =
( V, E )
where V = V H ∪ V 1 ∪ ... ∪ V N and
E = E H ∪ E Alice ∪ E Bob .
In Bera & Chakrabarti (2017), the authors show by a reduction to the DISJ N/ 3 N problem in communication complexity, that we need Ω( N ) space to distinguish the case when G has 0 triangles from the case when G has at least b 2 d triangles.
Given m and T , T =
Θ( m 1+ δ ) where 1 ≤ δ < 1 / 2 , as shown in Bera & Chakrabarti (2017), we can set b = N s and d = N s − 1 where s = 1 / (1 − 2 δ ) .
This will make m =
Θ( N 2 s ) and
T =
Θ( N 3 s − 1 ) , which will make the Ω( m 3 / 2 /T )
lower bound hold.
Note that at this time each edge will have an O ( 1 m ) -fraction of triangles or an O ( m 1 − 12 s ) -fraction of triangles.
Assume the threshold of the oracle c
= O ( m q ) .
When δ ≥ max(0 , q − 12 ) , there will be no heavy edges in the graph.
Theorem C.2 follows from the above discussion.
D F OUR -C YCLE C OUNTING IN THE A RBITRARY O RDER M ODEL
In the 4-cycle counting problem, for each xy ∈ E ( G ) , deﬁne N xy =
| z, w : { x, y, z, w } ∈ (cid:3) | as the number of 4-cycles attached to edge xy .
In this section, we present the one-pass (cid:101)
O ( T 1 / 3 + (cid:15) − 2 m/T 1 / 3 ) space algorithm behind Theorem 1.
and the proof of the theorem.
We begin with the following basic idea.
• Initialize: C ← 0 , S ← ∅ .
On the arrival of edge uv : – With probability p , S ← { uv } ∪ S .
– C ← C + |{{ w, z } : uw, wz and zv ∈ S }| .
• Return C/p 3 .
Following the analysis in Vorotnikova (2020), we have
[Equation]
where ∆ E and ∆ W denote the maximum number of 4-cycles sharing a single edge or a single wedge.
There are two important insights from the analysis: ﬁrst, this simple sampling scheme can output a good estimate to the number of 4-cycles on graphs that do not have too many heavy edges and heavy wedges (the deﬁnitions of heavy will be shown later).
Second, assuming that we can store all the heavy edges, then at the end of the stream we can also estimate the number of 4-cycles that have exactly one heavy edge but do not have heavy wedges.
For the 4-cycles that have heavy wedges, we use an idea proposed in McGregor & Vorotnikova (2020); Vorotnikova (2020): for any node pair u and v , if we know their number of common neighbors (denoted by k ), we can compute the exact number of 4-cycles with u, v as a diagonal pair (i.e., the four cycle contains two wedges with the endpoints u, v ), and this counts to (cid:0) k 2 (cid:1) .
Furthermore, if k is large (in this case we say all the wedges with endpoints u, v are heavy), we can detect it and obtain a good estimation to k by a similar vertex sampling method mentioned in Section 3.
We can then estimate the total number of 4-cycles that have heavy wedges with our samples.
At this point, we have not yet estimated the number of 4-cycles having more than one heavy edge but without heavy wedges.
However, McGregor & Vorotnikova (2020) shows this class of 4-cycles only takes up a small fraction of T , and hence we can get a (1 ± (cid:15) ) -approximation to T even without counting this class of 4-cycles.
The reader is referred to Algorithm 5 for a detailed version of our one-pass, 4-cycle counting algorithm.
Before the stream, we randomly select a node set S , where each vertex is in S with probability
, and we later store all edges that are incident to S during the stream.
In the meantime, we deﬁne the edge uv to be heavy if N uv ≥ T 2 / 3 and train the oracle to predict whether uv is heavy or not when we see the edge uv during the stream.
Let p = (cid:101) O
( (cid:15) − 2 /T 1 / 3 ) .
We store all the heavy edges and sample each light edge with probability p during the stream.
Upon seeing see a light edge uv , we look for the 4-cycles that are formed by uv and the light edges that have been sampled before, and then record them in set D .
At the end of the stream, we ﬁrst ﬁnd all the node pairs that share many common neighbors in S and identify them as heavy wedges.
Then, for each 4-cycle d ∈ D , we check if d has heavy wedges, and if so, remove it from D .
Finally, for each heavy edge uv indicated by the oracle, we compute the number of 4-cycles that are formed by uv and the sampled light edges, and without heavy wedges.
This completes all parts of our estimation procedure.
We now prove Theorem 1.3, restated here for the sake of convenience.
Theorem 1.3.
There exists a one-pass algorithm, Algorithm 5, with space complexity (cid:101)
O ( T 1 / 3 + (cid:15) − 2 m/T 1 / 3 ) in the arbitrary order model that, using a learning-based oracle, returns a (1 ± (cid:15) ) - approximation to the number T of four cycles with probability at least 7 / 10 .
Proof.
From the Lemma 3 in Vorotnikova (2020), we know that with probability at least 9 / 10 , we can get an estimate
A w such that
[Equation]
Where T w is the number of the 4-cycles that have at least one heavy wedge.
We note that if a 4cycle has two heavy wedges, it will be counted twice.
However, Vorotnikova (2020) shows that this double counting is at most O ( T 2 / 3 )
=
O ( (cid:15) ) T .
For the edge sampling algorithm mentioned in D, from the analysis in Vorotnikova (2020), we have
Var [ A l /p 3 ] ≤ O ( T/p 3 + T ∆ E /p + T ∆ W /p 2 ) .
Notice that in our algorithm, the threshold of the heavy edges and heavy wedges are N uv ≥ T 2 /and N w ≥ T 1 / 3 , respectively, which means Var [ A l p 3 ]
= O ( (cid:15) 2 T 2 ) .
Hence, we can get an estimate A l such that with probability at least 9 / 10 ,
[Equation]
where T l is the number of 4-cycles that have no heavy edges.
Similarly, we have with probability at least 9 / 10 ,
[Equation]
, where T h is the number of 4-cycles that have at most one heavy edge.
One issue is that the algorithm has not yet estimated the number of 4-cycles having more than one heavy edge, but without heavy wedges.
However, from Lemma 5.1 in McGregor & Vorotnikova (2020) we get that the number of this kind of 4-cycles is at most O ( T 5 / 6 )
=
O ( (cid:15) ) T .
Hence putting everything together, we have
[Equation]
which is what we need.
Now we analyze the space complexity.
The expected number of light edges we sample is
O ( mp ) =
(cid:101) O
( (cid:15) − 2 m/T 1 / 3 ) and the expected number of nodes in S and edges in E S is O (2 mp )
= (cid:101) O
( (cid:15) − 2 m/T 1 / 3 ) .
The expected number of 4-cycles we store in D is O
( Tp 3 )
= (cid:101) O
( (cid:15) − 6 ) .
We store all the heavy edges, and the number of heavy edges is at most O ( T 1 / 3 ) .
Therefore, the expected total space is (cid:101) O
( T 1 / 3 + (cid:15) − 2 m/T 1 / 3 ) .
E R UNTIME A NALYSIS
In this section, we verify the runtimes of our Algorithms 1, 2, 3, 4, and 5.
Proposition E.1.
Algorithm 1 runs in expected time at most (cid:101) O (cid:0) min( (cid:15) − 2 m 5 / 3 /T 1 / 3 , (cid:15) − 1 m 3 / 2 ) (cid:1) in the setting of Theorem 1.1, or (cid:101) O (cid:0) min( (cid:15) − 2 m 5 / 3 /T 1 / 3 , K · (cid:15) − 1 m 3 / 2 ) (cid:1) in the setting of Theorem 1.4.
Proof.
For each edge ab ∈ S L ∪ S M , we check whether we see both va and vb (lines 9-10) when looking at edges adjacent to v .
So, this takes time | S L | + | S M | per edge in the stream.
We similarly check for each edge in S aux (lines 11-12), so this takes time | S aux | .
Finally, for each edge vu (line 13), we note that lines 14-17 can trivially be implemented in O (1) time, along with 1 call to the oracle.
The remainder of the oracle simply involves looking through the set S aux and S M to search or count for elements.
Thus, the overall runtime is O ( | S L | + | S M | + | S aux | ) per stream element, so the runtime is O ( m · ( | S L | + | S M | + | S aux | )) .
This is at most O ( m · S ) , where S is the space used by the algorithm.
So, the runtime is (cid:101) O (cid:0) min( (cid:15) − 2 m 5 / 3 /T 1 / 3 , (cid:15) − 1 m 3 / 2 (cid:1) in the setting of Theorem 1.1, and is at most (cid:101)
O (cid:0) min( (cid:15) − 2 m 5 / 3 /T 1 / 3 , K · (cid:15) − 1 m 3 / 2 ) (cid:1) in the setting of Theorem 1.4.
Proposition E.2.
Algorithm 2 runs in time (cid:101) O
( (cid:15) − 2 ( α + mβ/T ) m ) .
Proof.
For each edge ab ∈ S i for each 0 ≤ i ≤ c(cid:15) − 2 , we check whether we see both ya and yb in the stream when looking at edges adjacent to y .
So, lines 7-8 take time O ( (cid:80) | S i | ) for each edge we see in the stream.
By storing each S i (and each Q i ) in a balanced binary search tree or a similar data structure, we can search for xy ∈ S i in time O (log n ) in line 11 , and it is clear that lines 12-16 take time O (log n ) (since insertion into the data structure for Q i , S i can take time O (log n ) ).
Since we do this for each i from 0 to c(cid:15) − 2 and for each incident edge yx we see, in total we spend
O ( (cid:15) − 2 log n + (cid:80) | S i | ) time up to line 16.
The remainder of the lines take time O ( (cid:15) − 2 · m log n ) , since the slowest operation is potentially deleting an edge from each S i and a value from each Q i up to m times (for each edge).
Overall, the runtime is (cid:101) O ( (cid:15) − 2 · m + m · (cid:80) | S i | ) = (cid:101) O ( m · S ) , where S is the total space used by the algorithm.
Hence, the runtime is (cid:101) O ( (cid:15) − 2 ( α + mβ/T ) m ) .
Proposition E.3.
Algorithm 3 runs in time (cid:101) O
( K(cid:15) − 2 ( α + mβ/T ) m ) .
Proof.
For each edge ab ∈ S ji for each 1 ≤ i ≤ O (log n ) , 1 ≤ j ≤ O (log log n ) , we check whether we see both ya and yb in the stream when looking at edges adjacent to y .
So, line 5 takes time
O ( (cid:80) | S ji | ) for each edge we see in the stream.
By storing each S ji in a balanced binary search tree or a similar data structure, we can search for xy ∈ S i in time O (log n ) in line 7 , and it is clear that lines 8-12 take time
O (log n ) (since insertion into the data structure for Q ji can take time O (log n ) ).
Since we do this for each i from 0 to c(cid:15) − 2 and for each incident edge yx we see, in total we spend poly log n · O ( (cid:80) | S i | ) time up to line 12.
Finally, lines 13-15 take time poly log n .
Overall, the runtime is (cid:101) O ( m · (cid:80) | S i | ) = (cid:101) O ( m · S ) , where S is the total space used by the algorithm.
Hence, the runtime is (cid:101) O
( K(cid:15) − 2 ( α + mβ/T ) m ) .
Proposition E.4.
Algorithm 4 runs in time (cid:101) O (cid:16) (cid:15) − 1 · m
[Equation]
/ √ T + (cid:15) − 1 · m
[Equation]
/ 2 (cid:17) .
Proof.
For each edge e in the stream, ﬁrst we check the oracle, and we either add e to H or to S L with probability p (lines 6-9), which take O (1) time.
The remaining lines (lines 10-18) involve operations that take O (1) time for each w which represents a pair ( e 1 , e 2 ) of edges where e 1 , e 2 ∈ S L ∪ H and ( e, e 1 , e 2 ) forms a triangle.
So, the runtime is bounded by the amount of time it takes to ﬁnd all e 1 , e 2 ∈ S L ∪ H such that ( e, e 1 , e 2 ) forms a triangle.
If the edge e = ( u, v ) , we just ﬁnd all edges in S L ∪ H adjacent to u , and all edges in S L ∪ H adjacent to v .
Then, we sort these edges based on their other endpoint and match the edges if they form triangles.
So, the runtime is (cid:101) O
( | S L | + | H | ) per edge e in the stream.
Finally, line 19 takes O(1) time.
Overall, the runtime is (cid:101) O ( m · ( | S L | + | H | ))
= (cid:101) O
( m · S ) where S is the total space of the algorithm.
So, the runtime is (cid:101) O (cid:16) (cid:15) − 1 · m 2 / √ T + (cid:15) − 1 · m 3 / 2 (cid:17) .
Proposition E.5.
Algorithm 5 runs in time (cid:101) O (cid:0) (cid:15) − 2 /T 1 / 3 · ( n 3 + m 2 ) (cid:1) .
Proof.
We note that for each edge uv in the stream (line 4), the code in the loop (lines 5-11) can be implemented using 1 oracle call and (cid:101)
O ( | E L | ) time.
The only nontrivial step here is to ﬁnd pairs ( w, z ) such that uw, wz, zv are all in E L .
However, by storing E L in a balanced binary search tree or similar data structure, one can enumerate through each edge wz ∈ E L and determine if uw and zv are in E L in O (log | E L | ) time.
So, lines 4-11 take time (cid:101) O
( | E L | ) per stream element.
Next, lines 12-16 can easily be implemented in time O ( n 2 · | S | ) , lines 17-19 can be easily implemented in time (cid:101)
O
( | D | ) if the vertices in W are stored properly, and lines 20-23 can be done in (cid:101) O
( | E H | · | E L | ) time.
The last part is true since we check each uv ∈ E H and e = ( u (cid:48) , v (cid:48) )
∈ E L , and then check if u, u (cid:48) , v (cid:48) , v form a 4 -cycle by determining if u, u (cid:48) and v, v (cid:48) are in E L ∪ E H (which takes time O (log | E L | + log | E H | ) time assuming E L , E H are stored in search tree or similar data structure).
Overall, we can bound the runtime as (cid:101) O
( m ·| E L | + n 2 ·| S | + | D | + | E H |·| E L | )
= (cid:101) O
( m ·| E L | + n 2 ·| S | ) .
As each edge is in E L with probability at most p and each edge is in S with probability at most p , the total runtime is (cid:101) O (cid:0) ( m 2 + n 3 ) · p (cid:1)
= (cid:101) O (cid:0) (cid:15) − 2 /T 1 / 3 · ( n 3 + m 2 ) (cid:1) .
F A DDITIONAL E XPERIMENTAL R ESULTS
All of our graph experiments were done on a CPU with i5 2.7 GHz dual core and 8 GB RAM or a CPU with i7 1.9 GHz 8 core and 16GB RAM.
The link prediction training was done on a single
GPU.
F. D ATASET DESCRIPTIONS
• Oregon: 9 graphs sampled over 3 months representing a communication network of internet routers Leskovec & Krevl (2014); Leskovec et al.
• CAIDA: 98 graphs sampled approximately weekly over 2 years, representing a communication network of internet routers Leskovec & Krevl (2014); Leskovec et al.
• Reddit Hyperlinks: Network where nodes represent sub communities (subreddits) and edges represent posts that link two different sub communities Leskovec & Krevl (2014); Kumar et al.
• WikiBooks: Network representing Wikipedia users and pages, with editing relationships between them Rossi & Ahmed (2015).
• Twitch:
A user-user social network of gamers who stream in a certain language.
Vertices are the users themselves and the links are mutual friendships between them.
Rozemberczki et al.
(2019) • Wikipedia:
This is a co-occurrence network of words appearing in the ﬁrst million bytes of the Wikipedia dump.
Mahoney (2011) • Synthetic Power law: Power law graph sampled from the Chung-Lu-Vu (CLV) model with the expected degree of the i th vertex proportional to 1 /i 2 Chung et al. (2003).
To create this graph, the vertices are ‘revealed’ in order.
When the j th vertex arrives, the probability of forming an edge between the j th vertex and i th vertex for i <
j is proportional to 1 / ( ij ) 2 .
F. D ETAILS ON LINK PREDICTION ORACLE
Our oracle for the Twitch and Wikipedia graphs is based on Link Prediction, where the task is to estimate the likelihood of the existence of edges or to ﬁnd missing links in the network.
Here we use the method and code proposed in Zhang & Chen (2018).
For each target link, it will extract a local enclosing subgraph around a node pair, and use a Graph Neural Network to learn general graph structure features for link prediction.
For the Twitch network, we use all the training links as the training data for the graph neural network, while for the Wikipedia network, we use 30% of the training links as the training data due to memory limitations.
For the Twitch network, our set of links that we will try to predict are between two nodes that have a common neighbor in the training network, but do not form a link in the training network.
This is about 3.8 million pairs, and we call this the candidate set.
We do this for memory considerations.
For the remaining node pairs, we set the probability they form a link to be 0 .
For the Wikipedia network, we randomly select a link candidate set of size 20 times the number of edges (about 1 million pairs) from the entire set of testing links.
These link candidate sets will be used by the oracle to determine heaviness.
Then, we use this network to do our prediction for all links in our candidate sets for the two networks.
Now we are ready to build our heavy edge oracle.
For the adjacency list model, when we see the edge uv in the stream, we know all the neighbors of u , and hence, we only need to predict the neighbors of v .
Let deg( v ) be the degree of v in the training graph.
The training graph and testing graph are randomly split.
Hence, we can use the training set to provide a good estimation to the degree of v .
Next, we choose the largest deg( v ) edges incident to v from the candidate set, in terms of their predicted likelihood given by the the neural network, as our prediction of N ( v ) , which leads to an estimate of R uv .
For the arbitrary order model, we use the same technique as above to predict
N ( u ) and N ( v ) , and estimate N uv .
F. P ARAMETER S ELECTION FOR A LGORITHM
We need to set two parameters for Algorithm 2: p , which is the edge sampling probability, and θ , which is the heaviness threshold.
In our theoretical analysis, we assume knowledge of a lower bound on T in order to set p and θ , as is standard in the theoretical streaming literature.
However, in practice, such an estimate may not be available; in most cases, the only parameter we are given is a space bound for the number of edges that can be stored.
To remedy this discrepancy, we modify our algorithm in experiments as follows.
First, we assume we only have access to the stream, a space parameter
Z indicating the maximum number of edges that we are allowed to store, and an estimate of m , the number of edges in the stream.
Given Z , we need to designate a portion of it for storing heavy edges, and the rest for storing light edges.
The trade-off is that the more heavy edges we store, the smaller our sampling probability of light edges would be.
We manage this trade off in our implementation by reserving 0 .
3 · Z of the edge ‘slots’ for heavy edges.
The constant 0 . 3 is ﬁxed throughout all our experiments.
We then set p = 0 .
7 · Z/m .
To improve the performance of our algorithm, we optimize for space usage by always insuring that we are storing exactly Z space (after observing the ﬁrst Z edges of the stream).
To do so and still maintain our theoretical guarantees, we perform the following procedure.
Call the ﬁrst Z edges of the stream the early phase and the rest of the edges the late phase.
We always keep edges in the early phase to use our space allocation Z and also keep track of the 0 .
3 -fraction of the heaviest edges.
After the early phase is over, i.e., more than Z edges have passed in the stream, if a new incoming edge is heavier than the lightest of the stored heavy edges, we replace the least heavy stored edge with the new arriving edge and re-sample the replaced edge with probability p .
Otherwise, if the new edge is not heavier than the lightest stored edge, we sample the new incoming edge with probability p .
If we exceed Z , the space threshold, we replace one of the light edges sampled in the early phase.
Then similarly as before, we re-sample this replaced edge with probability p and continue this procedure until one of the early light edges has been evicted.
In the case that there are no longer any of the light edges sampled in the early phase stored, we replace a late light edge and then any arbitrary edge (again performing the re-sampling procedure for the evicted edge).
Note that in our modiﬁcation, we only require our predictor be able to compare the heaviness between two edges, i.e., we do not need the predictor to output an estimate of the number of triangles on an edge.
This potentially allows for more ﬂexibility in the choice of predictors.
If the given space bound meets the space requirements of Theorem C.1, then the theoretical guarantees of this modiﬁcation simply carry over from Theorem 1.2: we always keep the heaviest edges and always sample light edges with probability at least p .
In case the space requirements are not met, the algorithm stores the most heavy edges as to reduce the overall variance.
F. A DDITIONAL F IGURES FROM A RBITRARY O RDER T RIANGLE C OUNTING E XPERIMENTS
Additional ﬁgures for our arbitrary order triangle counting experiments are given in Figures 4 and 5.
[Figure]
F. E XPERIMENTAL D ESIGN FOR A DJACENCY L IST E XPERIMENTS
We now present our adjacencly list experiments.
At a high level overview, similarly to the arbitrary order experiments, for our learning-based algorithm, we reserve the top 10% of the total space for storing the heavy edges.
To do this in practice, we can maintain the heaviest edges currently seen so far and evict the smallest edge in the set when a new heavy edge is predicted by the oracle and we no longer have sufﬁcient space.
We also consider a multi-layer sub-sampling version of the algorithm in Section B.2.
Here we use more information from the oracle by adapting the sub-sampling rates of edges based on their predicted value.
For more details, see Section F.5.
Our results are presented in Figure 6 (with additional plots given in Figure 7).
Our algorithms soundly outperform the MVV baseline for most graph datasets.
We only show the error bars for the multi-layer algorithm and
MVV for clarity.
Additional details follow.
We use the same predictor for N xy in Section 4 as a prediction for R xy .
The experiment is done under a random vertex arrival order.
For the learning-based algorithm, suppose Z is the maximum number of edges that we are allowed to store.
we set the k = Z/ 10 edges with the highest predicted
R uv values to be the heavy edges, and store them during the stream (i.e., we use 10% of the total
[Figure]
space for storing the heavy edges).
For the remaining edges, we use a similar sampling-based method.
Note that it is impossible to know the k -heaviest edges before we see all the edges in the stream.
However, in the implementation we can maintain the k heaviest edges we currently have seen so far, and evict the smallest edge in the set when a new heavy edge is predicted by the oracle.
We also consider the multi-layer sub-sampling algorithm mentioned in Section B.2, which uses more information from the oracle.
We notice that in many graph datasets, most of the edges having very few number of triangles attached to.
Taking an example of the Oregon and CAIDA graph, only about 3%-5% of edges will satisfy R e ≥ 5 under a random vertex arrival order.
Hence, for this edges, intuitively we can estimate them using a slightly smaller space.
For the implementation of this algorithm(we call it multi-layer version), we use 10% of the total space for storing the top k = Z/ 10 edges, and 70% of the space for sub-sampling the edges that the oracle predict value is very tiny(the threshold may be slightly different for different datasets, like for the Oregon and CAIDA graph, we set the threshold to be 5).
Then, we use 20% of the space for sub-sampling the remaining edges, for which we call them the medium edges.
F. F IGURES FROM A DJACENCY L IST T RIANGLE C OUNTING E XPERIMENTS
[Figure]
Additional ﬁgures from the adjacency list triangle counting experiments are shown in Figure 7.
They are qualitatively similar to the results presented in Figure
[Figure]
F. A CCURACY OF THE O RACLE
In this section
[Figure]
we evaluate the accuracy of the predictions the oracle gives in our experiments.
Value Prediction Oracle : We use the prediction of N xy in Section 4 as a value prediction for
R xy ( x ≤ s y ) , under a ﬁxed random vertex arrival order.
The results are shown in Figure 8.
For a ﬁxed approximation factor k , we compute the failure probability δ of the value prediction oracle as follows: δ = # /m , where m is the number of total edges and # equals to the number of edges e that p ( e ) ≥ kαR e + β or p ( e ) ≤ 1 k R e − β , respectively.
Here we set α = 1 , β = 10 for all graph datasets.
We can see for the smaller side, there are very few edges e such that p ( e ) ≤ 1 k R e − β .
This meets the assumption of the exponential decay tail bound of the error.
For the larger side, it also meets the assumption of the linear decay tail bound on most of the graph datasets.
F. D ETAILS ON O RACLE T RAINING O VERHEAD
The overhead of the oracles used in our experiments vary from task to task.
For the important use case illustrated by the Oregon and CAIDA datasets in which we are interested in repeatedly counting triangles over many related streams, we can pay a small upfront cost to create the oracle which can be reused over and over again.
Thus, the time complexity of building the oracle can be amortized over many problem instances, and the space complexity of the oracle is relatively small as we only need to store the top 10% of heavy edges from the ﬁrst graph (a similar strategy is used in prior work on learning-augmented algorithms in Hsu et al.
(2019b)).
To give more details, for this snapshot oracle, we simply calculate the N e values for all edges only in the ﬁrst graph.
We then keep the heaviest edges to form our oracle.
Note that this takes polynomial time to train.
The time complexity of using this oracle in the stream is as follows: when an edge in the stream comes, we simply check if it’s among the predicted heavy edges in our oracle.
This is a simple lookup which can even be done in constant time using hashing.
For learning-based oracles like the linear regression predictor, we similarly need to pay an upfront cost to train the model, but the space required to store the trained model depends on the dimension of the edge features.
For the Reddit dataset with ∼ 300 features, this means that the storage cost for the oracle is a small fraction of the space of the streaming algorithm.
Training of this oracle can be done in polynomial time and can even be computed in a stream via sketching and sampling
[Figure]
techniques which reduce the number of constraints from m (number of edges) to roughly linear in the number of features.
Our expected value oracle exploits the fact that our input graph is sampled from the CLV random graph model.
Given this, we can explicitly calculate the expected value of N e for every edge which requires no training time and nearly constant space.
For training details for our link prediction model, see Section F.2.
Note that in general, there is a wide and rich family of predictors which can be trained using sublinear space or in a stream such as regression Woodruff (2014), classiﬁcation for example using SVMs Andoni et al.
Rai et al.
(2009) and
even deep learning modelsGomes et al.
G I MPLICIT P REDICTOR IN P RIOR W ORKS
We prove that the ﬁrst pass of the two pass triangle counting Algorithm given in Section 3 .
1 of
McGregor et al.
(2016), satisﬁes the conditions of the K -noisy oracle in Deﬁnition 1.1.
Therefore, our work can be seen as a generalization of their approach when handling multiple related data sets, where instead of performing two passes on each data set and using the ﬁrst of which to train the heavy edge oracle, we perform the training once according to the ﬁrst related dataset, and we get a one pass algorithm for all remaining sets.
We ﬁrst recall the ﬁrst pass of (McGregor et al., 2016, Section 3.1):
Sample each node z of the graph with probability p =
C(cid:15) − 2 log m/ρ for some large constant C > 0 .
Let Z be the set of sampled nodes.
Collect all edges incident on the set Z . 3.
For any edge e = { u, v } , let (cid:101) t ( e )
= |
{ z ∈ Z : u, v ∈ N ( z ) }| and deﬁne the oracle as oracle
( e )
=
(cid:26)
LIGHT
if (cid:101) t ( e ) < p · ρ
HEAVY if (cid:101) t ( e ) ≥ p · ρ.
Lemma G.1.
For any edge e = ( u, v ) , oracle ( e ) =
LIGHT implies N e ≤ 2 ρ and oracle ( e ) =
HEAVY implies N e > ρ/ √ 2 with failure probability at most 1 /n 10 .
Proof.
For any edge e , it follows that (cid:101) t ( e ) ∼ Bin ( N e , p ) .
Therefore if N e > 2 ρ ,
Pr[ (cid:101) t
[Equation]
p
[Equation]
[Equation] [Equation]
Ω( p
[Equation]
) ≤ 1 /nby picking C large enough in the deﬁnition of p .
The other case follows similarly.
Lemma G.2.
The expected space used by the above oracle is O ( pm ) .
Proof.
Each vertex is sampled in Z with probability p
and we keep all of its incident edges.
Therefore, the expected number of edges saved is O ( p (cid:80) v d v ) =
O ( pm ) .
Note that the oracle satisﬁes the conditions of the K -noisy oracle in Deﬁnition 1.1.
For example, if
N e ≥ C (cid:48) ρ for C (cid:48)
(cid:29) 1 , Deﬁnition 1.1 only assumes that we incorrectly classify e with probability 1 /C (cid:48) , whereas the oracle presented above incorrectly classiﬁes e with probability exp( − C (cid:48) ) /nwhich is much smaller than the required 1 /C (cid:48) .
H L EARNABILITY R ESULTS
In this section, we give formal learning bounds for efﬁcient predictor learning for edge heaviness.
In particular, we wish to say that a good oracle or predictor for edge heaviness and related graph parameters can be learned efﬁciently using few samples if we observe graph instances drawn from a distribution.
We can view the result of this section, which will be derived via the PAC learning framework, as one formalization of data driven algorithm design.
Our results are quite general but we state simple examples throughout the exposition for clarity.
Our setting is formally the following.
Suppose there is an underlying distribution D which generates graph instances H 1 , H 2 , · · · all on n vertices.
Note that this mirrors some of our experimental setting, in particular our graph datasets which are similar snapshots of a dynamic graph across time.
Our goal is to efﬁciently learn a good predictor f among some family of functions F .
The input of each f is a graph instance H and the output is a feature vector in k dimensions.
The feature vector represents the prediction of the oracle and can encapsulate a variety of different meanings.
One example is when k
= | E | and f outputs an estimate of edge heaviness for all edges.
Another is when k << | E | and f outputs the id’s of the k heaviest edges.
We also think of each input instance H as encoded in a vector in R p for p ≥ (cid:0) n 2 (cid:1) (for example, each instance is represented as an adjacency matrix).
Note that we allow for p > (cid:0) n 2 (cid:1) if for example, each edge or vertex for H ∼ D also has an endowed feature vector.
To select the ‘best’ f , we need to precisely deﬁne the meaning of best.
Note that in many settings, this involves a loss function which captures the quality of a solution.
Indeed, suppose we have a loss function L : f
× H
→
R which represents how well a predictor f performs on some input H .
An example of L could be squared error from the output of f to the true edge heaviness values of edges in H .
Note that such a loss function clearly optimizes for predicting the heavy edges well.
Our goal is to learn the best function f ∈ F which minimizes the following objective:
[Equation]
Let f ∗ be such the optimal f ∈ F , and assume that for each instance H and each f ∈ F , f ( H ) can be computed in time T ( p, k ) .
For example, suppose graphs drawn from D possess edge features in R d , and that our family F is parameterized by a single vector θ ∈ R d and represents linear functions which outputs the dot product of each edge feature with θ .
Then it is clear that T ( p, k ) is a (small) polynomial in the relevant parameters.
Our main result is the following.
Theorem H.1.
There is an algorithm which after poly ( T ( p, k ) , 1 /(cid:15) ) samples, returns a function ˆ f that satisﬁes
[Equation]
with probability at least 9 / 10 .
We remark that one can boost the probability of success to 1 − δ by taking additional log(1 /δ ) multiplicative samples.
The above theorem is a PAC-style bound which shows that only a small number of samples are needed in order to ensure a good probability of learning an approximately-optimal function ˆ f .
The algorithm to compute ˆ f is the following: we simply minimize the empirical loss after an appropriate number of samples are drawn, i.e., we perform empirical risk minimization.
This result is proven by Theorem H.3.
Before introducing it, we need to deﬁne the concept of pseudo-dimension for a function class which is the more familiar VC dimension, generalized to real functions.
Deﬁnition H.2 (Pseudo-Dimension, Deﬁnition
Lucic et al.
Let X be a ground set and F be a set of functions from X to the interval [0 , 1] .
Fix a set S
= { x 1 , · · · , x n } ⊂ X , a set of real numbers R = {
r 1 , · · · , r n } with r i ∈ [0 , 1] and a function f ∈ F .
The set S f = { x i ∈ S | f
( x i ) ≥ r i } is called the induced subset of S formed by f and R .
The set S with associated values R is shattered by F if |{ S f | f ∈ F}| = 2 n .
The pseudo-dimension of F is the cardinality of the largest shattered subset of X (or ∞ ).
The following theorem relates the performance of empirical risk minimization and the number of samples needed, to the notion of pseudo-dimension.
We specialize the theorem statement to our situation at hand.
For notational simplicity, we deﬁne A be the class of functions in f composed with L :
[Equation]
Furthermore, by normalizing, we can assume that the range of L is equal to [0 , 1] .
Theorem H.3 (Anthony & Bartlett (1999)) .
Let D be a distribution over graph instances and A be a class of functions a : H
→
[0 , 1] with pseudo-dimension d A .
Consider t i.i.d.
samples
H 1 , H 2 , . .
, H t from D .
There is a universal constant c 0 , such that for any (cid:15) > 0 , if t ≥ c 0 · d A /(cid:15) 2 , then we have
[Equation]
for all a ∈ A with probability at least 9 / 10 .
The following corollary follows from the triangle inequality.
Corollary H.4.
Consider a set of t independent samples H 1 , . .
, H t from D and let ˆ a be a function in A which minimizes 1 t (cid:80) ti =
a ( H i ) .
If the number of samples t is chosen as in Theorem H.3, then
[Equation]
holds with probability at least 9 / 10 .
The main challenge is to bound the pseudo-dimension of our given function class A .
To do so, we ﬁrst relate the pseudo-dimension to the VC dimension of a related class of threshold functions.
This relationship has been fruitful in obtaining learning bounds in a variety of works such as Lucic et al.
; Izzo et al.
Lemma H.5 (Pseudo-dimension to VC dimension, Lemma 10 in Lucic et al.
For any a ∈ A , let B a be the indicator function of the region on or below the graph of a , i.e., B a ( x, y ) =
sgn ( a ( x ) − y ) .
The pseudo-dimension of A is equivalent to the VC-dimension of the subgraph class
B A =
{ B a | a ∈ A} .
Finally, the following theorem relates the VC dimension of a given function class to its computational complexity, i.e., the complexity of computing a function in the class in terms of the number of operations needed.
Lemma H.6 (Theorem 8 . 14 in Anthony & Bartlett (1999)) .
Let w : R α × R β
→ { 0 , 1 } , determining the class
[Equation]
Suppose that any w can be computed by an algorithm that takes as input the pair ( θ, x ) ∈
R α × R β and returns w ( θ, x ) after no more than r of the following operations:
• arithmetic operations + , − , × , and / on real numbers, • jumps conditioned on >, ≥ , <, ≤ , = , and = comparisons of real numbers, and • output 0 , 1 ,
then the VC dimension of W is O ( α 2 r 2 + r
α log α ) .
Combining the previous results allows us prove Theorem H.1.
At a high level, we are instantiating
Lemma H.6 with the complexity of computing any function in the function class A .
Proof of Theorem H.1.
First by Theorem H.3 and Corollary H.4, it sufﬁces to bound the pseudodimension of the class
A = L ◦ F .
Then from Lemmas H.5, the pseudo-dimension of A is the VC dimension of threshold functions deﬁned by A .
Finally from Lemma H.6, the VC dimension of the appropriate class of threshold functions is polynomial in the complexity of computing a member of the function class.
In other words, Lemma H.6 tells us that the VC dimension of B A as deﬁned in
Lemma H.5 is polynomial in the number of arithmetic operations needed to compute the threshold function associated to some a ∈ A .
By our deﬁnition, this quantity is polynomial in T ( p, k ) .
Hence, the pseudo-dimension of G is also polynomial in T ( p, k ) and the result follows.
Note that we can consider initializing Theorem H.1 with speciﬁc predictions.
If the family of oracles we are interested in is efﬁcient to compute (which is the case of the predictors we employ in our experiments), then Theorem H.1 assures us that only polynomially many samples are required (in terms of the computational complexity of our function class), to be able to learn a nearly optimal oracle.
Furthermore, computing the empirical risk minimizer needed in Theorem H.1 is also efﬁcient for a wide verity of function classes.
For example in practice, we can simply use gradient descent or stochastic gradient descent for a range of predictor models, such as regression or even general neural networks.
We remark that our learnability result is in similar in spirit to one given in the recent learningaugmented paper Dinitz et al.
There, they derive sample complexity learning bounds for the different algorithmic problem of computing matchings in a graph (not in a stream).
Since they specialize their analysis to a speciﬁc function class and loss function, their bounds are tighter compared to the possibly loose polynomial bounds we have stated.
However, our analysis above is more general as it allows for a variety of predictors and loss functions to measure the quality of the predictions.