S P SC
: A F AST AND P ROVABLE A LGORITHM FOR S AMPLING -B ASED GNN T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Neighbor sampling is a commonly used technique for training Graph Neural
Networks (GNNs) on large graphs.
Previous work has shown that samplingbased GNN training can be considered as Stochastic Compositional Optimization (SCO) problems and can be better solved by SCO algorithms.
However, we ﬁnd that SCO algorithms are impractical for training GNNs on large graphs because they need to store the moving averages of the aggregated features of all nodes in the graph.
The moving averages can easily exceed the GPU memory limit and even the CPU memory limit.
In this work, we propose a variant of SCO algorithms with sparse moving averages for GNN training.
By storing the moving averages in the most recent iterations, our algorithm only requires a ﬁxed size buffer, regardless of the graph size.
We show that our algorithm preserves the convergence rate of the original SCO algorithm when the buffer size satisﬁes certain conditions.
Our experiments validate our theoretical results and show that our algorithm outperforms the traditional Adam SGD for GNN training with a small memory overhead.
I NTRODUCTION
Graph Neural Networks (GNNs) have become the state-of-the-art models for machine learning tasks on graph-structured data.
By recursively aggregating the features of neighboring nodes,
GNNs learn an embedding of the nodes and use the embedding for downstream tasks such as node classiﬁcation (Kipf & Welling, 2017; Duran & Niepert, 2017) or link prediction (Zhang & Chen, 2017; 2018).
Due to the recursive neighbor aggregation, training GNNs on large graphs is computationally challenging.
To alleviate the computation burden, various neighbor sampling methods have been proposed (Hamilton et al., 2017;
Ying et al., 2018; Chen et al., 2018; Zou et al., 2019;
Li et al., 2018; Chiang et al., 2019; Zeng et al., 2020).
The idea is to compute an unbiased estimation of the aggregation result in each layer based on a sampled subset of neighbors.
These sampling techniques enable GNN training on large graphs.
However, due to the composition of the aggregation functions in multiple layers, the stochastic gradient obtained with sampled neighbor aggregation is not an unbiased estimation of the true gradient, which undermines the convergence property of SGD-based training algorithms.
Previous work has shown that sampling-based GNN training is actually a Stochastic Compositional
Optimization (SCO) problem (Cong et al., 2020; 2021).
Cong et al.
(2021) show that SCO algorithms can achieve faster convergence than the commonly used Adam SGD for GNN training on small graphs.
Despite their good convergence property, SCO algorithms are not widely adopted for GNN training due to two reasons.
First, although SCO algorithms achieve smaller training losses, the obtained GNN models usually have poor generalization – the validation and test accuracy are lower than the models trained by Adam SGD.
Second, SCO algorithms need to maintain the moving averages of aggregation results of all nodes in the graph.
For large graphs, the moving averages may exceed the memory capacity of the GPU.
While it is possible to store the moving averages in CPU memory, copying the data from CPU to GPU in each iteration is expensive, which may negate the beneﬁts of the faster convergence of SCO algorithms.
To address the above issues, we propose a Sparse Stochastic Compositional (SpSC) gradient method in this work.
Our main idea is to store the moving averages for nodes sampled in the most recent
[Figure]
iterations instead of all nodes.
As only a small number of nodes are stored, our algorithm has small memory consumption even for large graphs.
We provide a convergence analysis on SpSC and show that, when the number of stored iterations satisﬁes certain constraints, SpSC can preserve the asymptotic convergence rate of the original SCO algorithm.
In practice, the sparse moving averaging slightly slows down the convergence of SCO algorithm, but it surprisingly overcomes its poor generalization problem and achieves higher accuracy for sampling-based GNN training.
Compared with Adam SGD, our algorithm incurs a small overhead for updating the moving averages in each iteration, but the overhead can be easily justiﬁed by the faster convergence of our algorithm.
Our experiments with two GNN models on different input graphs validate our theoretical results and show that our algorithm achieves higher accuracy than Adam SGD with the same or less amount of training time.
B ACKGROUND AND M OTIVATION
To facilitate our discussion, we ﬁrst give background on sampling-based GNN training and its relation to stochastic compositional optimization.
2. GNN C OMPUTATION
The computation at each layer of a GNN is conducted in two steps: aggregate and update .
For each node v in the graph, the aggregate function gathers data from its neighboring nodes and returns the aggregation result as
[Equation]
Here, h ne [ v ] is the intermediate features of v ’s neighbors from the previous layer, x v is the input feature of v , and x ne [ v ] is the input features of v ’s neighbors.
The update function uses the aggregated value to produce the intermediate features of v as
[Equation]
By stacking the intermediate features and the input features of all nodes, the computation at layer l can be written as
[Equation]
Here, H ( l − 1) =
[ h ( l − 1), . . .
, h ( l ) N ] denotes the intermediate features of all nodes at layer l − 1 , Z ( l ) ∈
R N × d l is the aggregated features of all nodes at layer l , X =
[ x 1 , . .
, x N ] is the input features of all nodes, and W ( l ) is the learnable weights.
As an example, Graph Convolutional Network (GCN) (Kipf & Welling, 2017) has Agg( H ( l − 1) , X ) =
PH ( l − 1) where P is the normalized Laplacian matrix of the graph, and Upd( Z ( l ) , X, W ( l ) )
=
σ ( Z ( l ) W ( l ) ) where σ is a non-linear activation function.
Many other GNNs can be expressed in this form with different deﬁnitions of Agg and Upd (Zhou et al., 2018).
2. S AMPLING - BASED GNN T RAINING AS S TOCHASTIC C OMPOSITIONAL O PTIMIZATION
When the graph is large, the neigbhbor aggregation operation Agg incurs a large overhead, making the training of GNNs computationally challenging.
Therefore, prior work has proposed to replace the Agg function with a sampled neighbor aggregation operation (cid:103) Agg .
By sampling the neighboring nodes, an unbiased estimate of Z ( l ) is computed at each layer, i.e.,
[Equation]
) with E [ (cid:101) Z ( l ) ]
= Z ( l ) .
If we deﬁne the computation at layer l of the original GNN as a function
[Equation]
the computation with sampled neighbor aggregation can be written as a stochastic function
[Equation]
where ξ l represents the sampled neighbors at layer l .
Since (cid:101) Z ( l ) is an unbiased estimate of Z ( l ) , we have E [ f ( l ) ξ l ] =
f
( l ) , and the computation of a T -layer GNN can be written as
[Equation]
(7) where θ =
[ X, W (1) , ..., W ( T ) ] , f ( T +1) is the loss function, and f ( T +1)
ξ T +1 corresponds to the estimated loss with mini-batch sampling.
Note that we put all the learnable weights in θ to formulate the computation as a stochastic compositional function.
Our goal is to minimize F ( θ ) , which is exactly a multi-level SCO problem.
2. L ARGE M EMORY C ONSUMPTION I SSUE WITH A N AIVE I MPLEMENTATION
SCO has been well studied in the past few years, and many algorithms with guaranteed convergence have been proposed (Zhang & Xiao, 2019;
Yang et al., 2019; Chen et al., 2020; Yang et al., 2019;
Balasubramanian et al., 2020; Chen et al., 2020; Lian et al., 2017; Wang et al., 2017b;
Ghadimi et al., 2020).
It seems straightforward to adopt these SCO algorithms to achieve faster training of
GNNs.
However, these algorithms have large memory consumption when applied to GNN training and cannot run on GPUs for large graphs.
To see the problem, let us consider the implementation of the SCGD algorithm
(Yang et al., 2019) for
GNN training.
Formally, the algorithm is written as
[Equation] [Equation]
The key idea is to store an auxiliary variable y ( i ) to maintain the moving average of each composite function.
Since f ( l ) ξ l returns the exact values of W ( l ) , ..., W ( T ) , we only need to maintain a moving average of (cid:101) Z ( l ) for each layer.
The computation is shown in Figure 1a.
The moving average of the aggregated features is stored in Y ( l ) with each row for one node.
In each iteration, some nodes (rows) are sampled, and the estimated aggregation results (cid:101) Z ( l ) are merged into Y ( l ) based on Formula (8) and (9).
For the nodes that are not sampled, we simply multiply the corresponding rows of ¯ Z ( l ) by (1 − β k ) .
Since the number of rows in Y ( l ) is the number of nodes in the graph, Y ( l ) takes a lot of memory when the graph is large.
For example, for training a 3-layer GCN on a graph with two million nodes, suppose the hidden state dimension d l = 512 and a ﬂoating point has 4 bytes, Y takes 3 × 2 M × 512 × 4 = 12 GB of memory.
All of the existing SCO algorithms need to maintain this moving average, which impedes their application to large-scale GNN training.
S PARSE S TOCHASTIC C OMPOSITIONAL G RADIENT D ESCENT
To reduce the memory consumption of SCO algorithms for GNN training, we propose a Sparse
Stochastic Compositional (SpSC) gradient method.
Instead of storing the moving averages of all nodes in the graph, we only store the moving averages of nodes that are sampled in the most recent iterations.
As shown in Figure 1b, we maintain a ﬁxed size buffer of the moving averages.
The buffer is divided into t chunks with each chunk for the (cid:101) Z ( l ) of one iteration.
The size of each chunk is m l · d l where m l is the maximum number of the nodes that can be sampled at layer l and d l is the hidden state dimension.
Initially, the buffer is empty.
In every iteration, we ﬁrst check if the sampled nodes are in the buffer.
For the nodes that are found in the buffer, we collect the corresponding rows of the buffer and add them to (cid:101) Z ( l ) k based on Formula (8) and (9).
For the nodes that are not found in the buffer, we multiply the corresponding rows of (cid:101) Z ( l ) k by β k .
The updated (cid:101) Z ( l ) k is then written to chunk- ( k mod t ) .
All the other chunks are multiplied by (1 − β k ) .
Since the sampled nodes found in the buffer are updated to chunk- ( k mod t ) , the original values in chunk-0 and chunk-1 are invalidated, as shown by the shadowed rows in Figure 1b.
As the buffer size is a constant ( T · t · m l · d l ) regardless of the graph size, our algorithm can be employed to train GNN on very large graphs.
Our algorithm overwrites chunk-
( k mod t ) in iteration k .
The information of the overwritten nodes is lost.
The update of the moving averages can be written as
[Equation]
where
[Equation]
P ( ξ l,k − t / ( ξ l,k − t +1 ∪ · · · ∪ ξ l,k )) is a matrix with binary values representing the overwritten nodes in the current iteration k , i.e., the nodes that are sampled in iteration k − t and are not sampled in the following t iterations.
Since we only store the moving averages of nodes that are sampled in the most recent t iterations, the information of the overwritten nodes is lost.
These nodes are multiplied by (1 − β j ) in every iteration after iteration k − t . The values of the overwritten rows are [Equation]
[-+
-]=
[Equation] u [Equation] ) k .
Our algorithm simply replaces Formula (8) and (9) in the SCGD algorithm with Formula (11).
To study the convergence property of SpSC, we make the following assumptions that are commonly used in the analysis of SCO algorithms (Yang et al., 2019; Balasubramanian et al., 2020).
Assumption 1.
The composite functions f ( l ) are L l -smooth.
That is, for any y and y (cid:48) , we have
[Equation]
Assumption 2.
The stochastic gradients of the composite functions f ( l ) are bounded in expectation, i.e., E [ (cid:107)∇
[Equation]
(cid:107)
[Equation]
[Equation]
Assumption 3.
The estimated aggregation results obtained by sampled neighbor aggregation is unbiased, i.e., E [ f ( l ) ξ l ( y )]
= f ( l ) ( y ) , and the stochastic gradient of f ( l ) is unbiased, i.e.,
E [ ∇ f ( l ) ξ l ( y )]
= ∇ f ( l ) ( y ) .
Following the single-timescale analysis of the algorithm (Balasubramanian et al., 2020), we use large batches for estimating the composite functions and assume that the estimation variance is small.
Assumption 4.
The estimated aggregation results have small bounded variance, i.e., E [ (cid:107) f
( l ) ξ l ,k ( y ) − f ( l ) ( y ) (cid:107) ] ≤ β k V 2 .
This is a reasonable assumption for GNN training on GPUs as we always sample a batch of nodes for neighbor aggregation to achieve better utilization of the GPU parallelism.
In additional to the conventional assumptions, we make an assumption on the moving averages.
Assumption 5.
The moving average of the aggregated features are bounded, i.e., E [ (cid:107) y
( l ) (cid:107) 2 ] ≤ D 2 .
The convergence rate of our algorithm is summarized in the following theorem.
Theorem 1.
Under Assumptions 1-5, if we set β ≤ 1 − 2 ( − 1 / (2 T −
[Equation]
) and α = β · min [Equation]
, the model parameters { θ k } of our training algorithm with (11) for updating sparse moving averages satisfy
[Equation]
where C max = max ( C 22 C 23 · · · C 2 l , C 23 C 24 · · · C 2 l , · · · , C 2 l , 1) for l = 2 . . T , t is the number of buffer chunks used in our algorithm.
The last term on the RHS of (13) reveals how the convergence of the SCGD algorithm is affected by the sparse moving averages.
The larger t we use (i.e., the more chunks we have in the buffer), the smaller (1 − β ) 2( t − 1) and (1 − β ) 2 t {++
+} we have, and the faster convergence we achieve.
In theory, if we can make (1 − β ) 2( t − 1) =
O ( β 4 ) , the algorithm will achieve O ( (cid:112) 1 /K ) convergence rate.
As β → 0 , we need larger t to maintain the convergence rate, and eventually, we will need to store the moving average of all nodes in the graph.
Applying Sparse
Moving Averages to SCSC.
Our sparse moving average can also be applied to other SCO algorithms.
For example, the Stochastically Corrected Stochastic Compositional gradient method (SCSC) (Chen et al., 2020) has a correction term in the update of the moving averages.
Because of the correction term, SCSC needs a relaxed assumption on the estimation error of the composite functions.
More speciﬁcally, if we change (11) to[-y ( l ) k +
 = (1 − β [Equation]-]
[Equation]
and replace (8) and (9) with (14), we can relax Assumption 4 as
Assumption 6.
The estimated aggregation results have bounded variance, i.e., E [ (cid:107) f
( l ) ξ l ,k ( y ) − f ( l ) ( y ) (cid:107) ] ≤ V 2 .
The convergence rate of SCSC with sparse moving averages is summarized as follows.
Theorem 2.
Under Assumptions 1-3 and 5-6, if we choose α k
= α =
c α √ K and β k = β
[-= α (cid:80) N − 1 l =
-]
= c β √ K , the model parameters { θ k } of SCSC with (14) for updating sparse moving averages satisfy
[Equation]
where C u = max (cid:0) 4 C 2 l + γ l (cid:1) for l = 1 . . .
T .
The result suggests that, if we can set t such that (1 − β ) 2( t − 1)
=
O ( β 3 ) , the algorithm will achieve
O ( (cid:112) 1 /K ) convergence rate.
I MPLEMENTATION D ETAILS
Algorithm 1 describes an implementation of Formula (11) in our algorithm.
For each layer, we allocate a buffer ( buf ) of size tm l × d l for the moving averages.
The buffered nodes are maintained in a list node list ∈ R tm l where node list
[ i ] is the index of the node stored at buf [ i ] .
If buf [ i ] is empty, node list [ i ] is set to -1.
In every iteration k , we ﬁrst get the location of chunk-( k mod t ).
Then, we look up each of the sampled nodes in the buffer (line 3).
The LookUp function computes the intersection of S and node list and returns the indices of the overlapping nodes in the two arrays.
If a sampled node is not found in the buffer, we multiply the corresponding row of (cid:101) Z ( l ) k by β k .
If a sampled node is in the buffer, we read in its current moving average and update the corresponding row of (cid:101) Z ( l ) k
(line 4).
For buffered nodes that are not sampled, we simply multiply their moving averages by ( 1 − β k ) (line 5).
For buffered nodes that are sampled, we invalidate their original buffer by setting node list [ idx in buf ] to − 1 (line 6).
Last, we add the sampled nodes to the node list .
Most of the operations in Algorithm 1 are simple vector operations, and they incur little overhead.
The performance bottleneck is the LookUp function.
With a naive implementation, it has O ( tm l log | S | ) time complexity, assuming S is sorted.
In our implementation, we use an auxiliary array node loc ∈ R N to store the locations of all nodes in the buffer and accelerate the LookUp function.
Speciﬁcally, if node- i is in the buffer, we store its location in buffer in node loc [ i ] ; otherwise, node loc [ i ] is set to -1.
With the auxiliary array, the idx in z can be obtained by comparing node loc [ S ] with zero, and the idx in buf is simply node loc
[ S ][ idx in z ] .
Before updating the node list at line 7 of Algorithm 1, we remove the overwritten nodes from node loc by setting node loc [ node list [ start : end ]] to -1.
Finally, we store the locations of the newly sampled nodes to node loc by setting node loc [ S ] to [ start, start + 1 , . .
. , end ] .
It is easy to see that all these operations have O ( | S | ) time complexity.
E VALUATION 5. E XPERIMENTAL S ETUP
We conduct our experiments on a workstation with an Nvidia RTX 3090 GPU, an Intel Xeon Gold 6226R CPU, and 512GB RAM.
Our code is implemented with PyTorch 1.8.0 and PyTorch Geometric 1.7.0.
We evaluate our algorithm on ﬁve graphs as listed in Table 1.
The reddit and yelp graph are adopted from GraphSAINT (Zeng et al., 2020), and the arxiv , proteins , products are from the Open Graph Benchmark (Hu et al., 2020).
We apply our algorithm to two GNN models: GCN (Kipf & Welling, 2017) and GraphSAGE (Hamilton et al., 2017).
Both models have three convolutional layers.
We use Formula (11) for updating the moving average instead of Formula (14).
This is because Formula (14) requires two forward passes which incurs extra overheads.
The algorithm is run for 50 epochs.
We set β to 0.2 initially, and decrease it to 0.1 at epoch 20, and further decrease it to 0.05 at epoch 40.
The number of buffered chunks ( t ) is set to 8.
We adopt the layer-wise sampling method in Zou et al.
for neighbor sampling.
The batch size is set to 4096, and the number of sampled neighbors in each layer is set to 8192.
5. T RAINING R ESULTS
Validation Accuracy.
Figure 2a shows the validation accuracy of different algorithms for training a GCN on arxiv .
We can see that full neighbor aggregation (Adam Full) achieves the highest
[Figure]
[Table]
[Figure]
[Table]
[Figure]
accuracy.
This is reasonable because full neighbor aggregation returns unbiased estimates of gradients.
If neighbor sampling is used, the accuracy clearly drops with the same training algorithm
(Adam Sample).
Our algorithm (Sparse SCO) is able to improve the convergence of sampling-based
GNN training and achieves almost the same accuracy as full neighbor aggregation.
The results on reddit and yelp are similar.
On products graph, Adam Full runs out of memory, so we only show the results of sampling-based training in Figure 2d.
Interestingly, we ﬁnd that our algorithm with neighbor sampling achieves even higher accuracy than Adam Full on proteins graph, as shown in Figure 2c.
The original SCO algorithm (which stores the moving averages for all nodes in the graph) also has lower accuracy than Sparse SCO.
This is probably due to the overﬁtting of models by Adam Full and SCO.
Training Loss.
Figure 3 shows the training loss of different algorithms on different graphs.
For reddit and yelp , we are able to run full neighbor aggregation on our GPU.
As expected,
Adam Full achieves the smallest training loss.
Adam Sample, however, has the slowest convergence.
SCO achieves training loss close to Adam Full.
We run our algorithm with different t ’s. The larger t we use, the smaller training loss we obtain.
The results are consistent with our theoretical analysis and also suggesting that the poor accuracy of the original SCO algorithm is probably due to overﬁtting.
For products graph, we are not able to run full neighbor aggregation.
The results show a clearly faster convergence of our algorithm than Adam SGD for sampling-based training.
Test Accuracy.
Table 2 lists the test accuracy of the models trained by different algorithms.
For reddit and yelp , we follow the GraphSAINT paper (Zeng et al., 2020) and report the F1-micro score.
For proteins , we follow the OGB (Hu et al., 2020) and report the ROC-AUC.
We can see that our training algorithm achieves the highest test accuracy for both GCN and GraphSAGE on almost all the graphs.
We do not include the results for GCN on yelp graph because its accuracy is apparently lower than GraphSAGE with all training algorithms, probably due to the limited expressiveness of the GCN model.
While it is hard to draw a direct comparison with GraphSAINT because the sampling methods and the model architectures are different, our test accuracy on reddit and yelp matches the best accuracy reported by GraphSAINT (Zeng et al., 2020).
It is worth noting that our algorithm achieves higher accuracy than Adam SGD with both full neighbor aggregation and neighbor sampling on proteins and products graph.
The numbers are higher than the accuracy
[Figure] [Figure]
arxiv proteins
[Table]
[Figure]
[Table]
reddit
[Table]
arxiv proteins products
[Table]
of the same models reported in OGB Leaderboards (ogb, 2021).
The results suggest that there is an improvement space for the accuracy of GNN models by using better training algorithms.
Execution Time.
Table 3 lists the time per epoch of different training algorithms on different graphs.
Although full neighbor aggregation achieves good convergence in many cases, it incurs a much large computation overhead than sampled neighbor aggregation.
The execution time of Adam Full is 7x to 55x longer than that of Adam Sample.
Compared with Adam Sample, the SCO algorithm incurs a small overhead for updating the moving average of aggregation results.
Our algorithm runs slightly slower than SCO because the LookUp operation in Algorithm 1 incurs an extra overhead.
However, the execution time is still much smaller than full neighbor aggregation.
Memory Consumption.
Figure 4 shows the memory consumption of different algorithms.
We collect the numbers by calling the max memory allocated function in PyTorch at the end of the ﬁrst epoch.
For arxiv , reddit , proteins , and yelp , full neighbor aggregation takes much larger memory space than sampled aggregation.
SCO and Sparse SCO require additional memory for storing the moving averages.
Because we only store the moving average of nodes sampled in recent iterations, Sparse SCO uses less memory than SCO.
On products graph, SCO requires extremely large memory due to the massive number of nodes, while our Sparse SCO can run with only 2GB of GPU memory.
R ELATED W ORK
To overcome the scalability limitation of GNN training, various neighbor sampling methods have been proposed, including node-wise sampling (Hamilton et al., 2017;
Ying et al., 2018), layer-wise sampling (Chen et al., 2018; Huang et al., 2018; Zou et al., 2019), and subgraph sampling (Zeng et al., 2020;
Chiang et al., 2019).
These sampling-based training methods achieve good accuracy in practice (particularly on small graphs), but they lack theoretical justiﬁcation. [Table]
[Figure] Some recent works (Cong et al., 2020; 2021) point out that sampling-based GNN training is actually multi-level Stochastic Compositional Optimization (SCO).
However, they either use this connection to justify their sampling techniques and fall back to Adam SGD for training (Cong et al., 2020), or they directly adopt an SCO algorithm without considering the large memory consumption issue (Cong et al., 2021).
The research on SCO traces back to (Ermoliev, 1976) where a two-timescale stochastic approximation scheme was proposed for two-level problems.
Various SCO algorithms and convergence analyses have been proposed ever since (Wang et al., 2017a;b; Lian et al., 2017; Yang et al., 2019; Zhang &
Xiao, 2019; Chen et al., 2020; Balasubramanian et al., 2020; Ghadimi et al., 2020; Hu et al., 2019;
Lin et al., 2018).
Despite a substantial volume of work on SCO in recent years, none of the existing work has considered the large memory consumption issue and the data movement overhead of the algorithms.
Our work is the ﬁrst to establish a convergence analysis for SCO algorithms with sparse moving averages.
C ONCLUSION
In this work, we propose a new variant of SCO algorithm for training graph neural networks on large graphs.
Our main idea is to maintain a sparse moving average of the aggregation results in each convolutional layer.
We study the convergence property of our algorithm and show that the algorithm can achieve O ( (cid:112) 1 /K ) convergence rate when a sufﬁcient amount of moving averages are maintained.
Our experiments with two GNN models on different graphs validate our theoretical results and show a clear advantage of our algorithm against Adam SGD for GNN training.
APPENDIX
Link to our source code: https://anonymous.4open.science/r/iclr2022_ artifact
0FE
/ 8. P ROOF TO T HEOREM
Based on the discussion in Section 3, our proposed algorithm can be written as
[Equation]
[Equation] [Equation]
(18) with
[Equation] [Equation]
[Equation]
(19) and
[Equation]
8.1. S UPPORTING LEMMA
Lemma 3.
The change of y ( l ) k in every iteration is bounded, i.e.,
[Equation] [Equation]
[Equation] [Equation]
k
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
( l = 1) (21) and
[Equation]
[Equation] [Equation]
Proof.
According to (17)
[Equation]
we have
[Equation]
Taking the square of both sides and using Assumption
[Equation]
we have
[Equation]
Lemma 4.
The difference between the stochastic gradient and the true gradient is bounded, i.e.,
[Equation]
where A l := (cid:80)
Tm =
l +1 A l,m .
This lemma is commonly used in the analysis of SCO algorithms Chen et al.
; Yang et al.
Proof can be found in Chen et al.
(2020), page 20-23.
Lemma 5.
The difference between the composite function and its moving average satisﬁes
[Equation]
[Equation]
where C max = max ( C 22 C 23 · · · C 2 l , C 23 C 24 · · · C 2 l , · · · , C 2 l , 1) for l = 2 . . .
T .
Proof.
According to (17)
[Equation]
we have
[Equation]
Taking the square of both sides of (27) and taking the expectation conditioned on F k,n := (cid:110)
F k , y (1) k +1 , · · · , y ( l − 1) k +1 (cid:111) , we have:
[Equation]
[Equation]
Let us deﬁne
[Equation]
Formula (28) can be rewritten as
[Equation] [Equation]
Because
[Equation]
when l = 1 ,
[Equation]
[Equation]
plugging (32) into (30), we have
[Equation]
When k = 0 , || u 10 || 2 = 0 , and
S
[Equation]
[Equation]
[Equation]
Similarly
[Equation]
we have
[Equation]
Here, for i > t ,
[Equation]
for 0 < i ≤ t ,
[Equation]
Because S (1) = 0 , we can recursively write:
[Equation]
where w i,k = (1 − β k ) · · · (1 − β
i +1 )
β i . For l >
[Equation]
(30) can be rewritten as
[Equation]
(Lemma 3) ≤
[Equation] where
[Equation] It follows that
[Equation] Following the same steps as (38)
[Equation]
by recursively plugging S (2) k into S (2) k
+1 , we can obtain
[Equation]
[Equation]
According to (38), it is easy to ﬁnd (cid:80) kj
= i
w j,k < 1 .
Thus, we have
[Equation]
(cid:88)
[Equation]
Combining (43) and (42), we have
[Equation]
[-1) i (46) We let M k +
 = (cid:80) T +1-]Similarly, for any l > 1 , we have
[Equation]
where
[Equation]
Let C max = max ( C 22 C 23 · · · C 2 l
[Equation]
[Equation]
[Equation]
l
[Equation]
[Equation]
2 l , 1) , ∀ l ∈ [1 , T ] and Z maxi deﬁned as follows:
[Equation]
Plugging (46) into (45), we can obtain
[Equation]
Summing (48) from l = 1 to T , setting β k = β and α k
= α , then substituting ∆ (1) k
we have:
K
[Equation]
1 (cid:88)
[Equation] It follows that
[Equation]
If we set β ≤ 1 − 2 T [Equation] 1 (cid:113) 12 and α ≤
[Equation]
, we can ensure β[-) 2 − 2 QTC 
-]
[Equation] 1 − β )
[Equation] − 2 C 21 C max α 2 T 2 ≥ β 2 / 2 , and (50) can be simpliﬁed as
[Equation]
8.1. R EMAINING STEPS TOWARDS T HEOREM 1.
Using the smoothness of F ( θ k )
[Equation]
we have
[Equation]
[Equation]
Conditioning on F k and taking expectation of both sides w.r
[Equation]
t. ξ k , we have
[Equation]
F k (cid:105) + α k (cid:104)∇ F ( θ k ) , E [ ∇ F (
[Table]
k ) − ∇ k |F k ] (cid:105)
[Table]
[Table]
E (cid:104)
[Table]
F
[Table]
) (cid:107) 2 (cid:105) + L 2 C 21 · · · C 2 T α 2 k +
[Table]
E [ (cid:107)∇ F (
[Table]
) (cid:107) ] (cid:107) E [ ∇ k |F
[Table]
[Table]
∇ F
[Table]
[Table]
[Table]
[Table]
[Table]
E (cid:20)(cid:13)(cid:13)(cid:13)
y [Table] l ) k +1 − f
[Table]
l ) ( y ( l − 1) k +1 )
{+(cid:13)(cid:13)(cid:13) 
+}
|F k (cid:21) + L 2 C 21 · · · C 2 T α 2 k .
Summing from k = 0 to K − 1 , we have
[Equation]
Substitute (50) into the above equation, we have
[Equation]
If we set α ≤ 2 β (cid:80) Tl =1 A 2 l +16 C max C 21 T , we can ensure 1 − α 4 β (cid:80) Tl
[Equation]
1 A 2 l −
[Equation]
Tα β ≥ 12 , and thus, K − 1 (cid:88) [Equation] Combining with the condition for α and β in (50)
[Equation]
if we set
[Equation] we have [Equation]
This completes the proof of Theorem 1.If we can set t such that (1 − β ) 2( t − 1) =
O ( β 4 ) , the algorithm achieves O (1 / √ K ) convergence rate.
8. P ROOF TO T HEOREM
The SCSC algorithm with our sparse moving average can be written as
[Equation] [Equation]
61) with [Equation] P
[Equation]
62) and
[Equation]
8.2. S UPPORTING LEMMA
Lemma 6.
The change of y ( l ) k in every iteration is bounded, i.e.,
[Equation]
Proof.
According to (60)
[Equation]
we have
[Equation]
Taking the square of both sides of (65) gives us (64) [Equation]
Lemma 7.
The difference between the composite function and its moving average is bounded, i
[Equation]
e.,
[Equation]
Proof.
[Equation]
Conditioned on F k , taking expectation over ξ l,k , we have:
[Equation]
Taking the square of both sides of (67) and taking the expectation conditioned on F k
[Equation]
n := (cid:110)
F k [Equation] y (1) k +1 , · · · , y ( l − 1) k +1 (cid:111)
[Equation]
we have:
[Equation]
[Table]
[Equation] [Equation]
8.2. R EMAINING STEPS TOWARDS T HEOREM 2.
Using the smoothness of F ( θ k )
[Equation]
we have
[Equation]
Conditioning on F k and taking expectation of both sides w.r.t.
ξ k , we have
[Equation]
[Table]
[Equation]
The subsequent analysis builds on the following Lyapunov function:
[Equation]
where θ ∗ is the optimal solution of the problem.
Conditioning on F k and taking expectation of V k +1 w.r.t.
ξ k , we have
[Table]
y ( l ) k − f ( l ) ( y ( l − 1) k ) (cid:13)(cid:13)(cid:13) [-
-]2 |F k (cid:21) + 4 (1 + β k ) (1 − β k ) 2 C 21 E (cid:104) (cid:107) θ k − θ k − 1 (cid:107) [-
-]2 |F k (cid:105)
+
T (cid:88) l = (cid:2) 4 (1 + β k ) (1 − β k ) 2 C 2 l + γ l (cid:3) E (cid:104)(cid:13)(cid:13)(cid:13) y
( l − 1)
k +1 − y ( l − 1) k
(cid:13)(cid:13)(cid:13)
|F k (cid:105) + 2(1 + β k ) β 2 k NV 2 + 1 + β k β k k − 1 (cid:89)
j = k − t (1 − β j )
T (cid:88) l =
E (cid:104) (cid:107) u ( l ) k (cid:107) 2 (cid:105) −
T (cid:88) l = γ l E (cid:104)(cid:13)(cid:13)(cid:13)
y ( l − 1) k
+1 − y ( l − 1) k
(cid:13)(cid:13)(cid:13)
|F k (cid:105) − β k
T (cid:88) l =
E (cid:20)(cid:13)(cid:13)(cid:13)
y ( l ) k +1 − f ( l ) ( y ( l − 1) k +1 )
(cid:13)(cid:13)(cid:13) 2 (cid:21) ≤V k − α k (cid:32) 1 − α k 2 β k
T (cid:88) l =
A 2 l (cid:33) (cid:107)∇ F ( θ k ) (cid:107) 2 + L 2 C 21 · · · C 2 T +1 α 2 k + 2(1 + β k ) β 2 k NV+ 4 C 21 E (cid:104) (cid:107) θ k − θ k − 1 (cid:107) 2 |F k
(cid:105)
+
T (cid:88) l
= (cid:0) 4 C 2 l + γ l (cid:1) E (cid:20)(cid:13)(cid:13)(cid:13)
y ( l − 1) k
+1 − y ( l − 1) k
(cid:13)(cid:13)(cid:13) 2 |F k (cid:21) −
T (cid:88) l = γ l E (cid:104)(cid:13)(cid:13)(cid:13)
y ( l − 1) k
+1 − y ( l − 1) k
(cid:13)(cid:13)(cid:13)
|F k (cid:105) − β k
T (cid:88) l =
E (cid:20)(cid:13)(cid:13)(cid:13)
y ( l ) k +1 − f ( l ) ( y ( l − 1) k +1 )
(cid:13)(cid:13)(cid:13) 2 (cid:21) + 1 + β k β k k − 1 (cid:89)
j = k − t (1 − β j )
T (cid:88) l =
E (cid:104) (cid:107) u ( l ) k
(cid:107) 2 (cid:105)
[Table]
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
If we deﬁne γ l such that:
[Equation]
[Equation]
and
[Equation]
(75) then
[Equation] [Equation]
It follows that
[Equation]
[-(cid:107) 
-]
Setting β k
= α k (cid:80) T − 1 l =1 A 2 l , we have
[Equation]
Setting α k
= α =
c α √ K and β k = β
= c β √ K and summing up both sides of (79) from k =
to K − 1 , we have
[Equation]
where C u = max (cid:0) 4 C 2 l + γ l (cid:1) for l = 1 . . .
T .
This completes the proof of Theorem 2.
If we can set t such that (1 − β ) 2( t − 1) =
O ( β 3 ) , the algorithm achieves O (1 / √ K ) convergence rate.