GraB: Finding Provably Better Data Permutations than Random Reshuffling
Abstract
Random reshuffling, which randomly permutes the dataset each epoch, is widely adopted in model training because it yields faster convergence than withreplacement sampling.
Recent studies indicate greedily chosen data orderings can further speed up convergence empirically, at the cost of using more computation and memory.
However, greedy ordering lacks theoretical justification and has limited utility due to its non-trivial memory and computation overhead.
In this paper, we first formulate an example-ordering framework named herding and answer affirmatively that SGD with herding converges at the rate O ( T − 2 / 3 ) on smooth, non-convex objectives, faster than the O ( n 1 / 3 T − 2 / 3 ) obtained by random reshuffling, where n denotes the number of data points and T denotes the total number of iterations.
To reduce the memory overhead, we leverage discrepancy minimization theory to propose an online Gradient Balancing algorithm (GraB) that enjoys the same rate as herding, while reducing the memory usage from O ( nd ) to just O ( d ) and computation from O ( n 2 ) to O ( n ) , where d denotes the model dimension.
We show empirically 1 on applications including MNIST, CIFAR10,
WikiText and GLUE that GraB can outperform random reshuffling in terms of both training and validation performance, and even outperform state-of-the-art greedy ordering while reducing memory usage over 100 × .
Introduction
Many machine learning problems can be formulated as minimizing a differentiable (loss) function f : R d → R for a set of data examples { x i } ni =1 .
To train a parameterized model, the goal is to obtain a set of target parameters w ∗ =
arg min f ( w ) , where f ( w ) =
1 n (cid:80) ni
=1 f ( w ; x i ) , and f ( w ; x i ) denotes the loss incurred on the i -th data example x i (usually a mini-batch of images, sentences, etc.) with model parameters w .
A typical model training, or optimization process, is to iteratively update the model parameter w starting from some initial w (1) by running
[Equation]
where α denotes the step size, and σ : { 1 , . .
. , n } → { 1 , . .
. , n } denotes a permutation (ordering) from which the examples are chosen to compute the stochastic gradients 2 .
A widely adopted ordering protocol is Random Reshuffling (RR), where the optimizer scans in an order drawn at random without replacement over the entire training dataset multiple times during training (a common name for a single such scan is an “epoch”).
RR allows the optimizer to converge faster empirically and enjoys a better convergence rate in theory [1].
Despite RR’s theoretical improvement, it has been proven that
RR does not always guarantee a good ordering [2, 3]; in fact a random permutation is far from being optimal even when optimizing a simple quadratic objective [4].
In light of this, a natural research question is:
Can we find provably better orderings than Random Reshuffling—and do so efficiently?
Recent studies indicate the possibility of greedily constructing better permutations using stale estimates of each ∇ f ( w ; x j ) from the previous epoch [5, 6].
Concretely, Lu et al.
[5] proved that for any model parameters w ∈ R d , if sums of consecutive stochastic gradients converge faster to the full gradient, then the optimizer will converge faster.
Formally, in one epoch, any permutation σ minimizing the term (named average gradient error )
[Equation]
(2) leads to fast convergence.
Leveraging this insight, Lu et al.
[5] proposes to greedily select σ ( t ) one at a time for all t ∈ { 1 , . .
. , n } at the beginning of the ( k + 1) -th
epoch, using the stochastic gradients computed during the k -th epoch as an estimate.
This strategy works well empirically, but it remains an open question how its convergence rate compares to that of RR.
Moreover, the greedy ordering method intensively consumes O ( nd ) memory to store the gradients and O ( n 2 ) computation to order, which significantly hinders its usefulness in practice.
For instance, training a simple logistic regression on MNIST would easily cost more than 1 GB additional memory compared to RR.
In this paper, we address the limitations of previous order-selection approaches, culminating in proposing a new algorithm, GraB, which converges faster than RR both in theory and in practice without any blow-up in memory or computational time.
First, we address the theoretical gap by connecting the convergence of permuted-order SGD to the classic herding problem [7]: we show how any algorithm for solving the herding problem can be run on stale stochastic gradients to select an example ordering.
While the greedy selection method of previous work may fail to select a good ordering, using the best herding algorithms from the literature yields a better convergence rate: O ( T − 2 / 3 ) compared to RR
’s O ( n 1 / 3 T − 2 / 3 ) on smooth non-convex problems and ˜ O
( T − 2 ) compared to RR’s ˜ O
( nT − 2 ) under PL condition, where T is the total number of iterations.
Building on this, we leverage discrepancy minimization theory [8] to propose an online Gradient Balancing algorithm (GraB) that achieves the same O ( T − 2 / 3 ) rate but only requires O ( d ) memory and O ( n ) computation.
Perhaps surprisingly, on multiple deep learning applications we observe GraB not only allows fast minimization of the empirical risk, but also lets the model generalize better.
Our contributions in this paper can be summarized as follows:
• We formulate a general framework of sorting stochastic gradients with herding, and prove SGD with herding converges at a faster rate than Random Reshuffling (Section 3).
• We propose an online Gradient Balancing (GraB) algorithm that enjoys the same fast convergence rate as herding SGD while significantly reducing the memory from O ( nd ) to O ( d ) and computation from O ( n 2 ) to O ( n ) (Section 5).
• We conduct extensive experiments on MNIST, CIFAR10, WikiText and GLUE with different models.
We demonstrate that GraB converges faster in terms of both training and validation compared to Random Reshuffling and other baselines (Section 6).
Related Work
While traditional data ordering research mostly focuses on with-replacement samplers [9–11], withoutreplacement sampling is more common in practice [12].
Random Reshuffling (RR) [13] and the related shuffle-once (SO) method [14, 15] are among the most popular data permutation methods.
Recht and Ré [16] undertakes the first theoretical investigation of RR, while subsequent works like [2, 17] give counter examples where RR orders badly.
HaoChen and Sra [18], Gürbüzbalaban et al.
[19], and Mishchenko et al.
[1] discuss extensively on the conditions needed for RR to benefit.
Some recent works [5, 6] suggest constructing better data permutations than RR via a memory-intensive greedy strategy.
Concretely, Mohtashami et al.
[6] proposes evaluating gradients on all the examples first to minimize Equation (2) before starting an epoch, applied to Federated Learning;
Lu et al.
[5] provides an alternative of minimizing Equation (2) using stale gradients from previous epoch to estimate the gradient on each example.
Rajput et al.
[4] introduces an interesting variant to RR by reversing the ordering every other epoch, achieving better rates for quadratics.
Other approaches, such as curriculum learning [20], try to order the data to mimic human learning and improve generalization [21–23]: these approaches differ from our goal of finding good permutations for minimizing loss in a finite-sum setting.
Offline Stale-Gradient Herding
In this section, we study the use of stale gradients —the stochastic gradients for each example from the previous epoch, saved in memory—to construct a data permutation at the start of each epoch.
Lu et al.
[5] and Mohtashami et al.
[6] propose to use greedy ordering in this way—selecting examples one at a time to minimize Equation (2).
Here, we first show that this objective is closely related to the classic herding problem from discrepancy theory.
We demonstrate with examples from the herding formulation that in the adversarial setting, greedy selection is bad at herding and can underperform basic random reshuffling.
We conclude this section by proving SGD with proper herding converges faster than random reshuffling.
Herding.
The herding problem originates from Welling [24] for sampling from a Markov random field that agrees with a given data set.
Its discrete version is later formulated in Harvey and Samadi [7].
Given n vectors { z i } ni =1 that are d -dimensional (i.e. in R d ) and have norm ∥ z i ∥ 2 ≤ 1 , the herding problem we study 3 is the task of finding a permutation σ ∗ : [ n ] → [ n ] that minimizes
[Equation]
State-of-the-art algorithms guarantee an ˜ O (1) bound to this objective [7].
It is straightforward to observe the the formulation of Equation (3) generalizes Equation (2) if we replace each z i with the stochastic gradient computed on the example i .
In other words, any algorithm that minimizes
Equation (3) can be used to minimize Equation (2) and find a better data permutation.
The simplest way to do this is with the greedy algorithm for herding, which we show in Algorithm 1: this is
[Table]
essentially what the previous work was doing to select an order based on stale gradients.
Unfortunately, despite its elegance, it was first pointed out in Chelidze et al.
[25] that the greedy ordering can underperform random permutation.
adapt their result to show the following.
Statement 1.
There exist n vectors in R 2 such that when applying Algorithm 1, the objective in
Equation (2) becomes Ω( n ) ; a random permutation is guaranteed to achieve O ( √ n ) on average.
Herding SGD with Stale Gradients.
Based on the herding framework, a path to SGD with better data permutation becomes clear: when training models during a epoch, we can simply store all the stochastic gradients, and then herd them (with some herding algorithm) offline at the start of the next epoch to obtain the order for that epoch.
This stale-gradient approach is formally described in
Algorithm 2, and could be contrasted with a fresh-gradient approach (as in Mohtashami et al.
[6]) that herded with newly computed stochastic gradients at the start of each epoch—it is desirable to avoid the use of fresh gradients if possible as they double the gradient computations needed each epoch.
The greedy ordering of Lu et al.
[5] can be described as running Algorithm 2 using Algorithmas the Herding subroutine.
However, as greedy herding with Algorithm 1 does not have a good worst-case guarantee, we propose using some other herding algorithm instead.
There have been many algorithms proposed that can reduce the Equation (3) to ˜ O
(1) in polynomial time.
Here, we show that this herding-objective bound is sufficient to prove convergence of herded SGD and even yield a better convergence rate than Random Reshuffling.
We defer the details of the herding algorithm to
Section 5, as they do not affect the convergence rate of SGD.
We start by stating some assumptions for non-convex optimization, as well as our herding bound.
Assumption 1.
( Smoothness .)
For any example x j in the dataset (for j ∈ { 1 , . . . , n } ), the loss for x j is L 2 , ∞ -smooth and L ∞ -smooth
meaning that, for any w , v ∈ R d , it holds that
[Equation]
and
[Equation]
The averaged loss function is also L -smooth in that, for any w , v ∈ R d , it holds that
[Equation]
Assumption 2.
( PL condition .)
We say the loss function f fulfills the Polyak-Lojasiewicz (PL) condition if there exists µ > 0 such that for any w ∈ R d ,
[Equation]
where f ∗ = inf v ∈ R d f ( v ) .
Assumption 3.
( Bounded Gradient Error .)
For any j ∈ { 1 , · · · , n } and any w ∈ R d ,
[Equation]
Assumption 4.
( Herding Bound .)
In Algorithm 2, the subroutine Herding algorithm has the following property [26]: if given input vectors z 1 , . . .
, z n ∈ R d that satisfy ∥
z i ∥ 2 ≤ 1 and (cid:80) ni
=
z i = 0 , Herding outputs a permutation σ of { 1 , . . . , n } such that for all k ∈ { 1 , . . . , n } , ∥ (cid:80) k
i =
1 z σ ( i ) ∥ ∞ ≤ H for some constant H .
Remarks on the assumptions.
Assumptions 1, 2, and 3 are commonly used assumptions in the study of permutation-based SGD, except that we separate different smoothness constants with respect to different norms to tighten the final bound.
The cross-norm Lipschitz constant L 2 , ∞ is often used to obtain tight dimension-independent bounds [27, 28].
Note that these cross-norm assumptions can
[Figure]
[Equation]
be dropped by incurring extra factors of the dimension d , as L ∞ ≤ L 2 , ∞ ≤ √ dL for any function.
Assumption 4 holds for multiple algorithms for some H =
˜ O (1) : we will discuss this in depth in
Section 5, while treating H as a parameter here as we want to observe the dependence on this variable in the final bound.
Under these assumptions, the convergence rate of Algorithm 2 is given in the following theorem.
Theorem 1.
In Algorithm 2, under Assumptions 1, 3 and 4, if we set α to be
[Equation]
then it converges at the rate
[Equation]
where F = f ( w 1 ) − f ∗ .
Furthermore, under the additional PL condition (Assumption 2), with α as α = min
[Equation]
where W 0 denotes the Lambert W function, Algorithm 2 converges at the rate
[Equation]
Improved Rate Compared to RR.
Theorem 1 gives us a better rate than random reshuffling, in that there is no dependence on the number of examples n in the dominant term.
If we let T = nK be the total number of steps, then Theorem 1 shows Algorithm 2 converges at rate O ( 1 T 2 / 3 + nT ) .
In contrast, random reshuffling gives a rate of O ( n 1 / 3 T 2 / 3 + nT ) as given in Mishchenko et al.
[1], Theorem 4.
Similarly, herding improves the rate from ˜ O
( nT 2 ) to ˜ O
( 1 T 2 ) in the PL case.
This gives us an affirmative answer to our question from the intro: a faster rate can be obtained even if we herd with stale gradients estimates, without computing additional gradients.
Enabling Memory-Efficient Herding with Balancing
In Section 3 we introduce the herding problem, and prove that with proper herding, SGD converges at a faster rate than Random Reshuffling.
However, a downside of offline herding SGD is that it requires storing all the vectors (i.e., in the case of SGD, all the stochastic gradients) which incurs
O ( nd ) memory overhead.
In this section, we present background on the vector balancing problem , the solutions of which will enable us to solve the herding problem in an online fashion.
Background: Online Vector Balancing.
The problem of online vector balancing was first formulated in Spencer [29]: given n vectors { z i } ni =1 ∈ R d , arriving one at a time, the goal is to assign a sign ϵ i ∈ {− 1 , +1 } to each vector upon receiving it so as to minimize max k ∈ [ n ] ∥ (cid:80) ki
=1 ϵ
i
z i ∥ ∞ .
Over the last few decades, many efforts have been spent on bounding this ℓ ∞ -norm objective.
Specifically,
Aru et al.
[30] and Bansal and Spencer [31] shows it can be made to be on the order of ˜ O
(1) when the coordinates of each vector are independent .
Bansal et al.
[32] gives a poly (log( n ) , d ) bound when there are dependencies among coordinates.
The adversarial setting with interval discrepancy is discussed in [33].
Other works focus on balancing variants such as stochastic arrival [34] or box discrepancy [35].
Solving Herding via Balancing.
While the objectives of balancing and herding are different, an algorithm for balancing can be used as a subroutine to do herding as well.
Consider the simple case of vectors z 1 , . .
, z n in some order, where (cid:80) ni
=
z i = 0 , and suppose we have obtained a group of signs { ϵ i } ni =1 via some balancing algorithm: can we use this to determine an order of { z i } ni =1 that can solve the herding problem?
Harvey and Samadi [7], Chobanyan et al.
[36], and Chobanyan [37] approach this problem by a clever reordering that works as in Algorithm 3.
Concretely, a new order is obtained by first taking all the vectors with positive signs in order, followed by all the vectors with negative signs in reversed order from the original sequence.
This process is visualized in Figure 1.
Harvey and Samadi [7] shows that this balancing-and-then-reordering has the following effect on the herding objective in Equation (3).
Theorem 2.
([7], Theorem 10) Consider n vectors z 1 , . .
z n ∈ R d that fulfill (cid:80) ni =
z i =and ∥
z i ∥ ≤ 1 .
Suppose that for some permutation σ , the herding objective is bounded by max k ∥ (cid:80) ki =1 z σ ( i ) ∥ ∞ ≤ H for some constant H .
Suppose we have computed some signs ϵ 1 , . . .
, ϵ n such that the vector-balancing objective max k ∥ (cid:80) ki =1 ϵ i
z
σ ( i ) ∥ ∞ ≤ A for some constant A .
Then if we let σ ′ be the permutation output by Algorithm 3, the herding objective on this new permutation is bounded by max k ∥ (cid:80) ki =1 z σ ′ ( i ) ∥ ∞ ≤ ( A + H ) / 2 .
Theorem 2 shows that calling Algorithm 3 allows us to construct a new ordering from balancing, which reduces the current herding bound by a constant factor if A ≤ H .
This means that if we have a balancing algorithm that is guaranteed to reduce the balancing objective to less than A , by applying this “reordering” many times, we can also reduce the herding objective to be less than A : this approach is used in many offline herding algorithms that achieve ˜ O (1) bounds.
GraB: SGD with Gradient Balancing for Good Data Permutation
Building on the balance-then-reorder paradigm from the previous section, in this section we propose an algorithm named GraB that improves SGD with herding by choosing the ordering online —without storing gradients—thus using only O ( d ) memory and O ( n ) computation.
However, the conditions of applying the balancing-reordering framework bring up additional challenges in this online setting:
• Challenge I. To apply Theorem 2, we will need to pre-center all the vectors (as in line 2 of Algorithm 1) to ensure they sum to zero.
This cannot be done online, as we need to make decisions on-the-fly before seeing all the vectors.
• Challenge II.
The reordering strategy, as shown in Theorem 2, only reduces the herding bound by a constant factor, which does not give us the accelerated rate directly since applying Theorem
would require Equation (3) to be on the order of ˜ O
We can repeatedly call the balancingreordering subroutines, but that would again require storing all the stochastic gradients.
To overcome challenge I, we apply a two-step stale gradient estimate: for any k , we use the running average of stale gradients to “center” stochastic gradient in epoch k + 1 (this centering is itself stale, and does not guarantee the vectors sum to 0 , but this is still enough to prove convergence).
Then, the online balancing-reordering subroutine will determine the proper order to use in epoch k + 2 .
To address challenge II, we leverage a noisy reordering process: let σ k denote the order to use in epoch k , then the next order σ k +1 will be obtained from σ k .
The intuition here is that if we train the model with enough epochs, the reordering will in general push the herding bound towards A in Theoremwithout storing any additional vectors.
We present the full GraB algorithm in Algorithm 4.
We proceed to provide a convergence guarantee for Algorithm 4.
Since now we are working with balancing rather than herding, in the convergence analysis we replace Assumption 4 with the following one related to balancing.
Assumption 5.
( Balancing Bound. )
There exists a constant A > 0 such that in Algorithm 4, if the subroutine Balancing is given input vectors z 1 , . . .
, z n ∈ R d that satisfy ∥
z i ∥ 2 ≤ 1 , then
Balancing outputs signs ϵ i ∈ {− 1 , +1 } such that ∥ (cid:80) ki =1 ϵ i
z σ ( i ) ∥ ∞ ≤ A for all k ∈ { 1 , . . . , n } .
We defer discussing the complexity of A to the later part of this section, and first present the convergence guarantee for GraB (Algorithm 4) as follows.
Theorem 3.
In Algorithm 4, under Assumptions 1, 3 and 5, if we set α to be
[Equation]
then it converges at the rate
[Equation]
where F = f ( w 1 ) − f ∗ .
Furthermore, under the additional PL condition (Assumption 2) with α as α = min
[Equation]
where W 0 denotes the Lambert W function, it converges at the rate
[Equation]
Comparing Theorem 1 and 3, we can observe GraB
(Algorithm 4) achieves essentially the same fast asymptotic convergence rate as SGD with herding (Algorithm 2).
Asymptotic complexity for A and H .
So far we have covered the main design of GraB. A remaining yet unanswered question is how do A in Assumption 4 and H in Assumption 5 relate to other factors exactly, and what are the concrete algorithms to achieve small A and H .
Some recent progress in the theory community shows that there exists an online algorithm (Algorithm 6) that guarantees with high probability that A is on the order of ˜ O
(1) as shown in the following theorem.
Theorem 4.
(Alweiss et al.
In Algorithm 6, if all the vectors have ∥ z t ∥ ≤ 1 and the parameter c is set c = 30 log( nd/δ ) , then with probability 1 − δ , the maximum partial signed sum is bounded by max t ∥ (cid:80) tj =1 ϵ
[Equation]
This theorem gives us a high-probability bound on A that can easily be converted to a high-probability statement that the result of Theorem 3 holds with A replaced by ˜ O (1) .
Theorem 4 also yields a good offline herding algorithm and a bound on H : recall from Theorem 2 that given any herding bound
H on an ordering, running Algorithm 3 once allows us to obtain a new herding bound ( A + H ) / 2 .
Given Theorem 4 that a ˜ O (1) bound is guaranteed on A , then if we repeatedly run Algorithmas described in Section 4, we can eventually push the herding bound H towards A , which yields
H =
˜ O (1) .
This can be done with probability 1 offline by simply restarting on failure; however, as stated in Section 3, this still requires storing all the vectors.
On the normalization of vectors.
Note that a typical balancing subroutine (e.g. Algorithm 6) requires normalized vectors ∥
z i ∥ ≤ 1 .
∀ i .
This, in practice, implies we will need to estimate a large enough constant to normalize gradients when we run GraB. To alleviate this issue, we can instead adopt a balancing subroutine that is invariant to the normalizer, for instance, Algorithm 5.
As shown in the next section, this naive balancing algorithm suffices for non-trivial improvements.
Experiment
In this section, we evaluate GraB and other baseline algorithms on multiple machine learning applications.
We compare GraB
(Algorithm 4) with the following baselines: Random Reshuffling (RR), Shuffle Once (SO) [1], FlipFlop [4] and Greedy Ordering 4 [5].
All the experiments run on an instance configured with a 4-core Intel(R)
Xeon(R)
2.50GHz CPU, 32GB memory and an NVIDIA
GeForce RTX 2080 Ti GPU.
Although we used Algorithm 6 to show a ˜ O (1) bound can be obtained for herding, in our experiments, we adopt the simple balancing algorithm of Algorithm 5, which performs well in practice.
More results with other balancing algorithms can be found in the Appendix.
[Figure]
Model and Dataset.
We adopt the following model training tasks for evaluation: (1) training logistic regression on MNIST, (2) training LeNet on CIFAR10 [39], (3) training LSTM on wikitext-2 [40], and (4) finetuning BERT-Tiny
[41] on GLUE [42].
Detailed information regarding models, datasets and hyperparameters can be found in Appendix A.
Convergence Speedup.
The convergence plots for each algorithms are summarized in Figure 2.
We observe that in general, GraB is able to converge faster in terms of both training and validation loss on various tasks.
Of all the baseline algorithms, Greedy Ordering is able to achieve comparable convergence speed with respect to epochs to GraB. However, Greedy Ordering consumes much more memory and wall-clock time as shown in the figure.
In the BERT training case, we observe the
Greedy Ordering runs Out Of Memory (OOM) , causing an error on our machine.
Based on simple calculations, GraB only requires less than 1% the memory used by Greedy Ordering for all the tasks.
It is worth pointing out that throughout the experiments, we do not tune hyperparameters for GraB but let it reuse the hyperparameters from RR (details in the Appendix).
This implies GraB can provide in-place improvement without any additional tuning in practice, and a well-tuned GraB could benefit with even larger margin.
Shuffle Once (SO) performs worst in all the cases, and FlipFlop is generally the same as RR, which is aligned with the conclusion made in Rajput et al.
Ablation Study: are good permutations fixed?
In GraB, the example ordering σ is carried and improved over epochs, which is motivated from minimizing the herding bound.
This raises another question of whether the found permutation changes at later stage of training, and if so, can we find a fixed permutation that outperforms the RR in practice?
Here we conduct an ablation study on two fixed order strategies with GraB.
We include the following two variants:
[Figure]
• 1-step GraB : it only runs GraB for 1 epoch to obtain an ordering, and then use it as a fixed order for the rest of the training.
• Retrain from GraB :
First launch a full run of GraB, and then use the ordering in the final epoch as a fixed order in a new run.
We test these new variants on two tasks: LeNet on CIFAR10 (non-convex) and logistic regression on
MNIST (convex).
The training loss and validation accuracy curves can be found in Figure 3.
We observe that on both tasks, the 1-step GraB does not work well, which is aligned with our theory and motivation in Section 5 Challenge II.
On the other hand, Retrain from GraB achieves the comparable performance than original GraB on the convex task, but not on the non-convex one.
This is mainly because a good ordering is going to depend on the local optimum the algorithm is approaching and for a convex problem, there is only one such optimum.
It again verifies that GraB finds better orderings compared to baseline Random Reshuffling and Shuffle Once.
On the granularity of example ordering.
Throughout this paper, we have discussed how to find good data permutations via per-example gradients.
In practice, however, per-example gradients are usually not easy to obtain since data is usually loaded in batches, and many ML libraries (e.g.
PyTorch) directly accumulate gradients over the batch.
A direct workaround is to fix the data within batches and reorder the batches (treating them as coarse-grained examples).
This, however, would compromise the benefits of ordering since the total number of examples n is reduced by a factor of batch size, while the statistical improvement of herding (GraB) is in the order of O ( n − 1 / 3 ) .
To alleviate this, we provide two alternatives: (1) Use ML frameworks that support quick per-example gradients computation (e.g. JAX 5 ); (2) Leverage gradient accumulation steps as used in large language model training, i.e., use smaller batch sizes in the code but perform optimizer.step once every few steps so that we can obtain the finer-grained gradients on-the-fly while still optimizing the models with the desired batch size.
This is the default method we used throughout the experiments, we will include more details in the appendix.
Conclusion
In this paper, we formulate a herding framework for data-ordering in SGD.
We prove SGD with herding finds better data orderings, and converges faster than random reshuffling on smooth nonconvex and PL objectives.
We propose an online gradient balancing algorithm named GraB that finds a better ordering with little compute or memory overhead.
We substantiate our theory and the usefulness of GraB on multiple machine learning applications.
Acknowledgement
This work is supported by NSF-2046760.
Yucheng is supported by Meta PhD Fellowship.
The authors would like to thank Si Yi Meng and anonymous reviewers from NeurIPS 2022 for providing valuable feedbacks on this paper.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The details are given in text.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Experimental Details
Setup.
We first provide the details to models and datasets used in the experiments:
Logistic Regression on MNIST :
The MNIST consists of handwritten digits 0-9, it has a training set of 60 , 000 examples, and a test set of 10 , 000 examples.
It contains 28 × 28 = 784 features and 10 classes.
The model is of dimension d = 784 × 10 + 10 = 7850 (with bias).
LeNet on CIFAR10.
The CIFAR10 dataset consists of 60000 32 × 32 colour images in 10 classes, with 6000 images per class.
There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images.
The test batch contains exactly 1000 randomly-selected images from each class.
LeNet is a classic convoluntional neural network proposed by [43].
LSTM on WikiText-2.
In this task we train a 2-layer LSTM on the wikitext-2 dataset, which contains 2 million words.
We set the embedding size to be 32, number of hidden unit to be 32 and number of head to be 2.
We adopt the learning rate schedule from PyTorch example repo 6 .
We set the sequence length (bptt) to be 35. 4.
BERT on GLUE.
The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems.
GLUE consists of 11 different tasks.
In the main paper, we evaluate on the SST-2 and QNLI.
For this task, we adopt the BERT-Tiny model released by Google Research 7 .
BERT-Tiny contains 768 hidden layers.
For each task, we set maximum sequence length to be 32 and enable padding.
Hyperparameters.
We tune the hyperparameters for all the algorithms except GraB within a given range.
Then we reuse the hyperparameters for RR in GraB. This implies GraB can potentially provide in-place benefit without additional tuning.
We use momentum SGD (with its default value 0.9) for all the tasks.
The hyperparameters (ranges) for each task are as follows:
MNIST.
LR ∈ { 0 . 1 , 0 . 01 , 0 .
; BSZ = 64 ; GCC = 32 ; WD = 0 . 0001 .
CIFAR10.
LR ∈ { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } ; BSZ = 16 ; GCC = 2 ; WD = 0 . 0001 .
WikiText-2.
We set momentum to be 0.9 and let the learning rate follow ReduceLROnPlateau from Pytorch with variable: { mode =’min’, factor =0.1, patience =5, threshold =5}.
The initial learning rate is set to be 5. 4.
GLUE.
On the SST-2 we adopt BSZ = 1 ; GCC = 1 ; WD = 0 .
01 ; LR ∈ { 0 . 005 , 0 .
On the QNLI task we adopt BSZ = 2 ; GCC = 2 ; WD = 0 .
01 ; LR ∈ { 0 . 005 , 0 .
LR stands for learning rate, BSZ stands for batch size, GCC stands for the gradient accumulation steps and WD stands for the weight decay.
Gradient Accumulation.
As illustrated in the paper, one workaround to obtain fine-grained gradients (subgradients over one example, or a group of examples of size smaller than batch size) is to leverage the gradient accumulation step, especially in frameworks that do not support per-example gradient computation like PyTorch.
epoch in range(num_epochs):
batch in grab_ordered_batches:
related
grad_buffer.add_(grad / grad_accumulation_step)
if epoch_step % grad_accumulation_step == 0:
optimizer.apply_gradient(grad_buffer)
grad_buffer.zero_(
Listing 1: A simple workaround to obtain fine-grained gradients in some ML frameworks such as
PyTorch.
The effect of different balancing algorithms.
We extend the experiment from Figure 1 and run
Algorithm 5 and Algorithm 6 for different epochs.
Figure 4 summarizes the results: although the
[Figure]
two algorithms differ when single time sorting is used, but they obtain similar results when called repeatedly (10 times).
Note that the running of Algorithm 6 requires tuning a hyperparameter c and generate a series of random numbers.
To avoid these overhead, in practice we recommend using Algorithm 5.
Another interesting observation is that in terms of ℓ -2 norm, the naive balancing (Algorithm 5) outperforms Algorithm 6 in high-dimensional setting (b)(c) in Figure 4 when epoch
= 1 .
B Theoretical Analysis
In this section, we provide the detailed proof for all the theorems in the main paper.
Throughout the proof, for simplicity, we denote w k +1 = w (1) k
+1 =
w ( n +1) k for all the k ≥ 1 .
Additionally, we define the maximum backward deviation within an epoch to be
[Equation]
[Equation] [Equation]
K.
For any k , we let σ − 1 k
( i ) =
{ t = 1 , · · · , n | σ k ( t ) = i } to be the step when i -th example is visited.
Unless otherwise specified, we use ∥·∥ to denote the ℓ 2 -norm.
B.1 Details to the greedy statement
Proof.
[25] We first construct a group of 2-d vectors.
Without the loss of generality, we let n divides 2 .
And let n/ 2 vectors be [1 , 1] ⊤ and the other n/ 2 vectors be [4 , − 2] ⊤ .
Algorithm 1 will always select [1 , 1] ⊤ in the first n/ 2 steps based on the current sum being [ m, m ] ⊤ , ∀ m ≤ n/ 2 .
We show this with induction.
Note that when t = 1 , [1 , 1] ⊤ is selected.
Then suppose in the k -th selection the current sum is [ k, k ] ⊤ , the algorithm will again select [1 , 1]
⊤ since 2( k + 1) 2 <
( k + 4)
+ ( k − 2) 2 .
This makes the herding objective Ω( n ) with Algorithm 1.
On the other hand, consider using a random permutation.
Let random variable X t denote the value of herding objective at t -th selection.
Then we know that X t is a 2-d martingale.
So that from
Azuma-Hoeffding Inequality we know ∥
X k ∥ ≤ O ( √ n ) for all k .
B.2 Proof to Theorem
Theorem 1.
In Algorithm 2, under Assumptions 1, 3 and 4, if we set α to be
[Equation]
then it converges at the rate
[Equation]
where F = f ( w 1 ) − f ∗ .
Furthermore, under the additional PL condition (Assumption 2), with α as α = min (cid:40)
[Equation]
where W 0 denotes the Lambert W function, Algorithm 2 converges at the rate
[Equation]
Proof.
From Lemma 1, we get
[Equation]
On the other hand
[Equation]
from Lemma
[Equation]
we obtain
[Equation]
Combining them together gives us,
[Equation]
Note that for any x ∈ R d , ∥ x ∥ ∞ ≤ ∥ x ∥ 2 .
And so the last term can by bounded by its ℓ 2 -norm.
Moving it to the left side of the inequality gives us
[Equation]
Finally
[Equation]
we choose α so that
[Equation]
Denoting F = f ( w 1 ) − f ∗
[Equation]
we finally obtain This gives
[Equation]
This gives us the bound for general smooth-convex case.
We proceed to prove the PL case:
From Lemma 3 we have the following relation:
[Equation]
Square on both sides
[Equation]
w
[Equation]
∥ 2 .
Summing from k = 1 to K − 1 ,
K −
[Equation]
Recall from Lemma 3 that
[Equation]
this gives us,
[Equation] [Equation]
Using the fact that
[Equation]
Solve it
[Equation]
we obtain
[Equation]
First from Lemma 2 we get
[Equation]
where in the last step
[Equation]
we apply the learning rate bound that
[Equation]
The RHS of the inequality can be further bounded by
[Equation]
[Equation]
Take derivative with respect to α and set it to zero, we obtain
[Equation]
and
[Equation]
where W 0 ( · ) denotes the Lambert W function.
That completes the proof.
B.3 Proof to Theorem
Theorem 3.
In Algorithm 4, under Assumptions 1, 3 and 5, if we set α to be
[Equation]
then it converges at the rate
[Equation]
where F = f ( w 1 ) − f ∗ .
Furthermore, under the additional PL condition (Assumption 2) with α as
[Equation]
= min
[Equation]
where W 0 denotes the Lambert W function, it converges at the rate
[Equation]
Proof.
From Lemma
[Equation]
we get
[Equation]
On the other hand
[Equation]
from Lemma
[Equation]
we obtain
K (cid:88)
[Equation]
Combining them together gives us,
[Equation]
k
= ∥∇ f ( w k ) ∥ 2 .
Given
[Equation]
we get
[Equation]
This gives us the bound for general smooth-convex case.
We proceed to prove the PL case: From
Lemma 7 we know
K −
[-+ 
-]
[Equation]
[-+ 64 α 
 [( n + A ) L 2 , ∞ + 5 nL ∞ ] [Equation] n-]On the other hand
[Equation]
from Lemma 2 we get
[Equation]
where in the last step
[Equation]
we apply the learning rate bound that
[Equation]
The RHS of the inequality can be further bounded by
[Equation]
Take derivative with respect to α and set it to zero [Equation] we obtain
[Equation]
and
[Equation]
where W 0 ( · ) denotes the Lambert W function.
That completes the proof
[Equation]
B.4 Technical Lemmas
Lemma 1.
In Algorithm 2 and Algorithm [Equation] if αnL < 1 holds and Assumption 1 (except PL condition), 3 and 4 hold, then
[Equation]
Proof.
Note that in both algorithms, the update can be written as
[Equation]
By the Taylor Theorem, for all the k = 1 , · · · , K − 1 ,
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] In the second step, we apply −⟨ a , b ⟩ = − 12 ∥ a ∥ 2 − {+
+} ∥ b
∥
+ 12 ∥ a − b ∥ 2 , ∀ a , b .
In the third step, we use the condition that αnL < 1 .
Expanding the last term using Assumption 1, we get
[Equation]
In the second step we apply the Jensen Inequality.
Put it back, we obtain
[Equation]
Finally, summing from k = 1 to K − 1 , we obtain
[Equation]
That completes the proof
[Equation]
Lemma 2.
In Algorithm 2 and Algorithm 4, if αnL < 1 holds and Assumption 1 (including PL condition), 3 and 4 hold, then
[Equation]
Proof.
Since the this lemma is a special case of Lemma 1, we can just borrow the derivation there and get for all the k = 1 , · · · , K − [Equation]
[Equation]
Define 1 − αnµ/ 2 = ρ , then
[Equation]
Recursively apply it from k = 1 to K − 1 , we obtain
[Equation]
That completes the proof
[Equation]
Lemma 3.
In Algorithm [Equation] if the learning rate α fulfills
[Equation]
then the following inequalities hold:
[Equation]
and,
[Equation]
and finally,
[Equation]
Proof.
Without the loss of generality [Equation] for all the m ∈ { 2 , · · · , n + 1 } and all the k ∈ { 2 , · · · , K } ,
[Equation]
k − α m − 1 (cid:88) t = [Equation] f (cid:18) w ( σ [Equation]
[Equation]
k ( t )))
k
[Equation]
x σ k ( t ) (cid:19) −
[Equation]
m − 1 (cid:88) t = (cid:18) [Equation] f (cid:16) w ( t ) k
[Equation]
x σ k ( t ) (cid:17)
− ∇ f (cid:18) w ( σ − 1 k − 1 ( σ k ( t )))
k −; x σ k ( t ) (cid:19)(cid:19) .
Now add and subtract
[Equation]
which gives
[Equation]
We further add and subtract
[Equation]
to arrive at
[Equation]
We can now re-arrange, take norms on both sides and apply the triangle inequality,
[Equation]
There are four different terms on the right hand side, we will apply the Assumption 4 on the first term, and Assumption 1 on the last two terms.
First, for the first term,
[Equation]
[Table]
[Equation]
This implies if we denote
[Equation]
We can now use Assumption 4 to obtain a bound on the prefix sum
[Equation]
(cid:13) ∞ ≤ H, that is,
[Equation]
Now we have a bound for the first term in Equation (4), we proceed to bound the last two terms where we apply Assumption 1.
We can then rewrite Equation (4) into,
[Equation]
Furthermore, applying the triangle inequality to the norms in the last two terms
[Equation]
we obtain
[Equation]
and similarly,
[Equation]
This gives us
[Equation]
Note that Equation (5) only holds with k ∈ { 2 , · · · , K } and m ∈ { 2 , · · · , n + 1 } .
We now discuss the boundary cases.
Note that the bound of Equation (5) trivially holds with m = 1 for any k since the left hand side becomes zero.
On the other hand, when k = 1 , we have, w
[Equation]
take norms and apply the triangle inequality
[Equation]
we obtain
[Equation]
Now that we have the bounds for ∆ k , we next will sum them up.
Taking a max over m on both side in Equation (5), this implies for all the k ≥ 2 ,
[Equation]
as m − 1 ≤ n .
Considering the fact that αL ∞ n < 1 / 2 , we get
[Equation]
This completes the proof of the first inequality in the lemma.
Applying this recursively from any k ≥ 2 to 2 , this gives
[Equation]
Applying the learning rate conditions that 32 αnL ∞ ≤ 1 and 16 αHL 2 , ∞ ≤
[Equation]
we obtain
[Equation]
Square on both sides,
[Equation]
We can apply the similar trick to Equation (6) and get
[Equation]
This completes the proof of the second inequality in the lemma.
Summing from k = 1 to K , we will get
[Equation]
That completes the third inequality, and we have finished proving all three inequalities.
Lemma 4.
In Algorithm 4, if the learning rate α fulfills
[Equation]
then the following inequalities hold:
[Equation]
Proof.
With out the loss of generality, for all the m ∈ { 2 , · · · , n + 1 } and all the k ∈ { 3 , · · · , K } ,
[Equation]
Now add and subtract α
[Equation]
which gives
[Equation]
We further add and subtract
[Equation]
to arrive at
[Equation]
We can now re-arrange, take norms on both sides and apply the triangle inequality,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] Similar to the proof of Lemma [Equation] for the last two terms
[Equation]
we simply apply the Assumption 1 to obtain
[Equation]
[Equation]
[Table]
n [Equation] [Equation] [Table]
[Equation]
and
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] For brevity
[Equation]
we use r k [Equation] k ≥ 3 to denote [Equation]
Then the expression of ∆ k can be written as
[Equation]
Now square on both sides, it gives us
[Equation]
Now summing k from k = 3 to K on both sides of the inequality
[Equation]
we get
[-−+ 
-][Equation]
Now we apply Lemma 6 to replace the first term, K (cid:88)
[Equation]
where in the last step [Equation] we apply the learning rate requirement that
[Equation]
Now solve the LHS,
[Equation]
We have now proved k ≥ 3 .
We next discuss the k = 1 , 2 case, note that these cases follow exactly
Equation (6), so that we can reapply the k = 1 case from Lemma 3 and obtain
[Equation]
Now we can sum over all the epochs as follows
[Equation]
That completes the proof
[Equation]
Lemma 5.
Consider a group of time-variant input vectors z 1 ,k , . .
z n,k ∈ R d changing over time k = 3 , · · · , K that satisfy the following conditions:
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Now considering a sequence of ordering of { σ k } Kk =3 that fulfills condition: for any k ≥ 3 , σ k and σ k +1 are the input and output of Algorithm 3, then it holds that ,
[Equation]
Proof.
For convenience, we use the notation ˜ r k to represent
[Equation]
∞ .
For any k ≥ 3 , by the Triangle Inequality we have [Equation]
[Table]
[Equation]
We now derive the relation between σ k +1 and σ k .
Suppose we run Algorithm 3 and obtain the signs for all the vectors.
Denote ϵ k,j as the sign assigned to z j,k .
Let all the vector indices with positive signs form a set M + and indices with negative signs form a set M − .
Then we know that for any 1 ≤ t ≤ n ,
[Equation]
By using the triangle inequality, for any k
[Equation]
[Equation]
Given these bounds, now we can consider the permutation of σ k +1 , suppose when t = t 0 , the partial sum of σ k +1 reaches it maximum, then if t 0 ≤ | M + | , from the previous bound, we obtain
[Equation]
On the other hand, if the t 0 > | M + | , and we obtain
[Equation]
n (cid:88)
j = t {+
+}
+ z σ k +1 ( j )
[Equation]
k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞ ≤
[Equation]
+ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88) j ∈ M − ∩ [ k ] z σ k ( j )
[Equation]
k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞
[Equation]
And so that,
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
an
[Equation]
Square on both sides, we get ˜
[Equation]
Ac
[Equation]
Then we sum from k = 3 to K − 1 , we get ˜ r
+
K − 1 (cid:88) k
=
[Equation]
˜ r 2 k +1 ≤
[Equation]
r
[Equation]
[Equation]
K − 1 (cid:88) k
= ˜
r 2 k
[Equation]
[Equation]
which implies
[Equation]
moving the terms
and we finally get
[Equation]
Replace the ˜ r k , we finally get
[Equation]
That completes the proof.
Lemma 6.
In Algorithm 4, for k ≥ 3 , with notation of
[Equation]
we have
[Equation]
Proof.
We will apply Lemma 5 to the main derivation.
Denote
[Equation]
and so
[Equation]
We first need to derive a k , b k , c k in Lemma 5 in order to apply it, we will repeatedly use Assumption 1.
In the derivation, we will also use the relation that ∥ a ∥ ∞ ≤ ∥ a ∥ 2 .
First, we derive a k ,
[Equation]
[Table]
We proceed to derive b k ,
[Equation]
n (cid:88)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
k −
[Equation]
And finally for c k ,
[Equation]
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
L
[Equation]
[Equation]
2∆ k − 2 ) . Now we have the a k , b k and c k ready, we can now apply Lemma 5, and get K (cid:88) k
= r 2 k ≤ 2 r 23 + K (cid:88) k = [ Aς
[Equation]
AL
[Equation]
AL [Equation] ∆ k − Finally, we look at r 3 , note that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Square on both sides, we can get
[Equation]
Push it back we can finally obtain the bound as K (cid:88)
[Equation]
∆ 22 + 24 n 2 L 22 , ∞ ∆+ 6 A 2 ς 2 ( K − 2) + 24 n 2 L 2 ∞
K (cid:88) k = ∆ 2 {+k + 
+}
A 2 L 22 , ∞
K (cid:88) k
= ∆ 2 k − 1 + 384 n 2 L 2 ∞
K (cid:88) k
{+= ∆ 2 k −+ 
+}
A 2 L 22 , ∞ K (cid:88) k = ∆ 2 k − 2 + 384 n 2 L 2 ∞ K (cid:88) k
{+= ∆ 2 k −≤ 6 n 2 ς 2 + 
+}
A 2 ς 2 ( K − 2) + (792 n 2 L 2 ∞ + 78 A 2 L 22 , ∞ ) K (cid:88) k
{+= ∆ 2 k + (24 n 2 L 2 , ∞ + 
+}
A 2 L 22 , ∞ + 384 n 2 L 2 ∞ )∆+ (6 n 2 L 22 , ∞ + 78 A 2 L 22 , ∞ + 768 n 2 L 2 ∞ )∆ 22 . That completes the proof.
Lemma 7.
In Algorithm 4, under the PL condition, it holds that
K − 1 (cid:88)
[Equation]
Proof.
This lemma proves a weighted summation sequence for PL case convergence of GraB. Many of the steps are overlapping with Lemma 5 and Lemma 6.
For brevity, we will omit some of the steps, while their detailed derivations can be found in the proof of Lemma 5 and Lemma 6. First we define r k for any k ≥ 3 the same as Lemma
[Equation]
(cid:32) ∇ f (cid:18)
[Equation]
∞ .
From Lemma 5 and Lemma 6 we know that for any k ≥ 3 ,
[Equation]
where the definition of a k , b k and c k can be found in Lemma 6. We times ρ K − 1 − ( k +1) on both sides and obtain
[Equation]
Summing from k = 3 to K − 2 , we obtain
[Equation]
This implies that
[Equation]
We times 2 ρ on both sides, it gives us
[Equation]
Note that 2 ρ − 1 = 1 − αnµ ≥ 12 , so we can get
[Equation]
−
[Equation]
From Lemma 6 we know that
[Equation]
and [Equation] For the latter part, we can get [Equation]
Combine everything together,
[Equation] [Equation] Note that similar to Lemma 4, we get [Equation]
where in the last step
[Equation]
we apply the learning rate requirement that
[Equation]
Solve for the LHS we obtain
[Equation]
Note that when k = 1 , 2 ,
[Equation]
Combine everything together we get
[Equation]
That completes the proof.