New Bounds For Distributed Mean Estimation and Variance Reduction
Abstract
We consider the problem of distributed mean estimation (DME) , in which n machines are each given a local d -dimensional vector x v ∈ R d , and must cooperate to estimate the mean of their inputs µ = 1 n P nv =1 x v , while minimizing total communication cost.
DME is a fundamental construct in distributed machine learning, and there has been considerable work on variants of this problem, especially in the context of distributed variance reduction for stochastic gradients in parallel SGD.
Previous work typically assumes an upper bound on the norm of the input vectors, and achieves an error bound in terms of this norm.
However, in many real applications, the input vectors are concentrated around the correct output µ , but µ itself has large norm.
In such cases, previous output error bounds perform poorly.
In this paper, we show that output error bounds need not depend on input norm.
We provide a method of quantization which allows distributed mean estimation to be performed with solution quality dependent only on the distance between inputs , not on input norm, and show an analogous result for distributed variance reduction.
The technique is based on a new connection with lattice theory.
We also provide lower bounds showing that the communication to error trade-oﬀ of our algorithms is asymptotically optimal.
As the lattices achieving optimal bounds under ‘ 2 -norm can be computationally impractical, we also present an extension which leverages easy-to-use cubic lattices, and is loose only up to a logarithmic factor in d .
We show experimentally that our method yields practical improvements for common applications, relative to prior approaches.
Introduction
Several problems in distributed machine learning and optimization can be reduced to variants distributed mean estimation problem, in which n machines must cooperate to jointly estimate the mean of their d -dimensional inputs µ = 1 n P nv =1 x v as closely as possible, while minimizing communication.
In particular, this construct is often used for distributed variance reduction : here, each machine receives as input an independent probabilistic estimate of a d -dimensional vector ∇ and the aim is for all machines to output a common estimate of ∇ with lower variance than the individual inputs, minimizing communication.
Without any communication restrictions, the ideal output would be the mean of all machines’ inputs.
While variants of these fundamental problems have been considered since seminal work by Tsitsiklis & Luo (1987), the task has seen renewed attention recently in the context of distributed machine learning.
In particular, variance reduction is a key component in data-parallel distributed stochastic gradient descent (SGD), the standard way to parallelize the training of deep neural networks, e.g. Bottou (2010); Abadi et al.
(2016), where it is used to estimate the average of gradient updates obtained in parallel at the nodes.
Thus, several prior works proposed eﬃcient compression schemes to solve variance reduction or mean estimation, see e.g. Suresh et al.
; Alistarh et al.
Ramezani-Kebrya et al.
; Gandikota et al.
(2019), and Ben-Nun & Hoeﬂer (2019)
for a general survey of practical distribution schemes.
These schemes seek to quantize nodes’ inputs coordinatewise to one of a limited collection of values, in order to then eﬃciently encode and transmit these quantized values.
A trade-oﬀ then arises between the number of bits sent , and the added variance due of quantization.
Since the measure of output quality is variance, it appears most natural to evaluate this with respect to input variance, in order to show that variance reduction is indeed achieved.
Surprisingly, however, we are aware of no previous works which do so; all existing methods give bounds on output variance in terms of the squared input norm .
This is clearly suboptimal when the squared norm is higher than the variance, i.e., when inputs are not centered around the origin.
In some practical scenarios this causes output variance to be higher than input variance, as we demonstrate in Section 4.
Contributions.
In this paper, we provide the ﬁrst bounds for distributed mean estimation and variance reduction which are still tight when inputs are not centered around the origin.
Our results are based on new lattice-based quantization techniques, which may be of independent interest, and come with matching lower bounds, and practical extensions.
More precisely, our contributions are as follows:
• For distributed mean estimation , we show that, to achieve a reduction of a factor q in the input ‘variance’ (which we deﬁne to be the maximum squared distance between inputs), it is necessary and suﬃcient for machines to communicate Θ( d log q ) bits.
• For variance reduction , we show tight Θ( d log n ) bounds on the worst-case communication bits required to achieve optimal Θ( n )-factor variance reduction by n nodes over d -dimensional input, and indeed to achieve any variance reduction at all.
We then show how incorporating error detection into our quantization scheme, we can also obtain tight bounds on the bits required in expectation .
• We show how to eﬃciently instantiate our lattice-based quantization framework in practice, with guarantees.
In particular, we devise a variant of the scheme which ensures close-to-optimal communication-variance bounds even for the standard cubic lattice , and use it to obtain improvements relative to the best known previous methods for distributed mean estimation, both on synthetic and real-world tasks.
1. Problem Definitions and Discussion
MeanEstimation is deﬁned as follows: we have n machines v , and each receives as input a vector x v ∈ R d .
We also assume that all machines receive a common value y , with the guarantee that for any machines u, v , k x u − x v k ≤ y .
Our goal is for all machines to output the same value EST ∈ R d , which is an unbiased estimator of the mean µ = 1 n P v ∈ M x v , i.e. E [ EST ] = µ , with variance as low as possible.
Notice that the input speciﬁcation is entirely deterministic; any randomness in the output arises only from the algorithm used.
In the variant of VarianceReduction , we again have a set of n machines, and now an unknown true vector ∇ .
Each machine v receives as input an independent unbiased estimator x v of ∇ (i.e., E [ x v ] = ∇ ) with variance E (cid:2)
k x v − ∇ k 2 (cid:3) ≤ σ 2 .
Machines are assumed to have knowledge of σ .
Our goal is for all machines to output the same value EST ∈ R d , which is an unbiased estimator of ∇ , i.e., E [ EST ] = ∇ , with low variance.
Since the input is random, output randomness now stems from this input randomness as well as any randomness in the algorithm.
VarianceReduction is common for instance in the context of gradient-based optimization of machine learning models, where we assume that each machine v processes local samples in order to obtain a stochastic gradient ˜ g v , which is an unbiased estimator of the true gradient ∇ , with variance bound σ 2 .
If we directly averaged the local stochastic gradients ˜
g v , we could obtain an unbiased estimator of the true gradient G with variance bound σ 2 /n , which can lead to faster convergence.
Input Variance Assumption.
The parameter y replaces the usual MeanEstimation assumption of a known bound M on the norms of input vectors .
Note that, in the worst case, we can always set y = 2 M and obtain the same asymptotic upper bounds as in e.g.
Suresh et al.
; our results are therefore at least as good as previous approaches in all cases, but, as we will show, provide signiﬁcant improvement when inputs are not centered around the origin.
The reason for this change is to allow stronger bounds in scenarios where we expect inputs to be closer to each other than to the origin.
In particular, it allows our MeanEstimation problem to more eﬀectively generalize VarianceReduction .
Parameter y is a deterministic analogue of the parameter σ for VarianceReduction ; both y and σ provide a bound on the distance of inputs from their mean , rather than from the origin.
Accordingly, input variance σ 2 for a VarianceReduction instance corresponds (up to constant factors) to y 2 for a MeanEstimation instance.
For consistency of terminology, we therefore refer to y 2 as the input variance of the instance (despite such inputs being deterministic).
It is common in machine learning applications of VarianceReduction to assume that an estimate of the variance σ 2 is known (Alistarh et al., 2017; Gandikota et al., 2019).
To study both problems in a common framework, we make the analogous assumption about MeanEstimation , and assume knowledge of the input variance y 2 .
Even if the relevant bounds y or σ are not known a priori , they can usually be estimated in practice.
We discuss how we obtain estimates of input variance for our applications in Appendix G.
If one allows unrestricted communication, the straightforward solution to both problems is to average the inputs.
This is an exact solution to MeanEstimation with variance 0, and is an asymptotically optimal solution to VarianceReduction , of variance at most σ 2 n . {+
+}
However, doing so would require the exchange of inﬁnite precision real numbers.
So, it is common to instead communicate quantized values of bounded bit-length
(Alistarh et al., 2017), which will engender additional variance caused by random choices within the quantization method.
The resulting estimates will therefore have variance V ar quant for MeanEstimation , and σ 2 n + V ar quant for VarianceReduction .
We will show a trade-oﬀ between bits of communication and output variance for both problems; in the case of VarianceReduction , though, there is an ‘upper limit’ to this trade-oﬀ, since we cannot go below Ω( σ 2 n ) total output variance.
The other major diﬀerence between the two problems is that in MeanEstimation , distances between inputs are bounded by y with certainty, whereas in VarianceReduction they are instead bounded by O ( σ ) only in expectation .
This causes extra complications for quantization, and, as we will see, introduces a gap between average and worst-case communication cost.
Distributed Model. We aim to provide a widely applicable method for distributed mean estimation, and therefore we avoid relying on the speciﬁcs of particular distributed models. Instead, we assume that the basic communication structures we use (stars and binary trees) can be constructed without signiﬁcant overhead.
This setting is supported by machine learning applications, which have very high input dimension (i.e., d (cid:29) n ), and so the costs of synchronization or construction of an overlay (which do not depend on d , and are generally poly-logarithmic in n ), will be dominated by the communication costs incurred subsequently during mean estimation. For this reason, we do not include these model-speciﬁc costs in our stated complexities; any implementation of our techniques should take them into separate consideration. For simplicity, we will present our algorithms within a basic synchronous fault-free messagepassing model, in which machines can send arbitrary messages to any other machine, but they could naturally be extended to asynchronous and shared-memory models of communication. Our aim will be to minimize the number of bits sent and received by any machine during the course of the algorithm, i.e., we do not consider other measures such as round complexity.
Vector Norms. When dealing with vectors in R d , we will use names in bold, e.g. x , y . We will state most of our results in such a way that they will apply to any of the three most commonly-used norms on R d in applications: ‘ 1 norm k x k 1 := P di =1 | x i | , ‘norm k x k 2 := qP di =1 x 2 i , and ‘ ∞ norm k x k ∞ := max di =1 x i .
Throughout the paper we will therefore use the general notation k · k , which should be considered to be ﬁxed as one of these norms, other than for statements speciﬁc to particular norms.
Deﬁnitions which depend on norms, such as variance Var [ x ] := E (cid:2)
k x − E [ x ] k 2 (cid:3) , are therefore assumed to also be under the appropriate norm.
1. Related Work
Several recent works consider eﬃcient compression schemes for stochastic gradients, e.g. Seide et al.
(2014b)
[Table]
Wang et al.
[Table]
Alistarh et al.
[Table]
[Table]
Stich et al.
Wen et al.
; Wangni et al.
; Lu & Sa (2020).
We emphasize that these works consider a related, but diﬀerent problem: they usually rely on assumptions on the input structure—such as second-moment bounds on the gradients—and are evaluated primarily on the practical performance of SGD, rather than isolating the variance-reduction step.
(In some cases, these schemes also rely on history/error-correction (Aji & Heaﬁeld, 2017; Dryden et al., 2016;
Alistarh et al., 2018; Stich et al., 2018).)
As a result, they do not provide theoretical bounds on the problems we consider.
In this sense, our work is closer to Suresh et al.
(2017); Konečný & Richtárik (2018); Gandikota et al.
(2019), which focus primarily on the distributed estimation problem, and only use SGD as one of many potential applications.
For example, QSGD (Alistarh et al., 2017) considers a similar problem to VarianceRe- duction ; the major diﬀerence is that coordinates of the input vectors are assumed to be speciﬁed by 32-bit ﬂoats, rather than arbitrary real values.
Hence, transmitting input vectors exactly already requires only O ( d ) bits.
They therefore focus on reducing the constant factor (and thereby improving practical performance for SGD), rather than providing asymptotic results on communication cost.
They show that the expected number of bits per entry can be reduced from 32 to 2 .
8, at the expense of having an output variance bound in terms of input norm rather than input variance .
This is a common issue with existing quantization schemes, which leads to non-trivial complications when applying quantization to gradient descent and variance-reduced SGD (Künstner, 2017) or to model-averaging SGD (Lu & Sa, 2020), since in this case the inputs are clearly not centered around the origin.
The standard way to circumvent this issue, adopted by the latter two references, but also by other work on quantization (Mishchenko et al., 2019), is to carefully adapt the quantization scheme and the algorithm to remove this issue, for instance by quantizing diﬀerences with respect to the last quantization point.
These approaches, however, do not provide improvement as ‘one-shot’ quantization methods, and instead rely on historical information and properties of SGD or the function to optimize (such as smoothness).
They are therefore inherently application-speciﬁc.
Our method, by contrast, does not require “manual” centering of the iterates, and does not require storage of previous iterates, or any properties thereof.
Konečný & Richtárik (2018) study MeanEstimation under similar assumptions, and are the only prior work to use quantization centered around points other than the origin.
However, again prior knowledge about the input distribution must be assumed for their scheme to provide any improvements.
Suresh et al.
(2017) study the MeanEstimation problem deﬁned on real-valued input vectors.
They present a series of quantization methods, providing an O ( 1 n 2 P v ≤ n k x v k 22 ) upper bound, and corresponding lower bounds.
Recent work by Gandikota et al.
(2019) studies VarianceReduction , and uses multi-dimensional quantization techniques.
However, their focus is on protocols using o ( d )-bit messages per machine (which we show cannot reduce input variance).
They do give two quantization methods using Θ( d )-bit messages.
Of these, one gives an O ( 1 n max v ≤ n k x v k 2 2 ) bound on output variance, similar to the bound of Suresh et al.
(2017) for MeanEstimation (the other is much less eﬃcient since it is designed to achieve a privacy guarantee).
Mayekar & Tyagi (2020) obtain a similar error bound but with slightly longer Θ( d log log log(log ∗ d ))-bit messages.
All of the above works provide output error bounds based on the norms of input vectors.
This is only optimal under the implicit assumption that inputs are centered around the origin.
In Section G we provide evidence that this assumption does not hold in some practical scenarios, where the input (gradient) variance can be much lower than the input (gradient) norm: intuitively, for SGD, input variance is only close to squared norm when true gradients are close to 0 , i.e., the optimization process is already almost complete.
Our Results
In this work, we argue that it is both stronger and more natural to bound output variance in terms of input variance, rather than squared norm.
We devise optimal quantization schemes for MeanEstimation and VarianceReduction , and prove matching lower bounds, regardless of input norms.
We summarize the main ideas that lead to these results.
2. Lattice-based Quantization
The reason that all prior works obtain output variance bounds in terms of input norm rather than variance is that they employ sets of quantization points which are centered around the origin 0 .
We instead cover the entire space R d with quantization points that are in some sense uniformly spaced, using lattices .
Lattices are subgroups of R d consisting of the integer combinations of a set of basis vectors.
It is well-known (Minkowski, 1911) that certain lattices have desirable properties for covering and packing Euclidean space, and lattices have been previously used for some other applications of quantization (see, e.g., Gibson & Sayood (1988)), though mostly only in low dimension.
By choosing an appropriate family of lattices, we show that any vector in R d can be rounded (in a randomized, unbiased fashion) to a nearby lattice point, but also that there are not too many nearby lattice points, so the correct one can be speciﬁed using few bits.
Lattices contain an inﬁnite number of points, and therefore any encoding using a ﬁnite number of bits must use bit-strings to refer to an inﬁnite amount of lattice points.
To allow the receiver in our quantization method to correctly decode the intended point, we utilize the fact that we have a bound on the distance between any two machines’ inputs ( y for MeanEstimation , and O ( σ √ n ) (probabilistically, by Chebyshev’s inequality) for Vari- anceReduction ).
Therefore, if all points that map to the same bit-string are suﬃciently far apart, a machine can correctly decode based on proximity to its own input.
The simplest version of our lattice quantization algorithm can be described as follows
• To encode x u , randomly map to one of a set of nearby lattice points forming a convex hull around x u .
Denote this point by z .
• Send z mod q under the lattice basis: q is the quantization precision parameter.
• To decode with respect to x v , output the closest lattice point to x v matching z mod q .
By showing that x u is contained within a convex hull of nearby lattice points, we can round to one of these points randomly to obtain z such that the expectation of z is x u itself, thereby ensuring unbiasedness .
This is because a point within a convex hull can be expressed as a linear combination of its vertices with coeﬃcients in [0 , 1], which we can use as rounding probabilities.
Our reason for using mod q with respect to the lattice basis in order to encode lattice points into bit-strings is that by exploiting properties of the particular lattices we employ, we can show a lower bound on the distance between points encoded with the same bit-string, while also controlling the number of bits we use.
Then, since points encoded with the same string are suﬃciently far apart, our proximity-based decoding procedure can determine the correct point.
We also have a parameter (cid:15) which controls the granularity of the lattice used.
This method of lattice-based quantization gives the following guarantee for communicating a vector between two parties:
Theorem 1.
For any q = Ω(1) , any (cid:15) > 0 , and any two parties u , v holding input vectors x u , x v ∈ R d respectively, there is a quantization method in which u sends O ( d log q ) bits to v , and if k x u − x v k
=
O ( q(cid:15) ) , v can recover an unbiased estimate z of x u with k z − x u k
= O ( (cid:15) ) .
Details are given in Appendix A.
This result is very general, and can be applied not only to
DME but to any application in which high-dimensional vectors are communicated, in order to reduce communication.
2. Upper Bounds
Next, we show how to apply our quantization procedure to MeanEstimation and Vari- anceReduction .
We wish to gather (quantized estimates of) machines’ inputs to a single machine, which computes the average, and broadcasts a quantized estimate of this average to all other machines.
For simplicity of analysis we do so using a star and a binary tree as our communications structures, but any connected communication topology would admit such an approach.
The basic star topology algorithm can be described as follows:
• All machines u send x u , quantized with (cid:15) =
Θ( y ) and precision parameter q , to the leader machine v .
• Machine v decodes all received vectors, averages them, and broadcasts the result, using the same quantization parameters.
By choosing the leader randomly we can obtain tight bounds in expectation on the number of communication bits used per machine, and by using a more balanced communication structure such as a binary tree we can extend these bounds in expectation to hold with certainty (at the expense of requiring more communication rounds ).
Further details of these algorithmic results can be found in Appendix B.
Theorem 2.
For any q = Ω(1) , MeanEstimation can be performed with each machine using strictly O ( d log q ) communication bits, with O ( y 2 q ) output variance.
Theorem 3.
VarianceReduction can be performed using strictly O ( d log n ) bits, with
O ( σ 2 n ) output variance, succeeding with high probability.
These results give optimal communication-variance bounds for these problems.
However, to make this bounds practical, we address two main challenges.
Challenge 1: Input Variance.
One diﬃculty with our approach is that we assume a known estimate of input variance ( y 2 for MeanEstimation , σ 2 for VarianceReduction ).
Furthermore, in VarianceReduction , even if our input variance estimate is correct, some pairs of inputs can be further apart, since the bound is probabilistic.
To address this problem, we develop a mechanism for error correction , which allows the receiver to detect if the encode and decode vectors ( x u and x v respectively) are too far apart for successful decoding.
In this way, if our estimate of input variance proved too low, we can increase either it or the number of bits used for quantization until we succeed.
Details are presented in Appendix C; the main idea is that rather than using mod q to encode lattice points as bit-strings, we use a more sophisticated coloring of the lattice and a new encoding procedure to ensure if x u and x v are far apart, with high probability the encoder v chooses a color which is not used by any nearby point to x u , and therefore u can tell that x v was not nearby.
As an application, we obtain an algorithm for VarianceReduction which uses an optimal expected number of bits per machine (except for an additive log n , which in our applications is assumed to be far smaller than d log q ):
Theorem 4.
For any q = Ω(1) , VarianceReduction can be performed using O ( d log q + log n ) communication bits per machine in expectation, with O ( σ 2 q + σ 2 n ) output variance, succeeding with high probability.
Challenge 2: Computational Tractability.
Another issue is that known lattices which are optimal for ‘ 1 and ‘ 2 -norms can be computationally prohibitive to generate and use for problems in high dimension.
This is discussed further in Appendix D. We show that if we instead use the standard cubic lattice , which is optimal under ‘ ∞ -norm and admits straightforward ˜ O
( d )-computation encoding and decoding algorithms, in combination with a structured random rotation using the Walsh-Hadamard transform as proposed by Suresh et al.
(2017), we can come within a log-factor variance of the optimal bounds of Theorems 2, 3, and 4.
Theorem 5.
Using the cubic lattice with a random rotation, we achieve the following output variances under ‘ 2 -norm (succeeding with high probability):
• O ( y 2 log nd q ) for MeanEstimation , using strictly O ( d log q ) bits; • O ( σ 2 log d n ) for VarianceReduction , using strictly O ( d log n ) bits; • O ( σ 2 log nd q + σ 2 n ) for VarianceReduction , using O ( d log q + log n ) bits in expectation.
Furthermore, each machine need perform only ˜ O ( d ) computation in expectation.
Under the cubic lattice, our quantization procedure is particularly simple.
Since the lattice basis is orthogonal, to quantize a vector we can simply perform coordinate-wise rounding.
We round each coordinate either up or down to the closest multiple of the lattice side length, choosing the respective probabilities so that the expected value is the coeﬃcient of x u , similarly to Alistarh et al.
Likewise, to decode, we can ﬁnd the closest matching lattice point using coordinate-wise rounding.
So, the algorithm is then as follows:
• To encode x u , ﬁrst apply the structured random rotation HD (where H is the Walsh-Hadamard matrix and D is a random diagonal matrix: see Appendix D). • Round HD x u , to a lattice point z by coordinate-wise unbiased rounding. • Send z mod q under the lattice basis (which is the standard basis scaled appropriately), and D . • To decode, ﬁnd the closest lattice point to HD x v matching z mod q using coordinate-wise rounding.
• Apply the inverse rotation ( HD ) − 1 , and output the result.
We next show matching lower bounds on the communication required for MeanEstimation and VarianceReduction .
These results bound the number of bits a machine must receive (from any source) to output an estimate of suﬃcient accuracy, via an information-theoretic argument, and therefore apply to almost any model of communication.
Our proofs essentially argue that, if a machine receives only a small amount of bits during the course of an algorithm, it has only a small number of possible (expected) outputs.
We can therefore ﬁnd instances such that our desired output ( µ or ∇ ) is far from any of these possible outputs.
This argument is complicated, however, by the probabilistic nature of outputs (and, in the case of the VarianceReduction problem, inputs).
Theorem 6.
For any MeanEstimation algorithm in which any machine receives at most b bits in expectation ,
[Equation]
To achieve an output variance of O ( y 2 q ), we see that machines must receive Ω( d log q ) bits in expectation, matching the upper bound of Theorem 4.
Similarly, we have the following tight bounds for VarianceReduction :
Theorem 7.
For any VarianceReduction algorithm in which all machines receive (strictly) at most b bits,
[Equation]
This bound matches Theorem 3, since to reduce the variance expression to O ( σ 2 n ) (and, in fact, even to O ( σ 2 ), i.e., to achieve any reduction of output variance compared to input variance), we require b =
Ω( d log n ) bits.
Theorem 8.
For any VarianceReduction algorithm in which any machine receives at most b bits in expectation ,
[Equation]
Here we match the leading terms of Theorem 4: to reduce variance to O ( σ 2 q ), we require Ω( d log q ) bits in expectation.
Note that it is well known (and implied by e.g. Braverman et al.
that output variance cannot be reduced below O ( σ 2 n ) even with unlimited communication, so Theorem 8 implies that the full variance expression in Theorem 4 is tight.
These bounds are proven in Appendix E.
Experimental Validation
We investigate three distinct applications for our scheme: compressing gradients and models in variants of data-parallel SGD, and distributed power iteration for eigenvalue computation (Suresh et al., 2017).
We implement the practical version of our algorithm (Section 2.2), which for simplicity we call RLQSGD (or LQSGD for the version without Hadamard rotation).
The implemented variant ensures optimal variance, up to a log d factor (Theorem 5).
We compare against QSGD (Alistarh et al., 2017), the Hadamard-based scheme of Suresh et al.
(2017), as well as uncompressed baselines.
Please see Appendix G for a full description of the implementation, a complete experimental report, and additional experiments.
Example 1: Compressing Gradients.
We ﬁrst apply compression to parallel solver for least-squares regression, which is given as input a matrix A , partitioned among the nodes, and a target vector b , with the goal of ﬁnding w ∗ =
argmin w ||
A w − b
|| 22 .
In this example, we generate w ∗ ∈ R d and entries of A ∈ R S × d by sampling from N (0 , 1), and we set b =
A w ∗ .
Note that the input data (and gradients) are thus naturally normalized.
We then run distributed gradient descent with our quantization scheme, using 3 bits per coordinate for all methods, and examine the characteristics of the gradients sent, as well as the convergence and variance of the overall process.
Figure 1 (left) shows the gradient norms and distances for the case of two nodes executing gradient descent for d = 100 and 8K samples, while the center and right panels show the variance of the gradient estimate for each method and convergence, respectively.
[Figure]
Figure 1 (left) shows that, even in this simple normalized example, the distance between the two gradients throughout training is much smaller than the norm of the gradients themselves.
Figure 1 (center) shows that both variants of our scheme (RLQSGD and LQSGD, with and without rotation) have signiﬁcantly lower variance relative to Hadamard and standard
QSGD, and in fact are the only schemes to get below input variance in this setting.
Figure(right) shows that this leads to better convergence using our method.
The full report in
Appendix G contains experiments on other datasets, and node counts, as well as a largerscale application of our scheme to train neural networks.
This application shows that our scheme matches or slightly improves the performance of specialized gradient compression schemes, at the same bit budget.
This experiment is executed on 8 parallel workers; we found that the results are similar when throttling this parameter.
Example 2:
Local SGD.
A related example is that of compressing models in LocalSGD (Stich, 2018), where each node takes several SGD steps on its local model, followed by a global model averaging step, among all nodes.
(Similar algorithms are popular in Federated Learning (Kairouz et al., 2019).)
We use RLQSGD to quantize the models transmitted by each node as part of the averaging: to avoid artiﬁcially improving our performance, we compress the model diﬀerence ∆ i between averaging steps, at each node i .
RQSGD is a good ﬁt since neither the models nor the ∆ i are zero-centered.
We consider the same setup as for the previous example, averaging every 10 local SGD iterations.
We illustrate the convergence behavior and the quantization error in Figure 2, which shows better convergence and higher accuracy for lattice-based quantization. [Figure]
Example 3: Distributed Power Iteration.
The power iteration method estimates the principal eigenvector of an input matrix X , whose rows are partitioned across machines, to form input matrices
X i at the nodes.
Each row of the input matrix X is generated from a multivariate gaussian with ﬁrst two eigenvalues large and comparable.
In each iteration, each machine updates their estimate relative to its input matrix as u i =
X Ti X i x , and nodes average these estimates.
We apply 8-bit quantization to communicate these vectors u i , following Suresh et al.
Notice that our method is a good candidate here since these estimates clearly need not be zero-centered, a fact that is evident in Figure 3 (left).
Please see Appendix G.4 for a full description and additional results.
Figure 3 (center) shows convergence under diﬀerent quantization schemes, while Figure 3 (right) shows that our method provides signiﬁcantly lower quantization error across iterations. [Figure] Overall
[Figure]
the results suggest that our method can leverage its support of arbitrary centering to provide consistent improvements in these three diﬀerent settings
[Figure]
Conclusions
We have argued in this work that for the problems of distributed mean estimation and variance reduction, one should measure the output variance in terms of the input variance, rather than the input norm as used by previous works.
Through this change in perspective, we have shown optimal algorithms [Figure] and matching lower bounds, for both problems, independently of the norms of the input vectors.
This improves over the theoretical guarantees provided by previous work whenever the inputs are not known to be concentrated around the origin. Our experiments suggest that this also brings about improvements in terms of practical performance.
In future work, we plan to explore practical applications for variants of our schemes, for instance in the context of federated or decentralized distributed learning.
A Lattice-Based Quantization
Our quantization scheme will take the following course: to encode a vector x , we will round it (in an unbiased fashion) to a nearby point in some pre-determined lattice .
We then map lattice points to short bit-strings in such a way that points with the same bit-string are far apart and cannot be confused.
The decoding procedure can then recover the intended lattice point using proximity to a decoding vector , in our case the decoding machine’s own input.
We require the following deﬁnitions:
Deﬁnition 9.
A lattice Λ in d dimensions is an additively-closed subgroup of R d , deﬁned by a basis b 1 , . . .
, b d ∈ R d , and consisting of all integer combinations of the basis vectors.
For any lattice Λ ⊆ R d , the cover radius r c of Λ is the inﬁmum distance r such that S λ ∈ Λ B r ( λ ) = R d .
The packing radius r p of Λ is the supremum distance r such that for any λ 6 = λ 0 ∈ Λ , B r ( λ ) ∩ B r ( λ 0 ) = ∅ .
Based on these notions, can bound the number of lattice points in any ball in R d .
Lemma 10.
Let Λ ⊂ R d be a lattice, with cover radius r c and packing radius r p .
Then, for any x ∈ R d , δ > 0 , (cid:16) δ − r c r c (cid:17) d ≤ | B δ ( x ) ∩ Λ | ≤ (cid:16) δ + r p r p (cid:17) d .
Proof.
We ﬁx a point x ∈ R d and upper-bound the number of points in Λ within distance δ of it: consider the ball B δ + r p ( x ).
For any point y within distance δ of x , B r p ( y ) ⊂
B δ + r p ( x ).
It is also the case that for any z 6 = y ∈ Λ, B r p ( z ) ∩ B r p ( y ) = ∅ .
So,
[Equation]
Under any norm, the ratio V ol ( B r 1 ) V ol ( B r 2 ) of the volumes of two balls is ( r 1 r 2 ) d .
So: | B δ ( x ) ∩ Λ | ≤ (cid:18)
δ + r p r p
cid:19) d .
The lower bound follows similarly: B δ − r c ( x ) ⊂
S y ∈ Λ ∩ B δ ( x ) B r c ( y ), so
[Equation]
For our quantization purposes, we want lattices which have low r c (so that we can always ﬁnd a close lattice point to quantize to), and high r p (so that, by Lemma 10, there are not too many nearby lattice points, and therefore we can specify one with few bits).
Ideally we want r c
= O ( r p ).
Such lattices have long been known to exist:
Theorem 11.
For any d , under ‘ 1 , ‘ 2 , or ‘ ∞ -norm distances, there exists a lattice Λ ⊂ R d with r c ≤ 3 r p .
Proof.
For ‘ 1 or ‘ 2 -norm (or indeed any ‘ p norm, p ≥ 1), see Rogers (1950), or Proposition 1.1 of Henk (2016).
Under ‘ ∞ norm, the standard cubic lattice (i.e. with the standard basis as lattice basis) clearly has this property (in fact, r c = r p ).
We will call a lattice Λ an (cid:15) -lattice if (cid:15) =
r p ≤ r c ≤ 3 (cid:15) .
Note that r p , r c scale with the lattice basis, and therefore Theorem 11 implies that for any of the three norms, an (cid:15) -lattice exists for any (cid:15) > 0 by scaling appropriately.
We will denote such an epsilon lattice Λ (cid:15) .
A. Lattice Coloring
Rounding a vector x ∈ R d to a nearby lattice point is not suﬃcient for it to be communicated using a bounded number of bits, since there are still an inﬁnite number of lattice points (though we have reduced from the uncountably inﬁnite R d to the countably inﬁnite Λ).
So, we must encode an inﬁnite subset of lattice points using the same bit-string, and we want to ensure that the points in this subset are far apart, so that a receiver can identify which point the encoder intended.
There is a natural way to do this: we simply take a coordinate-wise (with respect to the lattice basis) mod operation, thereby mapping the inﬁnite number of points into a ﬁnite set of color classes .
That is, given a lattice Λ, and any positive integer q , we deﬁne a coloring procedure c q as follows: for any lattice point λ , represent λ as an integer combination of the canonical basis vectors of Λ, i.e. λ = α 1 b 1 + · · · + α d b d .
Then, we set c q ( λ ) i =
α i mod q , i.e. we obtain c q ( λ ) by applying the mod operation coordinate-wise to each of the entries α i .
We can then encode c q ( λ ) using d log q bits, since there are q possible integer values for each coordinate.
Lemma 12.
For any (cid:15) -lattice Λ (cid:15) , points λ 1 6 = λ 2 ∈ Λ (cid:15) with c q ( λ 1 )
=
c q ( λ 2 ) have k λ 1 − λ 2 k ≥ 2 q(cid:15) .
Proof.
Since c q ( λ 1 ) =
c q ( λ 2 ), the vector 1 q ( λ 1 − λ 2 ) must have integer coordinates under the canonical basis of Λ (cid:15) .
Therefore it is a point in Λ (cid:15) .
So, k 1 q ( λ 1 − λ 2 ) k ≥ 2 (cid:15) , since otherwise
B (cid:15) ( 1 q ( λ 1 − λ 2 ))
∩ B (cid:15) (0) 6 = ∅ , which cannot happen since r p = (cid:15) .
Then k λ 1 − λ 2 k ≥ 2 q(cid:15) .
A. Encoding Procedure
We are now ready to deﬁne our parameterized quantization procedure Q (cid:15),q : R d → { 0 , 1 } d log q which maps input vectors to subsets of a lattice , speciﬁed with d log q bits: let Λ (cid:15) be an (cid:15) -lattice, and let x be a ﬁxed vector to quantize.
We show that the convex hull of nearby lattice points contains x :
Lemma 13.
Any x is within the convex hull of B 7 (cid:15) ( x ) ∩ Λ (cid:15) .
Proof.
We show that for any nonzero vector w ∈ R d , there exists λ ∈ B 7 (cid:15) ( x ) ∩ Λ (cid:15) such that h w , λ i > h w , x i ; it is well-known that this implies that x is in the convex hull of
B 7 (cid:15) ( x ) ∩ Λ (cid:15) .
For ‘ 1 and ‘ 2 norms, let λ be the closest point in Λ (cid:15) to x + 4 (cid:15) w k w k .
Since r c ≤ 3 (cid:15) , k λ − ( x + 4 (cid:15) w k w k ) k < 3 (cid:15) (and so λ ∈ B 7 (cid:15) ( x )).
So,
[Equation]
Here the ﬁrst inequality uses Hölder’s inequality (and, in the case of ‘ 1 norm, the fact that k λ − [Equation] x + [Equation]
w
[Equation]
k ∞ ≤
[Equation] k [Equation] For ‘ ∞ norm, let w 0 be the vector 4 (cid:15) · sign ( w ), i.e., the vector containing with entries 4 (cid:15) at each positive coordinate of w and − 4 (cid:15) for each negative one.
Then, k w 0 k ∞ = 4 (cid:15) .
Let λ be the closest point in Λ (cid:15) to x + w 0 ; as before, we have k λ − ( x + w 0 ) k ∞ ≤ 3 (cid:15) and λ ∈ B 7 (cid:15) ( x ).
Then,
[Equation]
Again, the ﬁrst inequality uses Hölder’s inequality.
We can therefore show that we can probabilistically map x to these nearby lattice points in such a way that the expectation of the result is x :
Enumerate the points in B 7 (cid:15) ( x ) ∩ Λ (cid:15) as λ 1 , . . .
, λ z .
Since x is within the convex hull of
B 7 (cid:15) ( x ) ∩ Λ (cid:15) , there must be some sequence of non-negative coeﬃcients a 1 , . . .
, a z such that P zi =1 a
i
λ
i
= x .
Let z be a random vector taking value
λ i with probability a i P
z i =
1 a i , for all i .
Then, E [ z ] = x , and k x − z k < 7 (cid:15) (and so z is an unbiased estimator of x with variance most (7 (cid:15) ) 2 ≤ 49 (cid:15) 2 ).
We then set Q (cid:15),q ( x ) to be the color class of z in the coloring c q of Λ (cid:15) , which can be speciﬁed in d log q bits.
Q (cid:15),q ( x ) now corresponds to a subset Λ 0 (cid:15) containing z , such that any two elements of Λ 0 (cid:15) are of distance at least 2 q(cid:15) apart.
We summarize this process with pseudocode, taking as input the quantization parameters (cid:15) and q , Λ (cid:15) , and the vector x ∈ R d to quantize.
Algorithm 1 LatticeEncode , to compute Q (cid:15),q ( x )
Let λ 1 , . . .
, λ z ∈ B 7 (cid:15) ( x ) ∩ Λ (cid:15) , and a 1 , . .
. , a z ≥ 0 such that P zi =1 a
i λ i
= x .
Let z be a random vector taking value
λ i with probability a i P
z i =
1 a i , for all i .
Output c q ( z ), i.e., express z in the canonical basis of Λ (cid:15) and take mod q coordinate-wise.
We also summarize the properties of our quantization scheme in the following result:
Lemma 14.
There is a function Q (cid:15),q : R d → { 0 , 1 } b which maps each x ∈ R d to a string
Q (cid:15),q ( x ) of b =
d log q bits, specifying a subset Λ 0 (cid:15) ⊂ Λ (cid:15) with the following properties
: there exists z ∈ Λ 0 (cid:15) such that z is an unbiased estimator of x with k z − x k < 7 (cid:15) , and for all w ∈ Λ 0 (cid:15) \ { z } , k z − w k ≥ 2 q(cid:15) .
A. Decoding Procedure
We must now also deﬁne a procedure R (cid:15),q : { 0 , 1 } d log q × R d → R d to decode quantized values, using a machine’s own input x v (as the second input to the function): to do so, we simply take the point in the subset Λ 0 (cid:15) encoded by the received quantized value Q (cid:15),q ( x ) which is closest to x v :
Algorithm
, to compute R (cid:15),q ( Q (cid:15),q ( x ) , x v )
Let Λ 0 (cid:15) :
= { s ∈ Λ (cid:15) :
c q ( s ) =
Q (cid:15),q ( x ) } , the subset of lattice points matching Q (cid:15),q ( x ).
Output the closest point in Λ 0 (cid:15) to x v .
Lemma 15.
If k x − x v k ≤ ( q − 7) (cid:15) , then the decoding procedure R (cid:15),q ( Q (cid:15),q ( x ) , x v ) correctly returns the vector z which is an unbiased estimator of x .
Proof.
We upper-bound the distance to z :
[Equation]
, and lower-bound the distance to any other point y ∈ Λ 0 (cid:15) \ { z } :
[Equation]
= q(cid:15) .
Therefore z is the closest point in Λ 0 (cid:15) to x v , and will be returned by the decoding procedure.
Lemmas 14 and 15 together then imply our main theorem about the properties of our pairwise encode-decode process, Theorem 1.
B Algorithms
We now apply our quantization procedure to MeanEstimation and VarianceReduc- tion .
We ﬁrst give a simple algorithm using a star topology, in which machines communicate directly with a randomly chosen leader , in order to prove communication bounds in expectation.
Then, we show how to use a binary-tree communication structure in order to obtain these same bound with certainty on each machine.
B. Star-Topology Algorithm Algorithm 3 MeanEstimation
Nominate one machine v at random to be leader
All other machines u send Q (cid:15),q ( x u ) to v (and v simulates sending Q (cid:15),q ( x v ))
Machine v decodes (using x v ) and averages all received inputs to obtain ˆ µ
Machine v sends Q (cid:15),q ( ˆ µ ) to all machines All machines u decode (using x u ) and output the result
Theorem 16.
For any q = Ω(1) , Algorithm 3 performs MeanEstimation using O ( d log q ) communication bits per machine in expectation, with O ( y 2 q ) output variance.
Proof.
Let q ≥ 28, and (cid:15)
=
2 yq .
We then have ( q − 7) (cid:15) ≥ 34 q(cid:15)
= 1 .
5 y .
By Lemma 15, for any x , z with k x − z k ≤ 1 . 5 y ≤ ( q − 7) (cid:15) , decoding R (cid:15),q ( Q (cid:15),q ( x ) , z ) is successful.
Therefore, v correctly decodes all received messages (since all inputs are within distance y ), obtaining independent unbiased estimates of each machine’s input, each within distance 7 (cid:15) , and therefore with variance at most 49 (cid:15) 2 .
Then ˆ µ is an unbiased estimator of µ with variance at most 49 (cid:15) 2 n , and at distance at most 7 (cid:15) from µ .
ˆ µ is also at most distance y + 7 (cid:15) ≤ 1 .
5 y from any machine’s input, so again by Lemma 15, all machines correctly decode the encoded ˆ µ .
The output is therefore is unbiased estimate of ˆ µ with variance at most 49 (cid:15) 2 , i.e., an unbiased estimate of µ with variance at most 49 (cid:15) 2 + 49 (cid:15) 2 n ≤ 74 (cid:15)
= O ( y 2 q 2 ).
We thereby obtain a MeanEstimation algorithm achieving O
( y 2 q 2 ) output variance (and we simplify to O ( y 2 q ); this does not weaken the result because replacing q with q 2 does not asymptotically increase the number of bits required).
All nodes except v use using O ( d log q ) communication; v uses O ( nd log q ).
Since v is chosen uniformly at random, this gives the stated expected communication bounds.
We can perform a simple reduction to extend Algorithm 3 to VarianceReduction :
Theorem 17.
For any α > 1 , and any q between Ω(1) and O ( n 2 α ) , Algorithm 3 performs VarianceReduction using O ( d log q ) communication bits per machine in expectation, with
O ( αnσ 2 q ) output variance, succeeding with probability at least 1 − 1 α .
Proof.
By Chebyshev’s inequality, for any α > 1, Pr [ k x v − ∇ k ≥ σ √ αn ] ≤ 1 αn , and therefore by a union bound,
Pr (cid:2) All inputs within pairwise distance 2 σ √ αn (cid:3) ≥ Pr (cid:2)
All inputs within distance σ √ αn of ∇ (cid:3)
[Equation]
Therefore, we can reduce to a MeanEstimation instance with y = 2 σ √ αn , succeeding with probability 1 − 1 α .
Using q = O ( n 2 α ) (and (cid:15) = 2 yq = 4 σ √ αnq ), we obtain an unbiased estimate of µ with O ( αnσ 2 q ) variance, and therefore an unbiased estimate of ∇ with O ( σ 2 n + αnσ 2 q ) =
O ( αnσ 2 q ) variance, using O ( d log q ) expected communication per machine.
Notably, setting α = n c , for arbitrarily large constant c , and q = Θ( n 2 α ), Theorem 17 implies that we can achieve optimal O ( σ 2 n ) output variance using O ( d log n ) expected communication per machine, and succeeding with high probability in n (i.e., with success probability 1 − n − c ).
B. Tree-Topology Algorithm
Algorithm 3 uses a star topology for simplicity (all machines communicate via a randomly chosen leader v ). This accomplishes mean estimation in only two communication rounds , but has the drawback that the bounds on bits communicated are only in expectation, and machine v must perform signiﬁcantly more communication.
In this section, we show that, using a tree topology to more evenly distribute the work, we can obtain the communication bounds of Theorems 16 and 17 with certainty for all machines.
We note that, while we use a binary tree for simplicity, any connected communication graph would suﬃce.
However, we require communication rounds proportional to the diameter of the communication graph.
Therefore, our tree-topology algorithm requires O (log q ) rounds (though they need not be synchronous).
In systems where round complexity is a major concern, the two-round Algorithm 3 may be preferable.
Our algorithm to achieve worst-case bounds (as opposed to bounds in expectation) on communication cost is the following (Algorithm 4).
It uses a parameter m which controls both the quantization parameters and the number of input estimates to average; we will set m to achieve a trade-oﬀ in the same form as Theorem 16 later.
Sample a set T of min ( m, n ) machines uniformly at random.
Arrange nodes into a complete binary tree, with nodes in T as leaves.
Collect estimates from T to root of tree, averaging and encoding with Q ym 2 ,m 3 at every step.
Compute ﬁnal average at root, and broadcast to all machines (via a binary tree) encoded with Q ym 2 ,m 3 .
Decode and output result at all machines.
We now describe in more detail the steps of the algorithm:
Sampling a set of machines. We begin by sampling a set T of min ( m, n ) machines (so if m ≥ n , T is simply the set M of all machines).
Our goal will then be to estimate the average of the inputs x v , v ∈ T , and broadcast this estimate to all machines to output.
If m ≥ n then clearly µ T := 1 n P v ∈ T x v is equal to µ = 1 n P v ∈ M x v .
Otherwise, it is well-known that the sample mean µ T :
= 1 m P v ∈ T x v is an unbiased estimator of the population mean µ = 1 n P v ∈ M x v , with O ( y 2 m ) variance.
Therefore, if we can guarantee that all machines output the same value EST which is an unbiased estimator of µ T with variance O ( y 2 m ), we have correctly performed MeanEstimation with output variance O ( y 2 m ).
Arranging a communication tree. Our communication structure will be a complete binary tree, with the machines in T acting as the leaves (since we have choice of m and provide asymptotic bounds, we may assume it is a power of 2).
The roles of the remaining nodes of the communication tree may be taken by any arbitrary machines, so long as all machines take only O (1) roles.
We will then send messages up the tree (i.e., from child nodes to their parents), in order to convey estimates of the inputs of the leaf nodes (machines in
T ) to the root, which can then compute the ﬁnal average.
Collecting estimates from leaves to root. When a node in the communication tree receives and decodes a vector from both of its children, it takes the average, encodes with
Q ym 2 ,m 3 , and sends the result to its parent.
We will denote by A v the average input of all descendant leaves of a tree node v .
Our goal is then to show that the average computed by node v is an unbiased estimator of A v with low variance.
Since A r = µ T for the root r of the communication tree, we will then have an unbiased estimator of µ T (and therefore µ ) as desired.
To bound the estimator error at each step, we will employ an inductive argument.
Considering nodes by their depth in the tree (with leaves at depth 0 and the root r at depth log min( m, n )), we show the following:
Lemma 18.
A node v at depth i sends to its parent an unbiased estimator a v of A v with k A v − a v k ≤ 7 iym 2 , encoded with Q y m 2 ,m 3 .
Proof.
By induction.
Clearly the claim is true for leaf nodes at 0, which encode exactly A v (i.e. their own input).
Assuming the claim is true for i , we prove for i + 1:
Node v receives two values Q ym 2 ,m 3 ( a u ) and Q ym 2 ,m 3 ( a w ) from its children u and w .
By the inductive assumption, k A u − a u k ≤ 7 iym 2 , and so k x v − a u k ≤ k
A u − a u k + k x v − A u k ≤iym
+ y <
( m 3 − 7) ym 2 (for m at least a suﬃciently large constant).
By Lemma 15, therefore, v correctly decodes the message from u , to recover an unbiased estimator z u of a u with k z u − a u k < 7 ym 2 .
The same argument holds for w .
Node v then takes as a v the average of z u and z w .
Since k z u − A u k ≤ k
A u − a u k + k z u − a u k <iym 2 + 7 ym 2 =
( i +1)
y m(and the same holds for w ):
[Equation]
This completes the proof by induction.
Computing the ﬁnal average at the root. By Lemma 18, the root node r , at depth log min( m, n ), computes an unbiased estimator a r of µ T with k a r − µ T k ≤ 7 y logmin( m,n ) m≤ 7 y log m m. It then encodes this vector with Q ym 2 ,m 3 , and broadcasts it to all other machines via an arbitrary binary tree (all machines performing the role of one node in the tree).
Nodes in the tree relay the same message to their children until all nodes have received the message.
Then, all machines decode the message and output the resulting vector.
Decoding and outputting EST .
All machines have now received an unbiased estimator a r of µ T with k a r − µ T k ≤ 7 y log m m, encoded with Q ym 2 ,m 3 .
Any machine v has
[Equation]
and so by Lemma 15, v correctly decodes the message to recover an unbiased estimator z r of a r with k z r − a r k < 7 ym 2 .
All nodes therefore output z r , which is an unbiased estimator of µ T , with
[Equation]
Proof of Theorem 2.
As noted earlier, an unbiased estimator of µ T is also an unbiased estimator of µ (with O ( y 2 m ) additional variance), and z r is therefore a correct solution to
MeanEstimation with variance Var [ z r ] ≤ k z r − µ T k 2 + O
( y 2 m )
=
O ( y 2 m ).
We can bound the communication cost as follows: all machines have sent and received O (1) vectors encoded with Q ym 2 ,m 3 , which require O ( d log m ) bits.
We set m = q to reach an expression of the same form as Theorem 16, and obtain Theorem 2.
We can apply the same reduction as for Theorem 17 to obtain the following for Vari- anceReduction , using that a MeanEstimation algorithm with y = 2 σ √ αn solves VarianceReduction with probability at least 1 − 1 α :
Theorem 19.
For any α > 1 , and any q between Ω(1) and O ( n 2 α ) , Algorithm 4 performs VarianceReduction using O ( d log q ) communication bits per machine in total, with
O ( αnσ 2 q ) output variance, succeeding with probability at least 1 − 1 α .
As a corollary to this, we can obtain Theorem 3:
Proof of Theorem 3.
We set α = n c , for arbitrarily large constant c , and q = Θ( n 2 α ).
Theorem 19 then implies that we can achieve optimal O ( σ 2 n ) output variance using O ( d log n ) total communication per machine, and succeeding with high probability in n (i.e., with success probability 1 − n − c ).
C Error Detection In Quantization
In this section we equip our quantization method with a mechanism of error detection : that is, we wish to detect when the encode and decode vectors are too far for a successful decoding.
We ﬁrst replace the coloring used in Section A.1 with a new coloring method, which will allow us to detect this kind of error with high probability.
Lemma 20.
For any Λ (cid:15) and any r at least a suﬃciently large constant, there exists a coloring c r : Λ (cid:15) →
[ r O ( d ) ] such that for any q ∈ N with 4 ≤ q ≤ r and any λ 1 , λ 2 ∈ Λ (cid:15) with k λ 1 − λ 2 k ≤ 1 .
5 q q d (cid:15) , there are at most 2 q d points in S :
= Λ (cid:15) ∩ ( B r 3 (cid:15) ( λ 1 ) ∪ B r 3 (cid:15) ( λ 2 ))
whose colors are not unique in S .
Proof.
We show that such a coloring exists by the probabilistic method: we randomly construct a candidate coloring, and prove that it satisﬁes the necessary properties with positive probability.
Then, one such coloring must exist.
Let f : (cid:20)(cid:16) r q d (cid:17) d (cid:21) →
[ r 10 d ] be a uniformly random function.
Our candidate coloring c is then given by c ( λ ) =
f ( λ mod r q d ) (where mod is taken coordinate-wise with respect to the canonical basis of the lattice Λ (cid:15) ).
So, in eﬀect, we divide the lattice points into r dq d classes by taking mod r q d , and then randomly color the classes.
Any two points λ , λ 0 in the same class are of distance at least 2 · r q d (cid:15) apart, since r − q d ( λ − λ 0 ) is a non-zero lattice vector, and so is of length at least 2 (cid:15) .
Fixing some pair of points λ 1 , λ 2 ∈ Λ (cid:15) (and therefore some set S = Λ (cid:15) ∩ ( B r 3 (cid:15) ( λ 1 ) ∪
B r 3 (cid:15) ( λ 2 ))), we note that all pairs of points in S are at most distance 1 .
5 q q d (cid:15) +2 r 3 (cid:15) < 2 · r q d (cid:15) apart, so they are all in diﬀerent classes, and therefore their colors were chosen independently uniformly at random.
By Lemma 10, in any ball of radius r 3 (cid:15) , there are at most (cid:16) r 3 (cid:15) + (cid:15) (cid:15) (cid:17)
d = (cid:0) r 3 + 1 (cid:1) d lattice points.
Therefore, | S | ≤ 2 (cid:0) r 3 + 1 (cid:1) d .
Our goal now is to show that few points in S are given non-unique colors.
Order the points in S arbitrarily.
If there are more than 2 q d colors used by multiple points in S , then there are at least q d points assigned a color already used by another point prior in the ordering.
For any ﬁxed set S 0 of q d points in S , the probability that all receive a color already used by another point prior in the ordering is at most (cid:18) 2 ( r 3 +1 )
d r 10 d (cid:19) q d , since the colors are chosen uniformly and independently, and at most 2 (cid:0) r 3 + 1 (cid:1) d are used by prior points.
So,
Pr (cid:2) |{ points with non-unique colors in S }| > 2 q d (cid:3) ≤ X
[Equation]
< (cid:18) [Equation] r [Equation]
We take a union bound over all pairs of classes for λ 1 , λ 2 ; this incorporates all choices for λ 1 , λ 2 since the coloring is identical with respect to two pairs of points from the same pair of classes.
There are fewer than r 2 dq d such pairs, so the probability that any do not satisfy the condition is at most r − dq d .
Finally, we take a union bound over all q with 4 ≤ q ≤ r , and see that the probability that any such q does not satisfy the condition is less than P rq =4 r − dq d < 1.
Therefore, we have a positive probability that c satisﬁes the criteria of the lemma, so by the probabilistic method, such a good coloring must exist.
We utilize this coloring in the following way: when encoding a vector x u with a lattice point z , we can now ensure that for any decode vector x v , with high probability, the color of z is unique among points within r 3 (cid:15) of x u and x v .
Therefore, when machine v decodes, it either recovers z or knows that k x v − z k > r 3 (cid:15) .
Now that we have means of detecting whether the encode and decode vectors are too far apart for successful decoding, we can design a process that iteratively attempts to quantize using more bits until decoding succeeds:
Let ˆ x u be chosen uniformly at random from B q 2 (cid:15) ( x u ).
Map ˆ x u to a lattice point z ∈ B 7 (cid:15) (ˆ x ) ∩ Λ (cid:15) in unbiased fashion, as in Lemma 14.
r ← q loop
Machine u sends c r ( z ) to v
Machine v computes y ∈ Λ (cid:15) , the closest lattice point to x v
such that c r ( y )
=
c r ( z ) if k x v − y k ≤ 12 r 3 (cid:15) then v outputs y , procedure terminates else v sends the message Far to u r ← rend if end loop
Here the purpose of ﬁrst choosing a vector ˆ x u uniformly from B q 2 (cid:15) ( x u ) is to ensure that the probability that z takes any particular value is low, since there are a small number of points which do not have non-unique colors which we wish to avoid with high probability.
Lemma 21.
Lattice point z is an unbiased estimator of x u , with k z − x u k ≤ ( q 2 + 7) (cid:15) .
Furthermore, the probability of z taking any particular value ˆ z is at most ( 7 q 2 ) d .
Proof.
By our choice of z we clearly have k z − x u k ≤ ( q 2 + 7) (cid:15) ; unbiasedness follows from
Lemma 14.
For z to take some particular value ˆ z , we must have Pr [ k ˆ z − ˆ x k ≤ 7 (cid:15) ], which is the case with probability at most V ol ( B 7 (cid:15) )
V ol ( B q 2 (cid:15) )
= ( 7 q 2 ) d .
We next analyze what the properties of the communication procedure within the loop of
Algorithm 5, for some ﬁxed r :
Lemma 22.
During a loop of RobustAgreement :
• c r ( z ) takes O ( d log r ) bits to send.
• If k x u − x v k ≤ 12 r 3 (cid:15) , with probability at least 1 − ( 14 q ) d , v outputs z .
• If k x u − x v k ≤ q q d (cid:15) , with probability at least 1 − ( 14 q ) d , v either outputs z or sends Far to u .
Proof.
c r ( z ) is a color from [ r O ( d ) ], which takes O ( d log r ) bits to specify.
Let λ u be the closest lattice point to x u , and λ v be the closest lattice point to x v .
By the properties of an (cid:15) -lattice, k λ u − x u k and k λ v − x v k are at most 3 (cid:15) .
If k x u − x v k ≤ q q d (cid:15) , then k λ u − λ v k ≤ q q d (cid:15) + 6 (cid:15) ≤ 1 .
5 q q d .
So, by Lemma 20, there are at most 2 q d points with non-unique colors in S = Λ (cid:15) ∩ ( B r 3 (cid:15) ( λ u ) ∪ B r 3 (cid:15) ( λ v )).
We have z ∈ Λ (cid:15) ∩ ( B r 3 (cid:15) ( λ u ))
⊂ S , so the probability of its color being unique in S is at least 1 − 2 q d · ( 7 q 2 ) d
= 1 − ( 14 q ) d , by Lemma 21.
In this case, y is either z , or is outside B r 3 (cid:15) ( λ v ) .
If k x u − x v k ≤ 12 r 3 (cid:15) , then k y − λ v k ≤ r 3 (cid:15) , so we must have y =
z , and v correctly outputs it.
Otherwise, we may have y / ∈ B r 3 (cid:15) ( λ v ) , but if so, v sends Far to u , which is also permitted.
We can then show the properties of the algorithm as a whole:
Lemma 23.
If k x u − x v k ≤ q q d (cid:15) , with probability at least 1 − log log( 1 (cid:15) k x u − x v k ) · O ( q − d ) ,
Algorithm 5 provides machine v with an unbiased estimate z of x u , with k z − x u k ≤ ( q 2 +7) (cid:15) , and uses O (cid:0) d log( q(cid:15) k x u − x v k ) (cid:1) bits.
Proof.
After performing i iterations we have r = q 2 i .
Therefore, after iteration 2 log log( k x u − x v k /(cid:15) ) we have k x u − x v k ≤ 12 r 3 (cid:15) .
Then, so long as this iteration, and all previous iterations, are successful, v outputs an estimate z of x u with k z − x u k ≤ ( q 2 + 7) (cid:15) , by Lemma 22.
Since we have O (log log( k x u − x v k /(cid:15) )) iterations, and each fails with probability at most Ω( q ) − d , our total failure probability is log log( k x u − x v k /(cid:15) )
· O ( q − d ) by a union bound over all iterations.
The number of bits used per iteration is O ( d log r ); since log r doubles each iteration, the total is at most twice the bound for the ﬁnal iteration.
If this is the ﬁrst iteration, the bound is O ( d log q ); otherwise it is O
( d log( 1 (cid:15) k
x u − x v k )).
So, the total number of bits used is O
( d log( q(cid:15) k x u − x v k )).
C. Application to VarianceReduction
We now apply our error detection in order to improve our bounds on expected communication for performing VarianceReduction .
The idea is that, since we only have a probabilistic bound on distance between inputs, some pairs of inputs may be signiﬁcantly further away: we can now detect these cases using error detection, and increase the number of bits used for quantization accordingly.
We use the star-topology algorithm (Algorithm 3), and simply replace each pairwise encoding and decoding interaction with RobustAgreement ( (cid:15), q ): Algorithm
Nominate one machine v uniformly at random to be leader
All other machines u perform RobustAgreement ( (cid:15), q ) to send x u to v
Machine v averages all received estimates to obtain ˆ ∇
Machine v performs RobustAgreement ( (cid:15), q ) with each other machine to send ˆ ∇ , using the same choice of z
All machines output the resulting estimate
We are now ready to prove our main result applying error detection to VarianceReduc- tion , Theorem 4.
Proof of Theorem 4.
We set q to be q + cn 2 d for some suﬃciently large constant c , and run Algorithm 6 with q and (cid:15) =
σ/ ( q 4 ).
We ﬁrst consider the distances between encode and decode vectors in the ﬁrst stage of messages, where all other machines u send their input to the leader v using RobustAgreement .
In each case, E [ k x v − x u k ] ≤
E [ k x v − ∇ k + k x u − ∇ k ] ≤ 2 σ .
So long as we succeed (as speciﬁed by Lemma 23), node v then receives the estimate ˜ x of x u , with k ˜ x − x u k ≤ ( q 2 + 7) (cid:15) <
σ/ q (since q is chosen to be at least 3), and uses O (cid:0) d log( q (cid:15) k x u − x v k ) (cid:1) bits.
We have E (cid:2) d log( q (cid:15) k
x u − x v k ) (cid:3) =
O ( d log q ) =
O ( d log q + log n ).
The leader node v now averages these estimates to obtain ˆ ∇ .
Each of the received estimates is an independent unbiased estimator of ∇ with variance O ( σ 2 ) and therefore ˆ ∇ is an unbiased estimator of ∇ with variance O ( σ 2 /n ).
In the second stage of messages, v sends ˆ ∇ to all other nodes using RobustAgreement , taking the same choice of z in each, in order to ensure that machines all have the same output.
z is an unbiased estimate of ∇ with variance O ( σ 2 n + σ 2 q ) =
O ( σ 2 min n,q ).
Again, for each such node u we have E
h k x u − ˆ ∇ k i
=
O ( σ ), so we use O ( d log q + log n ) bits in expectation.
Therefore each node uses O ( d log q + log n ) total bits in expectation, except the leader v which uses O ( nd log q + n log n ).
Since v is chosen uniformly at random, we have an O ( d log q + log n )-bit bound on expected communication for each machine.
In all applications of RobustAgreement we perform, denoting the encode and decode vectors a , b respectively (i.e., in the ﬁrst stage a = x u and b = x v for each u , and in the second a = ˆ ∇ and b = x u ), we have E (cid:2)
k
a − b k 2 (cid:3) ≤ (2 σ ) 2 and so
Pr [ k a − b k ≤ 4 nσ ] ≤ 14 n 2 .
Since we perform RobustAgreement 2 n times, with probability at least 1 − 12 n , k a − b k ≤ 4 nσ in all cases by a union bound.
So, with high probability we always satisfy the condition k
a − b k ≤ q q d (cid:15) =
q q d − 4 σ , and have failure probability at most log log( 1 (cid:15)
k a − b k ) ·
O ( q − d ) =
O ( q − d log log( n q )) for each application of RobustAgreement .
So, the probability that all applications succeed is at least (1 − 12 n )(1 − O ( q − d log log( n q ))) ≥ 1 − 1 n .
Then, all machines correctly receive z , an estimate of ∇ with variance at most O ( σ 2 min n,q ).
D ‘ 2 Guarantees for the Cubic Lattice
One key question when applying our scheme in practice is the choice of lattice .
Ideally this would depend on the norm under which the results will be evaluated.
However, asymptotically optimal lattices for ‘ 1 and ‘ 2 norms can be computationally expensive to generate and use in practice.
For this reason, in our practical implementations we will employ the standard cubic lattice : this gives us computationally-eﬃcient ( ˜ O ( d )-time) quantization algorithms, is optimal under ‘ ∞ norm, and, as we will see, performs surprisingly well even evaluated under ‘ 1 or ‘ 2 norm.
We will investigate more complex lattices, tailored to ‘or ‘ 2 norm, in future work.
There are two possible approaches to doing so: one would be to apply general lattice algorithms (e.g. Laarhoven (2016)) in applications where d is still fairly low.
This is often the case in, for example, gradient descent for neural network training, where coordinates are already divided into fairly small buckets .
The second possible approach would be to ﬁnd speciﬁc lattices which admit more eﬃcient algorithms, and also have a good r c /r p ratio under ‘ 1 or ‘ 2 norm.
Using the cubic lattice, though, need not sacriﬁce too much by way of theoretical guarantees under ‘ 2 -norm, since, as noted in Suresh et al.
(2017), a random rotation using the WalshHadamard transform can ensure good bounds on the ratio between ‘ 2 norm and ‘ ∞ norm of vectors.
Let H be the d × d normalized Hadamard matrix H i,j = d − 1 / 2 ( − 1) h i − 1 ,j − 1 i , where h i, j i is the dot-product of the log 2 d -dimension { 0 , 1 } -valued vectors given by i , j expressed in binary (we must assume here that d is a power of two, but this does not aﬀect asymptotic results).
We use the following well-known properties of H (see e.g. Horadam (2007)):
• H is orthonormal, and so preserves ‘ 2 -norm distances; • H − 1 = H .
Let D be a d × d diagonal matrix where each D i,i is drawn uniformly at random from {− 1 , 1 } .
H is a ﬁxed, known matrix, and D costs d bits to communicate, so we can assume that both matrices are known to all machines at a cost of only O ( d ) bits of communication per machine.
Before applying our MeanEstimation or VarianceReduction algorithms, we apply the transformation HD to all inputs - we then invert the transform (i.e. apply ( HD ) − 1 =
D − 1 H ) before ﬁnal output.
As shown in Ailon & Chazelle (2009), both the forward and inverse transform require only O ( d log d ) computation.
Lemma 24.
For any set S of n 2 vectors in R d , with probability at least 1 − 2 d − 1 , all vectors x ∈ S have
[Equation]
Proof.
We follow a similar argument to Ailon & Chazelle (2009).
Fix some vector x ∈ S and some coordinate j ∈ [ d ].
Notice that ( HD x ) j = P di =
α
i x i , where each α i = ±
d − 1 /is chosen independently and uniformly.
By a Chernoﬀ-type bound (c.f. Ailon & Chazelle (2009)), we therefore obtain Pr [ | ( HD x ) j | ≥ s k x k 2 ] ≤ 2 e − s 2 d/ 2 .
Plugging in s = 2 q ln ndd , we get:
[Equation]
We then take a union bound over all x ∈ S and j ∈ [ d ] to ﬁnd that with probability at least 1 − 2 d − 1 , all x ∈ S have k HD x k ∞ =
O ( d − 1 / 2 k x k 2 √ log nd ).
We can now obtain analogue of Theorem 2 using the cubic lattice:
Theorem 25.
For any q = Ω(1) , MeanEstimation can be performed using the cubic lattice with each machine using O ( d log q ) communication bits in total, with O ( y 2 log nd q ) output variance under ‘ 2 norm, succeeding with probability at least 1 − 2 d − 1 .
Proof.
We apply the random rotation HD as in Lemma 24, and then proceed as usual using
Algorithm 4, applying the inverse rotation to the ﬁnal output.
Let y 2 denote the usual maximum input distance parameter under ‘ 2 -norm.
We ﬁrst show that after applying the random rotation we have y ∞ =
O ( d − 1 / 2 y 2 √ log nd ):
There are n machines v with input vectors x v ∈ R d .
We form a set S
= { x v − x u for all machines u, v } .
This set then has at most n 2 elements.
Applying the rotation
HD to all input vectors implicitly applies it to this set S of diﬀerences , since HD is a linear transformation.
Therefore, by Lemma 24, we have
[Equation]
so we can set y ∞
=
O ( d − 1 / 2 y 2 √ log nd ) and satisfy that k HD ( x v − x u ) k ∞ ≤ y ∞ for all machines u, v , with probability at least 1 − 2 d − 1 .
Now we apply Algorithm 4 using the cubic lattice.
Since the cubic lattice is optimal under ‘ ∞ norm, we obtain an output variance (under ‘ ∞ norm) of O ( y 2 ∞ q )
=
O ( d − 1 y 22 log nd q ) when using O ( d log q ) bits per machine, by Theorem 2 (but our output HD EST is currently an unbiased estimator of the rotated mean HD µ rather than µ itself).
Since, for any vector z ∈ R d , k z k 22 ≤ d k z k 2 ∞ , when applied with z = HD ( EST − µ ) we get an output variance under ‘ 2 -norm of O ( y 22 log nd q )
When applying the ﬁnal inverse rotation to the output vector HD EST , we also implicitly imply it to the error vector HD ( EST − µ ).
This inverse rotation preserves distances under ‘ 2 -norm, so we still have O ( y 22 log nd q ) ‘ 2 -norm variance, now of an unbiased estimator EST of the mean µ of our original inputs (unbiasedness is preserved by linearity of expectation, since HD and ( HD ) − 1 are linear transformations).
By applying the same analysis to Theorems 3 and 4, we obtain the VarianceReduction results of Theorem 5.
Proof of Theorem 5.
We perform the same procedure as in Theorem 25, applying the random rotation HD to obtain a bound on ‘ ∞ -norm, applying Algorithms 3 and 4 respectively, using the ‘ i nfty -norm-optimal cubic lattice, and then applying the inverse rotation before output to obtain an ‘ 2 -norm output variance bound at only a O (log nd ) factor higher than optimal.
Speciﬁcally, we can obtain unbiased estimates of input mean µ with variance O ( σ 2 log nd n) using strictly O
( d log n ) bits (with suﬃciently high constant within the asymptotic notation), and O ( σ 2 log nd q ) using O ( d log q + log n ) bits in expectation.
Since µ is itself an
O ( σ 2 n )-variance unbiased estimator of our true vector ∇ , we obtain ﬁnal output variances of O ( σ 2 log nd n+ σ 2 n ) =
O ( σ 2 log d n ) and O ( σ 2 log nd q + σ 2 n ) respectively.
E Lower Bounds
We prove matching lower bounds for MeanEstimation and VarianceReduction , using an argument based on bounding the volume of space for which a node can output a good estimate, if it receives a limited number of communication bits.
Our lower bounds will be against algorithms with shared randomness : that is, we assume access to a common random string s , drawn from some distribution, for all machines.
We begin with the simpler MeanEstimation bound:
Theorem 26.
For any MeanEstimation algorithm in which any machine receives at most b bits,
[Equation]
Proof.
We construct a hard input for mean estimation as follows: ﬁx a machine v , and arbitrarily ﬁx its input vector x v .
Machine v ’s output EST is dependent only on x v , the random string s , and the string of bits B v that v receives (from any other machines) during the course of the algorithm.
We denote by b v the number of such bits.
If b v < b for some b , then v has P b − 1 i =
i < 2 b possible strings B v , and therefore fewer than 2 b possible output distributions (over choice of s ).
For each B v we denote by OUT B v the set of points z ∈ R d for which P s [ k EST − z k ≤ δ ] > 12 , when B v is the string received.
For any B v , OUT B v has volume at most V ol ( B 2 δ ), since the δ -balls of any two points in OUT B v must intersect (as the probabilities of EST falling within the balls sum to more than 1).
We further denote OUT v to be the union of these sets over all B v .
Then:
[Equation]
We choose δ =
y 4 2 − bd , and see that:
[Equation]
Therefore there is some point z in B y 2 ( x v ) \ V ol ( OUT v ).
We choose µ to be such a point z , by setting one other machine’s input to 2 z − x v , and any others to z (note that this satisﬁes the condition that all inputs are within distance y ).
Then, regardless of B v , P s [ k EST − µ k ≤ δ ] ≤ 12 , and so E (cid:2) k EST − µ k 2 (cid:3) ≥ 12 · δ 2 = Ω( y 2 2 − 2 bd ).
This lower bound is for algorithms with a ﬁxed communication cost, but we show that it can easily be extended to apply to expected communication cost, proving Theorem 6:
Proof of Theorem 6.
By Markov’s inequality, a machine v which receives at most b bits in expectation receives at most 1 . 5 b bits with probability at least 13 .
Then,
E (cid:2) k EST − µ k 2 (cid:3) ≥ 1 3 E (cid:2) k EST − µ k 2 | v receives at most 1 . 5 b bits (cid:3) = Ω( y 2 2 − 3 bd ) .
The bounds for VarianceReduction are somewhat more complex since we must deﬁne a hard input distribution in which the input vectors are independent estimates of the true vector ∇ and have variance at most σ 2 .
We ﬁrst show a bound on the amount of communication bits that all machines must use (Theorem 8, before proceeding to Theorem 7 which shows that some machines must use more bits.
Proof of Theorem 8.
We deﬁne an input distribution I as follows: we choose ∇ uniformly at random from B d 2 σ ( 0 ).
We then independently choose each machine v ’s input x v uniformly at random from B σ ( ∇ ).
For any machine v , we deﬁne event
A v = {k x v k ≤ ( d 2 − 1) σ } .
[Equation]
Now, conditioning on A v , ∇ is distributed uniformly in B σ ( x v ).
We next show that in this case, v will, with high probability, not be able to closely estimate ∇ .
Using the same argument and deﬁnitions as in proof of Theorem 26, we have:
[Equation]
We choose δ =
σ 4 2 − bd , and see that V ol ( OUT v ) <
2 − d V ol ( B σ ).
Therefore, P I [ ∇ ∈ OUT v |
A v ] ≤ 2 − d , and so P I [ ∇ / ∈ OUT v ] ≤ (1 − 4 d )(1 − 2 − d ) ≤ 12 .
When ∇ / ∈ OUT v , with probability at least 1 2 , P s [ k EST − µ k ≤ δ ] ≤ 1 2 .
So, we have that:
[Equation]
Then, E {+(cid:2) k EST − ∇ k 2 (cid:3) > 0 . δ 
+}
= Ω( σ 2 n 2 − 2 bd ).
Theorem 7 is proven similarly, but using a diﬀerent input distribution that causes some inputs to be further from ∇ .
Proof of Theorem 7.
We deﬁne an input distribution I as follows: we choose ∇ uniformly at random from B
d 2 nσ ( 0 ).
We then independently choose each machine v ’s input x v as follows: with probability 1 n , x v is chosen uniformly from B σ √ n ( ∇ ), and otherwise x v = ∇ .
For any machine v , we are interested in the event
A v that
• x v is chosen from B σ √ n ( ∇ ), and • k x v k ≤ d 2 nσ − [Equation]
If we condition on ∇ ∈ B
d 2 nσ − 2 σ √ n ( 0 ), the second criterion will be true for all v , and so the events A v will occur independently with probability 1 n .
We use this to show that the probability that A v occurs for some v is greater than 12 :
[Equation]
Now, conditioning on A v , ∇ is distributed uniformly in B σ √ n ( x v ).
We next show that in this case, v will, with high probability, not be able to closely estimate ∇ .
Again we have:
[Equation]
We choose δ = σ √ n 4 2 − bd , and see that V ol ( OUT v ) <
2 − d V ol ( B σ √ n ).
Therefore, P I [ ∇ ∈
OUT v | A ] ≤ 2 − d , and so P I [ ∇ ∈ OUT v ] ≤ 12 .
When ∇ / ∈ OUT v , with probability at least 12 , P s [ k EST − µ k ≤ δ ] ≤ 12 .
So, we have that:
[Equation]
Then, E (cid:2) k EST − ∇ k 2 (cid:3) > 0 .
δ
=
Ω( σ 2 n 2 − 2 bd ).
Again, by Markov’s inequality, we can obtain that if machine v receives at most b bits in expectation , we have E (cid:2) k EST − ∇ k 2 (cid:3) =
Ω( σ 2 n 2 − 3 bd )
Theorems 26 and 8 imply that to obtain output variances of O ( y 2 q ) and O ( σ 2 q ) for MeanEstimation and VarianceReduction respectively, we require Ω( d log q ) communication bits per machine, thereby matching the upper bounds of Theorems 16 and 4 (the latter in expectation).
Theorem 7 further implies that to achieve any variance reduction at all (i.e. for output variance to be lower than input variance σ 2 ), at least one machine must receive Ω( d log n ) bits, matching the absolute upper bound on bits required to achieve the optimal output variance O ( σ 2 n ) by Theorem 3.
F Sublinear Communication
In this section we show an extension to the quantization scheme (and thereby also Algorithm 3) allowing us to use a sublinear (in d ) number of bits in expectation, providing a variance trade-oﬀ for the full range of communication for both MeanEstimation and VarianceReduction .
Here our methods apply only to ‘ 2 -norm.
Theorem 27.
For any q > 0 , MeanEstimation can be performed using O ( d log(1 + q )) communication bits per machine in expectation, with O ( y 2 q 2 ) output variance under ‘ 2 norm.
This communication expression matches the existing bounds of Theorem 16, since when q = Ω(1), it simpliﬁes to O ( d log q ). However, the method described in this section now works also for q = o (1); here, the expression simpliﬁes to O ( dq ) bits, which is sublinear in d .
We can further show good concentration on the amount of communication bits required, and extend to VarianceReduction as before.
Our sublinear-communication quantization method will involve mapping points to their closest lattice point (rather than randomly to a set of near by points who form a convex hull around them).
We will therefore be concerned with the (open) Voronoi regions of lattice points, since these are the regions that will be quantized to the same point:
Deﬁnition 28.
For any lattice Λ , the Voronoi region of a lattice point λ is the set of all x ∈ R d to whom λ is closer than any other lattice point, i.e., V or ( λ ) :
= { x ∈ R d : k x − λ k 2 < k x − λ
k 2 , ∀ λ 0 ∈ Λ \ { λ }} .
Voronoi regions of a lattice are known to have several useful properties, of which we will use the following:
Lemma 29.
The Voronoi regions of a lattice are:
open convex polytopes, 2.
symmetric, i e., λ + x ∈ V or ( λ ) ⇐⇒ λ − x ∈ V or ( λ ) , and 3.
identical,
i
e., λ + x ∈ V or ( λ ) ⇐⇒ λ 0 + x ∈ V or ( λ 0 ) for any λ 0 ∈ Λ .
Proof.
Property 1 follows since V or ( λ ) is the intersection of half-spaces formed by the set of points which are closer to λ than λ 0 , for any λ 0 ∈ Λ \ { λ }} .
Property 2 follows if there exists λ 0 ∈ Λ such that k λ + x − λ 0 k 2 ≤ k x [Equation] 2 , then ˆ λ = 2 λ − λis also a lattice point in Λ, and
[Equation]
Therefore λ + x / ∈ V or ( λ ) =
⇒ λ − x / ∈ V or ( λ ), which by contradiction proves the property.
Property 3 follows since the relative positions of all other lattice points are identical with respect to λ and λ 0 .
In particular, Property 3 implies that all Voronoi regions of a lattice have the same volume, which we denote V .
We note that, within any ﬁnite ball, the set of points which do not fall in a Voronoi region (i.e. have multiple equidistant closest lattice points) has measure(under the standard Lebesgue measure of R d ), and therefore do not aﬀect the probabilities of events under this measure.
We now wish to bound the number of Voronoi regions which are close to any point in R d , in expectation; these will be the regions of lattice points which are suﬃciently close to cause decoding errors if they are encoded with the same bit-string.
For this we introduce the concept of an expanded Voronoi region:
Deﬁnition 30.
The expanded Voronoi region V or + ( λ ) of a lattice point λ is the set
V or + ( λ ) of points within distance 2 q(cid:15) of V or ( λ ) .
We bound the volume of such sets:
Lemma 31.
For any λ in Λ (cid:15) , the expanded Voronoi region V or + ( λ ) has volume at most (1 + 2 q ) d V .
Proof.
Let (cid:91) V or ( λ ) be V or ( λ ) dilated by a factor of 1 + 2 q around λ ; that is:
[Equation]
or [Equation] Clearly the volume of (cid:91) V or ( λ ) is (1 + 2 q ) d V ; we now show that V or + ( λ ) ⊆
(cid:91) V or ( λ ).
V or ( λ ) is the intersection of open half-spaces bounded by the hyperplanes of points equidistant from λ and λ 0 , for any λ 0 ∈ Λ (cid:15) \{ λ } .
Dilation by a factor of 1+2 q therefore translates each of these hyperplanes by 2 q · λ 0 − λrespectively.
Since (cid:15) is the packing radius of Λ (cid:15) , k λ 0 − λ 2 k 2 ≥ (cid:15) , and therefore each such hyperplane is translated a distance of at least 2 q(cid:15) away from λ .
So, (cid:91) V or ( λ ) contains all points within distance 2 q(cid:15) of V or ( λ ), completing the proof.
We can now present the encoding algorithm
(Algorithm 7):
Algorithm 7 SublinearEncode , to compute Q 0 (cid:15),q ( x ) i ←loop
Let θ be a uniformly random vector in V or ( 0 ).
Let z be the closest lattice point in Λ (cid:15) to x + θ .
Let c 0 ∼ C be a random coloring Λ (cid:15) →
[1 + 2 q ] 3 d if there is no z 0 ∈ Λ (cid:15) with x + θ ∈ V or + ( z 0 ) and c 0 ( z ) =
c 0 ( z 0 ) then send c 0 ( z ) and i , terminate end if i ← i +end loop
The algorithm works as follows: we ﬁrst apply a random oﬀset vector θ , with the purpose of making the quantization unbiased.
We then round to the closest lattice point.
To convert this lattice point to a ﬁnite bit-string, we then apply a random coloring (from a distribution we will specify shortly); we show a lower bound on the probability that the color given to z is unique among those lattice points for whom x + θ falls in the expanded Voronoi region.
In this case, we send the color and the number i of the current iteration; if not, we repeat the whole process with fresh shared randomness.
We ﬁrst show a bound on the expected number of lattice points for which x + θ falls in the expanded Voronoi region:
Lemma 32.
The number V exp ( x + θ ) of expanded Voronoi regions containing x + θ is at most (1 + 2 q ) 2 d with probability at least 1 − (1 + 2 q ) − d .
Proof.
Since θ is uniformly distributed in V or ( 0 ), x + θ − z is uniformly distributed in
V or ( 0 ) (due to Voronoi regions being open, the probability distribution function diﬀers from uniformity, but only on a set of 0 measure).
Then, since Voronoi regions are identical (including in their intersections with expanded
Voronoi regions), and x + θ falls in exactly 1 Voronoi region in expectation, it falls in
V ol ( V or + ( 0 ))
V ol ( V or ( 0 )) ≤ (1+2 q ) d expanded Voronoi regions in expectation, by Lemma 31.
Therefore, by Markov’s inequality, the probability of falling within at least (1+2 q ) 2 d expanded Voronoi regions is at most (1 + 2 q ) − d .
We will need one further property of expanded Voronoi regions in order to show unbiasedness: that the number of expanded Voronoi regions containing a point is symmetric around any lattice point:
Lemma 33.
For any x ∈ R d , λ ∈ Λ (cid:15) , the points λ + x and λ − x are in the same number of expanded Voronoi regions V or + ( λ 0 ) .
Proof.
This follow from the symmetry of Λ (cid:15) with respect to λ : if λ + x ∈ V or + ( λ 0 ), then 2 λ − λ 0 is also a lattice point, and λ − x ∈ V or + (2 λ − λ 0 ).
We next deﬁne our distribution of colorings: we will ﬁrst apply the deterministic coloring c 3+2 q as described in Section A. Then, by Lemma 12, any two points of the same color are of distance at least 2(3 + 2 q ) (cid:15) apart.
Since V or ( λ ) ⊂ B r c ( λ ) ⊆
B 3 (cid:15) ( λ ), we have V or + ( λ ) ⊂
B (3+2 q ) (cid:15) ( λ ), and therefore there is no intersection between the expanded Voronoi regions of any points of the same color under c 3+2 q .
The purpose of this initial coloring is to limit the amount of randomness required for second random stage of the coloring process.
We then choose uniformly random colorings ˆ c : [3 + 2 q ] d → [1 + 2 q ] 3 d ; the ﬁnal coloring is then ˆ c ◦ c 3+2 q , and we deﬁne C to be the distribution of colorings generated in this way.
We will call such a coloring c 0 successful for a point x
+ θ if it meets the condition described in the algorithm, i.e., if there is no z 0 ∈ Λ (cid:15) with x + θ ∈ V or + ( z 0 ) and c 0 ( z ) = c 0 ( z 0 ).
Lemma 34.
Over choice of θ and c 0 , Pr [ c 0 is successful ] ≥ 1 − 2(1 + 2 q ) − d , and
E [ z − θ | c 0 is successful ]
= x .
Proof.
The probability of c 0 being successful is dependent entirely on the number V exp ( x + θ ) of expanded Voronoi regions containing x + θ .
By Lemma 32, this number is at most (1 + 2 q )
d
with probability at least 1 − (1 + 2 q ) − d .
In this case, the probability that z does not receive a unique color under c 0 is at most (1+2 q ) 2 d · (1+2 q ) − 3 d
=
(1+2 q ) 2 d · (1+2 q ) − d by a union bound.
Therefore, the total probability of c 0 not being successful is at most 2(1 + 2 q ) − d .
To show unbiasedness (that E [ z − θ | c 0 is successful] = x ), we note that x + θ − z is uniformly distributed in V or ( 0 ), and
therefore E [ z − θ ] = x ; by Lemma 33, V exp ( x + θ ), and therefore the probability of successful coloring, is symmetric around any lattice point, and so conditioning on successful coloring preserves unbiasedness.
Once we have a coloring in which there is no z 0 ∈ Λ (cid:15) with x + θ ∈ V or + ( z 0 ) and c 0 ( z ) =
c 0 ( z 0 ), the following simple decoding procedure can ﬁnd z so long as quantization input x and decoding vector x v are suﬃciently close:
Algorithm 8 SublinearDecode , to compute R 0 (cid:15),q ( Q 0 (cid:15),q ( x ) , x v )
Using i and shared randomness, reconstruct θ and c
Let ˆ z ∈ Λ (cid:15) such that B q(cid:15) ( x v + θ ) intersects V or ( ˆ z ), and c 0 ( ˆ z ) matches Q 0 (cid:15),q ( x )
Output ˆ z − θ
Lemma 35.
If k x − x v k 2 ≤ q(cid:15) , then the decoding procedure R 0 (cid:15),q ( Q 0 (cid:15),q ( x ) , x v ) correctly returns the vector z − θ which is an unbiased estimator of x .
Proof.
Clearly z is has the received color; we must show that B q(cid:15) ( x v + θ ) intersects V or ( z ), and does not intersect V or ( z 0 ) for any other z 0 with the same color.
The former is the case since x + θ ∈ V or ( z ) and k ( x + θ ) − ( x v + θ ) k ≤ q(cid:15) .
The latter holds since if B q(cid:15) ( x v + θ ) intersects V or ( z 0 ), then so does B 2 q(cid:15) ( x + θ ), and so x + θ ∈ V or + ( z 0 ).
However, the coloring was successful, so there is no z 0 6 =
z for which this is the case.
So, Algorithmmust successfully decode z .
Theorem 36.
Algorithms 7 and 8 give a quantization procedure with the following properties: If k x − x v k 2 ≤ q(cid:15) , then the decoder outputs an unbiased estimator ˆ z − θ of x with k x − ( ˆ z − θ ) k 2 ≤ 3 (cid:15) .
Each machine communicates O ( b ) bits in expectation (and with probability at least 1 − 2 (1 − b )2 b ), for b =
d log(1 + q ) .
Proof.
It remains only to prove a bound on the number of bits transmitted: transmitting c
( z ) requires log (cid:0) (1 + 2 q ) 3 d (cid:1)
=
O ( b ) bits.
To bound the number of encoding iterations required (and therefore the size of i ), we see that each iteration independently succeeds with probability at least 1 − 2(1 + 2 q ) − d ≥ 1 − 2 1 − b , by Lemma 34.
Then, for any j ∈ N ,
Pr (cid:2) i > 2 j (cid:3) ≤ 2 (1 − b )2 j .
Transmitting a value of i ≤ 2 j requires j bits, and therefore, setting j = b , we see that transmitting i uses b bits with probability at least 1 − 2 (1 − b )2 b , and in this case the total amount of communication used is O ( b ).
Note that we use Algorithms 7 and 8 primarily for the sublinear communication regime, i.e., when q < 1, and in this case we have b =
d log(1 + q ) =
Θ( dq ).
F. MeanEstimation and VarianceReduction with Sublinear Communication
If we wish to use o ( d )-bit messages, then by Theorems 26 and 8, we cannot achieve lower output variance than input variance.
Therefore, there is no longer any beneﬁt to averaging; we can instead simply choose a random machine to broadcast its quantized input to all other machines. Algorithm
Choose a source machine u uniformly at random
Machine u broadcasts Q 0 yq ,q ( x u ) to all other machines
Each machine v outputs R 0 y q ,q ( Q 0 yq ,q ( x u ) , x v )
Theorem 37.
For any q = O (1) , Algorithm 9 performs MeanEstimation with each machine using O ( dq ) communication bits in expectation (and with probability at least 1 − 2 (1 − dq )2 Θ( dq ) ), and with O ( y 2 q 2 ) output variance under ‘ 2 norm.
Proof.
The input of the chosen source x u is an unbiased estimator of the mean input µ with, k x u − µ k 2 ≤
y .
Since all machines have k x u − x v k 2 ≤ yq · q = y , all machines correctly decode a common unbiased estimate z of x u with k z − x u k 2 ≤ 3 yq .
For q = O (1), the output is therefore an unbiased estimate of µ with O ( y 2 q 2 ) variance.
If the broadcast is performed using a binary tree communication structure, then each machine receives at most one message and sends at most two.
We have b =
d log(1+ q ) =
Θ( dq ), and therefore each machine requires O ( dq ) communication bits, in expectation and indeed with probability at least 1 − 2 (1 − dq )2 Θ( dq ) .
Together with Theorem 16, this implies Theorem 27.
We again apply the same reduction as for Theorem 17 to obtain the following sublinearcommunication result for VarianceReduction (though the name is less ﬁtting, since variance is now necessarily increased ):
Theorem 38.
For any α > 1 , and any q = O (1) , Algorithm 9 performs VarianceReduction using O ( dq ) communication bits per machine (with probability at least 1 − 2 (1 − dq )2 Θ( dq ) ), with O ( αnσ 2 q 2 ) output variance under ‘ 2 norm, succeeding with probability at least 1 − 1 α .
Proof.
As in proof of Theorem 17, we have that a MeanEstimation algorithm using y = 2 σ √ αn performs VarianceReduction , succeeding with probability at least 1 − 1 α , which we plug into Theorem 37.
G Complete Experimental Report
We consider three distinct applications: data-parallel SGD, distributed power iteration for eigenvalue computation (Suresh et al., 2017), and adding communication compression to local SGD (Stich, 2018).
We implement the practical version of our algorithm (Section G.1), which for simplicity we call LQSGD, using a cubic lattice and ‘ ∞ to measure input variance, and a version we call RLQSGD with the Hadamard rotation to ensure close-to-optimal variance.
G. The Algorithm in Practice
Practically, the LQSGD algorithm with two machines ( u and v ) works as follows:
we ﬁrst oﬀset the cubic lattice by a uniformly random vector in [ − s 2 , s 2 ] d , using shared randomness.
This ensures that quantizing to the nearest lattice point now gives an unbiased estimator, avoiding the extra cost of the convex-hull method (though that can still be employed practically if shared randomness is unavailable, and for the cubic lattice is a simple coordinate-wise rounding procedure).
Machine u rounds to its closest lattice point and sends its mod- q color to machine v , who decodes it by ﬁnding its closest lattice point with the same color, and vice versa.
Finding the closest lattice point under the cubic lattice can be performed coordinatewise using ˜ O
( d ) computation, and gives a lattice point within s 2 ‘ ∞ norm distance.
Mod- q coloring can also be eﬃciently computed, and has the property that any two lattice points λ , λ 0 with the same coloring have k λ − λ 0 k ∞ ≥ qs .
Therefore, if the input gradients g 0 , ghave k g 0 − g 1 k ∞ ≤ ( q − 1) s, then decoding is successful.
So, assuming we have an estimate y such that for all g 0 , g 1 we have k g 0 − g 1 k ∞ ≤ y , we set our side-length s = 2 y q − 1 .
We also implement the algorithm using the structured random rotation described in Appendix D, which we call RLQSGD.
Here, we also generate the matrix D on machines using shared randomness, and then apply the transformation HD to inputs before quantization (and the inverse transform after decoding).
The algorithm then proceeds exactly as LQSGD; when setting lattice side length we use an estimate y R of ‘ ∞ -norm distance after applying HD , i.e., || HD ( g 0 − g 1 ) || ∞
≤ y R .
We will describe in each experiment how we set and update our estimate of y ; generally, this can be done by simply measuring ‘ ∞ norms between inputs during the course of the algorithm, and multiplying these by a small constant factor (1 . 5 to 3 . 5) to ensure suﬃcient slack that all decodes succeed.
G. Least-Squares Regression
The classic problem of least-squares is as follows: given as input some matrix A and target vector b , our goal is to ﬁnd w ∗ =
argmin w ||
A w − b
||
22 , i.e. the vector which, upon multiplication by A , minimizes Euclidean distance to b .
To obtain instances of the problem in order to test our approach, we generate w ∗ ∈ R d and entries of A ∈ R S × d by sampling from N (0 , 1), and we set b =
A w ∗ .
We then run distributed gradient descent using our quantization scheme, with the following settings:
S = 8192 samples in d = 100 dimensions 2.
n = 2 worker machines 3.
results are averaged over 5 random seeds [0, 10, 20, 30, 40]
In all experiments, in each iteration of gradient descent, the dataset (i.e., the rows of matrix
A ) will be randomly divided into two equal S 2 -sized groups and provided to the two machines.
We denote S 2 -batch gradients by g 0 , g 1 respectively.
We use large batch sizes in order to be in a regime where our algorithm’s assumptions can give a visible advantage over other bounded norm based approaches.
Experiment 1: Norms relevant to quantization schemes.
Our ﬁrst experiment is to provide justiﬁcation of our main message, that input variance (or similar measure of mutual distance depending on the setting) is often far lower than input norm in practical applications, and is a more natural quantity with which to measure output variance of algorithms.
In Figures 4 and 5 we compare four quantities, in the setting of least-squares on two worker machines:
• batch gradient distance k g 0 − g 1 k 2 (the closest proxy of our theoretical quantity y from MeanEstimation ), • k g 0 − g 1 k ∞ (the equivalent quantity under ‘ ∞ norm, which is the appropriate norm when using the cubic lattice), • k g 0 k 2 , the batch gradient norm, (which is used as the measure of input size in {+QSGD-L
+} (Alistarh et al., 2017), as well as most other prior work) • max( g 0 )
− min( g 0 ), the batch gradient coordinate diﬀerence, used as the measure of input size in the QSGD implementation (Alistarh et al., 2017)
We see that the former two quantities, used by our algorithms, are far lower than the latter two, i [Figure] e., input variance/distance is signiﬁcantly lower than input norm
[Figure]
and batch gradients are not centered around the origin.
Similar eﬀects are seen in a wide variety of applications and parameter settings [Figure] and justify our focus on input distances over norms.
Iterations in the ﬁgures are under gradient descent performed using full (unquantized) gradient. [Figure] [Figure]
Experiment 2: Variances using quantization methods.
We now apply quantization schemes to the communication gradients, and measure the variances induced, in order see whether the input norm versus variance diﬀerence demonstrated by Figures 4 andtranslates to lower output variance for LQSGD over other schemes. In this experiment
[Figure]
we perform distributed stochastic gradient descent, by communicating, averaging, and applying the quantized batch gradients g 0 and g 1 of the two worker machines in each iteration.
By construction, this resulting value EST will be an unbiased estimator of the full true gradient ∇ , and we will examine the associated output variance E [ k EST − ∇ k 22 ], for diﬀerent quantization schemes.
We experiment with q = 8 for all methods (for QSGD the equivalent quantity is referred to as qlevel), which means that messages comprise of log 2 8 = 3 bits per coordinate.
In norm-based methods like QSGD, there is an additional communication overhead of one/two 64 − bit ﬂoating point values for every quantization, used to convey input norms.
To allow machines using LQSGD to maintain an estimate y of distance between inputs, in the ﬁrst iteration alone a pre-computed estimate is provided to both machines.
(To ensure that this does not convey an unfair advantage, we allow the other methods to update using the full true gradient ∇ in the ﬁrst iteration.)
Henceforth, machines dynamically adjust y , to ensure that we always have k g 0 − g 1 k ∞ <
y , in order for decoding to be successful.
For each iteration t , denoting as Q ( g 0 ) , Q ( g 1 ) the encoded (i.e., rounded to closest lattice point) estimates, machines use the value y ( t + 1) = 1 . 5 · k Q ( g 0 ) − Q ( g 1 ) k ∞ for the next iteration.
Similarly, for RLQSGD, y R ( t + 1) = 1 . 5 · k HD ( Q ( g 0 )
− Q ( g 1 ))
k ∞ .
This value is common knowledge to both machines as long as decoding in iteration t is successful, and suﬃced, in this experiment, to ensure that all decodes succeeded.
Our baseline is naive averaging (i.e., what would be attained if we could communicate at full precision between the machines), and we compare against variants of QSGD (Alistarh et al., 2017), and the Hadamard -based quantization scheme of Suresh et al.
Figures 6 and 7 demonstrate that LQSGD provides the lowest output variance among the methods, and is the only method to achieve variance reduction , i.e. the output variance of the average of the quantized batch gradients is lower than the input variance of the individual batch gradients. [Figure] [Figure]
Experiment 3: Convergence using quantization methods.
We next measure the eﬀect on the convergence of the SGD process of the quantization schemes, using the same input data.
That is, machines now apply the unbiased estimate of ∇ they obtain by averaging quantized batch gradient in each iteration.
To clearly show the eﬀects of quantization on convergence, we will use a high learning rate of [Figure]
Estimation of y [Figure] and other parameter settings, remain the same as previously.
Figures 8 and 9 demonstrate faster convergence for
LQSGD over other methods
[Figure]
[Figure]
We also wish to test the performance of our sublinear quantization scheme from Section
F. Unfortunately, a naive implementation of the scheme (particularly, of ﬁnding the closest lattice point of a particular color under the chosen random coloring) is not computationally feasible under high enough dimension for meaningful results.
Therefore, here we will only simulate what the eﬀect of the quantization would be on output variance compared to state-of-the-art schemes; in future work we will explore how the scheme can be adapted for eﬃcient implementation.
In this experiment we again have two worker machines u, v , and we compare the variance induced when u sends its quantized batch gradient g 0 to v , who decodes it using g 1 .
The sublinear-communication quantization method that we compare with is the vQSGD cross polytope method with repetition (Gandikota et al., 2019).
We will perform our experiments using d 2 bits, i.e., 0.5 bits per coordinate (and so we set the number of vQSGD repetitions accordingly).
We use a slightly diﬀerent means of dynamically updating y , since variance from quantization means that the previous method no longer gives good estimates.
Now, once in 5 iterations, machine u receives 2 batches, which allows it to compute two gradient estimates g 0 and g 0 0 , and compute y = 1 . 6 · || g 0 − g 1 || ∞ (which suﬃces to ensure correct decoding in this experiment).
It then sends this to v as a 64-bit ﬂoating point value.
Note that this method of updating y generalizes to many machines, and that the constant factor 1 . 6 can be changed as necessary.
We use the cubic lattice, and as before denote its side length by s .
Given k g 0 − g 1 k ∞ <
y .
We can see from the analysis of the sublinear scheme (in particular, in proof of Lemma 32) that if we choose a number of colors (i.e., bit-strings) equal to the expected number of expanded Voronoi regions covering a point, quantization will succeed with at least constant probability.
This gives an expression of log 2 (1 + 2 y/r p )
d
= d log 2 (1 + 4 y/s ) bits.
To use 0 .
5 d bits, we set log 2 (1 + 4 y/s )
= 0 .
5, from which we get s = 4 y/ ( √ 2 − 1) (though our number of bits and therefore value of s slightly diﬀers from this in order to exactly match the communication used by vQSGD, where number of repetitions must be an integer).
Since the randomization in the quantization eﬀectively shifts each coordinate independently in [ − s/ 2 , s/ 2], we get variance ds 2 / 12, which is what we plot in the ﬁgures.
Figures 10 and 11 demonstrate that sublinear LQSGD could be competitive with state-ofthe-art methods, though it only outperforms when using large numbers of samples with respect to dimension.
The steps in the graph are due to the periodic updates to y .
[Figure] [Figure]
Experiment 5: Convergence on real dataset using quantization schemes with more than 2 machines.
We now test the performance of the (linear/superlinear) quantization scheme in a more realistic setting, on a real dataset using multiple machines.
We use the dataset cpusmall_scale from the LIBSVM datasets (Chang & Lin, 2011), which has
S = 8192 and d = 12.
We initialize the initial weight to a point far from the origin relative to w opt , speciﬁcally the vector of − 1000 in each coordinate.
This is to synthetically study the behavior one can expect on a general convex loss function when w opt is arbitrary.
We study the convergence behavior with q = 16, n = 8 , 16 and batch size =
Sn .
We apply Algorithm 3, with one randomly chosen machine acting as leader to collect all the quantized batch gradients Q ( g i ), and broadcast the average as a quantized vector i.e
Q ( P i Q ( g i ) /n ).
This machine also uses the quantized gradients Q ( g i ) to evaluate and broadcast y for the next round as a 64-bit ﬂoat using y = 3 · max i,j || Q ( g i ) − Q ( g j ) || ∞ , where the factor 3 is again chosen large enough to ensure that decoding is always successful in this experiment (and can be altered as necessary in other settings).
Figures 12 and 13 exhibit fast convergence compared to other methods. [Figure] [Figure]
G. Neural Network Training
Our next task is applying LQSGD for training neural networks in a data-parallel environment, where we use it to average gradients.
We note that this application slightly extends the setting for distributed mean estimation (Suresh et al., 2017), since other methods can, e.g., employ historical information from previous iterations (Stich et al., 2018; Karimireddy et al., 2019).
Experiment 6: Gradient compression for neural network training.
We train
ResNets (He et al., 2016) on a subset of 10 classes from the ILSVRC dataset (Deng et al., 2009), with full-sized images, as well as on the CIFAR dataset (Krizhevsky).
The former dataset is popular for model and hyperparameter calibration (Howard) relative to the full version of the dataset, and model performance on it is known to correlate well with performance on the full ILSVRC dataset.
The latter is a classic small dataset in computer vision.
The results in Figure 14 show the Top-1 validation accuracy on ResNet18 (11M parameters) for LQSGD with an average of 4 bits per entry, versus 4-bit QSGD (L2 and LInf normalized variants), PowerSGD (Vogels et al., 2019) with rank 16 (as suggested by the authors), and 1bitSGD/EFSignSGD (Seide et al., 2014a;
Karimireddy et al., 2019).
For LQSGD, each node uses one batch per epoch to estimate σ , and uses y = 3 σ as its upper bound estimate.
(This results in a rate of incorrect decodings of ∼ 3%, which we allow, but which has no impact on convergence.)
Results are averaged over 2 runs, since variance is small.
We note that all compression algorithms lose accuracy relative to the full-precision baseline.
(The large gap for EFSignSGD is reasonable, since it uses the fewest bits.)
Perhaps surprisingly, LQSGD loses the least accuracy relative to the baseline, although the diﬀerence relative to QSGD is small.
We conclude that, although gradient compression is not the main application for lattice-based quantization, it can yield competitive results for this task.
Figure 15 presents additional accuracy results for neural network training with compressed gradients, training ResNet20 on the CIFAR-100 dataset (Krizhevsky; Krizhevsky et al., 2014).
The network is trained for 200 epochs, with standard hyper-parameters.
Quantization is applied at the level of each layer, and we use 4 bits per coordinate (except for
EF-SignSGD which uses approximately one bit per component).
The results conﬁrm that
LQSGD can be competitive with state-of-the-art quantization methods.
G. Power Iteration
Power iteration is an algorithm for ﬁnding the principal eigenvector of an input matrix X .
In a distributed setting over two machines, the algorithm works as follows: the rows of our input matrix X are partitioned into two subsets X 0 , X 1 .
At the beginning of each round, both machines have the same unit vector x , which is the current estimate of the principal [Table] [Figure] eigenvector of X .
During the round, they must perform the update x ← X T X x
|| X
T X x
|| .
For this, machine i evaluates u i =
X T
i X i x and shares it with the other machine; both machines can then calculate X T X x = X T
X 0 x + X T 1 X 1 x =
u 0 + u 1 , and thereby perform the update step.
We will apply quantization methods to communicate these vectors u i , in order to test the performance of LQSGD in this setting.
Experiment 7: Quantization for compression during power iteration.
Each row of the input matrix X is generated from a multivariate gaussian with ﬁrst two eigenvalues large and comparable, which is to ensure that power iteration doesn’t converge too fast that we cannot observe the eﬀect of quantization.
The estimate x of the principal eigenvector is initialized to a random unit vector.
We use S = 8192 samples, dimension d = 128, q =i.e., 6 bits per coordinate.
From Figure 16, 17 one can see that the relevant norms ﬁt with our general message concerning norms: distance between quantization inputs (used in LQSGD and RLQSGD) is substantially lower than diﬀerence between coordinate values within a single quantization input (used as a surrogate for input norm in QSGD).
Also, we see that these norms settle quickly (relative to the number of iterations one would run to get a good estimate of the principal eigenvector) and are then near-constant.
Hence, one can run the baseline algorithm (either on a single machine, or assuming full-precision communication) for a few iterations until || u 0 − u 1 || ∞ stabilizes, and then set the value of y for LQSGD to 2 · max || u 0 − u 1 || ∞ (where the maximum is over all iterations currently seen).
Similarly, for RLQSGD we set y R = 2 · max || HD ( u 0 − u 1 ) || ∞ .
We then run LQSGD and RLQSGD from iteration 0, but using the computed value of y .
Upon doing so, Figure 16, 17 demonstrates substantially better estimation of the principal eigenvector for RLQSGD and LQSGD compared to other quantization methods.
[Figure]
Principal eigenvector is e 2 .
[Figure]