Paper: LASER: Linear Compression in Wireless Distributed Optimization (ICLR-2024)

Reviewer comment (Novelty):
Looks like LASER is a combination of well-known methods (low-rank approximation and noise variance minimization), so is there any major difference or difficulty in theoretical analysis besides the ones already sound in literature?

Author response:
We thank the reviewer for the constructive feedback and insightful comments. We refer to the common response about the  theoretical and  algorithmic novelty of LASER. 

- **Theoretical analysis:** As highlighted in the common response, a major difficulty in the theoretical analysis of LASER is handling the communication noise and the power budget. Since the channel noise interacts in a non-linear way with the gradients (Eq. (noisy channel)), we tackle this through introducing **channel-influence** factor which gives a handle to control the second-moments of the noisy gradients, which are crucial for convergence analysis. On the power front, Lemmas 7 and 8 in Appendix B.1 establish the optimal power allocation scheme among the rank components in order to obtain a tight characterization of the channel influence for various compression schemes. This further facilitates a thorough comparison with baselines such as Z-SGD and showcases the benefits of compression algorithms through convergence rates.

Reviewer comment (Theory):
Could you provide intuitive understanding about why low-rank approximation can induce lower noise variance? Specifically, explain the intuition on the comparison of noise variance in equation (5).

Author response:
We thank the reviewer for the constructive feedback and insightful comments. We refer to the common response about the  theoretical and  algorithmic novelty of LASER. 

- **Intuition behind Eq. (5):** As demonstrated in Section 3 of the paper, the underlying intuition here is that under a fixed power budget, if we transmit the full gradient matrix of size $m \times m$, this budget has to be utilized for $m^2$ entries. On the other hand, if we capitalize on its low-rank structure we effectively need to transmit $O(m)$ entries, thus we can allocate more power-per-entry hence resulting in a roughly $O(m)$ boost in the signal-to-noise ratio (SNR). This intuition mathematically translates into the reduction of the gradient variance, as captured by the channel-influence factor in Eq. (4). In view of this, Eq. (5) compares these noise variances for Z-SGD and LASER and confirms the above intuition.

Reviewer comment (Experiments):
The low-rank compressor seems to be time-consuming. Why not use top-k or other compressors?

Author response:
- **Processing time for PowerSGD and baselines:** As highlighted in Section 4.3 of the paper, PowerSGD [3], the low-rank compressor behind LASER, takes significantly smaller processing time as compared to other compressors such as Top-K or Random-K. For example, Table 4 in [3] highlights that PowerSGD achieves a significant 42 % reduction in processing time/batch as compared to Top-K, whereas it’s 55 % compared to Random-K for CIFAR-10. Similar computational gains of PowerSGD over the baselines are demonstrated further in [3] over various challenging datasets.

Reviewer comment (Evaluation):
It seems to be unfair to compare the energy budget in each iteration, because adding a compressor even with the error feedback, the algorithm needs more iterations to converge. Thus, it would be fair to compare total energy costs. Otherwise, the most energy-saving algorithm in this framework will be the top-1 compressor (sending only one value each time).

Author response:
- **Energy budget:** Indeed, our current comparison already considers the total energy costs in an implicit manner. Since we fix the power budget per each iteration across various baselines and compare the final accuracy after a specific number of epochs, this is equivalent to the former scenario. We observe similar gains for LASER even when the final number of epochs is increased further.

Reviewer comment (Theory):
In algorithm 1, the local error $e_i$ is only $\boldsymbol{M}_i - \text{DECOMPRESS}(\mathcal{C}_r(\boldsymbol{M}_i))$, without ever being multiplied by $\gamma$: is this correct? It seems that if so, the error may not be compensated correctly? (I may be mistaken)

Author response:
**Local error:** Following the standard convention in [1], in Line 7 of Algorithm 1 we add the local error $ \boldsymbol{e}_i $ to the scaled gradient $\gamma  \boldsymbol{g}_i $  to get the new gradient $  \boldsymbol{M}_i $. In Line 13, after its low-rank reconstruction through $ \boldsymbol{g}$, a normal SGD step is performed on $ \boldsymbol{\theta}$. We can see that when there is no compression error, $ \boldsymbol{e}_i =0 $, this falls back to the classical SGD. As suggested, one could in principle multiply the learning rate $\gamma$ to the local-error but this requires a slight change in the definition of local error and the SGD descent step, so that we could recover the classical SGD in the special-case as above.

Reviewer comment (Theory):
If I understand correctly, Assumption 5 is a new assumption introduced in the paper: unless I missed it in the paper, I think it would be good to elaborate a bit further on why such assumption should be verified in the settings considered, theoretically and/or experimentally (or even just with a discussion, just to provide some intuition on why such assumption should be verified). It seems that to verify it, either the compression should be very accurate (i.e. $\delta_r$ large), or $\lambda_{\text{LASER}}$ should be small. However, while taking a larger $r$ would make $\delta_r$ larger (more accurate compression), it would also make the bound on $\lambda_{\text{LASER}}$ larger, according to eq. (5). Therefore it seems a bit unclear whether Assumption 5 can be verified (or how to make it verified in practice).

Author response:
Regarding Assumption 5, had the compression $\mathcal{C}_r$ been an exact rank-$r$ compression, $\delta_r$ would have been $r/d$ as you noted. However, since we use PowerSGD for this rank-$r$ compression, it's only an approximation for the same and given the fact it uses power iteration based algorithm to compute the low-rank factors, it's a bit hard to theoretically characterize $\delta_r$. Nonetheless, as you have suggested, experimentally computing this quantity and verifying it in the context of DNNs is indeed a cool idea and worth exploring further.

Reviewer comment (Experiments):
For the experiments, I believe it would be useful to just recall how one can deal with parameters of DNNs, which are not, per se, 1 matrix. Looking into Vogels 2019, it seems that, to do that, the low-rank decompositions are done layer-wise (since each layer can be seen as a matrix, even in the convolutional case), but just a quick recall about this would be useful.

Author response:
**Layer-wise decomposition:** We fully agree with your comment. Indeed, we already specify this feature just below Algorithm 1 on page 4 in the paper, but we will further make it clear.

Reviewer comment (Theory):
In what scenario would the uplink communication be noisy but the downlink communication would be noiseless? "For the downlink communication from the server to the clients (broadcast channel), we assume that it is noiseless and thus the clients exactly receive what the server transmits..."

Author response:
- **Uplink and downlink noise:** In the uplink channel the resource-constrained workers communicate with the server whereas in the downlink communication the server to the clients. The downlink is a broadcast channel which is typically assumed in the literature to be noiseless [1-3]. The underlying intuition here is that the central server (e.g. base station) is not as resource/power-constrained as the individual clients (e.g. mobile phones) and hence it can ensure its transmission error-free relatively speaking. While few works [8, 9] study the impact of downlink noise on the convergence of Z-SGD, it’s an interesting future direction to extend these results for LASER-like compression algorithms.

Reviewer comment (Reproducibility):
(Chang & Tandon, 2020; Guo et al., 2020) imply that clients must first transmit information regarding the norms of their gradients before constructing a power budget policy -- does this imply 2 rounds of communication would be necessary assuming a dynamic power schedule is employed?

Author response:
- **Transmission of power scalars:** Yes, indeed. As highlighted in [4, 5], the nodes transmit only the gradient norms $\|\| \boldsymbol{g}_i \|\|$, which are scalars, to the server which then computes their maximum and transmits it back. Since this procedure only involves transmission of real scalars, it has negligible communication overhead as compared to the gradients being transmitted.