Paper: On the Effect of Batch Size in Byzantine-Robust Distributed Learning (ICLR-2024)

Reviewer comment (Theory):
There exist little theoretical improvements regarding to existing BRDL methods in terms of problem assumptions or convergence rates. In addition, this work only considers i.i.d. cases, which is kind of restrictive if not enough theoretical improvements are obtained.

Author response:
We would like to clarify that the main theoretical contribution of our work lies in the analysis of the effect of batch size in BRDL. Specifically, as far as we know, we are the first to show that a relatively large batch size is preferred in BRDL (please see our response to Weakness 2(a)).

In addition, we would like to point out that although in the same order, the convergence rate in our work is tighter than those in existing works for SGD-based methods in BRDL, as far as we know. Moreover, it has been shown in (Arjevani et al., 2019) that under Assumptions 1, 2, and 3 (please see Section 2), the convergence order $\mathbb{E}||\nabla F(w_T)||\leq O(1/T^\frac{1}{4})$ is optimal for SGD in the worst case. Therefore, we carefully think that the improvement by optimizing batch size in our work is meaningful.

Reviewer comment (Novelty):
The variance reduction trick using large batch size is a direct consequence of (Karimireddy et al., 2021)), so it is hard to claim that this is one of the main contributions of the current work. Furthermore, the optimization of B is conducted on the upper bounds of the convergence rates, so it does not necessarily leads to faster convergence if we set optimal B. This probably should be made clear in the paper.

Author response:
We thank the reviewer for the comment. Actually, we would like to politely clarify that the results of (Karimireddy et al., 2021) do not suggest using large batch sizes. Although increasing batch size has the positive effect of reducing the variance of each iteration, it also has the negative effect of increasing the gradient computation number for each iteration. Therefore, for a given total gradient computation number $\mathcal{C}$, it is uncertain whether a large or small batch size is better in BRDL based only on the results (Karimireddy et al., 2021) without further analysis. 

In traditional distributed learning (TDL) without attacks, the variance of stochastic gradients will also slow down the convergence. However, relatively small batch size is still preferred in TDL since it allows more iterations and leads to better empirical performance when $\mathcal{C}$ is fixed (or equivalently, when the total epoch number is fixed). As far as we know, almost all existing works on BRDL  directly follow the setting of small batch sizes in TDL. However, our theoretical results show that a relatively large batch size is preferred in BRDL, which is contrary to that in TDL without attacks. Moreover, our theoretical findings are supported by empirical results.

In summary, in this work, we have pointed out the improperly inherited setting of small batch size in BRDL by both theoretical analysis and empirical results. We would greatly appreciate it if the reviewer could re-evaluate the contribution of our work in light of our response above.  

We thank the reviewer for the insightful comment. We have added the following statement at the end of Section 3.1, which is marked in blue.

"In addition, we would like to clarify that although we obtain a better worst-case guarantee by optimizing $\mathcal{U}(B)$, it does not necessarily ensure a better empirical performance given the complexity and variety in real-world applications."

Reviewer comment (Novelty):
The technical elements in proving the convergence of ByzSGDnm are closely related to references such as (Cutkosky & Mehta, 2020), so I am not clear whether there are substantial contributions therein.

Author response:
We would like to point out that the theoretical results in our work cannot be obtained by simply combining the proof in (Cutkosky & Mehta, 2020) and (Karimireddy et al., 2021). Specifically, the term $(\frac{1}{\alpha T}+\sqrt{\alpha})$ in Theorem 2 cannot be obtained by directly following the proof in (Karimireddy et al., 2021). To overcome the problem, we provide a finer analysis for $\mathbb{E}||u_t-\bar{u}_t||^2$ in Lemma 2 (equation (29)). The term $[\alpha + (1-\alpha)^{2t}]$ is originally $[\alpha + (1-\alpha)^t]$ in Lemma 9 in Appendix E of (Karimireddy et al., 2021).

Reviewer comment (Writing):
SGDm is not defined before first used.

Author response:
We thank the reviewer for pointing out the undefined abbreviation. It has been modified to 'stochastic gradient descent with momentum (SGDm)' in the revised version.

Reviewer comment (Theory):
In proposition 2, equation (5), why the last term has a $\sigma^2$ term, while in the work (Cutkosky & Mehta, 2020), this term is only in $\sigma$. Can you briefly explain why?

Author response:
After checking our proof and the proof in (Cutkosky & Mehta, 2020), we carefully guess that there might be a typo in the proof of (Cutkosky & Mehta, 2020). It seems that the term $\frac{8\sigma\sqrt{T}}{\sqrt{RL}}=O(\sigma)$ in the last step of the proof of Theorem 1 in (Cutkosky & Mehta, 2020) should be $\frac{8\sigma^2\sqrt{T}}{\sqrt{RL}}=O(\sigma^2)$ since it is obtained by letting $\alpha=\frac{\sqrt{RL}}{\sigma\sqrt{T}}$ in the term $\frac{8\sigma}{\alpha}$.

Meanwhile, we are willing to hear from the reviewer if they have any further comments about this question.

Reviewer comment (Theory):
Can you explain why KR fails all cases? Is that because KR does not satisfy the definition 1?

Author response:
We would like to point out that in the experiments of previous works (Karimireddy et al., 2021; Xie et al., 2020), KR also failed under FoE attack in all cases. To the best of our knowledge, it is unknown whether KR satisfies Definition 1 or not. Although we do not focus on the performance of a specific aggregator under a specific attack in this work, we are willing to provide some comments on this phenomenon below. 

Firstly, the theoretical guarantee for Krum (KR) in (Blanchard et al., 2017) requires that $2f+2<n$ where $f$ is the number of Byzantine workers and $n$ is the number of all workers. The condition is not satisfied in our experiment where $3$ workers among $8$ workers are Byzantine. In addition, we find that when using KR under FoE attack, the model parameter falls into an area with a small gradient norm but a large training loss. Since theoretical results only guarantee a small gradient norm in expectation, the poor empirical performance of KR does not conflict with the theoretical results.

Reviewer comment (Evaluation):
Why does CM have degraded performance in table 4 after increasing batch size, this batch size should be right based on its performance in ByzSGDnm, can you provide some comments on that?

Author response:
As the detailed results in Appendix E.1 show, [ByzSGDm + CM] has the best performance when $B=128\times 8$ in this case. When $B<128\times8$, the performance of [ByzSGDm + CM] improves as $B$ increases. When $B>128\times8$, the performance of [ByzSGDm + CM] degrades as $B$ increases. It is consistent with our theoretical results and other empirical results.

For quick access, we present part of the results in Table 10 in the revised version below, which show the top-$1$ test accuracy of different methods when there are $3$ workers under FoE attack.

|Batch size|32$\times$8 | 64$\times$8 | 128$\times$8 | 256$\times$8 | 512$\times$8| 1024$\times$8|
|----|----|----|----|----|----|----|
|ByzSGDm + GM | 78.36\% | 81.98\% | 82.69\% | 82.20\% | **84.09\%** | 78.90\% |
|ByzSGDnm + GM | 88.55\% | 88.75\% | **90.99\%** | 90.23\% | 89.12\% | 88.38\% |
|ByzSGDm + CM | 83.97\% | **84.28\%** | 84.01\% | 83.48\% | 79.16\% | 78.76\% |
|ByzSGDnm + CM | 84.12\% | 84.77\% | 85.23\% | **85.74\%** | 84.65\% | 83.36\% |
|ByzSGDm + CC | 83.60\% | 84.26\% | 87.45\% | **88.48\%** | 86.24\% | 81.36\% |
|ByzSGDnm + CC | 88.99\% | 90.07\% | **90.69\%** | 90.54\% | 89.32\% | 88.20\% |

Reviewer comment (Experiments):
In the comparison with Byz-VR-MARINA, why do you use $512\times 8$ batch size for ByzSGDnm, but never uses that for Byz-VR-MARINA?

Author response:
This is mainly because Byz-VR-MARINA adopts the PAGE style, which prefers a relatively small batch size (Li et al., 2021, Gorbunov et al., 2023). Meanwhile, we have added the empirical results of Byz-VR-MARINA when $B=256\times 8$ and $B=512\times 8$ in Appendix D.5 in the revised version. The empirical results show that the performance of Byz-VR-MARINA does not improve when the batch size increases to $256\times 8$ or $512\times 8$.

Reviewer comment (Experiments):
In the experimental part of the work, the experiments do not support theoretical analysis. The authors do not compare the performance of the method with optimal batch size value and with another possible choice of it. It would be better if such experiments were in the work.

Author response:
Since the constants $L$ and $F_0$ for the ResNet-20 deep learning model on the CIFAR-10 dataset is hard to obtain, we try different batch sizes ranging from $2^5\times 8$ to $2^{10}\times 8$ in the experiment. As the empirical results in Table 1 and Table 2 show, for ByzSGDm with each of the four aggregators (KR, GM, CM, and CC), the optimal batch size $B^*$ that leads to the best top-$1$ test accuracy increases as $\delta$ increases from $0$ to $3/8$. Moreover, the top-$1$ accuracy increases as batch size $B$ increases when $B<B^*$, and decreases as $B$ increases when $B>B^*$. The empirical results are highly consistent with our theoretical results.

Similar empirical results for ByzSGDnm can be found in Table 7, Table 8, Table 9, Table 10, and Table 11 in Appendix E.1 due to limited space in the main text.

Reviewer comment (Experiments):
Also, the experiments with the comparison of convergence rates of the proposed method and previous methods. It would be better to provide some experiments like in this paper https://arxiv.org/pdf/2206.00529.pdf (see Figure 1).

Author response:
We sincerely thank the reviewer for the valuable suggestion. Due to limited space in the main text, we have added more results in the Appendix. Please see Figure 1 to Figure 9 in Appendix D.5 and Appendix E, which illustrate the empirical convergence rate of our method and the existing methods under different settings. The added results in the figures are highly consistent with those in the initial version.

We thank the reviewer for the timely response and the positive comments on our added experiments. We are sorry for misunderstanding the reviewer's initial comments. Meanwhile, we would like to point out that in non-i.i.d. cases, a point with small $||\nabla F(w_t)||$ can have a large loss $F(w_t)$. Therefore, $||\nabla F(w_t)||$ w.r.t. iteration number may not properly reflect the performance of a training method. Due to this reason, we did not record $||\nabla F(w_t)||$ in our experiments before, and the remaining time is not enough for us to re-run the experiments. 

Instead, to address the reviewer's concern, we added results of training loss $F(w_t)$ w.r.t. gradient computation number (or, equivalently, epoch number) in the second revision. Please refer to Figure 8 in Appendix E.1 on page 32 in the latest version. The added results further verify the effectiveness of large batch size and ByzSGDnm. 

Meanwhile, we would like to point out that as far as we know, even for non-convex objective functions, $F(w)$ w.r.t. iteration is much more frequently used as the metric in previous works than $||\nabla F(w_t)||$. Here are some references:

+ Figure 4-5 from (Xie et al., 2020);
+ Figure 5-7 in Appendix C from (Yang and Li, 2021);
+ Figure 6 in Appendix A.2.3 from (Karimireddy et al., 2022);

We sincerely thank the reviewer for the response again and hope that our response can address the reviewer's concern. Meanwhile, we will take the reviewer's suggestion into consideration and add results about $||\nabla F(w_t)||$ in future versions. We would greatly appreciate it if the reviewer could re-evaluate our work in light of our response.

---

Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking Byzantine-tolerant sgd by inner product manipulation. UAI 2020.

Yi-Rui Yang and Wu-Jun Li. BASGD: Buffered asynchronous sgd for Byzantine learning. ICML 2021.

Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Byzantine-robust learning on heterogeneous datasets via bucketing. ICLR 2022.

Reviewer comment (Experiments):
In the work the authors consider a homogeneous setting. Some results in heterogeneous setup can improve the contribution of this work dramatically.

Author response:
We thank the reviewer for the constructive suggestion. We have added the empirical results in heterogeneous setup in Appendix E.2. For quick access, we briefly summarize the empirical results below.

We sample from the training set of CIFAR-10 using the Dirichlet distribution with hyper-parameter $1.0$ to create the heterogeneous data. Among the $8$ workers, $1$ worker is under ALIE attack. Geometric median is used as the aggregator. The final top-$1$ test accuracy for different methods is presented in the table below.

|Batch size|32$\times$8 | 64$\times$8 | 128$\times$8 | 256$\times$8 | 512$\times$8|
|----|----|----|----|----|----|
|Byz-VR-MARINA $(p=0.05)$ | 23.07\% | 25.01\% | 22.25\% | 24.67\% |14.28\% |
|Byz-VR-MARINA $(p=0.1)$ | 27.16\% | 23.19\% | 23.08\% | 13.86\% | 11.45\% |
|Byz-VR-MARINA $(p=0.2)$ | 17.83\% | 28.10\% | 27.34\% | 18.67\% | 25.21\% |
|ByzSGDm | 10.16\% | 23.33\% | 27.78\% | 29.63\% | 32.90\% |
|ByzSGDnm | 23.87\% | 26.78\% | 28.42\% | 30.04\% | **35.24\%**|

As the empirical results show, under this non-i.i.d. setting, using a relatively large batch size can still lead to higher test accuracy. Moreover, ByzSGDnm with batch size $512\times 8$ has the best final top-$1$ test accuracy.

In addition, we have also tested the empirical performance of the three methods (ByzSGDnm, ByzSGDm, Byz-VR-MARINA) when combined with nearest neighbor mixing (NNM) technique under the non-i.i.d. setting. ByzSGDnm still has the best final top-$1$ test accuracy among the methods. Please refer to Appendix E.2. for more detailed results.

Reviewer comment (Theory):
There is no theoretical comparison between ByzSGDnm and ByzSGDm. Are there any theoretical benefits of ByzSGDnm compared to ByzSGDm?

Author response:
To the best of our knowledge, the theoretical upper bounds for methods based on SGD (without normalization) in non-convex cases are usually in the form of "$\mathbb{E}||\nabla F(w_t)||^2\leq ...$" while the upper bounds for those based on normalized SGD in the form of "$\mathbb{E}||\nabla F(w_t)||\leq...$". Therefore, the theoretical upper bound for ByzSGDnm cannot be directly compared to ByzSGDm. As we present in Section 4, the obtained convergence order $O(1/T^\frac{1}{4})$ for ByzSGDnm is optimal. Moreover, ByzSGDnm can recover the convergence rate of normalized momentum SGD when there is no attack ($\delta=0$).

Methods based on SGD (without normalization) and those based on normalized SGD have their own applications. Methods based on normalized SGD have been reported to have better empirical performance in large-batch cases (Goyal et al., 2017; Hoffer et al.,2017; Keskar et al., 2017; You et al., 2020). Given the findings in previous works above, we propose ByzSGDnm since our theoretical results show that a relatively large batch size is preferred under Byzantine attacks.

Reviewer comment (Experiments):
The proposed algorithm is only studied in a homogeneous setting ("i.i.d. case" in the paper), which is generally not the case in real applications where data between different clients are heterogeneous. Does the author have any ideas or perhaps experimental results on the behavior of the proposed algorithm in the presence of heterogeneous data?

Author response:
We thank the reviewer for the constructive suggestion. We have added the empirical results in the presence of heterogeneous data in Appendix E.2. For quick access, we briefly summarize the empirical results below.

We sample from the training set of CIFAR-10 using the Dirichlet distribution with hyper-parameter $1.0$ to create the heterogeneous data. Among the $8$ workers, $1$ worker is under ALIE attack. Geometric median is used as the aggregator. The final top-$1$ test accuracy for different methods is presented in the table below.

|Batch size|32$\times$8 | 64$\times$8 | 128$\times$8 | 256$\times$8 | 512$\times$8|
|----|----|----|----|----|----|
|Byz-VR-MARINA $(p=0.05)$ | 23.07\% | 25.01\% | 22.25\% | 24.67\% |14.28\% |
|Byz-VR-MARINA $(p=0.1)$ | 27.16\% | 23.19\% | 23.08\% | 13.86\% | 11.45\% |
|Byz-VR-MARINA $(p=0.2)$ | 17.83\% | 28.10\% | 27.34\% | 18.67\% | 25.21\% |
|ByzSGDm | 10.16\% | 23.33\% | 27.78\% | 29.63\% | 32.90\% |
|ByzSGDnm | 23.87\% | 26.78\% | 28.42\% | 30.04\% | **35.24\%**|

As the empirical results show, under this non-i.i.d. setting, using a relatively large batch size can still lead to higher test accuracy. Moreover, ByzSGDnm with batch size $512\times 8$ has the best final top-$1$ test accuracy.

In addition, we have also tested the empirical performance of the three methods (ByzSGDnm, ByzSGDm, Byz-VR-MARINA) when combined with nearest neighbor mixing (NNM) technique under the non-i.i.d. setting. ByzSGDnm still has the best final top-$1$ test accuracy among the methods. Please refer to Appendix E.2. for more detailed results.

Reviewer comment (Evaluation):
Table 6 shows the execution time of the different algorithms for different batch sizes, for a specific and fixed number of epochs. Can the authors explain why they chose to show the results for a fixed number of epochs and not for a specific accuracy achieved?

Author response:
We show the results for a fixed number of epochs in order to support the claim that using a relatively large batch size can more efficiently utilize the computation power of GPUs. In other words, we want to empirically verify that using a relatively large batch size can make more gradient computation finished in a unit of time.

We have followed the reviewer's suggestion and presented the wall-clock running time that different methods require to reach 50\%, 75\%, and 85\% top-$1$ test accuracy in Appendix E.1. The results show that ByzSGDnm requires less time to reach the target accuracy than ByzSGDm in most cases. Meanwhile, using a relatively large batch size can make ByzSGDm and ByzSGDnm reach the given accuracy faster. For quick access, we present the wall-clock running time required to reach 75\% top-$1$ test accuracy below. The missing values mean that the target test accuracy is not reached in $160$ epochs for the corresponding methods. 

|Batch size|32$\times$8 | 64$\times$8 | 128$\times$8 | 256$\times$8 | 512$\times$8|1024$\times$8|
|----|----|----|----|----|----|----|
|ByzSGDm + KR | -- | -- | -- | -- | **133.69s** | 148.70s|
|ByzSGDnm + KR | -- | -- | 328.49s | **123.12s** | 130.45s | 134.09s|
|ByzSGDm + GM | -- | -- | 114.38s | **59.19s** | 90.74s | 124.03s|
|ByzSGDnm + GM | -- | 252.19s | 66.13s | 54.12s | **45.07s** | 76.49s|
|ByzSGDm + CM | -- | -- | -- | 202.97s | **110.25s** | 204.42s|
|ByzSGDnm + CM | -- | -- | 322.83s | 150.25s | **63.85s** | 109.23s|
|ByzSGDm + CC | -- | 727.07s | 77.19s | 87.42s | **86.53s** | 150.59s|
|ByzSGDnm + CC | 1465.07s | 112.49s | 109.18s | **56.20s** | 61.30s | 81.28s|

Reviewer comment (Evaluation):
As far as I know, the speed gain obtained by choosing a larger batch size is naturally explained by the fact that we can benefit from the parallelization of computations (in the system side). However, I would find it interesting to understand whether the methods presented are faster to reach a given accuracy with a larger batch size.

Author response:
We show the results for a fixed number of epochs in order to support the claim that using a relatively large batch size can more efficiently utilize the computation power of GPUs. In other words, we want to empirically verify that using a relatively large batch size can make more gradient computation finished in a unit of time.

We have followed the reviewer's suggestion and presented the wall-clock running time that different methods require to reach 50\%, 75\%, and 85\% top-$1$ test accuracy in Appendix E.1. The results show that ByzSGDnm requires less time to reach the target accuracy than ByzSGDm in most cases. Meanwhile, using a relatively large batch size can make ByzSGDm and ByzSGDnm reach the given accuracy faster. For quick access, we present the wall-clock running time required to reach 75\% top-$1$ test accuracy below. The missing values mean that the target test accuracy is not reached in $160$ epochs for the corresponding methods. 

|Batch size|32$\times$8 | 64$\times$8 | 128$\times$8 | 256$\times$8 | 512$\times$8|1024$\times$8|
|----|----|----|----|----|----|----|
|ByzSGDm + KR | -- | -- | -- | -- | **133.69s** | 148.70s|
|ByzSGDnm + KR | -- | -- | 328.49s | **123.12s** | 130.45s | 134.09s|
|ByzSGDm + GM | -- | -- | 114.38s | **59.19s** | 90.74s | 124.03s|
|ByzSGDnm + GM | -- | 252.19s | 66.13s | 54.12s | **45.07s** | 76.49s|
|ByzSGDm + CM | -- | -- | -- | 202.97s | **110.25s** | 204.42s|
|ByzSGDnm + CM | -- | -- | 322.83s | 150.25s | **63.85s** | 109.23s|
|ByzSGDm + CC | -- | 727.07s | 77.19s | 87.42s | **86.53s** | 150.59s|
|ByzSGDnm + CC | 1465.07s | 112.49s | 109.18s | **56.20s** | 61.30s | 81.28s|