Paper: MoReDrop: Dropout Without Dropping (ICLR-2024)

Reviewer comment (Theory):
**Discrepancy in Mathematical Formulation:** There’s an inconsistency in Equation (4), where the expectation is placed inside the exponent, but during training, the expectation is positioned outside the exponent, leading to potential questions regarding the mathematical precision and validity of the given formulation. This inconsistency needs to be addressed to ensure the robustness and accuracy of the mathematical model presented.

Author response:
Thank the reviewer for pointing out the potential misunderstanding regarding Equation (4) in our manuscript. We would like to clarify there is no inconsistency in Eq (4). Concretely, this expectation over $S_D$ in $\mathbb{E}_{S\_{\mathcal{D}}}\left[l\left(\mathcal{D}, S\_{\mathcal{D}}; \theta\right)\right]$ is designed to approximate dropout as an ensemble of neural networks via weight sharing during the training phase, as defined in Eq (1). However, it does this **without an explicit expectation operator** over $S_D$. 

In contrast, during the training process with mini-batches, the expectation is taken over the data instead. To aid in understanding and implementing our approach, we have included a clear pseudocode formatted in a Pytorch-like style, in Appendix A under Algorithm 2. This should provide further clarification and assistance in grasping the nuances of our method.

Reviewer comment (Theory):
**Unclear Implications of Theoretical Concepts:** The paper introduces Theorem 1 but fails to elucidate its significance concerning Equation (4). It leaves the readers with ambiguity regarding the theorem's role and impact on the equation's interpretation or formulation.

Author response:
We would like to clarify that Theorem 3.1 is fundamental in establishing the validity and effectiveness of **the first-order form**, i.e., $L_1$, of our loss function with the non-decreasing function $g$:
$$
\left.\mathcal{R}=g\left(\mathbb{E}_{S\_{\mathcal{D}}}\left[l\left(\mathcal{D}, S\_{\mathcal{D}} ; \theta\right)\right]-l(\mathcal{D} ; \theta)\right]\right).
$$

Theorem 3.1 states that $l(\mathcal{D} ; \theta) \leq \mathbb{E}_{S\_{\mathcal{D}}}\left[l\left(\mathcal{D}, S\_{\mathcal{D}} ; \theta\right)\right]$. 

This inequality ensures that the expression $\mathbb{E}_{S\_{\mathcal{D}}}\left[l\left(\mathcal{D}, S\_{\mathcal{D}}; \theta\right)\right]-l(\mathcal{D}; \theta)$ is always greater than or equal to 0. 

Without Theorem 3.1, if we suppose $\mathbb{E}_{S\_{\mathcal{D}}}\left[l\left(\mathcal{D}, S\_{\mathcal{D}}; \theta\right)\right]$ is smaller than $l(\mathcal{D}; \theta)$, we would face the paradoxical situation to minimize the loss function, i.e., increasing  $l(\mathcal{D}; \theta)$, that does not align with our primary goal. Then, we have to choose the second order form, e.g., $L_2$,  to construct the regularizer.  

In summary, Theorem 3.1 is important which guarantees $x\geq 0$  and thus $\min_x \mathcal{R}= g(0)$ if $g$ is a non-decreasing function. We then can simply set the $g$ as the $L_1$ norm rather than the $L_2$ norm. The ablation study for $g$ in Appendix C.4.2 demonstrates the efficiency of $L_1$ choice, especially in high dropout rates.

Reviewer comment (Theory):
**Ambiguity in Backpropagation Process:** The explanation regarding the gradient backpropagation in Section 3.2 is unclear. It doesn’t adequately explain why the shared parameter sub-model $ M_i$ remains unaffected during backpropagation despite its involvement in the equations, leading to confusion about the actual process and mechanisms at play.

Author response:
We apologize for any misinterpretation of the claim that "the shared-parameter sub-model $M_i$ remains unaffected during backpropagation", and we have revised our paper for clarification. In response, this claim **does not exist in our manuscript**. Actually, the parameter of $M_i$ is passively updated through the shared nature with the dense model with active updates. To aid in understanding and implementing our approach, we have included a clear pseudocode formatted in a Pytorch-like style, in Appendix A under Algorithm 2. This should provide further clarification and assistance in grasping the nuances of our method.

Reviewer comment (Theory):
**Questionable Robustness at High Dropout Rates:** The claim regarding MoReDrop’s robustness at high dropout rates appears counterintuitive, and the paper doesn’t provide a comprehensive explanation to dispel this contradiction. It lacks clarity on how MoReDrop maintains effective regularization, especially with high values of $p$, which seems to diminish the effect of $M_i$ and potentially nullify the dropout effect.

Author response:
We thank the reviewer for pointing out this important question and we have revised our paper for better clarification. We here clarify for two main reasons:

**The training paradigm of MoReDrop**. In MoReDrop, the primary gradient update mechanism is derived from traditional dense training, distinct from the influence of the regularizer. The regularizer integrates the benefits of dropout models into the dense training framework. The parameter $\alpha$ is pivotal in modulating the extent to which the regularizer, denoted as $\mathcal{R}$, impacts the gradient update in the dense model. In scenarios with high dropout rates, a reduced $\alpha$ value minimizes the impact from the regularizer, preserving the performance of the dense model that primarily relies on conventional training methods sans dropout. 

This approach aligns with findings from our ablation study in Appendix C.4.1, indicating the importance of lower $\alpha$ values in sustaining model performance under increased dropout rates. In contrast, a higher $\alpha$ is advantageous in lower dropout scenarios to borrow advantages from dropout models, leveraging the reliable predictions of the dropout model.

**The bound nature of $\mathcal{R}$.** The inherent boundedness of the regularizer $\mathcal{R}$ facilitates constraining the gap to less than 1, coupled with the weight $\alpha$, exerts limited influence on the optimization of the dense model.

In summary, in high dropout rates, the performance is retained mostly from the dense model by setting a smaller $\alpha$, this inherits from the unique training paradigm of MoReDrop and the bound nature of the regularizer.

Reviewer comment (Experiments):
**Lack of Depth in Exploring Dropout Rates:** The paper doesn’t dive deep into exploring the implications of varying dropout rates in MoReDrop. There’s a missed opportunity to clarify the differences and effects at different dropout rates, such as distinguishing between $p=0$ and $p=1.0$, which would have contributed to a richer understanding of the methodology.

Author response:
I would like to clarify that we have performed a comprehensive dropout rate set, i.e., [0.1, 0.2, 0.3, 0.4, 0.5, 0.7, 0.9] in our experiments. Furthermore, we believe it is not entirely logical to conduct the extreme dropout cases. 
1. $p=0$: $p=0$ indicates that there is no dropout on the sub-model, which means the dense model and the sub-model are identical with no chance to leverage the advantages of dropout models. This also means that the regularization term $\mathcal{R}$ in Eq. 4 is 0 because $\mathbb{E}_{S\_{\mathcal{D}}}\left[l\left(\mathcal{D}, S\_{\mathcal{D}} ; \theta\right)\right]=l(\mathcal{D} ; \theta)]$. 
2. $p=1$: $p=1$ indicates that the sub-model is nothing because we mask all neurons, which means the sub-model does not exist and the prediction of it is totally random, where we do not expect any information or benefits from it. The value of $\mathbf{\hat{y}=f(\mathcal{D}; \theta)}$ will be a random prediction without any improvement. Practically, in a dropout setting with  $p=1$ **in layer $l$**, the output of the **layer $l$** will invariably be zero.

Reviewer comment (Evaluation):
The performance improvement showcased in the experiments is marginal. Across various datasets, there's only about a 1% enhancement. Given this, the results might fall within the error bar, which the authors should consider highlighting. Taking into account the longer training time that MoReDrop demands, the trade-off between model complexity and experimental outcomes seems limited.

Author response:
We thank the reviewer for suggestion on our experiments and we have added error bars and standard variance calculations to our experimental results under 5 different seeds in our new manuscript.  While we acknowledge that a 1% improvement might initially appear marginal, it is important to consider the context within the field of computer vision, where even small enhancements can be significant, especially when working with high-performing baseline models.

Reviewer comment (Experiments):
Were there any experiments conducted using a smaller model as the regularizer without employing dropout in that network, opting instead to reduce its complexity by modifying the number of layers or nodes? Btw, does the main model use any regularization (weight decay, etc.)?

Author response:
> W1: 1. Were there any experiments conducted using a smaller model as the regularizer without employing dropout in that network, opting instead to reduce its complexity by modifying the number of layers or nodes? 

Answer: We appreciate the reviewer's suggestion regarding model regularization for the dense model. However, we clarify that adopting a smaller model may not be reasonable for two reasons:
1. **Benefits we aim to leverage**: The primary motivation of this study is to harness benefits such as the robust generalization ability inherent in dropout models. We posit that only larger models have the potential to replicate these benefits akin to those observed in dropout models.
2. **Cost consideration:** Implementing a larger model for regularization, with the objective of achieving robust generalization ability, will invariably incur significant memory and computational costs.

In summary, while larger models may emulate the advantages of dropout models, they also bring substantial memory and computational costs. MoReDrop addresses these issues through regularization, implemented via a shared-parameter sub-model within the training dense model.

> W1: Btw, does the main model use any regularization (weight decay, etc.)?

Answer: Yes, our implementation closely follows the official open-source version, and we have maintained the same configuration settings as the original implementation. Specifically, we have incorporated weight decay in the implementation of the ViT-B setting, mirroring the original approach. Additionally, we use batch normalization in ResNet, as proposed in the original code. Further details about the data augmentation methods adopted in our work can be found in Appendix B.

Reviewer comment (Experiments):
Given that the regularizer network does not contribute to gradient flow and essentially remains untrained, were different weight and bias initialization methods explored for the regularizer network, and if so, what were the outcomes?

Author response:
> W2: Given that the regularizer network does not contribute to gradient flow and essentially remains untrained, were different weight and bias initialization methods explored for the regularizer network, and if so, what were the outcomes?

Answer: We appreciate the reviewer's query, as it contributes to a deeper understanding of MoReDrop. In response, conducting this ablation study with MoReDrop is not feasible due to the shared nature of the regularizer network with the dense training model. Initializing two separate networks—one for training and another for regularization—contradicts our established, such as the cost consideration, as detailed in response to W1.

Reviewer comment (Experiments):
It would be valuable to assess the performance and generalization capabilities of this approach on a wider range of datasets. Specifically, results on datasets like CIFAR-C, ImageNet-C, and domain generalization datasets such as VLCS would provide a broader perspective on the model's effectiveness.

Author response:
> W3: It would be valuable to assess the performance and generalization capabilities of this approach on a wider range of datasets. Specifically, results on datasets like CIFAR-C, ImageNet-C, and domain generalization datasets such as VLCS would provide a broader perspective on the model's effectiveness.

Answer: Thank the reviewer for your valuable suggestion regarding the evaluation of our model's robustness and generalization capabilities across a wider range of datasets. In line with your recommendations, we have extended our experimental framework to include additional datasets such as CIFAR-10-C, ImageNet-C (blur and digital) on WRN-28-10, and conducted an analysis under the PGD attack.

Due to time constraints, we standardized the experimental setup for all datasets by fixing $(p,\alpha)=(0.3,1)$ for CIFAR-10-C and PGD attack scenarios, and $(p,\alpha)=(0.1,1)$ for ImageNet-C. This decision allowed us to focus on comparing the effectiveness of our method, MoReDrop, across different datasets and conditions while maintaining a consistent experimental environment.

| method |CIFAR-10 | PGD|CIFAR-10-C | 
|-----|-----|-----|-----|
| WRN-28-10 |94.65 | 23.83 |76.27   |  
| +MoReDrop |**94.81** | **28.53** |**77.01**   |    

| method |ImageNet |ImageNet-blur|ImageNet-digital|
|-----|-----|-----|-----|
| ViT-B |84.01 | 51.86 |61.58 |
| +MoReDrop |**84.47**|**52.18**  | **62.30**|  

The tables presented above demonstrate that MoReDrop significantly improves model generalization and robustness. Notably, we enhanced the PGD accuracy without implementing any specific defense mechanisms. These results were attained without fine-tuning, relying solely on empirically derived hyperparameter settings. We believe that a proper parameter setting can further improve the robustness and generalization.

Reviewer comment (Novelty):
This work reminds me another paper (https://arxiv.org/abs/2207.01548) that focuses on regularization by using an additional model without batch normalization (BN). It might be interesting to delve into the connections and distinctions between these two approaches. This makes me more think of point #1 above.

Author response:
> W4: This work reminds me of another paper (https://arxiv.org/abs/2207.01548) that focuses on regularization by using an additional model without batch normalization (BN). It might be interesting to delve into the connections and distinctions between these two approaches. This makes me think of point #1 above.

Answer: we would like to share our grateful appreciate for this related work.  We delineate the Similarities and distinctions between our work and the referenced paper ([Taghanaki et al., 2022](https://arxiv.org/pdf/2207.01548.pdf)). 

**Similarities**: Both our research and [Taghanaki et al., 2022](https://arxiv.org/pdf/2207.01548.pdf) aim to augment the generalization and robustness of models. Notably, the training process in both instances involves the interaction of two models.

**Distinctions**
1. **Motivation**: [Taghanaki et al., 2022](https://arxiv.org/pdf/2207.01548.pdf) focus on addressing a problem specific to the Batch Normalization's (BN). In contrast, the motivation of our work focuses on how to leverage the benefits of dropout models and avoid the model distributional shift. 
2. **Methods**:  [Taghanaki et al., 2022](https://arxiv.org/pdf/2207.01548.pdf) adheres to the standard distillation training paradigm to solve the related work in BN, but MoReDrop has no relationship with the distillation paradigm but a sub-model with dropout. 
3. **Cost**: The distillation training paradigm requires the teacher and student models with different parameters, it requires extra memory cost but MoReDrop does not because of the shared parameter between the dropout model and the main dense model. 
4. **Scalability**: The problem that [Taghanaki et al., 2022](https://arxiv.org/pdf/2207.01548.pdf) aim to mitigate specific to the BN, yet not the same scenarios in other normalization methods, such as group normalization. However, our method is scalable and compatible with various dropout variants to further improve the performance.

Reviewer comment (Experiments):
I am not fully convinced by the use of equation 4 for the loss function. It would be good if additional empirical analysis can be done on the effect of different choices of $g(\cdot)$ have on the results.

Author response:
We greatly appreciate your assistance in strengthening our paper with regard to the loss function $g$. In response, our loss function ( $g(x)=\frac{e^x-1}{e^x+1}=2\cdot(sigmoid(x)-\frac{1}{2})$) and we clarify two main strengths:
1. The bound nature of this loss function between (-1, 1). 
2. The superior in contrast to loss functions of other dropout regularization methods. 
However, it is important to acknowledge the possibility of other, more effective loss functions, which we aim to explore in future research.

**Whether the bound nature between (-1, 1) efficient in the experimental view?**  For practicality and simplicity in implementation, we adopt ResNet-18 as our model backbone. We configure four distinct experimental scenarios: ResNet-18-2D, which incorporate 2 dropout layers. 

| $g(\cdot)$  | p=0.1 | p=0.3 |p=0.5|p=0.7 |p=0.9|
|-----|-----|-----|-----|-----|-----|
| $g(x)=x$    | 95.22    |  95.31  | 95.13  | 95.04 | 90.07  |
| $g(x)=x^2$     | 95.42    | 95.51   | 95.33   | 95.23 | 94.23  |
| $g(x)=\frac{e^x-1}{e^x+1}$    |  **95.56**   | **95.80**   | **95.72**  |  **95.47** | **95.23**  |

The tables above reveal distinct performance trends across different $g(\cdot)$ functions. Specifically,$g(x)=\frac{e^x-1}{e^x+1}$ outperforms other functions in all settings. With the increasing dropout rate, we can find the performance gap between $g(x) = x^2$ and $g(x) = \frac{e^x - 1}{e^x + 1}$ is also increasing, where $g(x) = \frac{e^x - 1}{e^x + 1}$ exhibits superior robustness in these extreme conditions. 

**$\alpha \cdot g(x)$ or $g(\alpha \cdot x)$?** We then perform an examination on these two forms of the loss function with respect to the location of $\alpha$, where the second one is also bounded between (-1, 1). To this end, an experiment was conducted using ResNet-18-4D with $p = 0.5$ shown in the following table: 

| $g(\cdot)$  | $\alpha=0.1$ | $\alpha=0.5$ |$\alpha=1$|$\alpha=2$ |$\alpha=5$ |$\alpha=10$ |
|-----|-----|-----|-----|-----|-----|-----|
| $\alpha \cdot g(x)$   | **95.55**    | 95.81   |  95.61 | **95.48** | 94.28|73.80|
| $g(\alpha \cdot x)$  | 95.52 | **95.83**   |  **95.66** | **95.48** |**95.36** |**95.22**|

We observe that the bounded loss function  $g(\alpha \cdot x)$ shows significant robustness in the extreme choices of $\alpha$, offering more versatility in addressing a range of issues. We acknowledge the limit discrepancy for the optimal solution between those two functions, i.e., $\alpha=0.5$. However, it is important to note that the optimal $\alpha$ may exhibit dramatic changes in some extreme tasks.

Reviewer comment (Experiments):
An additional ablation study on the quality of sub-model predictions have during the course of entire training would be very helpful in facilitating the understanding on exactly how the proposed method is helping regularizing the model.

Author response:
We appreciate the reviewer's input which has enhanced our paper's clarity. In Section 4.3, we present a thorough analysis of the loss function (the proxy to the prediction), comparing MoReDrop and R-Drop, two standard algorithms in this field. Our loss analysis also reveals that MoReDrop maintains the model generalization ability, despite imposed constraints. In contrast, R-Drop suffers from model distributional shift and compromised generalization, leading to its lower performance compared to MoReDrop.

Furthermore, we here want to highlight that the primary loss function from the dense model and gradient backpropagation solely to the dense model ensures model configuration consistency for training and evaluation. In other words, this design allows MoReDrop without model distributional shift, a clear comparison is shown in Appendix Table 4 and the first answer. The regularization in MoReDrop provides the dropout model benefits.

Reviewer comment (Novelty):
The proposed method feels very similar to the use of "exponential moving average (EMA) predictions" in semi-supervised learning [1]. I wonder if the proposed algorithm is just similar things to that. It also sounds similar to another line of research on self-distillation [2]. A baseline comparison and a literature review along these two lines of work would be helpful.

Author response:
We thank the reviewer for referring to more related work. In response, we think those two related work shows a significant discrepancy to our work,  especially the motivation, and the comparison is redundant. We present the reasons one by one: 

With [Tarvainen et al.,2017](https://proceedings.neurips.cc/paper/2017/file/68053af2923e00204c3ca7c6a3150cf7-Paper.pdf)
1. **Motivation**: [Tarvainen et al.,2017](https://proceedings.neurips.cc/paper/2017/file/68053af2923e00204c3ca7c6a3150cf7-Paper.pdf) aim to mitigate the limitation of Temporal Ensembling in semi-supervised learning. In contrast, the motivation of our work is to mitigate the model distributional shift problem with dropout. 
2. **Method**: [Tarvainen et al.,2017](https://proceedings.neurips.cc/paper/2017/file/68053af2923e00204c3ca7c6a3150cf7-Paper.pdf) improve the generalization ability by different data sources, i.e., the noisy input information. In contrast, our method preserves the generalization ability from the model perspective by the regularization of the dropout sub-model.
3. **Memory cost:** [Tarvainen et al.,2017](https://proceedings.neurips.cc/paper/2017/file/68053af2923e00204c3ca7c6a3150cf7-Paper.pdf) adopts the idea of distillation with teacher and student models, which requires extra memory to store these two models because they are not shared parameter. 

With [Zhang et al.,2019](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf) 
1. **Motivation**: [Zhang et al.,2019](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf)  advocate a self-distillation approach to enhance the performance on a smaller network, i.e., some modules from the entire model. However, the motivation of MoReDrop is to address the model distribution shift with the dense model and its shared-parameter sub-model, where these two models are the entire model with the same model size. 
2. **Method**: [Zhang et al.,2019](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf)  enhance shallow module predictions (a smaller model) using deep modules, whereas MoReDrop uses two equally sized models. Note that the sub-model definition in those two works is significantly different.
8. **Memory cost**: [Zhang et al.,2019](https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhang_Be_Your_Own_Teacher_Improve_the_Performance_of_Convolutional_Neural_ICCV_2019_paper.pdf)  necessitate additional training modifications such as supplementary softmax or classifier layers, which incur an additional memory cost. In contrast, our MoReDrop method does not require such adjustments, as we did not introduce any extra parameters during the training process.

To summarize, there is a notable divergence between our work and the two referenced studies, particularly in terms of motivation. Consequently, we consider comparisons with these studies to be less feasible and superfluous.

Reviewer comment (Evaluation):
Any additional explanation on why the proposed method can perform as well/better on high dropout-rate? It seems like higher dropout rate probably lead to less accurate predictions.

Author response:
We thank the reviewer for this question, which further enhances the understanding of MoReDrop. We here clarify for two main reasons:

**The training paradigm of MoReDrop**. In MoReDrop, the primary gradient update mechanism is derived from traditional dense training, distinct from the influence of the regularizer. The regularizer integrates benefits of dropout models into the dense training framework. The parameter $\alpha$ is pivotal in modulating the extent to which the regularizer, denoted as $\mathcal{R}$, impacts the gradient update in the dense model. In scenarios with high dropout rates, a reduced $\alpha$ value minimizes the impact from the regularizer, preserving the performance of the dense model that primarily relies on conventional training methods sans dropout. 

This approach aligns with findings from our ablation study in Appendix C.4.1, indicating the importance of lower $\alpha$ values in sustaining model performance under increased dropout rates. In contrast, a higher $\alpha$ is advantageous in lower dropout scenarios to borrow advantages from dropout models, leveraging the reliable predictions of the dropout model.

**The bound nature of $\mathcal{R}$.** The inherent boundedness of the regularizer $\mathcal{R}$ facilitates constraining the gap to less than 1, coupled with the weight $\alpha$, exerts limited influence on the optimization of the dense model.

In summary, in high dropout rates, the performance is retained mostly from the dense model by setting a smaller $\alpha$, this inherits from the unique training paradigm of MoReDrop and the bound nature of the regularizer.