Paper: SAFLEX: Self-Adaptive Augmentation via Feature Label Extrapolation (ICLR-2024)

Reviewer comment (Experiments):
Even though the paper proposes a first-order approximation method, it does not provide an evaluation of this method. In other words, the paper should provide a performance comparison with the usual bi-level optimization and a computation cost comparison with other data augmentation strategies such as Fast AutoAugment.

Author response:
**Response to Weakness 1:**

- We appreciate reviewer NKYa's valuable suggestion for an ablation study on the first-order approximation method. We agree that conducting such a study would further solidify our findings.


- However, due to space constraints, we prioritized presenting the core message of this paper: rectifying any upstream data augmentation techniques for downstream tasks. To effectively convey this message, we conducted extensive experiments encompassing diverse data types, comprehensive coverage of upstream augmentation techniques, various downstream tasks, and multiple backbone models. As a result, we were unable to include an extensive ablation study on the first-order approximation method.

- It is important to note that there is a significant body of research on first-order approximations of bilevel problems, such as the work by [R1]. Given the extensive nature of this field, we have opted to defer an in-depth exploration of first-order approximation methods to future investigations. This decision allows us to focus on the core contribution of this paper, which is the development of a novel data augmentation framework for downstream tasks. 
    > [R1] Liu, Bo, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu. "Bome! bilevel optimization made easy: A simple first-order approach." Advances in Neural Information Processing Systems 35 (2022): 17248-17262.

**Response to new comments on weakness 1:** We appreciate your understanding. Your suggestion to include the ablation study in the Appendix for the camera-ready version is well-received. We agree that this addition will enhance the paper's comprehensiveness and intend to incorporate it as suggested.

Reviewer comment (Writing):
The writing of the paper is not necessarily of high quality. For example, Theorem 1 is very difficult to read because it contains multiple claims that make up the entire solution. Theorems and corollaries should be split for each claim, or if the propositions are ambiguous, they should be replaced with detailed explanations for each component, rather than in the form of a theorem. In fact, the proofs provided by the Appendix are almost obvious and make little theoretical contribution.

Author response:
**Response to Weakness 2:**

We express our sincere gratitude to reviewer NKYa for their valuable suggestions on improving the accessibility of our paper. We have carefully considered reviewer NKYa's feedback and have implemented the following changes to enhance the clarity and readability of our manuscript:

- Specifically for Theorem 1, we separate the notation definitions and the final formula for the approximated soft label and sample weight. We have made every component more readable, including the gradient, Jacobian-vector product, and the final formula.

**Response to new comments on weakness 2:** We are glad to hear that the revisions have improved the readability of the paper.

Reviewer comment (Experiments):
Do you think the proposed method can be applied to consistency-based semi-supervised learning with data augmentation, e.g., FixMatch [a]? The study of estimating the importance of samples and labels is well studied in the field of semi-supervised learning rather than in the field of data augmentation (e.g., FreeMatch [b]). If it can be shown that the scheme of the proposed method can be implemented in semi-supervised learning, the impact of this paper on the community will be even greater.

Author response:
**Response to Question:** 

Indeed, at the end of section 3, we briefly discuss how to apply SAFLEX for contrastive learning, which is indeed the consistency-based self-supervised or semi-supervised learning suggested by reviewer NKYa. Due to space limitations, detailed equations are in Appendix A, equations (12) and (13). 

We have also verified this through experiments. Specifically, we did fine-tune a CLIP model (Radford et al., 2021) on the iWildCam dataset from Wilds (Koh et al., 2021), using a contrastive fine-tuning paradigm called “Finetune Like You Pretrain” (FLYP) (Goyal et al., 2023).

**Response to new comments on the qustion:**

Thank you for your clarification regarding the applicability of our method to semi-supervised learning, specifically in the context of pseudo-labels.

We acknowledge the potential of our method to contribute to consistency-based semi-supervised learning approaches, such as FixMatch. However, adapting our algorithm for this purpose would not be straightforward and would necessitate a range of new designs and experiments.

Several pertinent questions arise in this context:
1. Whether to apply our method to both weak and strong augmentations, or to focus on a single branch.
2. The choice between using the labeled set as a validation set in our algorithm, or developing a cross-validation method.
3. Deciding whether to simply soften the inferred pseudo-labels, or to devise a new method ensuring consistency between pseudo-labeling and soft-labeling.

While these questions extend beyond the scope of our current paper, we recognize their significance and potential impact on future research. We are inspired to explore these aspects in subsequent work, as they align with the broader objectives of our research. Your insights have been invaluable in highlighting these potential applications. We really appreciate your continued engagement and clarification on this matter.

Reviewer comment (Experiments):
The main issue is a lack of proper baselines. Papers such as [1] have already explored using soft labels for augmentations where the softness is derived on the basis of augmentation strength. This paper's novelty thus gets limited. There is no comparison with [1] in any of the experiments. The authors should do a proper comparison with [1] and justify how their approach is better than it.

Author response:
We thank reviewer GM6j for their constructive feedback. We are glad to see that reviewer GM6j acknowledges the motivation of our paper and our method of solving the issues with standard augmentation by learning sample weights and soft-label. 

Below, we would like to address the concerns reviewer GM6j has. 

---

> **Weakness 1:** The main issue is a lack of proper baselines. Papers such as [1] have already explored using soft labels for augmentations where the softness is derived on the basis of augmentation strength. This paper's novelty thus gets limited. There is no comparison with [1] in any of the experiments. The authors should do a proper comparison with [1] and justify how their approach is better than it.
[1] Soft Augmentation for Image Classification. Liu et al. https://arxiv.org/pdf/2211.04625.pdf

**Response to Weakness 1:**
We acknowledge that our SAFLEX approach shares a similar spirit with the Soft Augmentation method proposed by Liu et al. (2022), in that both methods consider soft labels/targets and soft samlpe weights (i.e., loss reweighting). However, we believe there are huge methodological differences between the two methods in how they model the soft labels and weights. These methodological distinctions lead to significant differences in applicability. Below, we elaborate on the methodological and applicability differences between the two approaches and provide empirical comparisons to further highlight the novelty and improved performance of our method.

1. **Methodology Differences**. Our *SAFLEX* employs a learnable, augmentation-method agnostic, and more automatic and principled approach for generating soft labels and sample weights.
    1. In *Soft Augmentation*, the authors implement a **specific approach to generating soft labels**, namely through label smoothing. Label smoothing modifies the indicator value '1' (representing the ground-truth class label) with $p = 1 − \alpha(\phi)$, where the adaptive smoothing factor $\alpha(\phi)$ is determined by the degree/strength $\phi$ of the specific sampled augmentation applied to input $x_i$. Notably, the remaining probability mass $\alpha(\phi)$ is **uniformly distributed** across all other class labels. The formula of $\alpha(\cdot)$ **requires human modeling with domain expertise**. And since different upstream augmentation methods have different definitions of the strength factor $\phi$, remodeling of $\alpha(\cdot)$ for each new augmentation method is required. The discussion in *Soft Augmentation* mainly **focuses on crop augmentations** on images, which impressively draws insights from human visual classification experiments.
    2. Our *SAFLEX*, in contrast, differs in these key aspects:
        (a) **Flexible Soft Labels:** *SAFLEX* employs a more flexible approach to modeling soft labels, moving beyond label smoothing's limitations. We believe that uniformly distributing the probability mass across all classes may not always be the most effective strategy. This limitation of *Soft Augmentation* is also acknowledged in the *Soft Augmentation* paper, which states: "the smoothed target label of a highly-occluded truck example could place more probability mass on other vehicle classes, rather than distributing it equally across all classes."
        (b) **Learned Soft Labels and Sample Weights:** In *SAFLEX*, both soft labels and sample weights are learned from a bilevel optimization problem, which is agnostic to the type and strength of the upstream augmentation method.
        \(c\) **Bilevel Optimization Problem:** *SAFLEX* confronts the inherent challenge of soft augmentation by framing it as a bilevel optimization problem. This approach represents the first rigorous formulation of the problem, underscoring an important theoretical contribution. Additionally, we introduce novel and efficient algorithms specifically designed to tackle this bilevel optimization challenge.

Reviewer comment (Experiments):
To solidify the experimental results the authors should also experiment with stronger architectures and datasets such as ResNet-101 over ImageNet as done in [1].

Author response:
> **Weakness 2:** To solidify the experimental results the authors should also experiment with stronger architectures and datasets such as ResNet-101 over ImageNet as done in [1].

**Response to weakness 2:** 

We appreciate reviewer GM6j's suggestion for further experiments with stronger architectures, such as ResNet-101 on the ImageNet dataset. However, we would like to emphasize that our current experimental evaluation is already comprehensive and demonstrates the effectiveness of our method across a variety of tasks and datasets, as reviewers EBBs, PN9Y, and NKYa have noted.

1. **Our evaluation encompasses a diverse range of data types**. We have assessed the performance of SAFLEX on medical image data, tabular data, and high-dimensional natural image data, including the CUB dataset and iWildCam from the Wilds dataset. This breadth of data types exceeds the scope of most existing studies.


2. **Our evaluation encompasses a broad spectrum of downstream tasks.** We have conducted experiments on classification tasks involving medical images and tabular data, fine-grained classification on natural images using the CUB dataset, out-of-distribution (OOD) generalization on the iWildCam dataset, and self-supervised contrastive fine-tuning. In contrast, most existing studies focus on a single task at a time.

3. **Our evaluation covers almost all existing augmentation as upstream augmentation techniques, showing SAFLEX's compatibility with all these methods and consistent improvements over these upstream augmentations.** The augmentation methods we test against range from RandAug, adversarial autonomous augmentation like LP-A3, to diffusion-model-based augmentation methods.

4. **Our evaluation covers various machine learning models, including state-of-the-art generative models.** We have included ResNet-50, Stable Diffusion v1.5,  CLIP model equipped with a ViT-B/16 encoder, etc. 

 
5. Indeed, we have employed ResNet-50 on ImageNet. We believe extending the results to ResNet-101 is not necessarily crucial, given our consistent performance improvements across diverse data types, different downstream tasks, a wide range of upstream augmentation methods, and various machine learning models.

---

Again, we express our sincere gratitude to reviewer GM6j for their valuable service in reviewing our manuscript.