Paper: Selective Mixup Fine-Tuning for Optimizing Non-Decomposable Objectives (ICLR-2024)

Reviewer comment (Experiments):
Can authors provide some ablation studies on the hyper-parameters, such as the imbalanced ratio and the number of samples for feedback?

Author response:
We request you look at Appendix K, where we have provided analysis on hyper-parameters. Here, we provide the ablation of the size of the validation set vs. the Mean Recall for the CIFAR-10 dataset with an imbalance ratio, $\rho_l=\rho_u=100$, where we observe robust performance of SelMix even when data is reduced to 10% of initial.

**Table 2: Comparison of SelMix's performance for different sizes of the feedback set.**
| Percentage of <br> samples used | Mean Recall |
|-|-|
| 5k                     | 85.5        |
| 2.5k                     | 84.8        |
| 1.25k                       | 84.7        |
| 0.5k                       | 84.4        |

We show the ability of SelMix to boost performance in various imbalanced settings. We consider the CIFAR-10 dataset with an imbalance ratio, $\rho_l=\rho_u=50, 100, 200$ and show the results for optimizing Mean Recall and Min. Recall.

**Table 2: Comparison of SelMix's performance on different imbalance factors (Setup as in Table 1).**

| Method       |   Mean Recall  $(\rho_l = \rho_u=50)$ |  Min Recall $(\rho_l = \rho_u=50)$ |  Mean Recall $(\rho_l = \rho_u=100)$ |   Min Recall $(\rho_l = \rho_u=100)$ |   Mean Recall $(\rho_l = \rho_u=200)$ |   Min Recall $(\rho_l = \rho_u = 200)$ |
|-|-|-|-|-|-|-|
| FixMatch(LA) | 84.5                            | 73.8                           | 79.7                             | 55.9                          | 74.3                             | 33.8                          |
| w/ SelMix    | 87.1                            | 81.1                           | 85.4                             | 79.1                          | 81.6                             | 64.6                          |

Further in the Appendix, we compare the performance of different sampling policies for different values of scaling parameters (Fig. K.1). We also compare against different substitute mixups for the input, such as PuzzleMix and CutMix (Tab. K.5). We have provided results for different permutations of imbalance ratios for the labeled and unlabeled data such as $\rho_l = \rho_u = 100$ in Tab. 2(L) , $(\rho_l, \rho_u)= (100, 1)$ and $(\rho_l, \rho_u) = (100, \frac{1}{100})$ in Appendix Tab. H.1 
We also compare the effectiveness of substituting cross-entropy loss with LA loss in FixMatch, which yields a better feature extractor, resulting in an overall boost in performance in Tab. K.4.

We hope our response has clarified your concerns; please feel free to reach out to us if there are any other questions.

Reviewer comment (Experiments):
**On a Fair Comparison with Other Methods:** The paper uses FixMatch + Logits Adjusted (LA) loss as a baseline, while other approaches are evaluated using the vanilla FixMatch. This discrepancy in the comparison may not provide a fair assessment of the true impact of SelMix. As can be seen in Table H.2 and H.1, FixMatch + LA shows already significant improvements over vanilla FixMatch. This makes it challenging to evaluate whether SelMix's improvements reported in the benchmarks are due to the approach or its combination with a more advanced baseline. A fair comparison would involve evaluating SelMix with vanilla FixMatch to better understand its relative performance, or comparing to other state-of-the-art methods when using FixMatch + LA.

Author response:
Thanks for pointing out this inconsistency due to **a typo of w/**; we have corrected typos in the revision. We want to clarify that all the methods we have compared in Table H.1 are methods **which are trained from scratch** and are not finetuning methods like the proposed SelMix (hence, w/ is not correct). Further, while comparing, we made sure that we were using the **best baselines** for proposed methods; for example, we used LA for DASO as their implementation as their implementation (https://github.com/ytaek-oh/daso) recommended it. Hence, to the best of our knowledge, we have **provided fair comparisons**.

> Results with FixMatch

One potential work that uses a similar framework is CoSSL, which is a two-stage framework comprising a pre-training and a fine-tuning stage. For performing fair comparison with SelMix finetuning, we use the same pre-trained model for both for the results below. It can be observed that SelMix finetuning outperforms CoSSL fine-tuning by a large margin.




**Dataset: CIFAR-10, IBR_L = 100, IBR_U = 100, N1=1500, M1=3000** (Table 1)
|                     | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |                   | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |
|-|-|-|-|-|-|-|-|-|-|-|-|
| FixMatch(LA)        | 79.7              | 55.9              | 76.7              | 78.3              | 79.7/0.056        | FixMatch           | 76.9              | 43.5              | 71.9              | 74.5              | 76.9/0.044        |
| FixMatch(LA) + CoSSL | 81.8              | 72.3              | 81.1              | 81.5              | 81.8/0.078        | FixMatch + CoSSL  | 82.6              | 72.0              | 81.9              | 82.1              | 82.6/0.076        |
| FixMatch(LA) + SelMix | 85.4              | 79.1              | 85.1              | 85.3              | 85.7/0.095        | FixMatch + SelMix | 85.4              | 79.5              | 84.2              | 84.3              | 85.4/0.096        |


**Dataset: CIFAR-10, IBR_L = 100, IBR_U = 1, N1=1500, M1=3000** (Table H.1)
|                     | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |                   | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |
|-|-|-|-|-|-|-|-|-|-|-|-|
| FixMatch(LA)        | 90.1              | 75.8              | 89.5              | 89.7              | 90.1/0.083        | FixMatch           | 82.8              | 59.1              | 80.6              | 82.3              | 82.8/0.059        |
| FixMatch(LA) + CoSSL | 89.9              | 75.9              | 89.4              | 89.7              | 89.9/0.085        | FixMatch + CoSSL  | 87.5              | 73.3              | 86.8              | 87.2              | 87.5/0.075        |
| FixMatch(LA) + SelMix | 91.4              | 84.7              | 91.3              | 91.3              | 91.4/0.096        | FixMatch + SelMix | 87.9              | 82.3              | 87.4              | 87.6              | 87.9/0.094        |




**Dataset: CIFAR-10, IBR_L = 100, IBR_U = 0.01, N1=1500, M1=3000** (Table H.1)

|                     | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |                   | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |
|-|-|-|-|-|-|-|-|-|-|-|-|
| FixMatch(LA)        | 75.5              | 45.1              | 71.1              | 73.3              | 75.5/0.046        | FixMatch           | 71.3              | 28.5              | 61.3              | 67.1              | 71.3/0.030        |
| FixMatch(LA) + CoSSL | 77.3              | 64.3              | 75.8              | 76.5              | 77.3/0.070        | FixMatch + CoSSL  | 77.5              | 63.3              | 75.9              | 76.6              | 77.5/0.070        |
| FixMatch(LA) + SelMix | 81.3              | 74.3              | 81.0              | 80.9              | 81.7/0.091        | FixMatch + SelMix | 79.8              | 72.6              | 78.9              | 79.1   | 79.8/0.082        |

Reviewer comment (Reproducibility):
What is the *prior distribution* on labels $\pi_i$ ? How do you define this in practice ?

Author response:
Prior distribution $\pi_i$ over labels $y=1...K$, represents the probabilities of obtaining a sample with a label $y=i$when sampled at random from a dataset before observing any specific data. 
We have obtained the prior distribution estimates, using the label statistics $y$ of the labeled training data present.

Reviewer comment (Writing):
What is the *unconstrained confusion matrix* $\tilde{C}$ ? It is never clearly defined in the paper, but is important to define the gain.

Author response:
Thanks for the interesting question, it actually captures mathematical beauty of the proposed approximation.
The unconstrained confusion matrix $\widetilde{C}$ is implicitly defined in the "Estimation of Gain Matrix" paragraph by $C_ {i} = \pi_ {i} softmax(\widetilde{C}_ {i})$, where $C_ {i}$ and $\widetilde{C}_ {i}$ are $i$-th row vectors of 
$C$ and $\widetilde{C}$, respectively.
Although the matrix $\widetilde{C}$ is not uniquely determined since the softmax function is not one-to-one,
the approximation (the first term of the RHS of eq (8)) is unique as the derivative of all these are unique.
We explain this more in detail.
In the approximation formula, we only need jacobian
$\partial \psi/\partial \widetilde{C} = \frac{\partial \psi}{\partial {C}} \frac{\partial {C}}{\partial{\widetilde{C}}}$.
We note that gradient of the softmax function can also be written as a function of the softmax function (i.e., $\frac{\partial \sigma_l}{\partial \xi_m} = \delta_{lm} \sigma_l(\xi) - \sigma_l(\xi) \sigma_m(\xi)$, where $\sigma_l(\xi) = softmax_l(\xi)$
for $\xi \in \mathbb{R}^K$, $1 \le l, m \le K$).
Therefore, the first term of the RHS of eq (8) is uniquely determined even if $\widetilde{C}$ is not uniquely determined.
We appreciate your suggestion and will clarify this in the final version.

Reviewer comment (Experiments):
I would like to see at least some comparison of SelMix using "vanilla" FixMatch to fairly compare with state of the art.

Author response:
Thanks for pointing out this inconsistency due to **a typo of w/**; we have corrected typos in the revision. We want to clarify that all the methods we have compared in Table H.1 are methods **which are trained from scratch** and are not finetuning methods like the proposed SelMix (hence, w/ is not correct). Further, while comparing, we made sure that we were using the **best baselines** for proposed methods; for example, we used LA for DASO as their implementation as their implementation (https://github.com/ytaek-oh/daso) recommended it. Hence, to the best of our knowledge, we have **provided fair comparisons**.

> Results with FixMatch

One potential work that uses a similar framework is CoSSL, which is a two-stage framework comprising a pre-training and a fine-tuning stage. For performing fair comparison with SelMix finetuning, we use the same pre-trained model for both for the results below. It can be observed that SelMix finetuning outperforms CoSSL fine-tuning by a large margin.




**Dataset: CIFAR-10, IBR_L = 100, IBR_U = 100, N1=1500, M1=3000** (Table 1)
|                     | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |                   | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |
|-|-|-|-|-|-|-|-|-|-|-|-|
| FixMatch(LA)        | 79.7              | 55.9              | 76.7              | 78.3              | 79.7/0.056        | FixMatch           | 76.9              | 43.5              | 71.9              | 74.5              | 76.9/0.044        |
| FixMatch(LA) + CoSSL | 81.8              | 72.3              | 81.1              | 81.5              | 81.8/0.078        | FixMatch + CoSSL  | 82.6              | 72.0              | 81.9              | 82.1              | 82.6/0.076        |
| FixMatch(LA) + SelMix | 85.4              | 79.1              | 85.1              | 85.3              | 85.7/0.095        | FixMatch + SelMix | 85.4              | 79.5              | 84.2              | 84.3              | 85.4/0.096        |


**Dataset: CIFAR-10, IBR_L = 100, IBR_U = 1, N1=1500, M1=3000** (Table H.1)
|                     | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |                   | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |
|-|-|-|-|-|-|-|-|-|-|-|-|
| FixMatch(LA)        | 90.1              | 75.8              | 89.5              | 89.7              | 90.1/0.083        | FixMatch           | 82.8              | 59.1              | 80.6              | 82.3              | 82.8/0.059        |
| FixMatch(LA) + CoSSL | 89.9              | 75.9              | 89.4              | 89.7              | 89.9/0.085        | FixMatch + CoSSL  | 87.5              | 73.3              | 86.8              | 87.2              | 87.5/0.075        |
| FixMatch(LA) + SelMix | 91.4              | 84.7              | 91.3              | 91.3              | 91.4/0.096        | FixMatch + SelMix | 87.9              | 82.3              | 87.4              | 87.6              | 87.9/0.094        |




**Dataset: CIFAR-10, IBR_L = 100, IBR_U = 0.01, N1=1500, M1=3000** (Table H.1)

|                     | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |                   | Mean Rec.         | Min Rec.          | HM                | GM                | Mean Rec./Min Cov |
|-|-|-|-|-|-|-|-|-|-|-|-|
| FixMatch(LA)        | 75.5              | 45.1              | 71.1              | 73.3              | 75.5/0.046        | FixMatch           | 71.3              | 28.5              | 61.3              | 67.1              | 71.3/0.030        |
| FixMatch(LA) + CoSSL | 77.3              | 64.3              | 75.8              | 76.5              | 77.3/0.070        | FixMatch + CoSSL  | 77.5              | 63.3              | 75.9              | 76.6              | 77.5/0.070        |
| FixMatch(LA) + SelMix | 81.3              | 74.3              | 81.0              | 80.9              | 81.7/0.091        | FixMatch + SelMix | 79.8              | 72.6              | 78.9              | 79.1   | 79.8/0.082        |

Reviewer comment (Novelty):
Literature review is not sufficient, and some sentences are over-claimed. Fine-tuning a pretrained network for non-decomposable objectives and even non-differential objectives has been extensively developed recently. Please refer to a recent paper [1] and the references therein for more details. Particularly, the objective derived $\mathbb{E}[G]=\sum_{i, j} G_{i, j} \mathcal{P}_{M i x}(i, \jmath)$ shares the same motivation as the natural evolution strategies [2] adopted in [1].

Author response:
We thank the reviewer for pointing towards a concurrent paper [1], which also tries to fine-tune a model, but it operates in the scenario where the feedback is only accesssible through queries (**black-box**), whereas *our work* SelMix works in a scenario where model gradients are not-computable for certain objectives but can be approximated with gradients on other objectives (**white-box**). We have cited the papers in the additional related work section of our revision. Since the method proposed in [1] is based on a black-box optimization method proposed in [2],
in [1] values of the objective $\psi$ can be observed only through function query, whereas
we assume the objective $\psi$ and its gradients w.r.t the confusion matrix $C$ are approximately known.
**Since our method is a white-box optimization method, it can converge faster than black-box optimization methods such as [1]**.
Because we were not able to find convergence analysis in [1], we compare our result to a standard result [Agarwal et al, 2010]
in the black-box optimization (bandit) literature.
Corollary 4 in [Agarwal et al, 2010] suggests that the convergence rate of the black box optimization method is $O(d^2 K^2/\sqrt{t})$,
while our convergence rate in Theorem 4.2 is given as $O(1/t)$,
which indicates that our approach is less prone to larger values of $d$ and $K$,
thus, our approach achieves better values of the objective compared to the black-box optimization approach [1].

Reviewer comment (Theory):
Some theorems are not formal and have overly strong assumptions. For instance, $z_k$ in Theorem 4.1 appears suddenly without intuitive explanation. There is no detailed explanation in the proof in the Appendix. “a reasonable directional vector for optimization” is not professional.

Author response:
Thank you for your suggestion. We have improved our paper based on your comments (please refer to the submitted revision). 


Although statements of Theorem 4.1 and 4.2 are formal, due to space constraint the statement of Theorem 4.3 is informal. The formal statement of Theorem 4.3 can be found in Section C.4.

The mean feature vector $z_i$ is defined in Theorem 4.1.
We provide an intuitive explanation why this is necessary.
Please note that $\psi$ is a function of the confusion matrix $C[h]$ of the classifier $h(x) = W^T g(x)$ (Sec. 3).
By definition the $(i, j)$-entry of the confusion matrix $C[h]$ is given as 

$$
      \mathbf{E}_ {x, y \sim D}[ 1 (y=i, \mathrm{argmax}_ {l} h(x)_ {l} = j) ]=
       P(y=i) \mathbf{E}_ {x \sim P(\cdot | y=i)}[ 1(\mathrm{argmax}_ {l} h(x)_ {l} = j) ]
$$

Here, $P(\cdot | y = i)$ is the class conditional probability distribution.
If we consider softmax instead of argmax, $C_ {ij}[h]$ is approximately given as 

$$
     C_ {ij}[h] \approx P(y=i) \mathbf{E}_ {x \sim P(\cdot | y=i)}[ \mathrm{softmax}_ {l} h(x)].
$$

Since we are interested in the change in the objective $\psi$ (Gain), by the above formula, we are interested in the change in the mean of the softmax 
of $h(x)$, where $x$ belongs to the class $i$.
Because $h(x) = W^T g(x)$ is a linear classifier, the change in the mean of the softmax is related to the mean feature vector
$z_i = \mathbf{E}_ x [g(x)]$, where $x$ belongs to the class $i$.

In Sec. 4.1, the words “a reasonable directional vector for optimization” appear twice.
The first appearance follows the following inequality and “a reasonable directional vector for optimization.” 
is an intuitive explanation of this inequality for the vector $\widetilde{V}^{(t)}$. We have modified it to "a vector which has a sufficient component in the direction of gradient" for optimization.

$$
\mathbf{E}[\widetilde{V}^{(t)}] \cdot 
        \frac{\partial \psi(W^{(t)}) }{\partial W}
         > c\|\frac{\partial \psi(W^{(t)}) }{\partial W}\|.
$$

Reviewer comment (Writing):
What is the difference between "Non-Decomposable Objective" and "Non-Differentiable Objective"?

Author response:
We would like to clarify that both "Non-Decomposable Objectives" and "Non-Differentiable Objectives" are two orthogonal categories. A differentiable objective is characterized by its ability to have a computable derivative with respect to the model's parameters and vice-versa. For instance, the $L_2$ error serves as a prime example of a differentiable objective commonly employed in regression tasks. In the realm of classification, the majority of objectives are non-differentiable, owing to their reliance on the indicator function $I_{y=y'}$. This function introduces non-differentiability due to its discrete nature. On the other hand, a decomposable objective is one that can be expressed as the mean or sum of smaller objectives. In the context of a classifier, these smaller objectives often hinge on the label-prediction pairs, as expressed by the following formula: $\psi  = \frac{1}{N} \sum_{i=1..N}F(y,y')$. Accuracy is a decomposable objective since $Acc = \frac{1}{N} \sum_{i=1..N}I_{y=y'}$ where $I_{y=y'}$ is the accuracy for the individual sample, yet it is non-differentiable. Although most useful objectives are both non-decomposable and non-differentiable, one does not imply the other as shown in the following table.

Table 1: Examples of objectives based on their decomposability and differentiability
|  | Decomposable | Non-Decomposable |
|----------|----------|----------|
| **Differentiable** | Mean $L_2$ error | H-mean of individual $L_2$ errors |
| **Non-differentiable** | Accuracy |  G-mean of Recall |

Reviewer comment (Theory):
It seems we can fitting a surrogate model to approximate the Non-Decomposable Objective, which can then be used for gradient-based model fine-tuning. It would be interesting to discuss the advantages of the first-order Taylor approximation compared to fitting a surrogate model.

Author response:
Thanks for an interesting question. We want to convey that all existing heuristic-based techniques like ABC, DASO, etc. utilize a surrogate loss to approximate the performance of the models on LT data. 
For instance, we note that the loss function considered by ABC (Lee et al., 2021) can be regarded as a surrogate loss for the mean recall across classes (we explain this more in detail in the "loss proposed by ABC" paragraph below).

We have shown in Table 1 and Fig. 3 *that our proposed method, SelMix, outperforms these heuristic surrogate-based methods (ABC, DASO etc.)*. 

Further, developing a surrogate function for each metric is hard, whereas using any metric with SelMix is straightforward. 

In the SelMix framework, it is easy to combine constraints, like coverage, into an optimization problem. Whereas it is hard to come up with surrogates which also incorporate constraints. 

We will discuss these in the final version of the draft; thank you for the suggestions.