Paper: RelationMatch: Matching In-batch Relationships for Semi-supervised Learning (ICLR-2024)

Reviewer comment (Writing):
My small comment concerns the details of the writing, especially the notations. There may be some typos or unclear statements.

Author response:
Thank you for your careful reading and suggestions; we have resolved these issues in the new submission.

Reviewer comment (Writing):
In section 2.1, it should read $\log q_1$ instead of $\log q_i$.

Author response:
Thank you for your careful reading and suggestions; we have resolved these issues in the new submission.

Reviewer comment (Writing):
In section 2.1, what's Eq.(2.1)?

Author response:
Thank you for your careful reading and suggestions; we have resolved these issues in the new submission.

Reviewer comment (Writing):
In Eq. 4, is $\tilde{Y}_s$ the model output of weak augmentations? In a line above it is written as $\tilde{Y}$. If they mean the same thing, the notation should be the same.

Author response:
Thank you for your careful reading and suggestions; we have resolved these issues in the new submission.

Reviewer comment (Writing):
In Definition 4.2, it should be $n=0$ instead of $i=0$.

Author response:
Thank you for your careful reading and suggestions; we have resolved these issues in the new submission.

Reviewer comment (Writing):
The author should check the notation carefully.

Author response:
Thank you for your careful reading and suggestions; we have resolved these issues in the new submission.

Reviewer comment (Reproducibility):
Since MCE can be simplified with a $l_2$-normalized matrix, what is the loss function used in the experiments? Is it still equation (1) with a non-normalized matrix?

Author response:
Thank you for your questions. In the experiment, in order to prevent errors caused by some unknown mistakes, we used equation(1) + row l2-normalized $X$ (recall $Q$ = $\frac{1}{b} X X^T$), which makes equation(2) equivalent to equation(1). However, for the sake of caution, we still used equation(1) as our loss in the experiments.

Reviewer comment (Novelty):
My primary concern pertains to the paper's novelty. SimMatch[1] has previously addressed the relationship between data points by applying consistency regularization at both the semantic and instance levels, promoting identical class predictions and maintaining similarity relations with other instances for different augmentations of the same instance. A detailed discussion and comparison between SimMatch and RelationMatch are essential to elucidate the distinct contributions of the latter.

Author response:
- Our approach is mainly a natural extension of CE (Cross-Entropy), and we find out that when using one-hot pseudo-labels in semi-supervised settings, our new method can be understood as capturing the relation. SimMatch, on the other hand, utilizes contrastive loss for consistency regularization of the relation, which differs from our starting point.

- One point that we need to emphasize is that our method has very good theoretical properties (as shown in Section ...), and at the same time, our method can be easily applied to most existing methods. Therefore, for the sake of fairness, we will not consider using SimMatch as a baseline for performance comparison in the current version. We will include SimMatch as a baseline when we have sufficient computational resources to conduct experiments applying our method to SimMatch.

  We have added these two discussions in the appendix (Section C).

Reviewer comment (Experiments):
The benchmark comparison appears outdated. The most recent method evaluated in the paper is from 2021, and although the authors mention some methods from 2022 and 2023, such as FreeMatch, MaxMatch, and NP-Match, these have not been included in the experimental comparisons. When compared with the latest methods, RelationMatch does not seem to meet the state-of-the-art standard.

Author response:
Thank you for your careful reading and suggestions. Due to the shortage of computational resources, we only had time to conduct complete experiments on FixMatch / FlexMatch + MCE.

 During the rebuttal period, we attempted to apply our method to FreeMatch. Due to the scarcity of computational resources, we are only able to finish experiments on CIFAR10 and STL-10. The results are shown in the following table, and it can be seen that our method has a significant improvement on the task of CIFAR10 with 40 labels. 

| Metric | CIFAR10 40 | CIFAR10 250 | CIFAR10 4000 | STL-10 40 | STL-10 250 | STL-10 1000 |
|--------|------------|-------------|----------------|-----------|------------|-------------|
| *FreeMatch* | 4.90 ±0.04 | 4.88 ±0.18 | 4.10 ±0.02 | 15.56 ±0.55 | - | 5.63 ±0.15 |
| *RelationMatch (w/ self-adaptive threshold)* | 4.80 ±0.06 | 4.89 ±0.14 | 4.08 ±0.05 | 15.60 ±4.09 | 10.42 ±0.78 | 5.62 ±0.18 |


What we need to point out is that our core idea is to propose a new loss, MCE, which is a universal enhancement that can be applied to most semi-supervised learning algorithms. We will supplement the experimental results with more advanced methods + MCE as soon as we have sufficient computing resources. Our current implementation of FixMatch / FreeMatch + MCE, when compared to the recent FreeMatch, MaxMatch, and NP-Match, is still not entirely fair.

Reviewer comment (Presentation):
Figure 1 can be improved. There are too many lines, which are confusing.

Author response:
Thank you for your suggestion, we will fix this issue in the next submission. We deeply apologize for any confusion caused to you.

Reviewer comment (Experiments):
Large dataset experiments are missing, e.g., ImageNet.

Author response:
Thank you for your suggestion. We will carry out the ImageNet-related experiments as soon as we have enough computing resources. However, we would like to emphasize that our method has performed very well on our existing benchmarks, especially when labels are scarce. For instance, there is an improvement of 15.21 on STL with 40 labels, an increase of 0.6 (FixMatch) on CIFAR10 with 40 labels, and a rise of 0.63 (FixMatch) on CIFAR100 with 40 labels.

Reviewer comment (Experiments):
Ablation studies on $\mu$ and $\gamma$ are missing.

Author response:
Thank you for your suggestion. Please note that we have already emphasized in the original text that our hyperparameter selection is made to ensure that the coefficient of the cross-entropy loss in the loss function is consistent with that of the baseline, in order to make a fair comparison.

Reviewer comment (Writing):
Formulations and notations are not clear. What's the definition of $Y_s$ and $X_s$ in eq(4)?

Author response:
Thank you for your question. We have made changes to the manuscript to make the equation clearer.

Reviewer comment (Writing):
How MCE connect with Relation in the introduction?

Author response:
Thank you for your question. We have discussed in Section 2.1 that the relation matrix can be easily generated by the predicted probability. Note MCE has minimization property(Proposition 4.8), so it can match the relation matrix of weakly and strongly augmented images from RelationMatch loss given in equation (4).

Reviewer comment (Theory):
MCE(P, Q) = tr(−P log Q + Q). For matrix cross-entropy, why +Q?

Author response:
This is for better theoretical properties for matrix Q when it is a positive semi-definite matrix and not a matrix whose trace is 1, we generalize the probability distribution defined in the density matrix (the standard result used in quantum information theory) into any positive semi-definite matrix, with multiple theoretical guarantees.

Reviewer comment (Experiments):
The experiment appears somewhat insufficient, as only two experiments were conducted in the main text, and they were tested on just two to three datasets. Additionally, I am curious as to why the STL-10 dataset was omitted from Table 1.

Author response:
Thank you for your suggestions; we will add more experiments in the future version. As for the omission of the STL-10 dataset in Table 1, this is mainly because STL-10 essentially is not suited for supervised training, since the majority of images it contains are unlabeled, and within the labeled images, each class has 800 test images and 500 train images, which is clearly an unbalanced ratio for conducting supervised experiments.

Reviewer comment (Evaluation):
Based on the results presented in Table 1, the displayed accuracy results show limited differentiation. The matrix cross-entropy outperformed by a margin of less than 0.3%. This could potentially be attributed to randomization and perturbations.

Author response:
Thank you for your careful reading and questions. In order to minimize the impact of randomness, we have repeated the experiment **3** times and reported the means and variances. Additionally, we must point out that, in supervised learning, when the baseline accuracy has already surpassed 95%, even an improvement of 0.3% is very significant.

Reviewer comment (Evaluation):
Potential failure modes or limitations not discussed.

Author response:
Thank you for your suggestion. We will try to incorporate potential failure modes as well as limitations after a thorough investigation.