Paper: EXCOST: Semi-Supervised Classification with Exemplar-Contrastive Self-Training (ICLR-2024)

Reviewer comment (Novelty):
Integrating the concept of contrastive learning into semi-supervised learning is not a novel approach, and several similar studies have been conducted previously [1,2,3,4]. The authors should analyze and experimentally compare their method with these previous approaches.

Author response:
Thank you very much for providing the references. We will include an introduction to [1,2,3,4] in the related work section of the next version.

Reviewer comment (Novelty):
As mentioned in the previous question, the Related Work in this paper is not comprehensive, lacks a summary and analysis of prior work, and fails to highlight the contributions of this paper.

Author response:
Thank you once again for pointing out our issue. We will include [1,2,3,4] in our related work section to provide a more comprehensive summary of previous work.

Reviewer comment (Experiments):
The experimental section lacks an introduction to baseline methods, and many experimental results are missing from Table 1. It would be preferable for the authors to mark the missing results with horizontal lines.

Author response:
We used the raw data from the original papers of all baseline models in Table 1. Since this data was missing in the original papers, we left those parts blank to ensure the accuracy of the information. We appreciate your valuable suggestion. We will take it into account and use horizontal lines to fill in the blank parts.

Reviewer comment (Writing):
The nomenclature in this paper should be more standardized. For instance, do ${\Phi _c}$ and ${\phi _c}$ represent the same thing?

Author response:
Thank you very much for your correction. We will investigate the issue of symbol standardization. $\Phi_C$ and $\phi_C$ represent different entities, where $\Phi_C$ is a function that generates a real value $\phi_C$ as its dependent variable based on the progress of training. Similar cases include $\Psi$ and $\psi$, as well as $\Phi_S$ and $\phi_S$. We will provide further clarification on their relationships in Sections 3.1 and 3.2.

Reviewer comment (Writing):
The article is difficult to understand, and the authors should restate the motivation of this paper and clarify the relationship between motivation and the proposed method. For example, how are prototype theory and exemplar theory reflected in the methodology, and what role does the typicality gradient play in this paper?

Author response:
Thank you very much for your suggestions, and I will make an effort to improve the clarity of this paper. Below, we will address the questions you raised:

The motivation of this paper is to simulate the human classification process at the psychological level. Therefore, we utilize the concepts of prototypes and exemplars, combining their strengths to obtain more reliable pseudo-labels. In the labeling algorithm of EXCOST, we always select the sample that is most in line with the prototype (considering the attribute-frequency theory [5,6], i.e., the sample with the highest confidence score from the network output) and has the highest similarity to the exemplars to assign pseudo-labels. The number of selected samples is controlled by the functions $\Phi_C$ and $\Phi_S$, which are monotonically increasing functions with respect to training progress (ranging from 0 to 1). Consequently, the model can utilize more pseudo-labels as the training progresses. This process is analogous to human learning, where the ability to make accurate classification decisions increases as one becomes more familiar with a certain category, while it is limited when facing less familiar instances.

The concepts of prototypes and exemplars are reflected in the labeling algorithm. For the prototype theory, we adopt the attribute-frequency theory, which we believe is equivalent to the confidence generated by the neural network. This is because neural networks optimize through the backpropagation algorithm, and the connections between features that frequently co-occur with a particular class label are strengthened. Therefore, we can infer that the more frequently a feature co-occurs with a class label, the higher the corresponding class probability will be. For the exemplar theory, we treat samples with true labels as exemplars and determine whether they belong to the same category by comparing the similarity between samples and exemplars.

The typicality gradient is mainly reflected in Category-Invariant Loss (CIL). Unlike the FixMatch algorithm [7], which uses hard pseudo-labels, CIL uses class probabilities generated from the original samples instead of using one-hot vectors. Consider the following scenario: if there is no label information in the training set, the model cannot generate class probabilities for samples through learning. Therefore, it can be inferred that the class probabilities of unlabeled samples are gradually learned through the guidance of labeled samples. According to the attribute-frequency theory, all supervised learning on labeled samples will cause the most confident samples to be closest to the prototypes. Thus, the class probability vector can be regarded as a representation of the typicality of a sample's category. We hope that the inputs with data augmentation can be close to this representation, rather than approaching the one-hot vector (losing typicality). Therefore, we believe that CIL contributes to the preservation of typicality gradients. 

For samples that are ambiguous in classification, when there is not enough information for the model to make a decision, we hope these samples can stay as close as possible to their original positions instead of actively moving towards the center of any category. The benefit of doing so is that when the model obtains more information about these ambiguous samples in the future, it can learn faster and avoid making excessive efforts to correct mistakes. Moreover, preserving the typicality gradient also provides basic information for distinguishing intra-class members. For example, penguins belong to the bird category, but we can all recognize their differences from other birds. Of course, in reality, our algorithm still falls short of achieving this goal, mainly because we artificially set functions ($\Phi_C$ and $\Phi_S$) to control the labeling rate and force the model to make choices at specified moments. However, in future work, combining with active learning or reinforcement learning may achieve a more human-like psychological behavior.


We have taken careful note of your comments and are committed to making the necessary revisions. Should you have any further inquiries or require additional clarification, we welcome the opportunity for continued discussion, and look forward to addressing your concerns in our upcoming revisions. Thank you once again for your constructive feedback.

Reviewer comment (Writing):
The motivation is not clear. Why combine prototype and exemplar information to improve the quality of pseudo-labels? The first paragraph does not explain well the role of this combination in semi-supervised learning.

Author response:
Thank you for your reminder. We may not have adequately explained why we took this approach. Our primary motivation is to guide semi-supervised learning from a psychological perspective by simulating the way humans categorize things. In cognitive psychology, prototypes and exemplars are used to guide human categorization, each having its own advantages and disadvantages. Specifically:

- Prototypes can be considered as the average of features within a category, and the individuality of different samples is lost through this averaging operation. Therefore, prototypes retain the commonalities of samples within that category, making samples with these commonalities more easily discernible as belonging to that category.
- Exemplars are each entity we encounter, and these entities carry comprehensive information, thereby retaining the individuality of each entity. These individual characteristics may not be common features across the entire category, but they may play a role in classification, especially when categorizing atypical samples.
- Assessing atypical members of a category through prototypes can be challenging. People may question whether "penguins belong to the bird category" because penguins are atypical members of birds and do not possess all the emphasized features of birds, such as the ability to fly.
- Exemplars retain too many detailed features, making them less generalizable than prototypes.

We can illustrate this with a practical example using the MNIST dataset: The digit 7 has some distinctive variations in its writing, with a horizontal line often present in the middle position. If we average the features of all samples belonging to the class of digit 7, this specific horizontal line may be overlooked as noise due to its infrequent occurrence. Depending on whether this distinctive variation is randomly selected as an exemplar (labeled sample), we will discuss the following two scenarios:

1. If a sample with this specific variation is randomly chosen as an exemplar, the features of this exemplar will be quite similar to the features of all other samples with the same distinctive variation. This ensures that digits with this special writing style of 7 will not be misclassified into the wrong category.
2. If a 7 with this specific writing style is not selected as an exemplar, it still ensures that these samples will not be prematurely classified into incorrect categories, as their features are dissimilar to any other category's exemplars. As the model matures through learning and with the increase in labeled data, there is a higher probability that the model will correctly classify these samples when forced to do so. This is akin to the human tendency to follow the principle of starting with easier tasks before tackling more challenging ones. In fact, when this special writing style of 7 is not included as an exemplar, there is not sufficient reason to assert that it belongs to the category of 7; the attribution of this writing style to the category of 7 is, in fact, a human-defined convention. Therefore, even in teaching humans, it would be appropriate to showcase this specific style of writing 7 to illustrate this point.

In summary, we aim to integrate prototypes and exemplars to jointly constrain the process of generating pseudo-labels, seeking to obtain higher-quality pseudo-labels and thereby reduce the emergence of confirmation bias. Thank you once again for your guidance, and we will revise the first paragraph in the next version to enhance the clarity of our motivation.

Reviewer comment (Novelty):
The novelty of this manuscript seems to be limited. The authors should discuss the relation between existing methods and the proposed method in details.

Author response:
Thank you very much for your valuable comments on our paper. Regarding the issue of the manuscript's innovativeness that you raised, we fully understand your concerns. We would like to emphasize here that, while you may perceive its innovativeness to be limited, we firmly believe that our approach exhibits unique innovation in specific aspects. The inspiration for our proposed method is drawn from relevant theories of category representation in cognitive psychology, with the incorporation of machine learning techniques such as self-training, consistency regularization, and contrastive learning. To illustrate the connections and distinctions between our model and the following models, we provide examples:

- Both UDA [1] and EXCOST incorporate the idea of consistency regularization. UDA calculates the cross-entropy between the predicted distributions of augmented samples and original samples as a form of consistency regularization loss. On the other hand, EXCOST achieves consistency by calculating the Category-Invariant Loss (CIL).
- Both MPL [2] and EXCOST draw inspiration from the idea of self-training. The key idea involves the teacher network continuously learning from the feedback of the student network to generate pseudo-labels that best assist the learning of the student network. In the case of EXCOST, reliable pseudo-labels are generated by combining constraints between prototypes and exemplars.
- FixMatch [3] and EXCOST both incorporate the ideas of self-training and consistency regularization. In the training process, FixMatch generates a prediction for each unlabeled sample through "weak augmentation." If the class probability in the prediction exceeds a threshold, the prediction is transformed into a "hard" pseudo-label. On the other hand, EXCOST generates pseudo-labels during the labeling phase by referencing confidence-based ranking (prototype) and exemplar-similarity-based ranking (exemplar). FixMatch requires the predictions of "strongly augmented" and "weakly augmented" for the same sample to be consistent, utilizing "hard" pseudo-labels and cross-entropy loss. EXCOST employs Category-Invariant Loss (CIL) to ensure consistency and emphasizes retaining typical gradients of classes.

Our approach not only aims to achieve better results in semi-supervised image classification tasks but also places a strong emphasis on simulating human behavior at a psychological level. Therefore, in our subsequent work, we will continuously improve the model to make it more similar to humans. The ultimate vision is to realize a human-like category representation system, providing abstract category concepts for artificial general intelligence.

However, to achieve this goal, we first need to establish a perceptual environment for the model that is similar to the environment encountered by humans, including the form of data and the way information is interacted with. This cannot be limited to semi-supervised image classification tasks because, in the visual aspect, humans encounter continuous scenes rather than static images. In terms of interaction, the task of semi-supervised image classification prescribes a passive way of information interaction, learning through limited information. Humans, on the other hand, usually ask questions when encountering unfamiliar things, and errors in understanding can be corrected by the environment.

Since the data augmentation in EXCOST does not restricted to a specific form (which sets it apart from MixMatch [4] and FixMatch), it is compatible with any form of data augmentation. We consider consecutive frames in a video constitute a "natural" form of data augmentation. The motion changes in these frames contain a lot of important information, forming the basis for the model to understand the real world. Our model aims to construct a category representation system at the psychological level, making it more interpretable.

Reviewer comment (Experiments):
The authors claim that as training progresses, more pseudo-labeled samples are incorporated into the learning process. However, there are no results to prove the point.

Author response:
Since both scheduling functions regarding the labeling rate (i.e., $\Phi_C$ and $\Phi_S$) are monotonically increasing, the labeling rate will gradually rise as the training progresses. This ensures that more pseudo-labels are incorporated into the learning process. Due to the relative obviousness of this conclusion and constraints on space, we have opted not to provide a formal proof.

Reviewer comment (Experiments):
The ablation study is confusing. Moreover, most of the results in Table 1 remain blank.

Author response:
We conducted rigorous ablation experiments on the MNIST dataset, thoroughly exploring various combinations of hyperparameters under different conditions to obtain comprehensive experimental results.

The results in Table 1 are sourced from the original data of baseline models' papers. Since the original data did not include this specific information, there are blank entries. We did not alter any original data to ensure its accuracy. For instance, we did not supplement results for the baseline models with blanks for MNIST, primarily because we couldn't ascertain the optimal hyperparameter settings for these models on this dataset. For example, for FixMatch and its improved models, determining the "weak augmentation" and "strong augmentation" seems less straightforward.

Reviewer comment (Reproducibility):
Some experimental details are missing, e.g., the threshold in Eq. (1).

Author response:
Thank you very much for pointing out our oversight. The symbol $\psi$ is generated by the function $\Psi$, and similarly, $\phi_C$ and $\phi_S$ are produced by the functions $\Phi_C$ and $\Phi_S$, respectively. We will supplement explanations in Sections 3.1 and 3.2 to clarify the relationships among them.

Reviewer comment (Evaluation):
Effectiveness: While the concept is interesting, the paper falls short in terms of performance when compared to SOTA methods. Notably, the proposed method exhibits considerably higher error rates on CIFAR-100 in comparison to other techniques.

Author response:
We appreciate your interest in the method proposed in this paper. From Table 1, it can be observed that except for the CIFAR-100 dataset, EXCOST achieves state-of-the-art (SOTA) or comparable performance levels on tasks such as MNIST, CIFAR-10, and SVHN (notably, these results were obtained using the Adam [1] optimization algorithm). However, it is important to note that our experimental conditions were quite limited, with only 4 RTX 3090 GPUs available. Therefore, it was challenging to identify the optimal hyperparameters, especially those related to the optimizer and data augmentation, which can have a significant impact on the results. Since our approach does not directly build upon existing mature models, we had to explore the hyperparameters from scratch. This is particularly challenging when computational resources are limited. However, the scarcity of hardware resources is a common situation faced by many researchers, which greatly reduces the possibility of achieving better experimental results. For the CIFAR-100 dataset, it took approximately 76 hours to validate a set of experimental configurations using one RTX 3090 GPU. Due to resource limitations, we could only adjust the parameters of data augmentation based on the concept of not severely distorting the images, which restricted our ability to discover better results on CIFAR-100.

Additionally, it is worth noting that, following common experimental practices, we provide an equal number of labeled samples for all categories. Specifically, both the training and testing datasets of CIFAR-100 are uniformly distributed across all classes. Therefore, techniques such as Distribution Alignment (DA) from ReMixMatch [2] or self-adaptive fairness (SAF) from FreeMatch [3] yield better results on CIFAR-100. However, their effectiveness on datasets with non-uniform class distributions remains uncertain. For we have abandoned this assumption, we cannot leverage these methods. In Section 4.1 of the FixMatch [4] paper, the third paragraph illustrates the benefits of DA on CIFAR-100 results.

Reviewer comment (Presentation):
Writing Quality: The clarity of the paper's presentation could be improved. For instance, in Table 1, there's a mix of reporting styles, with some methods providing the median error rate of the last 20 epochs, while others report the minimum error rate across all epochs. This inconsistency makes it challenging to make direct comparisons.

Author response:
We apologize for not clearly expressing ourselves and causing you to have a bad reading experience. We will make every effort to improve the clarity of our paper. It should be noted that in Table 1, we did not reproduce the baseline model but directly quoted the results from the original paper. This decision was based on two main considerations:

1. We did not have sufficient computing resources to reproduce all the models.
2. The authors of the original paper have the deepest understanding of the models they designed, and we trust the results they have achieved. Additionally, the evaluation methods in these papers vary (i.e., some papers report the minimum error rate of all epochs, while others report the median error rate of the last 20 epochs, and some papers did not provide relevant explanations about the error rate). Therefore, we believe it is necessary to make this distinction clear.

Reviewer comment (Presentation):
Experiment Setup and Results: To enhance the paper's organization, it's advisable to present the primary results in the main paper, while relocating additional details, like the results of the ablation study, to the appendix. This will streamline the main paper and maintain a focused narrative.

Author response:
Thank you for your excellent suggestions. We have taken your advice on board, and in the next version of the paper, the ablation study will be moved to the appendix.

Reviewer comment (Experiments):
One potential limitation of the Category-Invariant Loss (CIL) is its sensitivity to the choice of the threshold parameter. The text acknowledges that the threshold is introduced to manage the influence of irrelevant or outlier samples. However, determining the appropriate threshold value may not be straightforward and can significantly impact the loss function.

Author response:
Thank you for raising this question. In the ablation experiment (Table 3), we were able to compare the cases where the threshold gradually increased from $\frac{1}{10}$ to approximately 0.9 (denoted as $\mathcal{S}_1$) and where the threshold was constantly set at 0 (denoted as $\mathcal{S}_2$, which effectively corresponds to the range of $[0, \frac{1}{10}]$ for a 10-class problem) on the MNIST dataset. In terms of the minimum error rate among all epochs, the average error rate for $\mathcal{S}_1$ was $0.339$, and for $\mathcal{S}_2$, it was $0.449$. In terms of the median error rate of the last 20 epochs, the average error rate for $\mathcal{S}_1$ was $0.535$, while for $\mathcal{S}_2$, it was $0.839$. From this, it can be seen that the threshold does not significantly affect the model's learning but has a slight impact on the stability in the later stages of training.

Reviewer comment (Experiments):
The exemplar-contrastive self-training algorithm involves several hyper-parameters, including thresholds, margin values, and labeling rates. The sensitivity of these hyper-parameters could present challenges during practical implementation, as selecting the right values may require thorough experimentation and tuning.

Author response:
This is a great question, and it's a challenge faced by many deep learning models. In addition to the parameters related to data, backbone network, and optimizer that are common to all deep learning methods, EXCOST introduces a total of 5 parameters. These are $w_u$ (weight for CIL), $\delta$ (margin), and three scheduling functions: $\Psi$ (threshold for CIL), $\Phi_C$ (labeling rate based on confidence), and $\Phi_S$ (labeling rate based on exemplar similarity).

Let's also consider the parameter settings of other semi-supervised learning models: FixMatch [1] has 4 parameters: decay of EMA, $\lambda_u$ (unlabeled loss weight), $T$ (temperature), and $\tau$ (confidence threshold); MixMatch [2] has 5 parameters: decay of EMA, $\lambda_u$ (unsupervised loss weight), $\alpha$ (parameter for MixUp), $T$ (temperature), and $K$ (number of augmentations). Therefore, in terms of quantity, EXCOST does not involve an excessive number of hyperparameters.

For $\delta$ and $\Psi$, we use the same settings across all datasets, without the need for adjustments. Due to inherent differences in datasets and the lack of feedback information on the learning status in our algorithm, we need to set the labeling rate scheduling functions ($\Phi_C$ and $\Phi_S$) based on the dataset. This is similar to how humans exhibit variations in learning across different tasks. In future work, we aim to address this issue by incorporating active learning or reinforcement learning to mimic human-like learning in classification tasks.

It is worth noting that for two significantly different datasets, MNIST and CIFAR-10, even when employing completely identical settings for the 5 mentioned hyperparameters, EXCOST consistently achieves excellent results. Furthermore, our results, except for CIFAR-100, are obtained using the Adam [3] optimization algorithm, rather than meticulously tuning parameters with the SGD optimizer.