Paper: Enhancing Contrastive Learning for Ordinal Regression via  Ordinal Content Preserved Data Augmentation (ICLR-2024)

Reviewer comment (Reproducibility):
The detailed implementation of mask is not explained. It is important to clarify whether it's a category-wise operation or not.

Author response:
**Q7.Is the mask operation category-wise? If not, how does the model select whether or not to treat hair as ordinal information based on different age groups?**

As shown in Equation 2, $f_y$ is a category-specific transformation function, encoding categorical information. A uniform mask is applied, aiming to learn which latent dimensions correspond to ordinal content factors and which to non-ordinal factors under the minimal change constraint.

In the following, we provide a detailed explanation of the operations performed in Equation 2 for further clarification.

In Equation 2, our approach starts by projecting the input latent variable $\hat{z}$ into a category-specific distribution, achieved through the transformation function $f$. This results in a projected latent variable that is distinctly different from the input and is specific to a given category.

Next, we aim to isolate the dimensions associated with ordinal content. To accomplish this, we employ a mask operator, denoted as $M$. This operator is a learnable matrix, designed to match the shape of the input noise. Our adherence to the principle of minimal change is reflected in the way we constrain the mask $M$ to be sparse, ensuring that non-ordinal dimensions in the product are set to zero. We calculate the Hadamard product of $M$ with the projected input, effectively isolating the relevant dimensions.

The final step involves reintegrating style information. This is done to compensate for the zeroed elements in the latent variables. Consequently, the resulting processed latent variables, designated as $\hat{z}_{on}$, are effectively disentangled, separating ordinal content from other information.

Reviewer comment (Experiments):
Image augmentation is not necessary in supervised contrastive learning. The article lacks comparisons with contrastive learning without image augmentation and with contrastive learning using conventional image augmentation. The benefits of additional image augmentation are unclear.

Author response:
**Q2. Image augmentation is not necessary in supervised contrastive learning.** 

Thank you for your insights, We agree with the observation that the performance enhancement from strong image augmentation in supervised contrastive learning may not be as important as in unsupervised learning.

However, strong augmentations remain advantageous in supervised contrastive learning contexts. As evidenced in previous studies (Khosla et al., 2020), omitting strong image augmentations leads to noticeable performance degradation. To further demonstrate the value of data augmentation in our specific task, we will include a comparative analysis in the updated version of our paper. This analysis will contrast the performance of contrastive learning with and without data augmentations.

Reviewer comment (Experiments):
The article does not conduct experiments based on conventional GANs and does not demonstrate the superiority of the proposed generation methods.

Author response:
**Q3. The article does not conduct experiments based on conventional GANs and does not demonstrate the superiority of the proposed generation methods.**

Thank you for the suggestions. We will include comparison to conventional GANs in experiments for the updated version.

Reviewer comment (Novelty):
The proposed method is only appropriate for data augmentation in the supervised scenario, while most of the contrastive learning methods that have a strong dependence on augmentation are unsupervised.

Author response:
**Q4. The proposed method is only appropriate for data augmentation in the supervised scenario, while most of the contrastive learning methods that have a strong dependence on augmentation are unsupervised.**

Our focus is on the problem of ordinal regression, typically involving pairs of images and labels. A significant challenge in this domain is the subtlety of features in the original images that are crucial for determining ordinal classes. These features can become distorted through strong data augmentations in contrastive learning. Hence, our proposed method is specifically designed to tackle this issue in the context of ordinal regression, rather than addressing challenges in arbitrary unsupervised learning scenarios.

Reviewer comment (Theory):
How does minimal change ensure that content information related to ordinal is preserved?

Author response:
**Q6.  How does minimal change ensure that content information related to ordinal is preserved?**

Our method is grounded in the theoretical framework of minimal change (Kong et al. 2022), which is instrumental in achieving disentanglement. According to the theoretical insights from prior research, our method's applicability to ordinal regression becomes feasible. The principle implies that, given a sufficient number of images from different classes, the component $\hat{z}_o$ can be identified on a component-wise basis. This indicates that $\hat{z}_o$ undergoes a simple component-wise transformation solely from $z_o$. Consequently, $\hat{z}_o$ is devoid of information about the style component $z_n$, as it remains unaffected by $z_n$, thus achieving disentanglement.

The following provides the intuitive explanation about disentanglement:

- We categorize the latent factors into two groups: ordinal ($\hat{z}_o$) and non-ordinal ($\hat{z}_n$). In this framework, $\hat{z}_o$ serves as the repository for crucial ordinal content information that determines the ordinal category.

- For reconstruction, we utilize both the predicted ordinal factors ($\hat{z}_o$) and the predicted non-ordinal factors ($\hat{z}_n$). $\hat{z}_o$ is specifically employed for predicting ordinal labels and is subjected to minimal change, restricting its influence in generating new instances.

- Assuming the generative model attains minimal reconstruction error, $\hat{z}_o$, the factor designated for ordinal label prediction, effectively captures the ordinal content. By constraining it through the principle of minimal change, it encapsulates only the vital information pertinent to ordinal labels, while $\hat{z}_n$ accommodates other types of information.

- This approach leads to a clear disentanglement of ordinal and non-ordinal information, enhancing the model's effectiveness.

Reviewer comment (Reproducibility):
Is the mask operation category-wise? If not, how does the model select whether or not to treat hair as ordinal information based on different age groups?

Author response:
2. How the class-agnostic mask handles the problem of ordinal content information in some age groups but not ordinal content information in some age groups is not explained clearly.

The ordinal content information is handled by the category-specific transformation function $f_y(\cdot)$, where the specific $y$ is provided when doing augmentation. The mask operation will preserve such categorical information that differs from groups to groups.

Reviewer comment (Experiments):
Can you provide more generated images? The two sets of images in Figure 5 are not from the same age group. In real life, most infants do not have hair and most children have. If hair is not ordinal information, it may be more beneficial for improving classifier performance if part of the generated images have hair while others do not. What is your opinion?

Author response:
**Q8.Can you provide more generated images? The two sets of images in Figure 5 are not from the same age group. In real life, most infants do not have hair and most children have. I think that if hair is not ordinary information, it is more beneficial to improve the performance of the classifier if part of the generated images have hair while others do not have hair. What's your opinion?**


We have provided additional generated images in the supplementary materials. 

It is important to clarify that Figure 5 demonstrates the effect of disentanglement using the principle of minimal change. The two sets of images shown are from the same age group. With the influence of minimal change, the model preserves the 'no hair' feature for children, as evident in the second row. However, in the absence of this constraint, it becomes challenging to identify which dimensions should remain unchanged. Consequently, without applying the minimal change principle, 'hair' is incorrectly added to children in the post-augmentation images.

Reviewer comment (Experiments):
The experimental results appear to be based on a single run, and the author should consider conducting multiple experiments to reduce the influence of randomness. Notably, the performance on the MWR in Table 1 seems to exhibit minimal variation. The author should provide clarification regarding whether this consistency is a result of chance or if there are specific underlying reasons. Additionally, the author should explore novel ways to demonstrate the effectiveness of the proposed method.

Author response:
**Q1. The experimental results appear to be based on a single run, and the author should consider conducting multiple experiments to reduce the influence of randomness. Notably, the performance on the MWR in Table 1 seems to exhibit minimal variation. The author should provide clarification regarding whether this consistency is a result of chance or if there are specific underlying reasons. Additionally, the author should explore novel ways to demonstrate the effectiveness of the proposed method.**

We would like to clarify that the results presented are derived from multiple runs or cross-validation, and we will include the standard deviations in the updated version. Regarding performance, we observed that MWR (Shin et al., 2022) shows relatively less improvement compared to other ordinal regression methods.

Our hypothesis for this observation is that the ranking module in MWR inherently incorporates elements of contrastive training. This inherent feature possibly diminishes the benefits of explicit contrastive representation learning. Specifically, MWR's objective aligns with a generalized form of supervised contrastive regression (Zha et al., 2023), utilizing label orders to construct positive and negative pairs for training.

In addition, we have rigorously evaluated the model's performance across a variety of benchmark datasets, consistently observing improvements in performance across all datasets and for every model variant tested. This also highlights our significant contribution to the field of ordinal regression.

Reviewer comment (Experiments):
Formula 4 in the paper lacks sensitivity analysis for "lambda1."

Author response:
**Q3. Formula 4 in the paper lacks sensitivity analysis for "lambda1."**

Thank you for highlighting this aspect. We have updated the paper with sensitivity analysis for $\lambda_1$ in the appendix section.

Reviewer comment (Experiments):
Ablation experiments should be conducted to effectively demonstrate the impact and effectiveness of the proposed method.

Author response:
**Q4. Ablation experiments should be conducted to effectively demonstrate the impact and effectiveness of the proposed method.**

Thank you for your valuable suggestion.

- We have incorporated a sensitivity analysis of $\lambda_1$ into our ablation study.

- To demonstrate the efficacy of our proposed generative method, we have included a comparative analysis with conventional GANs.

Reviewer comment (Evaluation):
The improvement in effectiveness comes at the cost of increasing computational overhead. The time and space required to train the generated model is no less than (or even greater than) that of training the ordinal regression model, yet the performance gain is not so significant.

Author response:
**Q1. The improvement in effectiveness comes at the cost of increasing computational overhead. The time and space required to train the generated model is no less than (or even greater than) that of training the ordinal regression model, yet the performance gain is not so significant.**

We have rigorously evaluated the model's performance across a variety of benchmark datasets, consistently observing improvements in performance across all datasets and for every model variant tested. This highlights our significant contribution to the field of ordinal regression.

Besides, our proposed data augmentation method stands out for its generality. Our approach can be broadly applied to distinguish ordinal content from other information types and to generate “correct” new examples accurately. While we primarily tested it on image data, the conceptual framework of our method is versatile enough to be adapted to non-image data as well. This adaptability offers advantages not commonly found in traditional data augmentation methods.

Typically, existing augmentations in image data focus on altering factors like position, rotation, and color. However, these factors are specific to image data, and it remains unclear how to effectively augment non-image data. Our method transcends these limitations by automatically inferring latent ordinal content and other non-ordinal factors. By manipulating the inferred non-ordinal factors, we can generate new examples that maintain the crucial ordinal content. This capability makes our method broadly applicable and enhances the reliability of current machine learning techniques.

Reviewer comment (Theory):
It is unclear whether the proposed framework could guarantee that to what extent the content information can be maintained in the invariant ordinal content factors $\hat{\tilde{z}}_O$, which is crucial for the quality of generated images and further affect the performance of the ordinal regression model.

Author response:
**Q2. It is unclear whether the proposed framework could guarantee that to what extent the content information can be maintained in the invariant ordinal content factors z\_o\_tilde\_hat, which is crucial for the quality of generated images and further affects the performance of the ordinal regression model.** 

Our method is grounded in the theoretical framework of minimal change (Kong et al. 2022), which is instrumental in achieving disentanglement. According to the theoretical insights from prior research, our method's applicability to ordinal regression becomes evident. The principle implies that, given a sufficient number of images from different classes, the component $\hat{z}_o$ can be identified on a component-wise basis. This indicates that $\hat{z}_o$ undergoes a simple component-wise transformation solely from $z_o$. Consequently, $\hat{z}_o$ is devoid of information about the style component $z_n$, as it remains unaffected by $z_n$, thus achieving disentanglement.

The following provides the intuitive explanation about disentanglement:

- We categorize the latent factors into two groups: ordinal ($\hat{z}_o$) and non-ordinal ($\hat{z}_n$). In this framework, $\hat{z}_o$ serves as the repository for crucial ordinal content information that determines the ordinal category.

- For reconstruction, we utilize both the predicted ordinal factors ($\hat{z}_o$) and the predicted non-ordinal factors ($\hat{z}_n$). $\hat{z}_o$ is specifically employed for predicting ordinal labels and is subjected to minimal change, restricting its influence in generating new instances.

- Assuming the generative model attains minimal reconstruction error, $\hat{z}_o$, the factor designated for ordinal label prediction, effectively captures the ordinal content. By constraining it through the principle of minimal change, it encapsulates only the vital information pertinent to ordinal labels, while $\hat{z}_n$ accommodates other types of information.

- This approach leads to a clear disentanglement of ordinal and non-ordinal information, enhancing the model's effectiveness.

---

**References**

Kong, L., Xie, S., Yao, W., Zheng, Y., Chen, G., Stojanov, P., Akinwande, V., & Zhang, K. (2022). Partial disentanglement for domain adaptation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, & S. Sabato (Eds.), Proceedings of the 39th International Conference on Machine Learning (Vol. 162, pp. 11455–11472). PMLR.

Reviewer comment (Experiments):
Do the performance of the model sensitive to the setting of $\lambda_1$ in Eq. (4)?

Author response:
**Q3. Do the performance of the model sensitive to the setting of $\displaystyle \lambda _{1}$ in Eq. (4)?**

We are running sensitivity analysis on $\lambda _1$ for analyzing the performance of OCP-CL. This section and the paper will be updated after we obtain the results.

Reviewer comment (Evaluation):
Could you explain that to what extent the content information can be maintained in $\hat{\tilde{z}}_O$, either in theory or experiments?

Author response:
**Q2. It is unclear whether the proposed framework could guarantee that to what extent the content information can be maintained in the invariant ordinal content factors z\_o\_tilde\_hat, which is crucial for the quality of generated images and further affects the performance of the ordinal regression model.** 

Our method is grounded in the theoretical framework of minimal change (Kong et al. 2022), which is instrumental in achieving disentanglement. According to the theoretical insights from prior research, our method's applicability to ordinal regression becomes evident. The principle implies that, given a sufficient number of images from different classes, the component $\hat{z}_o$ can be identified on a component-wise basis. This indicates that $\hat{z}_o$ undergoes a simple component-wise transformation solely from $z_o$. Consequently, $\hat{z}_o$ is devoid of information about the style component $z_n$, as it remains unaffected by $z_n$, thus achieving disentanglement.

The following provides the intuitive explanation about disentanglement:

- We categorize the latent factors into two groups: ordinal ($\hat{z}_o$) and non-ordinal ($\hat{z}_n$). In this framework, $\hat{z}_o$ serves as the repository for crucial ordinal content information that determines the ordinal category.

- For reconstruction, we utilize both the predicted ordinal factors ($\hat{z}_o$) and the predicted non-ordinal factors ($\hat{z}_n$). $\hat{z}_o$ is specifically employed for predicting ordinal labels and is subjected to minimal change, restricting its influence in generating new instances.

- Assuming the generative model attains minimal reconstruction error, $\hat{z}_o$, the factor designated for ordinal label prediction, effectively captures the ordinal content. By constraining it through the principle of minimal change, it encapsulates only the vital information pertinent to ordinal labels, while $\hat{z}_n$ accommodates other types of information.

- This approach leads to a clear disentanglement of ordinal and non-ordinal information, enhancing the model's effectiveness.

---

**References**

Kong, L., Xie, S., Yao, W., Zheng, Y., Chen, G., Stojanov, P., Akinwande, V., & Zhang, K. (2022). Partial disentanglement for domain adaptation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, & S. Sabato (Eds.), Proceedings of the 39th International Conference on Machine Learning (Vol. 162, pp. 11455–11472). PMLR.

Reviewer comment (Experiments):
I think the experiments in this article did not fully verify that zo is an ordinal content factor and zn is a non-ordinal content factor. For example, the former obtains wrinkles or gray hair through the generator alone, while the latter obtains other features unrelated to age. From my point of view, the experimental part can only prove that the generated data-augmented images can maintain the ordinal content, which may benefit from the powerful generation ability of the generator itself. So I'd like to see more visual experiments verify this.

Author response:
In the updated supplementary section, we present additional visual experiments that involve combining the $\hat{z}_o$ of various age groups with a constant $\hat{z}_n$. These experiments reveal that the age of the same individual changes to match the corresponding age group of $\hat{z}_o$. This observation confirms that $\hat{z}_o$ represents an ordinal content factor, while $\hat{z}_n$ is a non-ordinal factor.

Reviewer comment (Evaluation):
In the data augmentation results produced in Figure 4, in the first group of augmentations from 4 to 6 years old, although the age range of the characters remains unchanged, the gender has changed. I think the ordinal invariant data augmentation produced by the proposed model may introduce additional noise, but there is no analysis of this additional noise in the paper. Will this additional noise generally affect the results?

Author response:
We categorize the latent factors into two categories: (1) ordinal content factors, which are the latent factors related to ordinal classes, and (2) non-ordinal factors, which encompass the remaining factors containing only styling information.

We would like to clarify that in the context of age estimation, gender is not considered an ordinal content factor. Introducing variability in non-ordinal factors, such as gender, can actually enhance the performance of the ordinal regression model. This is because it helps prevent the model from relying on non-ordinal factors for predictions. Consider a simplistic scenario where all training samples in the 4-6 age group are boys. In such a case, the model might incorrectly use gender as a predictor for this age group. Our method, by introducing diversity without altering the data's semantics, provides a more robust strategy for training ordinal regression models.

Reviewer comment (Experiments):
If one latent feature (non-ordinal content) does not contribute much to an ordinal regression downstream task, then how much help could its augmentation provide? I would like to see some analysis about it.

Author response:
**Q1. If one latent feature (non-ordinal factors) does not contribute much to an ordinal regression downstream task, then how much help could its augmentation provide?**

We categorize the latent factors into two categories: (1) ordinal content factors, which are the latent factors related to ordinal classes, and (2) non-ordinal factors, which encompass the remaining factors containing only styling information.

Suppose that 'contribution' in the original question is defined as the proportion of a certain type of feature present in an image, e.g., features unrelated to its ordinal class occupy only a small portion of the image.

In such cases, it is still challenging to guarantee that standard data augmentation will not distort ordinal information. This issue concerns not just proportion but also whether conflicts arise between certain augmentation methods and ordinal content. For example, in age estimation tasks, a person's hair color might be related to their age. This naturally conflicts with color distortion augmentation and is irrelevant to the proportion of 'hair' contributing to the image.

Reviewer comment (Experiments):
The authors use an example in Figure 1 to show that "the commonly used strong augmentations can distort or even erase these essential features in ordinal regression data." Can you try some strong augmentation methods to demonstrate this claim in Tables 1, 2, and 3?

Author response:
**Q2. The authors use an example in Figure 1 to show that "the commonly used strong augmentations can distort or even erase these essential features in ordinal regression data." Can you try some strong augmentation methods to demonstrate this claim in Tables 1, 2, and 3?**

Thank you for the suggestion. We will include visual analyses of all three datasets in the supplementary material to demonstrate how strong data augmentation might distort the ordinal content features visually.

Reviewer comment (Evaluation):
For results in Tables 1-4 and Figure 6, are they average of multiple runs? What about the standard deviations?

Author response:
**Q4. For results in Tables 1-4 and Figure 6, are they average of multiple runs? What about the standard deviations?**

For the age estimation task, we employ a five-fold cross-validation protocol in line with previous works (Li et al., 2021; Shen et al., 2018). The results reported are the average of the five folds. For the diabetic retinopathy rating and weather condition prediction tasks, the results are presented as averages of five runs. The standard deviations will be included in the updated version of the paper.

Reviewer comment (Novelty):
The proposed OCP-CL is somehow similar to feature selection for content disentangling. Can feature selection methods be applied to learn ordinal and non-ordinal content?

Author response:
**Q5. The proposed OCP-CL is somehow similar to feature selection for content disentangling. Can feature selection methods be applied to learn ordinal and non-ordinal factors?**

Traditional feature selection methods are applicable when features are predefined. These methods can identify the most relevant features for ordinal regression by leveraging various dependence relations within the data. However, in scenarios where features are not explicitly provided, such as with raw image data, selecting features on a pixel-by-pixel basis is impractical. Instead, it is more effective to first extract latent features (or concepts) before proceeding with feature selection.

Our method can be regarded as a feature selection approach tailored for latent features. It is specifically designed to automatically disentangle ordinal content factors (features) and non-ordinal factors based on the principle of minimum change. This strategy ensures that the identified ordinal content factors are pertinent and relevant for ordinal regression.

Reviewer comment (Reproducibility):
How did you get the numbers in Table 5? By setting a threshold for $M$?

Author response:
**Q6. How did you get the numbers in Table 5? By setting a threshold for $M$?**

We adjust the hyperparameter $\lambda_1$ to control the sparsity of the mask. Table 5 presents the number of ordinal content dimensions in the latent space corresponding to different values of $\lambda_1$.