Paper: Consistent Multi-Class Classification from Multiple Unlabeled Datasets (ICLR-2024)

Reviewer comment (Experiments):
The experimental comparison is somewhat limited, primarily featuring methods from Tang et al., 2022. While justified due to MCMU's novelty, including additional weakly supervised methods could provide valuable reference points. Moreover, presenting accuracies for fully supervised learning on the same train/test splits would enhance the evaluation.

Author response:
MCMU is a relatively new research area that was introduced very recently. To the best of our knowledge, there is only one work [1] that has studied this problem and this work proposes a number of effective methods to solve this MCMU problem. Therefore, we adopted all the methods in [1] as our baselines. We believe that the number of compared methods is sufficient, as we have already compared all the existing methods designed for MCMU. We would be very grateful if you could provide us with more weakly supervised learning methods that can be used for MCMU, and we will definitely include such methods in experimental comparisons.

[1] Tang Y, Lu N, Zhang T, et al. Learning from Multiple Unlabeled Datasets with Partial Risk Regularization. ACML 2022.

**Presenting accuracies for fully supervised learning on the same train/test splits would enhance the evaluation.**

Thank you for your advice. We have conducted experiments with fully supervised data. The experimental results have been incorporated into Table 2 and Table 3 for comprehensive evaluations.

Reviewer comment (Experiments):
The paper lacks a comprehensive study of the method's limitations, particularly with regards to constraints on parameters such as m (the number of unlabeled sets) and n_i (the number of data points in each set), which are crucial for practical applicability.

Author response:
We conducted experiments on variants $m$ and $n_i$ and showed the experimental results in Appendix E. Here we provide a detailed description of the additional experimental results.

We conducted additional experiments with different numbers of sets with fixed data points (i.e. with fixed set number times the set size) on the MNIST dataset. Random matrix and asymmetric matrix are adapted as the class prior matrix. The experimental results are shown in Table 1, Table 2, and Table 3. Our proposed RCM and CCM demonstrate stable performance with the increase in the number of sets and still outperform the baseline methods. When the number of sets becomes excessively large, some baseline methods may achieve poorer performance. This is because in such a case, the distribution $p(x)$ on training data might shift from testing data. Some papers [2,3] refer to this case as a covariate shift, which would lead to degraded performance.

Table 1: Classification performance of each method on the MNIST dataset using an asymmetric prior matrix with variant set numbers.

| Methods   | m=30         | m=50         | m=100        | m=200        | m=300         | m=500         | m=1000       |
| --------- | ------------ | ------------ | ------------ | ------------ | ------------- | ------------- | ------------ |
| Unbiased  | 84.30±2.55% | 86.85±1.06% | 86.74±0.88% | 87.75±0.47% | 88.17±0.68%  | 87.46±0.41%  | 62.78±5.07% |
| U-Stop    | 89.69±1.21% | 89.79±1.59% | 91.05±0.42% | 89.47±0.20% | 89.71±0.58%  | 88.37±0.21%  | 85.23±0.65% |
| U-Correct | 94.07±0.18% | 94.16±0.23% | 87.00±0.30% | 78.94±1.86% | 63.33±16.18% | 27.04±8.39%  | 26.80±7.59% |
| U-Flood   | 93.42±0.56% | 93.29±0.34% | 90.55±0.53% | 85.29±0.86% | 79.46±0.83%  | 68.43±10.75% | 56.70±3.33% |
| Prop      | 82.98±1.55% | 79.85±1.01% | 75.46±1.23% | 72.77±1.60% | 72.62±0.67%  | 71.61±0.51%  | 71.90±0.32% |
| U-PRR     | 91.11±0.29% | 89.32±0.89% | 79.28±1.34% | 70.88±2.36% | 67.80±0.69%  | 62.76±2.53%  | 67.28±2.11% |
| CCM       | 95.77±0.12% | 95.66±0.17% | 95.66±0.08% | 95.40±0.14% | 95.69±0.14%  | 95.57±0.20%  | **95.91±0.17%** |
| RCM       | **95.94±0.23%** | **96.02±0.08%** | **95.95±0.15%** | **95.70±0.23%** | **95.88±0.08%**  | **95.82±0.07%**  | 95.77±0.22% |

Table 2: Classification performance of each method on the MNIST dataset using a random prior matrix with variant set numbers.
| Methods   | m=30         | m=50         | m=100         | m=200         | m=300        | m=500        | m=1000       |
| --------- | ------------ | ------------ | ------------- | ------------- | ------------ | ------------ | ------------ |
| Unbiased  | 14.14±1.15% | 15.64±1.48% | 13.90±0.70%  | 11.18±2.30%  | 15.12±2.64% | 16.36±1.53% | 13.31±2.95% |
| U-Stop    | 53.12±3.76% | 71.61±0.52% | 38.41±18.44% | 40.62±24.60% | 18.33±6.65% | 18.27±2.32% | 25.89±3.77% |
| U-Correct | 63.84±4.61% | 66.64±1.21% | 75.00±0.50%  | 64.24±0.34%  | 64.85±1.51% | 71.59±0.20% | 77.01±0.83% |
| U-Flood   | 89.14±0.96% | 88.91±0.92% | 85.87±0.42%  | 75.59±0.72%  | 69.92±1.47% | 59.82±2.38% | 50.54±5.46% |
| Prop      | 81.84±1.36% | 77.29±1.51% | 66.61±2.35%  | 57.96±1.47%  | 54.38±1.12% | 53.38±1.26% | 51.44±1.16% |
| U-PRR     | 23.92±0.56% | 19.62±1.15% | 19.30±1.30%  | 18.15±1.22%  | 14.97±0.47% | 17.98±1.16% | 13.69±0.53% |
| CCM       | 92.90±0.29% | 92.49±0.14% | 92.86±0.39%  | 92.37±0.20%  | 93.23±0.37% | 93.00±0.08% | 93.06±0.20% |
| RCM       | **95.69±0.25%** | **95.63±0.12%** | **95.49±0.17%**  | **95.38±0.18%**  | **95.49±0.07%** | **95.46±0.15%** | **95.78±0.14%** |

Table 3: Classification performance of each method on the MNIST dataset using a random prior matrix with variant set numbers. ($m \geq k$)

| Methods | Unbiased        | U-Stop           | U-Correct       | U-Flood         | Prop            | U-PRR           | CCM             | RCM             |
| ------- | --------------- | ---------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- |
| m=5     | 18.23$\pm$8.08% | 45.88$\pm$12.29% | 34.71$\pm$7.20% | 58.97$\pm$4.25% | 72.48$\pm$2.37% | 26.34$\pm$6.46% | 88.80$\pm$2.43% | **89.16$\pm$4.29%** |
| m=6     | 14.61$\pm$6.72% | 42.02$\pm$22.68% | 40.06$\pm$8.00% | 68.00$\pm$3.39% | 82.06$\pm$4.49% | 28.45$\pm$5.02% | 89.80$\pm$3.59% | **92.71$\pm$0.73%** |
| m=7     | 15.56$\pm$2.38% | 42.39$\pm$6.74%  | 49.72$\pm$8.40% | 77.53$\pm$7.95% | 89.32$\pm$3.43% | 32.78$\pm$5.33% | 90.66$\pm$2.19% | **94.53$\pm$1.06%** |
| m=8     | 15.13$\pm$0.87% | 37.08$\pm$16.58% | 32.65$\pm$2.24% | 79.89$\pm$4.45% | 88.57$\pm$3.11% | 27.72$\pm$0.64% | 89.99$\pm$2.75% | **94.49$\pm$1.06%** |
| m=9     | 14.06$\pm$1.79% | 27.77$\pm$8.25%  | 33.80$\pm$1.88% | 82.83$\pm$2.78% | 88.31$\pm$3.68% | 29.79$\pm$4.22% | 91.42$\pm$1.26% | **94.44$\pm$0.91%** |

[2] Yiyang Zhang, Feng Liu, Zhen Fang, Bo Yuan, Guangquan Zhang, and Jie Lu. Clarinet: A one-step approach towards budget-friendly unsupervised domain adaptation. IJCAI 2020.

[3] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227–244, 2000.

Reviewer comment (Theory):
Theoretical results like Theorem 3.5, which establishes an upper bound for the difference between true and empirical risk, would benefit from discussion regarding their practical implications and their relationship to the actual classification task involving class labels.

Author response:
The practical implication of Theorem 3.5 lies in the fact that the performance of our method can be improved, by increasing the training data of MCMU. We can also find that the optimal classifier obtained by our method will converge to the optimal classifier (learned by minimizing the actual classification risk involving class labels), as the number of training data of MCMU increases.

Reviewer comment (Experiments):
The paper could further investigate the impact of small m (few unlabeled sets) and significant variations in label proportions across sets on the generalizability of its results with respect to m and n_i.

Author response:
Thank you for your valuable suggestion. We have conducted experiments specifically designed for the case of small values of $m$. The experimental results are reported in Table 3. Please refer to our response to Weakness 2.

Reviewer comment (Novelty):
A clarification regarding the difference between Lemma 3.1 in the paper and Theorem 1 in Lu et al., ICML 2021, would be valuable for readers.

Author response:
Lemma 3.1 can be considered as a multi-class extension of Theorem 1 (only for binary classification) in Lu et al., ICML 2021. As we indicated in our paper, CCM is a multi-class extension of Lu et al., ICML 2021. We would like to explain that the CCM is not the main contribution of this paper. The main contribution of our paper lies in the proposed RCM.

Reviewer comment (Experiments):
Given that label proportions in real-world scenarios may only slightly differ, the paper should outline constraints on parameters m and n for practical significance, especially considering that a large m may be necessary for the method's effectiveness.

Author response:
Please refer to our response to Weakness 2, where we have conducted experiments on various values of $m$ and $n$.

Reviewer comment (Theory):
A discussion on the practical relevance of the derived upper bound in Theorem 3.5 and its connection to the actual classification task involving class labels would provide valuable insights.

Author response:
Please check our response to Weakness 3.

Reviewer comment (Evaluation):
The observation that the proposed approach performs significantly worse under certain settings, such as a random class prior matrix and non-constrained m, should be supported by clearer details on how test accuracies were computed, whether test data were balanced, and if class size variations in datasets played a role.

Author response:
We computed the test accuracies on the test set of widely used benchmark datasets, including MNIST, Fashion, Kuzushiji, CIFAR-10, and SVHN. We simply used the given test set and did not make modifications to the class size of the provided test set.

We also conducted additional experiments on the MNIST, Kuzushiji, and Fashion datasets with imbalanced classes. We constructed the class-imbalanced prior matrix by following a similar procedure to [1]. Firstly, we generate a random prior matrix $\Theta$. Secondly, we generate a geometric progression $\boldsymbol{l}$ according to the imbalance ratio $\rho$, where $\rho$ denotes the ratio between the largest and smallest elements in $\boldsymbol{l}$, i.e., $\boldsymbol{l}=\{1,\rho^{1/k-1}\,\rho^{2/k-1}\,\dots,\rho\}$. Thirdly, we divide each row of the prior matrix by this geometric progression, i.e., $\Theta_{ij}=\Theta_{ij}/$ $\boldsymbol{l}{j}$. Finally, we normalize each row of $\theta$ by $\Theta_{ij}=\Theta_{ij}/\sum_{j=1}^k\Theta_{ij}$ where $k$ is the class number.

The experimental results are reported in Table 4. As can be seen from Table 4, our proposed methods RCM and CCM can still achieve satisfactory performance even under extreme class imbalance conditions. In MNIST and Kuzushiji datasets, RCM exhibited 
stronger robustness against class imbalance compared with CCM, as it was less affected as $\rho$ rises.

Table 4: Classification performance of RCM and CCM trained on imbalanced classes.

|   $\rho$    | Methods |      MNIST       |     Fashion      |    Kuzushiji    |
| :---------: | :-----: | :--------------: | :--------------: | :-------------: |
| 1 (balance) |   CCM   | 91.46$\pm$1.22%  | 76.09$\pm$07.57% | 72.85$\pm$2.52% |
|             |   RCM   | **94.80$\pm$0.64%**  | **80.49$\pm$2.50%**  | **78.36$\pm$2.89%** |
|     10      |   CCM   | 89.76$\pm$0.82%  | 73.21$\pm$7.96%  | 68.57$\pm$3.14% |
|             |   RCM   | **93.27$\pm$1.05%**  | **75.36$\pm$3.53%**  | **74.03$\pm$2.67%** |
|     100     |   CCM   | 72.49$\pm$14.81% | 64.77$\pm$2.46%  | 53.54$\pm$3.43% |
|             |   RCM   | **85.32$\pm$2.75%**  | **65.43$\pm$5.26%**  | **62.69$\pm$2.31%** |

[1] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.

Reviewer comment (Theory):
The statement “In MCMU, given the class priors, the data points in the same set are independent from each other, while in LLP, given the label proportions, the data points in the same set are dependent from each other” should clearly state whether data points are “independent from each other” or “dependent on each other.” If it implies independence, it raises a significant ambiguity regarding how MCMU and LLP fundamentally differ, as class priors and label proportions are practically equivalent concepts. There appears to be a conceptual misuse in the paragraph: data points within each class are typically assumed to be independent, conditioned on their class labels, yet the statement uses “class priors” in place of “class labels,” which could introduce confusion and should be clarified.

Author response:
Thank you for your valuable suggestion! We have revised the description in our paper. In the LLP problem, data points are dependent on each other. 
The label proportions and class priors are two different notions in our paper, and we describe the differences between them in the following response.

Reviewer comment (Writing):
The paper should consider using the term “label proportions” when referring to class priors within each set. What the paper denotes as the j-th class prior of the i-th unlabeled set can be expressed as the label proportion of class j in dataset i. This would distinguish label proportions from class priors estimated using data from all available unlabeled sets.

Author response:
Thank you for your thoughtful consideration of our paper.
In the MCMU problem, the unlabeled set is sampled according to class priors. In our humble opinion, we think that using class priors is more appropriate for MCMU. Although class priors and label proportions seem to be practically equivalent concepts, they actually describe two different sampling processes. In LLP, the notion of "label proportions" means that we calculate the proportion of instances belonging to each class after generating the data according to the underlying data distribution $p(x,y)$.

Reviewer comment (Writing):
In Section 4.1, the phrase “The n_i data points contained in i-th unlabeled set were randomly generated according to ...” should likely be “randomly sampled” or “drawn,” as data already exists and is not generated.

Author response:
Thank you for pointing out this issue and we have corrected it in our paper.

Reviewer comment (Writing):
The paper would benefit from providing an early explanation in the related work section regarding the reasons behind the possibility of negative empirical risk in these methods (Lu et al., 2019; Tsai & Lin, 2020; Tang et al., 2022), to help readers anticipate and better understand the issues the paper seeks to address.

Author response:
Thank you for this suggestion. There are two important reasons that we did not provide a detailed explanation/analysis of the negative empirical risk of previous work. Firstly, many previous studies on various weakly supervised learning problems have exhibited the negative empirical risk shared by common methods, such as positive-unlabeled learning [2], unlabeled-unlabeled learning [3], complementary-label learning [4], and so on. Thus we consider that this issue was commonly known and may not be necessary to be shown in this paper. Secondly, the page limit will also prevent us from adding a detailed explanation/analysis of the negative empirical risk of previous work. We really appreciate this suggestion and we will consider incorporating this part if our paper is accepted, because we are only allowed to use one more page for the camera-ready version. 

[2] Positive-unlabeled learning with non-negative risk estimator. NeurIPS 2017.

[3] Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets: A Consistent Risk Correction Approach. AISTATS 2020.

[4] Complementary-label learning for arbitrary losses and models. ICML 2019.

Reviewer comment (Writing):
On page 4, the index i is used for both the sets and data points, which may cause confusion.

Author response:
Thank you for pointing out this issue and we have corrected it in our paper.

Reviewer comment (Writing):
In Theorem 3.6, the probability p(y=j|\bar{y},x) is defined as the probability of x in \bar{y}-th unlabeled set whose ground-truth label is j. This should be clarified as the probability of x in \bar{y}-th unlabeled set belonging to class j, since x may or may not belong to class j and its ground truth label is not necessarily j.

Author response:
Yes, Your understanding is correct. $p(y=j|\overline{y},x)$ denotes the probability of $x$ in $\overline{y}$-th unlabeled set belonging to class $j$.

Reviewer comment (Theory):
Theorem 3.6 also assumes p(x|y=j)=p(x|y=j,\bar{y}) as a fact. This is not necessarily true, since the first is the conditional distribution of class j estimated using all data belonging to class j, whereas the second is estimated using only data from unlabeled set \bar{y}. From an empirical standpoint, these distributions are not equal, and this should be clarified.

Author response:
For the data generation of the MCMU problem, $p(x|y=j)=p(x|y=j,\overline{y})$ always holds. This is because, for MCMU, we need to first sample the information of true labels according to the class priors of each unlabeled set (i.e., $\overline{y} \rightarrow y$) and then sample the instances of each class according to the class-conditional density $p(x|y=j)$ (i.e., $y\rightarrow x$). Hence the whole data generation process is $\overline{y}\rightarrow y\rightarrow x$, which means, given the true label $y$, $x$ is independent of $\overline{y}$, i.e., $p(x|y=j)=p(x|y=j,\overline{y})$.

Reviewer comment (Theory):
This paper should give more explanations for the theoretical findings. For example, there are no discussions or descriptions on Theorem 3.5 and Theorem 3.7.

Author response:
Generally, the Rademacher complexity of the hypothesis space $H$ can be upper bounded by $C_{H}/\sqrt{n}$ for a positive constant $C_{H}$ [1,2]. Theorem 3.5 demonstrates that the empirical minimizer $\hat{f}_{\text{ccm}}$ would coverage to the minimizer $f^*$ on clean data as the training sample size approximates infinity. Theorem 3.7 shows that the empirical risk would converge to the expected risk as the sample size approaches infinity. 

[1] N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. COLT 2018.

[2] N. Lu, T.-Y. Zhang, G. Niu, and M. Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. AISTATS, 2020.

Reviewer comment (Theory):
What can we learn from Theorem 3.5 and Theorem 3.7?

Author response:
Generally, the Rademacher complexity of the hypothesis space $H$ can be upper bounded by $C_{H}/\sqrt{n}$ for a positive constant $C_{H}$ [1,2]. Theorem 3.5 demonstrates that the empirical minimizer $\hat{f}_{\text{ccm}}$ would coverage to the minimizer $f^*$ on clean data as the training sample size approximates infinity. Theorem 3.7 shows that the empirical risk would converge to the expected risk as the sample size approaches infinity. 

[1] N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. COLT 2018.

[2] N. Lu, T.-Y. Zhang, G. Niu, and M. Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. AISTATS, 2020.

Reviewer comment (Theory):
There is a lack of descriptions of Theorem 3.5 and Theorem 3.7. Additionally, could we compare RCM and CCM from a theoretical perspective?

Author response:
**Q1: There is a lack of descriptions of Theorem 3.5 and Theorem 3.7.**

Thank you for your advice, we have provided additional descriptions of Theorem 3.5 and Theorem 3.7 in our revised paper.

Generally, the Rademacher complexity of the hypothesis space $H$ is upper bounded by $C_H/\sqrt{n}$ [1,2] for a constant $C_H$. Theorem 3.5 demonstrates that the empirical minimizer $\hat{f}_{\text{ccm}}$ would coverage to the minimizer $f^*$ on clean data as the training sample size approximates infinity. Theorem 3.7 shows that the empirical risk would converge to the expected risk as the sample size approaches infinity. 

[1] N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. COLT 2017.

[2] N. Lu, T.-Y. Zhang, G. Niu, and M. Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In AISTATS, 2020.

**Additionally, could we compare RCM and CCM from a theoretical perspective?**

From a theoretical perspective, RCM tries to approximate or simulate the distribution of real clean data by utilizing the data distribution from the unlabeled set (by the importance-weighting schema). This means that RCM attempts to infer latent distribution patterns of real clean data from the unlabeled data. In contrast, CCM aims to better fit the distribution of the unlabeled set by maximizing a log-likelihood object. With a sufficient number of samples, RCM is more accurate in predicting the labels of unseen samples because it considers the restoration of the distribution of real clean data when modeling unlabeled data. This enables RCM to exhibit better generalization performance when facing unknown data, making more precise predictions on unseen samples.

Reviewer comment (Evaluation):
The analysis in section 4.3 is interesting. However, there is a lack of discussion about the observation from Fig. 1. So, could the authors provide more details of why CCM is more robust when few data points are provided?

Author response:
**Q2: The analysis in section 4.3 is interesting. However, there is a lack of discussion about the observation from Fig. 1. So, could the authors provide more details of why CCM is more robust when few data points are provided?**

We think it is because RCM is based on importance weighting, which requires a certain number of samples to accurately estimate the weights for reliably optimizing the objective. Hence, with a small sample size, CCM may outperform RCM.

Reviewer comment (Experiments):
I noticed that in all experimental settings, the number of sets is greater than or equal to the number of classes. Could the authors provide a discussion where the number of sets is less than the number of classes?

Author response:
**Q3: I noticed that in all experimental settings, the number of sets is greater than or equal to the number of classes. Could the authors provide a discussion where the number of sets is less than the number of classes?**

Thank you for your suggestion. We have additionally conducted experiments for the mentioned scenario where the number of sets is less than the number of classes, given a fixed number of data points. 

Table 1: Classification performance of each method on the MNIST dataset using a random prior matrix with variant set numbers. ($m \geq k$)

| Methods | Unbiased        | U-Stop           | U-Correct       | U-Flood         | Prop            | U-PRR           | CCM             | RCM             |
| ------- | --------------- | ---------------- | --------------- | --------------- | --------------- | --------------- | --------------- | --------------- |
| m=5     | 18.23$\pm$8.08% | 45.88$\pm$12.29% | 34.71$\pm$7.20% | 58.97$\pm$4.25% | 72.48$\pm$2.37% | 26.34$\pm$6.46% | 88.80$\pm$2.43% | **89.16$\pm$4.29%** |
| m=6     | 14.61$\pm$6.72% | 42.02$\pm$22.68% | 40.06$\pm$8.00% | 68.00$\pm$3.39% | 82.06$\pm$4.49% | 28.45$\pm$5.02% | 89.80$\pm$3.59% | **92.71$\pm$0.73%** |
| m=7     | 15.56$\pm$2.38% | 42.39$\pm$6.74%  | 49.72$\pm$8.40% | 77.53$\pm$7.95% | 89.32$\pm$3.43% | 32.78$\pm$5.33% | 90.66$\pm$2.19% | **94.53$\pm$1.06%** |
| m=8     | 15.13$\pm$0.87% | 37.08$\pm$16.58% | 32.65$\pm$2.24% | 79.89$\pm$4.45% | 88.57$\pm$3.11% | 27.72$\pm$0.64% | 89.99$\pm$2.75% | **94.49$\pm$1.06%** |
| m=9     | 14.06$\pm$1.79% | 27.77$\pm$8.25%  | 33.80$\pm$1.88% | 82.83$\pm$2.78% | 88.31$\pm$3.68% | 29.79$\pm$4.22% | 91.42$\pm$1.26% | **94.44$\pm$0.91%** |

The experimental results are reported in Table 1. In the experiments where $m$ is less than $k$, our proposed RCM and CCM consistently outperform other methods. Notably, RCM exhibits superior performance compared with CCM, and this performance difference becomes larger as $m$ increases.

Reviewer comment (Writing):
There is typo / the sentence got broken up here: “where d is a positive integer denotes the input dimension. [k] = “

Author response:
Thank you for your valuable comments. We have corrected the identified typos in the manuscript. Below, we provide a detailed response to further enhance the clarity and quality of our work.

Reviewer comment (Theory):
The author’s state that Section 2.2 that in MCMU has the data generating process P(X|y) P(y|c) P(c) which seems reasonable, but that the generating process of LLP is P(y|X)? That would make sense as the modelling distribution (learning a discriminative function), but not as a data generating process? Also where is the class priors in the data generating process of LLP?

Author response:
**Q1: The author’s state that Section 2.2 that in MCMU has the data generating process P(X|y) P(y|c) P(c) which seems reasonable, but that the generating process of LLP is P(y|X)? That would make sense as the modelling distribution (learning a discriminative function), but not as a data generating process? Also where is the class priors in the data generating process of LLP?**

The generation process in LLP can also be simplified by first generating a group of {$(x_{i},y_{i})$}$^n_{i=1}$ pairs randomly, and then calculating the corresponding class proportions based on the group of labels {$y_i$}$^n_{i=1}$. Thus the "class priors" in LLP are calculated by the group {$y_i$}$^n_{i=1}$ and we usually use the term "label proportions" (instead of "class priors") in LLP. In conclusion, for LLP, we first sample data pairs {$(x_i,y_i)$}$_{i=1}^n$ randomly according to probability density $p(x,y)$ then we could calculate the "label proportions".

In contrast, for MCMU, we first set the "class priors", and generate data pairs {$(x_i,y_i)$}$_{i=1}^n$ according to the "class priors" as we described in Section 2.4.

Reviewer comment (Presentation):
Section 2.4 is kind of randomly there without any introduction and it could be “cleaned up”.

Author response:
**Q3: Section 2.4 is kind of randomly there without any introduction and it could be “cleaned up”.**

Thanks for your suggestion. We have cleaned up it in Section 2.4.

Reviewer comment (Writing):
What is WSL?

Author response:
**Q4:  What is WSL?**

In our paper, WSL refers to Weakly Supervised Learning. We have provided an explanation of WSL in the revised paper.

Reviewer comment (Theory):
The main idea of CCM seems to be that by converting the problem from classifying multiple classes in multiple datasets, it can be simplified into classifying which dataset the X sample comes from? And this equivalency is due to a deterministic transition function T() which is based on \rho the probability a datapoint belongs to a dataset, \theta the proportions of each class for each dataset, and \pi the proportion of each “class” over all datasets?

Author response:
**Q6: The main idea of CCM seems to be that by converting the problem from classifying multiple classes in multiple datasets, it can be simplified into classifying which dataset the X sample comes from? And this equivalency is due to a deterministic transition function T() which is based on $\rho$ the probability a datapoint belongs to a dataset, $\theta$ the proportions of each class for each dataset, and $\pi$ the proportion of each "class" over all datasets?**

The main idea of CCM is to design a loss function by converting the problem of classifying multiple classes into multiple unlabeled datasets.
This conversion is based on our built connection relationship between $p(y|x)$ (the probability an instance $x$ belongs to a label) and $p(\overline{y}|x)$ (the probability an instance $x$ belongs to an unlabeled set) in Eq. (2), which is further represented as a transition function $T$ (i.e., $\overline{\eta}(x) = T(\eta(x))$ where $\overline{\eta}(x)=p(\overline{y}|x)$ and $\eta(x)=p(y|x)$).
As we aim to approximate $p(\overline{y}|x)$ by $T(g(x))$, we can infer that $p(y|x)$ can be approximated by $g(x)$ (where we use $g(x)$ to denote the to represent the Softmax output of the model), because $T$ is an injective function. The detailed proof can be found in Appendix B.3.
$\rho$ is the probability that a data point belongs to an unlabeled set, $\theta$ represents the priors of each class for each unlabeled set, and $\pi$ is the proportion of each "class" overall unlabeled sets.

Reviewer comment (Experiments):
The problem setup seems very restrictive / artificial. There has to be the same number of classes for each dataset and the classes are “ordered” / aligned across multiple datasets? Also while the problem of classifying which dataset a sample belongs intuitively is “easier”, it is not clear how that solves the problem of which class within which dataset a sample belongs to. Multiplying by the proportions of dataset size and class size will “on average” get you probabilities on the latter problem, but this is just a math trick to achieve a bound. It is not clear how this works practically. The experiments seems to indicate it works, it is just extremely unintuitively how / explained badly.

Author response:
**Q7: The problem setup seems very restrictive / artificial. There has to be the same number of classes for each dataset and the classes are “ordered” / aligned across multiple datasets? Also while the problem of classifying which dataset a sample belongs intuitively is “easier”, it is not clear how that solves the problem of which class within which dataset a sample belongs to. Multiplying by the proportions of dataset size and class size will “on average” get you probabilities on the latter problem, but this is just a math trick to achieve a bound. It is not clear how this works practically. The experiments seem to indicate it works, it is just extremely unintuitively how / explained badly.**

We would like to clarify that our setting does not have the constraint that the number of classes in each unlabeled set must be the same. There could be different numbers of classes in different unlabeled sets. For example, if there is no class $j$ exists in unlabeled set $i$. We can simply set $\theta_{ij}$ (the $j$-th class prior of the $i$-th unlabeled set) to 0, then we can still handle this setting by our proposed methods.

We admit that the classes should be ordered/aligned across multiple unlabeled sets. This is actually the basic requirement to learn a single classifier from multiple datasets simultaneously. Besides, in our humble opinion, it is also easy to satisfy this requirement during the data collection process, by simply assigning each collected class label a unique index. 

In our humble opinion, the MCMU problem is not artificial but can be encountered in various real-world scenarios. For example, predicting the demographic information of users in social networks is crucial for practical policy-making [1]. However, the collection of individual user information might be constrained by data privacy concerns. Fortunately, acquiring multiple unlabeled datasets is more feasible, and their class priors can be derived from existing census data.

[1] Culotta A, Kumar N, Cutler J. Predicting the demographics of twitter users from website traffic data. AAAI 2015.

Reviewer comment (Theory):
Employing the direct outputs of a network to estimate the posterior probabilities \(p(y=j∣x)\) can be imprecise, particularly under the extreme weak supervision scenario where only class prior probabilities serve as supervisory signals. In such a setting, the network predictions may not align well with the true posterior distributions, leading to suboptimal performance.

Author response:
We agree with the reviewer to some degree that the Softmax outputs may not precisely estimate the posterior probabilities. However, from the theoretical perspective, our provided Lemma 3.3 demonstrates that if certain loss functions are used (e.g., cross-entropy loss), the Softmax outputs of the model can ideally approximate the true posterior distributions. This theoretical result motivates us to use the Softmax outputs to estimate the posterior probabilities.

From the empirical perspective, it is true that the Softmax outputs may not precisely estimate the posterior probabilities. However, there are still many previous methods that have applied the Softmax outputs to approximate the posterior probabilities [1,3,4,5] and achieved excellent performance on various tasks. Hence, the rationale for employing Softmax outputs to approximate posterior probabilities has been widely acknowledged in previous studies. Actually, how to precisely estimate the posterior probabilities is another important task called model calibration [6], which is definitely out of the scope of this paper. In the current stage, we cannot expect there exists a perfect estimation strategy that has no estimation error. That is why we simply used the Softmax outputs to estimate the posterior probabilities, in our paper.

Reviewer comment (Theory):
In Theorems 3.5 and 3.7, the assumptions regarding the Lipschitz continuity of the loss function present an indeterminable strength which could affect the assessment of the model's robustness. Furthermore, the generalization error bound incorporates a summation over \(k\) categories of the Rademacher complexity, which may result in a rather loose bound.

Author response:
The Lipschitz continuity assumption was widely accepted in the machine learning community [1,3,4,5] for establishing generalization or estimation error bounds. This property has been chosen for its stability-enhancing characteristics in optimization and theoretical analyses, contributing to the theoretical underpinning of our work. Importantly, our use of Lipschitz continuity does not compromise the robustness assessment of the model under investigation.

Furthermore, it is noteworthy that Lipschitz continuity has become a fundamental property of many loss functions, such as cross-entropy loss, mean absolute error, square loss, and so on. The widespread applications of these loss functions have demonstrated the rationality of Lipschitz continuity.

For the generalization error bound and the summation over $k$ categories of the Rademacher complexity. The summation of $k$ Rademacher complexities is theoretically supported by the Rademacher vector contraction inequality [7], with the fact that MCMU is a $k$-class classification problem. According to [7], the summation of $k$ Rademacher complexities will not result in a loose bound.
It is also worth noting that similar bounds have been established in many previous studies on weakly supervised learning, e.g., [1,3,5].