Paper: Pi-DUAL: Using privileged information to distinguish clean from noisy labels (ICLR-2024)

Reviewer comment (Theory):
While PI is a concept introduced in prior works, this article doesn't offer significant innovations to it. The paper points out that PI-based methods underperform compared to no-PI-based methods. However, it fails to delve deep into the underlying principles causing this discrepancy. The conclusions seem to be drawn mainly from some experimental verifications rather than in-depth analysis.

Author response:
**Innovations to PI**

We respectfully disagree with the reviewer in that the innovations of Pi-DUAL are minor. As appreciated by the other reviewers, the goal of our paper is to build a simple and effective method that requires the minimal changes to existing training pipelines. In this regard, while Pi-DUAL is not the first in utilizing PI features to combat label noise, Pi-DUAL provides a solution that is both simpler and more efficient. Compared to TRAM, it has the following key distinctions:

1. **Network structure**: TRAM uses one model but with two heads, one for training and one for testing. In contrast,  Pi-DUAL uses three distinct subnetworks with clear roles during training and inference: the prediction network for fitting clean target distribution, the noise network for explaining away label noise, and the gate network for identifying noisy samples. These three roles can not be accounted for in the architecture of TRAM.
2. **Learning objective**: While the learning objective of TRAM is a weighted sum of two objectives (one for fitting the noisy target distribution, and another for transferring the knowledge to the no-PI head), the learning objective of Pi-DUAL is a simple cross-entropy loss over the noisy targets. For a better illustration of the differences between Pi-DUAL and TRAM, we conducted an ablation which trains Pi-DUAL with the loss function of TRAM, and show that this produces much worse results. This experiment has been added to Section C.8 in Appendix.   

| Dataset \ Method |    CE    |  Pi-DUAL | Pi-DUAL with TRAM loss |
|:----------------:|:--------:|:--------:|:----------------------:|
|     CIFAR-10H    | 51.1±2.2 | 71.3±3.3 |        61.6±4.8        |
|     CIFAR-10N    | 80.6±0.2 | 84.9±0.4 |        81.6±0.9        |
|    CIFAR-100N    | 60.4±0.5 | 64.2±0.3 |        59.0±0.6        |

3. **Modeling of label noise**. TRAM does not explicitly model label noise. Instead, it learns the conditional distribution of noisy labels with respect to both x and a, to eventually predict with an implicit marginalization over a. On the other hand, Pi-DUAL explicitly models per-instance-level label noise in its architecture, which permits it to identify noisy labels and to effectively reduce the negative impacts from noisy labels. In contrast to TRAM, which performs approximate marginalization over the PI,  the prediction network of Pi-DUAL learns the conditional distribution of clean labels with respect to only x. With this explicit modeling of label noise, Pi-DUAL delivers not only better performance, but also a much better interpretability as to how PI helps explain away label noise (e.g., how the noise network overfits to the wrong labels in Figure 2, and how the gate network identifies wrong labels in Figure 3).

With these non-trivial improvements, Pi-DUAL outperforms TRAM++ on all 5 benchmarks achieving up to 8.1 accuracy points more than TRAM++ on ImageNet-PI. Pi-DUAL also delivers much better performance in identifying label noise post-training due to the explicit modeling of label noise.

Reviewer comment (Experiments):
Certain ablation studies were not conducted, like the choice of the model backbone and parameters of the additional PI-related modules.

Author response:
**Architectural ablation studies**

We thank the reviewer for bringing the missing ablation studies on the model architectures to our attention, and we hope the following additional ablations help complete our understanding of the method.

> *Ablations for the model backbones*

In the paper we use a WideResNet-10-28 for all experiments on CIFAR. Here we provide the results when replacing the architecture with a ResNet-34 instead. ResNet-34 is a common choice for model backbones in CIFAR-level experiments in the noise label literature (Cheng et al., 2020, Liu et al., 2020, 2022). We have added this ablation in Appendix C. 6.1. The results using ResNet-34 comparing Pi-DUAL and CE baseline are reported in the following table:

|                  | WideResNet-10-28 |          | ResNet-34 |          |
|------------------|------------------|----------|:---------:|:--------:|
| Dataset \ Method |        CE        |  Pi-DUAL |     CE    |  Pi-DUAL |
|     CIFAR-10H    |     51.1±2.2     | 71.3±3.3 |  51.4±2.1 | 69.3±3.1 |
|     CIFAR-10N    |     80.6±0.2     | 84.9±0.4 |  80.7±1.0 | 84.5±0.4 |
|    CIFAR-100N    |     60.4±0.5     | 64.2±0.3 |  56.9±0.4 | 62.2±0.3 |

From the table, we observe that Pi-DUAL maintains its performance improvement over CE with a different model backbone, exceeding the performance of CE baseline by a notable margin in all three datasets.

> *Ablations for PI network structure*

Following the reviewer’s advice, we have added ablations on the width and depth of PI-related modules in Appendix C. 6.2. 

1. *Ablations for the width for PI-related modules*: In the paper we set the width of the PI-related modules (for both noise network and gate network) by default to 1024 for CIFAR-10H, and 2048 for all other experiments, without tuning. Here we provide the results for different widths for the PI networks on three CIFAR datasets. 
| Dataset \ Width |    512   |   1024   |   2048   |   4096   |
|:---------------:|:--------:|:--------:|:--------:|:--------:|
|    CIFAR-10H    | 72.8±2.9 | 71.3±3.3 | 71.4±3.6 | 71.2±3.8 |
|    CIFAR-10N    | 83.8±0.4 | 83.6±0.7 | 84.9±0.4 | 85.3±0.2 |
|    CIFAR-100N   | 62.3±0.2 | 63.7±0.4 | 64.2±0.3 | 64.4±0.2 |

From the table we observe that the performance of Pi-DUAL benefits from larger network width for the PI-related modules

2. *Ablations for the depth for PI-related modules*: In the paper we set the depth of the PI-related modules to 3 by default for all experiments, without tuning. Here we provide the results for different depths for the PI networks on three CIFAR datasets. Note that the width of the PI-related modules are set to the default value when tuning the depth.
| Dataset \ Depth |     2    |     3    |     4    | 
|:---------------:|:--------:|:--------:|:--------:|
|    CIFAR-10H    | 68.9±2.0 | 71.3±3.3 | 70.1±3.5 | 
|    CIFAR-10N    | 83.4±0.4 | 84.9±0.4 | 85.2±0.3 | 
|    CIFAR-100N   | 59.4±1.0 | 64.2±0.3 | 64.1±0.2 | 

From the table we observe that the depth of the PI-related modules have to be at least of 3 layers to maximize the performance of Pi-DUAL. We thank the reviewers for leading us to these interesting findings.

Reviewer comment (Novelty):
The primary distinction between PI-DUAL and TRAM [R1] lies in PI-DUAL's approach of separately modeling features and privileged information (PI), as opposed to using two heads on top of the feature vector $x$. While this approach provides some technical novelty, it may be considered somewhat limited.

Author response:
We thank the reviewer for taking the time to review our paper and for all the valuable feedback. We address their comments below:

**Differences between Pi-DUAL and TRAM**

We respectfully disagree with the reviewer in that the innovations of Pi-DUAL with respect to TRAM are minor and correspond only to a change in architecture. As appreciated by this reviewer, the goal of our paper is to build a simple and effective method that requires minimal changes to existing training pipelines. In this regard, while both TRAM and Pi-DUAL utilize PI features to combat label noise, Pi-DUAL provides a solution that is both simpler and more efficient. Compared to TRAM, it has the following key distinctions:

1. **Network structure**: TRAM uses one model but with two heads, one for training and one for testing. In contrast,  Pi-DUAL uses three distinct subnetworks with clear roles during training and inference: the prediction network for fitting clean target distribution, the noise network for explaining away label noise, and the gate network for identifying noisy samples. These three roles can not be accounted for in the architecture of TRAM.
2. **Learning objective**: While the learning objective of TRAM is a weighted sum of two objectives (one for fitting the noisy target distribution, and another for transferring the knowledge to the no-PI head), the learning objective of Pi-DUAL is a simple cross-entropy loss over the noisy targets. For a better illustration of the differences between Pi-DUAL and TRAM, we conducted an ablation which trains Pi-DUAL with the loss function of TRAM, and show that this produces much worse results:

| Dataset \ Method |    CE    |  Pi-DUAL | Pi-DUAL with TRAM loss |
|:----------------:|:--------:|:--------:|:----------------------:|
|     CIFAR-10H    | 51.1±2.2 | 71.3±3.3 |        61.6±4.8        |
|     CIFAR-10N    | 80.6±0.2 | 84.9±0.4 |        81.6±0.9        |
|    CIFAR-100N    | 60.4±0.5 | 64.2±0.3 |        59.0±0.6        |

3. **Modeling of label noise**. TRAM does not explicitly model label noise. Instead,  it learns the conditional distribution of noisy labels with respect to both x and a, to eventually predict with an implicit marginalization over a. On the other hand, Pi-DUAL explicitly models per-instance-level label noise in its architecture, which permits it to identify noisy labels and to effectively reduce the negative impact from noisy labels. In contrast to TRAM, the prediction network of Pi-DUAL learns the conditional distribution of clean labels with respect to x alone. With this explicit modeling of label noise, Pi-DUAL delivers not only better performance, but also a much better interpretability as to how PI helps explain away label noise (e.g., how the noise network overfits to the wrong labels in Figure 2, and how the gating network identifies wrong labels in Figure 3).

With these non-trivial improvements, Pi-DUAL outperforms TRAM++ on all 5 benchmarks achieving up to 8.1 accuracy points more than TRAM++ on ImageNet-PI. Pi-DUAL also delivers much better performance in identifying label noise post-training thanks to the explicit modeling of label noise.

Reviewer comment (Experiments):
I find the experimental results to be somewhat perplexing. It is reasonable that PI-DUAL doesn't need to be compared to two-stage methods like DivideMix or other semi-supervised pipelines. However, the test accuracy presented in Table 2 exhibits a notable gap compared to the results reported in the original papers or public leaderboards [R2]. Additionally, the test accuracy for TRAM on CIFAR-10N, as presented in the original paper, was 71.8, but in Table 2, it's only 64.9. While I acknowledge that differences in training settings may account for this, it would be beneficial to conduct specific experiments to compare each method under their optimal configurations.

Author response:
**Comparison with results in literature**

We absolutely agree with the reviewer that all methods should be compared under their optimal configurations and this has precisely been our intention in all our experiments. All our experiments, including the reimplementation of other methods, are built on the open-source uncertainty baselines codebase (Nado et al., 2021) and follow as much as possible the benchmarking practices standardized by Ortiz-Jimenez et al. (2023).

As detailed in Appendix B, we follow the same hyper-parameter tuning pipeline for all methods, namely we perform a grid search over their main hyper-parameters using a noisy validation set randomly split from the train set. In this regard, we want to highlight that, since Pi-DUAL only introduces one additional hyper-parameter compared to the cross-entropy baseline, it has the lowest tuning budget compared to the other methods that all have  at least two extra hyper-parameters.

Moreover, we want to clarify that there are no discrepancies between the results we report and those  from the literature, as the reviewer points out. Specifically:
- On CIFAR10-H, our results for TRAM agree with those released by the original TRAM authors in their follow-up paper (Table 1 in Ortiz-Jimenez et al. 2023). The performance gap between Collier et al. (2022) and Ortiz-Jimenez et al. (2023) is due to the fact that they use two different versions of  CIFAR10-H. Collier et al. (2022) use the “uniform” version while Ortiz-Jimenez et al. (2023) use the harder “worst” version; see discussion in Sec. 2.1 of Ortiz-Jimenez et al. (2023). All our experiments are trained using the latter version of the dataset.
- In all our tables, to ensure a fair comparison, we report results of the standard ELR and SOP methods which operate without any additional semi-supervised learning techniques. We note that the [leaderboard](www.noisylabels.com) reports separate results for ELR (our baseline) and ELR+ (including semi-supervised learning). For SOP however, we have found that the leaderboard mistakenly reports the results for SOP+ (including semi-supervised learning) instead of SOP, as one can see by comparing to the original SOP paper (cf. Table 4 of Liu et al. (2022)). **Taking this into account, we highlight that our results thus perfectly agree with those reported in the literature, and even exceed them due to better hyper-parameter tuning** (e.g. our ELR baseline is 5 points better on CIFAR100N than the one in the leaderboard).

We hope these clarifications will make the reviewer increase their confidence in our results. We fully understand the initial confusion of the reviewer due to the benchmarking disparities in the literature. Our goal was precisely to ensure a fair comparison of the different methods under the same settings to be able to draw solid conclusions.

Reviewer comment (Reproducibility):
What hardware requirements are needed for Pi-DUAL training for large datasets? If a training cost analysis is provided, I think it can be more useful for deploying your method in more scenarios.

Author response:
**Computational cost of Pi-DUAL**

In our experiments, we found that Pi-DUAL does not have extra computational requirements compared for hardware to the standard training baseline. The added cost of processing the PI features (which typically have much fewer dimensions) is indeed negligible with respect to processing the whole images. In this sense, training and inference using Pi-DUAL take roughly the same time as training with standard cross-entropy, TRAM++ or HET. For our large-scale datasets (i.e., ImageNet-PI), we used a TPU V3 with 8 cores for all our experiments.

|                        |   CE   | TRAM++ | Pi-DUAL |   HET  | SOP | ELR |
|:----------------------:|:------:|:------:|:-------:|:------:|:---:|:---:|
|  Number of parameters  |   26M  |   32M  |   36M   |   58M  | >1B | >1B |
| Training time per step | 0.510s | 0.541s |  0.566s | 0.575s |  -  |  -  |

We use the parameter count as a proxy for the memory cost. Since Pi-DUAL models the per-instance level of noise using the PI features, it does not require introducing parameters that scale with the number of samples or classes as SOP and ELR need to. In fact, the extra parameters needed in our experiments to account for the noise and gating networks are comparable to those of TRAM++ and even fewer than the ones introduced by the no-PI method HET.

We have added these comments to the Appendix D of our new version.

Reviewer comment (Experiments):
How does Pi-DUAL perform in terms of security and privacy protection? For example, is there a risk that privileged information may be compromised? If there is no PI exists, what the performance will be?

Author response:
**Privacy concerns of using PI**

We thank the reviewer for bringing this topic to our attention and we will make sure to discuss it in our camera-ready work. However, we note that none of our results require the use of personally identifiable annotator IDs. In fact, cryptographically safe IDs in the form of hashes work perfectly fine as PI. In this regard, we do not think that there are serious concerns about possible identity leakages stemming from our work if the proper anonymization protocols are followed. In addition, we have added an ethics statement to the paper regarding the privacy and safety protection requirements of Pi-DUAL.

**Performance without PI**

As mentioned throughout the paper, we would like to emphasize that Pi-DUAL has been explicitly designed to leverage PI features. In this regard, its good performance hinges on the availability of good PI features such as in the case of ImageNet-PI or CIFAR-10H (Ortiz-Jimenez et al. 2023).

However, as mentioned by the reviewer, it is still interesting to study the performance of Pi-DUAL when no PI is available, as we show in Table 4 in the paper (last row). In these cases, we can always construct some randomly-generated PI to train PI-DUAL, following the  procedure of Ortiz-Jimenez et al. (2023). As shown in Table 4 (and copied below for reference), Pi-DUAL with the randomly-generated PI still outperforms the cross-entropy baseline on the CIFAR datasets, but its performance is much worse than when using the real PI of the datasets.

|                        |  CIFAR-10H |  CIFAR-10N | CIFAR-100N | ImageNet-PI (high noise) | ImageNet-PI (low noise) |
|:----------------------:|:----------:|:----------:|:----------:|:------------------------:|:-----------------------:|
|           CE           |  51.1±2.2  |  80.6±0.2  |  60.4±0.5  |         68.2±0.2         |         47.2±0.2        |
|    Pi-DUAL (full PI)   |  71.3±3.3  |  84.9±0.4  |  64.2±0.3  |         71.6±0.1         |         62.1±0.1        |
| Pi-DUAL (only rand PI) |  53.5±2.2  |  83.7±1.3  |  61.8±0.3  |         68.4±0.1         |         47.0±0.4        |