Paper: Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models (ICLR-2024)

Reviewer comment (Experiments):
It seems that some hyperparameter tuning (detailed in appendix C) was done for all of the loss modification methods, but none was done for MLE. Because of this, perhaps some of the gains over MLE can be attributed to randomness in training, rather than to inherent goodness of the method.

Author response:
> perhaps some of the gains over MLE can be attributed to randomness in training, rather than to inherent goodness of the method.

To address the reviewer’s concerns, we performed additional tuning on the MLE baseline for our opus-MT experiments (Table 4), varying the learning rate and dropout probability:

| lr              | dropout | BLEU |
|-----------------|---------|------|
| 5e-4 (our reported baseline) | 0.1     | 30.1 |
| 1e-3            | 0.1     | 30.0 |
| 1e-4            | 0.1     | 28.4 |
| 7e-4            | 0.1     | **30.2** |
| 5e-4            | 0       | 29.1 |
| 5e-4            | 0.2     | **30.2** |
| 5e-4            | 0.3     | 29.8 |

**We tuned the baselines extensively.** We show the number of additional hyper-parameter configurations for each baseline in the following chart: 

 | Method         | Number of Hyper-parameters Tuned |
|---------------------|----------------------------------|
| TaiLr [3]           | 10                               |
| Loss Truncation [2] | 3                                |
| ENT-Threshold       | 3                                |
| ENT-Fraction        | 1                                |

Reviewer comment (Evaluation):
The results on real-world datasets (sections 5.3 and 5.4) are somewhat underwhelming. I see small gains (and perhaps small gains are a good result already given how simple the method is), but I'm also a little bit concerned whether these are interesting enough for practitioners to be excited and go back and implement/use this method. Overall, I feel like the paper lacks a big convincing results, such as significant improvements to SOTA on a dataset that people care about.

Author response:
> Overall, I feel like the paper lacks a big convincing results, such as significant improvements to SOTA on a dataset that people care about. 

Our main claim is that our method improves the robustness of text generation models against various types of data noise, outperforming **existing truncation methods based only on loss** across three different tasks, rather than our method achieving SOTA in language generation. SOTA methods for different tasks usually require task-specific handling (e.g., using bilingual pre-trained masked language models for translation [1], using contrastive learning between candidate summaries for summarization [4]), making them less generalizable. In contrast, our proposed approach **does not make such task-specific assumptions**, making it applicable to any text generation task to improve robustness.

Moreover, **our gains are not limited to synthetic settings**: We would like to highlight that our method yields large improvements in language modeling (-1.58 Perplexity, Table 3), and in Appendix F, we show that our method improves upon the MLE baselines from +0.5 to 0.9 BLEU on Paracrawl MT data. To further address the reviewer’s concerns, we added experiments to show that our method can be used in conjunction with SOTA task-specific methods [1, 4] to improve the performance:

| IWSLT 14 De-En  | Test BLEU |
|----------------------------------|-----------------|
| BiBERT (Table 2 in [1])      | 37.58     |
| BiBERT (reproduced) | 37.62     |
| BiBERT + ENT-Fraction        | **38.29**    |

| CNN/DM            | Rouge-1 | Rouge-2 | Rouge-L |
|-------------------|---------|---------|---------|
| BRIO (Table 2 in [4])   | 47.78   | 23.55   | 44.57   |
| BRIO (reproduced) | 47.81   | 23.52   | 44.52   |
| BRIO + ENT-Fraction  | **48.04**   | **23.61**   | **44.81**   |

We also conducted additional experiments in a more realistic setting: We use the Human+GPT mixture data in [5] and perform instruction tuning on LLama 2 7B. We show the results of adding ENT below:

|                                   | MMLU (0-shot Acc.) | TyDiQA (F1) |
|-----------------------------------|--------------------|-------------|
| Llama2-7b instruct (Table 5 in [5])     | 49.2               | 52.8        |
| LLama2-7b instruct (reproduced)   | 50.1               | 52.1        |
| LLama2-7b instruct + ENT-Fraction | **51.3**               | **54.0**        |

Reviewer comment (Experiments):
How was the thereshold hyperparameter tuned in all experiments in the experimental section?

Author response:
> How was the threshold hyperparameter tuned in all experiments in the experimental section?

We choose the best of {1.35, 1.38, 1.4} for ENT-Threshold in all our experiments. More details of how we tuned the hyper-parameters can be found in Appendix C.

Reviewer comment (Theory):
I was confused by the second equation in section 3, should it be a "less-than" sign rather than a "greater-than" sign?

Author response:
>  I was confused by the second equation in section 3, should it be a "less-than" sign rather than a "greater-than" sign?

We thank the reviewer for pointing this out, and we have made revisions to the manuscript (marked in blue).

Reviewer comment (Theory):
One motivation of the proposed method is that previous works treat all non-ground truth tokens as equally incorrect. However, I do not see how the proposed method solves this problem. In my opinion, all non-ground truth tokens are treated in the same way in the proposed method. Can you explain?

Author response:
We understand that there is a nuance in how we define “all non-ground truth tokens are treated in the same way”.  We meant that existing methods ignore the predicted distribution over the non-ground truth tokens - therefore treating all probability distributions over the non-ground truth tokens as equal, whereas our method differentiates between when the predicted distribution over non-target tokens is skewed versus uniform. Empirically, we found compelling evidence that the error norm clearly distinguishes between clean and noisy tokens (Figure 4). Thank you for pointing this out, and we have made revisions to clarify our motivation in Section 2 (Marked in blue).

Reviewer comment (Theory):
One potential improved version of the proposed method is to take into account the similarity between non-ground truth tokens and ground truth tokens when computing the l2 norm.

Author response:
Your point is well taken. We agree that a variant of our method that incorporates semantic similarity is an exciting future direction. However, one challenge is that the target and non-target token could be very similar and the data might still be an error (e.g., The capital of Maryland is Baltimore V.S. Annapolis). As also mentioned by Reviewer p8xo, Figure 4 provides compelling evidence that the error norm itself clearly distinguishes between clean and noisy data, therefore we opted to only use error norm for simplicity.

Reviewer comment (Theory):
Even though the results have proven the effectiveness of ENT, I think the motivation of using error norm to estimate data quality can be further discussed and provide more insights about how do they correlate.

Author response:
We show that using the error norm to detect noise is both **theoretically grounded** and **empirically effective**:

- **Theoretical:** The impact on the training loss when a single datapoint $x, y$ is removed is upper bounded by its gradient norm $||\nabla \mathcal{L}(x, y)||$ [1]. Moreover, the impact on test loss when a single datapoint is removed can be also upper bounded by the dot product between its gradient and the gradient on the test dataset $\nabla \mathcal{L}(x, y)^\top \nabla \mathcal{L}(x_\text{test}, y_\text{test})$ [2], indicating that the single sample gradient plays an important role in estimating the data utility. However, single sample gradients are impractical to obtain during batched training. We can approximate the single sample gradient with the error L2 norm [1]. **Therefore, by truncating tokens with a high error l2 norm, we are truncating tokens that have a high impact on the loss, which are likely to be noisy examples.** More discussions on relations with Influence Functions and other data utility measurement methods can be found in Appendix B.

- **Empirical:**  An established understanding of L2 norm is that it is more sensitive to outliers [3] than L1 norm (or equivalently, log-likelihood loss, proof in Appendix A). Therefore, outliers in the data would have larger L2 norms. As also mentioned by reviewer p8xo, Figure 4 provides strong empirical evidence on L2 norm clearly distinguishing between clean and noisy tokens.

If there are any details that need further improvements, we are open to your suggestions.

Reviewer comment (Theory):
Theoretically, whether an example is helpful to train a model not only depends on the correctness of its label but also on the uncertainty (or entropy) of this example, according to the lessons from active learning. As uncertainty also takes into account of the prob of non-targets as ENT does. Therefore, it would be important to discuss the relationship between uncertainty and ENT in this paper.

Author response:
Thank you for pointing this out, **we have included extended discussions between ENT and uncertainty sampling in Appendix B**. While both ENT and uncertainty sampling use predicted probabilities to select data, generally, uncertainty sampling for active learning either focuses on low predicted probability (high loss) without considering context entropy or on high entropy without considering probability on the target token. **In contrast, ENT truncates examples with both low predicted probability and low entropy, providing a more accurate estimation of data quality (see Figure 5).** Again, we have added a coverage of the related work for your attention. If there is anything that you’d like us to cover here, please let us know.

Reviewer comment (Experiments):
The authors claim that previous work would rely on hyper-parameter tuning (on fraction or threshold). But the proposed ENT method also requires these hyper-parameters. I am not sure the proposed method would be more efficient on hyper-parameter selecting.

Author response:
We report the number of hyper-parameters we tuned for each baseline below:

| Method         | Number of Hyper-parameters Tuned |
|---------------------|----------------------------------|
| TaiLr [1]           | 10                               |
| Loss Truncation [2] | 3                                |
| ENT-Threshold       | 3                                |
| ENT-Fraction        | 1                                |

In addition to fraction/threshold, the starting iteration at which we start to apply other baselines can have a significant impact on the final performance, as shown in Figure 6. **In contrast, ENT exhibits the best performance and least variance across different starting iterations.**

Reviewer comment (Experiments):
In $5.2, apart from the two settings, another setting "Over translated" (noted by the authors in Figure 2) should also be included.

Author response:
We included experiments by adding random words into the target sentence in opus En-Fr:

|     | 0%   | 10%  | 20%  | 30%  | 40%  | 50%  |
|-----|------|------|------|------|------|------|
| MLE | 36.5 | 36.3 | 36.1 | 36.0 | 35.8 | 35.4 |
| ENT-Fraction | 36.7 | 36.7 | 36.6 | 36.4 | 36.3 | 36.1 |

Our results show that adding random words into the target sentence does not harm the performance as much as untranslated target sentences, and **ENT was able to outperform the MLE baseline at all noise levels.**

Reviewer comment (Experiments):
In $5.4, it is inadequate to draw such a conclusion on using which method for pre-trained model and using which method for from-scratch model, with only a summarization task. More downstream tasks should be conducted to draw such a conclusion.

Author response:
We thank the reviewer for pointing this out. We have revised this section in our latest manuscript.

Reviewer comment (Evaluation):
In Table 1, when the untranslated ratio is 10%, MLE output performs all the methods with a relatively large margin. This is interesting and also confusing, comparing to other ratio. Do you have any explain or study on this? Moreover, in real scenoria, I think the untranslated ratio in the dataset is not as large as more than 20%, so does this result suggest that under common situation, using MLE is enough to beat other modifications?

Author response:
We suspect that the superior performance of MLE at small noise ratios can be attributed to certain engineering choices in the baseline. For example, existing work has shown that label smoothing and the choice of optimizers can have influence on robustness [1,2]. We agree that quantifying the strength of implicit robustness enhancement effect of label smoothing and optimizers, as well as how they interact with data truncation methods is an exciting direction for further research. 

While MLE outperforms others when 10% of the entire data is injected, we wouldn't conclude that MLE is enough to beat data truncation methods under a common situation. In practice, **the type and the amount of noise can vary depending on the dataset**, especially if we take the possibility of an adversary manually injecting sentences with multiple types of noise into the training corpus. 

Under a more practical setting, we validate our method on **naturally noisy Paracrawl data** and report the BLEU/COMET scores the WMT 22 test set: 

|               | En-Cs         | En-Ru         | En-Zh       |
|---------------|---------------|---------------|-------------|
| MLE           | 25.2/69.6     | 24.6/69.5     | 12.5/63.8   |
| ENT-Threshold | **25.7/70.9** | **25.5/70.2** | **13/65.5** |

**ENT achieves +0.63 BLEU and +1.23 COMET on average over the baseline in a practical, naturally noisy setting.**

Reviewer comment (Evaluation):
In Table 2, the advantages of ENT and also the Loss Trunc. and TaiLr are not significant against MLE when ratio <=30%. Also in real scenarios, the misordered ratio might not be that high, which also raises the same question as above.

Author response:
While MLE outperforms others when 10% of the entire data is injected, we wouldn't conclude that MLE is enough to beat data truncation methods under a common situation. In practice, **the type and the amount of noise can vary depending on the dataset**, especially if we take the possibility of an adversary manually injecting sentences with multiple types of noise into the training corpus. 

Under a more practical setting, we validate our method on **naturally noisy Paracrawl data** and report the BLEU/COMET scores the WMT 22 test set: 

|               | En-Cs         | En-Ru         | En-Zh       |
|---------------|---------------|---------------|-------------|
| MLE           | 25.2/69.6     | 24.6/69.5     | 12.5/63.8   |
| ENT-Threshold | **25.7/70.9** | **25.5/70.2** | **13/65.5** |

**ENT achieves +0.63 BLEU and +1.23 COMET on average over the baseline in a practical, naturally noisy setting.**