Paper: MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods (ICLR-2024)

Reviewer comment (Writing):
Some parts of the paper (e.g., the first three paragraphs of the introduction) talk about NLG in general but the paper focuses on machine translation. Reranking methods, in particular, rely on good quality estimation models that exist for MT but may not exist for other tasks. Of course, MBR finetuning can be applied to other NLG tasks (e.g., summarization) but this paper does not touch this problem. I think the authors should either revise the writing and make the scope more clear from the beginning, or perform experiments in other NLG tasks. The contribution is mainly empirical and only validated for MT.

Author response:
MT is a common testbed for NLG tasks. In the abstract, we state the paper's scope: " Using the canonical NLG task of Neural Machine Translation (NMT), ...". Moreover, while the first three paragraphs of the introduction discuss NLG in general (establishing the motivation and setup in the most general terms), the fourth paragraph already narrows the focus to MT: "In this work, we focus on the NLG task of Neural Machine Translation (NMT)...". In the conclusion, we propose "extending MBR and QE finetuning to other NLG tasks besides NMT" as an avenue for future work. 

Note that MBR decoding has been successfully applied to other NLG tasks including text summarization, image captioning, and data-to-text using BERTScore as the utility function, as described in https://arxiv.org/pdf/2311.05263.pdf (Model-based Minimum Bayes Risk Decoding; Jinnai et al., 2023). So applying MBR finetuning to these other tasks would be a natural extension of this work, and would likely perform well.

All of this said, given your concerns about the clarity of the scope, we have revised the first three paragraphs of the introduction, to make explicit from the very beginning that the scope is limited to MT.

Reviewer comment (Experiments):
The study is limited in terms of language coverage. They perform experiments on 2 language pairs only (mid & high resource), both for translation out of English. It’s not clear if the findings hold for lower resource languages and when translating into English. In particular, I’m expecting quality estimation models to be worse for low resource languages, which may end up affecting the quality of the translations produced with their method. Have you tried other languages? If so, what happens in that case?

Author response:
The quality of MT metrics on low-resource language pairs and into-English language pairs has been evaluated in the WMT Metrics Shared Tasks. See results from WMT'22 (https://aclanthology.org/2022.wmt-1.2.pdf)  and WMT'23 (https://www2.statmt.org/wmt23/pdf/2023.wmt-1.51.pdf), which show that 1) the MetricX metric (including the QE variant) generalizes well to language pairs unseen during training, including low-resource language pairs (e.g. see Hebrew-English in Table 8 of the WMT'23 Metrics Shared Task Report) and 2) BLEURT and MetricX-XXL perform well on into-English language pairs. Note that result 2) is not surprising given that most neural MT metrics, including BLEURT and MetricX-XXL, were finetuned on large language models which were pretrained mostly on English data. 

While this paper focuses on mid and high-resource language pairs, existing evidence about the quality of utility functions for low-resource language pairs and into-English language pairs suggests that MBR and QE finetuning would likely work in these settings as well. This would be a fruitful avenue for future work.

Reviewer comment (Experiments):
The related work discusses other methods for training NMT models beyond MLE (e.g., RL methods) but none of them is used as a baseline.

Author response:
For a correct comparison to RL baselines, we would need to implement them in our framework, to be able to run them on the same data and using the same utility function. Additionally, we would need to tune the RL settings to ensure a fair comparison. Thus, evaluating RL baselines is outside of the scope of this paper. Moreover, the MBR and QE finetuning methods proposed here use MLE, so comparison with RL methods would be tangential to this paper's focus.

Reviewer comment (Evaluation):
MBR decoding uses BLEURT and is prohibitively expensive. Note, however, that more efficient alternatives exist. See implementation details discussed in [1], Section 4.3. Is there a reason to choose BLEURT? While efficiency at inference time is an advantage, MBR/QE finetuning is expensive at training time. It would be good to see some numbers to understand the training/inference time difference when compared to other existing approaches.

Author response:
The two proposals in [1], Section 4.3 for increasing the efficiency of MBR decoding are to 1) use unique samples (deduping the candidate list) and 2) use non-neural utility functions (chrF++ and BLEU) or use COMET instead of BLEURT as the utility function. Both of these proposals increase efficiency at the cost of quality.

Deduping (using unique samples) removes the implicit weighting of hypotheses by their model probability (counter to the intention of MBR) and hurts performance substantially based on internal results. Using chrF++, BLEU, and COMET as MBR utility functions all underperform using BLEURT, based on internal experiments which were performed to choose the best utility function. We hypothesize that COMET's separate encoding of the candidate and pseudo-reference translations, while more efficient, hurts quality. All of this said, comparing different utility functions is outside the scope of this paper, even though we agree that it is an interesting avenue of research.

Reviewer comment (Theory):
In the introduction you say that MBR decoding requires the generation of a large number of candidates. While this is true if you get unbiased samples from the model, it does not seem to be the case when you bias the distribution (e.g., using nucleus sampling). I suggest you see the discussions in [2] and [3] and comment. Also, see my comment above about [1].

Author response:
It was shown in https://arxiv.org/pdf/2305.09860.pdf (Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation; Freitag et al., 2022) that epsilon sampling is the best sampling strategy for MBR decoding. In particular, it was shown that epsilon sampling outperforms nucleus sampling (according to human evaluation) when using a candidate size of 1024. 

Also see Table 3 and Table 4 in https://arxiv.org/pdf/2311.05263.pdf (Model-based Minimum Bayes Risk Decoding; Jinnai et al., 2023) for a comparison of MBR decoding using epsilon, top-k, nucleus, and ancestral sampling (with candidate size of 256), where epsilon sampling performs best for both English-German and Russian-English. (Also see Appendix C in the same paper, where it is shown that epsilon sampling outperforms all other sampling methods not only for machine translation, but also for text generation and image captioning tasks.)

Moreover, note that in Figure 4 in the Appendix of our paper, we show that QE reranking performance monotonically improves with increasing candidate size (while, interestingly, this is not the case for QE finetuning). This is true even when using epsilon sampling which, like nucleus sampling, biases the distribution.

Reviewer comment (Evaluation):
Results use COMET-20 instead of more recent versions already available online such as COMET-22. Is there a reason for using the 2020 version?

Author response:
We already report performance for both English-German and English-Japanese models on four diverse automatic metrics in Table 5 and Table 8, respectively. We chose COMET-20, rather than COMET-22, as our primary automatic evaluation metric since it is still the canonical metric used in the literature. 

As requested, we have also added COMET-22 scores to **Table 8** (for English-German) and **Table 11** (for English-Japanese). (We also added chrF as a representative lexical metric for completeness, as requested by another reviewer.)

Reviewer comment (Experiments):
Human evaluation results are important and should not be in the appendix (Table 7). In fact, I think they should be more highlighted in the paper! It would also be interesting to see if they generalize for En-Ja. Can you explain the reasoning for using 9 professional translators for En-De instead of using fewer and evaluate En-Ja as well?

Author response:
Human evaluation results have been moved to the main body of the paper. The human evaluation was only performed on English-German due to budget constraints and lack of availability of an English-Japanese rater pool within a reasonable timeframe.

Reviewer comment (Experiments):
What happens if you decode with reranking techniques (QE/MBR) using a model trained with your method, instead of using beam/greedy search? Even though this would make the method very inefficient at inference time, it would be interesting to see if it further boosts the performance.

Author response:
As requested, we have added **Table 19** with MBR/QE decoding results for the MBR/QE-finetuned models, in comparison against the more efficient decoding methods of beam search, greedy decoding, and epsilon sampling. (In addition, we have also added the performance of not only QE-finetuned, but also MBR-finetuned, models to **Table 4**.)

We see that after MBR finetuning, QE reranking works better than MBR decoding. Similarly, after QE finetuning, MBR decoding works better than QE reranking. This suggests that using the same decoding method both to generate the finetuning data and at inference time overfits to the metric used as the utility function. Also note that the gap between MBR/QE decoding and beam search/greedy decoding is small for the MBR/QE-finetuned models, and for Phases 2 and 3 of the QE-finetuned models, as well as Phase 3 of the MBR-finetuned model, beam search or greedy decoding is actually the top-performing decoding strategy.

Reviewer comment (Experiments):
According to Table 3, both beam and greedy search work well. Does this mean that your method helps solve the beam search curse [4]? What happens when you increase the beam width?

Author response:
Beam search with a small beam size (of 4, used here) should not be subject to the beam search curse. Moreover, the fact that greedy search performs on par with beam search certainly does not suggest that the beam search curse is solved with MBR/QE finetuning. If the beam search curse were solved, we should see that beam search with a beam size of 4 outperforms greedy, and this would motivate further experiments with a larger beam width. But that is not what we observe here. 
Since greedy decoding performs on par with beam search here, we are able to benefit from a very efficient decoding method for the MBR/QE-finetuned models at inference time.

Reviewer comment (Experiments):
The paper presents a section of the results from Palm2 finetuning and show large improvements. I think that is expected because it is a much stronger model. This set of experiments doesn't add too much value to the paper. The paper should also show the beam and MBR decoded results of the Palm2 model used for finetuning, so that the readers can understand how much of the teacher model performance can be transferred to the student via this finetuning.

Author response:
As requested, we have added the performance of the en-de and en-ja PaLM-2 QE-reranked, MBR-decoded, and greedy-decoded teacher models in **Table 10** and **Table 12** in the new revision, respectively. Note that for en-de, the student model is strong enough that MBR finetuning brings its performance almost on par with that of the PaLM-2 teacher model (student Comet20 = 63.62 vs teacher Comet20 = 63.76). Also note that across both language pairs, the MBR-decoded teacher is strongest. Moreover, even though the PaLM-2 QE-reranked teacher is weaker than the greedy-decoded teacher, PaLM-2-QE finetuning achieves much larger gains that PaLM-2-greedy finetuning (rows 3a versus 3c in Table 2). This suggests that QE finetuning is able to smooth out bad QE translations, e.g. due to utility function failure modes (while QE reranking at inference time is not robust to bad translations), allowing the student to exceed the performance of the teacher. In particular, the en-de PaLM-2-QE-finetuned student achieves a Comet20 score of 62.35, exceeding its teacher's score of 60.86.

Reviewer comment (Experiments):
After using MBR/QE finetuning, do we still see a difference between beam decoding and MBR/QE decoding? Table 3 summarizes the results using beam/greedy and sampling, but MBR/QE decoding results should also be included for completeness.

Author response:
As requested, we have added **Table 19** with MBR/QE decoding results for the MBR/QE-finetuned models, in comparison against the more efficient decoding methods of beam search, greedy decoding, and epsilon sampling. (In addition, we have also added the performance of not only QE-finetuned, but also MBR-finetuned, models to **Table 4**.)

We see that after MBR finetuning, QE reranking works better than MBR decoding. Similarly, after QE finetuning, MBR decoding works better than QE reranking. This suggests that using the same decoding method both to generate the finetuning data and at inference time overfits to the metric used as the utility function. Also note that the gap between MBR/QE decoding and beam search/greedy decoding is small for the MBR/QE-finetuned models, and for Phases 2 and 3 of the QE-finetuned models, as well as Phase 3 of the MBR-finetuned model, beam search or greedy decoding is actually the top-performing decoding strategy.

Reviewer comment (Evaluation):
In Table 1, we see 2a perform better than 1c) and 2c) perform better than 1b). This is a bit surprising because I would have assumed that finetuning performance will be upper bounded by MBR/QE used during run-time. I couldn't find any comment on this comparison in the paper.

Author response:
MBR decoding and QE reranking can produce translations with high utility score but low quality, due to honeypots/failure modes of their utility functions. While this directly degrades performance of the teacher models at inference time, the MBR and QE-finetuned students are able to smooth out these bad translations. This explains why student performance is not upper-bounded by performance of the teacher and can, in fact, exceed it.

This hypothesis is supported by work from Amrhein and Sennrich, 2022 (https://arxiv.org/pdf/2202.05148.pdf) which shows that, despite the strong performance of MBR decoding and QE reranking on average, these decoding methods suffer from failure modes which can be attributed to underlying problems with the utility function. They performed the study using Comet20 as the utility function, and showed that this metric is not sensitive enough to discrepancies in numbers and named entities. MBR and QE finetuning, on the other hand, would not necessarily be as susceptible to these same failure modes since, even though the model would be exposed to these extreme examples, its representation would be smoothed out over the course of training.

Reviewer comment (Evaluation):
The paper does not have any numbers on lexical metrics like chrF. Would have been nice to include those numbers at least in the Appendix, for interested readers.

Author response:
As requested, we have added performance on chrF to **Table 8** (for English-German) and **Table 11** (for English-Japanese) in the Appendix.

Reviewer comment (Evaluation):
While this paper reports significant gains with MBR and QE finetuning, the reasons for the improvements are not well analyzed. In some cases, QE finetuning performs better, while in others, MBR finetuning is superior. More details and an in-depth analysis of the advantages and limitations of each finetuning method would be expected.

Author response:
This paper does provide an explanation for the improvements observed with MBR and QE finetuning in paragraph 2 of the Discussion section (Section 7), and also provides a comparison of MBR versus QE finetuning in paragraph 3. Both paragraphs are pasted below for reference:

1. **Explanation for improvements (Section 7, paragraph 2)**: "Given the effectiveness of an external teacher model (which differs in architecture, training data, etc. from the student), the success of these finetuning methods can likely not be primarily explained by mitigation of the label exposure bias problem (Schmidt, 2019), since the data generated from an external teacher is less similar to the data the model has already seen (during previous training stages) than self-generated MBR or QE data. Instead, the effectiveness of MBR and QE finetuning seemingly can be attributed primarily to the high quality of the translations used as finetuning data."
* Also note that the experiments were designed so that each of the 3 phases answered a different question regarding the effectiveness of MBR and QE finetuning (see **Section 5.4**). For example, by comparing Phase 1 and Phase 2, we show that using a larger monolingual corpus to generate the MBR and QE finetuning data improves performance, and by comparing Phase 2 and Phase 3, we show that using a stronger teacher model also improves performance. 
* Also see **Appendix D** for extensive ablation studies which isolate the effect of various source-side and target-side variables on the performance of MBR and QE-finetuned models. For instance, we show that QE finetuning on forward-translated data is more effective than QE finetuning on backtranslated data (where the QE translations are on the source, rather than target, side). All of these ablations serve to understand the reasons for the observed improvements.
2. **Comparison of MBR vs QE finetuning (Section 7, paragraph 3)**: "When comparing MBR against QE finetuning, we see that QE tends to perform at least as well as MBR finetuning using a self-teacher, while MBR outperforms QE finetuning using the LLM teacher. We hypothesize that the sampling translations from the self-teacher do not closely approximate the true reference distribution, so using them as pseudo-references does not help (or even hurts), relative to using a reference-free (QE) metric. For higher-quality models (e.g. PaLM-2), the candidate translations are good enough approximations to references that they provide a useful signal."
* We also mention in the introduction that generating data via QE reranking is much more efficient than generating data via MBR decoding: "Despite its quality advantages, the slow inference speed of MBR decoding remains a limitation even when generating distillation data". In terms of computational cost, this is the key limitation of MBR finetuning. So the limitations of MBR relative to QE finetuning both in terms of quality (MBR finetuning requires a stronger teacher) and computational cost are included in the paper. Also see **Table 7** (added to the paper based on your comment #3). 

If 1. and 2. above do not satisfy the requirement for 1) an analysis of the reasons for the improvements due to MBR and QE finetuning and 2) a comparison of the advantages and limitations of MBR vs QE finetuning, please do clarify what is missing.

Reviewer comment (Experiments):
Please report the performance of the PaLM-2 Bison model.

Author response:
As requested, we have added the performance of the English-German and English-Japanese PaLM-2 greedy-decoded, QE-reranked, and MBR-decoded teacher models in **Table 10** and **Table 12** in the new revision, respectively. Note that for English-German, the student model is strong enough that MBR finetuning brings its performance almost on par with that of the teacher model (student Comet20 = 63.62 vs teacher Comet20 = 63.76). Also note that across both language pairs, the MBR-decoded teacher is strongest. Moreover, even though the PaLM-2 QE-reranked teacher is weaker than the greedy-decoded teacher, PaLM-2-QE finetuning achieves much larger gains that PaLM-2-greedy finetuning (rows 3a versus 3b in Table 2). This suggests that QE finetuning is able to smooth out bad QE translations, e.g. due to utility function honeypots/failure modes (while QE reranking at inference time is not robust to bad translations), allowing the student to exceed the performance of the teacher. In particular, the English-German PaLM-2-QE-finetuned student achieves a Comet20 score of 62.35, exceeding its teacher's score of 60.86.

Reviewer comment (Experiments):
For English-German, MBR finetuning seems to outperform QE finetuning. Why was only QE finetuning evaluated for English-Japanese instead of also evaluating MBR?

Author response:
First, to address the claim that "MBR finetuning seems to outperform QE finetuning": For English-German, MBR finetuning only outperforms QE finetuning for the stronger teacher models (self-teacher finetuned on references and PaLM-2 teacher), but QE finetuning outperforms MBR finetuning for the weaker teacher (self-teacher base model). This is brought up in the Discussion section (Section 7, paragraph 3). Also note that MBR and QE finetuning are not directly comparable since they use different utility functions. So we decided to only perform QE finetuning for English-Japanese, since we already ran experiments for both MBR and QE finetuning for English-German, and since QE data generation is much more efficient than MBR data generation. We were already able to achieve large gains with QE finetuning for English-Japanese, without having to resort to more expensive data generation, which is a win both in terms of quality and computational complexity.

Reviewer comment (Reproducibility):
Can you report the training time and compute requirements for the QE and MBR finetuning experiments? Providing the concrete training costs for comparison would be helpful.

Author response:
As requested, we have added **Table 7** (**Appendix B**) in our paper revision, which shows the time and compute requirements for performing MBR-BLEURT and QE-MetricX-XXL scoring, across all three experiment phases and across both language pairs. We use TPUs for BLEURT and MetricX-XXL scoring, as described in https://dl.acm.org/doi/fullHtml/10.1145/3360307. (The TPUv4 hardware is a faster and newer generation than TPUv3.) MBR scoring takes about 7.65 TPUv4 minutes/source sentence, while QE scoring takes about 4.2 TPUv3 seconds/source sentence. So even when using faster hardware for MBR, QE scoring is 109x faster.

As for training time and costs, finetuning was performed using 64 TPUv3s, with a rate of ~0.2 steps per second using a batch size of 32, so training for one epoch on the Common Crawl dataset (from Phases 2 and 3) took ~8.7 hours. The model was also evaluated on the validation set at every step, since the performance of MBR and QE finetuned models often peaked after only a few steps.

Reviewer comment (Writing):
The paper is quite dense at some parts, mainly at the experimental setting. It takes a few passes on those sections to completely understand the results and the settings but it is clear.

Author response:
Acknowledged that the experimental settings section of the paper is "dense". If there is anything unclear about the setup, please do specify and we will be happy to make changes to clarify. We have added **Table 6** in the Appendix (with a reference to this table in Section 5.4) with a summary of the experimental settings (in particular, the datasets used for each experimental phase). We hope this will make the experimental settings section more easily digestible with a single pass.

Reviewer comment (Reproducibility):
Is there any description of MetricX-XXL-QE in another paper? I could not find a paper describing except a paragraph in the Findings paper. How many languages does it support? This seems to be a very big QE model, 30B parameters.

Author response:
1. *Is there any description of MetricX-XXL-QE in another paper? I could not find a paper describing except a paragraph in the Findings paper. How many languages does it support? This seems to be a very big QE model, 30B parameters.*

The description of the reference-based counterpart to MetricX-XXL-QE (trained on the same human judgements data and with the same model architecture) can be found in https://www2.statmt.org/wmt23/pdf/2023.wmt-1.63.pdf (MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task; Juraska et al., 2023). The only difference between the original MetricX model and the QE version we use is that the QE model input is constructed as the concatenation of the source segment, rather than the reference, with the candidate translation (as described in paragraph 3 of Section 2.3 in our paper). This model has a mT5-XXL backbone with 13B parameters.

As shown in the WMT'23 Metrics Shared Task Report (https://www2.statmt.org/wmt23/pdf/2023.wmt-1.51.pdf), MetricX generalizes well to language pairs that it didn't see during training, including low-resource language pairs. See, for example, Hebrew-English performance in Table 8.

Reviewer comment (Experiments):
How much time is added to the training process when using the MBR and QE reranking? Maybe a table illustrating the added wallclock time would be interesting to contrast with the gains in performance.

Author response:
2. *How much time is added to the training process when using the MBR and QE reranking? Maybe a table illustrating the added wallclock time would be interesting to contrast with the gains in performance.*

To address your question, we have added **Table 7** (**Appendix B**) in our paper revision, which shows the time and compute requirements for performing MBR-BLEURT and QE-MetricX-XXL scoring, across all three experiment phases and across both language pairs. We use TPUs for BLEURT and MetricX-XXL scoring, as described in https://dl.acm.org/doi/fullHtml/10.1145/3360307. (The TPUv4 hardware is a faster and newer generation than TPUv3.) MBR scoring takes about 7.65 TPUv4 minutes/source sentence, while QE scoring takes about 4.2 TPUv3 seconds/source sentence. So even when using faster hardware for MBR, QE scoring is 109x faster. Please do let us know if you have additional questions about this new table.

Reviewer comment (Experiments):
How much does the data used to train the QE model and the utility function used in MBR affect the final results for specific domains? Do you have any insights on this? For example, for English-German, it seems most of the MT training data is similar to the data used to train the QE and BLEURT models. What if this is not the case? Do you observe the same improvements or does something change in the translation quality of the best models?

Author response:
3. *How much does the data used to train the QE model and the utility function used in MBR affect the final results for specific domains? Do yo have any insights on this? For example, for English-German, it seems most of the MT training data is similar to the data used to train the QE and BLEURT models. What if this is not the case? Do you observe the same improvements or does something change in the translation quality of the best models?*

In the WMT'22 Metrics Shared Task (https://aclanthology.org/2022.wmt-1.2.pdf), the metrics (including BLEURT and MetricX-XXL) were evaluated on various domains, and they performed well even on the domains that they were not finetuned on (see **Figure 2**).

Moreover, just as the utility functions generalize across domains, the same is true for the MBR-finetuned and QE-finetuned models which use these utility functions, as shown in **Table 9** in our paper. Also note that while the Phase 1 models were finetuned on past WMT test sets (with the source side partially overlapping the data used to train the utility functions), the Phase 2 and Phase 3 models were finetuned on Common Crawl data, and we do see that finetuning on Common Crawl improves generalization across domains.