Paper: MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR-2024)

Reviewer comment (Experiments):
I’m wondering if the comparison is fare for other systems since the proposed model is trained with multiple external resources such as XNLI and especially some data with parallel sentences, and is based on large base pre-trained encoders. I think it would be more convincing if there can be ways to directly compare different evaluation systems (ref-free vs ref-included, score-based vs comparison-based) with the same training resources and base models. (But surely, it would be indeed a benefit if the proposed system can better utilize extra resources.)

Author response:
1. **Concern about using the external data such as XNLI and MT-Prism:**
One of our baseline T5Score uses the same machine translation dataset MT-Prism to train their model. Other than SentBLEU all our baselines are based on pretrained-encoders. *It is fair to compare our systems against these systems since our system does not require costly human annotations of machine translation quality scores from humans.* Regarding comparing reference-included and reference-free systems, it is better to have a reference-free systems since in practice references are not always available. The fact that our system does not require the references during inference is a strength of our system.
2. **Concern about using the same amount of data as the baselines:**
Regarding comparing score-based vs comparison-based systems on same training resources, we have argued in our paper that it is much easier to gather synthetic data for comparison based approach, which means comparison based approach has the advantage of having larger synthetic training data. Since it is the advantage of our approach, constraining our approach to have the same amount of training data would be unfair to our systems.
3. **Concern about using different pretrained model from baselines:**
The reviewer also mentioned having the same base model as our baselines. Our best performing baseline T5-Score uses the mT5 model which is the same model we have used. Our second-best performing baseline used the XLM-Roberta-Large model to train their systems. To make sure our performance is not just due to using mT5 model, we have now trained a XLM-Roberta-Large based model using our method. The performance for this model which we call Comparator-XLMR on the DA20 benchmark is shown below. Comparator-XLMR achieves superior performance compared to baselines and similar performance to our Comparator-Large and Comparator-XXL models.

| Model           | cs-en | de-en | ja-en | pl-en | ru-en | ta-en | zh-en | Avg  |
|-----------------|-------|-------|-------|-------|-------|-------|-------|------|
| Comparator-XLMR | 9.6   | 47.3  | 26.9  | 10.1  | 16.2  | 23.6  | 14.6  | 21.2 |

| Model           | en-cs | en-de | en-ja | en-pl | en-ru | en-ta | en-zh | Avg  |
|-----------------|-------|-------|-------|-------|-------|-------|-------|------|
| Comparator-XLMR | 69.3  | 41.9  | 58.9  | 44.4  | 31.0  | 69.5  | 51.2  | 52.3 |

Reviewer comment (Experiments):
There should be more analysis on the proposed methods (such as those in Section 5). For example, more ablation studies on the training stages and especially the usage of different resources, and more detailed analysis on the metric and perturbation methods in Stage 3. Some of the result and setting sections may be shorten or moved to the appendix.

Author response:
We have performed further ablation studies of our different training stages. We trained ablated versions of our Comparator-Large model by removing different training stages. We show the performance of our ablated models on the DA20 dataset in the following table. Here sub-stage 3.1 refers to using the BERTScore based weak supervision and sub-stage 3.2 refers to the perturbation based weak supervision in our stage 3. The results show the specific contribution of metric and perturbation methods in Stage 3. The BERTScore based weak supervision has more contribution to the final performance. However, all the stages contribute positively to our final performance.
| All Stages | w/o 1 | w/o 2 | w/o 3 | w/o 3.1 | w/o 3.2 | w/o 1,2 | w/o 2,3 |
|------------|-------|-------|-------|---------|---------|---------|---------|
| 36.5       | 35.9  | 35.6  | 28.9  | 28.5    | 35.9    | 35.5    | 9.9     |

Reviewer comment (Experiments):
What pre-trained model did the baseline systems use? Did they use up to XXL models? (Same or similar base models should be utilized for fair comparisons.)

Author response:
1. One of our baseline T5-Score uses an XL sized model. Our other baselines systems use XLM-Roberta-Large and Info-XLM-Large models which have similar size as the mT5-Large model. Our Comparator-Large model which is based on mT5-Large beats all these baselines on the DA20 benchmark. We have also reported results with XLM-Roberta-Large previously in this response which also beats all the baselines. We also want to note that, in contrast to the baselines our system does not need costly human annotation data and is trained on weakly supervised synthetic data.

Reviewer comment (Evaluation):
Are there any ways to convert the relative comparisons to absolute scores? Sometimes, we might still need the scores (for example as rewards for RL).

Author response:
2. Given a set of machine translations, we can perform an all-pairwise comparison with our system and rank the translations based on how many times a translation is deemed better by our system. This ranking can be used to give a score. However, for training RL systems giving an absolute reward score is not always necessary. Recent NLP models that are trained using RL use human preference judgment annotations where human annotators only give better-worse judgments.[1] Our model can be used as a proxy to human annotators to synthetically generate these better-worse judgments, which then can be used to train the RL models.

Reviewer comment (Experiments):
In Stage 2 and the first part of Stage 3, references are required for the training purpose. If those two parts are ablated, how would it influence the results?

Author response:
3. The references are required in stage 2 and 3 to generate the synthetic data. If we ablate the references completely from our system then we are left with only stage 1. The ablation results shown in the previous table shows that without stage 2 and 3 the performance of the system is very poor. Thus access to the reference translations is required for training of our system. *Our focus in this work is to remove the reliance on costly human annotations of translation quality scores for training machine translation evaluation models.* Collecting these annotations are much costlier than collecting reference translations which can even be performed automatically by bi-text mining. Thus we believe reliance on reference translations during training is not a drawback of our system.

Reviewer comment (Novelty):
The novelty seems limited / overemphasized. In Quality Estimation (QE), reference-free ranking approaches have been used for MT quality estimates before it was re-invented in the context of MT metrics competitions. For example, in the very first QE task in 2012 (https://www.statmt.org/wmt12/quality-estimation-task.html) ranking-based evaluations (without references) were already designed, and as a result, ranking-based methods have been developed as well (e.g. Avramidis, Eleftherios. "Sentence-level ranking with quality estimation." Machine translation 27.3-4 (2013): 239-256.; Eleftherios Avramidis. 2012. Comparative Quality Estimation: Automatic Sentence-Level Ranking of Multiple Machine Translation Outputs. In Proceedings of COLING 2012, pages 115–132, Mumbai, India.)

Author response:
We thank the reviewer for bringing to our attention the related works that we missed. We note the following differences between our approach and the previous literature.
1. The previous systems rely on feature engineering and on external parser to generate some of the features. In contrast our approach does not rely on feature engineering and can work directly on the source sentence and the translation pair.
2. Our models show generalization to unseen language pairs during inference. In contrast, the previous systems only work on a single language pair.
3. Our system does not rely on costly human-annotated quality scores of machine translations and achieves superior performance with weakly supervised data. The previous systems mentioned by the reviewer were trained on human annotations of quality judgments.

Reviewer comment (Writing):
Can you explain what the sentence “relative ranking annotation from direct assessment with a large enough threshold has been used with as few as one annotation” (Intro) means? Where does the threshold come into play and what does it mean to have one annotation only (I assume one per input, but not only one input)?

Author response:
Direct assessment quality scores of machine translations collected from human evaluation is very noisy. It requires at least 15 annotations per machine translation to get reproducible scores. Since it is costly to get so many annotations per translation especially for low resource languages, in practice much less annotations may be available, sometimes as few as one. However, in this scenario suppose translation A gets one annotation of direct Assessment score 80 and translation B gets one annotation of Direct Assessment score 30, the difference in score being 50, we can still conclude that translation A is better than translation B. In previous literature a threshold of 25 score difference has been used to make similar conclusions.

Reviewer comment (Experiments):
Is there any train/test overlap of the training data of mT5 and the benchmarks’ test data?

Author response:
mT5 was pretrained on the mC4 corpus. The latest data in mC4 corpus is from Common Crawl dump 2020-34 which includes web scrapes upto August 15th 2020. The WMT20 dataset was released on August 23rd 2020. Other benchmark datasets we used were released much later. Thus we are confident that the mT5 models have not seen the benchmark test data.

Reviewer comment (Experiments):
What if references were removed from the fine-tuning sets in Stages 2 and 3? This would be a useful ablation to make the models truly free from references.

Author response:
The references are required in stage 2 and 3 to generate the synthetic data. If we ablate the references completely from our system then we are left with only stage 1. The ablation results in our paper and in the table later in this response shows that without stage 2 and 3 the performance of the system is very poor. Thus access to the reference translations is required for training of our system. *Our focus in this work is two-fold, i) to remove the requirement of having references during inference and ii) to remove the reliance on costly human annotations of translation quality scores for training machine translation evaluation models.* Collecting these annotations are much costlier than collecting reference translations which can even be performed automatically by bi-text mining. Thus we believe reliance on reference translations during training is not a drawback of our system.

Reviewer comment (Evaluation):
Which of the evaluation differences are significant?

Author response:
We performed the PERM-BOTH test following previous literature.[1] The results show that our Comparator-Large and Comparator-XXL model significantly outperforms all baselines on the DA20 benchmark dataset with a p-value < 0.05. On the MQM20 and MQM22 benchmark dataset our Comparator-XXL model significantly outperforms all baselines on en-de and zh-en language pairs with p-value <0.5. Even though our system does not significantly outperform other baselines in other settings, our system does not use any supervised data during training  in contrast to the baselines.

Reviewer comment (Experiments):
What if Stages 1 & 2 are dropped and only Stage 3 is performed? This would further illustrate the importance of that stage and make it clear for future use where most time investment should go.

Author response:
We have performed further ablation experiments. We show the results in the following table. The results of our ablation show the performance of the system drops by one point to 35.5 when we remove stage 1 and 2 and just perform stage 3.

| All Stages | w/o 1 | w/o 2 | w/o 3 | w/o 3.1 | w/o 3.2 | w/o 1,2 | w/o 2,3 |
|------------|-------|-------|-------|---------|---------|---------|---------|
| 36.5       | 35.9  | 35.6  | 28.9  | 28.5    | 35.9    | 35.5    | 9.9     |

Reviewer comment (Evaluation):
While pairwise evaluations of systems are useful, a more practical utility would be to rank multiple models. Score-based systems enable that easily. With pairwise ranking-based systems, multiple comparisons have to be run. Every time a new system has to be ranked, it has to be compared with multiple existing systems.

Author response:
In practice we do not have that many systems to compare. Usually it is less than a dozen. In this scenario, our system does not add much overhead. Given the simplicity of our system, not relying on human annotations during training, not requiring access to references and generalization to unseen languages, it is an attractive alternative to existing score-based approach.  
We can provide our models with more than two translations. On the pooling layer we can construct the pooled embedding for all pairs simultaneously. This can be done in parallel with full matrix operations. These pooled embeddings can be sent to classifier head in parallel to get rankings for all the translations. We are currently pursuing this approach to create a more efficient system.

Reviewer comment (Theory):
Equation 11 should be y=0?

Author response:
The reviewer is correct. We will fix the issue in our revised paper.

Reviewer comment (Experiments):
3 languages from the WMT-20 Metrics tasks have not been included in the evaluation. It would be good to include those results as well for getting a complete view of the WMT-20 Metrics task.

Author response:
We report the results for the three languages of DA20 in the following table.

| Model            | km-en | ps-en | iu-en | en-iu |
| ---------------- | ----- | ----- | ----- | ----- |
| Openkiwi-XLMR    | 24.4  | 10.6  | 5.6   | 6.0   |
| Comet            | 14.9  | 9.2   | 3.2   | \-6.3 |
| T5Score-XL       | 22.5  | 10.6  | 2.6   | \-2.6 |
| Comparator-Base  | 30.3  | 19.0  | 1.6   | 20.3  |
| Comparator-Large | 33.4  | 20.1  | 5.2   | 14.7  |
| Comparator-XXL   | 33.1  | 19.7  | 6.6   | 18.0  |

Our systems strongly beat the baseline systems on these languages even though our systems were not trained on human annotations. We unintentionally filtered out these languages in our evaluation. Additionally, for the iu-en and en-iu language pairs we noticed that we filtered out these language pairs also from our training. The strong performance on these two pairs also show the generalizability of our system to unseen language pairs during training.

Reviewer comment (Experiments):
**Experimental Settings**: Upon meticulous examination, I observe that the experimental setup deviates from the conventional practices of evaluating numerous systems simultaneously. The study opts to assess a custom set of 'Better-worse judgments' between pairs of system outputs. This focus narrows the scope of the evaluation and raises concerns about the validity of the correlation results. The paper's method shows a strong correlation with these judgments but fails to conclusively demonstrate the superiority of one system over another. The more critical challenge lies in integrating these pairwise comparisons into a comprehensive evaluation framework that can handle multiple systems. The current approach's limitations in addressing this challenge may undermine its utility and applicability.

Author response:
1. A straightforward approach to applying our system to perform system-level evaluation is to compare all machine translation pairs of pairs of MT systems. Since our model predicts a probability of which translation is better we can aggregate these probabilities across all the MT pairs to give a probability score for which MT system is better for each pair of. Then for each MT system we can average these probability scores to get a score for the system. These scores indicate how likely a system is to beat another system. For example, suppose we have three systems A, B and C under consideration. We apply our system to each pair and get the following scores. Each cell indicates the probability that the system on the row beats the system on the column. Thus, system A beats system B with a probability of 0.7. We take the average across rows to get the score for each system. Based on the scores we can conclude the ordering C>A>B.

|   | A   | B   | C   | Avg  |
| - | --- | --- | --- | ---- |
| A |     | 0.7 | 0.3 | 0.5  |
| B | 0.3 |     | 0.4 | 0.35 |
| C | 0.7 | 0.6 |     | 0.65 |

We apply this approach to system-level evaluation on the DA20 dataset and achieve competitive results with baseline models. In the following table we show the results. The reported values are Pearson correlation with the system level gold human evaluation scores.


| Model            | en-cs | en-de | en-ja | en-pl | en-ru | en-ta | en-zh  | Avg EN-X |
| ---------------- | ----- | ----- | ----- | ----- | ----- | ----- | ------ | -------- |
| OpenKiwi-XLMR    | 0.972 | 0.968 | 0.992 | 0.957 | 0.875 | 0.91  | \-0.01 | 0.809    |
| COMET-QE         | 0.989 | 0.903 | 0.953 | 0.969 | 0.807 | 0.887 | 0.375  | 0.840    |
| T5-Score-XL-SUP         | 0.985 | 0.963 | 0.964 | 0.968 | 0.887 | 0.952 | 0.960  | 0.954    |
| Comparator-Base  | 0.99  | 0.94  | 0.98  | 0.97  | 0.63  | 0.93  | 0.47   | 0.844    |
| Comparator-Large | 0.99  | 0.95  | 0.99  | 0.98  | 0.76  | 0.94  | 0.55   | 0.880    |
| Comparator-XXL   | 0.99  | 0.95  | 0.99  | 0.98  | 0.75  | 0.94  | 0.57   | 0.881    |

| Model            | cs-en | de-en | ja-en | pl-en | ru-en | ta-en | zh-en | Avg X-EN | Avg All     |
| ---------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | -------- | ----------- |
| OpenKiwi-XLMR    | 0.76  | 0.995 | 0.931 | 0.442 | 0.859 | 0.792 | 0.905 | 0.915    | 0.862       |
| COMET-QE         | 0.755 | 0.939 | 0.892 | 0.447 | 0.883 | 0.795 | 0.847 | 0.915    | 0.878       |
| T5-Score-XL-SUP         | 0.73 | 0.995 | 0.959 | 0.513 | 0.94 | 0.92 | 0.879  | 0.915 | 0.935   |
| Comparator-Base  | 0.8   | 1     | 0.94  | 0.52  | 0.91  | 0.92  | 0.88  | 0.853    | 0.849
| Comparator-Large | 0.8   | 0.99  | 0.94  | 0.6   | 0.92  | 0.93  | 0.89  | 0.867    | 0.874       |
| Comparator-XXL   | 0.8   | 0.99  | 0.94  | 0.58  | 0.9   | 0.92  | 0.88  | 0.859    | 0.870       |

Reviewer comment (Evaluation):
**Contextualizing with Recent Advances**: Recent developments in MT evaluation metrics, such as xCOMET (https://arxiv.org/abs/2310.10482) and SLIDE (https://arxiv.org/pdf/2309.08832.pdf), emphasize the importance of error span evaluations. While it is not mandatory to compare against the latest publications, incorporating insights from these advancements could significantly enhance the robustness and relevance of the proposed method.

Author response:
3. We note that the xCOMET paper was released after we submitted our paper to ICLR and the SLIDE paper was released only 12 days before our submission. We agree that fine-grained error span evaluations are an important direction for MT evaluation systems. We plan to explore this direction in our future work.

Reviewer comment (Evaluation):
When scaling up the evaluation to multiple systems, how might one resolve apparent ranking contradictions, such as the scenario where System A outperforms System B, System B outperforms System C, but System C outperforms System A?

Author response:
2. We have shown above how to get system level scores with our systems by averaging across the row of pairwise comparison probabilities. With the system-level scores, there are no contradictions. However, there is a chance of contradiction among the better-worse judgments of systems if we do not provide this system level score and rely on each pairwise comparison probabilities to make decisions. To that end, we have checked if there is any inconsistency among the pairwise system ranking generated by our system. Among the 14 language pairs on the DA20 datasets we have only found inconsistency on 7 of them. On the following table we show the percentage of triples that show inconsistency. We note the inconsistency level is less than one percent for all the 7 language pairs.

| zh-en | en-de | ru-en | ta-en | en-zh | pl-en | en-pl | Avg   |
| ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
| 0.54% | 0.14% | 0.91% | 0.27% | 0.91% | 0.41% | 0.27% | 0.49% |