MetricX-23
This is not an officially supported Google product.
GitHub repository: https://github.com/google-research/metricx
This repository contains the MetricX-23 models, a family of models for automatic evaluation of translations that were proposed in the WMT'23 Metrics Shared Task submission MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task. The models were trained in T5X and then converted for use in PyTorch.
Available Models
There are 6 models available on HuggingFace that vary in the number of parameters and whether or not the model is reference-based or reference-free (also known as quality estimation, or QE):
- MetricX-23-XXL
- MetricX-23-XL
- MetricX-23-Large
- MetricX-23-QE-XXL
- MetricX-23-QE-XL
- MetricX-23-QE-Large
We recommend using the XXL model versions for the best agreement with human judgments of translation quality, the Large versions for best speed, and the XL for an intermediate use case.
Changes to the WMT'23 Submission
These models available here are most similar to the primary submission to the WMT'23 Metrics Shared Task. They are initialized with mT5 then fine-tuned on a combination of direct assessment and MQM data. However, we made some changes that make these models different from the WMT'23 submissions.
First, the models are trained to regress the actual MQM score rather than a normalized score between 0 and 1. That means the output from the MetricX-23 models is a score in the range [0, 25] where lower is better (i.e., it predicts an error score).
Second, these models were trained with a larger variety of synthetic data that makes them more robust to translation edge cases like over- and undertranslation, described in more detail in the following section.
Synthetic Data
In order for our MetricX models to learn to identify certain types of bad translations that are not sufficiently (or at all) represented in the regular training data, we created synthetic examples and mixed them in during training. The synthetic training data was generated from the DA datasets ranging from WMT15 to WMT21 (~ 43 language pairs). In most cases, the synthetic examples have the candidate translation manipulated so as to turn it into a bad translation with a specific issue commonly unrecognized by learned metrics.
The table below provides an overview of the various failure modes that we considered, including brief descriptions of how we prepared the synthetic data to address them.
Failure mode | Synthetic example description |
---|---|
Undertranslation | Candidate translation with an arbitrary sentence removed (if multi-sentence); alternatively, candidate with a certain proportion of words removed from the end. |
Overtranslation | Candidate translation duplicated (with space in between). |
Fluent but unrelated translation | Arbitrary reference of a similar length from the dataset. |
Gibberish | Text of a similar length as the reference, generated by sampling words from the reference translation vocabulary (built from all references in the data). |
Missing punctuation | Reference translation with the end punctuation removed (11 punctuation symbols considered). |
Latin instead of Chinese/Japanese or Hindi/Bengali punctuation | Candidate translation with the language-specific punctuation symbol at the end replaced with the Latin equivalent (e.g., "." instead of "。" or "।"); alternatively, the punctuation symbol is replaced with the Latin equivalent in the reference, keeping the correct one in the candidate. |
Reference-matching translation | Reference translation copied as the candidate translation (unlike the rest of the synthetic data, these examples are meant to train the metric to predict a perfect score for candidates matching the reference). |
Examples from the first 4 categories were assigned a label corresponding to the worst score on the given rating scale (e.g., 25 when mixed with MQM training data), whereas the reference-matching translation examples are assigned the best score (e.g., 0 when used with MQM data). The missing/incorrect punctuation examples were labeled with a score slightly worse than perfect.
Note that some of the synthetic datasets are only meaningful in the reference-based scenario, and we thus excluded them when training a QE variant of MetricX. These are the Latin-vs-special punctuation and the reference-matching translation examples.
Most of the synthetic training sets were created using stratified sampling across target languages, taking 500 examples per target language. One exception is the missing punctuation set, which used a stratified sample across different punctuation symbols instead.
When training MetricX, a small proportion of the synthetic examples was mixed with the regular training examples. During the first-stage fine-tuning on DA data, each synthetic training set constituted between 0.1% and 1% of all training examples, whereas in the second-stage fine-tuning on MQM data we used an even smaller proportion, around 0.05%.
As for evaluating the effect of the synthetic training data on the model's performance, the DEMETR challenge set - which we originally used to evaluate the models submitted to the WMT23 Metrics Shared Task - was not adequate anymore. We therefore created a new DEMETR-style test set based on the WMT22 DA data, with examples constructed analogically to the synthetic training examples, as described above. This test set helped us determine the right proportions of synthetic data for fine-tuning in order to make MetricX robust for the failure modes in consideration, without sacrificing the system- and segment-level correlations with human ratings.
Usage
The code for using MetricX models can be found at https://github.com/google-research/metricx. The repository contains example prediction scripts, described below.
The metricx23/predict.py
script contains an example for how to run inference
on the models.
Reference-Based
Example usage for a reference-based model:
python -m metricx23.predict \
--tokenizer google/mt5-xl \
--model_name_or_path google/metricx-23-xl-v2p0 \
--max_input_length 1024 \
--batch_size 1 \
--input_file input.jsonl \
--output_file output.jsonl
input.jsonl
is expected to have 1 serialized JSON object per line with
"reference"
and "hypothesis"
fields. The output jsonl will be parallel
to input.jsonl
but additionally contain a "prediction"
field with the predicted score.
Note that the model was trained with a maximum input length of 1024 tokens, so significantly increasing that value may lead to unpredictable behavior.
Reference-Free
Example usage for a reference-free model:
python -m metricx23.predict \
--tokenizer google/mt5-xl \
--model_name_or_path google/metricx-23-qe-xl-v2p0 \
--max_input_length 1024 \
--batch_size 1 \
--input_file input.jsonl \
--output_file output.jsonl \
--qe
input.jsonl
is expected to have 1 serialized JSON object per line with
"source"
and "hypothesis"
fields. The output jsonl will be parallel
to input.jsonl
but additionally contain a "prediction"
field with the predicted score.
Meta-Evaluation
The metricx23/evaluate.py
script contains code to calculate various correlations
between the MetricX-23 scores and MQM ratings of translation quality using the
MT Metrics Eval library.
Example usage:
python -m metricx23.evaluate \
--dataset wmt22 \
--lp en-de \
--input_file input.jsonl \
--output_file output.json
input.jsonl
is expected to have one JSON object serialized per line.
Each JSON object is expected to contain 4 fields:
"system_id"
: The name of the system that generated the translation."segment_id"
: The 0-based index of the corresponding segment in the MT Metrics Eval data."label"
: The ground-truth translation quality score (with higher is better)."prediction"
: The model predicted translation quality score (with lower is better; the script negates the scores so higher is better).
The script will calculate the 4 agreement/correlations that were used in the WMT'23 Shared Task. Below are the results for the MetricX-23 models on the WMT'22 Metrics Shared Task data:
English-German:
Model | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
---|---|---|---|---|
MetricX-23-XXL | 0.795 | 0.835 | 0.546 | 0.619 |
MetricX-23-XL | 0.756 | 0.813 | 0.540 | 0.605 |
MetricX-23-Large | 0.769 | 0.759 | 0.507 | 0.595 |
MetricX-23-QE-XXL | 0.769 | 0.830 | 0.490 | 0.606 |
MetricX-23-QE-XL | 0.718 | 0.684 | 0.421 | 0.594 |
MetricX-23-QE-Large | 0.744 | 0.671 | 0.387 | 0.579 |
English-Russian:
Model | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
---|---|---|---|---|
MetricX-23-XXL | 0.905 | 0.943 | 0.477 | 0.609 |
MetricX-23-XL | 0.876 | 0.906 | 0.498 | 0.589 |
MetricX-23-Large | 0.876 | 0.841 | 0.474 | 0.569 |
MetricX-23-QE-XXL | 0.895 | 0.940 | 0.470 | 0.602 |
MetricX-23-QE-XL | 0.848 | 0.861 | 0.415 | 0.570 |
MetricX-23-QE-Large | 0.819 | 0.778 | 0.411 | 0.551 |
Chinese-English:
Model | System-Level Accuracy | System-Level Pearson | Segment-Level Pearson | Segment-Level Pairwise Acc |
---|---|---|---|---|
MetricX-23-XXL | 0.868 | 0.919 | 0.605 | 0.551 |
MetricX-23-XL | 0.868 | 0.924 | 0.584 | 0.543 |
MetricX-23-Large | 0.857 | 0.919 | 0.555 | 0.539 |
MetricX-23-QE-XXL | 0.857 | 0.928 | 0.573 | 0.544 |
MetricX-23-QE-XL | 0.802 | 0.879 | 0.546 | 0.529 |
MetricX-23-QE-Large | 0.758 | 0.904 | 0.522 | 0.529 |
The metricx23/evaluate_wmt23.py
script re-calculates the average correlation
score that was used to rank submissions from the
WMT'23 Shared Task.
Example usage:
python -m metricx23.evaluate_wmt23 \
--en_de predictions_ende.jsonl \
--he_en predictions_heen.jsonl \
--zh_en predictions_zhen.jsonl \
--output_file output.json
Each of the 3 input files is expected to be in the same format as described above. Each file should correspond to running inference on each of the language pairs from the WMT'23 dataset.
The results for each of the models is the following:
Model | Average Correlation |
---|---|
MetricX-23-XXL | 0.812 |
MetricX-23-XL | 0.813 |
MetricX-23-Large | 0.794 |
MetricX-23-QE-XXL | 0.797 |
MetricX-23-QE-XL | 0.767 |
MetricX-23-QE-Large | 0.762 |
Citation
If you use MetricX-23 in your research, please cite the following publication:
@inproceedings{juraska-etal-2023-metricx,
title = {{MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task}},
author = "Juraska, Juraj and
Finkelstein, Mara and
Deutsch, Daniel and
Siddhant, Aditya and
Mirzazadeh, Mehdi and
Freitag, Markus",
editor = "Koehn, Philipp and
Haddow, Barry and
Kocmi, Tom and
Monz, Christof",
booktitle = "Proceedings of the Eighth Conference on Machine Translation",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.wmt-1.63",
doi = "10.18653/v1/2023.wmt-1.63",
pages = "756--767",
}
- Downloads last month
- 151,498