Cometoid: Distilling Strong Reference-based Machine Translation Metrics into E ven Stronger Quality Estimation Metrics (2023)

Abstract: This paper describes our submissions to the 2023 Conference on Machine Translation (WMT-23) Metrics shared task. Knowledge distillation is commonly used to create smaller student models that mimic larger teacher model while reducing the model size and hence inference cost in production. In this work, we apply knowledge distillation to machine translation evaluation metrics and distill existing reference-based teacher metrics into reference-free (quality estimation; QE) student metrics. We mainly focus on students of Unbabel’s COMET22 reference-based metric. When evaluating on the official WMT-22 Metrics evaluation task, our distilled Cometoid QE metrics outperform all other QE metrics on that set while matching or out-performing the reference-based teacher metric. Our metrics never see the human ground-truth scores directly – only the teacher metric was trained on human scores by its original creators. We also distill ChrF sentence-level scores into a neural QE metric and find that our reference-free (and fully human-score-free) student metric ChrFoid outperforms its teacher metric by over 7% pairwise accuracy on the same WMT-22 task, rivaling other existing QE metrics.

## introduction the conference on machine translation (wmt) organizes an annual shared task for meta-evaluation of machine translation (mt) evaluation metrics @xcite , where numerous mt evaluation metrics are proposed and revised each year. the mt metrics are broadly categorized as: (i) reference-based metrics, which score mt hypothesis against one or more reference translations from humans, and (ii) reference-free metrics, which do not require references and instead score hypothesis directly against the source sentence. referencefree metrics, also known as quality estimation (qe) metrics, are an attractive choice in scenarios where reference translations are either unavailable or unreliable. however, currently, qe metrics lag behind the reference-based metrics by a considerable margin according to metrics meta-evaluation results @xcite . knowledge distillation (kd) @xcite is commonly used to create smaller student models that mimic larger teacher models @xcite which reduces computational cost when deploying models in production @xcite . other use cases of kd in mt include distillation from auto-regressive teacher translation models to non-autoregressive students @xcite where the students "suffer" from an information bottleneck (here: no access to their own previous output in a time sequence) which impedes their performance when trained on original data. the simplified and probably smoothed output distribution of the teacher is easier to "digest" and often results in improved performance for the student. in this work, we treat existing reference-based metrics as teachers and by applying knowledge distillation, we create reference-free student metrics that completely eliminate the need for references in evaluation. this is achieved by introducing a hard information bottleneck: just dropping the reference during training while keeping the original reference-based teacher score. ## results and analysis we report system level pairwise accuracy obtained using mt-metrics-eval, foot_5 the official metaevaluation pipeline used in wmt22 metrics task. ## related work reference-free ( qe ) metrics: comet20-qe @xcite and cometkiwi22 @xcite are popular qe metrics. unite @xcite supports inference in reference-free mode, in addition to reference-based mode. these metrics rely on scores from human evaluators during training and are limited by availability of high quality human ratings. our metrics are trained with scores from teacher models and are trained on larger training data than what has been rated by human evaluators. distillation: @xcite and @xcite apply knowledge distillation to the reference-based metrics, however, their distillation is aimed at reducing the model size for the sake of reducing computational cost during inference. our work differs from theirs, as we distill with the aim of removing the need for human references at inference time. ## conclusion we believe this work describes a perhaps simpler avenue towards more powerful qe metrics than proposed so far: build strong reference-based first, next distill into even stronger qe metrics. it further seems that performance improves with adding fully synthetic data (via adding larger amounts of inputs and automatically scored outputs). this effect seems also applicable to "dumb" metrics like chrf: we have arrived at chrfoid, a qe metric that has seen no human scores at all, and yet rivals the performance of the best previously available qe metrics. knowledge distillation combined with a strong information bottleneck (reference-based to reference-free) seems to be the key in this new approach.