Regressive Ensemble for Machine Translation Quality Evaluation (2021)

Abstract: This work introduces a simple regressive ensemble for evaluating machine translation quality based on a set of novel and established metrics. We evaluate the ensemble using a correlation to expert-based MQM scores of the WMT 2021 Metrics workshop. In both monolingual and zero-shot cross-lingual settings, we show a significant performance improvement over single metrics. In the cross-lingual settings, we also demonstrate that an ensemble approach is well-applicable to unseen languages. Furthermore, we identify a strong reference-free baseline that consistently outperforms the commonly-used BLEU and METEOR measures and significantly improves our ensemble’s performance.

## introduction automated evaluation of text generation is challenging due to many orthogonal qualitative aspects that the user expects from a text generation system. in machine translation, we can observe errors of the so-called critical category @xcite such as hallucinating @xcite , omitting parts of the input from translation, or the negation of meaning @xcite . consider an example of the last category: in this example, all bertscore @xcite , bleurt @xcite , and prism (thompson and post, 2020b) metrics rank hypothesis 2 higher than hypothesis 1. in bleurt and prism, this can be due to a known vulnerability of transformers, which rely on a lexical intersection @xcite if such a heuristic fits the prob-lem sufficiently well. trivially, just counting the negations can easily remedy this specific problem. however, such a heuristic would fail in many other cases, such as when we adjust hypothesis 1 to "it is an article of somebody else, i just edited it.". such cases motivate our ensemble approach that aims to expose both surface and deeper semantic properties of texts and subsequently learn to utilize these for the specific task of translation evaluation. even though the objective might constrain the particular metrics, data set, or systematically fail in some cases, another metric or their combination in the ensemble allows the flaw to be corrected. ## metrics for machine translation evaluation this section reviews the related work, focusing on the metrics we used in our ensemble. the standard and still widely-used surface-level metrics for the evaluation of machine translation quality are bleu @xcite , rouge @xcite , and ter @xcite . surfacelevel metrics are not able to capture the proximity of meaning in cases where one text paraphrases the other, which is an ability commonly observed in deep neural language models @xcite . one metric that addresses this flaw is meteor @xcite , which utilizes wordnet to account for synonymy, word inflection, or tokenlevel paraphrasing. evaluation of semantic text equivalence is closely related to a problem of accurate textual representations (embeddings). the traditional method that we identify as relevant for the evaluation of segment-sized texts is fasttext @xcite . fasttext learns representations of character n-grams from which it creates a unified representation of tokens by averaging. additionally, a distance of a pair of texts can be computed directly from the token-level embeddings using methods such as the soft vector space model (the soft co-sine measure, scm) (novotný, 2018), or by solving a minimum-cost flow problem (the word mover's distance, wmd) @xcite . similar matching is performed by bertscore @xcite , which uses internal token embeddings of a selected bert layer optimal for the task. although the token representations are multilingual in some models @xcite , which makes bertscore usable without references, we are not aware of prior work evaluating it as such. a possible drawback of cross-lingual alignment using a max-reference matching scheme of bertscore lies in a possibility of a significant mismatch of sub-word tokens in source and target text. in contrast, the metric that we refer to as wmd-contextual uses the same embeddings as bertscore but uses the network-flow optimization matching scheme of wmd. task-agnostic methods have recently been outperformed by methods that fine-tune a pre-trained model for a related objective: bleurt @xcite fine-tunes a bert @xcite model directly on direct assessments of submissions to wmt to predict the judgements using a linear head over contextual embeddings of a classification [cls] token. comet @xcite learns to predict direct assessments from tuples of source, reference, and translation texts with the triplet objective or the standard mse objective. some of the most recent work incorporates latent objectives and/or data sets. for instance, prism @xcite learns a languageagnostic representation from multilingual paraphrasing in 39 languages, thus being one of the few well-performing reference-free metrics. the orthogonality of its training objective might lower its correlation to other methods that use contextual embeddings. ## methodology our methodology aims to answer the following major question with additional supporting questions: ## results correlations to mqm judgements. the results suggest that a simple regressive ensemble can benefit from the variance of the predictors in a majority of the evaluated configurations, including the cross-lingual settings and other evaluated datasets. we observe the highest margins in correlations in the case of mqm judgements. ## discussion regressive ensemble. following the objectives that we set in section 3, we empirically confirm that an ensemble can push the quality of modeling the expert judgements in most of the configurations while performing close-to-the-best metrics on the others. additionally, we demonstrate that such ensemble is transferable to new language pairs and that its use is motivated by qualitative gains even in cross-lingual settings. at the same time, one must acknowledge the limitations that an ensemble system exposes compared to single and unsupervised metrics. an ensemble might inherit the systematic biases of each of its metrics. this problem is observable in the results of the source-based en-de pair of mqm in on the other hand, we argue that this might rarely be the case with the surface-level metrics that are mainly unsupervised. we suspect it to be unlikely with learnable metrics, too, having their output space constrained by the range of their imitated metrics. values of correlations in baseline ensemble. surprisingly, our baseline ensemble reg-base consistently outperforms other standard surface-based metrics such as bleu @xcite . this suggests possible applicability of surface-level metrics also in reference-free evaluation. we suspect that the baseline features of length based on a multilingual wordpiece tokenizer @xcite might reflect on the missing or inappropriately added segments more strictly than other surface-level metrics. at the same time, these errors are usually highly weighted in the overall score. of the other weak surface metrics into the baseline ensemble to alleviate some of its apparent flaws. ## conclusion this work evaluates the potential of ensembling multiple diverse metrics (regemt ) for an evaluation of machine translation quality and offers a new simple baseline metric reg-base that achieves better results than bleu and meteor by using just the source and reference lengths. we measure significant gains in spearman's correlation to mqm with regemt compared to standalone metrics and we demonstrate that even simple linear estimators can benefit from the expressivity that the methods of all levels of representation provide. additionally, as we demonstrate, the ensemble based on metrics supporting multilingualism can push the quality further even on unseen language pairs. we recognize the inherent limitations of the regressive ensemble, which is inevitably slower, resource-heavier, and prone to inherit latent inductive biases of underlying metrics or their combinations. however, regemt shows the agility of the simple ensemble approach, which is in contrast to attempts to learn the full complexity of quality estimation through a single objective and allows the quality estimator to avoid the blind spots of particular metrics. we hope that our results will motivate future work in the ensemble evaluation.