Reference-Free Word- and Sentence-Level Translation Evaluation with Token-Matching Metrics (2021)

Abstract: Many modern machine translation evaluation metrics like BERTScore, BLEURT, COMET, MonoTransquest or XMoverScore are based on black-box language models. Hence, it is difficult to explain why these metrics return certain scores. This year’s Eval4NLP shared task tackles this challenge by searching for methods that can extract feature importance scores that correlate well with human word-level error annotations. In this paper we show that unsupervised metrics that are based on tokenmatching can intrinsically provide such scores. The submitted system interprets the similarities of the contextualized word-embeddings that are used to compute (X)BERTScore as word-level importance scores.

## introduction in recent years, machine translation evaluation metrics constantly improved in their correlation with human judgements (e.g. @xcite . however, this improvement comes at a loss of understandability. early metrics such as bleu @xcite and meteor @xcite follow a clearly defined algorithm without learnable weights. therefore, these metrics are interpretable by design and could even be computed per hand. newer metrics such bertscore @xcite , bleurt @xcite , comet @xcite , monotransquest @xcite , moverscore @xcite or xmoverscore @xcite instead leverage transformer @xcite based language models. as these base their predictions on thousands of learned parameters, they are too complex to understand without employing further techniques. such techniques that aim to support the understanding of black-box models are the scope of xai (explainable artificial intelligence) (e.g. @xcite . this year's eval4nlp shared task @xcite considers to what extend xai techniques extract feature importance scores from metrics that correlate with word-level error annotations. some embedding based metrics, such as moverscore, xmoverscore and bertscore can be categorized as unsupervised matching @xcite . these metrics are unsupervised, as they are not fine-tuned on human annotated translation scores. and they perform matching, as the sentence-level score is calculated based on how well each token in one sentence matches to tokens in the other sentence. this work evaluates the usage of the tokenlevel matches of bertscore and xmoverscore as feature-importance explanation of the sentencelevel score. it was conducted as part of a master thesis by leiter (2021). ## related work this system paper is related to work in the fields of machine translation evaluation metrics and explainable artificial intelligence. ## feature importance from token-matching in this section we describe the extraction of wordlevel importance scores from xbertscore and xmoverscore. in specific, we consider that words that are well aligned between source and translation are important for the sentence-level score and are likely to be correct translations. if a word does not align well, it is likely to be an error. hence, the maximal similarity (or minimal dissimilarity) of each word between source and translation can be interpreted as word-level (importance) score. we choose @xmath0, ..., x n ) to represent a source sentence and @xmath1, ..., y m ) to represent a translation where x i and y j refer to arbitrary token embeddings in x and y. ## experiment setup we calculate word-and sentence-level scores for the dev sets foot_1 of the eval4nlp shared task @xcite , which are a subset of the mlqe-pe corpus by @xcite . the organizers provide 1000 samples for the ro-en (romanian-english) and et-en (estonian-english) language pairs each. for every sample they provide a source sentence, a translation, a sentence-level ground truth score and word-level ground truth labels for source and translation. on the word-level they label a word with 1 if it is erroneous and 0 if it is correct. @xcite show that the usage of language models fine-tuned for natural language inference (nli) improves the results of moverscore. therefore, we evaluate models fine-tuned for nli for xbertscore and xmoverscore. the results of the following configurations are reported: the result of (x)bertscore by @xcite depends on the choice of the layer to extract embeddings from. for the models already included in their library foot_5 , we use the layers they tested perform best in a reference-based setting. for xlmr-nli1 we choose layer 16 and for xmlr-nli2 we choose layer 17, which we determined to perform best on a small subset of et-en data from the mlqe-pe corpus. appendix a lists hashes produced by the bertscore library that summarize the configurations. for xmoverscore(xlmr ensemble )-keep we choose the same layers. the word-level scores are evaluated with area under the curve (auc), recall at top k (rtopk) and average precision (ap) using the implementation by the organizers of the eval4nlp shared task foot_6 . ## results in the test-phase of the shared task we submitted xbertscore(xlmr ensemble ), which achieved its highest rank for the zero-shot language pair ru-de (russian-german) and its lowest rank for de-zh (german-chinese). for the latter one, the sentence scores even had a negative correlation. the cause of this remains to be investigated in the future. ## conclusion in this paper we have evaluated xbertscore and xmoverscore for word-level error annotations in a reference-free setup. the best reported configurations are based on multiple xlmr models. for future work it might be interesting to apply xlmr models that are remapped with novel cross-lingual alignment techniques. also, it could be considered to incorporate the token-probabilities of the targetside language model of xms into the word-level scores.