Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model (2020)

Abstract: We propose an automatic evaluation method of machine translation that uses source language sentences regarded as additional pseudo references. The proposed method evaluates a translation hypothesis in a regression model. The model takes the paired source, reference, and hypothesis sentence all together as an input. A pretrained large scale cross-lingual language model encodes the input to sentence-pair vectors, and the model predicts a human evaluation score with those vectors. Our experiments show that our proposed method using Cross-lingual Language Model (XLM) trained with a translation language modeling (TLM) objective achieves a higher correlation with human judgments than a baseline method that uses only hypothesis and reference sentences. Additionally, using source sentences in our proposed method is confirmed to improve the evaluation performance.

## introduction automatic machine translation evaluation (mte) has been studied to substitute human evaluation in machine translation development because it is low-cost, handy, and stable to use. popular automatic mte metrics such as bleu @xcite calculate the evaluation score based on a surface-level similarity of a paired 1-to-1 reference and translated hypothesis sentences. bleu particularly evaluates the sentence similarity with the ngram word matching rate between a reference and hypothesis. however, the evaluation score drops when a reference and hypothesis are dissimilar in the surface even if they share the same meaning. to counter this problem, meteor @xcite is proposed to mitigate the word matching of synonyms with a synonym dictionary. yet still, with mitigation of word matching, surfacelevel similarity cannot fully compensate for seman-tics, thus word representation instead of word symbols is used in word mover's distance @xcite and bleu2vec @xcite . besides, sentence representation is known to be an efficient feature instead of word representation because sentence vectors can represent more global meanings. ruse @xcite and bert @xcite based mte, bert regressor @xcite , utilized sentence representation and performed well on wmt17 metric shared task @xcite . the metrics mentioned above compare a hypothesis translation to a reference. however, a reference translation represents only one possible translation and those mte metrics are unlikely to correctly evaluate all candidates that share the same meanings of the reference or have fatally different meanings due to a few translation errors. this problem can be mitigated by the use of multiple reference translations as argued by @xcite and @xcite , but preparing such multiple references is costly. hereby, we propose a method to incorporate source sentence into mte as another pseudo reference, since the source and reference sentences should be semantically equivalent. the proposed method uses cross-lingual language model (xlm) @xcite to handle source and target languages in a shared sentence embedding space. the proposed method with xlm trained with a translation language modeling (tlm) objective showed a higher correlation with human judgments than a baseline method using hypothesis and reference sentences. ## related work recent advances in sentence-level embedding have been used in mte. @xcite proposed an mte framework called ruse (regressor using sentence embeddings), which uses sentencelevel embeddings obtained by a large-scale pretrained model like infersent @xcite , quick thought @xcite , and universal sentence encoder @xcite . its regressor takes sentence vectors for a reference and translation hypothesis as inputs and returns a score, which is trained to correlate well with human evaluation @xcite . ruse achieved the best correlation score with human judgments in the wmt-2017 metrics shared task @xcite . bert regressor @xcite ) is a simple mte metric based on bert @xcite encoder. it is composed of bert encoder and a multi-layer perceptron (mlp) regressor attached to the last layer of bert. this bert encoder is a 12 layers bi-directional language model, referring to bert base (uncased) foot_0 , trained with masked language model (mlm) and next sentence prediction (nsp). bert regressor surpassed ruse on the wmt-2017 data. ## proposed method: automatic evaluation using xlm we propose an mte method using source language sentences as additional pseudo references. we use cross-lingual language models called xlm @xcite to encode both source and target language sentences into an embedding vector. xlm has three additional techniques to bert: language independent subword based on byte pair encoding @xcite , a language embedding layer, and a translation language modeling (tlm) objective that predicts masked words from surrounding words or a paired translation. the brief architecture of xlm is shown in figure 1 . @xcite reported that xlm trained with tlm objective obtains better performance than multilingual bert @xcite on the xnli cross-lingual classification task @xcite . the proposed method has two variants for the use of source language sentences, as illustrated in figure 2 . the first one called hyp+src/hyp+ref uses two sentence-pair vectors for hypothesissource and hypothesis-reference, encoded by a cross-lingual language model independently. these sentence-pair vectors are given to an mlp-based regression model to predict the human evaluation scores. this can be regarded as an ensemble model using a monolingual vector based on the reference and a cross-lingual vector based on the source sentence. the other one called hyp+src+ref takes a concatenation of hypotheses, source, and reference sentences as an input to a cross-lingual language model to obtain a sentence-pair vector. this sentence-pair vector is expected to be directly learned to represent the quality of the translation hypothesis given two correct sentences aligned aside. ## experiments we conducted experiments to evaluate the performance of the proposed method in mte by comparing with some existing methods. ## analysis training data size we conducted another experiment to see the effect of the training corpus size using randomly halved {de, ru, tr, zh}-en small corpus. from the results in ## conclusion in this paper, we proposed an mte framework that utilizes source sentences using xlm. we show that the proposed method with tlm-trained xlm showed a higher correlation with human judgments than the baseline method in the small corpus condition and stabilize the evaluation performance regardless of the quality of translation sentences by using additional source sentences. we also investigated why our proposed method worked poorly in the other conditions and found the importance of tlm training. in future work, we will work around the problem of evaluation errors in the low da range.