Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns (2020)

Abstract: This paper describes our submission of the WMT 2020 Shared Task on Sentence Level Direct Assessment, Quality Estimation (QE). In this study, we empirically reveal the mismatching issue when directly adopting BERTScore (Zhang et al., 2020) to QE. Specifically, there exist lots of mismatching errors between source sentence and translated candidate sentence with token pairwise similarity. In response to this issue, we propose to expose explicit cross lingual patterns, e.g. word alignments and generation score, to our proposed zero-shot models. Experiments show that our proposed QE model with explicit cross-lingual patterns could alleviate the mismatching issue, thereby improving the performance. Encouragingly, our zero-shot QE method could achieve comparable performance with supervised QE method, and even outperforms the supervised counterpart on 2 out of 6 directions. We expect our work could shed light on the zero-shot QE model improvement.

## introduction translation quality estimation (qe) @xcite @xcite aims to predict the quality of translation hypothesis without golden-standard human references, setting it apart from reference-based translation metrics. existing reference-based evaluation metrics, e.g. bleu @xcite , meteor @xcite , nist @xcite , rouge @xcite , ter @xcite , are commonly used in language generation tasks including translation, summarization, and captioning but all heavily rely on the quality of given references. recently, @xcite show that reference-based automatic evaluation metrics, e.g., bleu, are not always reliable because the human translated references are translationese @xcite назва ##но число го ##тов ##ящихся к от ##пра ##в ##ке в дон ##бас ##с нов ##ран ##об цев из украины the number of re ##c ##ruit ##s from ukraine preparing to be sent to don ##bas named has been 0.709 0.661 0.659 0.642 0.645 назва ##но число го ##тов ##ящихся к от ##пра ##в ##ке в дон ##бас ##с нов ##ран ##об цев из украины the number of re ##c ##ruit ##s from ukraine preparing to be sent to don ##bas named has been 0.56 0.529 0.527 0.642 0.645 example of mismatching error, @xmath0. on the left, token "назва" is mismatched to "the" with the maximal probability (within the red rectangle) only. on the right, guided by our proposed cross-lingual patterns, "назва" is correctly matched to the token "named" with the maximal probability (within the green rectangle.) @xcite . thus, an automatic method with no access to any references, i.e., qe, is highly appreciated. in this paper, we mainly focus on sentence level qe metrics, where existing studies categorize it into two classes: 1) supervised qe with human assessment as supervision signal: a feature extractor stacked with an estimator @xcite @xcite ; 2) unsupervised qe without human assessment, which normally based on the pre-trained word embeddings, for example, yisi @xcite and bertscore @xcite . our work follows the latter, where we adopt bertscore @xcite without extra fine-tuning. in particular, we implement our approach upon the pre-trained multilingual bert @xcite and xlm @xcite . we first empirically reveal the mismatching issue when directly adopting bertscore @xcite to qe task. specifically, there exist lots of mismatching errors between source tokens and translated candidate tokens when performing greedy matching with pairwise similarity. figure 1 shows an example of the mismatching error, where the russian token "назва" is mismatched to the english token "the" due to lacking of proper guidance. to alleviate this issue, we design two explicit cross-lingual patterns to augment the bertscore as a qe metric: ## related work mt evaluation taking sentence-level evaluation as an example, reference-based metrics describe to which extend a candidate sentence is similar to a reference one @xcite . bleu @xcite , meteor @xcite , nist @xcite , rouge @xcite measure such similarity through n-gram matching, which is restricted to the exact form of sentences. ter @xcite and char-acter @xcite use edit distance at word or character level to indicate the distance between candidate and reference. different from these metrics that are restricted to the exact form of sentences, recent dominated neural model metrics learn to evaluate with human assessment as supervision signal, such as beer @xcite and ruse @xcite , or oth-ers as yisi @xcite and bertscore @xcite , evaluate with pre-trained word embedding, without using human assessment. incorporating explicit knowledge several approaches have incorporated pre-defined or learned features into neural networks. @xcite demonstrate that incorporating structured semantic information could enhance the representations. @xcite feed the encoder cell combined embeddings of linguistic features including lemmas, subword tags, etc. @xcite leverage the domain knowledge to perform data selection to improve the machine translation models. @xcite incorporate the structure patterns of sentences, i.e., syntax, into the transformer network to enhance seq2seq modeling performance. @xcite utilize the pre-defined fixed patterns to replace the attention weights and show promising results. inspired by above works, we propose to augment zero-shot qe model with crosslingual patterns. ## conclusion and future work in this work, we revealed a mismatching issue in zero-shot qe modeling. to alleviate it, we introduced two explicit cross-lingual patterns based on bertscore backbone. extensive experiments indicated that our proposed patterns, without finetuning, the qe model can be improved marginally. notably, our zero-shot qe method outperforms supervised qe model on 2 out of 6 directions, shedding light on zero-shot qe researches. in the future, we plan to explore more strategies for incorporating various auxiliary information and better in-domain fine-tuning @xcite or introduce an non-autoregressive refiner @xcite to address our revealed mismatching issue. also, it will be interesting to apply qe metrics on document-level machine translations with considering the dropped pronoun @xcite .