K o BE : Knowledge-Based Machine Translation Evaluation (2020)

Abstract: We propose a simple and effective method for machine translation evaluation which does not require reference translations. Our approach is based on (1) grounding the entity mentions found in each source sentence and candidate translation against a large-scale multilingual knowledge base, and (2) measuring the recall of the grounded entities found in the candidate vs. those found in the source. Our approach achieves the highest correlation with human judgements on 9 out of the 18 language pairs from the WMT19 benchmark for evaluation without references, which is the largest number of wins for a single evaluation method on this task. On 4 language pairs, we also achieve higher correlation with human judgements than BLEU. To foster further research, we release a dataset containing 1.8 million grounded entity mentions across 18 language pairs from the WMT19 metrics track data.

## introduction reliable and accessible evaluation is an important catalyst for progress in machine translation (mt) and other natural language processing tasks. while human evaluation is still considered the goldstandard when done properly @xcite , automatic evaluation is a cheaper alternative that allows for rapid development cycles. today's prominent automatic evaluation methods like bleu @xcite or meteor @xcite rely on n-gram matching with reference translations. while these methods are widely adopted, they have notable deficiencies: to alleviate these issues, we propose knowledge-based evaluation (kobe), an evaluation method based on a large-scale multilingual knowledge base (kb). in our approach, we first ground each source sentence and candidate translation against the kb using entity linking @xcite @xcite @xcite . we then measure the recall for entities found in the candidate vs. entities found in the source for all sentence pairs in the test set. matching entities are ones linked to the same kb entry in both the source and the candidate. figure 1 shows our entity matches for two candidate translations vs. the source, where different surface forms that convey the same meaning are properly matched. our approach does not require reference translations, as it is based on linking entity mentions to the kb. this also makes it language-pair agnostic as long as the kb and entity linking systems cover the languages of interest. since different words that share the same meaning should be resolved to the same entry in the kb, our method will not penalize different valid translations of the same entity. as our method measures the recall of the entities found in the source sentence, it is useful as an absolute quality signal and not just as a relative one. finally, we can perform fine-grained error analysis using entity metadata to better understand where a system fails or succeeds in terms of entity types and domains. to test our approach, we experiment with the "quality estimation as a metric" benchmark (also named "metrics without references") from the wmt19 shared task on quality estimation @xcite . kobe performs better than the other participating metrics on 9 language pairs, and obtains better correlation with human raters than bleu on 4 language pairs, even though bleu uses reference translations and kobe is referenceagnostic. this demonstrates that kobe is a promising step towards mt evaluation without reference translations. to make our findings reproducible and useful for future work, we release the annotations we used together with scripts to reproduce our results. these entity linking annotations span over 425k sentences in 18 language pairs from 262 different mt systems, and contain 1.8 million entity mentions of 28k distinct entities. 1 to summarize, this work includes the following contributions: we then count the matches for each sentence pair; matches are all candidate entities that are linked to the same record in the kb as source entities. entities mentioned several times are counted as individual matches, and matches are clipped by the number of appearances of each entity in the source. as a pre-processing step, we ignore entity mentions in the candidate that are not in the target language using an in-house language identification tool, which we found to improve results in early experiments. we then compute recall by summing the number of matching entities across all sentence pairs and dividing by the the number of entities mentioned in all source sentences: our decision to ignore candidate entities that are not in the correct language came from an observation that for some low-resource language pairs, mt systems fail to translate the input and instead copy most of its content to the output -see @xcite for a similar observation. as our entity linking system is language agnostic, it was detecting the copied entities, which resulted in false matches. we found precision to have weaker correlation on most language pairs, as it rewards systems producing a lower number of entities -systems that usually produced lower quality translations. recall is more stable as the number of entities in the source is constant for all evaluated systems, and only the match count is changing. since recall may give inflated scores when over-producing entities, we introduce an entity count penalty (ecp), inspired by bleu's brevity penalty. ecp penalizes ## experimental setup the wmt conference holds a quality estimation track (qe) that aims to predict the quality of mt systems given the source sentences and candidate translations (without reference translations). while this was usually done at the word or sentence level, one of the novelties in wmt19 was introducing a new task for using qe as a metric at the corpus level, testing the generalization ability of qe approaches in a massive multi-system scenario (fonseca et al., 2019). to test our approach, we used the same setting as in this shared task. for every language pair of the 18 evaluated pairs, we use kobe to score the mt systems participating in same years news translation task. we then measure the pearson correlation of our scores for each system with its human direct-assessment (da) scores. to ensure a fair comparison, we recompute the correlations for the other participating metrics and confirm that we reproduce the reported scores. ## results we compare kobe with all participating metrics in the shared task. we refer the reader to @xcite for more details about the different metrics. we also compare our results with bleu to have a benchmark for a reference-based metric. in ## discussion and analysis summarizing the above findings, our method obtains the best results on 9 out of 18 language pairs, which makes it the method with the largest number of wins on the wmt19 metrics-without-references benchmark. this shows that knowledge-based evaluation is a promising path towards mt evaluation without references. in comparison, the next best method is yisi-2 @xcite which is based on token-level cosine-similarity using context-aware token representations from multilingual bert @xcite . we believe that combining our knowledge-based approach with such methods may result in even better correlation with human judgements, but leave this for future work. as our metric is based on the recall of entities in the target with respect to the source, it is important that the entities will be properly detected in the target. a failure to detect an entity in the source will just lead kobe to use less entities, while a failure to detect an entity in the target will lead kobe to penalize an entity that is actually present. our entity linking pipelines work best in english, which results in much higher correlations with human judgements when english is the target language ( our reproduction of the correlation results raises an issue with the current evaluation methodology in the shared task. in the published results @xcite , in order to support both lower-is-better metrics (e.g. @xcite ) and higheris-better metrics (e.g. bleu), the absolute values of the pearson correlations are reported. however, when looking in a possible drawback of our approach is that it only relies on entities, which do not fully cover the sentence semantics. however, in the quality estimation setting, we only have access to the source and candidate translation, which are in different languages. as different languages use different syntactic structures and vocabulary, it is hard to employ other structural cues -for example, the order of the entities may be different due to the grammatical differences between the languages. the strong correlation between our metric and human judgements shows that knowledge-based comparison is a strong indicator of translation quality in this challenging setting. this is in line with the results of @xcite who showed that bleu with extensively paraphrased references correlates better with human judgements than bleu with vanilla references -our method is "paraphrasing" or "stripping de-en fi-en gu-en kk-en lt-en ru-en zh-en mean bleu 0.849 0.982 0.834 0.946 0.961 0.879 0.899 0.907 kobe 0.906 0.961 0.85 0.961 0.901 0.954 0.947 0.926 public_companies authors ingredients leadership_roles categories 0 50 100 150 200 250 300 entity counts zh-en source entities (zh) online-x.0 matched entities (en) baidu-system.6940 matched entities (en) figure 3: statistics for entity categories for different chinese-to-english mt systems and the source side. ## related work quality estimation for mt has been studied extensively in recent years -see @xcite for a thorough overview. most work has been on the sentence or word level, using supervised approaches e.g. open-kiwi @xcite . using semantic knowledge for mt evaluation was proposed in different approaches: meteor @xcite used paraphrase tables for referencebased evaluation; yisi @xcite and meant @xcite used semantic role labeling (srl) annotations; @xcite used the ucca semantic annotations @xcite for human evaluation of mt; @xcite proposed a name-aware bleu score giving more weight to named entities. @xcite conducted a comparative evaluation of named entity recognition (ner) from mt outputs, concluding that the success rate of ner does not strongly correlate with human or automatic evaluation scores. we show contradicting results, which may stem from the better ner and mt systems available today, and from the entity linking step we add. to the best of our knowledge, our work is the first to introduce a reference-less mt evaluation method based purely on entity linking against a multilingual knowledge-base. ## conclusions and future work we proposed kobe, a method for reference-less machine translation evaluation using entity linking to a multilingual knowledge base. we demonstrated the applicability of our method by achieving strong results on the wmt19 benchmark for reference-less evaluation across 9 language pairs, where in 4 cases it also outperforms the referencebased bleu. our method is simple, interpretable and produces scores closer to human judgements on an absolute scale, while enabling more finegrained analysis which can be useful to find weak spots in the evaluated model. in future work, we would like to combine knowledge-based signals with unsupervised approaches like yisi @xcite and xmoverscore @xcite ) that use contextualized representations from cross-lingual lms like multilingual bert @xcite . as our method does not require reference translations, we would like to explore scaling it to use much larger or domain specific monolingual datasets. our knowledge-based approach can also be applied to other text generation tasks like summarization or text simplification where bleu was shown to be problematic @xcite . finally, performing outlier-aware meta-evaluation which was recently shown to be important in such settings @xcite ) could be beneficial.