Incorporate Semantic Structures into Machine Translation Evaluation via UCCA (2020)

Abstract: Copying mechanism has been commonly used in neural paraphrasing networks and other text generation tasks, in which some important words in the input sequence are preserved in the output sequence. Similarly, in machine translation, we notice that there are certain words or phrases appearing in all good translations of one source text, and these words tend to convey important semantic information. Therefore, in this work, we define words carrying important semantic meanings in sentences as semantic core words. Moreover, we propose an MT evaluation approach named Semantically Weighted Sentence Similarity (SWSS). It leverages the power of UCCA to identify semantic core words, and then calculates sentence similarity scores on the overlap of semantic core words. Experimental results show that SWSS can consistently improve the performance of popular MT evaluation metrics which are based on lexical similarity.

## introduction machine translation evaluation (mte) is to evaluate the quality of sentences produced by machine translation (mt) systems. most automatic mt evaluation metrics compare the candidate sentences from mt systems with reference sentences from human translation to produce a similarity score, in contrast some other reference-less metrics directly compare candidate sentences and source sentences. according to the observation of well-translated sentences, we find out that there are certain words or phrases appearing in all good translations of one source text. this phenomenon is consistent with the intuition of copying mechanism @xcite , which has been widely used in lots of text generation tasks. in the field of mt evaluation, meteor++ @xcite firstly proposes the concept of copy knowledge to define the words with word a word b ... word e word f ... ucca graph semantic core words within all words candidate reference word match f1 score output score penalties extract extract word a word c ... word d word e ... copy property, and it further incorporates the copy knowledge into meteor @xcite to improve its performance. specifically, it attempts to find copy words of references and candidate sentences, and uses the overlap of these words to modify the calculation of precision and recall of meteor. however, meteor++ uses named entities as an alternative to copy knowledge in its experiments, resulting in a limited range of selected copy words and a slight improvement. in this work, we argue that words undertaking important semantic meanings should be exactly expressed during the translation procedure, which we define as semantic core words. this concept is much more general and closer to linguistic intuition compared to the copy knowledge used in meteor++. in order to apply semantic core words in the process of mt evaluation, we design a mechanism named semantically weighted sentence similarity (swss) illustrated in figure 1 . firstly, swss extracts semantic core words according to the annotated semantic labels in universal conceptual cognitive annotation (ucca) @xcite , a multi-layered semantic representation. ucca is an appealing candidate for this mechanism as it includes a lot of fundamental semantic phenomena, such as verbal, nominal and adjectival argument structures and their inter-relations. also, semantic units in ucca are anchored in the text, which simplifies the aligning procedure a lot. with the assumption that all high-quality translations should have the same semantic core words, swss then calculates precision and recall based on the overlap of semantic core words between sentence pairs and their corresponding f1 scores. finally, we modify the f1 score according to the differences of two ucca representations. for example, scenes are involved in the penalties, which are essential nodes in ucca indicating actions and states of the sentences. our experimental results show that swss can be combined with other popular mt evaluation metrics to improve their performance significantly. ## conclusion in this paper, we propose semantically weighted sentence similarity (swss), which leverages the power of ucca to identify semantic core words, and then calculates a similarity score for machine translation evaluation. inspired by copying mechanism used in sequence generation tasks, we argue that semantic core words, which carry important meaning in the sentence, should exist in all good translations. additionally, swss also uses penalties based on the differences between ucca structures and sentence lengths, including the concept of scene in ucca, in order to make the output scores more accurate. experimental results show that swss can produce a higher correlation in mt evaluation when combined with lexical mt evaluation metrics such as bleu and meteor.