A Deep Analysis of the Impact of Multiword Expressions and Named Entities on C hinese- E nglish Machine Translations (2024)

Abstract: In this paper, we present a study on the impact of so-called multiword expressions (MWEs) and multiword named entities (NEs) on the performance of Chinese-English machine translation (MT) systems. Built on an extended version of the data from the WMT22 Metrics Shared Task (with extra labels of 9 types of Chinese MWEs, and 19 types of Chinese multiword NEs) which includes scores and error annotations provided by human experts, we make further extraction of MWE- and NE-related translation errors. By investigating the human evaluation scores and the error rates on each category of MWEs and NEs, we find that: 1) MT systems tend to perform significantly worse on Chinese sentences with most kinds of MWEs and NEs; 2) MWEs and NEs which make up of about twenty percent of tokens, i.e. characters in Chinese, result in one-third of translation errors; 3) for 13 categories of MWEs and NEs, the error rates exceed 50% with the highest to be 84.8%. Based on the results, we emphasize that MWEs and NEs are still a bottleneck issue for MT and special attention to MWEs and NEs should be paid to further improving the performance of MT systems.

## introduction evaluating machine translation (mt) systems on various fine-grained linguistic phenomena has become a trending practice @xcite . multiword expressions (mwes), making up approximately half of the lexicon @xcite , are shown to be an intractable problem to various tasks in the realm of natural language processing across different languages due to their idiosyncrasies in syntax and/or semantics @xcite @xcite . such an idiosyncratic nature of mwes is even crucial for mt since the translation systems rely on the complete understanding of the mwes as whole lin-guistic units to generate accurate translations in target languages. although there has been a lot of work indicating that mt systems still suffer from mwes @xcite and adding mwe-specialized components in current systems to deal with them will increase the overall performance @xcite @xcite , it still exists an obvious lack of systematic fine-grained investigation of how well mt systems can handle different types of mwes and how many translation errors can be accounted for by them. therefore, we conduct this study to give initial insights into the nuanced impacts of mwes on mt systems in terms of both the overall performance and the correlation between different types of translation errors and various categories of mwes, with a special focus on the chinese-english translation. according to @xcite , mwes can be defined as the "lexical items that: (a) can be decomposed into multiple lexemes; and (b) display lexical, syntactic, semantic, pragmatic and/or statistical idiomaticity". in the current study, we make use of an existing categorization scheme for chinese mwes proposed in our previous study @xcite . in this scheme, given that multiword named entities (nes) account for a large portion of mwe items and exhibit unique linguistic behaviors and forms compared to other types of mwes @xcite , they are discussed separately. specifically, the term 'ne(s)' covers 19 types of expressions according to the framework of the ontonotes project @xcite , and an additional category 'multiword domain terminology (ter)' is further included @xcite . while 'mwe(s)' only refers to 9 non-ne types of expressions based on the combination of the framework of parseme project @xcite and some related studies, e.g. @xcite . we adopt the same terms and denotations in the present study. in our previous study @xcite , we extended the chinese-english parallel corpus provided by the wmt22 metric shared task @xcite by annotating mwes and nes in chinese source sentences. following this annotated data, we make further identification of mweand ne-related translation errors in all the mt outputs in english based on the existing error annotations according to multidimensional quality metrics (mqm) framework @xcite , which also consist in the wmt22 metric shared task corpus @xcite foot_0 . the analysis of the impacts of different kinds of chinese mwes and multiword nes on mt systems is then conducted. we compare the average human scores of mt outputs on the sentence groups with and without a particular category of mwes and nes. the results show that most mt systems obtain lower scores on sentences with mwes and/or nes. besides translation scores, we explore whether and how different mwes and nes cause particular translation errors. our investigation also presents that mwes and nes which make up about twenty percent of chinese characters result in one-third of translation errors. furthermore, the error rates of 13 categories of mwes and nes exceed 50% with the highest being 84.8%. our study strongly suggests that the current mt systems still struggle with most kinds of mwes and nes. meanwhile, the fine-grained analysis from the perspective of translation errors can provide invaluable insights into the possible directions for the improvement of mt systems. the remainder of the paper is organized as follows: section 2 introduces the related work. section 3 describes the data and error extraction procedure in detail. section 4 presents our descriptive analysis and main experiments on the impacts of various mwes and nes on chinese-english translations from the perspectives of human evaluation scores and translation errors. in section 5, we focus on typical cases and propose possible directions for further improvement of mt systems in tackling mwes and nes. section 6 concludes the study and lays out our future work. ## related work translating mwes has long been recognized as a challenging task, both in theory and in practice @xcite . special attention has been paid to the common issues encountered in mt concerned with mwes @xcite . @xcite listed "ambiguity, discontiguity, noncompositionality, and variability" as the main challenges to mt systems when dealing with mwes. in the study of @xcite , they summarized six kinds of highly frequent difficulties when automatically translating chinese mwes into english, encompassing "common sense, super sense, abstract phrase, idioms, metaphor, and ambiguity". in addition to the qualitative analysis on mwe translations, esperança-rodier and didier (2016) made a quantitative analysis by semi-automatically annotating five types of french mwes and evaluating their english translations given by the mt systems. while they found an overall satisfactory performance of the mt system in translating french mwes into english, they called for more detailed discussions on translation errors. besides, tests with challenge sets conducted by @xcite ; @xcite for mt evaluation tasks also rendered mwes essential to investigate the performance of mt systems, further highlighting the contribution of the present study. as suggested by @xcite , improvements in mt performance by being aware of mwes can be achieved through specific orchestration strategies tailored to different types of mt systems. several studies have demonstrated that integrating information about mwes can benefit both statistical mt @xcite @xcite @xcite and neural mt @xcite @xcite in terms of automatic evaluations. to sum up, while the aforementioned studies have provided a broad understanding of the challenges posed by mwes in mt and highlighted the potential benefits of incorporating mwe information into mt systems, little is known about the specific and subtle effects of different mwes from the perspective of human scores and various translation errors caused by them, hindering the further advancements in mwe-enhanced mt systems. by @xcite and base our extraction of translation errors on a well-annotated dataset constructed accordingly. the scheme combines the general mwes categories proposed by the parseme project @xcite , the nes categories by the ontonotes project @xcite , and some other categories proposed by several existing studies, e.g. @xcite ; wang (2020). the mwe-and neextended dataset in our previous study @xcite therefore covers the labels of 9 types of chinese mwes and 19 types of chinese multiword nes by taking the chinese-english parallel corpus from the wmt22 metrics shared task @xcite as the basis. finally, in our dataset, 1,359 chinese source sentences containing 1,430 mwes and 2,827 nes (accounting for 15,585 tokens/characters) are marked out. more details are shown in the original parallel corpus of wmt22 consists of 1,875 chinese sentences (74,616 tokens/characters) along with their corresponding english translations. these translations were generated by 14 state-of-the-art mt systems that participated in the shared translation task of wmt22 @xcite , standing at the forefront in the field of mt. therefore, a total of 26,250 translation items are evaluated in the current study. ## impacts of mwes and nes on general performance of mt all these categories we include here due to their idiosyncrasy nature might cause problems for mt systems. mt systems need to fully understand the meaning of such linguistic units as a whole to generate the appropriate translations in the target language. on the other hand, although they all show semantic indecompositionality, their degrees vary across categories, which may affect mt systems differently. roughly, all the categories can be further divided into two groups, namely fixed and non-fixed. nid (noun-headed idiom) is usually manifested as a nominal unit possibly with modifiers of the head noun, e.g. 铁公鸡 tie gong ji 'iron cock' (miser). such nominal idioms are quite flexible regarding their key components. for example, different modifiers can be added to the head noun or the order of the components can be changed like 铁打的 (da de, 'made of') 公鸡. vids (verbheaded idioms) differ from nids in that they are usually verbal units, typically verb-object constructions, such as 下决心 xia jue xin (make up one's mind). vids also show high flexibility in that a variety of constituents can occur in the middle, like 下了 (le, 'perfect aspect') 决 心, 下定 (ding, 'firmly') 决心 to emphasize the status, and so on. besides, some nid and vid expressions are ambiguous between their literal meanings and idiomatic meanings, e.g. 戴帽子 dai mao zi 'wear a hat' or 'bear a given label'. to correctly translate such mwes, the mt system needs to be aware of the syntactic relation between the key components and their contexts, as well as understand the idiomatic meaning of such mwes in order to make a correct judgment. ionized words or separable words (ions) are a phenomenon where a verb is separated into two parts and acts like a verb-object construction. for example, the word 成像 cheng xiang (make an image) can be separated as in 成-不了-像 cheng bu liao xiang (cannot make an image), while an can not go through a semantic shift when it is separated. thus, to correctly translate ions requires syntactic awareness of the separated components and that the non-separated canonical form is within the vocabulary of the mt system. compared to the above categories, the left categories are more fixed including idi (idioms), most nes, and others. for example, idi, which particularly refers to the four-character chinese idioms, such as 雪中送炭 xue zhong song tan 'to send charcoal in snowy weather' (to send help in one's need). such idioms are always in the same form when used. the vocabularies of these categories are mostly fixed although new items occasionally appear. whether mt systems can translate them correctly also depends on the vocabulary of the mt systems collected from their training data. finally, there is a set of open categories (meaning that they have infinite vocabulary size) including quantity, money, cardinal, and so on. although the forms of such entities are mostly fixed, they require mt systems to grasp the potential numeric generation rules of both source and target languages since the training data is not able to include all possible instances of them. we postulate that flexible mwes and such open categories of nes will pose bigger challenges than the other categories. on the other hand, the fixed ones will be a good test of the potential 'vocabulary size' of the mt systems. to test our hypothesis, we focus on two measures: average scores and error rates, in our following experiments. ## discussion with the analysis presented in this study, it is confirmed that current mt systems still face challenges from most kinds of chinese mwes and nes. although mwes and nes only take up one-fifth of the tokens in source texts, nearly one-third of all translation errors are caused by them, and among all the errors, two-thirds are major errors, seriously affecting the accuracy and fluency of translations. secondly, according to a detailed error analysis, there are some interesting differences between the features of errors on mwes and nes. the errors on mwes tend to be concentrated on major errors, particularly accuracy/mistranslation, while those on nes usually show a larger amount, a higher error rate, and greater diversity. this pattern is consistent with the features of mwes and nes themselves in sentences, where nes tend to have a larger quantity, higher proportion, and richer categories. thirdly, we have hypothesized that open ne categories such as quantity, money, car-dinal, and mwes categories with high flexibility and idiosyncrasy or semantic non-transparency such as nid, vid, and idi will cause most problems to mt systems. the experimental results are consistent with our theoretical analyses. the fol-lowing shows some typical examples of mwes and nes that have been mistranslated by mt systems foot_1 . based on the analysis associated with the given examples, some hints of possibly improving the current mt systems can be derived. firstly, most mt systems have learned quite a bit of syntax. for example, even though the systems cannot translate some nids and vids correctly due to the flexibility in their syntactic structures, they still try their best to understand the meaning only through the semantic composition of the components. there are two possible ways of solving the problem. the first is to feed more training data containing examples of such mwes or nes, so the mt systems can learn them properly. however, due to the large vocabulary, it is potentially impossible to include all of them in the training data. the second possible solution is to allow the mt models to look up dictionaries in real-time and incorporate real-time information when generating the translations. the in-context learning scheme of large language models (llms) might also be possibly integrated into the mt task-specific models as well. secondly, for the open categories that show strict inner structures such as quantity, money, date, etc., a specific component for parsing such structures and generating some proper intermediate representations might still be necessary for modern deep learning architectures. correspondingly, linguistic resources that support solving the problems associated with mwes and nes as we exposed in this study should be constructed and leveraged. ## conclusion this study completes a deep dive into the performance of state-of-the-art mt systems on the task of translating chinese mwes into english. by comparing average scores and the three types of error rates in the automatic translations of 9 types of typical chinese mwes and 19 types of chinese multiword nes, we confirm that most chinese mwes and nes are still a bottleneck problem for mt systems, causing one-third of translation errors in general. categories such as nid, vid, fac, quantity, work_of_art, product, etc. significantly degrade the performance of mt systems due to their high error rates (above 60%). the study thus provides invaluable insights for facilitating further improvement of mt systems from the perspective of integrating and intensifying knowledge of different mwes.