A lpha MWE : Construction of Multilingual Parallel Corpora with MWE Annotations (2020)

Abstract: In this work, we present the construction of multilingual parallel corpora with annotation of multiword expressions (MWEs). MWEs include verbal MWEs (vMWEs) defined in the PARSEME shared task that have a verb as the head of the studied terms. The annotated vMWEs are also bilingually and multilingually aligned manually. The languages covered include English, Chinese, Polish, and German. Our original English corpus is taken from the PARSEME shared task in 2018. We performed machine translation of this source corpus followed by human post editing and annotation of target MWEs. Strict quality control was applied for error limitation, i.e., each MT output sentence received first manual post editing and annotation plus second manual quality rechecking. One of our findings during corpora preparation is that accurate translation of MWEs presents challenges to MT systems. To facilitate further MT research, we present a categorisation of the error types encountered by MT systems in performing MWE related translation. To acquire a broader view of MT issues, we selected four popular state-of-the-art MT models for comparisons namely: Microsoft Bing Translator, GoogleMT, Baidu Fanyi and DeepL MT. Because of the noise removal, translation post editing and MWE annotation by human professionals, we believe our AlphaMWE dataset will be an asset for cross-lingual and multilingual research, such as MT and information extraction. Our multilingual corpora are available as open access at github.com/poethan/AlphaMWE.

## introduction multiword expressions (mwes) have long been of interest to both natural language processing (nlp) researchers and linguists @xcite @xcite . the automatic processing of mwes has posed significant challenges for some fields in computational linguistics (cl), such as word sense disambiguation (wsd), parsing and (automated) translation @xcite @xcite @xcite . this is caused by both the variety and the richness of mwes as they are used in language. various definitions of mwes have included both syntactic structure and semantic viewpoints from different researchers covering syntactic anomalies, non-compositionality, nonsubstitutability and ambiguity @xcite . for instance, @xcite define mwes as "lexical items that: (i) can be decomposed into multiple lexemes; and (ii) display lexical, syntactic, semantic, pragmatic and/or statistical idiomaticity". however, as noted by nlp researchers for example in @xcite , there are very few bilingual or even multilingual parallel corpora with mwe annotations available for cross-lingual nlp research and for downstream applications such as machine translation (mt) @xcite . with regard to mwe research, verbal mwes are a mature category that has received attention from many researchers @xcite . verbal mwes have a verb as the head of the studied term and function as verbal phrases, such as "kick the bucket", "cutting capers" and "go to one's head". in this work, we present the construction of a multilingual corpus with vmwes annotation, including english-chinese, english-german and english-polish language pairs. the same source monolingual corpus is in english with its vmwe tags from the shared task affiliated with the siglex-mwe workshop in 2018 @xcite . several state-of-the-art (sota) mt models were used to perform an automated translation, and then human post editing and annotation for the target languages was conducted with cross validation to ensure the quality, i.e., with each sentence receiving post-editing and rechecking by at least two people. in order to get a deeper insight into the difficulties of processing mwes we carried out a categorisation of the errors made by mt models when processing mwes. from this we conclude that current state-of-the-art mt models are far from reaching parity with humans in terms of translation performance, especially on idiomatic mwes, even for sentence level translation, although researchers sometimes claim otherwise @xcite . the rest of this paper is organised as follows. in the next section we present related work and then detail the corpus preparation stages including selection of mt models. we then look at the various kinds of issues that mt has with mwes. this analysis, along with the public release of the corpora as a resource to the community, is the main contribution of the paper. ## related work there are a number of existing studies which focus on the creation of monolingual corpora with vmwe annotations, such as the parseme shared task corpora @xcite . the 2020 edition of this task covers 14 languages including chinese, hindi, and turkish as non-european languages. some work from monolingual english corpora includes the mwe aware "english dependency corpus" from the linguistic data consortium (ldc2017t01) that covers compound words used to train parsing models. also related to this are english mwes from "web reviews data" by @xcite that covers noun, verb and preposition super-senses and english verbal mwes from @xcite and @xcite that covers parseme shared task defined vmwe categories. however, all these works were performed in monolingual settings, independently by different language speakers without any bilingual alignment. these corpora are helpful for monolingual mwe research such as discovery or identification, however, it would be difficult to use these corpora for bilingual or multilingual research such as mt or cross-lingual information extraction. the work most related to ours is from @xcite , who created an english-hungarian parallel corpus with annotations for light verb constructions (lvcs). as many as 703 lvcs for hungarian and 727 for english were annotated in this work, and a comparison between english and hungarian data was carried out. however, the work did not cover other types of vmwes, for instance inherently adpositional verbs, verbal idioms, or verb-particle constructions, and it was not extended to any other language pairs. in our work, we annotate in a multilingual setting including far distance languages such as english, german, polish and chinese, in addition to the extension of vmwe categories. in other recent work @xcite , we performed an automatic construction of bilingual mwe terms based on a parallel corpus, in this case english-chinese and english-german. we first conducted automated extraction of monolingual mwes based on part-of-speech (pos) patterns and then aligned the two side monolingual mwes into bilingual terms based on statistical lexical translation probability. however, due to the automated procedure, the extracted bilingual "mwe terms" contain not only mwes but also normal phrases. part of the reason for this is due to the pos pattern design which is a challenging task for each language and needs to be further refined @xcite @xcite . mt compare four systems post-editing of machine output sentences annotate target mwes second round recheck from different person we selected deepl as our final mt output engine ensure fluency and adequacy, also paying attention to mwes the source and target mwes are bilingual well aligned and annotated to reduce human introduced / ignored translation errors top: corpora-flow; middle: work-flow; bottom: tools and techniques post-edited zh/pl/de/es… annotated mwes validated targets parseme english target languages figure 1: workflows to prepare alphamwe. ## experimental work in this section we describe our corpus preparation method, selection of the mt models used in our investigation, and the resulting open-source corpora alphamwe. ## mt issues with mwes we performed an analysis of the behaviour of various mt systems when required to translate mwes or mwes related context. due to space limitations, in this paper we focus on the @xmath0. we also highlight some issues on @xmath1, but leave the detailed analysis of other language pairs for future work. when mt produces incorrect or awkward translations this can fall into many different categories, and from our analysis we classify them as: common sense, super sense, abstract phrase, idiom, metaphor and ambiguity, with ambiguity further sub-divided. these classifications are to be further refined in the future, e.g. the differences between metaphor and idiom are usually fuzzy. we now list each of these with examples to support future mt research on improving the quality of mt when handling mwes. ## conclusion and future work in this paper, we presented the construction of multilingual parallel corpora, alphamwe, with vmwes as pioneer annotations by native speakers of the corresponding languages. we described the procedure of mt model selection, human post editing and annotation, and compared different state-of-the-art mt models and classified the mt errors from vmwes related sentence/context translations. we characterised the errors into different categories to help mt research to focus on one or more of them to improve the performance of mt. we performed the same process as described here for @xmath2, @xmath3. the en-@xmath4: (1) there are cases where the corresponding german translation of english mwes can be one word, which is partially because that german has separable verbs, (2) the automated translation to german is biased towards choosing the polite or formal form of the words, which is generally fine but depends on the context to decide which form is more suitable, and (3) english vmwes are often not translated as vmwes to german. in the main, @xmath5, literal translation errors and context unaware situation errors. we name our process as alphamwe to indicate that we will continue to maintain the developed corpora which are publicly available and extend them into other possible language pairs, e.g. spanish, french and italian (under-development). we also plan to extend the annotated mwe genres beyond the vmwes defined in the parseme shared task.