A lpha MWE - A rabic: A rabic Edition of Multilingual Parallel Corpora with Multiword Expression Annotations (2023)

Abstract: Multiword Expressions (MWEs) have been a bottleneck for Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks due to their idiomaticity, ambiguity, and non-compositionality. Bilingual parallel corpora introducing MWE annotations are very scarce which set another challenge for current Natural Language Processing (NLP) systems, especially in a multilingual setting. This work presents AlphaMWE-Arabic, an Arabic edition of the AlphaMWE parallel corpus with MWE annotations. We introduce how we created this corpus including machine translation (MT), post-editing, and annotations for both standard and dialectal varieties, i.e. Tunisian and Egyptian Arabic. We analyse the MT errors when they meet MWEs-related content, both quantitatively using the human-in-the-loop metric HOPE and qualitatively. We report the current state-of-the-art MT systems are far from reaching human parity performances. We expect our bilingual English-Arabic corpus will be an asset for multilingual research on MWEs such as translation and localisation, as well as for monolingual settings including the study of Arabic-specific lexicography and phrasal verbs on MWEs. Our corpus and experimental data are available at https://github.com/aaronlifenghan/AlphaMWE .

## introduction multiword expressions (mwes), such as "a cheap shot" (a cruel verbal attack) or "take it with a grain of salt" (regard something as exaggerated), are combinations of words that function as a single unit and have a specific meaning @xcite , typically regarded as a 'pain in the neck' to natural language processing (nlp) tasks, particularly in the field of machine translation (mt) @xcite and information extraction @xcite . translating mwes accurately poses a significant challenge for statistical and neural mt systems @xcite @xcite . the difficulty lies in the idiomatic, colloquial or culture-specific nature of mwes, which requires a deep understanding of their meaning, context, and cultural references @xcite . additionally, mwes can be interpreted into multiple possible meanings, further complicating their translation. therefore, a parallel corpus that incorporates mwe annotation is expected to be useful for improving the mt quality via system finetuning and error analysis. however, arabic seems to lack a satisfactory corpus for such use. the literature describes several english-arabic parallel corpora. however, to the best of our knowledge, none of these corpora includes the mwes annotation. in this paper, we describe our ongoing effort to extend alphamwe coordinated by @xcite , a multilingual parallel corpus with annotation of mwes, to the arabic language including both standard and dialectal ones, i.e. the egyptian and tunisian arabic. arabic is a morphologically rich language and has been challenging for state-of-the-art mt systems @xcite . fol-lowing alphamwe, our study primarily focused on verbal mwes (vmwes). a vmwe is defined as a mwe whose canonical form has a verb as its syntactic head @xcite @xcite with popular examples "kick the bucket", "take ... for granted", and "swallow someone's pride". we used state-of-the-art mt engines to facilitate the standard arabic corpus creation and we will discuss the pros and cons of different mt models on mwe-related translation errors. we carried out manual post-editing and annotations by native arabic speakers for this. regarding dialectal arabic corpus, we translated them from english from scratch, since the current mt models do not cover dialectal arabic translations and the quality from mt output is too low to be useful, which also indicated the value of our corpus creation. overall, in this work, we not only contribute to a series of parallel corpus on english-arabic with mwe annotations but also give qualitative and quantitative analysis on mt errors facing mwes, which we hope will be valuable for future mt research on this language pair. the rest of this paper is organised as follows: section 2 describes previous work dedicated to parallel arabic corpora and compares our contribution to the state of the art. section 3 is a brief introduction to the arabic language. section 4 explains the construction of the alphamwe-arabic corpus and the qualitative evaluation using examples from mt outputs. in section 5, we offer more quantitative and statistical analyses of the data annotation process using the human-in-the-loop metric hope @xcite . finally, section 6 concludes our paper and discusses perspectives for future work. ## related work the development of machine translation for lowresource languages is a widely studied challenge in nlp @xcite . many efforts have been made to create effective mt models. to train these models, various parallel resources have been proposed. @xcite created the united nations parallel corpus, which consists of over 2 million words of parallel texts in 6 official languages, including english and arabic. another work that includes arabic is the multilingual parallel corpus multiun @xcite . it extends the united nations parallel corpus by including texts from various sources such as the united nations and other international organisations. in addition, several researchers have undertaken efforts to construct resources for arabic dialects. @xcite built a bilingual dictionary that utilised explicit knowledge about the relationship between tunisian arabic and modern standard arabic (msa). @xcite translated dialectal arabic to msa as a bridge to translate to english. @xcite created a multi-dialectal arabic parallel corpus that contains 2000 sentences in msa, egyptian, tunisian, jordanian, palestinian, and syrian arabic. however, this previous work mainly focused on the creation of lexical and grammatical parallel resources using either manual or automatic methods, without annotation of linguistic phenomena such as mwes. to address this, there have been numerous studies aiming at creating monolingual corpora annotated with verbal mwes, such as the parseme shared task corpora @xcite . parseme is a multilingual initiative that targets the parsing of mwes in over 26 different languages, including msa @xcite , but they are not parallel data. to extend this effort, alphamwe @xcite ) not only focuses on the creation of multilingual parallel corpora but also incorporates the annotation of mwes in both the source and target languages. so far, 4 languages are covered in alphamwe, namely english, chinese, polish, and german. however, as we discussed earlier, there is a lack of such parallel corpora for arabic, even though it is one of the most spoken and used languages. in this work, we develop an arabic edition of alphamwe including both the standard language and dialectal varieties. ## on arabic language the term "arabic language" today can refer to either modern standard arabic (msa) or various spoken vernaculars referred to as arabic dialects. the classical form of msa is used in religious texts, poetry, and formal writing, whereas the dialectal form is used in everyday and colloquial conversation. we give in this section a brief overview of msa specificities. firstly, in msa, there are no capital letters and the use of punctuation marks is not widely adopted in current arabic texts (at least not reg-ularly). secondly, arabic tends to use long and complex sentences with right-to-left writing, making it common to find an entire paragraph without any punctuation. thirdly, as a semitic language, arabic has a complex morphology. indeed, it uses concatenative morphology (agglutinated or compound words), where words are formed via a sequential concatenation process foot_0 . for example, the sentence 'then they will write it' is presented in arabic as one word ‫.فسيكتبونها‬ in addition, the arabic language has some words that can add diacritical marks on top or below them to form new words that have new pronunciations and meanings, of which the new pronunciation is similar to the ones from the original word. as a result, texts without diacritical marks are highly susceptible to ambiguity. for instance, the word/symbol ‫علم‬ (pronunciation: alam) can be diacritised in 9 different forms @xcite including ‫ِلم‬ ‫ع‬ ("science", pronunciation: elm), ‫لم‬ َ ‫ع‬ ("flag", pronunciation: alam), and ‫م‬ َّ ‫ل‬ َ ‫ع‬ ("he taught", pronunciation: ellem), etc. finally, another special aspect of arabic is its flexible word order, where the rearrangement of certain words in a sentence does not affect its meaning. this is because the language uses case markers, particles, and other linguistic tools to clarify the connections between words, resulting in a more flexible syntax compared to languages with a more rigid word order. for example, "the boy went to the school" can be written in arabic in three forms: ‫المدرسة‬ ‫إلى‬ ‫ذهب‬ ‫الولد‬ (the boy went to the school), ‫المدرسة‬ ‫إلى‬ ‫الولد‬ ‫ذهب‬ (went the boy to the school), and ‫الولد‬ ‫ذهب‬ ‫المدرسة‬ ‫إلى‬ (to the school went the boy). these unique features make arabic a challenging language for nlp tasks. ## alphamwe-arabic following alphamwe @xcite , we used the parseme corpus for english as the source language. the parseme corpus is well established and provides a clear process of tagging and categorisation. the english corpus used in the parseme shared task was created by @xcite , where 832 vmwes were manually annotated across 7,437 sentences taken from various topics and domains, such as news, lit-erature, and it documents foot_1 . overall there are around 750 sentences extracted from the source parseme english corpus that include vmwe tags by alphamwe foot_2 . furthermore, alphamwe divided these 750 sentences into 5 portions (by files) with the same size, i.e. around 150 sentences each for cross-validation and system-tuning purposes. we followed this process for the creation of our three corpora: modern standard arabic, tunisian arabic, and egyptian arabic. we will first introduce the workflow for creating standard arabic msa including the usage of mt; then we introduce the ones for the dialectal varieties. ## statistical mt error analysis evaluation of hope tasks can be carried out both with and without a final human-generated reference translation. regardless, the evaluator assesses errors based on the hope quality metrics and assigns a score based on the severity of the errors using a penalty points system. in this task, we will generate a gold standard translation for the purpose of our open-source parallel corpus creation. figure 2 shows the statistics from the hope metric on the 'aa' portion, one of the five files (aa to ae) included in the alphamwe corpus, on the percentage of mt output sentences that falls into 'un-changed (correct)', 'minor errors', and 'major errors'. the scoreboard shows that only 35% of mt output sentences are correct, and there are 44% and 21% of sentences having minor and major errors. ## discussion and future work to bridge the gap in the parallel corpus of english-arabic with mwe annotations, we created alphamwe-arabic, an arabic edition of the alphamwe corpus. this is another step further to facilitate low-resource language processing including dialectal ones and can be useful to both multilingual and monolingual mwe-focused research. during our creation, we introduced two new error types to the hope metric, and the experimental results show that mwe-related errors have a big ratio out of all error types. this reflects that the current state-of-the-art mt systems are still far from reaching human parity as they falsely claimed sometimes, which was partially due to their limited evaluation setting @xcite @xcite . for the standard arabic corpus, we had two native speakers who carried out the post-editing and annotation. the corpus quality was ensured by cross-validation, i.e. having the second person check on the output from the other person's first edit. however, to quantitatively measure the interannotator agreement @xcite ) levels, in the future, we plan to design some experiments on calculating how much chance they agree with each other on the mt output quality and on the post-editing, e.g. to target mwes vs non-mwes. following the alphamwe open-source project, we plan to extend our corpus to a larger size and launch an open research project call where researchers can contribute and volunteer for the extension of the english-arabic corpus. we have shared tasks in mind by contributing our corpus as a mwe-focused mt challenge, e.g. using human-in-the-loop mt evaluation metric hilmeme that looks into mwes @xcite .