On the Evaluation of Terminology Translation Errors in NMT and PB - SMT in the Legal Domain: a Study on the Translation of A rabic Legal Documents into E nglish and F rench (2023)

Abstract: In the translation process, terminological resources are used to solve translation problems, so information on terminological equivalence is crucial to make the most appropriate choices in terms of translation equivalence. In the context of Machine translation, indeed, neural models have improved the state-of-the-art in Machine Translation considerably in recent years. However, they still underperform in domain-specific fields and in under-resourced languages. This is particularly evident in translating legal terminology for Arabic, where current Machine Translation outputs do not adhere to the contextual, linguistic, cultural, and terminological constraints posed by translating legal terms in Arabic. In this paper, we conduct a comparative qualitative evaluation and comprehensive error analysis on legal terminology translation in Phrase-Based Statistical Machine Translation and Neural Machine Translation in two translation language pairs: Arabic-English and Arabic-French. We propose an error typology taking the legal terminology translation from Arabic into account. We demonstrate our findings, highlighting the strengths and weaknesses of both approaches in the area of legal terminology translation for Arabic. We also introduce a multilingual gold standard dataset that we developed using our Arabic legal corpus. This dataset serves as a reliable benchmark and/or reference during the evaluation process to decide the degree of adequacy and fluency of the Phrase-Based Statistical Machine Translation and Neural Machine Translation systems.

## introduction machine translation (mt) is a subfield of computational linguistics that draws its fundamentals from linguistics, computer science, information theory, artificial intelligence, and statistics @xcite . phrase-based statistical machine translation (pb-smt) @xcite , a predictive modelling approach to mt, was the main paradigm in mt research for more than two decades. neural machine translation (nmt) @xcite @xcite , the current paradigm for mt research, is an approach to automatic translation in which a large neural network is trained by deep learning techniques. over the last five years, there has been incremental progress in the field of nmt @xcite @xcite to the point where some researchers claim parity with human translation @xcite . consistent term translation is an important facet of quality assurance for specialized translation. since terminologies are essential for communication among domain experts, term forms must be consistent and their translation must respond to the contextual requirements to maintain the integrity of the underlying conceptual system during knowledge exchange @xcite . nevertheless, some knowledge domains and languages still suffer from the lack of highquality mt results due to the mistranslation of terminology @xcite @xcite . this is the case, especially in the legal domain and the arabic language. consider example 1 from the moroccan family code, taking the terms ' ‫,'الفراش‬ ' ‫,'الطعن‬ ' ‫,'اللعان‬ ' ‫'القطع‬ into consideration: ## related work since the introduction of nmt to the mt community, researchers have been analyzing the pros and cons of nmt compared to pb-smt. @xcite examine several challenges to nmt and give empirical results on how well the technology holds up compared to pb-smt. to do this, they train both nmt and pb-smt for german-english on domains that are quite distant from each other (i.e., law (acquis), medical (emea), it, koran (tanzil), subtitles) obtained from opus @xcite . they note that the output of the nmt system is often quite fluent but completely unrelated to the input, while the pb-smt output betrays its difficulties with coping with the out-of-domain input by leaving some words untranslated. they conclude that despite the recent successes, nmt must still overcome various challenges, most notably performance in out-of-domain and underresourced conditions. @xcite conduct a survey related to arabic mt challenges which they split into two categories, namely linguistic (i.e., morphology richness, syntactic word reordering, word sense disambiguation, vocalization, dialectal variation, gender bias, etc.) and technical (i.e., low-resource language, domain mismatch, out-of-vocabulary, word alignment, sentence length, among others). several studies including @xcite ; @xcite ; @xcite prove the ineffectiveness of nmt systems, mainly google translate (gt) when producing arabic-english translations. in the context of domain-specific translation, particularly when dealing with legal texts, the problem escalates significantly. this is mainly due to domain mismatch @xcite , which @xcite tackle using multi-domain nmt. as long as the mt evaluation is concerned, researchers use different metrics such as word error rate @xcite , meteor @xcite , al-bleu @xcite metric which extends bleu @xcite to deal with arabic rich morphology. nevertheless, @xcite and another recent study by @xcite try to evaluate several automatic metrics, including the above ones. they prove that no conclusions can be drawn on the superior performance of any specific metric over others. they state that while automatic metrics such as bleu, capture the average case for how well an mt model translates sentences, they do not give insights into which linguistic aspects mt models struggle with producing fluent output. in this regard, some efforts investigate statistical error analysis of mt for arabic with native speakers so they can review linguistic aspects of mt errors @xcite ; al mahasees 2020), while others use neural networks to detect errors @xcite for arabic texts or to correct them @xcite . in another study on evaluating terminology translation in mt, @xcite examine why the automatic evaluation techniques fail to distinguish term translation in few cases, and identify the reasons (e.g., reordering, and inflectional issues in term translation). in this regard, they propose the termeval metric for the automatic evaluation of terminology translation in mt. nevertheless, the proposed metric supports only the english-hindi pair because of resources limitation. we now turn our attention to studies related to terminology translation in mt. haque et al. (2020) investigate legal domain term translation in pb-smt and nmt with two morphologically divergent languages, english and hindi. in their experiment, they adopt a technique that semiautomatically creates a gold standard test set from an english-hindi judicial domain parallel corpus. the sentences of the gold standard test set are translated with their pb-smt and nmt systems, and the patterns of the terminology translation errors on a sample set of translations is inspected 1 available here: https://github.com/kaitelfqih/goldstandard-terminology-translation-evaluation-data-set and classified. a comparative evaluation of pb-smt and nmt on terminology translation is then carried out. they find that nmt is less prone to errors than pb-smt as far as terminology translation is concerned (8.3% versus 9.9% and 11.5% versus 12.9% error rates in english-hindi and hindi-english translation tasks, respectively; differences in error rates are statistically significant). their empirical results present divergent outcomes in comparison to those reported in several prior investigations @xcite @xcite . in another scenario, @xcite study the performance of pb-smt and nmt systems on out-of-domain german-english opus data and german-romansh to define five domains (i.e., medical, it, koran, law, and subtitles). they find that in unknown domains, pb-smt and nmt suffer from different problems: pb-smt systems are mostly adequate but not fluent, while nmt systems are mostly fluent but not adequate. for nmt, they identify hallucinations (translations that are fluent but unrelated to the source) as a key reason for low domain robustness. several studies, including al-shehab (2013); @xcite ; junczys-dowmunt et al. (2016); @xcite , prove that although nmt systems are known to generalize better than phrase-based systems for out-of-domain data, it is unclear how they perform in purely in-domain setting, especially in the legal domain from arabic where terminology translation remains questionable and subject to continuous post-edition @xcite . given all the serious translation issues that arabic terminology in the legal domain faces, it remains a poorly explored area in mt research. hence, extensive research efforts are still needed to enhance and refine these aspects. ## experiments set-up and methodology to conduct our study, we semi-automatically create a gold standard dataset 1 from our legal corpus that we created using a variety of legal documents (i.e., codes, contracts, provisions, constitutions, and decrees) of different arab countries. the resource setup is described in detail in to address our research objectives, our methodology unfolds four distinct phases, to investigate key aspects of the study. they are as follow: the second phase of the experiment focuses on manual evaluation carried out by a native arabic speaker. it consists of a systematic analysis where we classify and annotate the errors (see section 4) of machine-translated out-of-context and incontext legal terms from ar to en and fr produced by different mt systems (gt and rc). figure 1 displays the number of terms and sentences containing errors, along with their corresponding percentages in ## evaluation and results the errors posed by machine-translated legal terms for arabic are classified into six error types for outof-context terms ( for ar-en, nmt appears to be more errorprone than pb-smt. nmt commits 36% of errors related to ae, clsre, re, and below 40% of errors related to te and gbe. whereas pb-smt presents 62% of errors related to ae, clsre, re, and below 33% of errors related to te and gbe. in addition, only 2% in nmt and 5% in pb-smt are correct translations. for the ar-fr pair, nmt preserves its status of being more erroneous than pb-smt, where nmt presents a percentage of 58% of errors related to ae, clsre, re, and 38% in favor of te and gbe. whereas pb-smt achieves 56% of ae, clsre, re, but outperforms nmt with 40% of errors related to te and gbe. in addition, 4% in nmt and 4% in pb-smt are correct translations. the judge ruled on 01/01/2011 all the duties of the wife to the husband of maintenance and pleasure. en (ht): on 01/01/2011, the judge sentenced that the husband must comply with all the wife's rights, including expenditure and compensation. fr (smt): le juge a statué le 01/01/2011 sur l'ensemble des devoirs de la femme envers le mari d'entretien et de jouissance. fr (ht): le 01/01/2011, le juge a condamné le mari à respecter tous les droits de sa femme, y compris les dépenses et l'indemnisation. the bold terms in example 1 are domainspecific and context-dependent these factors make their accurate translation a complex process. the ht, indeed, considers various elements, including context, exact terminology choice, structure, syntax, as well as their compatibility across languages and legal systems. whereas nmt and pb-smt systems fail in producing quality translations due to errors, such as: our corpus consists of judicial documents (i.e., contracts, provisions, codes, decrees) of different arab countries (morocco, algeria, tunisia, united arab emirates, saudi arabia, egypt). therefore, the use of distinct legal terminology to convey similar legal practices in different countries can significantly impact the outcomes of mt for arabic. due to variations in legal systems, cultural nuances, idiomatic expressions, linguistic variations, and the specific precision required in legal language, mt may struggle to accurately capture the intended meanings. this could lead to mistranslations, misinterpretations, and errors that have potentially serious legal consequences. for example, the term ' ‫'مأذون‬ is used mostly in qatar and egypt. it is used to refer to the person certified by the judge to perform certain legal formalities, especially to draw up or certify marriage contracts, deeds, and other documents for use in other jurisdictions 6 . rc, however, translates it as 'authorized' into english and 'autorisé' into french. whereas gt, as well, translates it as 'authorized' into english and 'autorisé' into french. therefore, we notice that both systems not only transform the grammatical category of the term from a noun, which represents a person into an adjective, but they also misinterpret the intended legal practice in the target legal systems. hence, in france, the equivalence of ' ‫'مأذون‬ is 'maire' (i.e., the person who chairs the municipal council 7 ), he/she is the one who oversees approving and drawing up marriage contracts. whereas in england the person in charge of approving and celebrating the marriage requests is called the 'superintendent registrar foot_1 ' of the district. this unveils that mt systems are not trained on a diverse and comprehensive dataset that covers a wide range of legal terminologies from different countries. in other words, mt systems need to be equipped with region-specific legal dictionaries and context-aware algorithms that consider the nuances of each country's legal language. additionally, leveraging parallel legal texts in different terms can help train mt models to better handle these variations. ## conclusion and future work in this paper, we conduct a comparative qualitative evaluation and comprehensive error analysis on legal terminology translation between pb-smt and nmt in two translation pairs: ar-en/ ar-fr. we also introduce a multilingual gold standard dataset that we developed using our arabic legal corpus, which serves as a reliable benchmark and/or reference during the evaluation process to decide the degree of adequacy and fluency of the pb-smt and nmt systems. we propose an error typology taking the legal terminology translation from arabic into account. we demonstrate our findings, highlighting the strengths and weaknesses of both approaches to mt in legal terminology translation for arabic. we found that nmt is more error-prone than pb-smt in both language pairs when translating out-ofcontext terms. whereas, for the ar-en pair, nmt seems to exhibit a higher percentage of errors compared to pb-smt concerning in-context machine-translated legal terms. concerning the ar-fr language pair, although nmt and pb-smt have the same overall error rate (94%) nmt produces more errors related to re, ae, clsre, and register errors. the findings also demonstrate that despite advances in mt, legal translation remains a 7 eesc/cor-fr, d'après le conseil des communes et régions d'europe (ccre), «gouvernements locaux et régionaux en @xcite (3.5.2022), page 26 challenging task that demands precision and adherence to specific legal nuances. for critical legal documents, human translation by professional legal experts is still the preferred approach to ensure the highest level of accuracy and consistency. mt, however, can be a helpful tool for initial draft translations or to aid human translators, but it should be used with caution, especially for legal content. as future work, a second annotator will undertake the annotation of the data concerning the mt errors, and the assessment of inter-annotator agreement will be conducted to enhance the reliability of the data. we will afterward focus on developing a highquality multilingual corpus from ar-en/ ar-fr in the legal domain to enhance the performance of mt systems. careful attention will be given to aligning sentences with precise legal terminologies to provide reliable and contextual translations.