Large Language Models as Legal Translators of A rabic Legislation: Do C hat GPT and G emini Care for Context and Terminology? (2024)

Abstract: Accurate translation of terminology and adaptation to in-context information is a pillar to high quality translation. Recently, there is a remarkable interest towards the use and the evaluation of Large Language Models (LLMs) particularly for Machine Translation tasks. Nevertheless, despite their recent advancement and ability to understand and generate human-like language, these LLMs are still far from perfect, especially in domain-specific scenarios, and need to be thoroughly investigated. This is particularly evident in automatically translating legal terminology from Arabic into English and French, where, beyond the inherent complexities of legal language and specialised translations, technical limitations of LLMs further hinder accurate generation of text. In this paper, we present a preliminary evaluation of two evolving LLMs, namely GPT-4 Generative Pre-trained Transformer and Gemini, as legal translators of Arabic legislatives to test their accuracy and the extent to which they care for context and terminology across two language pairs (AR→EN / AR→FR). The study targets the evaluation of Zero-Shot prompting for in-context and out-of-context scenarios of both models relying on a gold standard dataset, verified by professional translators who are also experts in the field. We evaluate the results applying the Multidimensional Quality Metrics to classify translation errors. Moreover, we also evaluate the general LLMs outputs to verify their correctness, consistency, and completeness. In general, our results show that the models are far from perfect and recall for more fine-tuning efforts using specialised terminological data in the legal domain from Arabic into English and French.

## introduction over the last five years, there has been incremental progress in the field of machine translation (mt) @xcite @xcite to the point where some researchers claim parity with human translation (ht) @xcite . even recently, with the huge advent of large language models (llms) such as generative pretrained transformer (gpt) @xcite and gemini @xcite , it is anticipated that they will become more accurate, robust, and adaptable to a wider range of languages, and domains, making them even more valuable tools in our interconnected world @xcite @xcite . llms significantly demonstrate excellent performance in diverse natural language processing (nlp) application tasks, including mt, due to their self-supervised and transfer learning abilities @xcite . this progress has led to focus on model scalability, efficiency, and humanlike understanding and translation capabilities. however, their potential in domain-specific mt remains under-explored. in other words, the extent to which these models are consistent and accurate in the translation of technical texts and the terminology used therein is uninspected. as a matter of fact, consistent term translation is an important facet of quality assurance for specialized translation. since terminologies are essential for communication among domain experts, term forms must be consistent and their translation must respond to the contextual requirements to maintain the integrity of the underlying conceptual system during knowledge exchange @xcite . nevertheless, some knowledge domains and languages still suffer from the lack of high-quality mt results due to the multidimensional mistranslations of terminology @xcite @xcite . this is the case, especially in the legal domain and the arabic language. consider example 1 from the moroccan family code, taking the terms { ‫,}الفراش‬ ‫,}الطعن{‬ ‫,}اللعان{‬ ‫}القطع{‬ into consideration: 1. ar: ‫يعتبر‬ ‫الفراش‬ ‫ثبوت‬ ‫على‬ ‫قاطعة‬ ‫حجة‬ ‫بشروطه‬ ‫يمكن‬ ‫ال‬ ‫النسب،‬ ‫الطعن‬ ‫طريق‬ ‫عن‬ ‫الزوج‬ ‫من‬ ‫إال‬ ‫فيه‬ ‫اللعان‬ ‫تفيد‬ ‫خبرة‬ ‫بواسطة‬ ‫أو‬ ، ‫القطع‬ . en (chatgpt-4): the consummation with its conditions constitutes conclusive evidence of the lineage, which can only be challenged by the husband through "li'an" or by means of expertise proving its falsity. fr (chatgpt-4) : the marital bed, under its conditions, is conclusive evidence of the establishment of lineage. it cannot be challenged except by the husband through cursing (li'an) or through expert testimony that provides certainty. en (gemini): the marriage bed, with its conditions, is considered conclusive evidence of lineage; it can only be challenged by the husband through invoking curses, or by expert testimony that provides conclusive evidence. fr (gemini): la consommation est considérée comme une preuve concluante de la lignée, qui ne peut être considerée que par le mari à travers la "li'an" ou par des moyens d'expertise prouvant sa fausseté. ht (en): marriage consummation is considered a strong proof of paternity, it can be rebutted only by the husband through accusation or through a certain evidence. (ht) (fr): la consommation du mariage est considérée comme une preuve solide signifiant la paternité, il ne peut être réfutée que par le mari soit à travers l'accusation ou bien une certaine preuve. the bold terms in example 1 are domain-specific and context-dependent, so their translation require the consideration of the context, as well as of the cultural, lexical, morphological, and semantic properties of the terms in addition to their equivalences across languages and legal systems (i.e., english, and french), as the ht does. both models, instead, produce results with high levels of multidimensional critical errors. this example highlights the fact that these models can indeed capture general language patterns, generate coherent responses, and perform excellent mt tasks in general but struggle with translating domain-specific texts and terminology due to their limited domain-knowledge and limited training data. in addressing this gap, many authors conducted a thorough evaluation of llms potentials in mt application including @xcite who find out that while some highresource languages such as english-to-french, english-to-spanish and even english-to-chinese show excellent results, other languages have lower support either because they are lowresource languages such as english-to-kinyarwanda or because of issues in the chatgpt-3.5 tokenizer such as english-to-arabic. @xcite survey reveals that current llms exhibit certain limitations in numerous tasks, notably reasoning and robustness tasks. they suggest the need for contemporary evaluation systems that ensure the accurate assessment of llms' inherent capabilities and limitations. @xcite presents a comprehensive study of the mt capabilities of the latest chatgpt models. their investigation covers 18 language pairs across four different domains, enabling a broad understanding of the models' general performance. their findings demonstrate that chatgpt-3.5 can produce highly fluent and competitive translation outputs especially and only for the high-resource language translations. all of them and others suggest that fine-tuning is an ultimate option to improve the in-context learning capability of llms and improve their translation quality and adherence to domain terminology and style @xcite @xcite @xcite @xcite . in this paper, we present a comprehensive evaluation and an indepth study of the mt capabilities of chatgpt-4 and gemini in in-context and out-of-context levels in the legal domain in two language pairs @xmath0. we also evaluate the potential of chatgpt-4 and gemini in mt of legal texts and terminology in arabic using a gold standard dataset developed specifically for assessing the quality and accuracy of machine-translated legal terms from arabic into english and french. this paper is structured as follow: section 1 presents the introduction, section 2 details the experimental set-up and methodology, section 3 discusses the evaluation and results, and section 4 presents conclusions and future work. ## experiment set-up and methodology despite their great ability to perform mt tasks, both chatgpt-4, developed by openai foot_0 , and gemini, by google deepmind foot_1 , encounter several and different pitfalls in accurately translating legal terminology from @xmath1, due to limited exposure to specialized terminological data in the legal domain during training. this leads to challenges in contextual and cultural understanding, besides inconsistencies, and potential misinterpretations of legal concepts in arabic. therefore, in this paper, we present a comprehensive evaluation of both models in the translation of arabic legislation and answer the question of whether these models care for context and terminology in two language pairs (@xmath2. to do this, i) we develop an arabic legal corpus (more details about the sources, size, coverage, terminology extraction and annotation etc., are mentioned in corpus and terminology extraction paragraph),, ii) generate translations from chatgpt-4 and gemini, iii) evaluate the accuracy of both llms outputs based on our gold standard dataset, which to the best of our knowledge is the first formalized legal terminological resource for arabic developed specifically for assessing the quality and accuracy of machine-translated legal terms from arabic into english and french, and then v) apply the mqm foot_2 @xcite framework to classify translation errors and calculate the severity levels presented by the models. our gold dataset comprises a collection of 1,015 arabic legal terms, each accompanied by 1,015 sentences where these terms are used, along with their translations into english and french. to make it reliable, our reference dataset is developed in accordance with the online gateways of eu laws, including eur-lex foot_3 , iate foot_4 , juremy foot_5 to retrieve the exact equivalence of legal terms in english and french, and is then submitted to two annotators to verify and validate it. the first is a legal expert whose language skills are excellent both in the source and the target languages. he validates the translations after checking their degree of accuracy and adequacy in the target languages. the second is a native arabic speaker with a linguistic background meticulously who annotates the partof-speech tags, geographical usage (following the iso 20771:2020 foot_6 standard for legal translation requirements, to indicate where a given term is adapted to express a legal practice). this validation method for such sensitive dataset is important for the sake of equivalence reference to ensure an adequate and accurate analysis. this dualannotator approach enhances the quality of the data by reducing the chances of errors and inconsistencies, and it provides a standardized point of reference for evaluating mt systems objectively and systematically in the area of legal terminology translation for arabic. corpus and terminology extraction. we use nooj foot_7 , an nlp software application, to develop a corpus of arabic legislation of different arab countries called the arabic legal corpus (arlc) that consists of 532,688 tokens @xcite @xcite including marriage contracts, divorce provisions, constitutions, decrees, codes, etc., (see write only the english/french translation without providing comments.' 'write the translations following the order provided.' prompts 1 and 2 generate from the llms the translation for out-of-context and in-context legal terms and sentences into english and then into french, each separately respecting the order provided. prompt 1 is instructed to provide translations without providing additional context or comments and prompt 2 is directed to provide no additional comment. the generated output, subject for evaluation, is refined and classified as follow: ## evaluation and results our evaluation consists of three fundamental phases. the first is to assess the obtained results of both models, gpt-4 and gemini, based on the accuracy criterion against the gold dataset. the second is the evaluation of the results applying the mqm @xcite , which is a framework for analytic translation quality evaluation (tqe) 9 that can be applied to evaluate machine-translated legal terms for arabic by chatgpt-4 and gemini against our gold dataset. the third consists of evaluating the general llms outputs to verify their correctness, consistency, and completeness in the mt of legal terminology for arabic. therefore, we apply the mqm @xcite framework. it includes a set of error types and a scoring model. error types are organized in a chatgpt-4 gemini error types error sub-types out-of-context @xmath5 subsequently, to be more precise towards the evaluation and the final judgement of the mt results produced by the models, we calculate the final quality of the translation results using the mqm scoring model as shown tables 4 , 5 , 6, 7, 8, 9, 10 & 11 in appendix a in by assigning scores to: conversely, in the in-context scenario ( in the in-context @xmath10, both chatgpt-4 and gemini demonstrate a high critical number of errors than the major and minor ones compared to the out-of-context (see our terminological dataset consists of judicial documents (i.e., contracts, provisions, codes, decrees, etc.) of different arab countries (morocco, algeria, tunisia, united arab emirates, saudi arabia, egypt). therefore, the use of distinct legal terminology to convey similar legal practices in different countries can significantly impact the outcomes of mt for arabic. due to variations in legal systems, cultural nuances, idiomatic expressions, linguistic variations, and the specific precision required in legal language, chatgpt-4 and gemini may struggle to accurately capture the intended meanings. this could lead to mistranslations, misinterpretations, and errors that have potentially serious legal consequences. for example, the term { ‫}مأذون‬ is used mostly in qatar and egypt. it is used to refer to the person certified by the judge to perform certain legal formalities, especially to draw up or certify marriage contracts, deeds, and other documents for use in other jurisdictions 13 . chatgpt-4, however, translates it as 'authorized' into english and 'autorisé' into french. whereas gemini, as well, translates it as 'authorized' into english and 'autorisé' into french. therefore, we notice that both models not only transform the grammatical category of the term from a noun, which represents a person into an adjective, but they also misinterpret the intended legal practice in the target legal systems. hence, in france, the equivalence of { ‫}مأذون‬ is 'maire' (i.e., the person who chairs the municipal council foot_11 ), he/she is the one who oversees approving and drawing up marriage contracts. whereas in england the person in charge of approving and celebrating the marriage requests is called the 'superintendent registrar foot_12 ' of the district. this unveils that these models are not trained on a diverse and comprehensive dataset that covers a wide range of legal terminologies from different countries. in other words, these models need to be equipped with region-specific legal dictionaries and context-aware algorithms that consider the nuances of each country's legal language. additionally, leveraging parallel legal texts in different terms can help train the models to better handle these variations. ## conclusion and future work the results show that gemini surpasses chatgpt-4 in achieving less critical errors in out-of-context scenario from @xmath11, conversely, in in-context scenario, chatgpt-4 demonstrates less critical results, but more minor and major errors compared to gemini with only slight differences regarding the critical error severity count. in @xmath12, chatgpt-4 and gemini achieve better results in out-of-context scenarios but face significant challenges with accurately capturing the nuances and complexities of legal terminology when the context is provided. this is because the complexity of legal language and the terminology used to express certain concepts that are culturally and religiously bound present critical barriers. secondly, these models are pre-trained on vast amount of text from diverse sources and on a large amount of unlabeled corpora, and they might not have access to sufficient amounts of domainspecific texts for every possible domain. in this regard, we believe that efforts should be invested on enhancing context sensitivity of llms' mt tasks by i) developing models capable of handling longer text sequences, such as transformer-xl @xcite , which can capture broader dependencies, ii) incorporating hierarchical modeling to process text at multiple levels (sentence, paragraph, document) to ensure better context continuity, iii) training with datasets that preserve discourse structure and using dynamic context integration techniques, like retrieval-augmented generation (rag) @xcite which combines generative models with retrieval mechanisms to dynamically incorporate relevant context from external sources during the generation process. indeed, mqm offers an insightful and comprehensive approach to evaluating machinetranslated legal terms for arabic by chatgpt-4 and gemini. this framework enables a granular assessment of the accuracy state of the models better than any automatic metric would do, by considering various dimensions such as terminology, style, linguistic convention, accuracy, and audience appropriateness, and their respective sub-dimensions. this detailed analysis provides us with actionable feedback for improvement as it presents detailed insights that go beyond automatic metrics. in the future, we will consider fine-tuning these models with extra context to improve their incontext learning abilities, as well as the translation quality and adherence to the domain terminology and style. we might also consider the use of dynamic context integration techniques like rag to enhance the models ability to handle complex and context-dependent translations by providing them with access to a broader and more relevant set of contextual information during the translation process. additionally, we will extend this evaluation framework to additional domains and language pairs and include more models to not only capture the understanding of the models capabilities and limitations but to also systematically improve the mqm dimensions so that to ensure it remains a relevant, accurate, and actionable metric for such models.