The MT @ BZ corpus: machine translation & legal language (2023)

Abstract: The paper reports on the creation, annotation and curation of the MT@BZ corpus, a bilingual (Italian–South Tyrolean German) corpus of machine-translated legal texts from the officially multilingual Province of Bolzano, Italy. It is the first human error-annotated corpus (using an adapted SCATE taxonomy) of machine-translated legal texts in this language combination that includes a lesser-used standard variety. The data of the project will be made available on GitHub and another repository. The output of the customized engine achieved notably better BLEU, TER and chrF2 scores than the baseline. Over 50% of the segments needed no human revision due to customization. The most frequent error categories were mistranslations and bilingual (legal) terminology errors. Our contribution brings fine-grained insights to Machine translation evaluation research, as it concerns a less common language combination, a lesser-used language variety and a societally relevant specialized domain. Such results are necessary to implement and inform the use of MT in institutional contexts of smaller language communities.

## introduction machine translation evaluation (mte) assesses the performance of machine translation (mt) systems. it can be human or automatic. while automatic metrics are quickly computed and offer an idea of how a system performs, human evaluation is timeconsuming and expensive but offers detailed insights into what machines get right or wrong when translating. human mte usually considers accuracy and fluency. accuracy measures "the extent to which the translation transfers the meaning of the source-language unit into the target", while fluency assesses "the extent to which the translation follows the rules and norms of the target language" @xcite . error classification and analysis may be considered a subtask of human mte. it requires a detailed error taxonomy and a group of annotators @xcite . in the past, different error taxonomies have been developed, but none was adapted or tested on the combination italian-south tyrolean german or on a lesser-used standard variety of a major european language. foot_0 a widely used error classification framework is the multidimensional quality metrics (mqm) @xcite , with a hierarchical list of categories and a flexible and customizable application that ensure different levels of granularity. despite its flexibility, in our project we opted for the scate taxonomy, as the possibility of linking target annotations to source spans helps interpret terminology issues, our main interest. besides, the availability of a ready-to-use annotation project with the scate taxonomy was an added value. despite the substantial improvements achieved thanks to neural technologies @xcite , mt still struggles with some language combinations in the legal domain @xcite , more than in other domains @xcite . this is due to the inherent complexity of legal discourse, with i) terminology coming from several fields, ii) a strong relation with general language (e.g., redefined words like 'hold'), iii) convoluted syntax and long sentences and iv) abundance of internal and external references @xcite @xcite . legal language is particularly difficult to translate @xcite and, consequently, to machine translate. the struggle becomes even more challenging when dealing with pluricentric languages, such as german, english or spanish. pluricentric languages are used in two or more countries with different official norms in grammar, orthography, and lexis @xcite . in addition, each country has a specific legal system, which is expressed by more or less diverging legal languages. this is the case of austria, germany, switzerland and south tyrol for german. in our study, we deal with south tyrol, where the legal system in force is the italian one and german is a co-official language at the local level. south tyrolean german is a standard variety of german, but no customized mt system has been developed for it so far. at the time of writing, freely accessible online mt systems have not implemented german varieties yet. in deepl foot_1 , two varieties are available only for english and portuguese, while none are available in google translate foot_2 . it is reasonable to assume that most texts used to train mt engines in german come from the european union and germany. their performance on south tyrolean german necessarily fails to consider typical local terminology and phraseology, as already proved by @xcite . against this background, we identified two major research gaps we aim to contribute to. to our knowledge, no corpus of machine translated legal texts has been annotated so far, nor does a corpus for the combination italian-south tyrolean german exist. the mt@bz corpus intends to: i) contribute to the evaluation of machine translated legal language; ii) set the basis for creating an mt system for south tyrolean public institutions. ## motivation the south tyrolean local administration is required to publish laws, decrees, circulars, communications, etc. in both italian and german (presidential decree 670/1972). this is done by translating them from either language. the task is usually carried out by civil servants, generally non-professional translators @xcite . they use also freely accessible online mt systems, which underperfom in the south tyrolean legal language (see section 1). to address the research gap related to customized mt for south tyrol, we considered an annotated corpus of mt errors a useful first step for the following reasons. first, no annotation scheme has been tested on the combination italian-german and no annotated corpus exists for this language pair. we consider it of utmost importance to assess the performance of mt systems on less common language combinations to identify language-dependent issues more clearly. furthermore, our research scenario deals with a lesser-used standard variety (south tyrolean german), for which the development of specific language technologies was hardly addressed so far. finally, legal language is an essential aspect of implementing linguistic human rights pertaining to language minorities, such as the right to understand the language within court proceedings @xcite . second, not many scholars working on mte focused on legal language. among those who did @xcite @xcite @xcite , only farzindar and lapalme assessed the quality of canadian court judgements translated between english and french with three human evaluators. however, they did not annotate mt mistakes. no fully annotated corpus of machine-translated legal texts has been created so far, to the best of our knowledge. third, a fine-grained error annotation could be used to advance research in the field of quality estimation (qe). the latest wmt shared task adopted mqm to produce the human gold standard for the task datasets @xcite because finegrained annotation schemas are more reliable for the metrics task @xcite . last, an annotated corpus is a valuable research output and may serve as input for further research. it sheds light on the mistakes of an mt system when translating legal texts in a given language pair, thus contributing to mt research on legal language. it can also represent useful input to further develop or fine-tune a customized mt system exploiting high-quality human, granular and refined analyses. ## data compiling the mt@bz corpus was compiled in late 2020 by downloading a set of provincial decrees from the local legal database lexbrowser in both italian and german. 5 we selected a range of decrees published from november 2020. in october 2020, contarino (2021) created a bilingual aligned corpus of texts from lexbrowser (lexb), which we used to train an mt system that translated the mt@bz corpus. it was therefore essential that the decrees of our corpus were not included in lexb. the aim of our test was to assess the performance of a customized mt system in a "real world" scenario. ## annotation we annotated our corpus to identify the more frequent error categories produced by a customized mt system when translating decrees in the language combination italian-south tyrolean german. this gave us a detailed summary of the major issues a neural mt system faces when dealing with legal discourse. among the many available, we selected the scate taxonomy @xcite 10 it is not possible to determine the source language for legal texts published in the multilingual setting of the province of bolzano: text drafting may occur in more than one language and extracts of published texts may be reused in either language. for this reason, we considered both versions of each decree either source or reference translation in the corresponding test settings. for several reasons. it has been used in a similar annotation campaign and, as such, allows for accuracy and fluency errors annotations. it allows to link accuracy errors in the target to relevant spans in the source language. it is provided with detailed guidelines. it is detailed but not to an unsustainable level. finally, it is easy to implement, as an annotation project carried out by the scate group was shared as a complete webanno project with the research community @xcite . ## data preparation and annotation workflow the annotation campaign was carried out between june and november 2022 with the help of four annotators with a degree in translation. two are italian native speakers, one is a native speaker of (south tyrolean) german, one is a balanced bilingual. all have at least a c1 level in their second language. one italian native speaker annotated only six texts. the three other annotators worked on both language directions due to a shortage of annotators. one is a master's student in translation, one has more than 5 years of experience and one has 20 years of experience in translation. two translators, one with 20+ years and one with 5+ years, were entrusted with the final curation step, which they performed on texts translated into their native language. ## conclusion we reported the creation, annotation and curation of the corpus mt@bz, a bilingual (italian-south tyrolean german) corpus of machine translated legal texts from the province of bolzano. to the best of our knowledge, this is the first annotated corpus of machine translated texts from the legal domain for a combination of languages that also includes a lesser-used standard language variety. it includes 52 decrees (26 in italian and the corresponding 26 in south tyrolean german) for an overall amount of 72,000 tokens. we selected and retrieved the texts from the institutional pages of the local administration of south tyrol and translated them with the help of the mmt engine plugged-in in the rws studio environment. a baseline translation was acquired with the default translation memory integrated in mmt, while a customized output came from the integration of a 230,000 segments trans-lation memory of bilingual legislation. the customized engine outperformed the baseline according to bleu, ter and chrf2 scores. we annotated translation errors on the customized machine translation outputs, using the scate taxonomy @xcite adapted to our language pair. three annotators annotated the entire corpus achieving a good level of agreement (iaa 0,74 -gamma-value). finally, we curated the corpus to produce a gold standard. we believe our contribution brings more finegrained insights to the field of machine translation evaluation, because we considered both a lessercommon language combination, a lesser-common language variety and a specialized domain, even if we focused exclusively on error classification and used only one mt system, which does not allow for generalisations. this kind of very granular evaluations seems necessary to integrate the use of mt in institutional contexts of smaller realities like south tyrol. we have shared 1 the corpus and the guidelines, as well as the project data, with the community to foster replication studies but also to encourage mt researchers to focus on lesserused languages in real life scenarios, such as public institutions in minority language communities.