The L eg IST yr Test Set: Investigating Off-the-Shelf Instruction-Tuned LLM s for Terminology-Constrained Translation in a Low-Resource Language Variety (2025)

Abstract: We investigate the effect of terminology injection for terminology-constrained translation in a low-resource language variety, with a particular focus on off-the-shelf instruction-tuned Large Language Models (LLMs). We compare a total of 9 models: 4 instruction-tuned LLMs from the Tower and EuroLLM suites, which have been specifically trained for translation-related tasks; 2 generic open-weight LLMs (LLaMA-8B and Mistral-7B); 3 Neural Machine Translation (NMT) systems (an adapted version of MarianMT and ModernMT with and without the glossary function). To this end, we release LegISTyr, a manually curated test set of 2,000 Italian sentences from the legal domain, paired with source Italian terms and target terms in the South Tyrolean standard variety of German. We select only real-world sources and design constraints on length, syntactic clarity, and referential coherence to ensure high quality. LegISTyr includes a homonym subset, which challenges systems on the selection of the correct homonym where sense disambiguation is deducible from the context. Results show that while generic LLMs achieve the highest raw term insertion rates (approximately 64%), translation-specialized LLMs deliver superior fluency (∆ COMET up to 0.04), reduce incorrect homonym selection by half, and generate more controllable output. We posit that models trained on translation-related data are better able to focus on source-side information, producing more coherent translations.

## introduction while neural machine translation (nmt) adaptation has demonstrated benefits from incorporating domain-specific terms @xcite , it has yet to ensure consistent and unambiguous terminology enforcement. the delicate trade-off between the continuous expansion of parallel training corpora and the enforcement of lexical choices without hampering fluency further complicates cross-lingual alignment of terminologically relevant tokens @xcite @xcite . this failure has even raised questions on the cost-effectiveness of maintaining termbases for mt purposes @xcite . terminology compliance has attracted notable interest and lead to the organization of shared mt tasks @xcite @xcite . terminology compliance is a crucial quality aspect in high-stakes domains, such as the legal domain. terminology mistakes in legal translation can have serious consequences (mattila 2018), including legal disputes and infringement of basic linguistic and human rights (e.g., through incorrect use of critical terminology during interpretation in criminal trials). furthermore, every legal system has its own specific set of rules and conceptual structures. legal terminology expresses such specificities and is therefore always bound to a specific legal system @xcite . this system-boundness of legal terminology often results in conceptual incongruency between legal systems, even between legal systems sharing the same language. consequently, correct terminology usage in languages with more than one recognized legal variety like arabic, english, spanish etc., pose notable translation challenges. in addition, the quality of mt in the legal domain hinges not so much on the language combination as on the legal subdomain @xcite . our experiments focus on south tyrolean ger-man, a minor standard variety of german spoken in northern italy that is used by the local public administration bodies and on legal terms from a range of different legal subdomains. researchers have addressed terminology enforcement using both nmt systems and large language models (llms). in 2024, two new llm suites specifically trained on translationrelated tasks (tt llms) were released to the public (tower and eurollm suites); yet to our knowledge their capabilities have not been adequately investigated thus far. to bridge the gap, we evaluate the instruction-following performances of these open models in terminology injection, comparing their term accuracy rate and overall translation quality against general-purpose llms as well as to adapted nmt models and their baseline. we also assess which models produce the most structured and clean outputs for easier downstream processing. to this end, we curate legistyr 1 , a test set comprising over 2,000 sentences from the legal domain in italian. each instance is annotated with both italian source and south tyrolean target terms, plus variants used in other germanspeaking legal systems whenever available. we select the sentences from termbase contexts and other real-world sources while enforcing constraints on length, syntactic clarity, and referential coherence. the test set covers the usage of each term in multiple contexts. it also features a subset on the challenge of disambiguation between homonyms where the correct sense is deducible from the domain or context of use. ## experimental target and limitations in light of these insights, we evaluate off-the-shelf models that have undergone both pre-training (eu-rollm) and supervised fine-tuning (tower) on translation tasks without additional adaptation to custom data. however, by deliberately excluding any modification to the model's hidden states representation, we restrict our investigation to the effect of terminology injection -particularly for terms likely under-represented during pre-training -on output fluency and decoding behavior under soft constraint conditions. this design choice may limit the performance upper bound for producing fully coherent south tyrolean text, as some of the most effective approaches for handling language varieties often involve large-scale pre-training or continued training @xcite , for machine translation @xcite ) and evaluation tasks @xcite ) alike. ## results as it can be appreciated in in addition, the magnitude of the comet score differential is non-trivial. according to @xcite 's tool foot_4 , a 2-point delta in comet-kiwi corresponds heuristically to a 95% agreement rate with expert human judgments. this result reinforces the hypothesis that tt llms prioritize fluent output more effectively than other model paradigms. homonym set results point to this hypothesis too. the difficulty of detecting the correct domain lowers overall term enforcement rates, with the most pronounced decline observed in generic llms, hence narrowing their performance lead. notably, generic llms exhibit a strong proneness to inserting any of the provided terms, even when contextual cues suggest otherwise. this results in more than twice as many incorrect homonym selections compared to tt llms -with critical consequences on meaning comprehension. it could be argued that this divergence may be attributable to the different objectives of the respective tuning procedures. specific training of tt llms on translation-specific tasks enables the models to learn source-target text alignment patterns, allowing its attention mechanisms to more effectively focus on source-side information. the higher learned attention to highly domain-relevant context in the source sentence reduces the likelihood of incorrect term selection. in contrast, instruction-tuning in generic llms often lacks explicit exposure to the parallel sentence structure typical of translation tasks. therefore, contextual attention weights may end up excessively biased towards less important parts of the prompt. additional insights emerge from the generation of terms belonging to other major varieties of legal german (ols) or other valid alternatives for south tyrol (ost), in the cases where these were measurable. the higher selection rate of ols variants by tt llms may signal that prompting embeddings cannot alone redress the token probability imbalance in the output distribution, which is presumably outmatched by major variant occurrences. however, we note that eurollm-9b achieves the highest ost selection rate among llms, despite these acceptable terms not being explicitly elicited in the prompt. given its pre-training from scratch on european language corpora, this result suggests that including regionallyfocused pre-training data may allow to better encode minor variant lexical forms. format-wise, tower and eurollm have proven to be most stable models, with all translations enclosed in the desired delimiters and no signs of verbosity. while llama 8b struggles to follow the requested format, spurious text remains a rare occurrence. eventually, mistral 7b shows a considerable amount of noise in the output, most notably in verbosity proneness, output in english language and repetition of parts of the prompt. ## discussion the process of integrating llms into translation workflows is still at an early stage, with a major divide opening between the use of generalpurpose models -leveraging scale-driven emergent capabilities -as opposed to models finetuned through task-specific, specialized training. while we register a higher overall success rate for generic llms, we also suggest that models trained specifically on translation data show greater promise to attend to the peculiar challenges of context-sensitive translation. on the technical side, future research should continue to explore strategies for conditioning more reliable terminology generation at decoding time. another promising direction involves exploiting the contextual metadata available in linguistic linked data (lld) resources (especially multilingual examples of use of term entries) for continued pre-training or task-adaptive fine-tuning. in this respect, while not having used terminology data as lld for terminology injection in this first stage, we consider it a necessary follow-up step. terminological data in machine-readable formats are likely to become crucial resources for terminology injection in future. we have shown that terminological data can be used to partly make up for the lack of training data in minor language varieties. where training data from major varieties risks overriding language use in minor varieties and even leading to critical mistakes in high-stakes domains like the legal domain, available terminological resources could help to partially fill the gap. the training data used for nmt systems and llm models are skewed towards high-resource world languages and language combinations containing english. further research is needed into non-english language combinations, which are a routine part of the work of many bodies with legislative, administrative and judicial powers that affect citizens daily lives. to name some examples in europe, there is translation from german into a minor legal variety of slovene in austria for the slovene-speaking minority in carinthia, translation from croatian into a minor legal variety of italian in croatia, translation from finnish into a minor variety of swedish in finland. all these and other minority communities would profit from efficient strategies for adapting machine translation to their specific varieties and/or from terminology injection to fine-tune translation results. ## conclusion and limitations we are aware of the limitations of our first experiments. although proprietary llms set the current upper bound for performance, the opacity surrounding their training data precludes any assessment of whether results stem from explicit exposure to translation tasks. this constraint limits our ability to isolate and attribute observed advantages to translation-specific (pre-)training, thereby confounding the validation of the research hypothesis. from a linguistic perspective, our data illus-trates phenomena that apply to the south tyrolean standard variety of german and to a non-english language combination. to assess whether similar results would apply, for example, to the standard variety used by the german-speaking community in belgium and to translation from dutch or french to another variety of german, we would need targeted studies. the same holds true for other minor (legal) varieties of major languages (e.g., swiss french, chilean spanish etc.) and the language combinations that might be predominant in other minor or minority variety contexts. a further limitation consists in the lack of qualitative analyses that could shed better light on the results, which we are planning for the future.