Glossary functionality in commercial machine translation: does it help? A first step to identify best practices for a language service provider (2021)

Abstract: Recently, a number of commercial Machine Translation (MT) providers have started to offer glossary features allowing users to enforce terminology into the output of a generic model. However, to the best of our knowledge it is not clear how such features would impact terminology accuracy and the overall quality of the output. The present contribution aims at providing a first insight into the performance of the glossary-enhanced generic models offered by four providers. Our tests involve two different domains and language pairs, i.e. Sportswear En–Fr and Industrial Equipment De–En. The output of each generic model and of the glossaryenhanced one will be evaluated relying on Translation Error Rate (TER) to take into account the overall output quality and on accuracy to assess the compliance with the glossary. This is followed by a manual evaluation. The present contribution mainly focuses on understanding how these glossary features can be fruitfully exploited by language service providers (LSPs), especially in a scenario in which a customer glossary is already available and is added to the generic model as is.

## introduction correctly translating terminology is one of the main challenges in translation, and this is also true for machine translation (mt). a first approach to achieve this goal lies in data preparation and possibly appropriate training algorithms. however, there are cases in which data is not available or training a model is not an option. a number of research works have explored ways to combine a bilingual glossary with an mt model at run-time, enforcing specific terminology in the output, e.g. @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite . the proposed approaches range from simple posttranslation replacement, to constrained decoding, down to methods that allow for soft constraints and are able to generate inflected forms of glossary terms - @xcite improved by @xcite . such recent breakthroughs might not have made it yet to commercial implementation. nevertheless, a number of commercial mt providers have started to offer features allowing users to enhance a generic mt model by leveraging a bilingual glossary. 1 while language service providers (lsps) have to rely on those solutions, "the commercial providers usually leave us in the dark about the technology that is used for the implementation of that feature" @xcite . users may expect the addition of a domain-specific glossary to a generic mt model to bring improvements both in terminology translation and, as a result, in the overall output quality. the present contribution aims at understanding if the glossary features offered by some of the main mt providers are meeting such expectations. more specifically, being a relevant scenario for lsps, we aim at understanding if a glossary extracted from a customer termbase can be leveraged as is, i.e. all experiments will be carried out using the glossaries without any preliminary cleaning up. we will further refer to this use case as naive use of glossary. four different mt providers will be tested in the sportswear (en-fr) and in the industrial equipment (de-en) domains, comparing their performance when the glossary feature is switched on and when it is not. more in detail, the impact of the glossary feature on terminology translation will be assessed by checking the extent to which the mt output complies with the glossary entries. a first evaluation will follow strict parameters, i.e. glossary term matching is case-sensitive and happens on a token level. we will refer to this evaluation as exact match. in a second evaluation (henceforth loose match), we aim at finding any terminology improvement by matching terms on a lemma level and without considering differences in casing. the effect of the glossary on the overall output quality will be measured with translation error rate (ter) @xcite . based on term matching and on ter, we will then categorize each sentence based on the terminological and/or qualitative improvements (if any). to conclude, a manual evaluation will provide a more detailed overview on the glossary impact on the sentence. the aim of the contribution is to start addressing the needs for best practices across the translation industry for the use of glossaries to improve mt output. given the availability of glossary features, how can we leverage pre-existing glossaries? the remainder of the present contribution is structured as follows. in sect. 2, a description of the experimental setup will be provided, including descriptions of the mt providers, the data sets, the metrics and the evaluation methods. the following section (sect. 3) will present the results obtained with the naive use of glossary approach. first we will focus on each provider's behavior on the whole data sets (sect. 3.1), then a sentence level analysis is carried out (sect. 3.2), and finally we will present the results of a manual annotation (3.3). this is followed by a discussion of the results obtained (sect. 4). ## conclusion and future work the experiments described in the previous sections illustrate how the naive use of a glossary may not always provide the expected outcome, i.e. a better terminological compliance together with an overall improved output quality. results depend on the implementation of the glossary feature by the mt provider (how entries are matched and enforced on the target side), on the language pair and on the glossary itself. regarding the differences between providers, those that are able to handle morphology (provider 3) have shown to produce more sentences where terminology improvements result in a better overall quality. most implementations seem to induce a number of undesirable sideeffects on casing, morphology, word order. moreover, some limitations remain for all providers tested. for example, none of them (including provider 3) is able to match glossary source terms when these occur in a compound term (e.g. matching the german term batterie if the source text contains batterietyp). this would impact all agglutinative languages. besides the specifications of the glossary features, we saw that some glossary entries brought a lower translation quality, which raises questions about the quality of the glossary itself (see example c in starting from the assumption that a customer glossary as is does not comply with some of the specifications set by the providers (see sect. 2.1), and focusing on a scenario in which we already chose which provider to use, how could we turn a preexisting termbase into an mtcompatible glossary? a manual revision of the whole glossary may be time-consuming and might not solve all issues. as mentioned by @xcite , we cannot expect the user to provide for each entry all casing forms, and even less so all inflected forms. automatic pos tagging could help identifying non-inflective entries, but will be prone to errors. on the one hand, in order to adapt to the currently available technology, lsps may have to define best practices. in future work, we intend to run similar tests with subsets of the client glossaries containing only entries that are compliant with the mt providers specifications. such tests would involve the assessment of different procedures and tools to clean up glossaries. besides being able to discard entries that are not relevant, a further step would be that of enhancing the glossary by identifying new terms that, if added to the entries, would bring further benefits to the output quality. on the other hand, the results of the recent research endeavours in the filed of terminology and mt are expected to build momentum for new implementations in commercial solutions, which should narrow the gap between what is currently offered by mt providers and what lsps are expecting.