Investigating automatic and manual filtering methods to produce MT -ready glossaries from existing ones (2022)

Abstract: Commercial Machine Translation (MT) providers offer functionalities that allow users to leverage bilingual glossaries. This poses the question of how to turn glossaries that were intended to be used by a human translator into MT-ready ones, removing entries that could harm the MT output. We present two automatic filtering approaches - one based on rules and the second one relying on a translation memory - and a manual filtering procedure carried out by a linguist. The resulting glossaries are added to the MT model. The outputs are compared against a baseline where no glossary is used and an output produced using the original glossary. The present work aims at investigating if any of these filtering methods can bring a higher terminology accuracy without negative effects on the overall quality. Results are measured with terminology accuracy and Translation Edit Rate. We test our filters on two language pairs, En-Fr and De-En. Results show that some of the automatically filtered glossaries improve the output when compared to the baseline, and they may help reach a better balance between accuracy and overall quality, replacing the costly manual process without quality loss.

## introduction the ability to correctly and consistently translate domain-specific or customer-specific terminology is key in the field of translation. to accommodate for this need, machine translation (mt) providers have started to offer terminology features that enforce glossary entries at runtime. 1 the availability of such features can be particularly advantageous for language service providers (lsps), giving them an opportunity to offer a terminology accurate mt output in a scenario in which training a model from scratch is not an option. however, such glossaries were created to be used by human translators, relying on their ability to, e.g., disambiguate terms before inserting them in the target text. also, glossaries are often created by customers without the help of terminologists. as a result, they might not be ready to be used by mt, since they might contain entries that harm the output quality @xcite @xcite . the creation of a pipeline to clean glossaries can help mt users leverage their terminology data base. the pipeline can be based on a manual intervention, which can be time-consuming, or rely on an automatic procedure. either way, two operations may be involved, i.e. removing entries that are not helpful and/or editing them. automatically editing entries is not a trivial task. for example, automatically editing term entries where several term alternatives are separated by slashes poses the question of which alternative(s) to keep. also, in some cases the slash is used to separate parts of a compound or of a multi-word term (e.g. the german term "abluft-/motorfilter" should be split into "abluftfilter" and "motorfilter"). editing terms with parentheses is not trivial either. in some cases what is inside the parentheses is part of the term, e.g. the german term "länge base" translated as "length (base)". in other cases it is not, and it should be removed, e.g. "kühlung (z. b. von notebooks)" translated as "cooling", where the content of the parentheses provide context for the term. for these reasons, we will rather focus on filtering out such invalid entries (more example provided in sect. 3.3). in this paper we present procedures to filter glossaries automatically and manually. we investigate the results each glossary yields in terms of terminology accuracy and overall output quality -as measured by automatic metrics -when it is leveraged by the glossary feature of commercial neural mt (nmt) providers. our main contribution is to investigate if any of these filtering methods brings improvements to terminology accuracy with respect to the baseline, without worsening the overall quality compared to the output where the whole glossary is used. ideally, a better terminology accuracy should bring a higher overall quality, but since not much is known about how mt providers implement their glossary feature, we also want to check if this feature introduces side effects. the results obtained with the filtered glossaries are compared to those obtained when no glossary is used and when the original one is applied. two automatic glossary cleaning techniques are presented (see sect. 3.3). one is based on rules to remove noisy entries. the second one also leverages a translation memory (tm) to remove entries that are not used consistently in the translated contents. both filtering techniques are applied to two use cases, i.e. sport equipment english-french and electrical devices german-english. two providers are tested and their performance is evaluated based on terminology accuracy and overall output quality (see sects. 3.2 and 3.5). some sentences are then manually inspected to highlight interesting patterns in the outputs. the remainder of the paper is structured as follows. section 2 provides a brief overview of the literature in the field of terminology and nmt. the experimental setup (data sets, mt providers, evaluation method, filtering methods and experiments carried out) is outlined in section 3 and its subsections, and the results are presented in section 4. section 4.3 offers a review of some examples. results are then discussed in section 5, together with suggestions on future work. ## background several different approaches have been developed to enforce glossary terms in the nmt output. a growing interest in this field is testified by the first shared task on machine translation using terminologies in the framework of wmt 2021 @xcite . the methods developed so far can be broadly grouped in two categories. some of them are based on the idea of injecting terms from a glossary into the mt output as constraints posed at decoding time @xcite @xcite . other works build on the idea of adding soft constraints by annotating the source side of the training data @xcite @xcite @xcite . commercial mt providers do not disclose how their glossary feature is implemented, thus we do not now if they apply one of the approaches mentioned so far, and little work has investigated the performance of commercial models when enhanced by a glossary. guerrero (2020) compared the work of translators post-editing the output with and without the glossary. @xcite have investigated how the performance of a number of mt models changes when a preexisting glossary is added. both works conclude that pre-existing glossaries should be filtered before being used for mt. the need of preparing glossaries so that they are mt-ready is also underlined by @xcite . however, to the best of our knowledge, the paper by @xcite is the only one to have compared the impact of different glossary filtering approaches on the mt output. in their work, noisy and inconsistent entries are automatically filtered ## conclusions and future work in this paper, we used various approaches to filter pre-existing glossaries and tested their usefulness in improving the terminology accuracy of an mt output without deteriorating the overall quality. the results show that using a filtered glossary may produce a better accuracy with similar or improved overall quality when compared to a baseline where no glossary is used. in several cases a filtered glossary led to a better ter than the whole glossary, which suggests that filtering removes matches from terms that are harmful for the overall quality. on the other hand, results show that using a whole glossary can be beneficial. the whole glossary usually outperformed the filtered ones in terms of accuracy -which is expected given the larger size of the former -and, especially in the case of sportswear en-fr, the ter improvements brought by filtering were rather small. in general, filtering -and in particular the tm-based automatic filter -helped find an acceptable balance between a higher accuracy and a good over-all quality. given the current experimental stage of the filtering tool, such results can be seen as promising. however, improvements to the filtering method and new tests are needed to check if filtering can bring larger quality improvements over the whole glossary, thus making the filtering effort worthwhile. the results in both use cases suggest that aiming at the highest possible accuracy may not always be the best choice in terms of quality. there appears to be an accuracy cut-off beyond which overall quality suffers. in the case of electrical devices, this could be due to the fact that the terminology is quite general -indeed, the baseline is already handling the majority of the terms correctly. the analysis in sect. 4.2 revealed that the tm-based filtering method with the most restrictive threshold introduced only minor changes with respect to the baseline output, reducing the number of sentences where ter was regressing. this behavior may be preferable to using a larger glossary, which can negatively impact more sentences, especially when the baseline is performing well in terms of accuracy and overall quality. the baseline in sportswear en-fr is struggling with terminology accuracy, indicating that terminology is highly domain/customer-specific. in terms of ter, two automatic filters outperform the whole glossary, whereas the manually filtered glossary achieves the highest accuracy, closely followed by the whole and rule-based glossaries. applying a strict filter does not improve the quality of these contents. when compared to the baseline, we discovered that each glossary affects at least 70% of the sentences (sect. 4.2). however, we still see an accuracy cut-off around 70% (see an interesting conclusion we can draw from our experiments is that a glossary filtered by a linguist according to task-specific guidelines does not necessarily bring relevant improvements over an automatically filtered glossary. in particular, the tm-based filter always outperforms the manual one in terms of ter score. the rule-based filter outperforms the manually filtered glossary for sportswear en-fr in terms of ter, and achieves a slightly lower accuracy score. given the high costs of manually filtering a glossary, this can be considered a relevant outcome, especially for lsps. in some cases, even for a linguist expert of the content type, it can be difficult to distinguish a generic term from a specific one, which is one of the key actions to take when filtering a glossary for mt. although results suggest that using a tm to identify terms that are not highly specific to one domain can be effective, we plan to test more accurate solutions to this problem, such as the use of inverse document frequency (idf) @xcite or word-alignment. in bergmanis and pinnis (2021a), both methods were tested for glossary filtering. we anticipate that an improved ability to filter out generic terms will be especially helpful in use cases such as the one of electrical devices de-en. the rule-based filter, which requires no bilingual data other than the glossary, has one of the highest accuracy and one of the best ter scores in sportswear en-fr. this result is especially relevant in cases where a glossary must be filtered but bilingual data are either not available or their quantity is limited. examples shown in sect. 4.3 suggested that there can be several reasons for quality improvements or regressions when terminology is added to the output. sometimes the output changes even if no term was matched in the sentence, which is probably due to the specific implementation of the glossary feature. to gain a better understanding of this, we plan to carry out in-depth manual analyses of the outputs produced by the baseline, by the whole glossary and by the filtered ones. the present contribution focused on term filtering. however, the ability to edit terms that can be improved may yield better results. in the future, we will concentrate on this, beginning with cases where terms can be easily improved using automatic editing rules. editing terms without the assistance of a linguist can be a difficult task at times. we therefore intend to conduct experiments in which the results of the automatic filters are provided to a linguist as an aid to help them perform their task. we anticipate that this will also help linguists in their decision making process, e.g. to determine when terms are generic. being able to see that a term is translated inconsistently in the tm, for example, can lead to a better decision as to label the term as generic/detrimental or not.