Terminology in Neural Machine Translation: A Case Study of the C anadian H ansard (2023)

Abstract: Incorporating terminology into a neural machine translation (NMT) system is a feature of interest for many users of machine translation. In this case study of English-French Canadian Parliamentary text, we examine the performance of standard NMT systems at handling terminology and consider the tradeoffs between potential performance improvements and the efforts required to maintain terminological resources specifically for NMT.

## introduction incorporating data from a specialized or particular lexicon is a commonly-desired property of neural machine translation (nmt) systems used in computer-aided translation settings. a number of approaches have been proposed for this, including modifications to decoding, training systems for special behavior, and training with external lexicons. results vary, highlighting the fact that they navigate a difficult compromise between imposing specific lexical choices on the decoder, and interfering as little as possible with its behavior (yvon and rauf, 2020). parallel to this, lexical resources developed by terminologists and translators are not necessarily designed and formatted with nmt requirements in mind, and not all terms they contain naturally lend themselves to incorporation: for example, it may be difficult to process terms with many morphological variants or terms whose translation depends on the context. extracting these resources' content for nmt and main-taining the two resources in sync may pose practical challenges. in light of this, it is reasonable to ask when, how, and whether it is worth implementing these methods in a real-life, practical setting. here, we use the scenario of canadian parliamentary translation as a case study to examine questions about terminology and machine translation performance. the data we use consists of transcriptions and translations of speech in the canadian house of commons (the proceedings, or hansard), with most speech originally in english (then translated to french), a much smaller part spoken in french (then translated to english), and a very small fraction in other languages. parliamentary translators have access to a document that provides guidance on terminology, from which we have manually extracted word and phrase pairs. we are interested in the following questions: with this data, we begin by examining just how "usable" the terminology actually is for nmt incorporation, and how consistently it is used in human translations. we then compare how an nmt system (without any special terminology handling) performs on these terms, through both automatic and manual evaluations. in our analysis, we highlight the following considerations for researchers and users of nmt interested in handling terminology: in this particular use case, we find that the terminology bank is appropriately designed for translator use rather than optimized for machine translation, the terms are relatively infrequent in the corpus, there is a mix of how consistent the term translations should be (even in high-quality human translations), and the nmt system performs reasonably well on the terms that are most unambiguous. for these reasons, there would be a relatively high cost in terms of human time (to produce and keep current an additional machinereadable version of the term bank) to handle terminology for a relatively small amount of potential improvement. depending on translator preferences and how much of a pain point terminology errors are, there may be appropriate alternatives, such as flagging potential terminology errors (though these also come with their own costs). we also discuss how the relative costs and payoffs may differ in other settings. ## analysis we begin by examining the frequency with which the terms of the pft ef appear in the text of the hansard. to handle issues of tokenization, we begin with raw/detokenized text and use nltk's word_tokenize (bird and loper, 2004) to tokenize the pft ef terms, the hansard source and reference, and the (detokenized) mt output. prior to tokenization, we perform apostrophe standardization, @xcite though this impacts only a small number of segments. in this analysis, we restrict ourselves to the data where the human translation direction matches the machine translation direction. there are 605 unique english terms in the pft ef and 600 unique french terms (599 after apostrophe standardization). the pft ef is directional and intended for english to french translation, so it is unambiguous in the english to french direction, and has some minor ambiguities in the french to english direction. this means that the most appropriate analysis is in the english-french direction, though we still include some analyses in the french-english direction (with caveats) in figure 1 shows the distribution of pft ef term occurrences in en-fr training data. five appear more than 10,000 times: climate change (14586), liberal party (16537), first nations (26702), conservatives (53883), and budget (67943). we focus our attention on the english-french portion of the ft-test data set. of the 7235 english text segments in the sample, 595 (8.2%) contain at least one (lowercase) match to one of the pft ef terms. as some segments contain more than one source term, there are a total of 694 instances of source terms in that data set. for 594 of corpus % of terms src. # % ref. % mt en-fr train 89.4% (541) 376680 78.0% -ft-test 13.2% (80) 694 85.4% 87.2% fr-en train 66.3% (397) 120845 81.0% -ft-test 5.0% (30) 230 75.7% 76.1% there are various reasons why a prescribed term might not appear in a translation, including @xcite again, when a segment contains multiple matches of the given source term, we verify that the reference translation contains at least as many occurrences of the corresponding target term as of the source term. a (human or machine) translation error. @xcite however, in many cases, a missing term does not imply an error. for example, a translation might have been formulated in such a way that the entity or notion to which the term refers is referred to with a paraphrase or a pronoun in the translation. in other cases, the context may render the term redundant or superfluous. sometimes a term occurrence is actually part of a larger term within which it should be translated differently; for example, while the prescribed french translation for climate change is changement climatique, the official french name for the "intergovernmental panel on climate change" (ipcc) is "groupe d'experts intergouvernemental sur l'évolution du climat" (giec). to better understand how humans and mt behave with regard to pft ef terms, we manually annotated a subset of the ft-test data set. we collected all ft-test segments that matched one or more source terms from the pft ef , but for which either the reference or the machine translation did not contain at least one occurrence of the prescribed translation for each matching source term. in all, there are 123 such source segments, each with two translations: 28 for which the reference translation uses the prescribed term but the mt doesn't; 40 for which the mt uses the prescribed term but the reference translation doesn't; and 55 for which both translations are missing a prescribed term. in order to get a better balance between the translations that use the prescribed terms and those that don't, we added 49 segments, randomly selected from ft-test that match both source and target terms. in all, our annotation set contains 172 distinct segments, containing 185 source term matches. for each of the 185 term instances, the reference and the machine translations were analyzed to determine whether the matched source term was correctly translated in the context. the question that annotators were asked was: "is the term highlighted in the source rendered correctly in the 9 it is worth noting that parliamentary translators are not always to blame for terminology errors found in the reference translations. in some cases, the hansard will contain excerpts from pre-existing documents, for which an official translation already exists. translators are not permitted to fix errors in these pre-existing translations. in other situations, the fault may lie with the speaker in the house of commons which may have used an incorrect or inexact term; it is then the translator's duty to attempt to fix this, by translating the speaker's intent rather than their words. translation?" foot_7 a first-pass annotation was performed by two of the authors. @xcite each annotator assigned one of three tags to each translation: correct, incorrect, or unsure. the two annotators then jointly produced consensus labels by reconciling their differences together. all term translations with a unsure label that remained after consensus were then submitted to a second-pass annotation (43 of the 370 translations). this second pass was done through individual interviews with three volunteer translators from the parliamentary service. @xcite from these judgments, we assign the majority label. @xcite reference mt target term appears: yes no yes no ## related work we now briefly discuss a number of approaches that have been applied to the problem of handling fixed terms, including modifications to decoding, training systems for special behavior, and training with external lexicons. for a much more extensive review of approaches to lexicons and terminology resources in nmt, see yvon and rauf (2020). these approaches can be applied independently or combined, and each has various strengths and weaknesses. decoding modifications, such as lexically constrained decoding (hokamp and liu, 2017; post and vilar, 2018) typically come with strong guarantees (i.e., that the desired term will appear in the output), do not require the lexicon to be known in advance, and do not necessarily require any modification to training procedures. downsides to these include that they may be overly strict (e.g., failing to inflect forms) and that forcing low probability output can harm overall translation quality ("reference aversion"). there is also no guarantee that the tokens are in the correct location, are produced by translating the correct source token, or are not concatenated with adjacent tokens. hasler et al. (2018) seek to improve terminology placement in constrained decoding by incorporating alignment (via attention) to tie the relevant source tokens to the desired target token output. susanto et al. (2020) modify the beam search procedure to enforce translation of words (as specified in xml-style input) or to perform look-ahead to ensure they are generated. training for special behavior, through placeholders (post et al., 2019) or factors (dinu et al., 2019) does not require a fixed lexicon in advance, but it does not offer the same strong guarantees of producing fixed terms. however, it sometimes successfully results in correctly inflected terms. bergmanis and pinnis (2021) expand on dinu et al. (2019), specifically with the goal of better handling morphological variants. if a lexicon is fixed in advance, it can be incorporated into nmt training (arthur et al., 2016; nguyen and chiang, 2018), though this does not hold strong guarantees of lexicon production and does not generalize to new lexicon entries in the future. exel et al. (2020) compare the approaches in dinu et al. (2019) with constrained decoding, and find that in their use case, this training for specific behavior "offers a good trade-off for terminology enforcement in a production setting." they also note that even baseline systems had fairly high performance on translation quality, though term translation did lag behind the specialized systems. ## discussion and conclusions our analysis shows that mt is twice as likely as humans to commit terminology errors in the hansard, for terms in the pft ef : when the mt system does not produce the target term, its term translation is incorrect approximately 34% of the time, as compared to 17% reference translations in the same scenario (see in this work we focused on the kinds of "fixed" fixed terms that could be most easily incorporated into lexicon-based approaches to nmt fixed term augmentation. in our setting, this meant excluding close to half the terms from the translators' term bank (48%). in particular, we excluded terms that would almost always require significant inflection (e.g., verbs), though some approaches to handling fixed terms are capable of handling morphological variation and future work may wish to broaden the use of terms to more fully capture the kinds of term banks used by translators, as argued by bergmanis and pinnis (2021). unlike prior work that has dealt with fixed terms by enforcing terminology in the test sets (alam et al., 2021), we leave the parallel text as it is, but also examine cases where, even within our more constrained setting, fixed terms are not "fixed" in the strictest sense. we observed situations where they are fluently replaced with pronouns (to avoid repetition), where the term is translated differently as part of a larger phrase, and other such sources of variation. we note that this may be particular to this corpus and term bank; a corpus that is heavy on highly-technical terminology (e.g., chemistry, medicine) might have a greater proportion of terms that are truly fixed. thus we encourage researchers and users to check how "fixed" the terminology is in real text, even if only at the shallow automatic level. in light of this, and in a scenario such as ours, it seems reasonable to ask whether it is worth implementing any of the methods outlined in section 4. to cover only the terms we analyzed here, most approaches would be suitable. however, we note that the aide-mémoire documents are periodically updated, which would require retraining in the case of approaches that require a known and fixed terminology in advance. @xcite even though the nmt system made twice as many terminology errors than the reference text did (when the target term was not produced), its term translations were still judged to be adequate the majority of the time. this raises the question: if we enforced term translation, what would happen in those sentences? would the result be just as good, or might it produce less-fluent translations? as we did not perform manual evaluation of quality beyond the terms, this is not a question that our current data can answer. one simple alternative to consider is to automatically flag to the translator's attention those translations (human or machine) that do not match the pft ef term when the source segment does. however, it should be noted that this too has a cost, not so much in software development, but in maintenance of the lexical resources, which must then be encoded in machine-readable format. this may include expanding morphological variants, as well as keeping the machine-readable term bank up-todate. this would need to be weighed against the time spent correcting machine translation errors, as well as the potential inconvenience or trust loss due to flags that are false positives. the time needed to correct mt errors should be weighed against the time needed to maintain and update the resource specifically as a tool for the nmt system. @xcite note here that we are not putting in question the aidemémoire documents themselves. as pointed out earlier, these documents are rich in information. they serve an invaluable role for parliamentary translators in documenting terminological decisions and for training newcomer translators to the service. importantly, they are designed for translator use, allowing for information about context and ambiguity that is often skimmed over in work on "fixed" terms.