Constrained Decoding for Technical Term Retention in E nglish- H indi MT (2021)

Abstract: Technical terms may require special handling when the target audience is bilingual, depending on the cultural and educational norms of the society in question. In particular, certain translation scenarios may require “term retention” i.e. preserving of the source language technical terms in the target language output to produce a fluent and comprehensible code-switched sentence. We show that a standard transformer-based machine translation model can be adapted easily to perform this task with little or no damage to the general quality of its output. We present an English-to-Hindi model that is trained to obey a “retain” signal, i.e. it can perform the required code-mixing on a list of terms, possibly unseen, provided at runtime. We perform automatic evaluation using BLEU as well as F1 metrics on the list of retained terms; we also collect manual judgments on the quality of the output sentences.

## introduction and motivation it is common for bilingual or multilingual speakers to borrow technical terms from other, usually high resource, languages into their native language. this may be for several reasons, e.g. the technical term in the high resource language may be much more popular and therefore better understood, or the required term may simply not exist in the language in question. this is very common, for example, in indian languages, where the language of education is frequently different from the regional native language. we can imagine, therefore, a scenario which requires the automatic translation of text or speech, with the constraint that a given list of english domain words appear untranslated in the hindi output. essentially, this can be seen as a special case of constrained decoding with a given source target terminology. we make the assumption that *equal contribution by these authors. the user knows the terms to be retained at run time, and can provide this information to the system before translating the sentence. foot_0 ## previous work the idea of constrained decoding has been recognized as useful in several works @xcite . usually, the constraints are in the form of a terminology list, as in the above works. to our knowledge, this is the first study on combining this concept with introducing codeswitching foot_1 (cs) into the output for a multilingual educational or technical setting. ## approach we set up an endtoend supervised learning scenario aimed at teaching the model to perform term retention. the basic idea is to train a machine translation model to obey a "signal", that we can then provide at run time on selected words. it is easy to see that such a model (the "tagged" model) would be independent of domain and could in theory perform term retention on any term for which the signal was provided. we also train a simple baseline for comparison; the baseline model sees the same training data as the tagged model, but does not receive any signal that would be highlighting the terms to retain. therefore, given input at run time, it must rely on past exposure on the specific terms and their (non)translation to perform term retention. we provide the mentioned signal in the form of tags i.e. <@xmath0 source sentence: you need to install these python libraries. term list: python, libraries input to the system: you need to install @xmath1: आपको इन python libraries को ा पत करना चा हय figure 1: example input to and desired output from the system dataset total sentences sentences with cs train 250 700 123 274 development 10 247 5 064 test seen 5 000 5 000 test unseen 768 768 test w/o cs 500 0 for "retained english word") to indicate that the enclosed term shall be retained during the translation, see figure 1 . this approach can be used in any type of transformerbased translation system and therefore can be implemented with little to no effort in current systems. ## synthetic data creation we used hindencorp 0.5 @xcite data set and we split it into multiple parts as seen in ## model we used a transformerbased model @xcite with vocabulary size of 32000 tokens and with hyperparameters as described in the university of edinburgh's neural mt systems for wmt17 @xcite for both of our models. we used marianmt framework @xcite to train the models; we let the model train until the bleu score @xcite did not improve on the development set for 5 epochs. we then selected the model with the highest bleu score as the model used for later experiments. the change of bleu score on the ## automatic evaluation there are two components of model performance: for the former, we calculate precision, recall, and f1 over the gold retained set of words and the set of retained terms in the output. our evaluation script compares the system output with the list of terms that should be untranslated in the given sentence. precision is the ratio of term occurrences in the system output that were anticipated in the reference, out of all produced latin terms. recall is the ratio of term occurrences produced by the system out of all term occurrences anticipated by the reference. for the latter, we use bleu score. the bleu scores on test sets can be seen in on the other hand on the unseen dataset, the tagged model receives explicit information to retain the term and therefore outperforms the baseline model. results for the retention metric can be seen in it can be seen that the tagged approach outperforms the baseline model on both the unseen and seen test set, demonstrating that it indeed learns to obey the provided signal, instead of simply relying on previous exposure as the baseline does. foot_5 ## manual evaluation we also performed a manual evaluation to complement the bleu score. this evaluation was solely for the purpose of judging the quality of the final output regardless of whether the model managed to retain the required words or not. ## conclusion the task of applying terminology constraints while dealing with codeswitched text seems especially important in current multilingual educational and other settings. we present a simple technique that can adapt a vanilla transformerbased mt tool for performing this task, by synthesizing training data that exhibits term retention. we demonstrate that our model performs well on unseen terminology, and that its general translation quality is not damaged. future research should consider using codeswitched parallel corpora, either for training or finetuning, in order to teach the models the various nuances of natural human codemixing.