Mitra: Improving Terminologically Constrained Translation Quality with Backtranslations and Flag Diacritics (2024)

Abstract: Terminologically constrained machine translation is a hot topic in the field of neural machine translation. One major way to categorize constrained translation methods is to divide them into “hard” constraints that are forced into the target language sentence using a special decoding algorithm, and “soft” constraints that are included in the input given to the model.We present a constrained translation pipeline that combines soft and hard constraints while being completely model-agnostic, i.e. our method can be used with any NMT or LLM model. In the “soft” part, we substitute the source language terms in the input sentence for the backtranslations of their target language equivalents. This causes the source sentence to be more similar to the intended translation, thus making it easier to translate for the model. In the “hard” part, we use a novel nondeterministic finite state transducer-based (NDFST) constraint recognition algorithm utilizing flag diacritics to force the model to use the desired target language terms.We test our model with both Finnish–English and English–Finnish real-world vocabularies. We find that our methods consistently improve the translation quality when compared to previous constrained decoding algorithms, while the improvement over unconstrained translations depends on the familiarity of the model over the subject vocabulary and the quality of the vocabulary.

## introduction in this paper, we present mitra, an end-to-end pipeline for terminology-constrained translation that combines a novel constrained beam search algorithm with backtranslation substitution. terminology-constrained machine translation is a popular topic in the field of machine translation, and has been a focus of several shared tasks in the wmt conference @xcite . in constrained translation, the system is given a lexicon, or a terminology, and it must use the words given in this terminology when translating sentences. while this was a trivial task in phrase-based statistical (cf. @xcite and rule-based machine translation systems (cf. @xcite , implementing it for neural systems has proved to be much more difficult due to their black-box nature. the existing methods can be divided into the so called "hard" and "soft" constraints. hard constraints use constrained decoding algorithms such as constrained beam-search @xcite , which first decides on the acceptable forms of constraints at the token level and then forces the decoder of an nmt system to abide by them. soft methods, on the other hand, use a neural network specifically trained for the purpose of constrained translation, and the constraints can be given to the encoder of the network as input (cf. @xcite . both of these methods have their own advantages: hard constraints can be enforced on any neural network without the need to train or fine-tune anything, while soft constraints are generally faster. typically, hard constraints only mandate that terms occur somewhere in the trans-lated sentence, while soft constraint approaches allow an explicit coupling of the source and target terms. implementing hard constraints for agglutinative languages has several difficulties. foremost, if a term has multiple possible inflected forms, and the correct form is not known beforehand, the constrained decoding algorithm must be given multiple alternative forms @xcite , a feature not widely supported by many algorithms such as @xcite ; @xcite . moreover, the system must be able to generate these alternative forms, requiring the use of languagespecific morphological generators, which might not be available for all languages. another problem of hard constraints is that the translation quality might be very poor if the machine translation model does not recognize the constrained words, a situation which in our experience is very common as terminologies often contain uncommon technical jargon, brand names, and other terms not appearing in the training data. in our specific case, we tested our method with a vocabulary provided by the finnish forest centre containing names of insects in finnish and english @xcite . many of the names have surprising translations: for example, a "violet tanbark beetle" is called "papintappaja" in finnish, which means "priest-slayer" if translated literally. if the translation model has not seen this term before, it cannot correctly translate it without terminology constraints. even then, the model has a hard time determining the correct location for the constraint in the output sentence (cf. @xcite . see section 2.3 for more details of this problem. we propose a combined method that tackles both of the aforementioned problems. to support heavily-inflected languages such as finnish, we introduce a "hard" finite-state automaton-based constraint recognition algorithm that can recognize arbitrarily large disjunctive constraints. for the problem of expressions that were not encountered at training time, we propose a "soft" backtranslation substitution algorithm that makes it possible to use terminology constraints even when the neural network sees no connection between the source-language term and the target-language term. both of these methods are integrated into an end-toend pipeline that takes source-language sentences and lexicons as input and produces lexically constrained translations. we argue that the "hard" and "soft" constraint methods complement each other and work together as a whole greater than the sum of its parts. in this paper, we first describe existing constrained beam search algorithms (section 2). we then give an overview of our pipeline, including detailed descriptions of the backtranslation and the constraint recognition algorithms (section 3). finally, we evaluate these algorithms against the existing algorithms (section 4). ## the mitra pipeline we designed our constrained translation pipeline with three goals: 1) allowing a high number of disjunctive constraint alternatives to support highly agglutinative languages such as finnish, 2) fix the problems caused by greediness in the previous constraint recognition algorithms, and 3) make constrained decoding a viable alternative even for rare terms not present in the training data of the neural network. the goals 1 and 2 are fulfilled by using a custom finite-state automaton-based constraint recognition algorithm, while goal 3 is fulfilled by a backtranslation substitution algorithm. the full end-to-end pipeline contains the following components: ## evaluation we use both automatic evaluation and human evaluation to measure the quality of the translations produced by our pipeline. the automatic methods include bleu, chrf, ter, and comet scores, as well as measuring the number of fulfilled constraints. in human evaluation, we asked a professional translator to evaluate all translated sentences and mark them either as ok, erroneous due to incorrectly applied constraints, or erroneous due to other cause. due to time constraints, we did not evaluate the swedish translation even though we implemented it. ## discussion and conclusions in this paper, we have presented mitra, a pipeline for terminologically-constrained machine translation that improves on the previous "hard" constraint methods with a finite state automaton-based constraint recognition algorithm and a backtranslation substitution step. when compared to the trie-based method based on the previously suggested algorithms @xcite ) without backtranslations, our method significantly increases the quality in human evaluation. we argue that our method fulfills the three goals we began with: allowing disjunctive constraints, solving the problems of greedy constraint recognition algorithms, and improving quality on "rare and obscure" terms. the finite state automata-based algorithm allows any number of alternatives in the disjunctive constraints, and does not suffer from the greediness of previous algorithms. furthermore, when combined with backtranslations, the number of errors caused by incorrectly applied constraints drops significantly on both of the evaluation datasets. while constrained translation significantly improved quality on the forest fire dataset, it unexpectedly decreased quality on the parliament dataset in human evaluation. we believe this discrepancy is due to two main factors. firstly, the parliament dataset's vocabulary is not well-suited for constrained generation (see appendix d for examples), and secondly, the opus-mt models we used are more familiar with the subject matter, using 49.46% of the constraint terms even when unconstrained, leaving less room for improvement. the forest fire dataset, on the contrary, is very unfamiliar to the model, as only 32.94% of the sentences were translated acceptably, and only 22.41% of the 10 see the major downside of "hard" constraints is their increased time requirement: the translation times were more than ten times larger on three of the four configurations evaluated (see of the general-purpose large language models evaluated, mixtral performed very well on the parliament dataset, producing higher-quality results than constrained decoding methods. similarly, poro provides results comparable to the mitra-t configuration in the human evaluation. although the percentage of fulfilled constraints is lower than that of constrained translation, ca. 82-84%, the number of "constraint errors" is also low, implying that at least some of the missing constraints can be explained by the models providing a satisfying translation using a synonym or other acceptable construct that does not match the constraint when they have trouble fitting the constraint word into the sentence. thus, the error mode of llms might be considered better than that of constrained decoding-based translation. since our pipeline is model-agnostic, it can be used with any nmt model or even with a generalpurpose language model. similarly, many of the "soft" constraint methods can also be combined with our method. we believe that our future research should focus on evaluating these combinations. as even a considerably basic soft method such as backtranslations can improve the translation quality significantly, our hypothesis is that more complex soft methods (such as bergmanis and pinnis 2021) can improve it even further. function getcandidates(hypotheses, v , k) is input :hypotheses from the previous iteration v the vocabulary k the parameter for top-k sampling output :set of candidate hypotheses candidates ← ∅ / * constraint continuations from the ndfst * / foreach h in hypotheses do if h is not finished then foreach token t that would advance the current ndfst state do append h + t to candidates end end end / * unconstrained candidates * / foreach h in hypotheses do append top-k continuations of h to candidates end return candidates end algorithm 2: the candidate selection algorithm. this is a simplified version of the algorithm described in (post and vilar 2018, section 3.1), combining their steps 1 and 3. function allocate(candidates, c, b, s) is input :the list of candidates c number of constraint tokens b target hypothesis bank size s maximum beam size output :a new list of hypotheses if b • (c + 1) > s then return allocate2(candidates, c, ⌊ s-1 c ⌋ + 1, s) else return allocate2(candidates, c, b, b • (c + 1)) end end function allocate2(candidates, c, b ′ , s ′ ) is input :the list of candidates c number of constraint tokens b ′ actual hypothesis bank size s ′ actual beam size output :a new list of hypotheses / * allocate hypotheses to banks they belong based on their number of fulfilled constraint tokens * / hypotheses ← [] foreach i in c, . . . , 0 do bank size ← 0 foreach candidate c from most probable to least probable do if bank @xmath0, . . . , 0 do foreach candidate c from most probable to least probable do if |hypotheses| = s ′ then return hypotheses end if number of constraint tokens in @xmath1 algorithm 3: the beam allocation algorithm. constraint is set in the same transition that generates the constraint end token, c(m) also corresponds to the set of constraints that have their flag set. c is the set of all constraint end tokens possible. definition. s(q, m) is the set of strings that the ndfst accepts from the initial state q and the initial output symbol list m. theorem. given a string s partitioned into two parts s 1 s 2 , so that the ndfst has consumed s 1 but not s 2 , and ndfst states (q 0 , m 1 ) and (q 0 , m 2 ), so that c(m 1 ) ⊊ c(m 2 ) and s 2 ∈ s(q 0 , m 1 ), then s 2 ∈ s(q 0 , m 2 ). proof. the only accepting node in the ndfst is fenced with flag diacritic symbols, each corresponding to a different constraint. thus, the accepting state can only be reached if all the flags are set, that is, c(m) = c. since s 2 ∈ s(q 0 , m 1 ), there must be a path q 0 → q i 1 → • • • → q @xmath2, m 1 ) is used as an initial state. as c(m 1 ) ⊊ c(m 2 ), the same path cannot be accepted when (q 0 , m 2 ) is used as the initial state, as for each @xmath3, the path contains a negative flag diacritic check that prevents the path from being accepted, as the flag for c is already set since @xmath4. we construct a new path that is accepted when (q 0 , m 2 ) is used as the initial state. for each @xmath5, we modify the path by replacing each transition beginning from @d.cc@:[cc] and ending to @p.cc.1@:[/cc] with q 0 → q 0 . as all of these c were already present in c(m 2 ), this modification only removes duplicate positive flag diacritic sets. since all the flags are set in the modified path, it is accepted by the ndfst. qed.