Language-Informed Beam Search Decoding for Multilingual Machine Translation (2024)

Abstract: Beam search decoding is the de-facto method for decoding auto-regressive Neural Machine Translation (NMT) models, including multilingual NMT where the target language is specified as an input. However, decoding multilingual NMT models commonly produces off-target translations – yielding translation outputs not in the intended language.In this paper, we first conduct an error analysis of off-target translations for a strong multilingual NMT model and identify how these decodings are produced during beam search. We then propose Language-informed Beam Search (LiBS), a general decoding algorithm incorporating an off-the-shelf Language Identification (LiD) model into beam search decoding to reduce off-target translations. LiBS is an inference-time procedure that is NMT-model agnostic and does not require any additional parallel data. Results show that our proposed LiBS algorithm on average improves +1.1 BLEU and +0.9 BLEU on WMT and OPUS datasets, and reduces off-target rates from 22.9% to 7.7% and 65.8% to 25.3% respectively.

## motivation with neural machine translation (nmt) @xcite becoming the state-of-the-art approach in the bilingual machine translation literature, multilingual neural machine translation (mnmt) has attracted much attention @xcite . mnmt has two main advantages: a) it enables one model to translate between multiple language pairs and thus reduces the model and deployment complexity from o(n 2 ) to o(1), and b) it enables transfer learning between high-resource and low-resource languages. one attractive feature of such transfer learning is zero-shot translation, where the multilingual model is able to translate between language pairs unseen during training. for example, after training from french to english and english to german mt data, the model could directly translate french to german. despite the theoretical benefits, recent studies have found an overwhelming amount of offtarget translation especially for the zero-shot directions @xcite , where the translation is not in the intended language. existing methods all aim to mitigate off-targets during training. @xcite ; @xcite apply back translation (bt) to generate synthetic training data for the zero-shot pairs. @xcite introduces a language prediction loss and regularizes the training gradients with a held-out oracle set. yet, none of the previous work has investigated the off-target issue at decoding time, i.e. how off -target translations emerge and come to outscore on-target translations during beam search decoding. in this work, we first examine when and how off-target translation emerges during beam search decoding, and then propose language-informed beam search (libs), a general algorithm to reduce off-target generation during beam search decoding by incorporating an off-the-shelf language identification (lid) model. our experiment results on two large-scale popular mnmt datasets (i.e. wmt and opus) demonstrate the effectiveness of libs in both reducing off-target rates and improving general translation performance. on average libs reduces off-target rates from 22.9% to 7.7% and 65.8% to 25.3% on wmt and opus respectively, which translates to +1.1 bleu and +0.9 bleu overall quality improvement. moreover, libs can be added post-hoc to reduce off-target translation of any existing multilingual model without requiring any additional data or training. ## experiment setup in this section, we illustrate the data and model setup we used, and the experimental results of our language-informed beam search algorithm. since we re-use the exact same model and data setup as @xcite ; @xcite , and we will provide a brief summary and direct readers to those works for more details. ## analyzing off-target occurrence during beam search to understand the off-target occurrence during beam search, we analyze the off-target error types on different language pairs, and conduct experiments with varying beam sizes. ## language-informed beam search (libs) the standard beam search process originating from the bilingual nmt model is target-languageagnostic and is found to produce an overwhelming number of off-target translations @xcite . yet the target language (i.e. the desired language for generation) is always known during decoding, thus it is straightforward to enforce the desired language to reduce the offtarget rates without any additional training or data. we thus propose language-informed beam search (libs), a general decoding algorithm to inform the beam search process of the desired language during decoding. to inform the beam search process of the desired language, we borrow an off-the-shelf language identification (lid) model to score the running beam search candidates with their probabilities in the correct language. since the candidates are normally ranked by the nmt model probabilities, we linearly combine the two log probabilities to ideally the detailed algorithm is illustrated in algorithm 1. for each step, we first pre-select topw candidates from each beam. then we sort all b • w active candidates by the linearly combined nmt and lid log probabilities, where we tune the linear coefficient α on the dev set. same as the fairseq @xcite ) implementation, we only store the finished ones within the top-b candidates, meanwhile save the top-b active candidates into the beam for the next step 10 . the decoding process stops when we have found b finished candidates, and at the end of the generation, we again rerank all finished candidates with the linearly combined log probabilities. ## experiment results to fully verify the performance of the libs algorithm, we compare libs against the baseline beam search decoding on both wmt and opus-100 datasets. ## conclusions our work conducts a comprehensive off-target error analysis with strong multilingual nmt models, to answer the question of how off-target translation wins over time during decoding. we additionally propose an empirical language-informed beam search algorithm to mitigate off-target errors during decoding time and with linear-scale overhead.