Anti- LM Decoding for Zero-shot In-context Machine Translation (2024)

Abstract: Zero-shot In-context learning is the phenomenon where models can perform a task given only the instructions. However, pre-trained large language models are known to be poorly calibrated for zero-shot tasks. One of the most effective approaches to handling this bias is to adopt a contrastive decoding objective, which accounts for the prior probability of generating the next token by conditioning on a context. This work introduces an Anti-Language Model objective with a decay factor designed to address the weaknesses of In-context Machine Translation. We conduct our experiments across 3 model types and sizes, 3 language directions, and for both greedy decoding and beam search. The proposed method outperforms other state-of-the-art decoding objectives, with up to 20 BLEU point improvement from the default objective in some settings.

## introduction decoding strategies in supervised neural machine translation (nmt) models are typically designed to select the maximum likelihood response under the given model. however, in the era of in-context learning with large language models (llms), selecting for the maximum likelihood response should be re-examined, as llms are known to be poorly calibrated for their tasks and exhibit a strong prior bias @xcite . there are two large classes of decoding strategies: sampling methods and decoding objectives. sampling methods do not change the probability ranking of the next token but influence how it is sampled, such as beam search @xcite , nucleus sampling @xcite , and top-k sampling @xcite . decoding objectives modify the probability of the next token before sampling takes place, typically by adding or subtracting scores, i.e., a contrastive objective. decoding objectives are one of the most effective approaches for improving the output of a generative model. these often require no training and are typically "frustratingly simple", because they only involve manipulation of output probability distributions at inference time. however, as outlined by @xcite in their recent large-scale survey, even for task-specific generation models, it is still surprisingly unclear what a good objective is for natural language generation. in this work, we propose an anti-language model (anti-lm) decoding objective with exponential decay, which is motivated by the observation that llms are performing bayesian inference under the hood @xcite , and are thus inclined to continue generating in the source language. we investigate this under a zeroshot setting where only the instructions are provided to the model without any examples. 1 we hypothesise that poor translations may be due to a strong prior bias of the dominant source language, but that this bias should diminish over future decoding steps. anti-lm modifies the original logits by taking the difference of the next token logits, conditioned on the source sentence to be translated. penalising the conditional source sentence logits discourages the model from continuing the non-translated generation from the source sentence or regurgitating it. we consider this negative scenario as a "failure to translate". our work falls under the category of contrastive objectives, which were popularised by @xcite . we compare our approach against other contrastive objectives: "conditional domain pmi" @xcite and "context-aware" decoding @xcite . our method consistently outperforms competitive baselines across language directions and model sizes, and the default objective by up to 20 bleu points. analysis shows that most of the gains come from the "failure to translate" case which the method was designed for. in addition, compared to the other contrastive decoding objectives, we only need to compute the contrastive logits (to be subtracted) once for the source sentence and not at every time step. regarding error analysis, tan and monz (2023) found that performance varies with respect to language direction, vocabulary overlap, and linguistic properties, and @xcite identified lexical similarity between source and target language as an issue. to combat the off-target problem, @xcite introduced language aware vocabulary sharing (lavs) to add language-specific tokens and decrease lexical similarity. @xcite , similar to our work, use a decoding method to improve zero-shot performance. they introduced ebbs (ensemble with bi-level beam search), where multiple "components" influence the final generation. other work uses a third language as a "pivot" @xcite . unlike these prior works, our method does not involve training new tokenizers, adding linear complexity with ensembling, or translating through other languages. contrastive decoding methods have been extensively explored for text generation, with different motivations behind each method. they have been used to reduce toxic language @xcite , improve general quality without further training @xcite , improve factuality @xcite , and reduce repetition @xcite . for summarization, @xcite used conditional pmi decoding to avoid model hallucinations and promote "faithfulness". @xcite , @xcite and @xcite adopt a weighted pmi based objective, conditioned on the context to "penalise surface forms". the key difference between our anti-lm formulation and other prior work, is that we compute the contrastive logits directly on x, the test source sentences to be translated, and not other "non-x context". our approach is motivated by improving the decoding of the target language by penalising source language continuations in zero-shot mt. within mt, concurrent work by @xcite also introduces a similar concept of "language contrastive decoding", but under a different formulation where they recompute the contrastive logits at each time step. in contrast, our method does not require recomputing the logits at every time step, which greatly speeds up inference. similarly, sampling methods are also designed for different purposes. nucleus sampling is good for creative generation @xcite while beam search is a popular choice for mt @xcite . decoding objectives can work in tandem with sampling methods but may have unexpected effects due to modification of the output probability space. we thus evaluate our objective with both greedy decoding and beam search. ## experiments decoding objectives we evaluate 4 decoding objectives in addition to the default maximum likelihood objective (summarised in models. we use three models: xglm (2.9b, 7.5b) @xcite , bloom (3b, 7b) @xcite and llama 2 (7b, 7b-chat) @xcite . 4 all models are available on huggingface @xcite , with the latter three having been advertised as "multilingual language models." to our knowledge, there have not been any reports of sentence-level parallel corpora in their training datasets (appendix a). in other words, these models were not trained with data that explicitly supports the translation task. ## results we observe that anti-lm objective is best across most objectives, language directions, and sampling strategies (see ## analysis failure to translate. models may fail to translate the provided sentence due to no generation or generation in the source (l1) language. even for the "large" multilingual models (xglm7.5b and bloom7b), the models still make a sizeable number of such errors (10%-45%). figure 1 shows the number of translation failures across models for pmi(x) and anti-lm(x) for en↔fr against the default (greedy) objective. we analyse the scores for the non-failure cases and find that there is largely equivalent proportion of sentences which are either better or worse than the baseline (figure 2 ). this indicates that the gains observed can be attributed to addressing failure to translate cases (see appendix d). ## conclusion decoding objectives are one of the most effective ways to improve a model's output, especially if it has strong prior bias from pre-training. we designed an anti-lm objective with decay for zeroshot machine translation which has a much smaller computational overhead and is more effective than existing approaches. our method outperforms strong baselines across language directions, model types and sizes, and decoding strategies, especially in failure to translate cases.