The Benefits of Being Uncertain: Perplexity as a Signal for Naturalness in Multilingual Machine Translation (2025)

Abstract: Model-internal uncertainty metrics like perplexity potentially offer low-cost signals for Machine Translation Quality Estimation (TQE). This paper analyses perplexity in the No Language Left Behind (NLLB) multilingual model. We quantify a significant model-human perplexity gap, where the model is consistently more confident in its own, often literal, machine-generated translation than in diverse, high-quality human versions. We then demonstrate that the utility of perplexity as a TQE signal is highly context-dependent, being strongest for low-resource pairs. Finally, we present an illustrative case study where a flawed translation is refined by providing potentially useful information in a targeted prompt, simulating a knowledge-based repair. We show that as the translation’s quality and naturalness improve (a +0.15 COMET score increase), its perplexity also increases, challenging the simple assumption that lower perplexity indicates higher quality and motivating a more nuanced view of uncertainty as signalling a text’s departure from rigid translationese.

## introduction translation quality estimation (tqe) is critical for machine translation (mt) system deployment, and is particularly challenging for low-resource languages. whereas reference-based evaluation metrics directly compare gold-standard humangenerated translations and model-generated translations, tqe aims to assess the quality of translations without such references. this paper employs a glass-box approach to investigate a twostage goal: first, whether a model's internal uncertainty, measured by perplexity (ppl), can serve as a lightweight signal to detect likely errors, and second, whether those errors can then be repaired using knowledge-guided prompting. one challenge to using perplexity for tqe is that autoregressive models prefer their own output distributions. our first contribution is to provide an empirical quantification of this phenomenon in a massively multilingual setting. we measure this model-human perplexity gap within the nllb-200-3.3b model @xcite , confirming that the model is systematically less perplexed by its own text than by professional human translations. our findings show that perplexity often measures conformity to translationese rather than actual translation quality. this finding motivates a more nuanced question that we explore: beyond simple quality, can perplexity serve as a signal for a translation's naturalness? this paper investigates the possibility that as a translation moves from literal translationese towards more fluid, human-like language, its perplexity, as judged by the original model, might paradoxically increase. finally, we conclude with a case study demonstrating the detect-and-repair workflow that motivates our research. we show how a lexical error, once identified, can be repaired using an instruction-tuned model guided by external knowledge, providing initial evidence for our naturalness hypothesis. ## related work our work builds on research into model uncertainty for tqe, model artifacts like translationese, and llm-based refinement. using model-internal probabilities for reference-free tqe is a wellestablished approach @xcite . however, optimizing solely for low perplexity can make models less human-like, as their surprisal patterns diverge from human reading patterns on key syntactic structures @xcite . our work quantifies how this sensitivity is modulated by language resource levels. we also connect this to the challenge of translationese-the distinct statistical characteristics of translated text, a known issue in mt evaluation @xcite , and contribute by measuring the model's preference for this form of language across a multilingual setting. ## method our primary analysis centres on the nllb-200-3.3b model. for our refinement case study, we use the llama 3 7b instruct model @xcite , since the nllb models are not instructiontuned and are therefore unsuited for the open-ended editing task required by our prompt. all translation data is from the 'devtest' split of the flores-200 dataset @xcite . we analyse three language pairs with english: spanish (highresource), japanese (medium-resource), and te reo māori (low-resource). our metrics include ppl, sacrebleu @xcite , and comet @xcite . for japanese evaluation, we use the default 'ja-mecab' tokenizer, which relies on the mecab morphological analyser @xcite . the specific comet model is 'unbabel/wmt22-comet-da' @xcite , the top-performing model from the wmt22 shared task @xcite . defining naturalness we define naturalness as a translation's fluency and resemblance to human writing. in our case study, we operationalize this by assessing two of its key components: an increase in overall translation quality, measured by the comet score, and the correction of a clear lexical error. ## analysing the model-human perplexity gap our first analysis involved generating translations for all language pairs and scoring the perplexity of both the mt and human reference for the same source sentence. ## discussion our experiments expose two key findings: a significant and consistent model-human perplexity gap exists, and the utility of perplexity as a quality signal is conditional on accounting for the resource level of the language pair. the quantification of the perplexity gap is a central finding. the nllb-3.3b model is systematically more confident in its own, often literal, outputs than the arguably more nuanced professional human translations. this indicates, not entirely surprisingly, that the model's internal sense of surprise is calibrated to its own output distribution-to its version of translationese-rather than to natural human language. we also observe an interaction between model scale and resource level. for the high-resource @xmath0, the correlation between perplexity and quality weakens with the larger model (from @xmath1.448 to @xmath2.299), hinting that once fluency reaches a ceiling, uncertainty becomes a less informative signal. token-level surprisal analysis further explains why perplexity can be a brittle quality indicator. in our study, high perplexity often stemmed from tokenization artifacts, such as rare subword fragments, rather than genuine semantic or syntactic difficulty. this suggests that raw perplexity scores may reflect superficial statistical anomalies more than deep meaning errors. accounting for this may improve use of ppl for translation quality improvement. finally, our case study illustrates that perplexity might signal naturalness rather than correctness. this reframes its potential role: instead of simply minimizing perplexity, future work might aim to align a translation's perplexity profile with that of high-quality human-generated text, or with text whose correctness has been improved or certified by other means. ## limitations and future work our analysis focussed on the nllb 3.3b model. results may differ for the largest nllb variants, which could exhibit different perplexity distributions. additionally, our evaluation relies exclusively on flores-200, which consists of multidirectional translations of encyclopedic text. testing across other domains will be essential to assess generalisability. our simulated rag refinement was a proof of concept with a hand-selected context. a next step would be to develop and evaluate a full rag system with automated retrieval strategies to determine if the observed improvements hold. more broadly, our results suggest that perplexity may serve as a signal for naturalness; scaling up the case study to correlate perplexity with human fluency judgements across diverse language pairs and domains could validate this. ## conclusion we quantified the model-human perplexity gap in a large multilingual model, showing that perplexity often measures conformity to translationese rather than semantic quality. we found that its utility as a quality signal is strongest in low-resource settings. our case study demonstrates that targeted refinements can improve a translation while increasing perplexity, challenging the view of perplexity solely as a metric to minimize. a richer interpretation treating it as a useable signal for naturalness could open new directions for improving translation quality.