When LLM s Struggle: Reference-less Translation Evaluation for Low-resource Languages (2025)

Abstract: This paper investigates the reference-less evaluation of machine translation for low-resource language pairs, known as quality estimation (QE). Segment-level QE is a challenging cross-lingual language understanding task that provides a quality score (0 -100) to the translated output. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios and perform instruction fine-tuning using a novel prompt based on annotation guidelines. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models. Our error analysis reveals tokenization issues, along with errors due to transliteration and named entities, and argues for refinement in LLM pre-training for cross-lingual tasks. We release the data, and models trained publicly for further research.

## introduction traditional methods of obtaining references for machine-translated texts are costly, and prone to subjectivity and inconsistency @xcite @xcite . to address these challenges of evaluating imperfect translations, quality estimation (qe) has emerged as a crucial area, enabling the assessment of mt output in the absence of a reference @xcite . our work investigates segment-level qe @xcite @xcite , which is conventionally modelled as a regression task and aims to predict a segment-level quality score, also known as the direct assessment (da) score @xcite . due to the underlying subjectivity in human translation quality evaluation, da score is computed as a mean of three or more human annotations on a scale of 0 -100. while large language models (llms) claim superlative performance for different natural language processing (nlp) tasks @xcite , evaluation of machinetranslated output poses a unique challenge where both syntactic accuracy and cross-lingual semantic match are relevant, for the prediction of da scores. llms are applicable for many nlp tasks, including machine translation (mt) @xcite @xcite and quality estimation @xcite @xcite . there are significant disparities in the reported performance of llms between high-and low-resource languages @xcite . llms exhibit better performance in evaluating the quality when references are available @xcite ; however, they are challenging to scale due to the cost associated with manual translation. this work focuses on the reference-less scenario, evaluating the efficacy of llms in settings like zero-shot, few-shot/in-context learning (icl), and instruction fine-tuning with an adapter @xcite . we present a novel prompt which utilizes annotation guidelines within prompt instructions and improves task performance. additionally, we perform experiments for both independent language-pair training (ilt -training instances from one language pair), and unified multilingual training (umt -training instances from all language pairs) settings. our contributions are: ## background transformer-based approaches which leverage supervised fine-tuning of regression models significantly improved the performance of qe models @xcite . recently proposed approaches like cometkiwi @xcite , @xcite and transquest @xcite from wmt qe shared tasks @xcite are based on pre-trained encoder-based language models. however, recent claims have propelled the use of llms across various nlp tasks @xcite . following suit, @xcite introduced the gemba prompt-based metric for evaluating translation quality. their approach focuses on zero-shot prompt-based evaluation, comparing four prompt variants across nine gpt model variants for three high-resource language pairs. the paper discusses experiments with both settings, with and without reference, claiming sota performance by including the reference for da prediction. our experiments reproduce their prompt in a referenceless setting utilizing only publicly available llms and compare prompting strategies by adding relevant context. @xcite examined how llms use source and reference information for translation evaluation and they observed that reference information improves accuracy and correlations, while source information shows a negative impact, highlighting limitations in llms' cross-lingual semantic matching capability, which is essential for a task such as qe. @xcite perform qe by pre-tuning the adapter using a large parallel corpus of english-indic languages over machine translation task. they fine-tune the model again using supervised qe data and show that pre-tuning the model using mt does not help. other approaches to qe such as mqm, include fine-grained error annotation and detailed explanations, which are often not viable for lowresource languages due to lack of annotated data. ## results given the ag prompt's strong zero-shot performance, our icl investigations focused solely on it. in the icl setting, 4 language pairs (en-gu, en-mr, en-te, ne-en) performed best with 7-icl, 3 language pairs (en-hi, en-et, si-en) with 5-icl, and 1 language pair (en-ta) with 3-icl. openchat consistently achieved the highest correlation scores across all low-resource pairs, with et-en, ne-en, and si-en outperforming other english-indic pairs in both zero-shot and icl. in appendix tables 4 , 5 , and 6, for the zeroshot setting, we note that the number of dropped rows for the te prompt is the highest whereas the same when using ag prompts is the lowest, likely because ag prompt specifies the score ranges explicitly. umt setting as shown in comparing ilt and umt setting results, the umt performs better for most low-resource language pairs. this suggests that incorporating diverse linguistic data enhances the model's ability to generalize and accurately evaluate translations across various low-resource languages. considering the overall best results, fine-tuned encoder-based models demonstrate the best performance. ## discussion zero-shot-in comparison to the gemba and te prompts, the ag prompt demonstrated the best overall performance in zero-shot experiments with llms across the majority of language pairs. this indicates that in the absence of training data, the additional context provided in the ag prompt-acting as an annotation guide, enhances the effectiveness of llm-based quality estimation more effectively than llms functioning as translation evaluators (te template) or simply assigning scores based on a straightforward request like in the gemba prompt. the structured guidelines in the ag prompt offer a clearer framework for evaluating translation quality, which supports more accurate scoring in zero-shot settings. icl-outperformed zero-shot for most language pairs (en-gu, en-te, et-en, ne-en, si-en), suggesting that adding examples improves llms' ability to predict translation quality. however, the effect of increasing examples varied across language pairs and models (see appendix k). when the number of examples in the icl prompts was increased, the en-gu and ne-en language pairs with the gemma-7b model, as well as the en-mr and ne-en language pairs with the openchat model, consistently showed improved performance. however, for other language pairs and models, the performance gains were not always evident, suggesting that increasing the number of examples does not necessarily lead to better results. fine-tune -we observed a notable improvement in correlation scores when moving from zeroshot to fine-tuning, compared to zero-shot to icl (appendix k). this indicates that instruction finetuning with task-specific data is more effective than providing detailed examples in prompts. in fine-tuning experiments, pre-trained encoder-based models with umt settings outperformed llms. despite this, llms are significantly larger in size and contain more parameters compared to pre-trained encoder models (appendix l). while llms can handle various nlp tasks and show decent performance in translation evaluation for some low-resource language pairs, they are not specifically trained for regression tasks like pretrained encoders. this difference likely contributes to llms' lower performance in qe. notably, the openchat model consistently outperformed other llms when provided with sufficient context as annotation guidelines. a noteworthy observation is that english, when used as the target language in machine translation, consistently achieved higher correlation scores for qe in zero-shot and icl experiments with llms. similarly, figure 2 , which highlights setting-agnostic best performance for fine-tuned llms vs. transquest-infoxlm vs. comet, figure 3: error types and their percent contribution. figure 2 indicates llms are outperformed for most language pairs by transquest-based and comet models. interestingly, for the only language pair where llms match comet performance, en-mr, the results are statistically insignificant. among the indic-target language pairs, en-mr shows a consistently higher correlation, but statistically insignificant in most cases ( tokenization analysis to explore the reasons behind the better performance of fine-tuned pretrained encoders over llms in reference-less qe tasks, we conducted an analysis of token counts generated by llms and pre-trained encoders, such as transquest's infoxlm and cometkiwi's xlm-r-xl. for comparison, high-resource language pairs from the wmt22 test data @xcite were included to assess tokenization differences across languages with varying resources. we selected 100 sentences per language pair from our test set and created a tokenization pipeline for each model. both source and translation texts were input to observe token counts. figure 4 shows word and token counts for three language pairs, revealing slight differences between llama-2-7b and openchat-3.5 despite using the same tokenizer. the tokenization outcomes for all language pairs are detailed in appendix e. the token counts generated by llms (gemma, openchat, llama) for low-resource non-english languages significantly deviate from the original word counts, while pre-trained encoders like infoxlm and xlmr-xl show smaller discrepancies. rich morphological languages like marathi, tamil, and telugu, which feature agglutinative foot_5 phrases, and hindi, which includes compounding, experience skewed tokenization, affecting semantic matching between source and translation (appendix e). in contrast, for english, the tokenized count closely matches the word count, regardless of the model used. this highlights the need for improved tokenization strategies for cross-lingual semantic matching with llms for low-resource languages to enhance performance on the qe task. we also identified that the et-en language pair consistently achieved the highest performance across all experimental settings. as illustrated in the appendix e, the difference between the token counts generated by language models vs. the original word counts is evidently smaller than that observed for other low-resource languages. this holds true even for llms as well. this reduced tokenization discrepancy, likely due to both languages (en and et) using the latin alphabet, may explain why et-en performs better in all experimental settings. looking at the tokenization for high-resource non-english languages (see appendix e), it can be seen that the language pairs en-de (german) and ro (romanian)-en exhibit limited disparity in the number of tokens from the original word count, and for en-zh (chinese) it significantly higher. de/ro uses latin-based scripts too. this analysis suggests that english and other latinscript-based languages, benefit from more efficient tokenization in language models, which leads to improved performance in tasks like qe. in contrast, other languages, exhibit greater disparities in token counts, indicating the need for more advanced tokenization strategies with llms to enhance performance. this underscores the importance of developing better tokenization methods to ensure equitable model performance across different language pairs. ## conclusion and future work this paper investigates reference-less quality estimation for low-resource language pairs using large language models. we reproduce results with existing sota prompts and propose a new ag prompt, which performs best in zero-shot settings. further experiments with icl and instruction finetuning settings are performed with ag prompt which achieves closer performance with the pre-trained encoder-based approaches. our findings indicate how llm-based qe can be challenging for morphologically richer languages without much data in the pre-training stage. based on our findings, we highly recommend the addition of qe datasets to llm evaluation task suits given the significant cross-lingual challenge posed by this task. we perform a detailed tokenization analysis which highlights that crosslingual machine understanding for low-resource languages needs to be addressed at the stage of tokenization @xcite , and within pretraining data @xcite . additionally, error analysis highlights significant challenges in handling context, syntax, and domain-specific terms, suggesting that further refinement in model training and adaptation is necessary. in the future, we aim to employ regression head-based adapters within the llm pipeline for qe, eliminating the challenges in the reliability of extracting the scores from the outputs.