Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models (2025)

Abstract: The evolution of Neural Machine Translation (NMT) has been significantly influenced by six core challenges (Koehn and Knowles, 2017) that have acted as benchmarks for progress in this field. This study revisits these challenges, offering insights into their ongoing relevance in the context of advanced Large Language Models (LLMs): domain mismatch, amount of parallel data, rare word prediction, translation of long sentences, attention model as word alignment, and sub-optimal beam search. Our empirical findings show that LLMs effectively reduce reliance on parallel data for major languages during pretraining and significantly improve translation of long sentences containing approximately 80 words, even translating documents up to 512 words. Despite these improvements, challenges in domain mismatch and rare word prediction persist. While NMT-specific challenges like word alignment and beam search may not apply to LLMs, we identify three new challenges in LLM-based translation: inference efficiency, translation of low-resource languages during pretraining, and human-aligned evaluation.

## introduction in the natural language processing (nlp) community, one of the most critical and longstanding tasks is machine translation (mt), which aims to convert human languages from one form to another @xcite . as the demand for effective translation systems continues to grow, researchers have been striving to develop models that can tackle the inherent challenges of this complex task. in this context, the six challenges about mt proposed by @xcite have been widely recognized and studied by numerous studies, with many efforts revolving around them @xcite @xcite . the emerging large language models (llms) have been a significant breakthrough in nlp @xcite @xcite . llms have demonstrated remarkable capabilities, outperforming traditional approaches and setting new benchmark performance for various applications such as machine translation @xcite @xcite . llms exhibit remarkable translation capabilities for major languages, owing to their extensive pretraining on vast amounts of unpaired data. this implies a significant advancement over conventional techniques. consequently, recent studies have employed llms for translation tasks @xcite , achieving remarkable performance. however, it is unclear how llms fare against the six classical challenges, this intriguing question warrants further investigation and discussion. to gain insights into llm-based mt research and identify paths for advancement, we train llama2 models as german-to-english translation systems and evaluate their abilities in addressing the six classic challenges. note that english is a high-resource language in the llama2 pretraining data @xcite and german is a relatively high-resource language, ensuring the model's competence in these languages. furthermore, we identify two llm-specific challenges: pretraining resource imbalance and human-like challenges experiments outcomes enc2dec llms takeaways domain llama2-7b/13b ✗ ✗ llms demonstrate overall advancements yet still face significant domain variance issues. mismatch opus datasets amount of llama2-7b/13b ✗ ✓ llms diminish the dependence on bilingual data for high-resource pretraining languages. parallel data wmt23 datasets rare word llama2-7b/13b ✗ ✗ llms consistently struggle with predicting infrequent words. prediction wmt23 datasets long sentence llama2-7b/13b ✗ ✓ llms address the task of translating long sentences and display remarkable performance at the document level. translation wmt23 datasets word alignment llama2-7b ✓ ✗ the attention weights in llms are unsuitable for word alignment extraction, yet they provide valuable insights into model interpretability. ## new challenges within the research field of llms, two pressing challenges arise. the first challenge concerns the translation quality for language pairs that are underrepresented during the pretraining figure 8 : bleu scores of bi-directional translation results using llama2-7b based translation models across multiple language pairs, where english, german, chinese, ukrainian, and hebrew are sorted based on the level of available resources, with english having the highest resources and hebrew the least in the pretraining stage. ## conclusions our research highlights several key findings. on the positive side, llms have effectively removed the dependence on bilingual data for translation into major languages. additionally, despite being fine-tuned solely on sentence-level translation pairs, llms demonstrate impressive capabilities in handling long sentences and even document-level translations. however, challenges persist. llms face difficulties in adapting to multi-domain tasks and predicting rare words. our experiments suggest that larger models possess greater potential to acquire multi-domain translation knowledge, offering a promising path to mitigating the domain shift issue. yet, performance for low-resource languages remains suboptimal, and inference delays pose a significant bottleneck. addressing these limitations, particularly improving translation for underrepresented languages, will be crucial. our findings prove that leveraging bilingual corpora more efficiently could offer a viable solution. furthermore, the need for more robust, human-aligned evaluation metrics remains an urgent area for future research.