A Benchmark for Translations Across Styles and Language Variants (2025)

Abstract: As machine translation (MT) rapidly advances in bridging global communication gaps, there is growing interest in variety-targeted translation for fine-grained language variants and specific translation styles. This translation variant aims to generate target outputs that are not only contextually accurate but also culturally sensitive. However, the lack of comprehensive evaluation benchmarks has hindered progress in this field. To bridge this gap, this work focuses on the translation across styles and language variants, aiming to establish a robust foundation for the automatic evaluation of fine-grained cultural and stylistic nuances, thereby fostering innovation in culturally sensitive translations. Specifically, we evaluate translations across four key dimensions: semantic preservation, cultural and regional specificity, expression style, and fluency at both the word and sentence levels. Through detailed human evaluations, we validate the high reliability of the proposed evaluation framework. On this basis, we thoroughly assess translations of state-of-the-art large language models (LLMs) for this task, highlighting their strengths and identifying areas for future improvement.

## introduction machine translation (mt) has made significant strides in breaking down communication barriers around the world, particularly for widely spoken languages like chinese and english at a broad level. as mt technologies continue to advance, there is growing interest in variety-targeted translation, targeting fine-grained language variants such as regional dialects @xcite , and specialized stylistic adaptations, including formality-aware mt @xcite @xcite and personalized mt @xcite . this evolution in mt aims to ensure that translations are not only contextually accurate but also culturally sen-sitive, thereby facilitating cross-cultural communication @xcite . the emphasis on integrating translations with different regions, cultural contexts, and specific styles highlights the unique challenges of this task compared to general machine translation. as a result, traditional evaluation metrics such as bleu are no longer adequate to measure the quality of these fine-grained translations @xcite . progress in this area has been hampered by the lack of comprehensive, highquality evaluation benchmarks to assess stylistic and cultural variations in translations. to bridge this gap, this work explores automatic evaluation metrics for translations across styles and language variants. specifically, we focus on the translation scenario from english to chinese variants, targeting social media translations in mainland mandarin (zh_cn), taiwanese mandarin (zh_tw), and the web-minority singaporean mandarin (zh_sg). to comprehensively capture cultural and regional nuances as well as the desired expression style in translations, we assess translations at both word and sentence levels across four key dimensions: semantic preservation, cultural and regional specificity, expression style, and fluency. at the word level, we evaluate lexical terms that explicitly reflect regional and cultural nuances, focusing on: 1) models' ability to accurately understand and translate region-specific vocabulary; 2) the alignment of lexical choices in models' translations with local references, showcasing its grasp of domain-or culture-specific expression patterns. at the sentence level, we leverage implicit linguistic expression features to evaluate the model's overall performance in meaning preservation, regional cultural adaptation, and expression style transfer. in summary, the key contributions of this work are three-fold: ## conclusion to fill the gap in a thorough evaluation of varietytargeted machine translation, this work proposes a benchmark for automatically assessing machine translation across language variants and styles. a detailed human assessment validates the high reliability of the proposed evaluation framework. leveraging the proposed metrics, we perform a comprehensive evaluation of recent llms on this task and highlight key challenges for future research.