Decoding Machine Translationese in E nglish- C hinese News: LLM s vs. NMT s (2025)

Abstract: This study explores Machine Translationese (MTese) — the linguistic peculiarities of machine translation outputs — focusing on the under-researched English-to-Chinese language pair in news texts. We construct a large dataset consisting of 4 sub-corpora and employ a comprehensive five-layer feature set. Then, a chi-square ranking algorithm is applied for feature selection in both classification and clustering tasks. Our findings confirm the presence of MTese in both Neural Machine Translation systems (NMTs) and Large Language Models (LLMs). Original Chinese texts are nearly perfectly distinguishable from both LLM and NMT outputs. Notable linguistic patterns in MT outputs are shorter sentence lengths and increased use of adversative conjunctions. Comparing LLMs and NMTs, we achieve approximately 70% classification accuracy, with LLMs exhibiting greater lexical diversity and NMTs using more brackets. Additionally, translation-specific LLMs show lower lexical diversity but higher usage of causal conjunctions compared to generic LLMs. Lastly, we find no significant differences between LLMs developed by Chinese firms and their foreign counterparts.

## related work first introduced by @xcite , the concept of translationese describes the systematic influence of a source language on the target language during translation. when applied to mt, @xcite emphasize the pivotal role of mtese in shaping the characteristics of post-edited texts, analyzing 55 linguistic features ranging from pos tags to dependency parsing. expanding on this, @xcite explore lexical density and diversity, revealing that post-edited literary mts tend to be more simplified, normalized, and influenced by the source text compared to human translations (hts), where mt outputs exhibit lower lexical density than hts, with the neural system showing even lower density than those from the statistical system. however, @xcite report contrasting findings from a different genre. for news texts, mts show slightly higher lexical density and richness than hts, whereas for literary texts, mts demonstrate slightly lower lexical density but comparable lexical richness to hts. in a similar vein, loock (2020) investigate mtese by analyzing linguistic deviations in english-to-french mt texts compared to original, untranslated texts, providing a broader perspective on the systematic over-representation of linguistic features and their implications for translator training and post-editing practices. de @xcite , working on the same language pair, used 22 linguistic features to distinguish between the original and mted french. they showed that average sentence length and four features related to formulaicity could discriminate between original and mted french. however, for linguistically distant language pairs like e2c, research on mtese remains rel-atively sparse. @xcite examine a corpus of english translations of modern chinese literary texts, including texts translated by nmt and humans. they confirm the presence of translationese in both human and machine translations compared to original texts in some coherence metrics. a recent study by @xcite revealed that simplification is a notable characteristic of nmt texts across genres in the e2c direction, such as a loss of lexical complexity. a general conclusion drawn from the works above is that translations produced by mt engines consistently exhibit a loss of lexical and syntactic richness @xcite . studies tend to apply finegrained linguistic features to reveal consistent distributional patterns. similar phenomena are also observed across various language pairs. despite these findings, it remains unclear whether llms exhibit distinct features of mtese or whether their outputs can be reliably differentiated from those of nmt engines, particularly in general text types as news discourse. therefore, our study addresses this gap by focusing on the distant language pair of e2c, emphasizing general news texts, and constructing a larger and more comprehensive dataset and feature set for analysis. we adopt the study design of @xcite and de clercq et al. (2021), and compare mt 2 outputs generated by different systems with original texts. we address the following research questions: • rq1: does mtese exist in e2c mted news texts (nmts and llms)? if so, which linguistic features contribute most to this distinction? • rq2: how do llms differ from nmts in their manifestation of mtese across linguistic features? • rq3: do translation-specific and generic llms differ from each other? additionally, how do llms developed by chinese companies compare with those developed by foreign companies in this regard? 3 methodology 3.1 dataset the dataset used in this study encompasses four corpora (detailed information is shown in all texts underwent careful preprocessing, including cleaning, denoising, part-of-speech (pos) tagging, and dependency (dep) tagging. chinese is a language without explicit word boundaries, which requires word segmentation in advance. to achieve state-of-the-art performance, we utilized the language technology platform (ltp) foot_5 , a comprehensive natural language processing toolkit @xcite . we used ltp's advanced deep learning models (base2) to perform word segmentation, pos tagging, and syntactic analysis. its reported performances reach 99.18%, 98.69%, and 90.19% for segmentation, pos tagging, and dependency parsing, respectively. 8 the mt data are generated by translating oen articles into chinese using each engine on a one-byone basis, processing each text individually with each engine, thereby minimizing potential interference from varying text topics. as a result, each mt engine produces approximately 200 translations 9 . the dataset includes five nmt engines, comprising three international systems (google translate, deepl, and microsoft translator) and two chinese-developed systems (baidu translate and youdao translate). additionally, six llms are incorporated, including models developed by chinese firms (kimi and chatglm), one tailored for mt-specific applications (towerinstruct), and leading models such as chatgpt, claude, and gemini. all these systems represent sota engines on the llm arena foot_8 at the time of the experiment (october to december 2024). the mt process for llms involves prompt engineering, with prompt design following andrew ng's course guidelines foot_9 and the crispe framework 12 . this approach resulted in a standardized and structured user instruction, which was consistently applied across all engines during the translation process. the full prompt is provided in appendix a for reference. to address the potential issue of the ocn corpus having insufficient coverage and variability when limited to the same sample size (200) as other subcorpora, we adopted the corpus structure outlined in de @xcite and increased the size of the ocn by incorporating more original chinese news to 2,000 texts in total. this expansion ensures a comparable dataset size between ocn and mts. also, we did not impose strict limitations on specific topics within the news genre. constraining the dataset to a particular domain could lead to data scarcity, as certain topics may not be consistently available over the given period. to maintain balance and comparability, we ensured that all selected news articles were consistent in terms of lexical length and time period. ## conclusion this study applies classification, clustering and feature selection methods in machine learning experiments, with the aim to identify mtese of llms and nmts systems in an e2c news settings. our findings suggest that mtese is still present in llms (rq1). mtese is evident in both nmt and llm systems, with averaged acc reaching almost 99%. ocn-nmts and ocn-llms yield similar results, suggesting that llm translations with prompt engineering still differ significantly from original chinese writing styles. key features include fewer characters per sentence in mts and higher frequencies of adversative conjunctions compared to original chinese. for rq2, a comparison between llms and nmts showed classification accuracy of around 70%. the similarities are most likely due to the fact that both systems translate the same source text, have a similar transformer architecture and have overlapping training data. the differences are reflected in llms exhibiting higher mtld (for lexical diversity) than nmts, meaning greater lexical variation and stylistic flexibility. and nmts use brackets more frequently than llms, possibly due to additional rules embedded in nmt engines for specific proper noun translations. for rq3, which examined subcategories of llms, a comparison between translation-specific and generic systems shows that the specific lto engine exhibits a lower mtld than certain generic llms, but demonstrates a higher proportion of causal conjunctions. we did not find evidence to support the distinction between chinese llms and foreign ones. as a final remark, while llms have made some distinctions from nmts, they remain far from matching the so-called "human-parity" (poibeau, 2022) with stylistic and aesthetic qualities of original chinese writing. future advancements in llms should prioritize minimizing "machine translationese" to better align with native language characteristics and avoid potential contamination towards everyday communication.