G en AI ese - A Comprehensive Comparison of GPT -4o and D eep S eek-V3 for E nglish-to- C hinese Academic Translation (2025)

Abstract: This study investigates the translation performance of two large language models–ChatGPT-4o and DeepSeek-V3–in translating English academic papers on on language, culture, and literature into Chinese at the discourse level. Using a corpus of 11 academic texts totaling 3,498 sentences, we evaluated translation quality through automatic metrics (COMET-KIWI), lexical diversity indicators, and syntactic complexity measures. Our findings reveal an interesting contrast \colon while DeepSeek-V3 achieves higher overall quality scores, GPT-4o produces translations with consistently greater lexical richness (higher type-token ratio, standardized TTR, average sentence length, and word entropy) and syntactic complexity across all five measured metrics, such as Incomplete Dependency Theory Metric (IDT), Dependency Locality Theory Metric (DLT), Combined IDT+DLT Metric (IDT+DLT), Left-Embeddedness (LE), and Nested Nouns Distance (NND). Particularly notable are GPT-4o’s higher scores in Left-Embeddedness and Nested Nouns Distance metrics, which are specifically relevant to Chinese linguistic patterns. The divergence between automatic quality estimation and linguistic complexity metrics highlights the multifaceted nature of translation quality assessment.

## introduction quality estimation (qe) of machine translation (mt) products has long been a key area of interest to both mt developers and translation scholars. a wide range of qe methods have been developed to provide information on improving and selecting mt systems. at the same time, specific features in machine-involved translation products, such as high levels of semantic and syntactic literality and unidiomatic target language expressions, have also attracted great research interest, offering implications for further understanding of mt systems and the design of more reliable and valid qe methods. for statistical and neural machine translation (nmt) systems, mt outputs tend to be easily influenced by source text structure, exhibiting a stronger structural shining-through effect, lower level of target-text normalization and reduced linguistic richness when compared to fromscratch human translation products @xcite . those features, referred to as "machine translationese", will further bring an effect on downstream post-editors, leading to "post-editese" -features distinguishing post-editing products from from-scratch translation products, including lexical simplification and more salient syntactic influence from the source text @xcite . the performance of large language models (llms) applied to translation tasks has been proven promising. llms outperformed commercial nmt models in various language pairs when it comes to document-level translation @xcite . @xcite found that the strength of gpt-powered translation is more salient when it comes to human evaluation, possibly due to its advantage in contextual coherence and naturalness of the target language. the better ability of context awareness in gpt-powered translation over nmt systems is proved by @xcite , who found the advantage in all tested language pairs except a low-resource pair (english-irish). llm-powered translation is also found to be less literal compared to nmt systems when translating out of english @xcite , showing its potential in tackling issues of machine translationese. however, a comparison of translation error types between chatgpt and nmt systems found more frequent over-translation and mistranslation errors in gpt-powered translation, suggesting the problem of hallucinations in llm-powered translation @xcite . despite the research effort in investigating llmpowered translations, previous studies mostly followed quality assessment methods used in assessing mt outputs, either on a sentence level or on a document level of a short length. such a short evaluation unit might prevent us from exploring linguistic issues that will probably bring negative effects to the users in real-life translation settings, for example, cohesion and coherence issues. this concern is particularly significant in the translation of academic papers, as these texts are typically lengthy, feature complex syntactic structures, and demand high accuracy in terminology. rendering academic papers on topics such as language, culture and literature is even more challenging. these papers are often rich in cultural-loaded expressions, wordplays, and quotations of literary works, making them complex combinations of technical and creative texts. therefore, translating such papers requires a high level of accuracy and creativity. previous research highlights that translators often face challenges in maintaining consistency, choosing precise terminology, and adapting to evolving technical language (al-smadi, 2022). moreover, the complexity of subject matter and the need for adherence to proper academic style make the translation process time-consuming and meticulous @xcite . ensuring high-quality translations requires not only linguistic proficiency but also careful proofreading and a deep understanding of the field's specialized knowledge. if llms can be effectively tested and proven capable of handling these challenges, they could significantly assist professional translators and streamline the translation process by improving consistency, reducing repetitive manual corrections, and assisting with complex syntactic structures. therefore, there is a pressing need to explore both llm-powered mt capabilities and corresponding qe methods at the discourse level. this dual focus would not only advance the development of more sophisticated translation technologies but also ensure reliable quality assessment methods that can effectively evaluate long-form translations, considering factors such as terminology consistency, cross-reference accuracy, and overall document coherence. the present paper reports a study on the performance of llm-powered mt products under the context of academic translation. our focus is on comparing the performance of two non-reasoning models, namely, chatgpt-4o and deepseek-v3, in english-chinese translation. deepseek-v3 is a free-access model recently released in december 2024. although user feedback on the use of deekseek in translation tasks is generally positive, experts in the language industry remain skeptical of its translation capability, pointing out that it did not outperform other mainstream llms in some language pairs and domain-specific use cases @xcite . this calls for a systematic evaluation of deepseek-powered translation products before reaching a solid conclusion on its translation performance. the significance of this study lies in its examination of deepseek's translation capabilities since its recent launch. the investigation comes at a critical moment in the way that machine translation increasingly handles complex document-level tasks. this research addresses two notable gaps in the current literature: first, it provides early empirical evidence of deepseek's translation performance, contributing to our understanding of emerging large language models; second, it offers a systematic comparison with chatgpt at the discourse level, moving beyond the more common sentencelevel evaluations. this comprehensive analysis at the discourse level is particularly valuable as it better reflects real-world translation scenarios and reveals how these models handle broader context and maintain consistency across longer texts. ## methodology the sts totaled 3,498 sentences (93,865 tokens), covering a range of topics including language, culture, and literature. examples appeared in some articles that involve languages other than english (1,044 tokens) were removed when calculating readability indices. profiling of the sts is reported in flesch-kincaid grade level (rdfkgl) estimates the us school grade level required to understand the text. higher scores indicate that more advanced education is needed to comprehend the text. for example, a score of 12 suggests a high school senior level, while 16+ suggests college graduate level. in our data, st9 has the highest rdfkgl (18.157), suggesting it requires post-graduate level education to comprehend. coh-metrix l2 readability (rdl2) is specifically designed to assess text difficulty for second language learners. it incorporates additional linguistic features beyond sentence and word length, including cohesion, syntactic complexity, and lexical diversity. lower rdl2 scores indicate text that would be more challenging for non-native speakers. in our dataset, st11 has the lowest rdl2 (7.161), suggesting it would be particularly difficult for second language readers. overall, the readability scores indicate that the sts generally fall within the reading proficiency of college to graduate students who are native english speakers and are relatively difficult to comprehend by l2 english speakers @xcite . the sts were translated by chatgpt-4o and deepseek-v3 with the same prompt: "you are a professional translator working with academic texts. translate this from english to chinese: st". we chose the models for two reasons. first, they are likely to be accepted by users in need of english-chinese machine translation. in a recent survey among chinese professional translators, 75.5% respondents reported use of chatgpt as translation aid @xcite . deepseek-v3 is reported to achieve "performance comparable to leading closed-source models, including chatgpt-4o and claude-3.5-sonnet, on a series of standard and open-ended benchmark" and "surpasses these models in chinese factual knowledge" @xcite , showing its potential in conducting englishto-chinese translation tasks. second, they can generate the entire target text more effectively than newer reasoning models. in our pilot study, we tested the state-of-the-art reasoning model deepseek-r1, released on january 20th. we observed that the r1 model tended to omit parts of sentences or entire sentences to a large extent in its translations. as shown in ## automatic quality estimation in this paper, we use comet-kiwi foot_1 as the automatic tool for qe, as it provides a more comprehensive and linguistically informed evaluation of mt outputs. unlike bleu @xcite , which primarily measures lexical and syntactic similarity based on n-gram overlaps, comet-kiwi captures deeper semantic relationships between the source and translation. this ability allows it to assess meaning more effectively, making it robust to variations in word choice and paraphrasing that @xcite . bleu often fails to capture fluency and adequacy effectively, especially in cases where an nmt system produces highly fluent yet paraphrased translations. in contrast, comet-kiwi's deep learning-based approach aligns more closely with how humans assess translation quality, making it a more reliable metric. according to the results of overall comet scores at text level, deepseek-v3 (ds-v3) demonstrates superior performance overall with an average comet score of 0.7790, compared to chatgpt-4o (gpt-4o)'s 0.7655. this 0.0135 point advantage, while seemingly modest, is consistent across nearly all texts (10 out of 11) and suggests a meaningful difference in translation quality. as shown in figure 1 , ds-v3 shows more consistent performance across different sts. its scores range from 0.6847 to 0.8171, whereas gpt-4o shows greater variability, with scores ranging from 0.6514 to 0.8074. this suggests ds-v3 may offer more reliable quality across diverse topics or text complexity levels. the most revealing insights emerge from examining specific sts. st3 (thought-language identification) exhibits the largest performance gap (0.1292) between ds-v3 and gpt-4o despite having only moderate complexity scores (rdfre: 33.76, rdfkgl: 14.41, rdl2: 12.58). gpt-4o's dramatic underperformance on this specialized linguistic content indicates a significant weakness in handling certain conceptual ambiguity. conversely, st1 (sociolinguistic scales), the formally most complex text (lowest rdfre: 18.206, second lowest rdl2: 8.086), shows strong and nearly identical performance from both models (ds-v3: 0.8171, gpt-4o: 0.8041). a likely explanation is that st1's content domain may contain terminology and concepts that are well-represented in the training data of both models. even though the text is structurally complex, the semantic content may be more accessible to these models compared to other domains. alternatively, the text may be complex but internally consistent in its terminology and logical reasoning patterns, making it more manageable for the llms to translate despite its high readability scores. the correlation between performance gap and readability measures is minimal (rdfre: @xmath0.11; rdfkgl: @xmath1.005; rdl2: @xmath2.05; all @xmath3.05), reinforcing that domain-specific knowledge rather than general readability differentiates these models' translation performance. while comet-kiwi and similar qe metrics provide a general assessment of mt quality of both llms, their performance in terms of lexical diversity and syntactic complexity is worth further investigation as part of their translation quality evaluation @xcite . ## comparison of lexical diversity between ds-v3 and gpt-4o lexical diversity refers to the variety and richness of vocabulary used in a text, representing a crucial factor in assessing translation quality @xcite . it is typically measured through metrics such as type-token ratio (ttr), moving-average ttr (mattr), and measure of textual lexical diversity (mtld) @xcite . while advancements in llms promise improved translation quality, their ability to maintain lexical diversity remains an area of active investigation. in this paper, we compare the lexical diversity of translation outputs produced by gpt-4o and ds-v3 at discourse level using our dataset of academic papers on topics such as language, culture, and literature. the translation of such papers demands both precision and creativity, making them ideal test sets for evaluating advanced llms' ability to maintain lexical richness while preserving semantic accuracy. we first calculate traditional lexical diversity metrics using the wordsmith tool to analyze surface linguistic features in the translated texts by both gpt-4o and ds-v3. these metrics include type-token ratio (ttr), which is the ratio of unique words to total words; standardized type-token ratio (sttr), which is ttr calculated per 1,000-word segments to control for text length effects; and average sentence length (asl), measured by the word count of each target text segment corresponding to a source text sentence. to provide a more comprehensive assessment of lexical diversity, we also incorporate word entropy (we) as an additional indicator. while ttr and sttr measure vocabulary diversity based on the proportion of unique words relative to total words, they do not capture the randomness and unpredictability of word usage within a text. entropy, derived from information theory @xcite , quantifies the degree of uncertainty in word distribution, offering a more refined understanding of lexical variation in the translated text. a higher we value indicates greater word unpredictability and diversity, suggesting a richer and more varied vocabulary, though potentially increasing reading difficulty @xcite . our results of lexical diversity metrics, including ttr, sttr, asl, and we show that gpt-4o consistently outperforms ds-v3 across all four lexical diversity metrics when translating english academic papers on language, culture, and literature into chinese. gpt-4o shows a notably higher ttr (approximately 21.80% vs 19.99% for ds-v3), which suggests its translations contain a richer vocabulary variety. the average 9.06% higher ttr indicates gpt-4o produces less repetitive translations than ds-v3 in our dataset. in terms of sttr, gpt-4o maintains an advantage (approximately 42.24 vs 39.08 for ds-v3). the 8.08% higher sttr confirms that gpt-4o's higher lexical diversity is consistent even when controlling for text length. for asl, gpt-4o produces consistently longer sentences (around 26.5 words compared to 22.3 words for ds-v3). this nearly 19% difference in average sentence length aligns with our findings in regarding we, gpt-4o demonstrates slightly higher word entropy (approximately 5.85 vs 5.74 for ds-v3). the 1.78% higher we indicates gpt-4o translations have a more balanced distribution of word frequencies, leading to a richer and more varied vocabulary. this suggests that gpt-4o tends to produce more information-rich content with less predictable word choices. as illustrated in the radar chart of figure 2 , the differences between these two models are not uniform across all metrics. the most pronounced differences appear in asl and ttr, while we shows the smallest difference. this pattern not only suggests that gpt-4o employs a broader vocabulary range, but may also indicate that its translation approach preserves more complex sentence structures of the original texts than ds-v3. we will further examine this possibility through syntactic complexity metrics in the following section. ## comparison of syntactic complexity between translations of ds-v3 and gpt-4o in this study, we computed five syntactic metrics for each of the 3,498 segments in our dataset, including the incomplete dependency theory metric (idt), dependency locality theory metric (dlt), combined idt+dlt metric (idt+dlt), left-embeddedness (le), and nested nouns distance (nnd). the idt, dlt, and idt+dlt metrics are based on linguistic complexity theories derived from gibson's incomplete dependency theory (idt) and dependency locality theory (dlt) @xcite . the le metric is adapted with slight modifications from coh-metrix analysis @xcite , while nnd was introduced by @xcite . le and nnd were selected due to their relevance in capturing syntactic differences between english and chinese @xcite . unlike previous studies that utilize categorical proof nets @xcite for syntactic representation, these metrics adopt universal dependencies @xcite . this framework provides a consistent approach to annotating grammar, encompassing part-of-speech tagging, morphological features, and syntactic dependencies. additionally, blache's reformulation of incremental dependency theory (idt) and dependency length theory (dlt) @xcite is applied for analyzing dependency relations. to parse segments in our dataset, stanford stanza @xcite is employed to generate dependency trees. the decision to use dependency tree parsing over categorical proof nets is primarily driven by the availability of high-quality, scalable parsers like stanza, which support a wide range of languages. furthermore, previous research has demonstrated that dependency trees yield reliable and interpretable results, reinforcing their suitability for this study. the definition and implementation of these metrics are detailed in @xcite ; @xcite . developing these metrics allows us to assess whether gpt-4o retains more syntactic complexity of the sts compared to ds-v3. additionally, it enables us to examine whether the syntactic complexity of the sts has a greater influence on translations generated by gpt-4o versus ds-v3 in the context of english-to-chinese academic translations of papers on language, culture, and literature. the results shown in figure 3 reveal that gpt-4o consistently exhibits higher syntactic complexity than ds-v3 across all five measured metrics. these differences are statistically significant (@xmath4.05) for all metrics, suggesting a systematic difference in the linguistic structures produced by the two models. these differences may stem from the models' underlying architectures and training data. incomplete dependency theory (idt) counts the number of incomplete dependencies between tokens. gpt-4o shows significantly higher average idt values (136.30) compared to ds-v3 (122.58), representing a 10.07% difference. the higher idt values suggest that gpt-4o's outputs contain more complex syntactic structures with greater numbers of incomplete dependencies that span across the target texts. this may manifest as more sophisticated sentence constructions with multiple embedded clauses or modifying phrases. the dependency locality theory metric (dlt) metric, which counts discourse referents (nouns, proper nouns, and verbs) between a head token and its longest leftmost dependent, is 5.28% higher in gpt-4o (13.12) compared to ds-v3 (12.43). the higher dlt values indicate gpt-4o produces target texts with longer dependency distances containing more nouns and verbs between head words and their dependents, potentially creating greater processing demands on target readers. not surprisingly, the combined metric shows the values of gpt-4o (149.42) are on average 9.65% higher than ds-v3 (135.01). this comprehensive measure reinforces that gpt-4o's target texts con-tain both more incomplete dependencies and more discourse referents within those dependencies. the significant higher values in this combined metric suggest gpt-4o's tendency toward greater overall syntactic complexity. the left-embeddedness (le) metric, which counts non-verb tokens before the main verb, shows a significant 9.89% higher value in gpt-4o (22.07) compared to ds-v3 (19.89). this difference is particularly meaningful for translations for the english-to-chinese language pair because chinese syntax fundamentally relies on left-embedded structures where substantial information is placed before the main verb. topicalization in chinese requires placing important contextual elements at the beginning of sentences, and temporal, locative, and adverbial information naturally precedes the predicate. gpt-4o's significantly higher le score might indicate it better captures this essential characteristic of chinese syntax, producing target text that follows the natural information flow patterns expected by the audience of native chinese speakers. the nested nouns distance (nnd) metric shows the largest percentage difference (13.20%) between the two models, with gpt-4o scoring 3.54 compared to ds-v3's 3.08. this metric is particularly relevant for english-to-chinese translation because chinese noun phrases follow different structural patterns than english, with modifiers strictly preceding the head noun. chinese permits complex nested nominal structures with multiple embedded modifiers, and the distance relationships between nested nouns follow language-specific conventions. gpt-4o's significantly higher nnd suggests that it might better replicates the characteristic distances and relationships between nested nominal elements in chinese. this may contribute to more authenticsounding modifier structures and more natural nominal phrases that align with native chinese linguistic expectations. ## discussion and conclusion this study has examined the performance of chatgpt-4o and deepseek-v3 in translating english academic papers on language, literature, and culture to chinese at the discourse level. our analysis reveals that deepseek-v3 demonstrates better overall translation quality than chatgpt-4o according to automatic quality estimation results from comet-kiwi. deepseek-v3 achieved a higher average score (0.7790 versus 0.7655) and showed more consistent performance across different source texts, suggesting it may offer more reliable quality across diverse topics and complexity levels. interestingly, despite its lower comet-kiwi scores, gpt-4o exhibits greater lexical richness according to all four indicators we measured. gpt-4o consistently produced translations with higher typetoken ratio (21.80% versus 19.99%), standardized type-token ratio (42.24 versus 39.08), average sentence length (26.5 versus 22.3 words), and word entropy (5.85 versus 5.74). this suggests gpt-4o translations contain a more varied vocabulary and less repetitive language patterns than deepseek-v3. we also found that gpt-4o's translations have greater syntactic complexity across all five segment-level metrics we examined. the higher values in incomplete dependency theory (idt), dependency locality theory (dlt), combined idt+dlt, left-embeddedness (le), and nested nouns distance (nnd) metrics indicate gpt-4o produces more complex syntactic structures. particularly notable are the higher scores in le and nnd, which were specifically selected as english-chinese pair-specific metrics. the higher syntactic complexity in le and nnd metrics might correspond to more authentic chinese patterns, but could alternatively reflect unnecessarily complicated structures that native speakers would find awkward or unnatural. whether this increased complexity actually results in more naturalsounding chinese would require human evaluation by native speakers of simplified chinese, as syntactic complexity metrics alone cannot definitively determine naturalness or fluency in the target language. our findings contribute to the growing body of research on what we term "genaiese" -the distinctive linguistic characteristics of text generated by large language models. just as previous research identified "translationese" in statistical and neural mt outputs, our study suggests that different llm architectures may produce systematically different translation patterns that could potentially be identi-