The Devil is in the Details: Assessing the Effects of Machine-Translation on LLM Performance in Domain-Specific Texts (2025)

Abstract: Conflict scholars increasingly use computational tools to track violence and cooperation at a global scale. To study foreign locations, researchers often use machine translation (MT) tools, but rarely evaluate the quality of the MT output or its effects on Large Language Model (LLM) performance. Using a domain-specific multi-lingual parallel corpus, this study evaluates the quality of several MT tools for text in English, Arabic, and Spanish. Using ConfliBERT, a domain-specific LLM, the study evaluates the effect of MT texts on model performance, and finds that MT texts tend to yield better results than native texts. The MT quality assessment reveals considerable translation-induced distortions, reductions in vocabulary size and text specialization, and changes in syntactical structure. Regression analysis at the sentence-level reveals that such distortions, particularly reductions in general and domain vocabulary rarity, artificially boost LLM performance by simplifying the MT output. This finding cautions researchers and practitioners about uncritically relying on MT tools without considering MT-induced data loss.

## introduction political scientists, like many other domain-specific users, often rely on computational tools to make sense of large volumes of data. in particular, conflict scholars increasingly use computational methods to analyze global dynamics of political conflict and cooperation in foreign locations. to do so, researchers frequently rely on machine translations (mt) to translate political text from different languages @xcite . despite the growing research on mt quality @xcite @xcite , social scientists seldom evaluate the quality of the mt output nor its consequences on model performance. careful researchers may be concerned about mt quality due to data loss or incorrect translations, particularly for low-resource languages @xcite @xcite or specialized domains requiring precise terminology @xcite . mt-induced changes to the source text, known as translationese effect @xcite , may result in considerable alterations of the output text, making translationese especially crucial to investigate in domain-specific translations where seemingly minor distortions of the output text may lead to incorrect inference. moreover, there is little work analyzing the impact of mt quality on large language model (llm) performance @xcite . consequently, the quality of the mt text often gets overlooked, and its effects on llm performance remain ignored. for researchers tracking conflict around the world, disregarding translationese or its effects on llm performance may lead to missing important signals about security threats or cooperation. by using a multilingual parallel corpus from the united nations @xcite , this study analyzes the quality of various mt tools for english, arabic, and spanish and evaluates the effects of mt distortions on llm performance on tasks related to political conflict and cooperation. in particular, the study evaluates four mt tools, google translate (gt) (google cloud, 2024), deepl translate (deepl) @xcite , google translate within the deep learning @xcite , and opus machine translation (opus) @xcite , and evaluates the performance of their mt outputs using conflibert @xcite , a domainspecific llm specialized on political conflict. this research offers several contributions. the study carefully evaluates the quality of various mt tools using a domain-specific parallel corpus in english, arabic, and spanish. contrary to the ex-pectation that llms work better when processing native-speaker written/translated texts (nst), experiments in this study indicate that conflibert performs better using mt output. by disentangling sentence-level characteristics, the analysis reveals mt distortions related to nouns, verbs, lemmas, vocabulary complexity, and sentence structure. the study further explores the effects of mt distortions on llm performance at the sentence level using regression analysis. the results reveal that mt distortions, primarily the vocabulary loss of general and domain-specific rarity, generate simplified text representations that artificially boost model performance, particularly for translated text from arabic and spanish to english. such simplification favors model performance on mt output over nst. these results represent a double-edged sword for researchers using mt tools who may face a tradeoff between achieving higher llm performance at the expense of domain-specific words that may be relevant to their subject of study. ## related works pre-trained language models (plm) such as bert @xcite achieve great results by using continued pre-training on domain-specific data to capture its unique vocabulary, semantics, and language style @xcite . taking advantage of this capability, political scientists use different models to study party manifestos @xcite , voter partisanship @xcite , social movements @xcite @xcite , dictionary development @xcite , codebook-based classification (halterman and keith, 2024) and annotations @xcite , among other tasks. similarly, conflict scholars use specialized language to study political violence and cooperation. for that purpose, conflibert @xcite is a domain-specific model specialized on political conflict that yields superior performance compared to generic llms. researchers using non-english text generally rely on mt tools to pre-process the original text. however, mts can heterogeneously distort the data, thus affecting the model performance @xcite . assessing mts from spanish to english, previous research shows a net summarization of the original text, which reduces the verboseness of the original text and results in an adaptation of the target text to linguistic characteristics of en-glish. this adaptation to english linguistic standards yields high quality-metric results for mt text @xcite . the summarization effect can further artificially enhance conflibert en's performance, as the english language generally favors more concise text @xcite . when evaluating the translation quality from arabic to english, @xcite found that mts artificially extend the source text. this data increase in mts from arabic is penalized in english, as metrics show a notable decrease in translation quality relative to the original arabic text. however, this data increase appears to introduce more linguistic elements that artificially boost conflib-ert en's performance on the mt corpus. the predominant body of research is in favor of languages with abundant resources; thus, more recent studies use translation tools to mitigate the scarcity of training data, including @xcite , which used google cloud (2024) (gt) to translate official transcripts of european parliament debates written in the official majority of the eu's languages into english. to improve inference in prompting multilingual llms, @xcite translated from languages that are comparatively less represented in available llms, like spanish, into english to leverage the fact that english makes up the majority of training data in multilingual llms. other recent studies further compare translation tools @xcite @xcite and quality translation metrics @xcite @xcite . ## data and annotations this research uses the united nations parallel corpus (unpc) @xcite , containing 86,307 official united nations (un) security council documents translated by professional un translators. since the un operates in six official languages, these translations are considered the gold standard record (gsr). out of the official un languages, this study uses nst texts written in english (en), spanish (es), and arabic (ar). in total, the unpc contains 11,365,709 fully aligned sentences across languages. this study uses a random sample of 11,326 sentences from un security council documents related to human rights, the protection of civilians, and terrorism. the resulting sample provides a uniquely valuable multilingual parallel corpus in the domain of political conflict and co-operation. having the same gsr content across multiple nst sentences within the unpc provides a ceteris paribus leveled field to compare the effects of different mt tools on model performance. this study uses the unpc sentences previously annotated by @xcite , 1 which classify the content of the sentences according to plover (open event data alliance, 2018), an ontology often used in political science to categorize different types of material and verbal interactions based on the cooperative or conflictive conduct of the parties involved. annotators classified the full sample of sentences according to three tasks. relevance is a binary classification identifying whether a sentence is relevant for political conflict or cooperation or not. quadclass is a multi-class classification task categorizing whether the sentences indicate verbal conflict, verbal cooperation, material conflict, material cooperation, or non-relevant sentences. finally, binquad is a binary classification for each quad-class category identified above, indicating whether the sentence can be categorized as the respective plover category or not, thereby representing one of the other three categories. the annotations have the following distributions 2 . in the relevance binary task, coders identified 52% sentences as not relevant and the rest 48% as relevant. for the multi-class quadclass task, coders identified 14% sentences as material conflict, 13% as material cooperation, 8% as verbal conflict, 11% as verbal cooperation, and 53% as not relevant. finally, the binquad binary task of quadclass categories produced the following distribution for material conflict (yes 14%, no 86%), material cooperation (yes 13%, no 87%), verbal conflict (yes 8%, no 92%), and verbal cooperation (yes 11%, no 89%). all experiments used balanced datasets, with the number of randomly selected sentences capped to match the smallest category size in each task. ## translation quality assessment using unpc text in english (en), spanish (es), and arabic (ar), we conduct a series of mts using different tools. our analysis uses bidirectional mt to convert the entire sample of spanish and arabic texts into english (es to en, ar to en) and vice versa (en to es, en to ar). to conduct the translations, we use four commonly used building on @xcite , we use four quality assessment metrics: sacrebleu @xcite , meteor @xcite , bertscore @xcite , and comet (crosslingual optimized metric for evaluation of translation), using the wmt20-comet-da model @xcite . sacrebleu, and meteor are lexical-based metrics measuring the similarity between nst and mt text using mathematical or heuristic methods, comet is a neural-based metric @xcite , whereas bertscore uses an embedding-based metric that relies on deep learning methods @xcite . quality scores range from 0 to 1, with high values indicating greater nst-mt similarity @xcite . the metrics can be ranked according to their degree of flexibility. sacrebleu employs the moses tokenizer, an advanced preprocessing tool that facilitates score comparability and was first created for the moses statistical machine translation system @xcite . the moses tokenizer uses heuristics and rules unique to a given language to normalize text and to handle punctuation or special characters. meteor is more flexible and calculates the similarity of word alignments. comet is a state-of-the-art neural-based mt evaluation metric. bertscore is the most flexible metric as it considers contextual correctness and synonyms. using each unpc nst text as a reference, figure 1 presents the quality scores from the different metrics applied to each mt output. results show that different tools generate varying degrees of quality across languages. while for ar to en and es to en mts, sacrebleu, meteor, and bertscore indicate that deepl provides the best quality output, comet considers opus to be the most accurate mt tool for these language combinations. for en to es, opus yields the best mt quality based on all metrics, while for en to ar, comet disagrees with all other metrics and considers deepl the best-performing mt tool. while these metrics offer a first assessment of the mt quality, they do not permit an in-depth understanding of mt-induced translationese effects on the source text or assess more subtle changes in meaning and nuance. consequently, these metrics do not fully capture whether mt-induced changes influence llm performance. the following section evaluates llm performance across mt texts to see if the results align with quality assessment suggestions. ## model performance across mt tools following the quality assessment, we test the effect of mts on llm performance. to do so, we use conflibert @xcite , a domain-specific pre-trained language model specifically designed to analyze political texts, to evaluate the unpc nst and mt texts for three classification tasks: relevant (binary) classification, quadclass (multiclass) classification, and binquad (binary) classifi-figure 1: quality assessment metrics cation of each quadclass category. for each task, the fine-tuning uses three versions of the conflib-ert family, namely conflibert arabic @xcite , conflibert spanish @xcite , and conflibert english @xcite , with cased and uncased variations, resulting in a total of 6 different models. all models use balanced datasets for each task. by keeping the unpc content and the use of conflibert constant, we analyze variations in performance derived from different mt tools, including deep, deepl, gt, opus, and the nst texts. first, we split the data into training, testing, and developing using 70-15-15 rule. second, for each model, we perform the evaluation using 10 seeds and 5 epochs. finally, we run a total of 114 fine-tuning tasks on those models and their corresponding datasets. we used a hpc system with a single a100 gpu 20gb and a single v100 gpu 32gb, and a learning rate of 4e-05, with a training batch size of 8 and a maximum sequence length of 512 for both binary and multiclass classifications. figures 2 3 4 present the f1 scores highlighting the top-performing models in red. overall, results show that processing mt text yields better results than analyzing nst text. this is puzzling since domain-specific models would be expected to perform better with nst texts. ## corpus rarity and vocabulary loss to disentangle the characteristics of mt outputs that yield marginally superior conflibert performance compared to processing nst texts, this section analyzes mt-induced distortions to the original corpora. first, we measure the total vocabulary size after preprocessing using spacy's en_core_web_trf transformer pipeline for english @xcite ), spacy's es_dep_news_trf transformer pipeline for spanish @xcite , and the farasa segmenter (al-shaibani, 2021) for arabic. the vocabulary size for each language represents the total number of unique words included in the mt corpora. figure 5 shows the vocabulary sizes. we find that the mt corpora consistently have a lower vocabulary size than the respective nst corpus. this finding aligns with the characteristics of translationese, where translated text tends to show reduced lexical diversity compared to original text @xcite . therefore, there may be a convergence of mt on similar phrasing, reducing the need to learn diverse patterns as in the native text. to further explore this reduction effect, we measure lexical rarity per sentence. following (proisl, 2022), we define lexical rarity as the proportion of tokens in a text that does not appear in the 5,000 most common tokens for a domain. we consider two types of rarity: general and domain. general rarity relies on the 5,000 most common tokens for a language, regardless of subject. domain rarity uses the 5,000 most common tokens in the sentences from the unpc to measure rarity as it relates to a political corpus. we use rarity as a proxy for lexical complexity and consider it the prime indicator for the reduction of text complexity and loss of context in the mt texts. a reduction in rarity for mt text represents a decline in the number of unique tokens compared to the nst text. consequently, a reduction in the mean rarity of the mt corpus represents a loss of language complexity compared to the nst corpus. the loss of rarity may be particularly relevant for domain-specific researchers where key terms or technical words may carry particular substantive value. in addition to rarity, we measure the number of unique lemmas, nouns, and verbs in each sentence as additional measures of linguistic features (see appendix h). figure 6 shows the mean general and domain rarity scores. using a pairwise wilcox test from the stats r package (r core team, 2023), we compare the mt text to the nst corpora in terms of rarity and find that the english and arabic mts all have statistically significantly lower general and domain rarity scores than the respective nst corpus. spanish, however, does not display the same effect. in contrast, mts using deep, deepl, and gt all have statistically significant higher general rarity scores than the spanish nst corpus. however, regarding domain rarity, the difference between the spanish nst text and the mt output using deep, deepl, and gt is not significant. opus translations are not significantly different from spanish nst text in general rarity, but show a statistically significant reduction in domain rarity. these results and the reduced vocabulary size show that mts into english and arabic generate in a significant loss of rare tokens. while this loss may simplify the text and facilitate model classification, these translationese-induced simplifications may lead to the loss of critical context where specific words and their substantive meaning are essential. the significant increase in rarity for some mt tools into spanish is likely due to a bias toward brevity using more complex words. while native speakers may opt for longer but simpler phrasing in nst text, the mt text may result in a brief but vocabulary-heavy phrasing. this may be due to the training data for translation into spanish favoring these characteristics. the reduction in text complexity further indicates that there could be overfitting to mt text in fine-tuning. models trained on the mt text, which has lower text complexity, may not perform well when tasked with classifying nst text or even text from another mt tool that introduces higher or different types of complexity. this finding, consequently, warrants additional consideration when fine-tuning using mt text. these findings resonate with major challenges in neural machine translation (nmt). first, nmt systems perform poorly in specialized domains for which the system has not been trained for. second, nmt systems are weak at translating lowfrequency (rare) words, especially in cases where there are many inflections (as in verb conjugations in spanish). third, nmt systems struggle with long sentences, which are disproportionately underrepresented in the unpc @xcite . finally, @xcite similarly find that mt systems fail to reach the diversity of phrasing and vocabulary of natural human language. therefore, there is a loss of information, context, and the unique voice of the speaker/writer of the source text in this process. while event classification may not suffer from this loss, other tasks, such as named entity recognition (ner), may experience poorer performance using mt. ## dependency distance to explore the mt effects on model performance, the study also analyzes the dependency distance mean (ddm) of each sentence across languages and the dependency distance mean difference (ddm d ) of each mt output to their corresponding nst text. ddm is the average syntactical distance between the root of a sentence to other parts of speech and is generally regarded as an indicator of sentence complexity @xcite . a high ddm refers to highly complex sentences. relatedly, ddm d is interpreted here as the distortion caused by the mt tool when compared to its target nst text, such that a negative ddm d indicates syntactical simplification and a positive ddm d shows increasing syntactical complexity by the mt tool. to get the ddm, we use textdescriptives, spacy, spacy_transformers, and libraries with en_core_web_sm, bert-large-arabertv02, and es_core_news_sm models for english, arabic, and spanish, respectively. for example, ## sentence-level prediction confidence to better understand conflibert's performance across nst and mt texts, we analyze the effects of different sentence-level characteristics on model performance. we first estimate the degree of con to calculate prediction confidence at the sentence level, we use the conflibert-uncased model in the binary relevant classification task. the methodology generates label predictions and confidence scores by applying the softmax function to its output logits @xcite . the methodology processes the logits of each sentence through softmax, converting them into probabilities ranging from (0,1), thus indicating the model's confidence in correctly assigning the chunk to a specific class. in this way, the prediction reflects the probability of conflibert's correct classification. for sentences longer than 512 tokens, we apply a chunking strategy, splitting them into segments of 512 tokens each @xcite , and independently classify and generate a predicted label and confidence score for each segment. to ensure an accurate sentence-level prediction, we apply majority voting to determine the final label and average the confidence scores across all chunks in a sentence. this method ensures that the final confidence prediction reflects the model's certainty across the entire sentence, allowing us to handle longer texts without losing context or compromising accuracy. averaging the confidence scores provides a robust measure of the model's overall confidence. ## explaining model performance to explore the determinants of model performance, we further analyze the sentence-level prediction confidence for the binary classification task using a linear regression model as indicated in equation 1. following @xcite and brandt et al. (2022), we evaluate the contribution of each variable on the probability of correct classification by comparing the contribution of each sentence-level characteristic to the regression root mean standard error (rmse) using stepwise elimination. rmse is the standard deviation of the residuals away from the regression line. a low rmse indicates that the observations closely revolve around the regression line, which suggests a good model fit. the stepwise elimination approach consists of first running the full regression and calculating the rmse, then dropping one variable at a time from equation 1 and comparing the change in the rmse from each subsequent model. a large rmse increase after eliminating a certain variable indicates a greater model fit loss, suggesting that this variable largely contributes to the model performance. since each regression has its own rmse (see appendix k), we favor the comparability of results by calculating the model fit loss as a percentage using as baseline the full model's rmse. this provides a standardized measure for cross-model comparison in which lower model fit loss values indicate worse model performance after each variable elimination. figure 7 presents the model fit loss by stepwise elimination across native and mt texts. the baseline in each panel is the full model rmse from equation 1. the model fit gradually decreases after subsequently dropping each independent variable in each elimination step; the magnitude of the drop indicates the contribution of each eliminated variable. in general, figure 7 shows that all models experience substantial performance loss after eliminating the general and domain rarity variables. this shows that general rarity and domain-specific rarity have considerable leverage in explaining conflib-ert performance for binary classification. text translated using opus seems particularly sensitive to the contributions of general and domain rarity in any translation direction. these results offer an important finding suggesting that highly specialized words are crucial for explaining model performance. as mt tools generally reduce the vocabulary richness (see figure 5 ) and decrease the number of specialized or domainspecific terms in the mt text (see figure 6 ), the resulting translation output is a simplified representation of the native text containing sentences with fewer tokens and simpler words. this mt text generally makes it easier for llms to process. however, this performance gain comes at the cost of lower vocabulary richness in key terms. ## discussion and conclusion third, although llms are expected to perform better with native documents than with mt texts, results across downstream tasks indicate that llms generally perform better with mt texts than with native corpora. yet, no single mt tool consistently reports the top performance across languages. finally, using regression analysis and a stepwise deletion approach to assess the contributions of different sentence-level characteristics on model performance, the analysis indicates that highly specialized words-represented by general and domainspecific rarity-have the most leverage in explaining model performance for binary tasks. based on a specific application in the field of political science, this study suggests an important trade-off for the use of mt tools that could be extended to other domains. on the one hand, results indicate that mt tools may substantially reduce the time and effort for human analysis to process large volumes of text, and such mt texts tend to yield better results when using specialized llms for a variety of tasks. in simple terms, it seems that machines talking to machines tend to generate better results. on the other hand, the use of mt tools tends to produce translationese outputs that reduce vocabulary richness, particularly for rare terms that may be of high substantive value to domain experts. for human translators operating in highly technical fields, such vocabulary loss may prove unacceptable despite the artificially superior machine-to-machine performance.