Automatically Generating C hinese Homophone Words to Probe Machine Translation Estimation Systems (2025)

Abstract: Evaluating machine translation (MT) of user-generated content (UGC) involves unique challenges such as checking whether the nuance of emotions from the source are preserved in the target text. Recent studies have proposed emotion-related datasets, frameworks and models to automatically evaluate MT quality of Chinese UGC, without relying on reference translations. However, whether these models are robust to the challenge of preserving emotional nuances has been left largely unexplored. To this end, we introduce a novel method inspired by information theory which generates challenging Chinese homophone words related to emotions, by leveraging the concept of *self-information*. Our approach generates homophones that were observed to cause translation errors in emotion preservation, and exposes vulnerabilities in MT models struggling to preserve relevant emotions. We evaluate the efficacy of our method using human evaluation and compare it with an existing one, showing that our method achieves higher correlation with human judgments. The generated Chinese homophones, along with their manual translations, are utilized to generate perturbations and to probe the robustness of existing quality evaluation models, including models trained using multi-task learning, fine-tuned variants of multilingual language models, as well as large language models (LLMs). Our results indicate that LLMs with larger size exhibit higher stability and robustness to such perturbations. We release our data and code for reproducibility and further research.

## introduction machine translation (mt) of chinese-english news articles has been claimed to achieve human parity in recent years @xcite . however, research on machine translation of user-generated 1 https://github.com/surrey-nlp/homo_gen content (ugc) like tweets has revealed additional challenges including problems with handling slang, emotion, and literary devices like sarcasm and euphemisms @xcite , as shown in the example translated by chatgpt foot_0 and google translate in figure 1 . evaluating mt quality of such texts has become a challenging and urgent task for the improvement their translation quality @xcite . traditional ways of evaluating mt quality involve metrics such as bleu @xcite , bleurt @xcite or bertscore @xcite to compare the mt output with one or several reference translations. when references are unavailable, quality estimation (qe) methods are often used to predict scores which approximate human evaluation @xcite . one approach for qe is fine-tuning multilingual pre-trained language models (ptlms) using human evaluation scores. frameworks like multi-dimensional quality metrics (mqm) @xcite , an error-based evaluation scheme, are commonly employed to obtain the human evaluation scores for this purpose. for machine translation of ugc, @xcite recruited professional translators to evaluate translations of a chinese ugc dataset using google translate, based on an mqm-adapted framework. they found that homophone slang words used by netizens are the most common cause of errors in the translation of emotions. they proposed different types of qe models based on fine-tuning, multi-task learning (mtl) and large language models (llms) for automatic evaluation @xcite and claimed that their models achieved state-of-the-art performance in evaluating mt quality of ugc. in this paper, we investigate whether their models are robust enough to cope with newly generated homophones or human-figure 1 : an example of the challenges for translating chinese ugc improved translations. in this regard, we propose a method to automatically generate chinese homophone words to probe the robustness of these qe systems towards new homophone words and human-improved translations. our contributions can be summarized as follows: the rest of the paper is organized as follows: section 2 reviews related work on quality evaluation of ugc and chinese homophone words. section 3 introduces the main dataset used in this study. section 4 details the existing generation approach, our proposed method, and the human evaluation and perturbation methods. section 5 presents and discusses the results of these evaluations. section 6 concludes the study and outlines future directions, while section 7 addresses limitations and ethical considerations. ## related work section 2.1 provides an overview of related work on the evaluation of ugc translation, and section 2.2 explores studies focused on chinese ugc and the generation of homophones. ## data we used the human annotated dataset for quality assessment of emotion translation (hadqaet) foot_1 from @xcite to sample ugc that contains chinese homophone slang for automatic generation. hadqaet was chosen because, 1) its source texts contain many homophone slang; 2) it has quality evaluation data such as qe scores for the mt texts, error words related to emotion preservation and reference translations, and 3) there are qe systems trained on it (explained in § 4.3). the source texts of hadqaet originated from the dataset released by the evaluation of weibo emotion classification technology on the ninth china national conference on social media processing (smp2020-ewect). it originally has a size of 34,768 instances. each instance is a tweetlike text segment in chinese, which was manually annotated with one of the six emotion labels, i.e., anger, joy, sadness, surprise, fear and neutral @xcite . @xcite randomly kept 5,538 instances and used google translate to translate them to english. to evaluate translation quality for emotion preservation, they proposed an emotion-related mqm framework and recruited two professional translators to annotate errors and their corresponding severity. words/characters in both source and target that cause errors were highlighted for error analysis. in addition, they hired a translation company to post-edit the mt output to get reference translations @xcite . more details about hadqaet can be found in @xcite . we tokenized the source texts using jieba (sun, 2013) and extracted the words that were highlighted as causes of error. following @xcite , we made a frequency list of these error words and picked those that contain homophone slang with a frequency higher than 10, under the supervision of a chinese native speaker. this produced a list of 5 homophone slang words (as shown in ## methodology this section presents our methodology for homophone generation and the evaluation of generated homophones in § 4.1 and § 4.2, respectively. qe models for robustness test as well as the perturbation methods are elaborated in § 4.3 and § 4.4. ## results and discussion this section presents and discusses the results of evaluation of our generated homophone words as well as the results of our perturbation methods. ## conclusion and future work this paper investigates how robust emotion-related qe systems are towards emotion-loaded homophone words. for this purpose, we proposed to use self-information to automatically generate and select chinese homophone words that frequently cause translation errors. we evaluated the efficacy of our method based on human evaluation and compared it with the baseline, percentile score. we find that our method can achieve higher correlation with human evaluation than the baseline. we picked 5 generated homophone words and replaced the original homophones with our generated ones in the source as perturbations to test the robustness of the qe systems trained by @xcite , including fine-tuned cometkiwi, transquest and mtl models as well as llms. at the same time, we replaced the mt output with improved translations to test how robust qe systems are towards improved translations. our results indicate that llms with a larger size such as deepseek-67b exhibited great stability and robustness to all our perturbation groups. for future work, we plan to generate homophones at a larger scale and invite more linguists to evaluate their usefulness in real-world scenarios on social media.