PROSE : A Pronoun Omission Solution for C hinese- E nglish Spoken Language Translation (2023)

Abstract: Neural Machine Translation (NMT) systems encounter a significant challenge when translating a pro-drop (‘pronoun-dropping’) language (e.g., Chinese) to a non-pro-drop one (e.g., English), since the pro-drop phenomenon demands NMT systems to recover omitted pronouns. This unique and crucial task, however, lacks sufficient datasets for benchmarking. To bridge this gap, we introduce PROSE, a new benchmark featured in diverse pro-drop instances for document-level Chinese-English spoken language translation. Furthermore, we conduct an in-depth investigation of the pro-drop phenomenon in spoken Chinese on this dataset, reconfirming that pro-drop reduces the performance of NMT systems in Chinese-English translation. To alleviate the negative impact introduced by pro-drop, we propose Mention-Aware Semantic Augmentation, a novel approach that leverages the semantic embedding of dropped pronouns to augment training pairs. Results from the experiments on four Chinese-English translation corpora show that our proposed method outperforms existing methods regarding omitted pronoun retrieval and overall translation quality.

## introduction in recent years, neural machine translation (nmt) technology has made significant progress in lowering communication barriers between individuals from different language backgrounds. however, nmt systems often struggle when translating sentences from a pro-drop ('pronoun-dropping') language, such as chinese, korean and japanese @xcite , to a non-pro-drop language, such as english, french and german @xcite ). while the pro-drop phenomenon has been widely studied in the research community (nagard figure 1 : examples of pro-drop in daily spoken chinese with corresponding english translations provided by our model and several mature commercial nmt systems (google, deepl, baidu and chatgpt, respectively, data collected on january 13th, 2023). the chinese pronouns in brackets are dropped, and the dropped subject and object pronouns are marked in red and green, respectively. and @xcite @xcite @xcite , advanced commercial nmt systems occasionally fail to faithfully recover dropped pronouns in the source language. in some cases, leaving missed pronouns unrecovered could result in severe semantic distortion and alter the intended meaning of the translated text, as demonstrated in figure 1 . to tackle this issue, researchers have proposed two primary strategies: (1) incorporating additional pro-drop resolution systems to provide supplementary syntactic information @xcite @xcite . for instance, @xcite modeled empty categories within the framework of governmentbinding theory; (2) treating pro-drop resolution as a regularization component of nmt task directly @xcite . this approach suggests filling dropped pronouns @xcite or predicting pro-drops in the chinese text encoder component of a seq2seq model @xcite . despite the studies done on resolving chinese pro-drop in nmt so far, relevant benchmarks evaluating the effectiveness of pro-drop mitigation are highly limited, and chinese-english spoken translation datasets with fine-grained annotation are even fewer. in this study, we present prose, a pronoun omission solution for chinese-english spoken language translation. to facilitate research in this area, we introduce a novel dataset for document-level chinese-english spoken language translation that includes abundant and diverse pro-drop instances with contextual and pro-drop annotations across four spoken language genres (talk, drama, movie, and vlog). the analysis of this dataset reveals that the negative impact of pro-drop on chinese-english spoken language translation. furthermore, we propose the mention-aware semantic augmentation approach, which utilizes a mention encoder to capture the semantic embedding of dropped pronouns and employs a mention-side data augmentation technique to generate additional training pairs. experiment results on four chinese-english translation corpora demonstrate that our proposed approach significantly increase translation quality and the recover rate of missed pronouns, in comparison with baseline methods on both automatic and human evaluation metrics. additionally, we conducted ablation studies to provide further insights on the effect of designated losses. our contributions are summarized as follows: ## dataset creation to mitigate the scarce of benchmarks evaluating pro-drop in chinese-english spoken language translation, we collect and construct a new benchmark, prose, a high-quality chinese-english bilingual dataset of four different genres, including talk, drama, movie and vlog. ## pronoun-dropping analysis to gain more insights into the phenomenon of prodrop in the translation of spoken chinese into english, we examine the prevalence of pro-drop in spoken chinese and its impact on the quality of chinese-english spoken language translation. spoken chinese contains more pro-drop than literary language formally, pro-drop refers to a reference position that is filled with amorphologically unrealized form, and is one of the most common referential options in many languages such as chinese @xcite , japanese @xcite , korean @xcite , and thai @xcite . previous studies have revealed that spoken chinese language tends to contain more pro-drops than literary language @xcite @xcite . however, quantitative studies on pro-drops in different genres of spoken chinese, remain scarce. as demonstrated in pro-drop harms the quality of chinese-english spoken language translation subjective and objective pronouns are frequently omitted in spoken chinese, but should be recovered in non-prodrop languages like english. the question arises whether the current nmt system is able to accurately translate spoken chinese sentences with dropped pronouns into english, a non-pro-drop language, as illustrated in figure 1 . terminology error 38.89% inaccurate translation 23.61% missing translation 15.28% ellipsis 11.11% polysemy 2.78% broken english 2.78% others 5.56% figure 2 shows the distribution of chinese-to-english translation errors in our online simultaneous machine translation system. the primary use case of our translation system is chinese-to-english translation (primarily spoken chinese) in meetings and conferences. moreover, we have experienced labeling experts to categorize the bad cases generated by the online system. it can be seen from figure 2 that the proportion of errors caused by pro-drop is relatively high, constituting more than 11% of all errors. this is one of the major factors contributing to the degradation of the translation quality of our system. to investigate the potential of reinstated pronouns in chinese spoken sentences to improve the quality of chinese-english spoken language translation, we conduct experiments using spoken chinese sentences with omitted pronouns complemented by humans. we first train a transformerbase @xcite model on the cwmt dataset, and then report the bleu @xcite scores with sacre-bleu foot_2 @xcite on test sets of our four cor-pora (i.e., talk, drama, movie and vlog). next, the spoken chinese in test sets that is detected as pro-drop are completed manually, as shown in the content in brackets in figure 1 . the experimental results before and after human completion are shown in ## conclusion this study provides valuable insights into the phenomenon of pro-drop in chinese and its impact on chinese-english spoken language translation. furthermore, we introduced a new dataset that improves upon existing corpora in terms of representativeness, diversity, and informational value. lastly, our proposed approach, mention-aware semantic augmentation, demonstrates superior performance over existing methods in addressing the challenges posed by pro-drops. our study underscores the critical importance of taking into account pro-drops in nmt systems, and offers valuable benchmarks and insights to guide future advancements in this field.