Modeling Bilingual Conversational Characteristics for Neural Chat Translation (2021)

Abstract: Neural chat translation aims to translate bilingual conversational text, which has a broad application in international exchanges and cooperation. Despite the impressive performance of sentence-level and context-aware Neural Machine Translation (NMT), there still remain challenges to translate bilingual conversational text due to its inherent characteristics such as role preference, dialogue coherence, and translation consistency. In this paper, we aim to promote the translation quality of conversational text by modeling the above properties. Specifically, we design three latent variational modules to learn the distributions of bilingual conversational characteristics. Through sampling from these learned distributions, the latent variables, tailored for role preference, dialogue coherence, and translation consistency, are incorporated into the NMT model for better translation. We evaluate our approach on the benchmark dataset BConTrasT (English<->German) and a self-collected bilingual dialogue corpus, named BMELD (English<->Chinese). Extensive experiments show that our approach notably boosts the performance over strong baselines by a large margin and significantly surpasses some state-of-the-art context-aware NMT models in terms of BLEU and TER. Additionally, we make the BMELD dataset publicly available for the research community.

## introduction a conversation may involve participants that speak in different languages (e.g., one speaking in english and another in chinese). fig. 1 shows an example, where the english role r 1 and the chinese role r 2 are talking about the "boat". the goal of chat translation is to translate bilingual conversational text, i.e., converting one participant's language (e.g., english) to another's (e.g., chinese) and vice versa @xcite . it enables multiple speakers to communicate with each other in their native languages, which has a wide application in industry-level services. although sentence-level neural machine translation (nmt) @xcite @xcite @xcite has achieved promising progress, it still faces challenges in accurately translating conversational text due to abandoning the dialogue history, which leads to role-irrelevant, incoherent and inconsistent translations @xcite @xcite . further, context-aware nmt @xcite @xcite @xcite @xcite can be directly applied to chat translation through incorporating the dialogue history but cannot obtain satisfactory results in this sce-nario @xcite . one important reason is the lack of explicitly modeling the inherent bilingual conversational characteristics, e.g., role preference, dialogue coherence, and translation consistency, as pointed out by @xcite . for a conversation, its dialogue history contains rich role preference information such as emotion, style, and humor, which is beneficial to rolerelevant utterance generation @xcite . as shown in fig. 1 , the utterances x 1 , x 3 and x 5 from role r 1 always have strong emotions (i.e., joy) because of his/her preference, and preserving the same preference information across languages can help raise emotional resonance and mutual understanding @xcite . meanwhile, there exists semantic coherence in the conversation, as the solid green arrow in fig. 1 , where the utterance x 5 naturally and semantically connects with the dialogue history (x 1∼4 ) on the topic "boat". in addition, the bilingual conversation exhibits translation consistency, where the correct lexical choice to translate the current utterance might have appeared in preceding turns. for instance, the word "sail" in x 1 is translated into "jiàchuán", and thus the word "sailing" in x 3 should be mapped into "jiàchuán" rather than other words (e.g., "hángxíng" foot_0 ) to maintain translation consistency. on the contrary, if we ignore these characteristics, translations might be role-irrelevant, incoherent, inconsistent, and detrimental to further communication like the translation produced by the "s-nmt" in fig. 1 . although the translation is acceptable at the sentence level, it is abrupt at the bilingual conversation level. apparently, how to effectively exploit these bilingual conversational characteristics is one of the core issues in chat translation. and it is challenging to implicitly capture these properties by just incorporating the complex dialogue history into encoders due to lacking the relevant information guidance @xcite . on the other hand, the conditional variational auto-encoder (cvae) @xcite has shown its superiority in learning distributions of data properties, which is often utilized to model the diversity @xcite , coherence @xcite and users' personalities @xcite , etc. in spite of its success, adapting it to chat translation is non-trivial, especially involving multiple tailored latent variables. therefore, in this paper, we propose a model, named cpcc, to capture role preference, dialogue coherence, and translation consistency with latent variables learned by the cvae for neural chat translation. cpcc contains three specific latent variational modules to learn the distributions of role preference, dialogue coherence, and translation consistency, respectively. specifically, we firstly use one role-tailored latent variable, sampled from the learned distribution conditioned only on the utterances from this role, to preserve preference. then, we utilize another latent variable, generated by the distribution conditioned on source-language dialogue history, to maintain coherence. finally, we leverage the last latent variable, generated by the distribution conditioned on paired bilingual conversational utterances, to keep translation consistency. as a result, these tailored latent variables allow our cpcc to produce role-specific, coherent, and consistent translations, and hence make the bilingual conversation go fluently. we conduct experiments on wmt20 chat translation dataset: bcontrast (en⇔de foot_1 ) @xcite and a self-collected dialogue corpus: bmeld (en⇔ch). results demonstrate that our model achieves consistent improvements in four directions in terms of bleu @xcite and ter @xcite , showing its effectiveness and generalizability. human evaluation further suggests that our model effectively alleviates the issue of role-irrelevant, incoherent and inconsistent translations compared to other methods. our contributions are summarized as follows: ## chat nmt we aim to learn a model that can capture inherent characteristics in the bilingual dialogue history for producing high-quality translations, i.e., using the context for better translations (farajian y2: nǐhǎo,wǒ de yígè péngyǒu hé wǒ... x1: hi there, how can i help today? y3: dāngrán, shuǐzúguǎn shì fēicháng hǎo de ... y2k: xīyātú fùháo yǐngyuàn。 y1: nǐhǎo, jīntiān yǒu shénme né nggóu xiàoláo de? x2: hey. a friend of mine and i ... x3: sure, aquaman is a great ... x2k: seattle regal cinemas. x2k+1: great, regal cinemas... y2k+1: tàibàng le, dìwáng diànyǐngyuàn... turn number 1 2 3 2k source-language turns set cx target-language turns set cy 2k+1 ... r 1 r 1 r 1 r 2 r 2 ... ... nmt figure 2: a dialogue example (en⇔ch) when translating the utterance x 2k+1 where @xmath0, |t |-1 2 ] and t is the total number of turns (assumed to be odd here). before we dig into the details of how to utilize c, we define three types of context in c (as shown in fig. 2 ): (1) the set of previous role-specific source-language turns, denoted as c role @xmath1, x 3 , x 5 , ..., x 2k+1 } 4 where @xmath2, |t |-3 2 ] and t is the total number of turns; (2) the set of previous source-language turns, denoted as c @xmath3, x 2 , x 3 , ..., x 2k }; and (3) the set of previous target-language turns, denoted as ## our methodology fig. 3 demonstrates an overview of our model, consisting of five components: input representation, encoder, latent variational modules, decoder, and training objectives. specifically, we aim to model both dialogue and translation simultaneously. therefore, for the input representation ( § 4.1), we incorporate dialogue-level embeddings, i.e., role and dialogue turn embeddings, into the encoder ( § 4.2). then, we introduce three specific latent variational modules ( § 4.3) to learn the distributions for varied inherent bilingual characteristics. finally, we elaborate on how to incorporate the three tailored latent variables sampled from 4 c role @xmath4, y4, y6, ..., y 2k } is also role-specific utterances of the interlocutor, which is used to model the interlocutor's consistency in the reverse translation direction. here, we take one translation direction (i.e., en⇒ch) as an example. the distributions into the decoder ( § 4.4) and our two-stage training objectives ( § 4.5). ## analysis 6.1 ablation study we conduct ablation studies to investigate how well each tailored latent variable of our model works. when removing latent variables listed in tab. 3, we have the following findings. ## related work chat nmt. it only involves several researches due to the lack of human-annotated publicly available data @xcite . therefore, some existing work @xcite @xcite mainly pays attention to designing methods to automatically construct the subtitles corpus, which may contain noisy bilingual utterances. recently, farajian et al. (2020) organize the wmt20 chat translation task and first provide a human postedited corpus, where some teams investigate the effect of dialogue history and finally ensemble their models for higher ranks @xcite @xcite @xcite . as a synchronizing study, @xcite use multitask learning to auto-correct the translation error, such as pronoun dropping, punctuation dropping, and typos. unlike them, we focus on explicitly modeling role preference, dialogue coherence, and translation consistency with tailored latent variables to promote the translation quality. ## conclusion and future work we propose to model bilingual conversational characteristics through tailored latent variables for neural chat translation. experiments on en⇔de and en⇔ch directions show that our model notably improves translation quality on both bleu and ter metrics, showing its superiority and generalizability. human evaluation further verifies that our model yields role-specific, coherent, and consistent translations by incorporating tailored latent variables into nmt. moreover, we contribute a new bilingual dialogue data (bmeld, en⇔ch) with manual translations to the research community. in the future, we would like to explore the effect of multimodality and emotion on chat translation, which has been well studied in dialogue field @xcite . stages en⇒de de⇒en en⇒ch ch⇒en base the first stage 5d 7d 4d 3.5d fine-tuning stage 4h 5h 3h 2h big the first stage 10d 12d 7d 6d fine-tuning stage 4.5h 5.5h 4h 2.5h in the case of blind testing or online use (assumed dealing with en⇒de), since translations of target utterances (i.e., english) will not be given, an inverse de⇒en model is simultaneously trained and used to back-translate target utterances @xcite , similar to all tasks.