Context-aware Neural Machine Translation for E nglish- J apanese Business Scene Dialogues (2023)

Abstract: Despite the remarkable advancements in machine translation, the current sentence-level paradigm faces challenges when dealing with highly-contextual languages like Japanese. In this paper, we explore how context-awareness can improve the performance of the current Neural Machine Translation (NMT) models for English-Japanese business dialogues translation, and what kind of context provides meaningful information to improve translation. As business dialogue involves complex discourse phenomena but offers scarce training resources, we adapted a pretrained mBART model, finetuning on multi-sentence dialogue data, which allows us to experiment with different contexts. We investigate the impact of larger context sizes and propose novel context tokens encoding extra-sentential information, such as speaker turn and scene type. We make use of Conditional Cross-Mutual Information (CXMI) to explore how much of the context the model uses and generalise CXMI to study the impact of the extra sentential context. Overall, we find that models leverage both preceding sentences and extra-sentential context (with CXMI increasing with context size) and we provide a more focused analysis on honorifics translation. Regarding translation quality, increased source-side context paired with scene and speaker information improves the model performance compared to previous work and our context-agnostic baselines, measured in BLEU and COMET metrics.

## introduction traditionally nmt models such as transformers @xcite approach the task of machine translation (mt) focusing on individual sentences without considering the surrounding information, such as previous utterances or underlying topics. as a result, the output often lacks discourse coherence and cohesion, which is problematic for mt applications such as chat translation systems @xcite . thus, it is still an open research question to what degree these models can take advantage of contextual information to produce more accurate translations. to answer this question, several context-aware nmt @xcite @xcite studies have been conducted by adding surrounding sentences to the models and testing if it helps to capture better specific linguistic phenomena requiring context (e.g. coreference resolution). however, there is limited work on discourse or dialogue datasets, and most of it is focused on high-resource or indo-european (ie) languages @xcite . therefore, there is a need to investigate how well do the proposed approaches capture discourse phenomena in non-ie or low-resource languages. this work aims to address the aforementioned gap by focusing on english-japanese (en-ja) translation for business dialogue scenarios in order to examine if current context-aware nmt models @xcite actually use the additional context, and what kind of context is useful regarding the translation of linguistic phenomena pertaining to japanese discourse, such as honorifics. we specifically propose the use of novel extra-sentential information as additional context and show that it improves translation quality. overall, the main contributions of this study are threefold: (1) we demonstrate that it is possible to adapt a (non-context-aware) large pretrained model (mbart; @xcite ; @xcite ) to attend to context for business dialogue translation and propose an improved attention mechanism (coattmask) with significant performance gains for source-side context, even on small datasets; (2) we propose novel extra-sentential information elements such as speaker turn and scene type, to be used as additional source-side context; and (3) we compare the use of context between our context-aware models using cxmi @xcite , a mutual-information-based metric and perform a more focused analysis on the translation of honorifics. several methods using a transformer-based architecture @xcite have been proposed for context-aware nmt, frequently categorised into single-encoder and multi-encoder models @xcite . single-encoder models concatenate the source sentence with (a) preceding sentence(s) as the contexts, with a special symbol to distinguish the context and the source or target in an encoder @xcite . multi-encoder models pass the preceding sentence(s) used as context through a separate encoder modifying the transformer architecture @xcite . according to @xcite , the observed performance gap between the two models is marginal, but the single-encoder models are relatively simpler architectures without modifying sequence-to-sequence transformers. apart from concatenating preceding sentences on the source-side, some works focus on the target-side context, i.e., show some benefits from attempting to decode multiple sequential sentences together @xcite . depending on the use-case, source-side, target-side, or a combination of contexts has proven beneficial @xcite @xcite . additionally, some works focused more on context related to discourse phenomena, with @xcite proposing the use of variational autoencoders to model dialogue phenomena such as speaker role as latent variables @xcite . we examine here a simpler approach, that directly encodes such speaker and scene information and allows the model to use it as additional context. in more recent work, the impact of pretraining on larger out-of-domain (ood) data has also been studied to aid in downstream mt tasks with limited resources @xcite . for english-japanese translation, there have been some context-aware nmt studies that used variations of single-encoder models in the news and dialogue domain @xcite @xcite . specifically for dialogue, @xcite experimented with context-aware mt that employs source-side factors on ja-en (japanese-english) and en-ja (english-japanese) discourse datasets. they propose to concatenate the preceding sentence(s) from the same document followed by a tag-token to separate the context from the original sentence and use binary token-level factors on top of this to signify whether a token belongs to the context or source sentence. ## datasets we use business scene dialogue corpus (bsd) @xcite as the main dataset. additionally, only to compare the performance in a certain setup with the main dataset, we also use ami meeting parallel corpus (ami) @xcite as a supplemental dataset. they are both document-level parallel corpora consisting of different scenes (dialogue sequence scenarios) or meetings and include both out-of-english and into-english translations, of which we use the english-japanese translation direction. we focus our analysis on the bsd dataset, as it contains more scenarios and extra-sentential information which we use as additional context. in the main dataset bsd, each document consists of a business scene with a scene tag (face-to-face, phone call, general chatting, meeting, training, and presentation), and each sentence has speaker information that indicates who is speaking. contents of bsd are originally written either in english or japanese by bilingual scenario writers who are familiar with business scene conversations and then translated into the other language to create a parallel corpus. as for ami, the contents are translations to japanese from 100 hours of meeting recordings in english. since it originates from naturally occurring dialogue it contains shorter utterances than bsd, including multiple single-word sentences with filler and interjection words. the data split statistics for bsd and ami are shown in ## methodology in this section, we analyse our context-aware nmt approach in a dialogue setup in two steps: firstly, we consider what type of information might be useful as context and how it should be encoded to generate useful input representations, and secondly, we discuss modifications in the original encoder-decoder architecture that facilitate learning to attend to context even when tuning on small datasets. ## experimental results we compare our work to previous approaches evaluated on bsd, namely this of @xcite who combined multiple en-ja datasets to train a model for en-ja dialogue translation and @xcite who also used a context-aware variant of @xcite combined with factors to encode dialogue context. additionally, we compare with our context agnostic baseline. ## conclusion and future work this paper explored to what degree encoded context can improve nmt performance for english-japanese dialogue translation, and what kind of context provides useful information. with our proposed method, we were able to tune mbart on small dialogue datasets and obtain improved mt performance using context. we found that source-side context was more beneficial towards performance and that complementing our source-side context with scene and speakerturn tags provided further performance improvements. we further analyse the impact of our proposed context-aware methods on the translations obtained, with a focus on translation of japanese honorifics. in future work, we aim to further investigate context for dialogue translation, expanding to a multilingual setup, larger datasets, and additional extra-sentential context.