MTC ue: Learning Zero-Shot Control of Extra-Textual Attributes by Leveraging Unstructured Context in Neural Machine Translation (2023)

Abstract: Efficient utilisation of both intra- and extra-textual context remains one of the critical gaps between machine and human translation. Existing research has primarily focused on providing individual, well-defined types of context in translation, such as the surrounding text or discrete external variables like the speaker’s gender. This work introduces MTCue, a novel neural machine translation (NMT) framework that interprets all context (including discrete variables) as text. MTCue learns an abstract representation of context, enabling transferability across different data settings and leveraging similar attributes in low-resource scenarios. With a focus on a dialogue domain with access to document and metadata context, we extensively evaluate MTCue in four language pairs in both translation directions. Our framework demonstrates significant improvements in translation quality over a parameter-matched non-contextual baseline, as measured by BLEU (+0.88) and Comet (+1.58). Moreover, MTCue significantly outperforms a “tagging” baseline at translating English text. Analysis reveals that the context encoder of MTCue learns a representation space that organises context based on specific attributes, such as formality, enabling effective zero-shot control. Pre-training on context embeddings also improves MTCue’s few-shot performance compared to the “tagging” baseline. Finally, an ablation study conducted on model components and contextual variables further supports the robustness of MTCue for context-based NMT.

## introduction research in neural machine translation (nmt) has advanced considerably in recent years, much owing to the release of the transformer architecture @xcite , subword segmentation @xcite and back-translation @xcite . this resulted in claims of human parity in machine translation @xcite , which in turn prompted researchers to look beyond the sentence level: at how a translation still needs to be compatible with the context it arises in. the task of contextual adaptation to more nuanced extra-textual variables like the description of the discourse situation has been largely overlooked, in spite of earlier work suggesting that conversational machine translation may benefit from such fine-grained adaptations @xcite . most existing work on contextual nmt has focused on document-level context instead, aiming to improve the coherence and cohesion of the translated document (e.g. @xcite . some research has successfully adapted nmt to extra-textual context variables using supervised learning frameworks on labelled datasets, targeting aspects such as gender @xcite @xcite , formality @xcite , translators' or speakers' style @xcite and translation length @xcite , sometimes controlling multiple attributes simultaneously @xcite . however, to our knowledge, no prior work has attempted to model the impact of continuous extra-textual contexts in translation or combined the intra-and extratextual contexts in a robust framework. this is problematic since translating sentences without or with incomplete context is akin to a human translator working with incomplete information. similarly, only a handful of earlier research has contemplated the idea of controlling these extratextual attributes in a zero-shot or few-shot fashion @xcite ; such approaches are essential given the difficulty of obtaining the labels required for training fully supervised models. in some domains, extra-textual context is paramount and nmt systems oblivious to this information are expected to under-perform. for instance, for the dubbing and subtitling domain, where translated shows can span different decades, genres, countries of origin, etc., a one-size-fits-all model is limited by treating all input sentences alike. in this domain, there is an abundance of various metadata (not just document-level data) that could be used to overcome this limitation. however, such adaptation is not trivial: (i) the metadata often comes in quantities too small for training and with missing labels; (ii) it is expressed in various formats and types, being difficult to use in a standard pipeline; (iii) it is difficult to quantify its exact (positive) effect. in this paper, we address (i) and (ii) by proposing mtcue (machine translation with contextual universal embeddings), a novel nmt framework that bridges the gap between training on discrete control variables and intra-textual context as well as allows the user to utilise metadata of various lengths in training, easing the need for laborious data editing and manual annotation (figure 1 ). during inference, when context is provided verbatim, mtcue falls back to a code-controlled translation model; by vectorising the inputs, it exhibits competitive performance for noisy phrases and learns transferrability across contextual tasks. while (iii) is not directly addressed, our evaluation encompasses two translation quality metrics and two external test sets of attribute control, showing the impact on both translation quality and capturing relevant contextual attributes. mtcue can generalise to unseen context variables, achieving 100% accuracy at a zero-shot formality controlling task; it learns to map embeddings of input contexts to discrete phenomena (e.g. formality), increasing explainability; and it exhibits more robust few-shot performance at multiattribute control tasks than a "tagging" baseline. the main contributions of this work are: 1. mtcue ( §2): a novel framework for combining (un)structured intra-and extra-textual context in nmt that significantly improves translation quality for four language pairs in both directions: english (en) to/from german (de), french (fr), polish (pl) and russian (ru). 2. a comprehensive evaluation, showing that mtcue can be primed to exhibit excellent zero-shot and few-shot performance at downstream contextual translation tasks ( §4 and §5). 3. pre-trained models, code, and an organised version of the opensubtitles18 @xcite dataset with the annotation of six metadata are made available. this paper also presents the experimental settings ( §3), related work ( §6) and conclusions ( §7). ## proposed architecture: mtcue mtcue is an encoder-decoder transformer architecture with two encoders: one dedicated for contextual signals and one for inputting the source text. the signals from both encoders are combined using parallel cross-attention in the decoder. below we describe how context inputs are treated in detail, and later in §2.2 and §2.3 we describe the context encoder and context incorporation, respectively. ## ablation study we discuss the robustness of mtcue with an ablation study on the model components as well as a complementary ablation on types of context (metadata vs document). we evaluate three language pairs (@xmath0,fr,pl) and report results from single runs ( removing the context encoder (output of the linear layer is combined with source straight away) or the position embeddings has only a minor effect on the comet score; replacing minilm-v2 with a discrete embedding function hurts performance the most. positional embeddings seem more important to the eamt22 task than iwslt22 -possibly because eamt22 focuses on sentence-level phenomena, so the order of past context matters. replacing minilm-v2 with a discrete embedding function removes the zero-shot effect in both tasks. an interesting finding is that between metadata and document-level data, it is the latter that brings more improvements to contextual tasks; this means that our model potentially scales to domains without metadata. finally, using random context degrades performance w.r.t. full model implying that the gains come from signals in data rather than an increase in parameters or training time. ## related work although contextual adaptation has been discussed in other tasks (e.g. @xcite , in this section we focus on nmt, as well as set our work side by side with research that inspired our approach. existing studies on incorporating context into nmt have primarily focused on document-level context. these approaches include multi-encoder models (e.g. @xcite , cache models @xcite , automatic post-editing @xcite , shallow fusion with a document-level language model @xcite , data engineering techniques @xcite or simple concatenation models @xcite . another line of research aims to restrict hypotheses based on certain pre-determined conditions, and this includes formality @xcite , interlocutors' genders (e.g. @xcite , or a combination of both @xcite . other conditions include translation length and monotonicity @xcite , vocabulary usage @xcite or domain and genre @xcite . while wider contextual adaptation in nmt has been discussed theoretically, most empirical research falls back to gender @xcite or formality control @xcite . one exception is @xcite who adapt nmt for each of many speakers by adding a "speaker bias" vector to the decoder outputs. our work is motivated by the cue vectors @xcite and their application in personalised language models for film and tv dialogue @xcite . cue vectors represent context computed by passing sentence embeddings of the input context through a dedicated encoder. @xcite show that incorporating cue in language modelling improves perplexity, while vincent et al. use them to personalise language models for on-screen characters. in contrast, we reformulate cue for contextual machine translation, provide a detailed analysis of incorporating cue into the model, emphasise the importance of vectorising the context prior to embedding it, and examine the benefits for zero-shot and few-shot performance in contextual nmt tasks. ## conclusions we have presented mtcue, a new nmt architecture that enables zero-and few-shot control of contextual variables, leading to superior translation quality compared to strong baselines across multiple language pairs (english to others, cf.