Simultaneous Machine Translation with Visual Context (2020)

Abstract: Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible. The translation thus has to start with an incomplete source text, which is read progressively, creating the need for anticipation. In this paper, we seek to understand whether the addition of visual information can compensate for the missing source context. To this end, we analyse the impact of different multimodal approaches and visual features on state-of-the-art SiMT frameworks. Our results show that visual context is helpful and that visually-grounded models based on explicit object region information are much better than commonly used global features, reaching up to 3 BLEU points improvement under low latency scenarios. Our qualitative analysis illustrates cases where only the multimodal systems are able to translate correctly from English into gender-marked languages, as well as deal with differences in word order, such as adjective-noun placement between English and French.

## introduction simultaneous machine translation (simt) aims to reproduce human interpretation, where an interpreter translates spoken utterances as they are produced. the interpreter has to dynamically find the balance between how much context is needed to generate the translation reliably, and how long the listener has to wait for the translation. in contrast to consecutive machine translation where source sentences are available in their entirety before translation, the challenge in simt is thus the design of a strategy to find a good trade-off between the quality of the translation and the latency incurred in producing it. previous work has considered rulebased strategies that rely on waiting until some white chat blanc cat a un figure 1 : an illustration of a 1-word latency system that makes use of visual grounding to resolve the gender of the article 'un' and to predict the noun 'chat' after reading its qualifier 'white'.anddenote read and write, respectively. thus far, all prior research has focused on unimodal interpretation foot_0 . in this paper, we explore simt for multimodal machine translation (mmt) @xcite , where in addition to the source sentence, we have access to visual information in the form of an image. we believe that having access to a complementary context should help the models anticipate the missing context (figure 1 ) by grounding their decisions about 'when @xcite to translate. to test our hypothesis, we explore heuristic-based decoding and fixed-latency wait-k policy foot_1 , and investigate the effectiveness of different visual representations ( §3). based on experiments on the multi30k dataset @xcite ( §4), we provide quantitative and qualitative analyses for @xmath0. our contributions highlight that: (i) visual context offers up to 3 bleu points improvement for low-latency wait-k policies, and consistently lowers the latency for wait-if-diff @xcite decoding, (ii) explicit object region features are more expressive than commonly used global visual features, (iii) training wait-k mmts offers remarkably better grounding capabilities than decoding-only wait-k for linguistic phenomena such as gender resolution and adjective-noun ordering, and (iv) with twice the runtime speed of decoder-based visual attention, the encoder-based grounding is promising for application scenarios. ## methods in this section, we first describe the underlying nmt architectures and baseline simultaneous mt approaches, to then introduce the proposed multimodal extensions to simt. ## conclusion we present the first thorough investigation of the utility of visual context for the task of simultaneous machine translation. our experiments reveal that integrating visual context lowers the latency for heuristic policies while retaining the quality of the translations. under low-latency wait-k policies, the visual cues are highly impactful with quality improvements of almost 3 bleu points compared to unimodal baselines. from a multimodal perspective, we introduce effective ways of integrating visual features and show that explicit object region information consistently outperforms commonly used global features. our qualitative analysis illustrates that the models are capable of resolving linguistic particularities, including gender marking and word order handling by exploiting the associated visual cues. we hope that future research continues this line of work, especially by finding novel ways to devise adaptive policies -such as reinforcement learning models with the visual modality. we believe that our work can also benefit research in multimodal speech translation @xcite where the audio stream is accompanied by a video stream.