Towards Multimodal Simultaneous Neural Machine Translation (2020)

Abstract: Simultaneous translation involves translating a sentence before the speaker’s utterance is completed in order to realize real-time understanding in multiple languages. This task is significantly more challenging than the general full sentence translation because of the shortage of input information during decoding. To alleviate this shortage, we propose multimodal simultaneous neural machine translation (MSNMT), which leverages visual information as an additional modality. Our experiments with the Multi30k dataset showed that MSNMT significantly outperforms its text-only counterpart in more timely translation situations with low latency. Furthermore, we verified the importance of visual information during decoding by performing an adversarial evaluation of MSNMT, where we studied how models behaved with incongruent input modality and analyzed the effect of different word order between source and target languages.

## introduction simultaneous translation is a natural language processing (nlp) task in which translation begins before receiving the whole source sentence. it is widely used in international summits and conferences where real-time comprehension is one of the essential aspects. simultaneous translation is already a difficult task for human interpreters because the message must be understood and translated while the input sentence is still incomplete, especially for language pairs with different word orders (e.g. svo-sov) @xcite . consequently, simultaneous translation is more challenging for machines. previous works attempt to solve this task by predicting the sentence-final verb @xcite , or predicting unseen syntactic constituents @xcite . given the difficulty * these authors contributed equally to this paper of predicting future inputs based on existing limited inputs, @xcite proposed a simple simultaneous neural machine translation (snmt) approach wait-k which generates the target sentence concurrently with the source sentence, but always k tokens behind, satisfying low latency requirements. however, previous approaches solve the given task by solely using the text modality, which may be insufficient to produce a reliable translation. simultaneous interpreters often consider various additional information sources such as visual clues or acoustic data while translating @xcite . therefore, we hypothesize that using supplementary information, such as visual clues, can also be beneficial for simultaneous machine translation. to this end, we propose multimodal simultaneous neural machine translation (msnmt) that supplements the incomplete textual modality with visual information, in the form of an image. it will predict still missing information to improve translation quality during the decoding process. our approach can be applied in various situations where visual information is related to the content of speech such as presentations with slides (e.g. ted talks foot_0 ) and news video broadcasts foot_1 . our experiments show that the proposed msnmt method achieves higher translation accuracy than the snmt model that does not use images by leveraging image information. to the best of our knowledge, we are the first to propose the incorporation of visual information to solve the problem of incomplete text information in snmt. the main contributions of our research are as follows. we propose to combine multimodal and simultaneous nmt, therefore, discovering cases where such multimodal signals are beneficial for the end-task. our msnmt approach brings significant improvement in simultaneous translation quality by enriching incomplete text input information using visual clues. as a result of a thorough analysis, we conclude that the proposed method is able to predict tokens that have not appeared yet for source-target language pairs with different word order (e.g. @xmath0. by providing an adversarial evaluation, we showed that the models indeed utilize visual information. ## related work for simultaneous translation, it is crucial to predict the words that have not appeared yet. for example, it is important to distinguish nouns in svo-sov translation and verbs in sov-svo translation @xcite . snmt can be realized with two types of policy: fixed and adaptive policies @xcite . adaptive policy decides whether to wait for another source word or emit a target word in one model. previous models with adaptive policies include explicit prediction of the sentencefinal verb @xcite and unseen syntactic constituents @xcite . most dynamic models with adaptive policies @xcite @xcite @xcite have the advantage of exploiting input text information as effectively as possible due to the lack of such information in the first place. meanwhile, @xcite proposed a simple wait-k method with fixed policy, which generates the target sentence only from the source sentence that is delayed by k tokens. however, their model for simultaneous translation relies only on the source sentence. in this research, we concentrate on the wait-k approach with fixed policy, so that the amount of input textual context can be controlled to analyze better whether multimodality is effective in snmt. multimodal nmt (mnmt) for full-sentence machine translation has been developed to enrich text modality by using visual information @xcite @xcite . while the improvement brought by visual features is moderate, their usefulness is proven by @xcite . they showed that mnmt models are able to capture visual clues under limited textual context, where source sentences are synthetically degraded by color deprivation, entity masking, and progressive masking. however, they use an artificial set-ting where they deliberately deprive the models of source-side textual context by masking. however, our research has discovered an actual end-task and has shown the effectiveness of using multimodal data for it. compared with the entity masking experiments @xcite , where they use a model exposed to only k words, our model starts by waiting for the first k source words and then generates each target word after receiving every new source token, eventually seeing all input text. in mnmt, visual features are incorporated into standard machine translation in many ways. doubly-attentive models are used to capture the textual and visual context vectors independently and then combine these context vectors in a concatenation manner @xcite or hierarchical manner @xcite . some studies use visual features in a multitask learning scenario @xcite . also, recent work on mnmt has partly addressed lexical ambiguity by using visual information @xcite @xcite showing that using textual context with visual features outperform unimodal models. in our study, visual features are extracted using image processing techniques and then integrated into an snmt model as additional information, which is supposed to be useful to predict missing words in a simultaneous translation scenario. to the best of our knowledge, this is the first work that incorporates external knowledge into an snmt model. ## multimodal simultaneous neural machine translation architecture our main goal is to investigate if image information would bring improvement on snmt. as a result, two tasks could benefit from each other by combining them. in this section, we describe our msnmt model, which is composed by combining an snmt framework wait-k @xcite and a multimodal model @xcite . we base our model on the rnn architecture, which is widely used in mnmt research @xcite @xcite @xcite . the model takes a sentence and its corresponding image as inputs. the decoder of the msnmt model outputs the target language sentence in a simultaneous and multimodal manner by attaching attention not only to the source sentence but also to the image related to the source sentence. foot_2 ## analysis in this section, we provide a thorough analysis to further investigate the effect of visual data to produce a simultaneous translation by (a) providing adversarial evaluation; and (b) analyzing the impact of different word order for @xmath1. ## conclusion in this paper, we proposed a multimodal simultaneous neural machine translation approach, which takes advantage of visual information as an additional modality to compensate for the shortage of input text information in the simultaneous neural machine translation. we showed that in a wait-k setting, our model significantly outperformed its text-only counterpart in situations where only a few input tokens are available to begin translation. we showed the importance of the visual information for simultaneous translation, especially in the low latency setup and for a language pair with word-order differences. we hope that our proposed method can be explored even further for various tasks and datasets. in this paper, we created a separate model for each value of wait-k. however, in future work, we plan to experiment on having a single model for all k values @xcite . furthermore, we acknowledge the importance of investigating msnmt effects on more realistic data (e.g. ted), where the utterance does not necessarily match a shown image while speaking and/or where its context can not be guessed from the shown image.