Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation (2021)

Abstract: This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual information to support this process by providing additional (visual) contextual information which may be available before the textual input is produced. For that, we propose a multimodal approach to simultaneous machine translation using reinforcement learning, with strategies to integrate visual and textual information in both the agent and the environment. We provide an exploration on how different types of visual information and integration strategies affect the quality and latency of simultaneous translation models, and demonstrate that visual cues lead to higher quality while keeping the latency low.

## introduction research into automating real-time interpretation has explored deterministic and adaptive approaches to build policies that address the issue of translation delay @xcite @xcite . in another recent development, the availability of multimodal data (such as visual information) has driven the community towards multimodal approaches for machine translation (mmt) @xcite @xcite . although deterministic policies have been recently explored for simultaneous mmt @xcite , there are no studies regarding how multimodal information can be exploited to build flexible and adaptive policies for simultaneous machine translation (simt). applications of reinforcement learning (rl) for unimodal simt have highlighted the challenges for the agent to maintain good translation quality while learning an optimal translation path (i.e. a sequence of read/write decisions at every time step) @xcite @xcite . incomplete source information will have detrimental effect especially in the cases where significant restructuring is needed while translating from one language to another. in addition, the lack of information generally leads to high variance during the training in the rl setup. we posit that multimodality in adaptive simt could help the agent by providing extra signals, which would in turn improve training stability and thus the quality of the estimator and translation decoder. in this paper, we present the first exploration on multimodal rl approaches for the task of simt. as visual signals, we explore both image classification features as well as visual concepts, which provide global image information and explicit object representations, respectively. for rl, we employ the policy gradient method with a pre-trained neural machine translation model acting as the environment. as the simt model is optimised for both translation quality and latency, we apply a combined reward function that consists of a decomposed smoothed bleu score and a latency score. to integrate visual and textual information, we propose different strategies that operate both on the agent (as prior information or at each step) and the environment side. in experiments on standard datasets for mmt, our models achieve the highest bleu scores on most settings without significant loss on average latency, as compared to strong simt baselines. a qualitative analysis shows that the agent benefits from the multimodal information by grounding language signals on the images. our main contributions are as follows: ## related work in this section, we first present background and related work on simt, and then discuss recent work in mmt and multimodal rl. ## methods we first present the architectures for consecutive and baseline fixed policy simultaneous mt (section 3.1). then we introduce our rl approaches, both the baseline, the proposed multimodal extension (section 3.2), and the visual features used by all multimodal approaches (section 3.3). ## results in this section, we first provide the results from our experiments (section 5.1) and then analyse the behaviour of the (multimodal) agents (section 5.2). ## conclusion in this paper we presented the first thorough exposition of multimodal reinforcement learning strategies for simultaneous machine translation. we demonstrate the efficacy of visual information and show that it leads to adaptive policies which substantially improve over the deterministic and unimodal rl baselines. our empirical results indicate that both agent-side and environment-side visual information can be exploited to achieve higher quality translations with lower latency. throughout the experimental journey, we observed that the optimisation of simultaneous machine translation for dynamic policies is non-trivial, due to the two competing objectives: translation quality versus latency. for unimodal simultaneous machine translation, rl approaches tend to achieve translation quality on par with the quality of the deterministic policies within the same average lag. we believe that the fundamental issue is related to the high variance of the estimator for sequence prediction, which increases sample complexity and impedes effective learning. on the other hand, the approaches with deterministic policies are simple and effective, as they are positively biased for language pairs that are close to each other. but the latter suffer from poor generalisation. in the multimodal simultaneous machine translation setting, however, the variance of the estimator from rl models can be substantially reduced with to the presence of additional (visual) information.