Meet Changes with Constancy: Learning Invariance in Multi-Source Translation (2020)

Abstract: Multi-source neural machine translation aims to translate from parallel sources of information (e.g. languages, images, etc.) to a single target language, which has shown better performance than most one-to-one systems. Despite the remarkable success of existing models, they usually neglect the fact that multiple source inputs may have inconsistencies. Such differences might bring noise to the task and limit the performance of existing multi-source NMT approaches due to their indiscriminate usage of input sources for target word predictions. In this paper, we attempt to leverage the potential complementary information among distinct sources and alleviate the occasional conflicts of them. To accomplish that, we propose a source invariance network to learn the invariant information of parallel sources. Such network can be easily integrated with multi-encoder based multi-source NMT methods (e.g. multi-encoder RNN and transformer) to enhance the translation results. Extensive experiments on two multi-source translation tasks demonstrate that the proposed approach not only achieves clear gains in translation quality but also captures implicit invariance between different sources.

## introduction neural machine translation (nmt) systems in general translate one source language to a target language. various one-to-one attentional encoder-decoder architectures @xcite @xcite were designed to learn word and structure mappings including formations, grammatical correspondences between source and target languages. recently, multi-source nmt @xcite , which simultaneously takes multiple different languages @xcite @xcite as input when translating to another one, is emerging. its intuitive idea is ambiguity between one source language and the target language could be reduced by another source language via the "triangulation" proposed by @xcite . for example, it is hard to tell who was with a telescope by the ambiguous english sentence in figure 1 @xcite . but it could be more easily to be translated to spanish if provided with the corresponding chinese phrase "我通过望远镜", which corresponds to "i see through a telescope" in english. besides, the complementary source could be extended to non-language input such as images, also known as multimodal nmt @xcite . it takes an image with description in source languages, that is then translated into a target language. the images are expected to provide additional signals for better translations, which abides by similar assumptions in multi-source nmt with different languages. a specific instance is illustrated in figure 1 @xcite . in this paper, we consider general multisource nmt which contains more than one source as input. our proposed model could be naturally adapted to these two specific sub-tasks, namely multi-lingual translation and multi-modal translation. for simplicity and convenience to describe, we use multi-source nmt to denote both tasks. a woman on the floor covered in cake . vi a alguien en la montaña a través de un telescopio. une femme sur le sol couverte de gâteau . french spanish i saw someone on the hill with a telescope. ## background in this section, we introduce two general multi-encoder based multi-source nmt frameworks, namely rnn-based and transformer-based approaches. they basically have the same architecture with multiple encoders and a single decoder corresponding to different sources and one target, respectively. ## the proposed model in this section, we detail our source invariance network (sin). our sin is inspired by domain separation networks (dsn) @xcite , which is used for learning domain-shared representations in transfer learning field. while in this paper, sin attempts to learn the invariance among different inputs in multi-source translation tasks and alleviate the occasional conflicts of inputs. in this section, we first detail the architecture of our sin, then illustrate how to integrate it with rnnbased and transformer-based multi-source nmt frameworks. furthermore, we raise a discussion on how sin can strengthen the performance on multi-encoder based frameworks. encoder 1 … encoder 2 … difference loss similarity loss invariant encoder private encoder private encoder source invariance network difference loss invariant encoder shared weights figure 3: the architecture of source invariance network. it takes in the source representation of each encoder, separating the invariance and variance through invariant and private encoder, respectively. ## experimental set-up we conduct our experiments on two tasks: multi-lingual and multi-modal nmt. in this section, we describe the datasets, baselines, the evaluation protocol and implementation details. ## experimental results in this section, we exhibit the overall performance among our methods and baselines in sec 6.1, examine the conflicts caused by the inconsistency among different sources in sec 6.2, and evaluate the effectiveness of learning invariance in sec 6.3. ## conclusion in this paper, we observed the drawbacks of multi-encoder based multi-source nmt on neglecting the potential inconsistency among sources. we thus proposed sin to learn the invariance among multiple sources. with the equipped invariant encoder, the invariant features are explored by minimizing the maximum mean discrepancy between source representations. meanwhile, the private encoder is devised to distill the corresponding particular features of different sources. sin can be easily integrated with conventional multi-encoder based multi-source nmt frameworks. we conducted extensive experiments in two main multi-source tasks, and the results demonstrate the effectiveness of the proposed model.