Priming Neural Machine Translation (2020)

Abstract: Priming is a well known and studied psychology phenomenon based on the prior presentation of one stimulus (cue) to influence the processing of a response. In this paper, we propose a framework to mimic the process of priming in the context of neural machine translation (NMT). We evaluate the effect of using similar translations as priming cues on the NMT network. We propose a method to inject priming cues into the NMT network and compare our framework to other mechanisms that perform micro-adaptation during inference. Overall, experiments conducted in a multi-domain setting confirm that adding priming cues in the NMT decoder can go a long way towards improving the translation accuracy. Besides, we show the suitability of our framework to gather valuable information for an NMT network from monolingual resources.

## introduction priming is a well studied human cognitive phenomenon, founded on the establishment of associations between a stimulus and a response @xcite . multiple studies have shown how external stimuli (cues) may have a profound effect on perception. in the case of language translation, external stimuli having such effects are said to prime language understanding and potentially have a impact the actions of a human translator. imagine for instance a translator facing the ambiguous sentence i was in the bank, and the effect on translation accuracy if primed with the cue river. most likely, the human translator would consider the "edge of river" sense rather than "financial institution" for translation. in the context of human translation, cross-lingual priming is particularly effective as cues in the target language may notably influence the final translation word choice. several research works have introduced the priming analogy in deep neural networks. in computer vision priming has been broadly studied: for instance, in @xcite , the authors introduce a cue about the presence of a certain class of object in an image that significantly improves object detection performance. concerning language generation, brown et al. (2020) use a combination of prompt and example to guide the gpt-3 network when performing a task, where the prompt is a sentence that describes the task (i.e. "translate from english to french"); and is followed by an example of the task (i.e. "sea otter ; loutre de mer"). in the context of nmt, experiments reported @xcite @xcite aim at influencing translation inference with respectively politeness, domain and terminology constraints. more related to our work, @xcite introduce a simple and elegant framework where similar translations (cues) are used to prime an nmt model, effectively boosting translation accuracy. in all cases, priming is performed by injecting cues in the input stream prior to inference decoding. in this paper, we extend a framework that mimics the priming process in neural networks, in the context of machine translation. following up on previous work @xcite , we consider similar translations as external cues that can influence the translation process. we push this concept further: a) by proposing a novel scheme to integrate similar translation cues into the nmt network. we examine the attention mechanism of the network and confirm that priming stimuli are actually taken into account; b) by extending an efficient network to train distributed representations of sentences that are used to identify accurate translations used as priming cues 1 ; c) by analyzing how on-the-fly priming compares to micro-adaptation (fine-tuning). finally, we show that our priming approach can also be used with monolingual data, providing a scenario where nmt can be effectively helped by large amounts of available data. our proposal does not require to change the nmt architectures or algorithms, relying solely on input preprocessing and on prefix (forced) decoding @xcite , a feature already implemented in many nmt toolkits. the remainder of the paper is organized as follows: section 2 gives details regarding our priming approach. the experimental framework is presented in section 3. results and discussion are respectively in sections 4 and 5. we review related work in section 6 and conclude in section 7. ## nmt priming on-the-fly this section describes our framework for priming neural mt with similar translations. we follow the work by @xcite and build a translation model that incorporates similar translations from a translation memory (tm) to boost translation accuracy. in this work, tms are parallel corpora containing translations falling in the same domain as test sentences. we first describe the methods employed in this work to compute sentence similarity. we then introduce various augmentation schemes considered to prime the nmt network with retrieved similar translations. overall, we pay special attention to efficiency, since retrieval is applied on a sentenceby-sentence basis at inference. ## results retrieval algorithms employed in this work are significantly faster than nmt transformer decoding, thus implying a limited decoding overhead. all nmt models are built using the concatenation of the original parallel corpora in the second block of results in the third block compares translation results obtained when identifying similar translations by s2v and cbon. in both cases, the s+t 5 scheme is used. the choice for 5-best similar translations and em(s i , s j ) ≥ 0.8 threshold is made after running optimization work on a held out development set. sentences identified by cbon outperform those selected by s2v. the idiosyncrasy of fuzzy matching does not enable to find multiple similar sentences for a given input sentence. overall best results are obtained by the cbon s+t 5 configuration. note that as expected, the number of similar translations found using distributed representations is larger than those found by fuzzy matching. finally, the last block in ## conclusions inspired by the human psychological phenomenon of priming, we have presented a simple framework for priming nmt networks. following other research works, we used similar translations as priming cues to influence the nmt network. we presented a novel method that injects similar translations in the nmt network as prefixes of the decoder. the proposed method obtains higher translation accuracy results and reduces the undesirable effect observed in previous methods of copying unrelated words when performing translations. we also proposed an extension to sent2vec that considers larger n-gram orders. it allows us to identify similar sentences (cues) that yield higher accuracy rates as measured on translation test sets. we evaluate results on a multi-domain setting using a single model trained on a heterogeneous data set, built from multiple corpora and domains, achieving better results when compared to previous micro-adaptation approaches. in addition, we showed the suitability of our approach to gather valuable information from large monolingual corpora. in our future work, we would like to explore alternative algorithms to compute distributed sentence representations from word embeddings, such as tf-idf. furthermore, we would like to consider source sentence coverage when selecting n-best similar translations. as regards distributed representations we plan to experiment with cross-lingual networks to retrieve similar translations directly from human-crafted monolingual data in order to eliminate the noise introduced by synthetic translations.