Two-Headed Monster and Crossed Co-Attention Networks (2020)

Abstract: This paper investigates a new co-attention mechanism in neural transduction models for machine translation tasks. We propose a paradigm, termed Two-Headed Monster (THM), which consists of two symmetric encoder modules and one decoder module connected with co-attention. As a specific and concrete implementation of THM, Crossed Co-Attention Networks (CCNs) are designed based on the Transformer model. We test CCNs on WMT 2014 EN-DE and WMT 2016 EN-FI translation tasks and show both advantages and disadvantages of the proposed method. Our model outperforms the strong Transformer baseline by 0.51 (big) and 0.74 (base) BLEU points on EN-DE and by 0.17 (big) and 0.47 (base) BLEU points on EN-FI but the epoch time increases by circa 75%.

## introduction attention has emerged as a prominent mechanism extensively adopted in neural modules in a wide range of research problems @xcite @xcite @xcite @xcite @xcite such as vqa, reading comprehension, textual entailment, image captioning and speech recognition. its remarkable success is also embodied in machine translation tasks @xcite . this work proposes an end-to-end co-attentional neural structure named crossed co-attention networks (ccns) to address machine translation, a typical sequence-to-sequence nlp task. we customize the transformer @xcite featured by non-local operations @xcite with two input branches and tailor the transformer's multihead attention mechanism to the needs of information exchange between these two parallel branches. a higher-level and more abstract paradigm generalized from ccns is denoted as "two-headed monster" (thm), representing a broader class of neural structures benefiting from two parallel neural channels that would be intertwined with each other through co-attention mechanism as illustrated in fig. 1 . needless to say, co-attention is widely adopted in multi-modal scenarios @xcite @xcite @xcite , the basic idea of which is to make two feature maps from different domains to attend to each other and thus output summarized representations for each domain. in this work, we emphasize a parallel and symmetric manifold operating on two input channels and possessing two output channels but do not assume that the two channels of input must be disparate. our coattention mechanism is designed in a "transformer" style, and to the best of our knowledge, our proposed crossed co-attention network is one of the first implementations of co-attention on transformer model @xcite and we are the first to apply transformerbased co-attention on machine translation tasks. in particular, we apply our model on the popular wmt machine translation tasks where two input channels are in one same domain. our code also leverages half-precision floating-point format (fp16) training and synchronous distributed training for inter-gpu communication (we do not discard gradients calculated by "stragglers") which dramatically accelerate our training procedure @xcite . ## model architecture in this section, we first define co-attention as a generic concept, following @xcite 's definition of non-local operation. after that, we propose an end-to-end neural architecture based on the transformer to address machine translation tasks where the model takes input from two channels. in particular, we design a crossed co-attention mechanism to make our model capable of attending to two parallel information flows simultaneously in both the encoding and the decoding stages. our co-attention mechanism is naively realized by a crossed connection of value (v), key (k) and query (q) gates of a regular multihead attention module, so we term our model crossed co-attention networks. ## conclusion and future work we propose two-headed monster (thm) neural structures consisting of two parallel encoders and a decoder connected with each other using co-attention, which is the core module for information flow in thm. first, we formulate the coattention in a general sense as a non-local operation. then we design crossed co-attention networks (ccns) based on our defined co-attention under the umbrella of thm. our implementation of ccn adopts a specific way of connections of attention gates and improve the machine translation tasks by 0.17 ∼ 0.74 bleu points and enhance the capability of model selection. however, the time efficiency is reduced since the number of parameters increases. while we investigate the advantages and disadvantages of our proposed model on machine translation, we still think that the work is at a preliminary stage and our future work will focus on two aspects: (1) adopt input from two different domains and test our model on tasks such as multi-modal machine translation; (2) investigate how different ways of co-attention connections will influence the model performance on different tasks.