Differentiable Window for Dynamic Local Attention (2020)

Abstract: We propose Differentiable Window, a new neural module and general purpose component for dynamic window selection. While universally applicable, we demonstrate a compelling use case of utilizing Differentiable Window to improve standard attention modules by enabling more focused attentions over the input regions. We propose two variants of Differentiable Window, and integrate them within the Transformer architecture in two novel ways. We evaluate our proposed approach on a myriad of NLP tasks, including machine translation, sentiment analysis, subject-verb agreement and language modeling. Our experimental results demonstrate consistent and sizable improvements across all tasks.

## introduction computing relative importance across a series of inputs can be regarded as one of the important advances in modern deep learning research. this paradigm, commonly known as attention @xcite , has demonstrated immense success across a wide spectrum of applications. to this end, learning to compute contextual representations @xcite , to point to the relevant part in the input @xcite , or to select windows or spans @xcite from sequences forms the crux of many modern deep neural architectures. despite aggressive advances in developing neural modules for computing relative relevance @xcite , there has been no general purpose solution for learning differentiable attention windows. while span selectionbased pointer network models typically predict a start boundary and an end boundary @xcite , these soft predictions generally reside at the last layer of the net-work and are softly optimized. to the best of our knowledge, there exists no general purpose component for learning differentiable windows within networks. although the practical advantages of learning differentiable windows are plenty, this paper focuses on improving attentions with differentiable windows. the key idea is to enable more focused attention, leveraging dynamic window selection for limiting (and guiding) the search space for the standard attention modules to work within. this can also be interpreted as performing a form of dynamic local attention. we make several key technical contributions. first, we formulate the dynamic window selection problem as a problem of learning a discrete mask (i.e., binary values representing the window). by learning and composing left and right boundaries, we show that we are able to parameterize the (discrete) masking method. we then propose soft adaptations of the above mentioned, namely trainable soft masking and segment-based soft masking, which are differentiable approximations that can not only be easily optimized in an end-toend fashion, but also inherit the desirable properties of discrete masking. while these modules are task and model agnostic, we imbue the state-of-the-art transformer @xcite model with our differentiable window-based attention. to this end, we propose two further variants, i.e., multiplicative window attention and additive window attention for improving the transformer model. within the context of sequence transduction and self-attention based encoding, learning dynamic attention windows are beneficial because they can potentially eliminate noisy aggregation and alignment from large input sequences. on the other hand, it is good to note that hard attention @xcite , which replaces the weight average of soft attention with a stochas-tic sampling model, tries to achieve similar ends, albeit restricted to token-level selection. hence, our proposed differentiable windows are more flexible and expressive compared to hard attentions. we evaluate our transformer model with differentiable window-based attention on a potpourri of nlp tasks, namely machine translation, sentiment analysis, language modeling, and subjectverb agreement. extensive experimental results on these tasks demonstrate the effectiveness of our proposed method. notably, on the english-german and english-french wmt'14 translation tasks, our method accomplishes improvements of 0.63 and 0.85 bleu, respectively. on the stanford sentiment treebank and imdb sentiment analysis tasks, our approach achieves 2.4% and 3.37% improvements in accuracy, respectively. we further report improvements of 0.92% in accuracy and 2.13 points in perplexity on the subjectverb agreement and language modeling tasks, respectively. we make our code publicly available at https://ntunlpsg.github.io/project/ dynamic-attention/ . ## background the attention mechanism enables dynamic selection of relevant contextual representations with respect to a query representation. it has become a key module in most deep learning models for language and image processing tasks, especially in encoder-decoder models @xcite @xcite . ## dynamic differentiable window our proposed attention method works in two steps: (i) derive the attention span for each query vector to attend over, and (ii) compute the respective attention vector using the span. in this section, we present our approaches to step (i) by proposing trainable soft masking and segment-based soft masking. in the next section, we present our methods to compute the attention vectors. to give the necessary background to understand what can be expected from our method, we first present the discrete masking case. ## dynamic window attention methods having presented our method to compute the mask vector that defines the attention spans, we now present our methods to incorporate the mask vectors into the attention layers. ## experiment in this section, we present the training settings, experimental results and analysis of our models in comparison with the baselines on machine translation (mt), sentiment analysis, subject verb agreement and language modeling (lm) tasks. ## conclusion we have presented a novel differential window method for dynamic window selection, and used it to improve the standard attention modules by enabling more focused attentions. specifically, we proposed trainable soft masking and segmentbased masking, which can be applied to encoder/decoder self-attentions and cross attention. we evaluated our models on four nlp tasks including machine translation, sentiment analysis, subject verb agreement and language modeling. our experiments show that our proposed methods outperform the baselines significantly across all the tasks. all in all, we demonstrate the benefit of incorporating the differentiable window in the attention. in the future, we would like to extend our work to make a syntactically-aware window that can automatically learn tree (or phrase) structures.