How Does Selective Mechanism Improve Self-Attention Networks? (2020)

Abstract: Self-attention networks (SANs) with selective mechanism has produced substantial improvements in various NLP tasks by concentrating on a subset of input words. However, the underlying reasons for their strong performance have not been well explained. In this paper, we bridge the gap by assessing the strengths of selective SANs (SSANs), which are implemented with a flexible and universal Gumbel-Softmax. Experimental results on several representative NLP tasks, including natural language inference, semantic role labelling, and machine translation, show that SSANs consistently outperform the standard SANs. Through well-designed probing experiments, we empirically validate that the improvement of SSANs can be attributed in part to mitigating two commonly-cited weaknesses of SANs: word order encoding and structure modeling. Specifically, the selective mechanism improves SANs by paying more attention to content words that contribute to the meaning of the sentence.

## introduction self-attention networks (sans) @xcite have achieved promising progress in various natural language processing (nlp) tasks, including machine translation @xcite , natural language inference @xcite , semantic role labeling @xcite and language representation @xcite . the appealing strength of sans derives from high parallelism as well as flexibility in modeling dependencies among all the input elements. recently, there has been a growing interest in integrating selective mechanism into sans, which has produced substantial improvements in a variety * work done when interning at tencent ai lab. although the general idea of selective mechanism works well across nlp tasks, previous studies only validate their own implementations in a few tasks, either on only classification tasks @xcite or sequence generation tasks @xcite . this poses a potential threat to the conclusive effectiveness of selective mechanism. in response to this problem, we adopt a flexible and universal implementation of selective mechanism using gumbel-softmax @xcite , called selective selfattention networks (i.e., ssans). experimental results on several representative types of nlp tasks, including natural language inference (i.e., classification), semantic role labeling (i.e., sequence labeling), and machine translation (i.e., sequence generation), demonstrate that ssans consistently outperform the standard sans ( §3). despite demonstrating the effectiveness of ssans, the underlying reasons for their strong performance have not been well explained, which poses great challenges for further refinement. in this study, we bridge this gap by assessing the strengths of selective mechanism on capturing essentially linguistic properties via well-designed experiments. the starting point for our approach is recent findings: the standard sans suffer from two representation limitation on modeling word order encoding @xcite and syntactic structure modeling @xcite , which are essential for natural language understanding and generation. experimental results on targeted linguistic evaluation lead to the following observations: ## nlp benchmarks to demonstrate the robustness and effectiveness of the ssans, we evaluate it in three representative nlp tasks: language inference, semantic role labeling and machine translation. we used them as nlp benchmarks, which cover classification, sequence labeling and sequence generation categories. specifically, the performances of semantic role labeling and language inference models heavily rely on structural information @xcite , while machine translation models need to learn word order and syntactic structure @xcite . ## evaluation of word order encoding in this section, we investigate the ability of ssans of capturing both local and global word orders on the bigram order shift detection ( §4.1) and word reordering detection ( §4.2) tasks. ## evaluation of structure modeling in this section, we investigate whether ssans better capture structural information of sentences. to this end, we first empirically evaluate the syntactic structure knowledge embedded in the learned representations ( §5.1). then we investigate the attention behaviors by extracting constituency tree from the attention distribution ( §5.2). ## conclusion in this work, we make an early attempt to assess the strengths of the selective mechanism for sans, which is implemented with a flexible gumbel-softmax approach. through several well-designed experiments, we empirically reveal that the selective mechanism migrates two major weaknesses of sans, namely word order encoding and structure modeling, which are essential for natural language understanding and generation. future directions include validating our findings on other san architectures (e.g., @xcite ) and more general attention models @xcite .