Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used in NLP tasks to capture the longterm and local dependencies respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multidimensional, i.e., feature-wise. A light-weight neural net, "Directional Self-Attention Network (DiSAN)", is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention block with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite this simple form, DiSAN outperforms complicated RNN/CNN models on both prediction quality and efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by about 1.0% on the Stanford Natural Language Inference (SNLI) dataset, and shows the state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Sentences Involving Compositional Knowledge (SICK), TREC Question-type Classification and Multi-Genre Natural Language Inference (MultiNLI) datasets.
REF proposed DiSAN, which is composed of a directional self-attention with temporal order encoded.
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding
ArXiv
ArXiv
Computer Science