Interactively-Propagative Attention Learning for Implicit Discourse Relation Recognition (2020)

Abstract: We tackle implicit discourse relation recognition. Both self-attention and interactive-attention mechanisms have been applied for attention-aware representation learning, which improves the current discourse analysis models. To take advantages of the two attention mechanisms simultaneously, we develop a propagative attention learning model using a cross-coupled two-channel network. We experiment on Penn Discourse Treebank. The test results demonstrate that our model yields substantial improvements over the baselines (BiLSTM and BERT).

## introduction implicit discourse relation recognition (idrr) is required to determine the relationship between arguments, under the condition that there is lack of a connective signaling the relationship. an argument generally stands for a narrative sentence or clause. for example, the arguments (i.e., arg1 and arg2) in figure 1 hold a causal relation, where the possible connective "because" is not given. ## approach (ipal) we show the overall framework of discourse relation classification model in figure 2 , where the pairwise argument analysis is performed. in the basic layer, bilstm is used to encode the arguments. in each bottom-up channel, self-attention is computed over the encoded argument, so as to produce the self-attentive representation. in the interactive layer, the resultant self-attentive representation in one channel will be delivered to the other channel. consequently, in each channel, the interactive attention is computed using the delivered self-attentive representation as the external evidence. this allows the self-attentive information to be propagated to the generator of interactively-attentive representations, in a cross-channel manner. eventually, the representations output by the two channels will be concatenated to produce the final representation. over the final representation, relation classification is conducted, where the dense and softmax layers constitute a multilayer perceptron. ## related work in the earlier work, the traditional machine learning techniques (svm and bayes classifiers) were used for discourse relation classification. meanwhile, a great interest has been devoted to the empirical findings, as well as the exploitation of effective features. one of the most important feature engineering approaches uses interrelated word pairs as the reliable features @xcite since they imply semantic relationships. hereafter, part-of-speech (pos) @xcite , syntactic structures and dependencies @xcite and semantic properties @xcite were used as novel features. recently, neural networks have been widely studied for argument representation learning @xcite , which is admitted to be the crucial issue for discourse relation recognition. due to the capacity of generating low-dimensional continuous representations for arguments, rnns with bi-lstm are used during encoding. @xcite couple bi-lstm with a gated relevance model. @xcite use multi-layer attention computation over the output of bi-lstm. meanwhile, @xcite build a multi-task learning framework with convolutional neural network (cnn) for argument encoding. by contrast, @xcite integrate bi-lstm into the multi-task framework and couple it with the attention mechanism. @xcite utilize the interaction mechanism to weight the representations emitted by bi-lstm, and perform a deeper encoding by tensor network. @xcite use bi-lstm to bring paragraph-level contextual information into argument representations. in addition, @xcite build a hybrid neural model which couples two gated cnns to extract both word-level and semantic-level convolutional features. further, @xcite integrate generative adversarial networks into multi-task learning network. hereafter, bai and zhao (2018) establish multi-task network using multi-layer gated cnns. the network is additionally coupled with residual networks and interactive attention mechanisms. @xcite enhance bai and zhao (2018)'s multi-layer cnns-based multi-task learning by minimizing the divergence between connective-level and relation-level embeddings. ## conclusion the binary classification of implicit discourse relations still remains a great challenge. although sophisticated representation learning models are deliberately used, the best performance achieved for a specific relation so far is less than 76% or even worse. we suggest that, in many cases, discourse relation recognition heavily relies on common-sense knowledge. in the future, we will attempt to acquire closely-related knowledge for attentive words and model their knowledge graphs. on the basis, attention-aware graph convolutional networks will be integrated into the existing idrr architectures.