Encoding and Fusing Semantic Connection and Linguistic Evidence for Implicit Discourse Relation Recognition (2022)

Abstract: Prior studies use one attention mechanism to improve contextual semantic representation learning for implicit discourse relation recognition (IDRR). However, diverse relation senses may benefit from different attention mechanisms. We also argue that some linguistic relation in between two words can be further exploited for IDRR. This paper proposes a Multi-Attentive Neural Fusion (MANF) model to encode and fuse both semantic connection and linguistic evidence for IDRR. In MANF, we design a Dual Attention Network (DAN) to learn and fuse two kinds of attentive representation for arguments as its semantic connection. We also propose an Offset Matrix Network (OMN) to encode the linguistic relations of word-pairs as linguistic evidence. Our MANF model achieves the state-of-the-art results on the PDTB 3.0 corpus.

## introduction implicit discourse relation recognition (idrr) is to detect and classify some latent relation in between a pair of text segments (called arguments) without an explicit connective word. it is of great importance for many downstream natural language processing (nlp) applications, such as question answering @xcite , machine translation @xcite , information extraction @xcite , sentiment analysis @xcite , and etc. however, due to the absence of an explicit connective word, inferring discourse relations from the contextual semantics of arguments is still a challenging task. conventional machine learning based methods usually train a relation classifier by using many handmade features to capture lexical, syntactic regularity and contextual information of arguments, which is time-consuming and labor-intensive @xcite . deep learning based methods design diverse neural networks to automatic learn the contextual semantic representation of each argument, such as the shallow conventional neural network (scnn) @xcite , tree-like long short-term memory (tree-lstm) @xcite , and bilstm-cnn framework @xcite . although these neural networks can autonomously learn a kind of deeper contextual semantics of arguments, they do not differentiate arguments' words in the representation learning. recently, some attention mechanisms have been employed in neural networks to unequally treat words in representation learning. for example, the self-attention computes the local contextual importance of each word in one argument, which generally prioritizes content words for better learning substantive meaning of an argument @xcite . the interactive attention weights each word in one argument according to its interaction with the representation of another argument, which usually focuses on the rhetorical device of two arguments, like prioritizing some function words with little substantive meaning but potentially indicating the connection of two arguments @xcite . both kinds of attention mechanisms have been proven effective for idrr, as each can well exploit either content semantics or rhetorical devices of an argument pair. we regard these contextual semantic information derived from argument content as a kind of semantic connection for relation recognition. however, the idrr task normally needs to recognize diverse senses of relations, while different senses may benefit from different attentions. to enjoy both advantages, we propose to learn two kinds of argument representation, each based on one attention mechanism. they are next fused to encode an argument pair as the semantic connection for relation recognition in this paper. besides semantic connection, we argue that a kind of linguistic evidence can be obtained from word distributed representation for relation recognition. indeed, many pre-trained language models, like the word2vec @xcite and bert @xcite , are learned from a large amount of unlabeled text by encoding the linguistic regularities and patterns in an unsupervised way. as a word embedding contains inherent meaning of the word, it can be used to infer some linguistic relation in between two words by linear translation. this motivates us to encode such linguistic relations of word-pairs as linguistic evidence. in this paper, we propose a multi-attentive neural fusion (manf) model to encode and fuse both semantic connection and linguistic evidence for the idrr task. the manf model contains two modules. one is a dual attention network (dan) : it builds upon a bilstm to first encode a selfattentive representation and an inter-attentive representation for each argument. to adapt to different relation senses, we next use a fusion gate to integrate the two representations into the semantic connection representation. another is an offset matrix network (omn): it first computes the offset between word embeddings of a word-pair that contains one word from the first argument and another word from the second argument. upon the offset matrix, we next design an offset attention layer and a multilayer perceptron to encode the linguistic evidence representation. finally, we design another fusion gate to integrate both semantic connection and linguistic evidence representation for relation recognition. our manf model achieve the stateof-the-art results on the pdtb 3.0 corpus. our main contributions are as follows: ## the multi-attentive neural fusion model fig. 1 illustrates our manf model, including the dan, the omn, and a hierarchical fusion mechanism. ## related work the idrr task is usually approached as a classification problem, and the key is the argument representation. machine learning approaches, like using a naive bayes, support vector machine (svm) classifier, have designed various features to capture lexical, syntactic regularity and contextual information as argument representation @xcite @xcite . however, manually crafting features is not only time-consuming and labor-intensive, but also suffers from data sparsity problem due to the use of one-hot feature encoding. deep learning models have prevailed for their capabilities of automatic learning argument representation @xcite . for example, the scnn model @xcite obtains each argument representation via a single convolution layer, and the concatenation of two arguments' representations is used for relation classification. @xcite employ a lstm network to capture word contextual semantics for argument representation. some hybrid models have attempted to combine cnn, lstm, graph convolutional networks and etc. for more sophisticated argument representation @xcite @xcite . these approaches, however, have ignored the fact that different words may contribute differently in argument representation learning. attention mechanisms can guide a neural model to unequally encode each word according to its contextual importance for argument representation @xcite @xcite @xcite . for example, zhou et al. @xcite apply selfattention to weight a word according to its similarity to its belonging argument. guo et al. @xcite ) adopt an interactive attention to differentiae words in one argument, where a word is weighted according to the similarity between its encoding and another argument representation. liu and li @xcite design a multi-level attention to repeatedly compute word importance in a hierarchical way. ruan et al. @xcite propose a pipeline workflow to apply interactive attention after self-attention. word pair features have been exploited in machine learning and deep learning approaches for argument representation @xcite @xcite . for example, biran and mckeown @xcite compute the appearance probabilities of aggregated word pairs to train a logistic regression classifier. chen et al. @xcite ) construct a relevance score word-pair interaction matrix based on a bilinear model @xcite and a single layer neural model @xcite . the proposed manf model is a deep neural model, employing a hierarchical fusion mechanism to fuse two kinds of attentive word encodings as well as word pair offset encodings in argument representation learning. ## concluding remarks in this paper, we argue that implicit relation recognition can benefit from both semantic connection and linguistic evidence between arguments. motivated from such considerations, we have designed the manf model to encode and fuse them for the idrr task. the manf model consists a dan module to fuse both self-attentive and interattentive contextual semantics for learning representation of semantic connection, and a omn module to attentively encode word-pair offsets for learning representation of linguistic evidence. both kinds of representations are finally fused for rela-tion recognition. experiments on the latest pdtb 3.0 corpus have validated our design objectives for the new benchmark performance established by our manf model. this paper has employed the pre-trained word embeddings trained by the word2vec and bert; while other pre-training models shall also be adopted and tested in our future work. the performance differences of recognizing different relation senses also motivate to further investigate other advanced fusion mechanisms.