Using a Penalty-based Loss Re-estimation Method to Improve Implicit Discourse Relation Classification (2020)

Abstract: We tackle implicit discourse relation classification, a task of automatically determining semantic relationships between arguments. The attention-worthy words in arguments are crucial clues for classifying the discourse relations. Attention mechanisms have been proven effective in highlighting the attention-worthy words during encoding. However, our survey shows that some inessential words are unintentionally misjudged as the attention-worthy words and, therefore, assigned heavier attention weights than should be. We propose a penalty-based loss re-estimation method to regulate the attention learning process, integrating penalty coefficients into the computation of loss by means of overstability of attention weight distributions. We conduct experiments on the Penn Discourse TreeBank (PDTB) corpus. The test results show that our loss re-estimation method leads to substantial improvements for a variety of attention mechanisms, and it obtains highly competitive performance compared to the state-of-the-art methods.

## introduction the goal of pairwise sentence-level discourse analysis is to determine the relation that is held by a pair of arguments @xcite , where an argument generally stands for a narrative sentence. implicit discourse relation classification is a challenging subtask. it is required to determine the relation on the condition that the explicit connective (i.e., a syntactic conjunction) is not given. for example, the arguments in figure 1 are a pair of semantically-related arguments, where the possible connective "but" that may signal the comparison relation has been omitted. argument 1 (arg1): robert s. enrlich resigned as chairman, president and chief executive. argument 2 (arg2): mr. enrlich will continue as a director and a consultant. ## approach we utilize bert @xcite as the baseline encoder, and connect it with a multi-layer perceptron (mlp) to form the discourse relation classifier. in addition, we reproduce three attention mechanisms, including self @xcite , interactive @xcite and multi-layer @xcite attention mechanisms. on the basis, we couple them with the baseline (bert) encoder. similarly, they are also connected with a mlp respectively for discourse relation classification. it is noteworthy that the attention mechanisms mentioned above have been carefully studied on pdtb v2.0. though, they were built over some slightly weak word embeddings. for fair comparison, we choose to couple them with the pre-trained bert encoder. bert is fine-tuned in all of our experiments. assume that a certain neural attention mechanism is defined as f(h 1 , h 2 , θ), we tend to optimize θ. in the equation, h 1 and h 2 denotes the encoder states of a pair of arguments which are obtained using the pre-trained bert, and θ stands for the shorthand of all parameters of the attention mechanism. we specify the parameters of all the considered attention-based computational models in ## experimentation we evaluated our model on the benchmark pdtb v2.0 @xcite . the four main relation classes are considered in the experiments, including comparison (abbr., com), contingency (con), expansion (exp) and temporality (tem). we follow the previous work @xcite to split datasets, using sections 02-20 as the training set, sections 00-01 the development set and sections 21-22 the test set. for comparison purpose, we use f1 as the evaluation metric. we utilize bert that outputs word embeddings with the hidden size of 768. there are 12 self-attention heads considered in bert. the max length of the input sequence is set to 163, in which the maximum length n of an argument is set to 80. in addition, the batch size is set to 15, and gradient descent is set separately: β 1 =0.9 and β 2 =0.999. the learning rate is set to 5e-5 and dropout rate is set to 0.1. we set the hyperparameter λ to 1e-3 for both the multi-layer and self-attention mechanisms, and 1e-2 for the interactive attention mechanism. the main test results are shown in we compare our method to the state-of-the-art. as shown in ## related work recently, neural networks have been widely studied for argument representation learning @xcite , which is admitted to be the crucial issue for discourse relation recognition. due to the capacity of generating low-dimensional continuous representations for arguments, rnns with bi-lstm are used during encoding. chen et al (2016) couple bi-lstm with a gated relevance model. liu and li (2016) use multi-layer attention computation over the output of bi-lstm. meanwhile, liu et al (2016) build a multi-task learning framework with convolutional neural network (cnn) for argument encoding. by contrast, lan et al (2017) integrate bi-lstm into the multi-task framework and couple it with the attention mechanism. guo et al (2018) utilize the interaction mechanism to weight the representations emitted by bi-lstm, and perform a deeper encoding by tensor network. dai and huang (2018) use bi-lstm to bring paragraph-level contextual information into argument representations. in addition, @xcite build a hybrid neural model which couples two gated cnns to extract both word-level and semantic-level convolutional features. further, @xcite integrate generative adversarial networks into multi-task learning network. hereafter, @xcite establish multitask network using multi-layer gated cnns. the network is additionally coupled with residual networks and interactive attention mechanisms. @xcite enhance @xcite 's multi-layer cnns-based multi-task learning by minimizing the divergence between connective-level embeddings and relation-level embeddings. he et al (2020) develop a joint learning architecture which updates both geometric and semantic features during encoding. ## conclusion our experiments demonstrate that the utilization of penalty coefficients for loss re-estimation can effectively strengthen the attention-based implicit discourse relation classification. nevertheless, our survey shows that some attention-worthy words fails to be effectively perceived by the current attention mechanisms. more importantly, the semantics of such kind of attention-worthy words can be well-encoded only through the understanding of related common sense. therefore, in the future, we will utilize commonsense knowledge graph to enhance the attention modeling method.