Discourse Self-Attention for Discourse Element Identification in Argumentative Student Essays (2020)

Abstract: This paper proposes to adapt self-attention to discourse level for modeling discourse elements in argumentative student essays. Specifically, we focus on two issues. First, we propose structural sentence positional encodings to explicitly represent sentence positions. Second, we propose to use inter-sentence attentions to capture sentence interactions and enhance sentence representation. We conduct experiments on two datasets: a Chinese dataset and an English dataset. We find that (i) sentence positional encoding can lead to a large improvement for identifying discourse elements; (ii) a structural relative positional encoding of sentences shows to be most effective; (iii) inter-sentence attention vectors are useful as a kind of sentence representations for identifying discourse elements.

## introduction discourse describes how a document is organized. this paper focuses on the task of discourse element identification (dei) in argumentative student essays. discourse elements represent the function and contribution of every discourse unit to the discourse. @xcite formulate discourse elements as 5 categories: introduction, thesis, main idea, supporting and conclusion, while argument components such as major claim, claim and premise are used as discourse elements in argumentation structure parsing in persuasive essays @xcite . dei can benefit automated essay scoring in many aspects: modeling organization, inferring topics and opinions or used as features for scoring systems @xcite @xcite . despite its importance, dei is challenging. first, the ambiguity of sentences makes learning models difficult to distinguish some discourse elements. for example, the thesis is defined as expressing the central claim of the author and the main ideas support the thesis from specific aspects. however, it is hard to distinguish them from their content and style. second, the discourse element of a specific sentence depends on context. as a result, considering individual sentences only would have difficulties in identifying discourse elements. the relations and relatedness among multiple sentences should be explored. third, the data imbalance problem is serious, e.g., the number of elaboration sentences could be 10 times more than the number of thesis sentences. the minority discourse elements (such as thesis, main ideas or major claim) are harder to be recalled although they have important roles in many scenarios, e.g., evaluating the organization of essays @xcite . in this paper, we propose a method to explicitly model sentence positions and relations to improve discourse element identification in argumentative student essays. our idea is partially motivated by the self-attention mechanism such as @xcite . self-attention is usually applied to capture dependencies between words. we aim to apply self-attention mechanism to describe relations between sentences. on one hand, position information is important for dei to give clues on discourse elements beyond content and style, because authors usually hold some conventions to organize content. position is one of the most useful feature classes in feature-based dei @xcite . previous neural network models usually cast dei as a classification or sequence labeling task and do not explicitly model position information. motivated by the positional encoding of words, we propose a simple structural positional encoding strategy for a sentence by considering its relative position in its essay, relative position of its paragraph in its essay, and its relative position within its paragraph. on the other hand, relatedness among sentences may also indicate properties of discourse elements. for example, thesis sentences should have close relations to the whole essay; main ideas usually locate in similar positions and have high relatedness. relatedness between discourse elements has shown to be an important indicator of essay coherence @xcite . we compute inter-sentence attention vectors to represent either element-wise or content-wise relations to other sentences, which bring in additional information beyond individual sentences and enhance sentence representation without extra information. experiments show that the proposed approach can get considerable improvements compared with feature-based and neural network based baselines on a chinese dataset and obtain competitive results compared with the state-of-the-art method on an english dataset. the structural positional encodings of sentences show effectiveness to achieve obvious overall improvements. the inter-sentence attention vectors enhance sentence representation helping identify discourse elements as well. ## baseline we use hierarchical bilstm (hbilstm), which is similar to @xcite , as the base model to model sentence and discourse level representations. the task is to assign discourse element labels @xmath0, ..., y n ) to sentences (x 1 , ..., x n ) in a text, where x i , 1 ≤ @xmath1, is a sentence of a sequence of words and y @xmath2, y is a set of pre-defined discourse elements. ## discourse self-attention we propose the discourse self-attention (disa) layer to improve the baseline by explicitly modeling sentence positions and inter-sentence interactions. the architecture is illustrated in figure 1 . the sentences in an essay are converted to sentence embeddings c through the bilstm encoder introduced in section 3.1, which are used as the input of disa. disa explicitly represents sentence positions, which are integrated with the content representations of sentences to get element representations. disa also has an inter-sentence attention module to get both element-wise and content-wise attention vectors of sentences to capture sentence interactions. the attention vectors and element representations are concatenated and fed to a linear layer and a softmax layer for prediction. ## experiment 6.1 experimental settings the max length of sentences is set to 40 words. sentences are padded or truncated according to this length. the tencent pre-trained word embeddings @xcite were used for experiments on the chinese dataset. the dimension of word embeddings is 200. the bert tokenizer and embeddings were used for experiments on the english dataset. the dimension of all the bilstm hidden layers is 256 on chinese dataset, and 128 on english dataset. so is the dimension of d k . the dimension of the attention vectors is 15. the optimizer is stochastic gradient descent (sgd) with a learning rate 0.1. the best models were selected for all settings based on the results on the validation data, which is 10% of the training data. we use accuracy (acc.) and macro-f1 as evaluation metrics. ## conclusion we presented a method disa to identify discourse elements in argumentative student essays by explicitly modeling structural positions and inter-