Contextual Argument Component Classification for Class Discussions (2020)

Abstract: Argument mining systems often consider contextual information, i.e. information outside of an argumentative discourse unit, when trained to accomplish tasks such as argument component identification, classification, and relation extraction. However, prior work has not carefully analyzed the utility of different contextual properties in context-aware models. In this work, we show how two different types of contextual information, local discourse context and speaker context, can be incorporated into a computational model for classifying argument components in multi-party classroom discussions. We find that both context types can improve performance, although the improvements are dependent on context size and position.

## motivation in a typical argument mining system, the first task is identifying spans of text consisting of argumentative discourse units (adu), i.e. argument component identification (aci). the next task, argument component classification (acc), consists of assigning a label to each adu according to an argument model, e.g. claims, evidence, etc. for example, row 1 in while "context" has been used in the argument mining literature to refer to several phenomena, we consider context to be auxiliary textual information outside the span of an adu. it is generally accepted that context is important in argument mining. @xcite as well as @xcite use context features extracted from the sentence containing an adu to improve acc. @xcite , @xcite and @xcite similarly use contextual features in joint aci/acc models. @xcite analyze a previous argument mining system and find that, for its predictions, it relies on context more than it does on adu content. @xcite indirectly model context in acc by fine-tuning a bert model to predict the next sentence (i.e. the context). @xcite analyzed several neural models for jointly performing aci, acc and argument relation extraction. all of these works share several limitations: (i) the context is either limited to a single configuration (e.g. one sentence before/after the adu) or optimized along a single dimension (typically size but not position); (ii) only a subset of the features extracted for the target adu are also extracted for context; (iii) context is typically based on adu adjacency, although other ways of building context (e.g. based on speakers in multi-party dialogues) are possible. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . row id adu ac 1 7 i feel that she included herself in the book to give more of an unbiased point of view to it. claim 2 7 because, obviously the family was going to be completely against the doctors and what they did, and all that. evidence 3 7 but i feel by her including herself in it, she did show both sides of the story. warrant 4 10 i'd argue against that keeping her as unbiased. because i think it puts more of a bias on the lacks family's side. claim 5 7 i also feel she included herself because this was such a big part of her career from learning all this. and she became attached to the family members. claim in this work, we improve upon baseline acc models for multi-party discussions by incorporating two types of contexts. we define local context as adus preceding and/or following a target adu, regardless of speaker. speaker context consists of previous adus that a specific speaker previously voiced during the discussion. our results show that both context types can individually and jointly improve acc performance, with performance gains dependent on context size and position. ## dataset and models the dataset foot_0 used to build and evaluate our proposed acc models consists of 3,135 adus in a corpus of 29 text-based (i.e. centered around a book, play, or other literature piece), multi-party classroom discussions between high school students (average 15 students per discussion) @xcite . the discussions (average length of 34 minutes) were audio-recorded, manually transcribed, and student turns were manually segmented into multiple adus when needed. adus were then manually annotated according to a simplified version of toulmin's argumentative model @xcite consisting of three labels: (i) claims, arguable statements that voice a specific interpretation of a text; (ii) evidence, facts or documentation used to support a claim; (iii) warrant, reasoning given to explain how certain evidence supports a claim. an inter-rater reliability analysis showed a krippendorff α u of 0.96 for segmentation and cohen kappa of 0.74 for argument components. the dataset contains 3,135 adus: 65.3% claims, 24.3% evidence, and 10.4% warrants. adus are additionally labeled with the id of the speaker who voiced the utterance. to evaluate the utility of context, we introduce two baseline acc models and propose several contextual extensions. the source code for all models (and all parameters) is available at https: //github.com/lucalugini/coling2020_argmining . baseline models. our first model (hybrid baseline) is based on the model of @xcite which was developed for a similar type of dataset, where an embedding generated through a convolutional neural network (cnn) is concatenated to a set of handcrafted features, and a softmax classifier is used to predict argument components. given the limited size of our dataset, however, we only use a subset of the original model's handcrafted features, namely those used by @xcite ; this reduces the number of handcrafted features from over 7,000 to 114 and avoids overfit. the speciteller feature set consists of pretrained word vectors (average of the word vectors for each word in the adu), as well as number of connectives, number of words, number of numbers, number of symbols, number of capital letters, number of stopwords normalized by adu length, number of subjective and polar words (from the mpqa @xcite and the general inquirer (stone and hunt, 1963) dictionaries), average word familiarity (from mrc psycholinguistic database @xcite ), average characters per word, and inverse document frequency statistics (minimum and maximum). the dimensionality of the final feature vector is 2,514 (114 for the handcrafted features and 2,400 for the cnn). our second model (bert baseline) is based on recent advances related to transformer architectures @xcite : a bert pretrained model @xcite generates word embeddings of dimensionality 768; average pooling is used to compute a fixed-size adu embedding; a softmax classifier predicts argument components. ## experiments and results all models are evaluated using ten fold cross-validation, and results are shown in local context. we extended both baseline models with local context extracted in three different ways: only adus preceding the target, only following adus, and both preceding/following adus. we report three main observations when adding context to the hybrid baseline. first, with respect to position, all models including prior adus in local context significantly outperformed the baseline (@xmath0.01), while the same is not true for models including next adus. including both prior and next adus, though, resulted in the best performance for local context models (row 2 in ## conclusions and future work in this paper we analyzed the impact of context for predicting argument components in multi-party discussions. we defined two types of context, local context and speaker context, and analyzed how different models perform when varying context size. we also investigated the effect of different positions for local context. we performed evaluations of the two context types separately as well as simultaneously on two types of neural network models. experimental results support our claim that both context size and position are important when incorporating context in argument mining systems, therefore sentences beyond the ones immediately surrounding an adu should be considered. our results also show that speaker context can improve performance in multi-party discussions. finally, we investigated the use of an attention mechanism for optimizing context size and found that its effectiveness is dependent on the type of model used. our future plans include two main directions: 1) repeating our local context experiments on larger datasets, including other domains; 2) evaluating the effectiveness of speaker context on multi-party web discussions, where discussions are usually longer and author id is typically available.