Boundary Detection and Categorization of Argument Aspects via Supervised Learning (2022)

Abstract: Aspect-based argument mining (ABAM) is the task of automatic _detection_ and _categorization_ of argument aspects, i.e. the parts of an argumentative text that contain the issue-specific key rationale for its conclusion. From empirical data, overlapping but not congruent sets of aspect categories can be derived for different topics. So far, two supervised approaches to detect aspect boundaries, and a smaller number of unsupervised clustering approaches to categorize groups of similar aspects have been proposed. With this paper, we introduce the Argument Aspect Corpus (AAC) that contains token-level annotations of aspects in 3,547 argumentative sentences from three highly debated topics. This dataset enables both the supervised learning of boundaries and categorization of argument aspects. During the design of our annotation process, we noticed that it is not clear from the outset at which contextual unit aspects should be coded. We, thus, experiment with classification at the token, chunk, and sentence level granularity. Our finding is that the chunk level provides the most useful information for applications. At the same time, it produces the best performing results in our tested supervised learning setups.

## related work during the last years, several approaches to grouping arguments into some type of semantic categories were published in the field of argument mining. to describe their task, these approaches rely on heterogeneous names, theoretical concepts, and mining strategies. a first group of approaches builds on framing theory that is commonly used in empirical communication and media research. in argument mining, the notion of a frame is adopted as the aspect of a discussion that is emphasized by an argument. sets of aspects can be of varying breadth and depth. also, approaches differ whether they assume frames to be issue-specific or should generalize across topics. @xcite , for instance, define a frame as a set of arguments that focus on the same aspect. to identify references to the same aspects, they use an unsupervised clustering on argumentative texts. by definition then, each cluster supposedly represents one frame. however, the resulting clusters do not necessarily describe semantic frames in the sense of repeatedly occurring aspects of the corresponding discussion. the large number of optimal clusters as described in the paper also drastically reduces the usefulness for any further study. @xcite define frames as the aspects a talking point discusses. they address the shortcomings of frames that are too generic and frames that are too issuespecific by clustering user-generated, specific labels into general frame categories from classic media research. although they have shown that their approach is able to automatically identify media frames to some extent, they do not provide welldefined sets of issue-specific aspects that would allow for a deductive analysis of public debates. @xcite describe a clusteringbased grouping of arguments based on aspects for better search results. they use agglomerative hierarchical clustering of contextualized word embeddings, such as bert-embeddings @xcite , on sentence-level argument pairs. the resulting clusters based on similarity metrics also do not necessarily provide useful aspect categories, let alone semantically meaningful labels. @xcite introduced key point analysis to generate a summary for large collections of arguments by finding key points. their work also inspired the argmining 2021 shared task @xcite which contained one task for matching arguments to key points, and one task for the generation of key points. hereby, key points are defined as higher-level arguments that occur frequently in debates on a given topic. key points are formulated as full sentences and with an indication of a clear pro or contra stance on the debated issue. besides the difference that in our definition aspects are independent of any stance, key points can play a similar role in argument classification as our proposed aspects. they also acknowledge the difficulty of the problem of argument grouping and the ineffectiveness of unsupervised methods based on contextual embeddings. addressing the problem of unsupervised approaches, @xcite propose abam as a multi-class sentence classification task and provide a corpus containing argumentative sentences from the nuclear energy debate with manually annotated class labels. in a further development of this work, our approach to abam is designed as a token-level sequence tagging task that allows for multiple aspects to being mentioned in one sentence, and for the extraction of the decisive sentence parts determining these aspects. annotating and predicting aspects on the token level is also performed in the works of trautmann (2020) and @xcite . @xcite defines aspects analog to aspect-based sentiment analysis @xcite . he proposes the task of aspect term extraction (ate) and presents a supervised sequence tagging approach to detect the most common token n-grams that address argument aspects. however, no semantically meaningful aspect categories are created from the extracted token sequences. similar to ate, @xcite perform aspect boundary detection as a supervised sequence tagging task trained on argumentative sentences in which token sequences were labeled with a bio-tagging scheme to indicate the beginning (b), inside (i) and outside (o) of token aspect spans. they also address the problem of fuzzy span boundaries that motivated our research and present a crowdsourcing task based on automatic candidate ranking and manual candidate selection to create a gold standard with high inter-coder agreement. regarding this task of aspect boundary detection, their approach to abam mostly resembles ours. we, however, extend the tagging and extraction of aspect terms to a classification of the predicted spans into issue-specific aspect categories. ## the argument aspect corpus with this paper, we publish the argument aspect corpus (aac) that contains manually annotated aspect labels on token spans from argumentative sentences. the argumentative sentences were extracted from the ukp sentential argument mining corpus (ukp sam) @xcite . for the aac, we selected only those sentences that have been annotated as either expressing a pro or a contra stance on one of the three topics: minimum wage (mw), nuclear energy (ne), and marijuana legalization (mj). the topics were chosen with respect to their importance within recent european political discourses. as bar-haim et al. (2020) and jurkschat et al. (2022) have already pointed out, labeling aspects in arguments is a complex task. this is mainly due to the fact that the granularity of aspects cannot be determined in a data-driven manner, but must be specified in a methodically rigorous process of developing the coding scheme. with this comes the necessity to develop definitions of aspect categories that are as precise as possible to separate the sometimes overlapping meanings of argumentative components from one another. to fulfill these requirements and, at the same time, address the heterogeneity of the empirical data, we followed a process that combined unsupervised clustering with group discussions to reach consensus definitions of our aspect categories. as a starting point, we employed unsupervised k-means clustering of sentence embeddings from s-bert @xcite . analog to previous research on argument frames, we expect that semantic-similarity-based clusters already group aspect information to some extent. we decided on a fixed number of 15 initial clusters as a rough estimate of how many aspects per topic we expect. however, our subsequent development of aspect categories would allow for the creation of more or fewer aspect categories. with a group of three annotators, students, and researchers from the field of (computational) social science, we listed aspects that occur in these clusters as a first summary of a topic. with these initial aspects, we created a preliminary codebook and annotated a sample of 200 sentences per topic. arguments in these samples were sorted by cosine similarity of their s-bert representation. annotators reported that this sorting was beneficial for speeding up the annotation and, at the same time, increasing its coherence. annotators were encouraged to write comments about aspect categories and extend the list of aspects if necessary. next, the inter-coder agreement (krippendorff's alpha) for each aspect was calculated on a sentence level in order to find aspects that need clarification. in extensive discussion rounds, the category definitions were sharpened and refined. in the second and following rounds of annotating samples, we switched from the sentence level classification to an annotation of token spans to be able 1) to justify label decisions directly on text snippets, and 2) to allow for aspect term extraction in a subsequent step of machine learning. this iterative process of annotation, agreement evaluation, and discussion was repeated until a consensus for all aspect definitions was reached and the list of aspects covered the large majority of arguments for a topic. the full dataset was then annotated by all three annotators with the final codebook resulting from the aforementioned iterative process. a major challenge during annotation was determining token span boundaries since in many cases it is not possible to unambiguously decide where the mentioning of an aspect in a sentence actually starts or ends. we decided to instruct annotators to label the smallest number of tokens that provide sufficient information to label the aspect on its own. still, this resulted in substantial disagreement about aspect boundaries in many cases while, at the same time, sentence-level agreement of labeled aspects was high. this observation led to the decision to further investigate the question of which granularity level of context units abam should be performed (cp. section 5). final gold labels for the aac dataset on the token level were derived in a two-step process. first, on the sentence level, we determined all labels that have been annotated by a majority of annotators as gold labels. arguments without any majority label were reviewed once again to determine a final label. second, for each token in a sentence, we copied the sentence gold label if at least one annotator included it in his/her annotation span. again, rare conflicts of overlaps of sentence majority labels for individual tokens have been resolved in a final review. this strategy results in potentially more extensive gold labels on the token level compared to those of the single annotators. ## experimenting with aspect boundaries in a significant number of cases, annotators agreed upon which aspects were present in an argument but labeled slightly different token sequences as indicative of an aspect. therefore, the strict tokenlevel annotator agreement was relatively low compared to the agreement on the sentence level. a qualitative look into boundary disagreement for a small sample revealed that different individual annotations could be considered valid regarding our guidelines. this challenge to achieve a high agreement for exact matches of token span boundaries during aspect annotation led us to the more general questions: what would be the most suitable level of granularity of context units, and what would be the best corresponding modeling approach to perform abam as a machine learning task? to answer these questions, we experiment with different modifications of the aac dataset. since the category other was used to annotate any sentence that either did not fit any aspect definition or was deemed not argumentative, we excluded the category from training. then, we split the annotated data per topic randomly into a training (70 %), validation (10 %), and test set (20 %), finally, we created different formats of these sets to test different abam task variants: task variant sentence representation examples sequence tagging [after] [the] [wage] [increase] [,] [that] [same] [basket] prices [cost] prices [@xmath0] [ 315] [.] chunk classification [that same basket [sep] after the wage increase , that same basket cost @xmath1315.] prices ## conclusion in this paper, we further defined the task of supervised aspect-based argument mining based on experiments with a newly created dataset containing aspect annotations of token spans in argumentative sentences from three different topics. with our experiments, 8 we showed that abam performs best on a granularity level finer than multi-label sentence classification (cp. exp. 2). we also showed that best results are achieved by fine-tuning a stateof-the-art language model such as roberta on a token sequence tagging task. despite satisfactory results up to 70 % f1-score (cp. exp. 1), we see that especially disagreement on span boundaries for annotated aspects is a source of error. normalizing token labels in the gold dataset to identical labels within syntactic chunks can mitigate this effect to some extent (cp. exp. 2). compared to sentences that can refer to multiple aspects, chunks are short enough to carry information for only one aspect. compared to tokens, chunks contain more