“Sharks are not the threat humans are”: Argument Component Segmentation in School Student Essays (2021)

Abstract: Argument mining is often addressed by a pipeline method where segmentation of text into argumentative units is conducted first and proceeded by an argument component identification task. In this research, we apply a token-level classification to identify claim and premise tokens from a new corpus of argumentative essays written by middle school students. To this end, we compare a variety of state-of-the-art models such as discrete features and deep learning architectures (e.g., BiLSTM networks and BERT-based architectures) to identify the argument components. We demonstrate that a BERT-based multi-task learning architecture (i.e., token and sentence level classification) adaptively pretrained on a relevant unlabeled dataset obtains the best results.

## introduction computational argument mining focuses on subtasks such as identifying the argumentative discourse units (adus) @xcite , their nature (i.e., claim or premise), and the relation (i.e., support/attack) between them @xcite @xcite @xcite @xcite . argumentation is essential in academic writing as it enhances the logical reasoning, as well as, critical thinking capacities of students @xcite . thus, in recent times, argument mining has been used to assess students' writing skills in essay scoring and provide feedback on the writing @xcite @xcite . ## related work the majority of the prior work on argument mining addressed the problems of argument segmentation, component, and relation identification modeled in a pipeline of subtasks @xcite @xcite except a few research @xcite . however, most of the research assumes the availability of segmented argumentative units and do the subsequent tasks such as the classification of argumentative component types @xcite @xcite , argument relations @xcite , and argument schemes @xcite . previous work on argument segmentation includes approaches that model the task as a sentence classification to argumentative or nonargumentative sentences @xcite @xcite @xcite @xcite , or by defining heuristics to identify argumentative segment boundaries @xcite @xcite . although we conduct segmentation, we focus on the token-level classification to directly identify the argument component's type. this setup is related to @xcite where authors analyzed students' diagnostic reasoning skills via token level identification. our joint model using bert is similar to @xcite . however, we set the main task as the token-level classification where the auxiliary task of argumentative sentence identification assists the main task to attain a better performance. as stated earlier, most of the research on argumentative writing in an educational context focuses on identifying argument structures (i.e., argument components and their relations) @xcite as well as to predict essays scores from features derived from the essays (e.g., number of claims and premises, number of supported claims, number of dangling claims) @xcite . related investigations have also examined the challenge of scoring a certain dimension of essay quality, such as relevance to the prompt @xcite , opinions and their targets @xcite , argument strength @xcite among others. majority of the above research are conducted in the context of college-level writing. for instance, @xcite investigated argument structures in toefl11 corpus @xcite which was also the main focus of @xcite and @xcite analyzed writing of university students and @xcite used data from "essayforum.com", where college entrance examination is the largest forum. although, writing quality in essays by young writers has been addressed @xcite @xcite , identification of arguments was not part of these studies. computational analysis of arguments from school students is in infancy except for a few research @xcite @xcite . we believe our dataset (section 3) will be useful for researchers working at the intersection of argument mining and education. ## data we obtained a large number of english essays (over 10k) through the writing mentor educational app. this app is a google docs add-on designed to provide instructional writing support, especially for academic writing. the addon provides students to write argumentative or narrative essays and receive feedback on their writings. we selected a subset of 145 argumentative essays for the annotation purpose. essays were either self-labeled as "argumentative" or annotators identified their argumentative nature from the titles (e.g., "should artificial sweeteners be banned in america ?"). 3 essays covered various social issues related to climate change, veteran care, e↵ects of wars, whether sharks are dangerous or not, etc. we denote this corpus as arg2020 in the remaining sections of the paper. we employed three expert annotators (with academic and professional background in linguistics and education) to identify the argument components. the annotators were instructed to read sentences from the essays and identify the claims (defined as, "a potentially arguable statement that indicates a person is arguing for or arguing against some-thing. claims are not clarification or elaboration statements.") that the argument is in reference to. next, once the claims are identified, the annotators annotated the premises (defined as, "reasons given by either for supporting or attacking the claims making those claims more than mere assertions"). 4 earlier research has addressed college level writing, and even such resources are scarce except for a few corpora @xcite (denoted as sg2017 in this paper). on the contrary, arg2020 is based on middle school students writing, which di↵ers from college level writing sg2017 in several aspects briefly discussed in the next paragraph. first, we notice that essays in sg2017 maintain distinct paragraphs such as the introduction (initiates the major claim in the essay), the conclusion (summarizes the arguments), and a few paragraphs in between that express many claims and their premises. however, essays written by middle school students do not always comply with such writing conventions to keep a concrete introduction and conclusion paragraph, rather, they write many short paragraphs (7-8 paragraphs on average) per essay while each paragraph contains multiple claims. second, in general, claims in college essays in sg2017 are justified by one or multiple premises, whereas arg2020 has many unsupported claims. for instance, the excerpt from the annotated essay in figure 1 contains two unsupported claims (e.g., "diet soda, sugar . . . without the calories" and "artificial sweeteners . . . health e↵ects"). third, middle school students often put opinions (e.g., "sugar substitutes produce sweet food without feeling guilty consequences") or matter-of-fact statements (e.g., "even canned food and dairy products can be artificially sweetened") that are not argumentative claims but structurally they are identical to claims. fourth, multiple claims frequently appear in a single sentence that are separated by discourse markers or commas. fifth, many essays contain run-on sentences (e.g., "this is hard on the family, they have a hard time adjusting") that make the task of parsing even tricky. we argue these reasons make identifying argument claims and premises from arg2020 more challenging. the annotators were presented with specific guidelines and examples for annotation. we conducted a pilot task first where all the three annotators annotated ten essays and exchanged their notes for calibration. following that, we continued pair-wise annotation tasks (30 essays for each pair of annotators), and finally, individual annotators annotated the remaining essays. since the annotation task involves identifying each argumentative component's words, we have to account for fuzzy boundaries (e.g., in-claim vs. not-in-claim tokens) to measure the iaa. we considered the krippendor↵'s ↵ @xcite metric to compute the iaa. we measure the ↵ between each pair of annotators and report the average. for claim we have a modest agreement of 0.71 that is comparable to @xcite and for premise, we have a high agreement of 0.90. out of the 145 essays from arg2020 we randomly assign 100 essays for training, 10 essays for dev, and the remaining 35 essays for test. ## experimental setup majority of the argumentation research first segment the text in argumentative and nonargumentative segments and then identify the structures such as components and relations @xcite . @xcite mentioned that the granularity of computational approaches addressing the second task of argument component identification is diverse because some approaches consider detecting components at the clause level (e.g., approaches focused on the sg2017 corpus @xcite @xcite ) and others at the sentence levels @xcite . we avoided both approaches for the following two reasons. first, middle school student essays often contain run-on sentences, and it is unclear how to handle clause level annotations because parsing might be inaccurate. second, around 62% of the premises in the training set appears to be in the same sentence as their claims. this makes sentence classification to either claim or premise impractical (figure 1 contains one such example). thus, instead of relying on the pipeline approach, we tackle the problem by identifying argument components from the token-level classification akin to @xcite . our unit of sequence tagging is a sentence, unlike a passage @xcite . we apply a five-way tokenclassification (or sequence tagging) task while using the standard bio notation for the claim and premise tokens (see we explore three separate machine learning approaches well-established for studying token-based classification. first, we experiment with the sequence classifier conditional random field (crf) that exploits state-of-theart discrete features. second, we implement a bilstm network (with and without crf) based on the bert embeddings. finally, we experiment with the fine-tuned bert models with/without multitask learning setting. ## results and discussion we present our experiments' results using the crf, bilstm, and bert models under different settings. we report the individual f1, accuracy, and macro-f1 (abbrev. as "acc." and "f1") scores for all the categories in we apply the discrete features (structural, syntactic, lexico-syntactic ("lexsyn")) together and individually to the crf model. we observe the structural and syntactic features do not perform well individually, especially in the case of premise tokens (see the overall performance(s) improve when we apply the bilstm networks on the test data. accuracy improves by 5.3% in the case of bilstm against the embeddings+lexsyn features. however, results do not improve when we augment the crf classifier on top of the lstm networks (bilstm-crf). instead, the performance drops by 0.8% accuracy (see in the case of bert based experiments, we observe bert bl , obtains an accuracy of 73% that is comparable to the bilstm performance. in terms of the individual categories, we observe bert bl achieves around 7.5% improvement over the bilstm-crf classifier for the b-premise tokens. we also observe that the two adaptive-pretrained models (e.g., bert imho and bert essay ) perform better than the bert bl where bert essay achieves the best accuracy of 74.7%, a 2% improvement over bert bl . although bert imho was trained on a much larger corpus than bert essay , we assume since bert essay was trained on a domain relevant corpus it achieves the highest f1. likewise, in the case of multitask models, we observe bert mt performs better than bert bl by 1.3%. this shows that using argumentative sentence identification as an auxiliary task is beneficial for token-level classification. with regards to the adaptive-pretrained models, akin to the bert bl based experiments, we observe bert mtessay perform best by achieving the highest accuracy over 75%. ## conclusion we conduct a token-level classification task to identify the type of the argument component tokens (e.g., claims and premises) by combining the argument segmentation and component identification in one single task. we perused a new corpus collected from essays written by middle school students. w our findings show that a multitask bert performs the best with an absolute gain of 7.5% accuracy over the discrete features. we also conducted an in-depth comparison against the standard segmentation step (i.e., classifying the argumentative vs. nonargumentative units) and proposed a thorough qualitative analysis. middle school student essays often contain run-on sentences or unsupported claims that make the task of identifying argument components much harder. we achieve the best performance using a multitask framework with an adaptive pretrained model, and we plan to continue to augment other tasks (e.g., opinion and stance identification) under a similar multitask framework @xcite . we plan to generate personalized feedback for the students (e.g., which are the supported claims in the essay?) that is useful in automated writing assistance.