An Exploratory Study of Argumentative Writing by Young Students: A transformer-based Approach (2020)

Abstract: We present a computational exploration of argument critique writing by young students. Middle school students were asked to criticize an argument presented in the prompt, focusing on identifying and explaining the reasoning flaws. This task resembles an established college-level argument critique task. Lexical and discourse features that utilize detailed domain knowledge to identify critiques exist for the college task but do not perform well on the young students’ data. Instead, transformer-based architecture (e.g., BERT) fine-tuned on a large corpus of critique essays from the college task performs much better (over 20% improvement in F1 score). Analysis of the performance of various configurations of the system suggests that while children’s writing does not exhibit the standard discourse structure of an argumentative essay, it does share basic local sequential structures with the more mature writers.

## introduction argument and logic are essential in academic writing as they enhance the critical thinking capacities of students. argumentation requires systematic reasoning and the skill of using relevant examples to craft a support for one's point of view @xcite . in recent times, the surge in ai-informed scoring systems has made it possible to assess writing skills using automated systems. recent research suggests the possibility of argumentationaware automated essay scoring systems @xcite . most of the current work on computational analysis of argumentative writing in educational context focuses on automatically identifying the argument structures (e.g., argument components and their relations) in the essays @xcite @xcite and by predicting essay scores from features derived from the structures (e.g., the number of claims and premises and the number of supported claims) @xcite . related research has also addressed the problem of scoring a particular dimension of essay quality, such as relevance to the prompt @xcite , opinions and their targets @xcite , argument strength @xcite , among others. while argument mining literature has addressed the educational context, it has so far mainly focused on analyzing college-level writing. for instance, @xcite investigated argument structures in toefl11 corpus @xcite and @xcite analyzed writing of university students; @xcite used data from "essayforum.com", where college entrance examination is the largest forum. computational analysis of arguments in young students' writing has not yet been done, to the best of our knowledge. writing quality in essays by young writers has been addressed @xcite @xcite , but identification of arguments was not part of these studies. in this paper, we present a novel learning-andassessment context where middle school students were asked to criticize an argument presented in the prompt, focusing on identifying and explaining the reasoning flaws. using a relatively small pilot data collected for this task, our aim here is to automatically identify good argument critiques in the young students' writing, with the twin goals of (a) exploring the characteristics of young students' writing for this task, and (b) in view of potential scoring and feedback applications. we start with describing and exemplifying the data, as well as the argument critique annotation we performed on it (section 2). experiments and results are pre-dear editor, advertising aimed at children under 12 should be allowed for several reasons. first, one family in my neighbourhood sits down and watches tv together almost every evening. the whole family learns a lot, which shows that advertising for children is always a good thing because it brings families together. second, research shows that children can't remember commercials well anyway, so they can't be doing kids any harm. finally, the arguments against advertising aren't very effective. some countries banned ads because kids thought the ads were funny. but that's not a good reason. think about it: the advertising industry spends billions of dollars a year on ads for children. they wouldn't spend all the money if the ads weren't doing some good. let's not hurt children by stopping a good thing. if anyone doesn't like children's ads, the advertisers should just try to make them more interesting. the ads are allowed to be shown on tv, so they shouldn't be banned. ## dataset and annotation the data used in this study was collected as part of a pilot of a scenario-based assessment of argumentation skills with about 900 middle school students @xcite . 1 students engaged in a sequence of steps in which they researched and reflected on whether advertising to children under the age of twelve should be banned. the test consists of four tasks; we use the responses to task 3 in which students are asked to review a letter to the editor and evaluate problems in the letter's reasoning or use of evidence (see students were expected to produce a written critique of the arguments, demonstrating their ability to identify and explain problems in the reasoning or use of evidence. for example, the first excerpt below shows a well-articulated critique of the hasty generalization problem in the prompt: however, many students had difficulty explaining the reasoning flaws clearly. in the second excerpt, the student thought that an argument from the family in the neighborhood is not strong, but did not demonstrate an understanding of a weak generalization in his explanation. other common problems included students summarizing the prompt without criticizing, or providing a generic critique that does not adhere to the particulars of the prompt (excerpt (3)). the goal of the argument critique annotation (described next) was to identify where in a response good critiques are made, such as the one in the first excerpt. ## discussion the difference in f1 between bert bl , bert bl+lm , and bert pairs+lm is almost exclusively in recall -they have comparable precision at about 0.6, with recall of 0.52, 0.64, and 0.74, respectively. partitioning out 10% of the training data for a development set, we found that bert bl+lm detected 13 more arg sentences than bert bl in the development data. these fell into two sequential patterns: (a) the sentence is followed by another that further develops the critique (7 cases) -see excerpts ( 4 ) and ( 5 ) below; (b) the sentence is the final sentence in the response (6 cases); excerpt (6). our interpretation of this finding is that bert bl+lm captured organizational elements in children's writing that are similar to adult patterns. beigman @xcite found that adult writers often reiterate a previously stated critique in an extended conclusion and spread critiques across consecutive support sentences. thus, even though alignment of critiques with "standard" discourse elements such as conclu-sion and support is not recognizable in children's writing (as witnessed by the failure of the structural features to detect critiques), some basic local sequential patterns do exist, and they are sufficiently similar to the ones in adult writing that a system with its language model tuned on adult critique writing can capitalize on this knowledge. interestingly, bert pairs learned similar sequential patterns -indeed 7 of the 13 sentences gained by bert bl+lm over bert bl are also recalled by bert pairs . this further reinforces the conclusion that young writers exhibit certain local sequential patterns of discourse organization that they share with mature argument critique writers. ## conclusion and future work we present a computational exploration of argument critiques written by middle school children. a feature set designed for college-level critique writing has poor recall of critiques when trained on children's data; a pre-trained bert model fine-tuned on children's data does better by 18%. when bert's language model is additionally fine-tuned on a large corpus of college critique essays, recall improves by further 20%, suggesting the existence of some similarity between young and mature writers. performance analysis suggests that bert capitalized on certain sequential patterns in critique writing; a larger study examining patterns of argumentation in children's data is needed to confirm the hypothesis. in future, we plan to fine-tune our models on auxiliary dataset, such as the convincing argument dataset from @xcite .