Modeling Persuasive Discourse to Adaptively Support Students’ Argumentative Writing (2022)

Abstract: We introduce an argumentation annotation approach to model the structure of argumentative discourse in student-written business model pitches. Additionally, the annotation scheme captures a series of persuasiveness scores such as the specificity, strength, evidence, and relevance of the pitch and the individual components. Based on this scheme, we annotated a corpus of 200 business model pitches in German. Moreover, we trained predictive models to detect argumentative discourse structures and embedded them in an adaptive writing support system for students that provides them with individual argumentation feedback independent of an instructor, time, and location. We evaluated our tool in a real-world writing exercise and found promising results for the measured self-efficacy and perceived ease-of-use. Finally, we present our freely available corpus of persuasive business model pitches with 3,207 annotated sentences in German language and our annotation guidelines.

## introduction argumentation is an omnipresent rudiment of daily communication and thinking @xcite . the ability to form convincing arguments is not only fundamental to persuading an audience of novel ideas but also plays a major role in strategic decision-making, negotiation, and constructive civil discourse @xcite . however, humans often struggle to develop argumentation skills owing to a lack of individual and instant feedback in their learning process @xcite , since providing feedback on the individual argumentation skills of learners is time-consuming and not scalable if conducted manually by educators @xcite . furthermore, novel distance learning scenarios such as massive open online courses (moocs) @xcite come with addi-figure 1 : argumentation annotation scheme. first, a text sentence is classified into an argumentative component (claim, premise, major claim, noneargumentative). second, the same annotator captures the basic discourse structure between the components. third, the components and the pitch are scored for the persuasiveness scores (specificity, evidence, strength, relevance) based on our annotation guideline on a 1to-5 scale. one possible solution to this dilemma are adaptive argumentation support systems that enable individuals to train their argumentation skills, e.g., in collaborative learning settings @xcite or by providing tailored argumentation feedback independent of an instructor, time and place @xcite . such tools are increasingly utilizing recent developments in computational linguistics in the form of computerassisted writing @xcite to provide tailored feedback about textual documents @xcite . in this context, argumentation mining (am) research is a crucial field for the development of support systems that identify arguments in unstructured texts @xcite . however, corpora that are applicable for the design and development of adaptive argumentative writing systems in pedagogical scenarios are rather scarce. to the best of our knowledge, there are only two collections from the educational domain which are based on student-written texts and annotated for argumentative discourse structures @xcite . we propose a novel argumentation annotation scheme for persuasive student-written business model pitches. therefore, we introduce a corpus of 200 student-written persuasive pitches with 3,207 sentences that are annotated for argument components, their relations, and persuasiveness scores to judge the argumentation quality of the single arguments. we trained different models and embedded them as feedback algorithms in a novel writing support tool that provides students with individual argumentation feedback and recommendations in a persuasive writing exercise. the design of our tool is based on the self-evaluation mechanism for students to improve self-efficacy and argumentation learning outcomes during a learning process (i.e., self-regulated learning theory @xcite ; zimmerman and schunk ( 2001 )). we asked students to conduct a persuasive writing exercise and provided them with argumentation self-evaluation. the measured argumentation @xcite , the perceived self-efficacy @xcite , and the perceived usefulness @xcite in an evaluation provided promising results for using our approach in different large-scale learning scenarios to offer quality education with individual feedback independent of an instructor, time, and location. hence, we contribute to research by (1) deriving an annotation scheme for a new data domain for am based on argumentation theory and previous work on annotation schemes for student-written texts @xcite @xcite , ( 2 ) presenting an annotation study based on 50 persuasive business model pitches and two annotators to show that the annotation of student-written pitches is reliably possible, (3) offering our final and freely available corpus of 200 student business pitches consisting of 3,207 annotated sentences collected from a lecture about digital business models in german, and (4) embedding and evaluating our annotation approach as predictive models in a writing support system in a real-world writing exercise. we, therefore, hope to encourage future research on argumentation discourse and persuasiveness levels in student-written texts and on writing support systems for argumentation. ## corpus construction our corpus consists of 200 student-written business model pitches in which students present an entrepreneurial idea of a digital business model. business model pitches -also called entrepreneurial or business pitches @xcite -are described as "a brief description of the value proposition of an idea or company" @xcite with the objective to convince a group of stakeholders of the novelty of an idea. the formulation of persuasive business model pitches is increasingly used in modern pedagogical scenarios, e.g., to train the entrepreneurship mindset or agile work (i.e., oecd (2019)). students are asked to write a concise but persuasive summary of the "what, why, and how" of their (business) idea in order to convince a peer. this pedagogical scenario is domain-independent, easy to implement in different settings (e.g., in moocs), and can be utilized to train skills such as logical argumentation. in fact, in their study about entrepreneurial business pitches, fernández-vázquez and álvarez-delgado (2019) found out that "the lack of rational arguments determines the failure of the entrepreneur's efforts to be persuasive, regardless of the emotional appeals that are introduced into the pitch". therefore, fernández-vázquez and álvarez-delgado (2019) calls for more emphasis on logical argumentation chains in business pitches. however, linguistic research on business model pitches is a growing but still small field (ducasse, 2020). therefore, it is not surprising that no pitch corpus exists that is annotated for argumentation discourse structures based on an appropriate argumentation scheme @xcite . we propose a new annotation scheme to model argument components, their relations as well as argumentation quality labels that reflect the argumentative discourse structures in persuasive business model pitches. we based our annotation scheme on the model of @xcite and the studies of @xcite ; @xcite ; @xcite ; @xcite . following a 4-step methodology to build a robust corpus, we (1) searched literature and scientific theory on argumentation discourse structures and argumentation models in different text domains; ## providing students adaptive feedback modelling argumentation structures after constructing and analyzing our corpus, we leveraged the novel data to train a machine learning model. our objective was to embed a classification algorithm in the back end of an argumentative writing support system to provide students with individual argumentation feedback in the writing process. the task is considered a sentence-based classification task, where each sentence can be either a major claim, a claim, a premise, or non-argumentative. therefore, we trained and tuned a long short-term memory (lstm) model @xcite to classify the argumentative components of a given text. we tokenized the texts and transformed them into word embeddings. the data set was split into training and test sets using an 80:20 split. for the component classification we received an accuracy of 54.12%, a precision of 55.90% and a recall of 54.12% on the test data. we benchmark our approach against a bert model @xcite . however, we received a rather unsatisfying accuracy of 47.50%, a precision of 46.66% and a recall of 47.50%. more information about the modeling can be found in section b of the appendix. argumentation writing support system we designed and built an adaptive writing support sys4 for detailed statistics see section a of the appendix. evaluation in a writing exercise we embedded the tool into a persuasive writing exercise where students were asked to write an argumentative pitch about a business idea. during this writing task, they received adaptive feedback on their argumentation level based on our model. the evaluation was conducted as a part of an exercise with students from a western-european university, and thus designed and reviewed according to the ethical guidelines of the university. to keep data privacy standards, the students' data were additionally anonymized. we conducted a field experiment to see if and how individual argumentation self-evaluation with adaptive feedback can assist students in writing more persuasive writings. we created a pedagogical scenario in which participants had to write a 300-word persuasive business pitch. the declared goal was to write a convincing pitch to persuade potential investors. students were not required to participate in the assignment in order to pass the class; nevertheless, by successfully completing the assignment, they may increase their final mark by 2.2 percent. the persuasiveness of the business presentation had no influence on the assignment's grading and, as a result, no impact on final marks. after the treatment, we measured the perceived ease-of-use according to @xcite by asking the following three items: "it would be easy for me to become adept at using the reasoning tool", or "i find the reasoning tool easy to interact with", and "learning how to use the reasoning tool would be easy for me". moreover, we measured the self-efficacy of students for the task of argumentation skill learning based on three items following @xcite to control for self-regulated learning. the items included, "in comparison to other users, i will write a good argumentative text", "i am sure that i could write a very good argumentative text", and "i think i now know quite a bit about argumentative writing." both constructs were measured with a 1-to-7 point likert scale (1: totally disagree to 7: totally agree, with 4 being a neutral statement). furthermore, we asked three qualitative questions: "what did you particularly like about the use of the tool?", "what else could be improved?", and "do you have any other ideas?" and captured the demographics. ## conclusion we propose an argumentation annotation scheme and introduce an annotated corpus of persuasive student-written business model pitches extracted from a pedagogical scenario. we offer a corpus of 200 student-written business model pitches with 3,207 sentences annotated for argument components, their relations, and six persuasiveness scores on different levels. by presenting an annotation study based on 50 persuasive pitches, we demonstrate that the annotation of student-written business model pitches is reliably possible. finally, we embed and evaluated a trained model based on our corpus in an argumentation writing support tool for students. we thus aim to encourage fellow researchers to leverage our annotation scheme and corpus to design and develop argumentation support systems for students in large-scale scenarios.