ALEN App: Argumentative Writing Support To Foster E nglish Language Learning (2022)

Abstract: This paper introduces a novel tool to support and engage English language learners with feedback on the quality of their argument structures. We present an approach which automatically detects claim-premise structures and provides visual feedback to the learner to prompt them to repair any broken argumentation structures. To investigate, if our persuasive feedback on language learners’ essay writing tasks engages and supports them in learning better English language, we designed the ALEN app (Argumentation for Learning English). We leverage an argumentation mining model trained on texts written by students and embed it in a writing support tool which provides students with feedback in their essay writing process. We evaluated our tool in two field-studies with a total of 28 students from a German high school to investigate the effects of adaptive argumentation feedback on their learning of English. The quantitative results suggest that using the ALEN app leads to a high self-efficacy, ease-of-use, intention to use and perceived usefulness for students in their English language learning process. Moreover, the qualitative answers indicate the potential benefits of combining grammar feedback with discourse level argumentation mining.

## introduction novel advances from natural language processing (nlp) and machine learning (ml) are increasingly utilized and embedded in learner-centered writing support tools (e.g., @xcite ; @xcite ; @xcite ). for example, researchers have successfully embedded novel argumentation mining models to identify persuasive components and their relations in order to provide students adaptive writing feedback @xcite . as @xcite highlighted argumentation learning consists of at least two different dimensions: a) to train argumentation skills (learning to argue) and b) to use argumentation as a dialectical method to achieve other learning outcomes (arguing to learn), such as critical thinking, problem-solving or factual knowledge @xcite @xcite . while the former dimension of argumentation is steadily investigated in the context of nlp-based feedback with argumentation mining on students' learning processes (e.g., @xcite ; @xcite ), the latter described learning context bears still promising potential for nlp-based argumentation feedback opportunities to foster other learning outcomes of students @xcite . in this vein, @xcite has suggested that providing english language learners with feedback on their essays from a discourse perspective can enhance text coherence and comprehension. nevertheless, little work exists which demonstrates the embedding of argumentation mining in writing support tools to investigate the effects of "arguing to learn", e.g., to engage and foster secondary language learning @xcite . in fact, different methods from nlp and ml have been used to provide students feedback on their grammatical errors or syntactical sentence structures to foster language learning (e.g., @xcite ; @xcite ; @xcite ), but insights on the effects and concepts of discourse level feedback based on argumentation modelling on students learning process are few and far between. hence, in this paper, we demonstrate the alen app. the learning application provides english language learners with discourse level feedback in persuasive writing exercises. the underlying model is trained on a corpus of 1000 student-written texts to detect claims and premises as well as their relations @xcite . to investigate, if persuasive feedback on language learners' essay writing tasks engages and supports them in learning better english language, we evaluated our tool in two field-studies with 28 students from a german high school. our objective is to conduct a proof-ofconcept study to explore the impact of prompting english language learners to repair any broken argumentation structure. hence, we asked students to conduct an cambridge english assessment task for the language level b2 and use the alen app to write and evaluate their text's argumentation level and persuasiveness. based on the literature stream of "arguing-to-learn" (e.g., @xcite ), our hypothesis is that adaptive argumentation feedback might engage students to evaluate their text, reflect about their discourse level writing and thus learn better english. the results from our small-scale evaluation provide first suggestions that adaptive argumentation feedback in english language learners essay writing task leads to a high self-efficacy, ease-of-use, intention to use and usefulness for students in their language learning process. future work is needed to investigate the effects of adaptive argumentation support in large-scale field studies to measure the long-term learning success on students language learning outcomes. ## related work for the most part, nlp and ml have been used in education technology for language learners in ways which relate to word-level feedback and text scoring. popular mobile applications such as duolingo tend to focus on vocabulary and phrase learning, a writing assistant such as grammarly gives feedback on spelling and grammar, as does the essay practice website write & improve whilst also providing essay scores pinned to the cefr proficiency scale @xcite @xcite . at the same time, there is now a growing interest in providing automated feedback at the discourse level, and efforts have been made to accumulate and analyse the training materials needed for feedback on argument quality -namely with the gaqcorpus @xcite . thus far only a few practical tools have been developed to provide learners with argumentation feedback. for instance, margot is available as a web application and processes a text that is input in the corresponding editor field @xcite . the text is analyzed, claims are displayed in bold font, whereas premises are displayed in italic style. or in targer a user can analyze the persuasive structure of an input text. @xcite trained multiple models on three different corpora along with three different pre-trained word embeddings. thus, the user not only puts in a text to analyze, but different argumentation models may be selected. the results are then presented below the input, with claims being highlighted in red and premises being marked in green. neither margot nor targer are easy-to-use in normal pedagogical scenarios, since the student has to select from several different models (the nuances of the choices may not be clear) and then copy her text into the input field. this excludes students who are unsure about choosing from different models. moreover, the models are not all trained on text extracted from the educational domain and therefore, might not be applicable to every pedagogical scenarios. besides, argumentation mining was successfully embedded in al (short for argumentation learning), a learner-centered tool which improved students persuasive writing skills with adaptive feedback @xcite . moreover, @xcite presented ar-guetutor, a dialogue-based argumentation learning tool, which tutors students with adaptive scaffolds and theory-explanation through a persuasive writing task. however, to the best of our knowledge literature on the embedding and demonstration of argumentation mining approach to foster language learning through engaging students in persuasive writing exercises are rather rare. ## design of alen to build alen, we followed a three step methodology (see figure 2 ). first, we analyzed the current state of argumentation learning and argumentation mining achievements in literature. therefore, we reviewed multiple papers from the fields of educational technology, such as @xcite al. (2009 second, we investigated different corpora and trained models for argument detection and classification across multiple domains. we started by searching the literature for a corpus that fulfilled the following criteria: 1) the corpus contains annotated persuasive student essays, 2) it has a sufficient corpus size to be able to use the trained model in a real-world scenario, and 3) the annotations are based on a rigorous annotation guideline for guiding the annotators towards a moderate agreement. the business model peer review corpus published in @xcite fulfilled all these requirements. the corpus consists of 1000 business model peer feedback essays written by students extracted from a large-scale lecture scenario. we used the algorithm of @xcite , to train a multi-class classifier on the sentence level to detect the argument components and their relations. for argument component classification, a support vector machine (svm) achieved the best results, with an accuracy of 65.4% on the test set. regarding the persuasive relation classification, a binary classification task, an svm again achieved the best results on the corpus, obtaining an accuracy of 72.1% on the test set. more information on the model and the replicated features we used can be found in @xcite . third, we designed and built an adaptive writing support system that provides students with individual feedback on their argumentation skill level during an english essay writing task based on our model. for the design of the tool, we followed the design principles of @xcite . alen provides the user with a simple text input field with a word count in which they can write or copy a text (see figure 1 ). next to the text input, the user can ask for feedback on the argumentation structure of their text in a personal learning dashboard. the dashboard provides different granularity levels of feedback, which enables the user to control the amount of feedback information displayed @xcite . a visual graph-based representation of a text's argumentation structure and three summarizing scores provide a first assessment of the text's quality. to offer the user with a visual representation of argument structures in their essay, the identified claims are highlighted in green and the premises are highlighted in yellow in the written text. a visual graph-based representation of text-based augmentations has been found to be an effective element to guide learners argumentation (i.e., representational guidance theory @xcite ). a more detailed perspective of the argument's discourse can be obtained by clicking on the highlighted text fields or the nodes in the graph. this displays whether a claim is well-supported or if it is missing a premise. moreover, best practices and explanations about argumentation and argumentation theory are provided by clicking on the "explanation" or "help" button . three summarizing scores, calculated following wambsganss et al. (2020a) -readability, coherence and persuasiveness -provide the student with an assessment of their text to provide automatic proficiency feedback. the methodology for computing the scores, as well as actual tips, action steps, and explanations on how the learner can improve her score level, can be found by clicking on the scores or on details. ## evaluating alen our objective was to empirically investigate the effect of our adaptive argumentation feedback on students' english language learning and their perception of usefulness in a real-world educational writing scenario. therefore, we created a field experiment design in which language students were instructed to complete a persuasive writing exercise while receiving adaptive argumentation feedback from alen. 1 the study was conduct in cooperation with the english department of a german speaking high-school. we conduct two different studies based on a similar field-experimental design in two different english classes in the 12th grade. in both studies, we asked students to conduct a persuasive english language writing tasks. the only difference between study 1 and study 2 were the post-survey measurements (see following paragraph). the experiments were both conducted in the computer room of the high-school on desktop devices. in total, 28 students participated in both studies. the participants were on average 17.17 years old (@xmath0.5384); 11 were male, 11 were female, and 6 non-binary. the experiment design was two-fold (see figure 3 ): 1) a persuasive writing task and 2) a post-survey. for study 2 (18 participants) our goal was to control for the self-efficacy of students for the task of english language learning based on seven items following @xcite to control for self-regulated learning. exemplary items included, "compared to other students in this class, i expect to do well.", or "i am confident that i will be able to solve the problems and tasks set for me in this course.". all constructs were measured on a likert scale from 1 to 7 (1: totally disagree; 7: totally agree, with 4 being a neutral statement). finally, we captured some demographic information and asked three qualitative questions: "what did you particularly like about the use of the argumentation tool?", "what else could be improved?" and "do you have any other ideas?". ## discussion and conclusion we have presented alen, a novel writing support tool that provides students with persuasive feedback during an english language learning task. we embedded the svm model of @xcite to identify claim-premise structures in learners' texts and evaluated the proof-of-concept in two field-studies with 28 students. based on the literature stream of "arguing-to-learn" (e.g., @xcite ), our hypothesis was that adaptive argumentation feedback might engage students to evaluate their text, reflect about their discourse level writing and thus learn better english. our results suggest that the alen app leads to a high self-efficacy in the task of english essay writing and a high technology acceptance (intention to use, perceived usefulness and easeof-use) for k12 language learners. our study extends the current literature stream of nlp-based learning tools for argumentation (e.g. for future work, we suggest to combine discourse level argumentation feedback with grammar feedback for language learners to provide them with more nuanced guidance in their language learning process. moreover, further studies are needed which investigated the human-computer interaction of discourse-level writing support tools for language learners. finally, future research is needed to investigate the effects of adaptive argumentation support in large-scale field studies to measure the long-term learning success on students language learning outcomes.