Teach Me How to Argue: A Survey on NLP Feedback Systems in Argumentation (2023)

Abstract: The use of argumentation in education has shown improvement in students’ critical thinking skills, and computational models for argumentation have been developed to further assist this process. Although these models are useful for evaluating the quality of an argument, they often cannot explain why a particular argument score was predicted, i.e., why the argument is good or bad, which makes it difficult to provide constructive feedback to users, e.g., students, so that they can strengthen their critical thinking skills. In this survey, we explore current NLP feedback systems by categorizing each into four important dimensions of feedback (Richness, Visualization, Interactivity and Personalization). We discuss limitations for each dimension and provide suggestions to enhance the power of feedback and explanations to ultimately improve user critical thinking skills.

## introduction argumentation is the field of elaborating and presenting arguments to engage in debate, convince others, and eventually reach agreements. in this context, an argument is made of a conclusion (i.e., a claim) supported by reasons (i.e., premises) @xcite . computational argumentation emerged as a way to support argumentation. it is a subfield of natural language processing (nlp) that deals with the automated representation, evaluation, and generation of arguments. it includes tasks such as mining arguments @xcite , assessing arguments' quality @xcite , reconstructing implicit assumptions in arguments @xcite or even providing constructive feedback for improving arguments @xcite , to name a few. in education, learning how to argue (e.g., writing argumentative essays, debates, etc.) has been shown to improve students' critical thinking skills @xcite ; behar-horenstein @xmath0. yuan et al. bonial et al. jin et al. naito et al. kikteval et al. singh et al. becker et al. lauscher et al. chernodub et al. fang et al. zhang et al. rach et al. wambsganss et al. zhang et al lugini et al petasis et al wambsganss et al aicher et al mirzababai et al wambsganss et al wachsmuth and alshomary hunter et al. putra et al. and @xcite . to further improve critical thinking skills, several researchers have been working on computational argumentation and specifically argumentative feedback systems to provide support and to assist learners in improving the quality of their arguments @xcite @xcite . although argumentative feedback systems are proven to assist students' learning and reduce teachers' workload @xcite , such systems still lack the ability to deeply explain how an argument can be improved; i.e., not only providing a holistic label or score, but explaining particularly why this result was given by automatic evaluation rubrics. such explanations as feedback can ultimately explain and visualize the results comprehensively for the users so that users can understand and improve their argumentation skills. the lack of ability in current systems to provide deep explanations as feedback motivated our interest in investigating the current state of argumentative feedback generation. in this survey, we focus on different kinds of feedback given to learn how to argue. inspired by the sections tutorial feedback and architecture and technology mentioned in @xcite , we combine features of feedback systems, formulate four distinct dimensions and categorize existing papers into these dimensions (figure 1 ): figure 2 shows four different dimensions of feedback (richness, visualization, interactivity, and personalization), for a given argument consisting of two claims and one premise. in this example, in the richness dimension, a faulty generalization in the argument is identified (cf. what) and explained (cf. why). visualization would add symbols and highlight important feedback elements to make it more understandable. interactivity would allow the user to ask for more explanations to the model. personalization would consider that the user is a child and provide appropriate feedback on that basis. towards better argumentative feedback, this survey aims to give an overview of argumentative feed-back systems. we explore work that provide feedback answering one or multiple questions among the types: what/why ( §4), how ( §5), who ( §6), and to whom ( §7). finally, we discuss remaining challenges and potential ways to overcome them ( §8) in order to develop systems that provide feedback or detailed explanations in a way so that learners can improve their critical thinking skills. we believe this survey can aid researchers in understanding current explanations in argumentation and broaden their horizon on argumentative feedback. foot_0 2 related work several surveys have been done in the field of argumentation @xcite @xcite and explainability @xcite @xcite . as we would like to focus on how well a model can explain its results as a type of feedback for learners, we present here recent surveys related to feedback or explainability in argumentation. beigman klebanov and madnani (2020) present the progress in automated writing evaluation, using @xcite to frame the presentation. in this survey, the succinct feedback section enumerates different systems for writing assistant and highlights the inconclusiveness of research on effectiveness of automated writing evaluation. @xcite highlights the potential of argumentation in explainable ai systems. they provide an exhaustive overview of argumentation systems by grouping them based on domain, such as law. for each domain, papers are compared by tasks (e.g., argument classification). despite the extensiveness of their survey, some topics to improve explanations in argumentative systems received little attention. for example, frameworks that include arguments with commonsense knowledge and diverse attack relations between them have rarely been discussed, even though they can enhance the model's explainability @xcite . @xcite focus on the different frameworks, types, and forms of explanations. they distinguish intrinsic approaches (i.e., models using argumentative methods) from post-hoc approaches (i.e., non-argumentative models that provide complete or partial explanations). they discuss multiple forms of argumentation, such as dialogue. their final roadmap covers the need to focus more on properties and computational aspects of argumentation-based explanations. whereas they focus on how argumentation can be used to enhance the explainability of models, our work discusses what kind of feedback (i.e., explanations) on argumentation models can provide. moreover, our work distinguishes itself from the surveys previously mentioned by giving an overview of automatized feedback on argumentation from the angle of rich ( §4), visual ( §5), interactive ( §6), and personalized ( §7) explanations inspired by @xcite . ## pedagogy before discussing the four dimensions mentioned a priori, it is essential to know the pedagogy used to teach argumentation and adopted by computational models. this section presents some standard pedagogical methods used in teaching how to argue. toulmin model the toulmin model @xcite , often seen as the foundation of teaching argumentation, is a popular framework for constructing, analyzing and evaluating arguments, and can contribute to the improvement of students' argumentative writing @xcite as well as critical thinking skills @xcite . this approach deconstructs an argument into six elements (appendix, figure 4 ), and students are taught to identify each element within an argument. by identifying elements from the toulmin model, models can provide users with rich feedback. rhetorical structure theory based on @xcite , the rhetorical structure theory was originally developed in the context of computer-based text generation in order to attribute a formal structure to a text @xcite . this theory employs graphical representations, such as mind maps or graphs, to illustrate the relationships between different components of the text's architecture. this visual approach can help students visualize the connections between different concepts and enhance their understanding of complex topics @xcite . the advent of tools like tiara @xcite has given rise to the deployment of the rhetorical structure theory, i.e. the generation of visual feedback. collaborative argumentation in collaborative argumentation-based learning, also described as cable by @xcite , individuals work together to construct, refine, and evaluate arguments on a particular topic or issue. the main goal of collaborative argumentation is to foster constructive dialogue, critical thinking, and the exploration of different perspectives. @xcite differentiate four dimensions of cable: ## richness -what is an error and why? to improve students' critical thinking skills, we first need to evaluate their argumentative texts, i.e., identify argumentative errors. in this section, we focus on models providing shallow explanations, i.e., models that identify what should be corrected in the arguments. we discuss relevant works that identify properties such as the structure of arguments which is helpful in this process. components identifying argumentative components is one of the fundamental tasks in argumentation @xcite @xcite . such works primarily focus on identifying components such as claims and premises. more recently, the usefulness of identifying such components can be seen in tasks such as counterargument generation. for example, in @xcite , weak premises are identified and ranked to generate counter-arguments. relations after identifying the different components of an argumentative text, it is necessary to distinguish the multiple relations between them, ultimately to assert the arguments' quality. indeed, supporting or refuting a claim is made of complex logical moves, such as promoting, contradicting, or acknowledging a fact. to identify the different relations patterns, @xcite focus on finding interactive argument pairs, whereas mim et al. (2022) enables annotating complex attack relations. schemes in addition to components and relations, @xcite proposed a set of roughly 80 logical argumentation schemes to categorize the underlying logic. each scheme has a set of critical questions which provide a template to assess the strength of the argument depending upon the associated scheme. since the first work on automatically detecting argumentation schemes in argumentative texts @xcite , the use of such schemes has been explored in tasks such as essay scoring @xcite . fallacies although a good structure with a claim and premises is necessary for a good argument, it is not sufficient. an argument has more complex properties, such as its logical, dialectical, and rhetorical aspects. a fallacy is a logical error or deceptive argument that undermines the validity of a conclusion or reasoning, which poses a substantial issue due to its propensity to generate miscommunication. towards teaching students to avoid making errors in logical reasoning, logical fallacies have received attention @xcite @xcite . motivated by the gamification method made by @xcite , @xcite aimed to capture similar fallacy types for news articles, but the low distribution of fallacy types in the wild makes identification challenging. however, most natural texts do not have recurrent specific patterns, compared to current datasets, like the logic and logicclimate datasets @xcite . moreover, given the large number of logical fallacies that exist (over 100 types), long arguments can be grouped into multiple fallacies, resulting in difficulties in classification @xcite . ## visualization -how to show the error? the effectiveness of any argument does not solely rely on its content but also on its presentation. this is where visualization of argumentative feedback emerges as a crucial factor. visualizing feedback empowers individuals to perceive the intricacies of an argument in a more comprehensive manner. by using visual aids like graphs, feedback becomes more accessible and engaging, fostering constructive discussions. in this section, we discuss how visualization impacts argumentative feedback. highlights a simple approach to visualization is highlighting, i.e., application of visual emphasis on a specific pattern with the intention of drawing the viewer's attention to this specific pattern. for example, lauscher et al. (2018) use a text editor which highlights components, a graph view which shows the argumentative structure, and a score view showing the user's performance. based on the classroom-setting evaluation, students using such systems wrote texts with a better formal quality of argumentation compared to the ones using the traditional approach. nevertheless, the current accuracy of such systems' feedback still leaves a large improvement space in order for users to be motivated to use them. more recent work such as @xcite incor-porate feedback generated by state-of-the-art llms in their graphical systems. nonetheless, factual inaccuracies, as well as inconsistent or contradictory statements, are still generated, exposing the user to confusion and leaving room for improvement. ## personalization -to whom is it for? even if the feedback mentioned in §4 are a step towards good guidance, they are static, which can be problematic. beginners and professionals in argumentation do not need the same amount of feedback. a child and an adult have different levels of understanding and knowledge. therefore, it is essential that a model knows to whom it should explain the errors and hence how to adapt its output by providing personalized explanations. levels of explanations a first approach to personalization is to discretize different users' proficiency levels in argumentation into a small number of categories. for instance, with the system described in @xcite and @xcite , users can select their own level among the following categories: novice, advanced, competent, proficient, expert. although @xcite and @xcite propose different granularity levels of explanations, their study is restrained to students from their university. having end-users from different backgrounds may imply the need for new levels of explanations. @xcite show that the explainee's age affects the way an explainer explains the topic at hand. thus, we consider that information such as the learner's age should be considered in future interactive argumentative feedback systems, where terminology such as fallacy and their existence would require different explanation approaches for younger students (i.e., elementary) compared to older students. self-personalization for more personalized feedback, systems such as @xcite and putra et al. (2020) rely on user's inputs. they allow users to make their custom tags or to choose their preferences among a set of rubrics. nevertheless, manually personalizing the system can be overwhelming and time-consuming for users. next directions @xcite argue that the next direction for personalized argumentative feedback would be to develop argumentation chatbots for persuasion and infer the user's stance based on the discussion. chatbots' personalization capabilities enable them to tailor their responses to individual learners' needs and learning styles, potentially enhancing the effectiveness of the tutoring process @xcite . however, bridging the gap among personalized chatbots @xcite , personalized educational methods @xcite @xcite and argumentation has remained unexplored. thus, we think researchers should focus in the future on providing more personalized explanations (i.e., precisely adjusted by considering the learner's background) to improve the users' critical thinking skills efficiently. ## discussions teaching how to argue through nlp systems holds significant promise for enhancing students' learning process. however, existing research in this area presents various open issues. in this section, we explore some difficulties in designing and evaluating computational models for argumentation and discuss some methods for mitigating them. evaluating different systems the evaluation of nlp systems often relies on human assessment, which is insightful. however, this reliance makes it hard to reproduce the evaluation and to compare different systems. to the best of our knowledge, no research has focused on comprehensive comparative studies of different end-to-end systems. the lack of direct comparisons between similar systems hampers the understanding of their relative advantages and limitations. as researchers and educators, it becomes overwhelming to discern which system best fits specific pedagogical objectives. a possible reason for this issue resides in the restricted access to various tools. indeed, many systems may not be accessible, limiting researchers to test them. additionally, the lack of guidelines to evaluate systems for learning argumentation exacerbates the difficulty in evaluating these systems in a systematic manner. current systems' performance is evaluated with metrics such as coherence. nevertheless, new evaluation methods such as the ones described in @xcite should be explored. therefore, we should promote open-source projects and the research of standard guidelines. ## conclusion in our survey, we explored several works providing feedback in argumentation, following various dimensions: richness ( §4), visualization ( §5), interactivity ( §6), and personalization ( §7). figure 3 summarizes the pedagogy, current achievements and potential future directions of each dimension. as potential areas for improvement to enhance the quality of educational argumentative systems, we highlighted the following points: (1) generate accurate, constructive feedback for a real-life input( §4-5), (2) tailor the output based on the user's background ( §6-7), (3) evaluate and compare endto-end systems more deeply( §8), (4) improve models' abilities to adapt to unknown topics( §8), (5) collaborate with pedagogical teams and actual students( §8), and finally (6) take into consideration ethical issues( §8). for instance, in challenge (2), the use of conversational agents becomes increasingly frequent. however, such systems still leave room for improvement, particularly their ability to tailor discussions based on the user's background. we hope our survey contributes to enriching the research community focused on argumentation with a comprehensive understanding of current perspectives in nlp systems for teaching how to argue. in our future work, we will focus further on reallife and end-to-end systems (challenges (1) and ( 3 )). we plan to prototype a system to measure the effects of different feedback on users and evaluate it in actual classrooms (appendix, figure 5 ).