Graph Embeddings for Argumentation Quality Assessment (2022)

Abstract: Argumentation is used by people both internally, by evaluating arguments and counterarguments to make sense of a situation and take a decision, and externally, e.g., in a debate, by exchanging arguments to reach an agreement or to promote an individual position. In this context, the assessment of the quality of the arguments is of extreme importance, as it strongly influences the evaluation of the overall argumentation, impacting on the decision making process. The automatic assessment of the quality of natural language arguments is recently attracting interest in the Argument Mining field. However, the issue of automatically assessing the quality of an argumentation largely remains a challenging unsolved task. Our contribution is twofold: first, we present a novel resource of 402 student persuasive essays, where three main quality dimensions (i.e., cogency, rhetoric, and reasonableness) have been annotated, leading to 1908 arguments tagged with quality facets; second, we address this novel task of argumentation quality assessment proposing a novel neural architecture based on graph embeddings, that combines both the textual features of the natural language arguments and the overall argument graph, i.e., considering also the support and attack relations holding among the arguments. Results on the persuasive essays dataset outperform state-of-the-art and standard baselines’ performance.

## introduction argumentation is the process by which arguments are constructed, compared, evaluated in several respects and judged in order to establish whether any of them is warranted. argumentation is an effective approach for solving various theoretical and practical problems @xcite , like explaining and justifying the decision making outcomes and reasoning under inconsistent and incomplete information. roughly, each argument is a set of premises or assumptions that, together with a claim, is obtained by a reasoning process. the overall goal of argumentation is to increase or decrease the acceptability of claims by supporting or attacking them with new arguments. a major component of the argumentation process concerns the assessment of a set of arguments and of their conclusions to establish their justification status, and therefore compute their acceptability degree @xcite . both qualitative and quantitative approaches have been proposed in the literature to assess the acceptance of an argument. however, the assessment of the arguments acceptability is only a (basic) part of the complex assessment tasks required in argumentative processes in many everyday life applications and contexts, e.g., in medicine and education. the issue of assessing an argumentation is particularly critical when considering the different aspects of artificial argumentation, from the identification of real natural language arguments and their relations in text, to the computation of the justification status of abstract arguments @xcite , to the gradual assessment of arguments @xcite based, e.g., on the trustworthiness of the argument proponents @xcite or on the value promoted by the argument @xcite . in particular, despite some approaches addressing the automatic assessment of natural language arguments @xcite @xcite , this issue remains largely unexplored and unsolved. in this paper, we address this open issue and we answer the following research questions: (i) what are the basic quality dimensions to characterise natural language argumentation? and (ii) how to automatically assess these quality dimensions on natural language argumentative text? more specifically, we propose an argument mining @xcite @xcite approach to iden-tify and classify natural language arguments along with quality dimensions. we first define and annotate three prominent quality dimensions for natural language argumentation, i.e., cogency, rhetoric and reasonableness, on an existing dataset of student persuasive essays @xcite . more specifically, cogency estimates the acceptability of the premises that are relevant to the argument's conclusion and their sufficiency to draw the conclusion, rhetoric determines the rhetorical strategy employed in the argument's conclusion (if any), and reasonableness rates if the argument adequately rebuts its counterarguments. we then train a transfomer-based neural classifier with an attention mechanism called longformer @xcite empowered with graph embeddings to address the task. our core contribution is twofold: the work we present in this paper is motivated by the lack of existing resources of natural language argumentation annotated with quality dimensions, and the need for effective methods to address this task. our contribution advances the state of the art with a novel resource and an effective method. ## related work recent approaches in argument(ation) mining (am) @xcite @xcite tackle specific argument qualities features, such as argument relevancy @xcite , convincing arguments @xcite and overall argument quality @xcite . previous work on student essays aimed to assess clarity @xcite , organization @xcite and argument strength @xcite . @xcite target the automatic prediction of the quality of student reflective responses, showing how expert-coded quality ratings and quality predictions based on their features positively correlate with student learning gain. defining the characteristics of a good and successful argument is a hard task. different approaches have been proposed to assess logical, rhetorical, and dialectical quality dimensions of natural language arguments. @xcite derive a taxonomy of argumentation quality that systematically decomposes quality assessment based on the interactions of 15 widely accepted quality dimensions. the three main characteristics are cogency, effectiveness and reasonableness. as a follow up, @xcite investigate how effectively each dimension can be automatically assessed, modelling features such as content, style, length and subjectivity. this textonly assessment yields moderate learning success for most of the evaluated dimensions. in another text-only approach, @xcite describe a large argument quality corpus with data extracted from forums. they propose the first computational model to automatically evaluate cogency, reasonableness, effectiveness and overall quality. @xcite present an argument quality assessment method defined as a graph classification task. the authors reconstruct the graph structure of the arguments within the argument quality dataset of @xcite , showing that this is feasible only in some cases. the reconstructed structures are composed of claims and evidence connected by a support relation, disregarding important elements like counterarguments and rebuttals. results indicate that discourse-based argument structures reflect qualitative properties of the arguments. for rhetorical aspects, @xcite show the impact of the different rhetorical strategies used in political discourse. for automatic essay scoring, @xcite show how human-labelled evidence scores can be replaced with other automated essay quality signals, such as word count and topic distribution similarity. in this paper, we advance the state of the art of natural language argument quality assessment by investigating three main quality properties of persuasive essays grounding on social science argument quality assessment scores @xcite . moreover, we propose a novel method to evaluate the reasonableness of an argument by combining cogency properties with the argumentation graph structure. ## quality dimensions of persuasive essays to annotate the quality dimensions on persuasive essays, we rely on the corpus built by @xcite , containing 402 persuasive essays annotated with the argument components (i.e., evidence, claims and major claims) and relations (i.e., support or attack). this results in 402 argument graphs where the argument components are the nodes of the graph, and the argumentative relations are the edges of the graph. we add a new annotation layer by manually labelling for each argument in the essays the following three quality attributes: cogency, reasonableness and argumentation rhetoric, following the taxonomy proposed by @xcite . taking advantage of the relation annotations, we use the argument graph (i.e., argument components and their relations) to assist annotators in their annotation process. annotation guidelines. given that our goal is to assess persuasive essays written by students, we rely on the quality evaluation process proposed in social sciences showing how these essays are assessed by professors. @xcite propose a scoring rubric for persuasive writing that integrates the assessment of both argumentative structural elements and reasoning quality by manually analyzing argumentative essays made by 125 students in hong kong. this rubric contemplates several characteristics of the standard definition of cogency and reasonableness, such as relevancy, acceptability, and soundness as well as the presence of counterarguments and rebuttals. tables 1 , 2 and 3 show the analytic scoring rubrics proposed by @xcite . a scale of 0, 10, 15, 20, 25 is given to assess the cogency and reasonableness of a given argument. cogency. an argument should be seen as cogent if it has individually acceptable premises that are relevant to the argument's conclusion and that are sufficient to draw the conclusion @xcite . annotators were provided with annotators were asked to annotate both the reasonableness counterargument and the reasonableness rebuttal for each claim in the essays which is attacked by a counterargument. whilst the definitions of reasonableness and cogency are similar, the key difference is that with cogency we evaluate the premises of the argument and with reasonableness the whole argumentation graph involving the argument to be assessed (including its counterarguments and their rebuttals). assessing the reasonableness of an argument implicates analysing its counterarguments and the related rebuttals. in the example of figure 1 , we assess the reasonableness quality dimension following tables 2 and 3 : for counterargument claim e, we can see that no reasons, or premises, provided to support it. this falls under the criteria for score 0 for reasonableness counterargument. for the rebuttal, we can see that claim f correctly points out the weakness of the counterargument, providing an acceptable and sound premise and with a reasoning quality stronger than of the counterargument, therefore falling under the criteria for score 25. score: 25 score: 20 score: 15 a. provides multiple reasons for the counterargument claim(s) /alternative view(s), and b. all counterarguments/reasons for the alternative view(s) are sound/acceptable and free of irrelevancies a. provides multiple reasons for the counterargument claim(s)/alternative view(s), and b. most counterarguments/reasons for the alternative view(s) are sound/acceptable and free of irrelevancies, but one or two are weak a. provides one to two reasons for the counterargument claim(s) /alternative view(s), and b. some counterarguments/reasons for the alternative view(s) are sound/acceptable, but some are weak or irrelevant score: 10 score: 0 a. provides only one reason for the counterargument claim(s)/alternative view(s), or b. the counterargument/reason for the alternative view is weak or irrelevant a. no reasons are provided for the counterargument claim(s)/alternative view(s); or b. none of the reasons are relevant to/support the counterargument claim(s)/alternative view(s) argumentation rhetoric. annotators were asked to evaluate at the argument level which rhetoric strategy the argument is following among ethos, logos, and pathos @xcite . logos is the act of appealing to the audience through reasoning or logic, by citing facts and statistics, historical and literal analogies. ethos is the act of appealing to the audience through the credibility of the author's beliefs or authority. pathos means to persuade an audience by appealing to their emotions. some examples of persuasive essays annotated with argumentation rhetoric are available in the appendix. in example 2 the claim (in bold) appeals to emotions pathos when the author describes how "people are better taken care' in the premises 1 and 3 (in italic). in example 3 the authors employ ethos, we can notice that the author refers to personal experiences in premises 1 and 2. example 4 employs logos, the author refers to a formal study, in premise 2, in order to support its claim. example 2 the advanced medical care brings with it more benefits than disadvantages. [the main advantage of high tech medical care is that people are better taken care so that they have a good health]1. [healthy workers can create more productivity]1 [they can contribute effectively to the development of the economy]2. [they do not have to spend more time in health checking or treatment]. 3 [this saves an amount of time as well as cost]4. example 3 people should sometimes do things that they do not enjoy. [in personal live, we have some responsibilities towards to other people, there is nobody who likes all of these responsibilities]1. [housework is very difficult for me, although my husband helps me some of them, but it is my responsibility]1. [i really don't like any of them, however i should do]2, [most people's lives are filled with tasks that they don't enjoy doing]3. example 4 following celebrities can be dangerous for the youth. [this has an overall effect on personality and future of an individual, following celebrities blindly affects the health of adolescents.]1 [many young people indulge themselves in drugs and start smoking at an early age]1. [in a survey carried out in a university, it was asked to students that why did they start smoking, then around forty percent of individuals answered that they wanted to look like their favorite screen actor while smoking cigarettes]2 [imitating celebrities has a negative influence on health of young individuals]3. ## automatic assessment of argumentation an overview of the automatic argument quality assessment framework we propose is visualized in figure 2 . starting from the persuasive essays where argument components and their relations are identified, the goal is to assess the quality of each argument (i.e., the quality of each claim). three scores are computed: a cogency score in the range {0, 15, 25}, an argumentation rhetoric label among ethos, logos, and pathos, and a reasonableness score in the range {0, 10, 15, 20, 25}. two different methods are combined to effectively assess the quality dimensions of the arguments: (i) the cogency score and the argumentation rhetoric labels are predicted using an attention-based neural architecture which employs the argumentation graphs through graph embeddings, and (ii) the reasonableness score is computed by means of an algorithm, combining the cogency score predicted at step (i) and the graph structure of each persuasive essay. in the following, we present the features we extracted from the persuasive essays to predict the cogency score and argumentation rhetoric labels, the neural architecture we define to predict these two quality dimensions, and we conclude with the reasonableness algorithm used to assess this score. ## evaluation in the following, we report on the experimental setup, the obtained results and the error analysis. experimental setup. for argument quality prediction, the embeddings (see section 4) were combined with either (i) a random forest, (ii) a lstm, (iii) a dense layer, or (iv) a svm. additionally, the best performing static and dynamic embeddings were concatenated and evaluated as if they were one single embedding. the pytorch framework @xcite version 1.10 was used for implementing the lstm model with a learning rate selected from 0.05, 0.1, rnn layers 1, 2, dropout 0.1, 0.3, 0.5, and batch size from 8, 16, 32 and a hidden size of 128. for longformer, bert and t5 pre-trained models, we use the pytorch implementation of huggingface @xcite version 4.16.2. for the graph feather-g embeddings, the karate club framework @xcite was used with the standard hyperparameters. the scikit-learn @xcite framework was employed for the implementation of the random forest and svm models. we trained the svm models for each quality attribute, optimizing the gamma and c hyperparameter (tested c range: 10 -4 to 10 3 , gamma range: 10 -4 to 10 0 ) on the training data set given by the original split @xcite in the dataset. for the rhetoric attribute, we trained the svm models concatenating the longformer and feather-g embeddings with the emotion word embedding. the latest was obtained by (i) running the fine-tuned t5 model to detect emotions, and (ii) either using that label as an input on glove, or extracting directly from the model the representation of the labels by summarizing the hidden states of the last four layers in the model. to train the binary classification, we con- we addressed a comparison with the state-of-theart approaches for the cogency assessment, despite the fact that we focus on a different dataset and divergent features (e.g., graph embeddings in our case). we retrained our model with the dataset of (wachsmuth and werner, 2020) following their same configuration. in this dataset of forum data, each argument instance is associated to 3 different gold labels for cogency, one for each annotator. they also separate them into 16 different topics and train each model with 15 of them, testing on the excluded one. we followed the same process for each annotator with our baseline model (longformer embedding + svm). given that they do not provide any graph structure we cannot test our best model on their data to compare. however, the results obtained are a mean absolute error of .64, .38 and .52 for expert #1, #2 and #3, respectively. comparing with (wachsmuth and werner, 2020), we can see that for expert #2 our baseline model outperforms their best model (.38 vs .57). in the case of expert #1, we obtain the same result as their baseline, and for expert #3 we perform similarly (.52 vs .50). the results we obtained on cogency (.78 f1) are, to the best of our knowledge, the best result obtained so far in the literature @xcite . ## concluding remarks we presented a novel approach to the task of automatic quality assessment of natural language argumentation. we built a new resource of 402 students' persuasive essays annotated with 3 different quality dimensions, i.e., cogency, rhetoric and reasonableness. through our extensive evaluation, we show that our neural architecture relying on a transformer with an attention mechanism and graph embeddings is able to successfully classify arguments along with these quality dimensions, outperforming standard baselines and similar approaches in the literature. our quality assessment method conjugates the empirical evaluation of the cogency dimension with the graph-based computation of the reasonableness one, which encompasses the quality (expressed in terms of cogency) of the counterarguments and the argumentation structure. in the context of ai in education, we aim to include our automatic argument quality assessment pipeline into a larger framework where the system engages the student into an explanatory rule-based dialogue to assess her essays, explain why they obtained a certain quality score and how to improve them along with the considered quality dimensions.