Graph-based Argument Quality Assessment (2021)

Abstract: The paper presents a novel discourse-based approach to argument quality assessment defined as a graph classification task, where the depth of reasoning (argumentation) is evident from the number and type of detected discourse units and relations between them. We successfully applied state-of-the-art discourse parsers and machine learning models to reconstruct argument graphs with the identified and classified discourse units as nodes and relations between them as edges. Then Graph Neural Networks were trained to predict the argument quality assessing its acceptability, relevance, sufficiency and overall cogency. The obtained accuracy ranges from 74.5% to 85.0% and indicates that discourse-based argument structures reflect qualitative properties of natural language arguments. The results open many interesting prospects for future research in the field of argumentation mining.

## introduction argumentation modelling and mining are steadily gaining attention of the broad natural language processing and engineering community. in many studies and applications, assessment of the argument quality plays an important role. the ability to construct good arguments and engage in argumentative discussions is assessed by argumentation systems focusing on training hypothetical reasoning, creating and structuring arguments @xcite , preventing opinion manipulation, detecting inconsistent arguments in online discussions and addressing different standpoints, attacking or supporting claims with evidence (debategraph foot_0 and truthmapping foot_1 ) as well as on the use of multimodal rhetorical devices @xcite . assessment of argument quality, its organization, clarity, adherence and strength, are approached by several authors as sub-tasks in the evaluation of written essays @xcite @xcite . online content is searched to filter or weight the validity of statements and factoids @xcite , to identify fake news and false claims @xcite and to detect opinion manipulation @xcite . while the acceptability of an argument in the presence of other supporting or attacking arguments has been addressed @xcite , 'local' argument quality still deserves our attention -an argument built on a certain set of conditions, is logically strong, rhetorically convincing, socially undistorted by virtue of its intrinsic properties. in this paper, we present a novel approach to assessing the structural strength and inferential weakness of arguments as merits of argument cogency. the approach relies on the discourse-based reconstruction of argumentation schemes. for this, we apply state-of-the-art discourse parsers and machine learning models to reconstruct argument graphs where the identified discourse units are represented as nodes and the classified discourse relations between them as edges. a graph neural network (gnn) model is built to predict the quality (low vs high) of the reconstructed argument graphs in terms of argument acceptability, relevance, sufficiency and overall cogency. the paper is structured as follows. section 2 defines the conceptual framework within which the study is performed. we provide the definition of an argument and elaborate on its internal structure. in section 3, we survey related work on argument quality assessment. section 4 presents the argument graph reconstruction approach. the performed gnn-based quality assessment experiments are discussed and results are reported in section 5. section 6 summarises our findings and outlines directions for future research. ## argument and its structure an argument may be considered as an atomic entity without an internal structure. for instance, an argument is defined as an overall position held by a person towards an idea or attitude, e.g. a stance in 'favour' or 'against' a certain motion @xcite . a structured argumentation model is an essential element for the tasks aiming at understanding and emulating of human inference, investigating patterns of reasoning, focusing at extraction and validity assessment of arguments. a simple argument structure is then defined as consisting of a claim that is supported by evidence(s) @xcite . a claim is an assertion that an argument aims to prove, i.e. a claim is a conclusion whose merit must be established. evidence comprises propositions which give reasons or grounds for drawing the conclusion. this general argument definition has been translated into several discourse-based schemes for analysing and evaluating natural language arguments @xcite ). an argument is modelled as a group of argumentative discourse units (adus) -text segments corresponding to propositions that are argumentatively relevant and have their own argumentative function @xcite ). an edu can function as a claim, as an evidence or as a conclusion. an adu can be identified as a collection of several elementary discourse units (edus) which correspond to clauses in written discourse and to dialogue acts in spoken discourse @xcite . discourse relations such as justification, motivation, cause, and exemplification can be used to identify how propositions are related to each other, inferring the type of support that is expressed. a claim may be summarized or re-stated in a conclusion. figure 1 depicts a general discourse-based argument structure. adus reflect different ways to provide support for a claim, i.e. links between them express the level of support that evidence provides to the claim and the level of their sufficiency to draw a valid conclusion. figure 2 provides an example of an argument. edus (solid-line rectangles) are combined by means of discourse relations into adus (dottedline rectangles) which are connected to each other evidence re-state statement result cause exemplify elaborate motivate conclude summarize claim conclusion justify, etc. figure 1: argument structure observed in spoken debate arguments, adapted from petukhova et al. (2017a). solid-line rectangles represent edus and dotted-line rectangles represent adus. by support links. evidence may either together (linked support, e.g. evidence 2.1 and 2.2 ) or independently (multiple support, e.g. evidence 1, 2 and 4) support a conclusion. a premise may provide support for another premise and indirectly support a conclusion (serial support, e.g. evidence 3 and 2). a special form of lending support to a claim is that of providing examples (example support, e.g. evidences 4.1 and 4.2). foot_2 clear properties of a good argument and successful argumentation are not easy to define. @xcite proposed a unified taxonomy of argumentation quality assessment that resulted from an extensive analysis of numerous existing approaches. the assessment comprises three quality dimensions: cogency, effectiveness and reasonableness. argument quality assessment aims at answering the question how logical, persuasive or convincing the given argument is, and how rhetorically appealing it is for the targeted audience. evaluation of argument cogency is based on the truthfulness and logical coherence of arguments. an argument is cogent if it has acceptable premises that are relevant and sufficient to support the conclusion @xcite . a premise is acceptable if it is rationally worthy of being believed to be true @xcite . according to @xcite , a premise is locally acceptable if it is supported by a cogent sub-argument or another cogent argument; it is a matter of common knowledge, testimony or expert view (appeal to authority). a statement a is positively relevant to another statement b if and only if the truth of a counts in favour of the truth of b. this means that according to the national association for pet container resources, pet water containers are now the most recycled container in curb side program by weight and by number! evidence 3 more than a million in the united states purchase bottled water every day which is helping the economy to exit the current recession. evidence 1 1.1 1.2 bottled water also only makes less than 1% the world's waste and 1% is a trivial proportion. 2.1 2.2 evidence 2 in addition, bottled water is a safer alternative as compared tap water in some countries. evidence 4 4.1 4.2 for example, according to research, tap water in panama, nicaragua, egypt and peru is unsafe for drinking banning plastic bottles would be a huge mistake in this very moment claim/conclusion cause exemplification elaboration figure 2: argument example from dagstuhl-15512 argquality corpus (wachsmuth et al., 2017) annotated with core iso 24617-8 core discourse relations (bunt and prasad, 2016) and support links observed. a provides some evidence for b, or some reason to believe that b is true. an argument is locally sufficient if all premises together provide sufficient reasons to accept the conclusion. the preconditions of the argument sufficiency are rooted in its local acceptability and its local relevance @xcite . the local sufficiency of an argument is often called inferential sufficiency and it holds if one of the following logical patterns is applicable: deductive entailment, conductive support, inductive support and analogy. argument quality is found to correlate with the argument's actual persuasive success. persuasion is defined as a process of encouraging people to do or believe something through argument. here, many factors are relevant, including psychological effects of argument memorisation, replication and reviewing @xcite . certain argumentation patterns are acknowledged as more persuasive than others, however they may differ in different domains. @xcite experimentally investigated lay people's expectations about the persuasiveness of anecdotal, statistical, causal, and expert evidence, and compared these expectations with the actual persuasiveness of these evidence types. van @xcite defined symptomatic (sign), comparison (resemblance) and causal (consequence) argumentation, and specified what argumentative patterns are more suitable/persuasive for what communicative types in various domains. for persuasive essays, different quality dimensions of argumentation were studied such as essay's organization @xcite , thesis clarity @xcite , prompt adherence @xcite and argument strength @xcite . these studies exploit a complex feature-rich approach to predict a score for each essay based on its content or style along with all of these categories. the study of @xcite looks at the argument persuasiveness from a different point of view: it does not try to estimate how persuasive an argument is but attempts to explain why an argument is experienced as unpersuasive. research has also targeted various interactive aspects, e.g. capturing the interactions between participants on argument level @xcite and providing feedback regarding the argument persuasiveness @xcite . many studies explore the aspect of argument convincingness foot_3 . in contrast to cogency, which is based on the truthfulness and logical coherence of arguments, convincingness is related to subjective perception by the audience @xcite . experiments were performed to detect more convincing arguments @xcite and evidence @xcite . the rhetorical force of an argument should not be underestimated. due to the use of powerful rhetorical devices, even a not very cogent argument may be perceived as convincing @xcite . people generally associate certain speech, personality and interaction features with what they think is a persuasive argument. more broadly, the persuasion literature of the last decades has shown that an argument that has higher perceived competence (e.g., evidence-based expert knowledge) and/or higher warmth (e.g., more likeable and trustworthy) is more convincing @xcite . the study of @xcite suggests an argument quality assessment approach that focuses solely on the argument structure, and defines statistical patterns in the structure of essays and define novel features that are evaluated in argumentation-related essay scoring tasks. the present study investigates the structural properties of a cogent argument and assesses its inferential strength, i.e. structures of inference (argumentation schemes) predicted from the associated amount, depth and type of evidence provided to the claim. ## argument graph reconstruction we define argument graph reconstruction to involve: (1) segmentation of a text into edus; (2) discourse relation detection and classification between them; (3) identification and classification of adus based on the classified discourse relations; and (4) argument completion, i.e. reconstruction of the implicit units to achieve a complete argument structure, see figure 3 . we performed joint and two-stage segmentation and classification of edus. for the joint segmentation and classification, the full pdtb parser developed by @xcite was applied. we observed that the parser failed to identify many edu spans. 5 rather low overall f1 scores of 21.20% for exact segment boundaries match and 5-class discourse relation classification were achieved on the penn discourse tree bank 1.0 corpus (pdtb 1.0, @xcite ), see the right part of the two-stage segmentation and classification was performed applying the bilstm-crf based segmentation model neuraleduseg developed by @xcite and the xlnet-large discourse relations classification model by @xcite . a segmentation performance of 68.55% in terms of f1 score was achieved when testing on the pdtb 1.0 and pdtb 2.0 datasets (pdtb 2.0, @xcite ). 6 for discourse relation recognition with the xlnet model designed by @xcite , we first carried out a binary classification to establish whether any relation exists between the identified units, i.e. discriminate between the rel class which includes any type of discourse relations and norel types. the former comprises explicitly marked (explicit), implicitly marked (implicit) and alternatively lexicalized (altlex) discourse relations, the later includes entrel relation which is not a discourse relation between clauses but an entity-based coherence relation. subsequently, we performed five-class toplevel (l1) and ten-class fine-grained (l2) relations classification. since class distributions were unbalanced in all classification settings, re-sampling was performed: up-sampling of the under-represented norel class in binary classification by adding synthetic samples. for this random edus from different textual units were combined. in addition to this, down-sampling of the majority classes in the multi-class settings was performed. for the training and evaluation procedures, we fine-tuned each encoder model following the suggestions of @xcite and trained it for 10 epochs using a learning rate of to demonstrate the applicability of the approach beyond pdtb, we applied the two-stage segmentation and classification procedure and finetuned models on the argumentative dagstuhl15512 argquality corpus which is a collection of 304 argumentative texts annotated according to 15 argument quality criteria @xcite . the output was manually examined and corrected. we observed that some argument components, often claims, are implicit, see also @xcite . without the claim or conclusion, an argument structure is incomplete. therefore, we reconstructed a claim for every topic in the corpus, either 'for' or 'against' stance it may present. the reconstructed claim is a simple sentence corresponding to a single edu, see the identified dagstuhl arguments are of different length and have various, often complex discourse-based structures distinguishable through diverse linking patterns and number of evidences provided by an arguer to support a claim. figure 4 provides an example of the identified discoursebased argumentation scheme. the upper node represents the claim books are better than tv which was supported by seven evidence statements, six of them connected to the claim by means of contingency.cause relation and one by expansion.instantiation. five of the evidence statements correspond to one edu, whereas the other two are more complex and consist of two edus. finally, argument structures were represented as graphs where the detected edus spans are represented as nodes and the classified discourse relations -as edges visualising number and level of supporting evidence through links to the claim and premises. 2,278 reconstructed arguments have 303 structures (argumentative schemes) specifying 172 unique reasoning patterns. figure 6 in appendix iii provides the most frequent examples of the reconstructed discourse-based argument structures.