Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks (2023)

Abstract: The lack of annotated data on professional argumentation and complete argumentative debates has led to the oversimplification and the inability of approaching more complex natural language processing tasks. Such is the case of the automatic evaluation of complete professional argumentative debates. In this paper, we propose an original hybrid method to automatically predict the winning stance in this kind of debates. For that purpose, we combine concepts from argumentation theory such as argumentation frameworks and semantics, with Transformer-based architectures and neural graph networks. Furthermore, we obtain promising results that lay the basis on an unexplored new instance of the automatic analysis of natural language arguments.

## introduction the automatic evaluation of argumentative debates is a natural language processing (nlp) task that can support judges in debate tournaments, analysts of political debates, and even help to understand the human reasoning used in social media (e.g., twitter debates) where argumentation may be difficult to follow. this task belongs to the computational argumentation area of research, a broad, multidisciplinary area of research that has been evolving rapidly in the last years @xcite . classically, computational argumentation research focused on formal abstract logic and computational (i.e., graph) representations of arguments and their relations. in this approach, the evaluation of arguments relied exclusively in logical and topological properties of the argument representations @xcite . furthermore, these techniques have been thoroughly studied and analysed, but from a theoretical and formal viewpoint considering specific cases and configurations instead of large, informal debates @xcite . the significant advances in nlp have enabled the study of new less formal approaches to under-take argumentative analysis tasks @xcite . one of the most popular tasks that has gained a lot of popularity in the recent years is argument mining, a task aimed at finding argumentative elements in natural language inputs (i.e., argumentative discourse segmentation) @xcite , defining their argumentative purpose (i.e., argumentative component classification) @xcite , and detecting argumentative structures between these elements (i.e., argumentative relation identification) @xcite . even though most of the nlp research applied to computational argumentation has been focused in argument mining, other nlp-based tasks have also been researched such as the generation of natural language arguments @xcite , the assesment of the persuasiveness of natural language arguments (el @xcite , and the automatic generation of argument summaries @xcite among others. however, it is possible to observe an important lack of research aimed at the evaluation of complete argumentative debates approached with nlp-based algorithms. furthermore, most of the existing research in this topic has been contextualised in online debate forums, considering only short text arguments and messages, and without a professional human evaluation @xcite . in this paper, we propose a hybrid method for evaluating complete argumentative debates considering the lines of reasoning presented by professional debaters and taking into account the human evaluations provided by an impartial jury. our method combines concepts from the classical computational argumentation theory (i.e., argumentation frameworks and semantics), with models and algorithms effectively used in other nlp tasks (i.e., transformer-based sentence vector representations and graph networks). this way, we take a complete professional debate including all the argumentation and rebuttal phases as an input, and predict the win-ning stance (i.e., in favour or against) for a given argumentative topic. for that purpose, we define an original computational modelling of complete professional natural language debates that allows us to improve the prediction of the winning stance of complete argumentative debates compared to more conventional approaches. we present a complete comparison of approaches relying exclusively on nlp algorithms and approaches relying exclusively on argumentation theory concepts. from our findings, we can observe that the hybrid approach proposed in this paper is the more adequate to tackle a complex nlp challenge such as predicting the winning stance in complete debates. ## related work classically, the computational representation and assessment of arguments has been conducted through argumentation frameworks and argumentation semantics @xcite . however, this line of research has been focused on abstract argumentation and formal logic-based argumentative structures, and has not been properly extended to the informal natural language representation of human argumentation. the automatic assessment of natural language arguments is a relatively new topic of research that has been addressed from different nlp viewpoints. most of this research has been focused on performing an individual evaluation of arguments or argumentative lines of reasoning @xcite instead of a global, interactive viewpoint where complete debates consisting of multiple, conflicting lines of reasoning are analysed. typically, the automatic evaluation of natural language arguments has been carried out comparing the convincingness of pairs of arguments @xcite ; analysing user features such as interests or personality to predict argument persuasiveness @xcite ; and analysing natural language features of argumentative text to estimate its persuasive effect @xcite . the use of graph-based approaches to evaluate individual argument structures has been recently explored in @xcite . in the same direction, @xcite proposes a framework for evaluating three dimensions of arguments (i.e., cogency, rhetoric, and reasonableness) by producing natural language embeddings from individual argument structures (e.g., claim -premise). however, none of them considers the problem of argument evaluation in an argumentative dialogue as in the case of the debates. the global (i.e., debate) approach on the evaluation of natural language arguments was initially researched in @xcite where recurrent neural networks were used to evaluate non-professional debates in a corpus of limited size and structure. following this trend, in @xcite , the authors propose a method based on the persuasiveness to predict the outcome of online debates using a support vector machine. recently, in @xcite , the authors present an algorithm for predicting the outcome of nonprofessional debates of limited length and depth in online forums. furthermore, in the previous work the considered argumentative structures are simple, and the proposed methods depend exclusively on natural language features. all these works have two main aspects in common: first, they are focused exclusively in online text-based debates, where information is easy to obtain, but very limited from an argumentative viewpoint; and second, the debates brought into consideration present short interactions and simpler arguments than the ones that can be found in a professional debate. interestingly, a recent trend in the proposed argument assessment algorithms from the more theoretical side of the computational argumentation area of research also consists of leveraging structural information of the graph to estimate the acceptability of arguments by using neural networks instead of classic solvers @xcite @xcite . however, in these cases arguments are treated as abstract entities without specific natural language being attributed to them, making their results more difficult to contextualise in a real situation with natural language arguments. the previously reviewed work evidences that fundamental concepts from computational argumentation theory are typically overseen in the argument-related nlp literature, and the used corpora contain debates far removed from the concept of a professional debate. thus, we propose a new method that combines the advantages of both areas of research: formal argumentation theory and nlp. this way, our proposal enables the analysis of complete professional argumentative debates in both, length and argumentative depth, a task that has not been addressed in the literature yet. ## data in this paper, we approach the automatic prediction of the winning stance in complete natural language professional debates. for that purpose, we use the vivesdebate foot_0 corpus @xcite to conduct all the experiments and the validation of our proposed method. this corpus contains the annotations of the complete lines of reasoning presented by the debaters in a debate tournament, inspired by the aif @xcite standard, and the professional jury evaluations of the quality of argumentation presented in each debate. it is important to emphasise this aspect, as the average length of the debates we analysed in this paper is 4819 words (30-40 minutes of length), and large language models have problems when working with long sequences of natural language text @xcite foot_1 . previously published corpora for the analysis of natural language argumentation always tended to simplify the annotated argumentative reasoning, by only considering individual arguments, pairs of arguments, or considering a small set of arguments, instead of deeper and complete lines of argumentative reasoning. for example, in argument mining (e.g., us2016 @xcite and qt30 (hautli-janisz et al., 2022)), argument assessment (e.g., ibm-eviconv @xcite ), or natural language argument generation/summarisation (e.g., gpr-kb @xcite , debatesum , (roush and balaji, 2020)). furthermore, online debates with their crowd-sourced evaluations were compiled in @xcite , but argumentation was produced in short written paragraphs, and evaluations were based on anonymous votes from the community that did not require any justification. therefore, the vivesdebate corpus is the only identified publicly available corpus that enables the study of the automatic evaluation of natural language professional debates in their complete form. the vivesdebate corpus contains 29 complete argumentative debates (139,756 words) from a university debate tournament in catalan. each debate is annotated entirely without partitions, and capturing the complete lines of reasoning presented by the debaters. the natural language text is segmented into argumentative discourse units (7,810 adus) @xcite . each adu contains its own text, its stance (i.e., in favour or against the topic of the debate), the phase of the debate where it has been uttered (i.e., introduction, argumentation, and conclusion), and a set of argumentative relations (i.e., inference, conflict, and rephrase) that make possible to capture argumentative structures, the sequentiality in the debate, and the existing major lines of reasoning. additionally, each debate has the scores of the jury that indicate which team has proposed a more solid and stronger argumentative reasoning. an in-depth analysis of the corpus structure and statistics can be found in @xcite . ## method the human evaluation of argumentative debates is a complex task that involves many different aspects such as the thesis solidity, the argumentation quality, and other linguistic aspects of the debate (e.g., oral fluency, grammatical correctness, etc.). it is possible to observe that both, the logic of argumentation and the linguistic properties play a major role in the evaluation of argumentative debates. therefore, the method proposed in this paper is designed to capture both aspects of argumentation by combining concepts from computational argumentation theory and nlp. our method is divided into two different phases: first, (i) determining the acceptability of arguments (i.e., their logical validity) in a debate based on their logical structures and relations; and second, (ii) scoring the resulting acceptable arguments by analysing aspects of their underlying natural language features to determine the winner of a debate. figure 1 presents an scheme with the most important phases and elements of the proposed method. before describing both phases of our method, it is important to contextualise our proposal within the area of computational argumentation research. we assume that the whole argument analysis of natural language text has already been carried out: the argumentative discourse has been segmented, the argument components have been classified, and argument relations have been identified among the segmented argumentative text spans (see @xcite ). thus, a graph structure can be defined from a given natural language argumentative input. as depicted in figure 1 , the argument analysis containing the text of the arguments (i.e., node content), their stance (i.e., node colour), inference relations (i.e., green edges), conflict relations argument analysis framework encoding argumentation semantics learning sample sample init. debate [...] [...] [...] language model graph network [...]' [...]' [...]' predictor .... .... winner looser phase i phase ii (i.e., red edges), and rephrase relations (i.e., yellow edges) among arguments can be a valid starting point to the proposed method. @xcite ## discussion in this paper, we have defined an original hybrid method to approach the winning stance prediction of complete natural language argumentative debates. for that purpose, we present a new instance of the debate assessment task, where argumentative debates and their underlying lines of reasoning are considered in a comprehensive, undivided manner. the proposed method combines aspects from formal logic and computational argumentation theory, with nlp and deep learning. from the observed results, several conclusions can be drawn. first, it has been possible to determine that our method performed better than approaching independently the debate evaluation task from either the argumentation theory or the nlp viewpoints. furthermore, we have observed in our experiments that conflict-free semantics produce a higher number of acceptable extensions from each af compared to the admissibility-based semantics. this helped to improve the learning process of the task in a similar way to that achieved by data augmentation techniques. thus, a better probabilistic models of natural language distributions that are not too constrained to formal logic and graph topology can be learnt by our model. this paper represents a solid starting point of research in the evaluation of complete natural language professional debates. as future work, we are interested in considering finer-grained features for the evaluation of argumentation such as thesis solidity, argumentation quality, and adaptability. we also plan to extend our method with acoustic features, considering aspects such as the intonation or the fluency.