A rg A nalysis35 K : A large-scale dataset for Argument Quality Analysis (2023)

Abstract: Argument Quality Detection is an emerging field in NLP which has seen significant recent development. However, existing datasets in this field suffer from a lack of quality, quantity and diversity of topics and arguments, specifically the presence of vague arguments that are not persuasive in nature. In this paper, we leverage a combined experience of 10+ years of Parliamentary Debating to create a dataset that covers significantly more topics and has a wide range of sources to capture more diversity of opinion. With 34,890 high-quality argument-analysis pairs (a term we introduce in this paper), this is also the largest dataset of its kind to our knowledge. In addition to this contribution, we introduce an innovative argument scoring system based on instance-level annotator reliability and propose a quantitative model of scoring the relevance of arguments to a range of topics.

## introduction parliamentary debate is an extemporaneous form of debating. one of the major intersections of natural language processing and debating was ibm project debater @xcite , an end-toend system that mines arguments in a text @xcite , determines argument quality @xcite , and through a combination of modules can debate against a human being. the purpose of this paper is to propose a new dataset foot_0 that adds a new dimension to the field of argument quality detection in the context of parliamentary debating, eventually enabling the creation of a system that can beat a human debater in a parliamentary debate. the dimension that we introduce here is a detailed explanation of why the argument made is true, applicable or impactful, henceforth referred to as "analysis". analysis is defined as logical links provided to defend a statement, an example of which can be seen in these logical links need to be seen as one "analysis" instead of multiple claims and subclaims because each subsequent link needs to be seen in the context of the links that come before to build the overall reason for defending the argument. (good primary @xmath0. here, each individual sub-claim does not defend the overall argument, but rather the collection of links in order that performs that function. argument motions a child is still growing, physically and mentally, cosmetic surgery should not be considered until they are an adult and able to make these decisions we should ban cosmetic surgery for minors racial profiling unfairly targets minorities and poor we should end racial profiling argument relevance is an important indicator of persuasiveness according to @xcite . in a parliamentary debating context, the same argument can be applied to a variety of topics and can be differently persuasive for each topic. arguments like "accountability is important" can be used in debates about governments, churches, corporations, schools, etc. similarly, arguments that deal with the premise of free speech being important can be used to defend free speech for members of the lgbtq community, as well as to defend people's right to protest against a corporation. the quantification of relevance of the argument to the topic under discussion is defined as the relevance model which attempts to capture this complexity. application of instance-based annotator reliability to argumentation is another important contribution described in this paper. some annotators might know a lot more about art than about the criminal justice system, hence might judge certain arguments as more or less persuasive using their knowledge; secondly, because of the element of bias that comes in when ranking arguments. annotators might be biased about a certain argument on race, for example, because of the strong sentiments they feel towards them in their daily life, but they may not be biased when judging an argument on art. we propose a system that enables us to keep the scores of these annotators instead of dropping them, like previous systems have, and show how this leads to a better overall dataset with a more uniform distribution of scores. the dataset is crucial to designing systems that can interact efficiently with humans. arguments generated with this system can analyze arguments better, and create effective rebuttals using high scoring arguments of the other side. the dataset can also be used to judge a debate by assigning scores to arguments as per their level. any interactive system, such as ibms project debater needs this dataset as a preliminary base to analyze and win debates with a human. in summary, our major contributions detailed in this paper are: (1) argument-analysis pairs collected from a variety of sources on a variety of topics; (2) introduction of a relevance model that enables the use of multiple arguments in different contexts; (3) introduction of an instance based annotator scoring system that reduces bias and makes argument scores more accurate. ## related works there have been several datasets in the field of argument quality using empirical methods that focus on finding arguments and evidence. @xcite collects policy arguments and evidence from national speech and debate association, while hua and wang (2017) categorises argument analysis african american groups should fight for economic reparations from the government. reparations are required because african americans were asked to pay equal taxes while being treated unequally with laws such as jim crow laws, 3 5 citizen rule, etc. racial appearance changes should be banned because it leads to discrimination. anti discrimination legislation is prefaced on the fact that all races should be treated equally because race is something you cannot change, this is undermined when the government allows changing of race. lastly, a major contribution in this work is the proposal of a relevance model. @xcite suggested a model that decomposes quality to 15 dimensions to determine the qualities that make an argument persuasive. they discover that relevance is an important factor that determines argument quality. @xcite uses this as the basis to discover that global relevance (how related an argument is to the topic) has the highest difference between low and high scoring arguments, hence proving that it is the most important factor that determined how persuasive annotators found it. we use this theory as the basis to create a relevance model that judges this quantitatively. @xcite finds relevance using the number of other arguments that use it as a premise. our method is different from this as it does not depend on other arguments and can be used independently on every argument. ## scoring functions scoring an argument-analysis pair is an inherently subjective task. in order to make it as objective as possible, we have reduced the annotator involvement to two binary questions. however in order to make our dataset usable and interfaceable with others in the field @xcite , we need to convert these annotations to a quality score. in order to do this, we have used the two methods used in the creation of ibm-30k as well as a third, recently proposed method @xcite that models annotator reliability on a per instance basis. ## relevance model in this section, we describe the relevance model that quantifies the applicability of each argumentanalysis pair to a topic. the underlying assumption is that each argument-analysis pair has a degree of applicability to at least one and likely more topics. this assumption is made on the basis of the personal experience that we have gathered while debating and discussions with experts in the field, where we often find that arguments repeat across multiple topics and motions. @xcite conducted a qualitative evaluation of the correlation between relevance or applicability of an argument and a topic and how that is one of the factors by which we can understand why a particular argument is good. we believe that the approach can be extended in a quantitative manner by application of topic modeling and topic analysis. ## conclusion and future works in this work, we create arganalysis35k and validate it using a variety of methods. this system can be integrated with existing models to create a system that is able to debate more efficiently, be more persuasive, and as a result win more debates.