How Will I Argue? A Dataset for Evaluating Recommender Systems for Argumentations (2021)

Abstract: Exchanging arguments is an important part in communication, but we are often flooded with lots of arguments for different positions or are captured in filter bubbles. Tools which can present strong arguments relevant to oneself could help to reduce those problems. To be able to evaluate algorithms which can predict how convincing an argument is, we have collected a dataset with more than 900 arguments and personal attitudes of 600 individuals, which we present in this paper. Based on this data, we suggest three recommender tasks, for which we provide two baseline results from a simple majority classifier and a more complex nearest-neighbor algorithm. Our results suggest that better algorithms can still be developed, and we invite the community to improve on our results.

## introduction argumentation is an important tool of human communication and interaction. arguments allow us to justify our views and opinions and persuade others. they also play an important role when it comes to decision-making. not only in terms of law and justice @xcite , but also for each and every personal decision we make on a daily basis. taking a position on a controversial issue can be difficult, especially when there are many pro and contra arguments to consider. finding the arguments that are most important and convincing for oneself is an important aspect in the process of decision-making. for a wide range of fields, recommender systems already facilitate our decisions, using collaborative and content-based filtering algorithms @xcite , filtering the great load of information that can be found online @xcite . a recommender system for argumentations could help users to make decisions more confidently and also gain a better understanding of the whole issue discussed. first applications like the predictive and relevance based heuristic agent @xcite and our platform deliberate @xcite were presented to address this task. they try to present arguments to users which are most relevant for them. but large-scale datasets to systematically test and evaluate such recommender systems for argumentations outside a laboratory setting are missing. in this work, we provide a dataset including more than 900 arguments and 600 user profiles, obtained as part of a larger study on political opinionforming. in this study, we let participants interact with our platform deliberate, exposing them to arguments we gathered beforehand, concerning two different controversial questions on nutrition policy. the participants could rate the overall strength of the displayed arguments, indicate whether they find them convincing, and add own arguments. they were exposed to the topics at different points of time, such that the user profiles grow over time and the dataset can be used to test predicting future user behavior. the dataset we provide here should serve to test and evaluate metrics and algorithms for argument recommender systems. as a baseline, we provide our results from two different algorithms on three different tasks which are predicting the user conviction towards an argument, the assigned strength of an argument, and the top-3 convincing arguments. the baseline results are obtained using a plain majority classifier and the existing recommender algorithm of deliberate to test its performance. to our knowledge, we provide the first large-scale dataset on the task of argument recommendation which contains user attitudes at different points of time. the paper is structured as follows. in section 2, the theoretical basics on argumentation and the terms used in this paper are defined. the data we collected is described in detail in section 3. section 4 introduces the three challenges and subtasks for argument recommendation we propose in this work, for which we provide two baseline results which are subsequently discussed. section 5 gives an overview of related research, and finally, we summarize our work and look at future work. ## definitions in this paper, we use terms based on the ibis model @xcite , but our dataset can also be interpreted in bipolar dung-style (dung, 1995) argumentation frameworks. the atomic building blocks of argumentations are textual statements. two statements, called premise and conclusion, form an argument. the premise can either support or attack the conclusion. a controversial statement which is argued about is called position, e.g., "plastic packaging for fresh food should be prohibited," and is typically an action which can be performed. positions do not have a conclusion, but they can be used as conclusions when arguing why the position is sensible or not. all statements define an argumentation graph where statements are nodes and the edges are arguments, i.e., they represent the argumentative relation between statements. for simplicity, userinterfaces like deliberate often call the premises themselves arguments to hide the technical definition of argument from the user. when the conclusion talked about is fixed, an argument can be uniquely identified by its premise. individual persons can have different opinions on the statements in an argumentation, e.g., agree or disagree with them with different strengths (i.e., the person can be (un)sure about their opinion). in real-world applications, a person's opinion on a statement can be unknown, leading to sparse data. furthermore, a person can consider an argument more or less convincing than another argument with the same conclusion; we call this weight, and we use a value from the interval [0, 6] to represent it, where higher values correspond to stronger weights; this interval directly corresponds to the likert scale we used during data collection. we call the collection of weights and opinions of a person in an argumentation attitude. a person's attitude and their user name form a user profile. s will refer to a set of statements. for a statement @xmath0, c(s) ∈ {0, 1} indicates whether the argument is considered convincing (1) or not (0) by a user, and w(s) ∈ [0, 6] is the associated weight. predicted values for conviction and weight produced by a prediction algorithm are referred to as ĉ(s) and ŵ(s), respectively. the set of all user profiles is called u and can be represented as big sparse matrix with user profiles in the rows (i.e., in our case, with columns for the user name, position agreement strength, and, for each argument, columns for premise conviction and argument weight). ## description of the dataset we present our new argumentation dataset with arguments on two different positions on nutrition policies in germany (see the two discussed issues have been identified as the most topical and polarizing ones from a preselected set of controversial questions through a pre-test survey before our main study. in the original main study, we examined whether the use of artificial intelligence methods to pre-select arguments participants can see has an impact on the political opinion forming of individuals in the field of nutrition policies. now, we first explain the general data collection and the demographics of the participants. afterwards, we expound on the pieces of information collected for our data set. finally, we explain how the dataset looks like and where to obtain it. ## challenges and baseline results for recommender systems based on our dataset, we introduce three different classification and recommendation tasks where the opinions on statements and weights of arguments have to be predicted. we provide baseline results from a majority classifier and a neighbor-based recommendation algorithm to get a first feeling for the hardness of the tasks. ## related work similar datasets have been published before, and similar recommender tasks have been considered. @xcite suggested the task of predicting convincingness of web argument pairs. they annotated and published a large-scale dataset of 16k argument pairs on 32 topics for the task of convincingness prediction and argument ranking. different from our work, the task was not predicting the attitudes for each user for a given argument, but compare arguments in pairs and de2019 ) presented a dataset with 16 positions on 4 issues, for which 309 students gave their attitudes by adding arguments and indicating their level of agreement with that argument on a scale from -1 (total disagreement) to 1 (total agreement). using the information about argument agreement, the agreement with the position was calculated. in our work, however, we explicitly ask for the agreement with a position, which allows a user to have an opinion which is inconsistent with their arguments. the authors also compared different algorithms for predicting user opinions on positions, where the best algorithm was a kind of soft cosine measure, which exploited feature similarity using position correlation. @xcite tested different recommender agents in laboratory argumentation settings where arguments probably used next in a discussion were suggested. different features were considered, i.a., the distance of arguments in the argumentation graph, a calculated argument strength, and the current context in the discussion. several machine learning algorithms like svms and neural networks were evaluated. this is different from our work because we only recommend statements which are a premise for a given statement, although considering a broader suggestion strategy, which suggests statements from a different context, might be more appropriate for specific applications. chalaguine and hunter (2020) presented a chat bot which should select appropriate counterarguments, using cosine and concern similarity, with the goal of persuading a human to change their opinion. they compared their algorithms with a random baseline and got significantly better-thanrandom results for selecting relevant arguments. a crowd-sourced dataset with arguments about uk university fees was used @xcite . in contrast to our work, this dataset only contains arguments, but no user profiles with the attitudes of different persons on the arguments. the same applies to other corpora, like the internet argument corpus @xcite . ## conclusion and future work in our work, we introduce an extensive dataset which contains more than 900 arguments for two political positions and the user attitude data from more than 600 individuals, collected at different points of time. this dataset can be used for evaluating argument recommender systems, which can, e.g., be used to help people finding personally relevant arguments in discussions with many arguments. we suggest three different recommender tasks and provide baseline results from a simple majority predictor and a more sophisticated nearestneighbor algorithm, which yields better results. our baseline results can still be improved on, and we invite everyone to develop better algorithms. possible first improvements are considering linguistic information, and using different metrics for the nearest-neighbor classifier. what is more, other tasks could be defined on our dataset, e.g., predicting t 3 data from t 1 or non-convincing arguments. furthermore, we want to research the effects of different recommendation strategies for argumentation on the formation of opinion when they are used to pre-filter content a user can see. other evaluations in terms of novelty and utility should also be considered in the future.