Reports of personal experiences and stories in argumentation: datasets and analysis (2022)

Abstract: Reports of personal experiences or stories can play a crucial role in argumentation, as they represent an immediate and (often) relatable way to back up one’s position with respect to a given topic. They are easy to understand and increase empathy: this makes them powerful in argumentation. The impact of personal reports and stories in argumentation has been studied in the Social Sciences, but it is still largely underexplored in NLP. Our work is the first step towards filling this gap: our goal is to develop robust classifiers to identify documents containing personal experiences and reports. The main challenge is the scarcity of annotated data: our solution is to leverage existing annotations to be able to scale-up the analysis. Our contribution is two-fold. First, we conduct a set of in-domain and cross-domain experiments involving three datasets (two from Argument Mining, one from the Social Sciences), modeling architectures, training setups and fine-tuning options tailored to the involved domains. We show that despite the differences among datasets and annotations, robust cross-domain classification is possible. Second, we employ linear regression for performance mining, identifying performance trends both for overall classification performance and individual classifier predictions.

## introduction although personal narratives and experiences naturally fill an important place in our everyday discussions, they still do not conform to the classic ideal of a "good" argument. a "good" argument contains facts and logical conclusions, but as soon as more personal or emotional nuances are involved, it deviates from the norm. according to the theory of deliberative democracy @xcite , the discourse that precedes political decisions plays a central role. here, too, until the so-called affective turn in social sciences @xcite , the assumption was that an exchange of arguments that is as rational as possible can lead to better political decisions. recent studies in deliberative theory, however, are increasingly concerned with the interplay between classical argumentation and alternative forms of argumentation, which include personal experiences, narratives and emotions, and their positive effects on discourse @xcite @xcite . a norm that only allows rational and logical argumentation firstly does not correspond to human realistic communication and secondly bears the danger that less educated groups or groups in which other communication standards prevail are marginalized. arguments with personal experiences are therefore important to fulfill one of the core deliberative standards, namely inclusivity @xcite . the following example is taken from a discussion about regulations to ban peanut products on airlines and illustrates further possible positive effects of arguments with personal experiences: my daughter has been tested 4 times for her allergy to peanuts. she is in the highest category of reactivity which means if peanuts are being ingested in her vicinity, she could die. a buffer zone simply doesn't work in a confined space such as an airline. 1 the perspective of the frightened parent and their allergy-affected daughter is likely to elicit emphatic reactions from the other participants in the discussion and illustrates the rationale of proponents of a peanut ban. our work represents the first step towards identifying these arguments at a large scale by developing models for automatic detection of contributions with personal experiences and stories (to which we refer with the general label of reports in the paper). this is the necessary first step to be able to further examine the role of such arguments in reasoning and their relationship with argument quality @xcite and discourse quality @xcite , to tackle this task we collect available datasets and investigate the best strategy to merge the available sources. in two of the datasets we employ, produced by the argument mining community (regulation room and change my view) our targeted phenomenon is annotated as testimony. in the third dataset, from the social science community (europolis) it is annotated as storytelling. the two categories are obviously not fully overlapping, but we hypothesize that they share a conceptual core which can be leveraged for robust classification. we perform a large set of in-domain, out-domain and cross-domain experiments including different domain-adaptation strategies for the pre-trained language models used in the classification. we analyze our results using performance mining and show that the cross-domain training setup leads to the most robust results and also has the most positive effect when used with a domain-adapted lm. we also conduct regression-based error analysis and compare the most salient features of the two largest datasets that get picked up by the most robust model and show that prototypical textual properties of reports can push the model in the right direction (probability of reports close to decision boundary) but can also lead to over-generalization (higher probability of reports for false positives). ## related work argument mining & nlp the automatic detection of arguments with personal experiences was first tackled by @xcite with the goal to classify claims as verifiable or unverifiable and to be able to detect what type of evidence would be necessary as a consequence. they defined the subcategory verifiable experiential for verifiable claims that contain personal experiences. their dataset contains comments from regulation-room, 2 an e-rulemaking platform with the goal of enabling online deliberation: governmental institutions or companies can have their proposals about new regulations discussed by citizens to get feedback. @xcite conducted classification experiments with a svm using different feature sets hypothesizing that the amount of past tense and first personal pronouns would be most predictive for verifiable experientials. they 2 http://regulationroom.org/ achieved a f1-score of ∼ 70% on the regulation-room dataset. this work was further extended in @xcite and reformulated as a sequence classification task. @xcite and @xcite developed a new annotation scheme targeted at elementary units in arguments. this schema introduces testimony as an elementary unit which corresponds to a proposition about the author's personal state or experience. a similar evidence type, called anecdote and defined as a personal experience of the author or a narration of a concrete example or event was classified in news editorials @xcite . @xcite build a database for claims and suitable anecdotes focusing on stories with a clear narrative structure and popular main characters (e.g. the dalai lama). @xcite model different persuasion strategies in dialogues targeting social good (e.g. fund raising): personal stories which exemplify positive outcomes and benefits of a donation are one of these strategies. ## datasets regulation room (regroom) our experiments are based on the final version of the cornell erulemaking corpus (cdcp) @xcite it contains 725 comments from regulation room, discussing consumer debt collection practices in the united states. the comments are annotated with different proposition types, and our category of interest is testimony. change my view (cmv) contains 344 comments from the subreddit changemyview 3 @xcite and is annotated with a similar schema as introduced in @xcite . our reference category is testimony. europolis @xcite , already introduced in section 2, contains a total of 856 transcribed speech contributions whose original language was german, french, and polish (only available in the english translation). 4 europolis is annotated along many deliberative quality dimensions @xcite , and our reference category is storytelling. from now on, we will use the neutral term re-port as our positive label, of which testimony and storytelling are dataset-specific declinations. note that, while storytelling is annotated at the document level in europolis, regroom and cmv contain span-level annotation of testimony: for our experiments, a document containing a testimony span is considered as a positive instance of report. for all datasets, reports are the minority class (re-groom: 41%; cmv: 37%, europolis: 35%). ## experiments task we perform binary classification at the document level: forum posts in regroom and cmv, spoken contributions in europolis. we create 10 random train / dev / test splits using 15 % as development and 20 % as test data and we ensure that every document is part of the test set at least once. ## analysis to get a statistically informed understanding of the different factors influencing the behavior of our models on the different datasets, we employ linear regression. our dependent variables are aggregated performance (f1 macro) in section 5.1 and model predictions on individual items (probability of report) in section 5.2. for the aggregated performance analysis in section 5.1, our independent variables (iv) are: the different experimental configurations i.e., combinations of classifier architectures (referred to as "model" in the tables), training setup and test corpus as well as their interactions, as specified in the formula: 7 f1macro ∼ (model + training setup + test corpus)ˆ3 for the model predictions (section 5.2) our independent variables are: the contribution-level features in 3.1 and a subset of the experimental configurations (training setup and test corpus), as well as the interactions between the contribution-level features and the experimental configurations. in both cases, our analysis proceeds in three steps. first, we fit incrementally complex models and assess their fit in terms of adjusted r 2 (proportion of explained variance) and significance with respect to the less complex models (i.e., models with fewer independent variables). at this step, we also check for multicollinearities. next, once we identify the most explanatory regression model (i.e., the set of independent variables that maximizes the fit to the dependent variables), we proceed to identify its most explanatory ivs in terms of explained variance and significance (e.g., does the choice of training setup determine a strong difference in the performance of our classifiers?). last, we identify the best values for the ivs (e.g., which of the training setups guarantees best f1?) by visualizing predicted performance with the help of effect displays @xcite , which show the partial effect of one (or more) parameters by marginalizing over all other parameters. ## conclusion this work targeted the automatic identification of personal experiences and stories in argumentation. leveraging available annotation in three argumentative datasets (two, regulation room and change my view, from the argument mining community; one, europolis, from the deliberative theory community), we evaluated different classifier architectures and training setups. mixing training data from different domains leads to robust results across all corpora and models and boosts the performance of the classifiers based on the domain-adapted lms. 13 our experiments established an empirical foundation that will allow us to investigate the target phenomenon at a larger scale and will lead to a better understanding of the compatibility of the underlying annotations (storytelling from deliberative theory and testimony in argument mining) as well as of the impact of storytelling/testimony on the quality of an argument. chenhao tan, vlad niculae, cristian danescu-niculescu-mizil, and lillian lee. 2016. winning arguments: interaction dynamics and persuasion strategies in good-faith online discussions. in proceedings of the 25th international conference on world wide web, www '16, page 613-624, republic and canton of geneva, che. international world wide web conferences steering committee. henning wachsmuth, nona naderi, ivan habernal, yufang hou, graeme hirst, iryna gurevych, and benno stein. 2017a. argumentation quality assessment: theory vs. practice. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), pages 250-255, vancouver, canada. association for computational linguistics. henning wachsmuth, nona naderi, yufang hou, yonatan bilu, vinodkumar prabhakaran, tim alberdingk thijm, graeme hirst, and benno stein. 2017b. computational argumentation quality assessment in natural language. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 1, long papers, pages 176-187, valencia, spain. association for computational linguistics. henning wachsmuth, nona naderi, yufang hou, yonatan bilu, vinodkumar prabhakaran, tim alberdingk thijm, graeme hirst, and benno stein. 2017c. dagstuhl-15512-argquality. technical report. henning wachsmuth and till werner. 2020. intrinsic quality assessment of arguments. in proceedings of the 28th international conference on computational linguistics, pages 6739-6745, barcelona, spain (online). international committee on computational linguistics. xuewei wang, weiyan shi, richard kim, yoojung oh, sijia yang, jingwen zhang, and zhou yu. 2019. persuasion for good: towards a personalized persuasive dialogue system for social good. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5635-5649, florence, italy. association for computational linguistics.