S tory ARG : a corpus of narratives and personal experiences in argumentative texts (2023)

Abstract: Humans are storytellers, even in communication scenarios which are assumed to be more rationality-oriented, such as argumentation. Indeed, supporting arguments with narratives or personal experiences (henceforth, stories) is a very natural thing to do – and yet, this phenomenon is largely unexplored in computational argumentation. Which role do stories play in an argument? Do they make the argument more effective? What are their narrative properties? To address these questions, we collected and annotated StoryARG, a dataset sampled from well-established corpora in computational argumentation (ChangeMyView and RegulationRoom), and the Social Sciences (Europolis), as well as comments to New York Times articles. StoryARG contains 2451 textual spans annotated at two levels. At the argumentative level, we annotate the function of the story (e.g., clarification, disclosure of harm, search for a solution, establishing speaker’s authority), as well as its impact on the effectiveness of the argument and its emotional load. At the level of narrative properties, we annotate whether the story has a plot-like development, is factual or hypothetical, and who the protagonist is. What makes a story effective in an argument? Our analysis of the annotations in StoryARG uncover a positive impact on effectiveness for stories which illustrate a solution to a problem, and in general, annotator-specific preferences that we investigate with regression analysis.

## introduction narratives and argumentation are deeply related: this is a well established observation in psychology and social science. although stories per se express something individual and concrete, they allow people to draw conclusions about matters of general interest, for example, social problems and injustices -something general is expressed through something concrete and can thus often be better understood @xcite . in addition, stories have a unique effect on the recipient(s) (e.g., the other participants in a discussion): they offer room for interpretation, therefore encourage reflection, and precisely because they are individual, the recipient is required to take on the perspective of the other @xcite , a quality that is becoming increasingly important in times of growing political polarization. on the side of computational argumentation research, however, the role of narratives and personal experiences has barely been investigated, since in argumentative contexts they are often regarded as rather second-class (not logical, not verifiable). with our paper, the resource it presents, and the analysis we carry out, we aim at building a finegrained empirical picture of this phenomenon, crucial both in terms of its persuasiveness within an argument and its contribution to interpersonal communication. while there are existing datasets that make it possible to develop classification methods to detect stories in argumentative texts @xcite @xcite the next step to be made is to understand these stories in terms of both their argumentative function and narrative properties. this paper presents storyarg, a novel dataset that can be used to get a finer-grained picture of this phenomenon, helping filling an important gap in the study of "everyday" argumentation. storyarg has several novel features. first, it is based on a compilation of datasets that are well-established in computational argumentation (changemyview @xcite , regulation-room @xcite ) and social sciences (europolis @xcite ). this will allow us and others to exploit already available annotations to explore further research questions. additionally, we included a newly collected sample: user comments to new york times articles on veganism. second, our interdisciplinary annotation schema is unique in that it integrates both the argumentative and the narrative perspective. the argumentative layers we annotate are related to the argumentative function of the story (disclosure of harm, search for a solution, clarification, establishing speakers' authority) as well as to the effectiveness of the argument, its stance and main claim. additionally, it has been shown that emotions play a role in the persuasiveness of a story as they enable the listener to better empathize with it @xcite . at the narrative level, we annotate whether the story has a clear plot or not, who is the protagonist (an individual, a group), whether the story is hypothetical or factual, as well as the narrative perspective (first hand vs. second hand) as a result, storyarg contains 9 annotation layers, is annotated by 4 annotators and consists of a total of 2,451 instances in the context of 507 documents over the four corpora. do stories make an argument stronger? the annotations in storyarg allow us to tackle a crucial question in the social sciences in the context of deliberative theory @xcite : i.e. how do narratives affect the quality of a contribution? our analysis shows that stories that illustrate a solution to a problem are perceived as more effective. annotator-specific preferences highight the subjectivity of the task: in the spirit of recent developments in perspectivism in nlp @xcite we don't disregard them but integrate them in our regression analysis. in argument theory and argument mining, narratives and experiences are most frequently analyzed when serving as premises and have been analyzed as part of different argument schemes @xcite . the most common schema is the argument of analogy @xcite (the narrative or experience serves as an example from which a general conclusion can be derived) and the argument from authority / expert @xcite (a statement is valid because this person is an expert in a certain field of competence). these schemes also serve as the basis for existing work in computational linguistics that develop different annotation frameworks for argumentative texts in order to automatically classify types of claims and premises @xcite , study different flows of evidence types @xcite or their effectiveness as a persuasion strategy @xcite . depending on the research focus, the target phenomenon is termed and defined differently, for example, as anecdote @xcite , testimony @xcite @xcite , experiential knowledge @xcite or personal story @xcite . this includes personal accounts, concrete events but also personal experiences with no narrative structure. social science while this type of premise is studied in linguistics and computational linguistics more in terms of formal and structural properties, social science focuses on the role of narratives in the context of communication or deliberation with other people. the different types of narratives in arguments are often summarized under the more general term 'storytelling'. this phenomenon is considered, for example, in deliberation theory as an alternative form of reasoning and both positive and negative effects on the success of the deliberation process are examined here @xcite . apart from the fact that storytelling, as a simpler form of reasoning, allows all kinds of groups and social classes to access and participate in discourses, it plays a key role regardless of social background, as it takes on important cognitive and social functions, such as individual and collective identity formation, sharing socio-cultural knowledge, empathy and perspective-taking and guiding decision processes @xcite @xcite . the existing literature shows that there is no prevailing definition of arguments and narratives. the phenomenon includes complex personal experiences, as well as micro-stories, everyday narratives, anecdotes, and historical events. narratives can be fully fleshed out (plot-like structure) or fragmented and implied. with this work, we propose a unified definition of narrative in argumentation which includes all the above mentioned variants. we do not limit ourselves to one type of narrative but rather annotate certain characteristics of the diverse types of narratives we find in argumentation. these characteristics allow for the grouping of the stories according to certain criteria. thus, future research contributions can use the dataset together with the criteria to apply their desired definition of narratives in a specific context. with respect to the functions of narratives in argumentation, our annotation is based on the social science framework proposed by maia et al. (2020), which we discuss in detail in section 4.3. we deliberately choose an interdisciplinary perspective here, as this has not yet been sufficiently explored with respect to the phenomenon in computational linguistics. ## corpus construction we select sources from argument mining and social science that have already been annotated with some notion of storytelling, and add a sample of user comments about a controversial topic: veganism. ## annotation in what follows, we talk the reader through the annotation layers. the full annotation guidelines can be found in appendix section c, along with more annotation layer labels property document level stance clear, unclear argumentative claim free text argumentative span level experience type story, experiential knowledge narrative protagonist1 individual, group, non-human narrative protagonist2 individual, group, non-human narrative proximity first-hand, second-hand, other narrative hypothetical true, false narrative argumentative function clarification, disclosure of harm, search for solu-tion, establish background argumentative effectiveness low, medium, high argumentative emotional appeal low, medium, high ## quantitative properties experiences spans and types out of 507 documents, 483 documents contain at least one experience and the annotators extracted a total of 2,451 experiences out of which 2,385 are connected to clear argumentative position. for most of the documents, the number of extracted spans for each document ranges between 1 and 5 spans the majority of the spans range between 20 and 500 tokens; again there is a long tail of spans that deviate from this range and are very long (more than 1000 tokens). as expected, stories have more tokens on average (@xmath0. comparing the different sub-corpora we can see that cmv and peanuts contain the highest number of spans, while europolis, nyt comments and cpcp contain a less spans (figure 1 ; cmv also has the longest average token length and nyt the shortest). on top of that we can observe that stories are less frequent than experiential knowledge. proximity and protagonist while more personal experiences (first-or second-hand) often talk about individuals (first-@xmath1, second-@xmath2, stories whose narrative per-claim marked span properties ban of serving peanuts if allergic people are on the flight we have several times had issues with airlines not caring about the allergies. one continental flight attendant once insisted on that it was a rule that she had to serve peanuts to us and everyone around us even though we had informed them before hand that we had peanut allergies. i believe continental since has stopped serving peanuts, but it was very unpleasant and we had to give benadryl to our then 2 year old as he started wheezing. it was not until he was wheezing that the flight attendant was kind enough to inform the captain and take back the peanuts! spective is more general or from an observer's point of view (other) more often talk about groups or institutions (@xmath3, non-@xmath4. thus, the experiences can be arranged on a scale between personal (here individuals rather play the main role) and general (a collective or certain, social circumstances are in the foreground). we can also observe differences with respect to proximity and protagonists when comparing the different sub-corpora. if we compare the distribution of narrative proximity across the sub-corpora we can see that first-hand stories are most frequent (76%) and second-hand stories are quite rare (10%) (more cases can be found in peanuts (15%). for europolis, on the other hand, most experiences are reported from an external perspective (@xmath5, first-@xmath6. we can observe a similar trend when we compare the main characters of the stories. the individual plays a more important role in cmv (57%), cdcp (52%) and peanuts (66%) while for europolis and the nyt comments stories are more often about collectives, such as groups or institutions (europolis: @xmath7, non-@xmath8; nyt comments: @xmath9, non @xmath10. on the one hand, this makes sense, since the topics of immigration and veganism are political topics of interest to society as a whole, whereas the other discussions tend to involve everyday topics with less social relevance. on the other hand, the setup of the discussions also plays a role: the discussion in europolis is deliberative and conducted on a european level, therefore the participants see themselves as representatives of a larger collective (their country) and consequently more often take a broader perspective. argumentative function regarding the distribution of argumentative functions, we find that the amount of establish background and clarification is a lot higher than the more specific types disclosure of harm and search for solution (@xmath11, @xmath12, @xmath13, @xmath14. comparing the two more specific functions, nyt comments shows a lot more solution-oriented experiences than disclosures of harms (15% vs. 3%). in this discourse, people often share positive experiences with the vegan lifestyle to illustrate the benefits of this on everyday life. there are also more solution-oriented experiences in europolis (11%) -a corpus with a strong deliberative focus in which moderators facilitate productive and solution-oriented discussion. in peanuts and cdcp many experiences about harm are shared (12% and 21%, respectively), for example, by allergy sufferers who feel unfairly treated and disadvantaged and who want to trigger empathy and understanding in the other discourse participants by highlighting their suffering, to achieve a change in the regulations. ## analysis: what makes experiences in order to investigate which characteristics of experiences influence the annotators' perceived effectiveness of the experience in the argument, we perform a regression analysis on our dataset. which types of experiences are perceived as more or less effective? the regression model contains effectiveness on a continuous scale (1 -3, from low to high) as a dependent variable (dv) and the annotated properties (narrative and argumentative) of the experiences as independent variables (iv). each annotated instance with a clear argumentative position represents a data point, we drop all instances with missing values in any of the annotation layers or an unclear stance (@xmath15,367). besides the annotated properties we add the number of tokens as a continuous iv and convert the labels of emotional appeal to a continuous scale (1 -3). since we saw that the perceived effectiveness of experiences is subjective, we add the annotator as an iv to the model. this allows us to uncover general trends but also annotator-specific differences. the following formula describes the full model with 8 ivs and all two-way interactions. foot_0 . effectiveness ∼ (experiencetype + argfunction + emotionalappeal + hypothetical + proximity + protagonist+ tokens + annotator)ˆ2 we perform a step-wise model selection foot_1 to reduce the complexity of the model. we estimate the best fit in terms of adjusted r 2 (proportion of explained variance). the final model explains 31% of the variance. the most explanatory variables are the annotator (13.41%), the experience type (3.42%), the argumentative function (4.38%), the number of tokens (2.7%) and emotional appeal (1.4%). which properties have the greatest effect on the perceived effectiveness? the forest plot in figure 3 illustrates which values of the corresponding properties have the greatest impact on the effectiveness. in general experiences with stronger narrative character (experi-@xmath16. these findings are consistent with findings from psychology: stories are particularly compelling when they 'transport' the listener to another world (narrative transportation), or in other words, when they stimulate a stronger narrative engagement @xcite . for the categorical ivs protagonist and argumentative function we can compare all values with the effect plots in appendix figure 3 . 4 we can observe that predicted effectiveness increases with specificity of argumentative function (increase from clarification to background to harm to solution), and search for solution predicts the highest effectiveness indicating a preference for solutionoriented experiences. with regard to the protagonists, the effectiveness increases from individual to general. experiences in which a collective is the focus (group or country / institution) are perceived as more effective. ## conclusion the role played by personal narratives in argumentation is widely acknowledged in the social sciences but so far not investigated in computational argumentation. storyarg, the resource released in this paper, and the analysis we conduct is the first step towards filling this gap. the interdisciplinary annotation scheme of stor-yarg makes it unique in the landscape of research on computational argumentation: we integrate argumentative layers and narrative layers, thus uncovering interaction between the different facets of the phenomenon (e.g., positive impact on effectiveness for longer stories with a plot-like development). crucially, the annotator-specific preferences uncovered in our annotations place our work in the broader debate on perspectivism and the importance of looking at disagreements as a resource and not as a bug. storyarg is sampled from existing reference corpora (plus a novel, out-of-domain sample), making the year-long effort invested in its annotation sustainable as our annotations can be compared with available ones for the same datasets. the dataset and annotation guidelines can be accessed via https://github.com/blubberli/ storyarg .