Narrative Datasets through the Lenses of NLP and HCI (2022)

Abstract: In this short paper, we compare existing value systems and approaches in NLP and HCI for collecting narrative data. Building on these parallel discussions, we shed light on the challenges facing some popular NLP dataset types, which we discuss these in relation to widely-used narrative-based HCI research methods; and we highlight points where NLP methods can broaden qualitative narrative studies. In particular, we point towards contextuality, positionality, dataset size, and open research design as central points of difference and windows for collaboration when studying narratives. Through the use case of narratives, this work contributes to a larger conversation regarding the possibilities for bridging NLP and HCI through speculative mixed-methods.

## introduction human beings are myth-makers; we use stories and imagination to create communities and make sense of the world and our place in it @xcite . narratives are powerful modes of expression, with physical, emotional, and social benefits for both the narrator and the audience @xcite @xcite @xcite . they can also be powerful methods for understanding human behavior and beliefs @xcite . crucially, narratives are situated; they are told and take place in specific social contexts @xcite . natural language processing (nlp) methods can analyze patterns across large datasets, putting stories into context. but narrative datasets in nlp are often removed from the narratives' original contexts (e.g., scraped internet datasets) or are designed without any explicit context or social grounding (e.g., short and artificial stories). in contrast, contextuality is of the utmost importance in qualitative human-computer interaction (hci) approaches to narrative. hci researchers frequently borrow social science methods including surveys, interviews, focus groups, and ethnography for closer investigations that address the diversity of human life and experiences @xcite . qualitative hci methods are often constrained to small sample sizes and susceptible to observer biases, but narrative research and portraiture methods enable creative and holistic engagement with participants' experiences and meaning-making processes @xcite @xcite . these differences make narrative datasets an useful case study when considering tensions and possible collaborations between nlp and hci. both disciplines face challenges in their study and analysis of narrative. while nlp datasets contain a high volume of data points, their labels are constrained to a specific task; in contrast, smaller hci datasets, in particular data collected through qualitative methods such as ethnography and interview, are open-ended in research scope but situated in a particular context. combining these methods can contribute to designing multifaceted datasets while not losing the sight of individual experiences and perspectives in a large volume of stories. in the following sections, we outline dominant framings of narrative and narrative dataset collection in nlp and hci. placing these framings side-by-side highlights a set of tensions-including dataset size, contextuality and positionality, and dataset design-that we finally consider as material for synthesis and mixed methods approaches to narrative data. ## nlp framings of narrative in a recent overview of nlp and humanist approaches to "narrative understanding", @xcite formulate narrativity as a scalar construct rather than a binary class; texts can include some or all narrative features (e.g., narrator, audience, se-quential actions). most nlp narrative tasks focus on building abstractions from narratives by extracting these features and measuring relationships among them. these tasks include extracting narrative structure, like scripts, plot units, or narrative arcs @xcite @xcite @xcite ; modeling connections between characters @xcite @xcite ; generating new stories or summaries @xcite @xcite ; answering questions about the story @xcite , and identifying a correct story ending @xcite . as in other areas of nlp, some narrative research falls into shared tasks, where artificial story datasets are often (though not always) used for testing a particular technical ability of a system. these datasets are sometimes created and often labeled by crowdworkers, and they include brief scenarios not explicitly connected to broader social contexts and narratives. for example, one of the widely used corpora for testing performance on the story cloze task is rocstories dataset which is a collection of 100,000 crowdsourced "five-sentence commonsense stories" @xcite . narrative research in nlp also includes corpusbased studies, where researchers use narrative models to learn about a particular dataset and its authors. corpus-based studies depend on curated datasets that range widely, e.g., fictional works (e.g., novels, fairytales) @xcite , news stories @xcite , biographies @xcite , and personal stories shared orally or on social media @xcite @xcite . these curated datasets were authored in social contexts separate from the nlp research study and are gathered afterwards. curated datasets can also be used for shared tasks, e.g., coreference resolution @xcite , story generation @xcite . there are a small number of naturalistic nlp narrative datasets that lie outside of the above categories. for example, @xcite collected autobiographical stories and retellings of these stories from crowdworkers; this data was shared as part of the research study but was also grounded in the authors' personal experiences. and finally, many modern nlp methods for nar-ratives rely on large, pretrained models @xcite . these models are trained on massive and (mostly) undocumented datasets, containing a mixture of documents from unrelated domains to generalize to other domains and tasks (after finetuning). these pretraining datasets, like the aptlynamed pile @xcite , are too large for full datasheet descriptions @xcite and can encode human biases @xcite . ## hci framings of narrative four key themes are associated with hci's sensibility of narrative: (a) fact (universal/objective truth) (b) experience (global, local, and day-to-day experiences) (c) interpretation (perceived understanding of a and b) (d) fiction (imaginations and cultural value-system based storytelling) @xcite @xcite . hci researchers often ask questions to understand problems better and care about accuracy, legitimacy, and materiality (i.e., why and how certain issues are important) of information. many subdomains of hci refer to and build on users' experiences regarding narratives (feuston and piper, 2019). hci practitioners' and designers' interactions with social settings and/or professional environments frequently influence their experiences in a given time and situation, and so they consciously refrain from making generalizable statements and encourage the mention of contextuality, which is strongly associated with the narratives @xcite . experience-centered research also values empathy to understand the researchers' orientation to the user, and whether they are motivated to empathize with the users' needs and emotional responses @xcite . hci uses both qualitative and quantitative techniques to gather and examine narratives. both quantitative techniques (e.g., surveys and computational analysis of social media data) and qualitative approaches (e.g., interviews, observation, and focus groups) and artistic techniques are frequent in hci. in this paper, we focus on qualitative hci methods for narratives, as they differ from nlp approaches in terms of ontology and epistemology, representing two distinct worldviews @xcite . in surveys, researchers conduct statistical analyses and evaluate the responses based on standard tests and sets of metrics. using qualitative text coding in cases of free-text responses within the survey is also common. more specifically, hci scholars have conducted surveys to investigate people's motivations, goals, and challenges with regard to posting narratives on social media @xcite , as well as factors that influence their decision-making when it comes to online disclosure @xcite these approaches are motivated by understanding a specific set of people in a specific setting and social context. ethnography, observations, interviews, focus group discussions, and story-telling are common methods used by qualitative hci researchers @xcite . other than transcriptionbased qualitative text coding, image, audio and video coding are also used for analysis @xcite . in this regard, not only the texts, images, audio, and videos of the participants are used, but also pictures of the environment, background noise, and even reasons and results of unintended interruptions can contribute to the richness of narratives. hci researchers also use curated social media data and conduct statistical analyses and qualitative text coding on this data to understand certain research problems. hci researchers in some domains also use online ethnographic techniques on social media and collaborative platforms (e.g., gaming, e-commerce) to grow deeper understandings of these communities (mim and ahmed, 2020). many hci researchers and participants use artistic techniques like drawings and installations for addressing research queries @xcite . it is quite common for artists to conduct an autoethnography with themselves, in which their creation of art is their narrative. in many cases, such narratives are symbolic and contextual. finally, many hci researchers adopt mixed methods where they use both qualitative and quantitative approaches to grow a wider and deeper understanding of their research problems. for example, a recent feminist-hci design used a survey to understand the spread of gender harassment on social media and also conducted interviews and focus group discussions for participatory design and user evaluation of unmochon @xcite . ## interdisciplinary tensions volume and depth of narratives. on one hand, nlp techniques can analyze more narratives (and often more normalized) with reduced researcher workload, though at the loss of qualitative detail. on the other hand, qualitative methods in hci offer a deeper understanding of narratives based on a more limited sample size. despite smaller datasets, hci often depends on theoretical saturation of narratives, in which all important themes are represented, while in nlp, even if the number of datapoints is greater, researchers interested in a particular community often rely on an extracted sample decided by someone else (i.e., curated datasets) which might not capture all relevant themes. ## towards a mixed methods approach we argue that mixed methods research, drawing from both nlp and hci, could allow for richer narrative datasets and more holistic understandings of narrative and the social impacts of narrative. this triangulation of methods not only minimizes the biases of researchers and enhances the validity of the findings, but also reveals different dimensions of the phenomenon being investigated @xcite . while prior work has suggested mixed methods in many other nlp contexts (e.g., grounded topic modeling @xcite ), narratives are particularly well-suited because of the strong research interest on both sides and the tensions enumerated above. ## conclusion as both a research tool and as an object of study, narrative datasets have been widely used in both nlp and hci. this short work is not intended to describe all approaches to narrative in these disciplines, nor is it intended to provide solutions to all the described challenges. boundaries between disciplines are fluid, especially in regards to stories shared on social media, where platform design, moderation, and many other hci concepts have shaped the stories studied via computational nlp methods. many different fields (e.g., literary studies) are concerned with narratives; we have constrained our discussion to datasets in nlp and qualitative hci because we see room for crosspollination and conversations. stories can be powerful tools of persuasion and expression, and richer methods that draw from both nlp and hci can raise new questions and open up new directions.