Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus. (2020)

Abstract: Current story writing or story editing systems rely on human judgments of story quality for evaluating performance, often ignoring the subjectivity in ratings. We analyze the effect of author and reader characteristics and story writing setup on the quality of stories in a short storytelling task. To study this effect, we create and release STORIESINTHEWILD, containing 1,630 stories collected on a volunteer-based crowdsourcing platform. Each story is rated by three different readers, and comes paired with the author’s and reader’s age, gender, and personality. Our findings show significant effects of authors’ and readers’ identities, as well as writing setup, on story writing and ratings. Notably, compared to younger readers, readers age 45 and older consider stories significantly less creative and less entertaining. Readers also prefer stories written all at once, rather than in chunks, finding them more coherent and creative. We also observe linguistic differences associated with authors’ demographics (e.g., older authors wrote more vivid and emotional stories). Our findings suggest that reader and writer demographics, as well as writing setup, should be accounted for in story writing evaluations.

## introduction reading or writing a story is an inherently subjective task that depends on the experiences and identity of the author, those of the reader, and the structure of the writing process itself @xcite @xcite . despite this subjectivity, many natural language processing tasks treat human judgments of story quality as the gold standard for evaluating systems that generate or revise text. in creative applications, such as machine-inthe-loop story writing systems @xcite , it is important to understand sources of variation in judgments if we hope to have reliable, reproducible estimates of quality. in this work, we investigate how an author's and reader's identity, as well as overall writing setup, influence how stories are written and rated. we introduce and release storiesinthewild, foot_0 containing 1,630 short stories written on a volunteerbased crowdsourcing platform, paired with author demographics and personality information. for each story, we obtain three sets of ratings from third-party evaluators, along with their demographics and personality. our findings confirm that author identity, reader identity, and writing setup affect story writing and rating in storiesinthewild. notably, people in general preferred stories written in one chunk rather than broken up into multiple stages. raters age 45 and over generally rated stories as less creative, more confusing, and liked them less compared to raters under age 45. additionally, we find that, in our corpus, men were more likely than women to write about female characters and their social interactions, and compared to younger authors, older authors wrote more vivid and emotional stories. we also find evidence of reader and author personality, and their interaction, influencing ratings of story creativity. our new dataset and results are first steps in analyzing how writing setup and author and reader traits can influence ratings of story quality, and suggest that these characteristics should be accounted for in human evaluations of story quality. ## background and research questions to guide our study, we craft several research questions informed by existing literature on story writing and the relationship between author identity and language, outlined below. rq1 how are author gender, age, and personality traits associated with language variation in stories? a wealth of work has shown an association between an author's mental states and their language patterns. variation in pronoun usage, topic choices, and narrative complexity correlates strongly with the author's age and gender @xcite and moderately with their personality @xcite . we aim to confirm these differences in a prompted storytelling setting, since most work has focused on self-narratives (e.g., diaries and social media posts; pennebaker and @xcite @xcite , with the exception of the essays studied by @xcite . rq2 how are rater gender, age, and personality traits associated with variation in story quality ratings? ratings of stories are often only used to evaluate a story writing system's output (e.g., @xcite or to develop automatic evaluation metrics (e.g., @xcite , ignoring the rater's identity. however, prior work has shown differences in crowdsourcing worker's behavior or annotations based on task framing @xcite @xcite or the annotator's own identity or experiences @xcite . we seek to confirm and characterize these differences in our story rating task. as a follow-up to rq2, we also investigate the interaction between author and rater demographics on story ratings. ## storiesinthewild collection we introduce and release storiesinthewild, containing 1,630 short stories ( §3.1) paired with author demographics and personality information. 2 we pair these stories with third-party rat- ## analyses we investigate the effects of author and rater characteristics on the story's language and ratings. unless otherwise specified, we only consider the male and female gender labels foot_5 and use a continuous representation of age and personality. we also explore the impact the writing setup-whether authors wrote stories all at once or in sequential chunks-has on story ratings. 7 note that our findings are simply measuring associations between aggregate categories (e.g., number of pronouns used, authors over age 45) and should not be interpreted as applying to individual data points with specific contexts. ## conclusion in this study we find that differences in author characteristics are associated with linguistic differences in stories and that rater characteristics are associated with differences in ratings. for authors, men were more likely than women to write about female characters and their social interactions, and compared to younger authors, older authors wrote more vivid and emotional stories. raters preferred stories written all at once rather than broken up into multiple stages, and raters age 45 and older rate stories significantly lower than raters under age 45. we release our dataset, sto-riesinthewild, containing 1,630 stories with quality ratings and anonymized author and rater demographics. our results suggest that author and reader characteristics (e.g., demographics, personality) could explain variations in story writing evaluations. while work has shown that some study designs are more robust against this variation, (e.g., by ranking instead of rating yannakakis and martínez, 2015), rater differences could still lead to variation in annotations. we recommend that evaluations include some ability to collect characteristics, such as a short demographics and personality questionnaire, in order to assess any influence of these variables. furthermore, future work could explore alternative ways of collecting author and reader characteristics during evaluations. while demographic questionnaires are common and short (e.g., to collect gender and age would require two questions), full personality questionnaires are time consuming, asking multiple questions for each characteristic. study designers could instead use reduced questionnaires, such as the ten item personality inventory @xcite . alternatively, focusing on fewer, more highly trained raters-that represent a diverse set of demographics and personality-could reduce the cost of collecting many rater demographics. finally, future work should investigate whether annotator variance might be better captured with psychological factors related to reading (e.g., propensity for liking long sentences or fiction) rather than stable traits such as personality or demographics. our results that author personality and gender were associated with topic selection and story writing also suggest that studies could leverage the behavior of participants to predict personality characteristics. while these results are not yet strong enough to provide robust measures of personality or demographics, future studies could explore how to leverage these associations to predict author characteristics in story writing or other writing evaluations rather than relying on questionnaires.