F anfiction NLP : A Text Processing Pipeline for Fanfiction (2021)

Abstract: Fanfiction presents an opportunity as a data source for research in NLP, education, and social science. However, answering specific research questions with this data is difficult, since fanfiction contains more diverse writing styles than formal fiction. We present a text processing pipeline for fanfiction, with a focus on identifying text associated with characters. The pipeline includes modules for character identification and coreference, as well as the attribution of quotes and narration to those characters. Additionally, the pipeline contains a novel approach to character coreference that uses knowledge from quote attribution to resolve pronouns within quotes. For each module, we evaluate the effectiveness of various approaches on 10 annotated fanfiction stories. This pipeline outperforms tools developed for formal fiction on the tasks of character coreference and quote attribution

## introduction a growing number of natural language processing tools and approaches have been developed for fiction @xcite @xcite . these tools generally focus on published literary works, such as collections of novels. we present an nlp pipeline for processing fanfiction, amateur writing from fans of tv shows, movies, books, games, and comics. fanfiction writers creatively change and expand on plots, settings, and characters from original media, an example of "participatory culture" @xcite . the community of fanfiction readers and writers, now largely online, has been studied for its mentorship and support for writers @xcite and for the broad representation of lgbtq+ characters and relationships in fan-written stories @xcite . fanfiction presents an opportunity as * denotes equal contribution. a data source for research in a variety of fields, from those studying learning in online communities to social science analysis of how community norms develop in an lgbtq-friendly environment. for nlp researchers, fanfiction provides a large source of literary text with metadata, and has already been used in applications such as authorship attribution @xcite and character relationship classification @xcite . there is an vast amount of fanfiction in online archives. as of march 2021, over 7 million stories were hosted on just one fanfiction website, archive of our own, and there exist other online archives of similar or even larger sizes @xcite . we present a pipeline that enables structured insight into this vast amount of text by identifying sets of characters in fanfiction stories and attributing narration and quotes to these characters. knowing who the characters are and what they do and say is essential for understanding story structure @xcite . such processing is also useful for researchers in the humanities and social sciences investigating identification with characters and the representation of characters of diverse genders, sexualities, and ethnicities @xcite @xcite . the presented pipeline, which extracts text related to characters in fanfiction, can assist researchers building nlp tools for literary domains, as well those analyzing characterization in fields such as digital humanities. for example, the pipeline could be used to explore how characters are voiced and described differently when cast in queer versus straight relationships. the presented pipeline contains three main modules: character coreference resolution, quote attribution, and extraction of "assertions", narration that relates to particular characters. we incorporate new and existing methods into the pipeline that perform well on an annotated set of 10 fanfiction stories. this includes a novel method using fanfiction is written by amateur writers of all ages and education levels worldwide, so it contains much more variety in style and genre than formal fiction. it is not immediately clear that techniques for coreference resolution or quote attribution that perform well on news data or formal fiction will be effective in the informal domain of fanfiction. we demonstrate that this pipeline outperforms existing tools designed for formal fiction on the tasks of character coreference resolution and quote attribution @xcite . contributions. we contribute a fanfiction processing pipeline that outperforms prior work designed for formal fiction. the pipeline includes novel interleaving of coreference and quote attribution to improve the resolution of first-and secondperson pronouns within quotes in narrative text. we also introduce an evaluation dataset of 10 fanfiction stories with annotations for character coreference, as well as for quote detection and attribution. ## fanfiction and nlp data from fanfiction has been used in nlp research for a variety of tasks, including authorship attribution @xcite , action prediction @xcite , finegrained entity typing @xcite , and tracing the sources of derivative texts @xcite . computational work focusing on characterization in fanfiction includes the work of @xcite , who found that fanfiction writers are more likely to emphasize female and secondary characters. using data from wattpad, a platform that includes fanfiction along with original fiction, @xcite find that portrayals of gendered characters generally align with mainstream stereotypes. we are not aware of any text processing system for fanfiction specifically, though booknlp @xcite is commonly used as an nlp system for formal fiction. we evaluate our pipeline's approaches to character coreference resolution and quote attribution against booknlp, as well as against other task-specific approaches, on an evaluation dataset of fanfiction. ## fanfiction processing pipeline we introduce a publicly available pipeline for processing fanfiction. 1 this pipeline is a commandline tool developed in python. from the text of a fanfiction story, the pipeline extracts a list of characters, each mention of a character, as well as what each character does and says (figure 1 ). more specifically, the pipeline first performs character coreference resolution, extracting character mentions and attributing them to character clusters with a single standardized character name (section 3.1). after coreference, the pipeline outputs quotes uttered by each character using a sieve-based approach from @xcite (section 3.2). these quote attribution results are optionally used to aid the resolution of first-and second-person pronouns within quotes to improve coreference output (section 3.3). in parallel with quote attribution, the pipeline extracts "assertions", topically coherent segments of text that mention a character (section 3.4). ## fanfiction evaluation dataset to evaluate our pipeline, we annotate a dataset of 10 publicly available fanfiction stories for all mentions of characters and quotes attributed to these characters, which is similar in size to the test set used in litbank @xcite . we select these stories from archive of our own foot_1 , a large fanfiction archive that is maintained and operated by a fan-centered non-profit organization, the organization for transformative works @xcite . to capture a representative range of fanfiction, we choose one story from each of the 10 most popular fandoms on archive of our own when we collected data in 2018 ( these stories illustrate the expanded set of challenges and variety in fanfiction. in one story, all of the characters meet clones of themselves as male if they are female, or female if they are male. this is a variation on the practice of "genderswapping" characters in fanfiction @xcite . coreference systems can struggle to keep up with characters with the same name but different genders. another story in our test set is a genre of fanfiction called "songfic", which intersperses song lyrics into the narrative. these song lyrics often contain pronouns such as i and you that do not refer to any character. for quote attribution, challenges in the test set include a variation of quotation marks, sometimes used inconsistently. there is also great variation in the number of indirect quotes without clear quota-tives such as "she said". this can be a source of ambiguity in published fiction as well, but we find a large variety of styles in fanfiction. one fanfiction story in our evaluation dataset, for example, contains many implicit quotes in conversations among three or more characters, which can be difficult for quote attribution. annotation details and inter-annotator agreement for this evaluation dataset are described below. an overview of inter-annotator agreement is provided in ## pipeline evaluation we evaluate the pipeline against booknlp, as well as other state-of-the-art approaches for coreference resolution and quote attribution. ## ethics though most online fanfiction is publicly available, researchers must consider how users themselves view the reach of their content @xcite . anonymity and privacy are core values of fanfiction communities; this is especially important since many participants identify as lgbtq+ @xcite . we informed archive of our own, with our contact information, when scraping fanfiction and modified fanfiction examples given in this paper for privacy. we urge researchers who may use the fanfiction pipeline we present to consider how their work engages with fanfiction readers and writers, and to honor the creativity and privacy of the community and individuals behind this "data". ## conclusion we present a text processing pipeline for the domain of fanfiction, stories that are written by fans and inspired by original media. large archives of fanfiction are available online and present opportunities for researchers interested in community writing practices, narrative structure, fan culture, and online communities. the presented text processing pipeline allows researchers to extract and cluster mentions of characters from fanfiction stories, along with what each character does (assertions) and says (quotes). we assemble state-of-the-art nlp approaches for each module of this processing pipeline and evaluate them on an annotated test set, outperforming a pipeline developed for formal fiction on character coreference and quote attribution. we also present improvements in character coreference with a post-processing step that uses information from quote attribution to resolve first-and second-person pronouns within quotes. our hope is that this pipeline will be a step toward enabling structured analysis of the text of fanfiction stories, which contain more variety than published, formal fiction. the pipeline could also be applied to other formal or informal narratives outside of fanfiction, though we have not evaluated it in other domains.