Heroes, Villains, and Victims, and GPT -3: Automated Extraction of Character Roles Without Training Data (2022)

Abstract: This paper shows how to use large-scale pretrained language models to extract character roles from narrative texts without domain-specific training data. Queried with a zero-shot question-answering prompt, GPT-3 can identify the hero, villain, and victim in diverse domains: newspaper articles, movie plot summaries, and political speeches.

## introduction what makes a good story? according to some leading theories of narrative (e.g. propp, 1968), a good story has at least three ingredients: a hero, a villain, and a victim. in a classic recipe, the hero defeats the villain to gain justice for the victim. the cohesiveness of these character roles extends even beyond fiction, with social-science research showing the effectiveness of the hero-villain-victim framework in explaining what motivates consumer behavior and political processes @xcite @xcite . motivated by the relevance of narratives to culture and society, a literature in natural language processing (nlp) has arisen to automatically detect characters in texts and annotate their character roles @xcite @xcite . in particular, prior work has used dictionary methods to identify heroes, villains, and victims in texts @xcite . the previous methods have mixed results, motivating the present research. this paper explores the use of large pre-trained language models for the task of character role labeling. operationalizing the problem as a machine reading comprehension task (mrcp), we provide an input document and ask the language model who is the hero (or villain or victim). as illustrated by the prompt in figure 1 , we directly ask "who is the hero" (or villain or victim). thus, we can extract character roles from plain-text documents without in-domain training data. who is the villain in the following text? text: in questioning the use of hydraulic fracturing in new york to help produce natural gas, you do not note that the technology has been employed and continuously improved for more than 50 years and that studies by the environmental protection agency and the ground water protection council have not identified a single instance of groundwater contamination. wells where fracturing is used are specially constructed to protect drinking water sources. regulatory oversight is extensive. the fluids mostly water that are forced into a well to create pressure to fracture rock are pushed back out by the oil and gas flowing upward for safe processing. protecting our water supplies is important, as are reductions in greenhouse gas emissions through use of clean-burning natural gas. banning hydraulic fracturing would be unwarranted and shortsighted, preventing production of large amounts of natural gas that could directly benefit new york consumers for decades and create thousands of good jobs. villain: the villain in this text is the person who is questioning the use of hydraulic fracturing in new york. hero: the hero in this text is the person who wrote it, who is trying to convince the reader that hydraulic fracturing is a safe and effective way to produce natural gas victim: the new york consumers we find that a large pre-trained language model, gpt-3 @xcite , is more effective in labeling these character roles than previous methods, across a diverse set of narrative domains. first, we investigate our method on a labeled corpus of newspaper articles about fracking where the three roles-hero, villain, victim-are manually annotated. in that dataset, our language-model approach is over twice as accurate as an existing baseline using a dictionary approach. second, as an example of possible memorization during pre-training, we report the annotations produced on a selection of disney movie plot summaries. these results comport well with subjective judgment. in our third experiment, we apply the method to a corpus of u.s. state of the union addresses, 2001-2018. matching up the character role annotations with the party affiliation of the president, we explore partisan differences in the framing of heroes, victims, and villains. to make this process feasible, we explore clustering of the gpt-3 output, which produces more legible sets of character assignments. these results demonstrate the promise of the method for empirical research in social science and the digital humanities. these results are of broad interest given the narrative centrality of character archetypes @xcite . they are of more specific interest in the literature analyzing narrative framing in news media and policy discourse @xcite . a robust and efficient computational method to extract character roles in text without training data opens up a wide array of research questions to quantitative analysis. ## related work this paper adds to the work in nlp on automated extraction of character roles from natural language accounts, and in particular the identification of heroes, villains, and victims. the closest paper is @xcite , who similarly focus on the detection of heroes, villains, and victims in news articles and provide a dictionary-based approach which we will use as a baseline. on the broader problem of extracting stereotypical character roles, prior work has explored a variety of methods, including the detection of personas using annotated data combined with feature engineering and regression @xcite ; parsing and lexical matching tools to identify a consistent set of personas (e.g. doctor, nurse, doula) across testimonials about childbirth and then assess the relative power dynamics @xcite ; annotations of german news and social media sentences for villains and rogues and transformer models to machine-tag these roles @xcite ; clustering of structural plot information from folktales @xcite ; and a combination of ner and clustered phrase embeddings to identify repeatedly occurring entities, along with semantic role labeling to identify how entities are connected by actions @xcite . our method does not rely on labeled data, but we employ some of these techniques (e.g., clustering) to support the legibility of our results. the second related literature is treating role extraction as a machine reading comprehension (mrcp) task, which for example has been proposed for semantic role labeling @xcite . most related to our work, @xcite and @xcite interpret event extraction as an mrpc task and leverage pre-trained language models to extract events, producing state-of-the-art results in event extraction and leading us to apply this method for detection of character roles. in principle, any nlp task can be framed as mrcp or question answering (qa) tasks (see e.g. @xcite . interpreting tasks (such as event extraction) as question answering enables us to leverage zero-shot capabilities of pre-trained models. moreover, these methods are not necessarily dependant on domainspecific features, but solely on plain text. given the zero-shot capabilities of mrcp tasks across domains @xcite , it is more likely that this procedure transfers across domains. our proposed task has many similarities with the computational identification of framing @xcite and agenda setting @xcite , as well as with automated bias measurement @xcite . these various tasks all seek to identify the author's written perspective; the same topic can be portrayed differently by different authors, just as the hero or victim might be assigned differently by different authors @xcite . our identification of the hero, villain, and victim provides yet another method to describe the particular viewpoint expressed in a particular text and to draw comparisons between these various viewpoints over large datasets. ## results this section presents the results, with the three empirical domains reported in turn.