A Straightforward Approach to Narratologically Grounded Character Identification (2020)

Abstract: One of the most fundamental elements of narrative is character: if we are to understand a narrative, we must be able to identify the characters of that narrative. Therefore, character identification is a critical task in narrative natural language understanding. Most prior work has lacked a narratologically grounded definition of character, instead relying on simplified or implicit definitions that do not capture essential distinctions between characters and other referents in narratives. In prior work we proposed a preliminary definition of character that was based in clear narratological principles: a character is an animate entity that is important to the plot. Here we flesh out this concept, demonstrate that it can be reliably annotated (0.78 Cohen’s κ), and provide annotations of 170 narrative texts, drawn from 3 different corpora, containing 1,347 character co-reference chains and 21,999 non-character chains that include 3,937 animate chains. Furthermore, we have shown that a supervised classifier using a simple set of easily computable features can effectively identify these characters (overall F1 of 0.90). A detailed error analysis shows that character identification is first and foremost affected by co-reference quality, and further, that the shorter a chain is the harder it is to effectively identify as a character. We release our code and data for the benefit of other researchers

## introduction characters are some of the most central elements of narratives, and the concept of character plays an important role in most definitions of narrative. as an example, monika fludernik defines a narrative as "a representation of a possible world . . . at whose centre there are one or several protagonists of an anthropomorphic nature . . . who (mostly) perform goal-directed actions . . . " @xcite . this definition clearly states that characters are central to stories per se. therefore, it is natural to assume that character identification is an important step in automatic approaches to story understanding. a number of approaches have been proposed for automatically identifying characters. some approaches, for example, have sought to solve the character identification task using domain-specific ontologies @xcite or reasoning by reference to an existing case base @xcite . others have taken supervised machine learning approaches @xcite , where a classifier is trained over data annotated by people. some approaches, e.g., examining characters' social networks @xcite , take character identification for granted, implementing heuristic-driven approaches over named entities or coreference chains that are not examined for their efficacy. regardless of approach, all prior work of which we are aware has, unfortunately, had a relatively impoverished concept of character, at least from a narratological point of view. in particular, a key aspect of any character is that it contributes to the plot-characters are not just any animate entity in the narrative-and all prior work essentially ignores this point. here we build on a prior proposal of ours @xcite to incorporate this narratologically grounded definition of character into automatic character identification. we first define and operationalize our concept of character, and use that concept to generate annotated data (170 narrative texts drawn from 3 different corpora) with high inter-annotator agreement. then we demonstrate a straightforward supervised machine learning model using seven features that performs quite well on these data. our error analysis reveals several choke points in the performance of the system, most importantly the quality of the co-reference chains. the paper proceeds as follows. first, we discuss the definition of the character as presented by narratologists, contrasting this concept with those used in prior computational work, and describe an operationalized concept of character that can be annotated with high inter-annotator agreement ( §2). we next describe the data to which we applied this concept ( §3), following which we discuss the experimental setup, including the features and classification model ( §4). we present the results ( §5) and analyze the error patterns of the system, discussing various aspects, which leads us to a discussion of future work ( §6). although we have discussed prior work briefly in the introduction, we summarize work related to this study ( §7) before we conclude by enumerating our contributions ( §8). ## data and annotation we annotated characters on 170 texts across three corpora, one with 46 texts (the extended propplearner corpus), the second with 94 texts (a subset of the ontonotes corpus), and the third with 30 texts (a subset of the corpus of english novels). the first two authors double annotated this corpus at the coreference chain level for character, achieving an agreement of 0.78 @xcite . this level of agreement represents substantial overall agreement @xcite . the authors discussed any disagreements and corrected them to generate figure 2 : sample text fragment of pl corpus a gold-standard annotation. our high agreement measures are in accordance with prior work that has shown that dramatis personae (i.e., main characters) can be annotated with high reliability. in particular, @xcite showed that dramatics personae can be annotated with agreements of f 1 > 0.8 and κ > 0.6. because of the high agreement for this annotation task, we single-annotated the remaining two corpora for the sake of efficiency. ontonotes @xcite is a large corpus containing a variety of genres, including news, conversational telephone speech, broadcast news transcripts, talk show transcripts, among others, in english, chinese, and arabic. we extracted 94 english broadcast news transcripts that had gold-standard coreference chain annotations. the first author annotated the coreference chains as to character. despite having clear narrative elements, including characters and events, the news texts have very different goals and textual properties. for example, the plot is only partially represented in a news text, while we have a full plot in many narrative texts. george w. bush wants to modernize social security, as he puts it, by allowing workers to take a small share of their social security, payroll tax, to increase themselves. ## results & discussion the extended propplearner we performed some preprocessing on this corpus, primarily involved in correcting minor errors in the coreference chain annotation. this included removing duplicate coreference chains generated by stanford corenlp, merging coreference chains with the same chain heads, and merging pronouns with the correct chain heads. as expected, we obtained good results using this corpus as the coreference chains are of high quality (i.e., we started with gold standard chains and corrected the small number of errors we found). ontonotes we evaluated our model in three ways on ontonotes data. first, we trained and tested the character model on the complete ontonotes data with all features, achieving an f 1 of 0.66. because the ontonotes coreference chains are not completely clean (containing some duplicates and incorrect chains), we used direct sampling @xcite to select a subset of the chains foot_2 and manually corrected them, and trained and tested the full model over this subset. this achieved an improved f 1 of 0.82, suggesting that the classes are imbalanced because the model voted for majority class only. finally, as the classes are imbalanced (there are many fewer character chains with non-character chains), we performed over-and under-sampling in the same fashion as for the propplearner data. when oversampling only, we achieved an improved performance of 0.91 f 1 . when over-and under-sampling simultaneously, we achieved a performance of 0.92 f 1 . ## error analysis a detailed error analysis of the results revealed problems for the character identification model that depend mainly on the external tools we have used and the quality of the data. first, the character model uses the output of the animacy detector and so if a character was not marked animate, the character model also missed it. conversely, sometimes inanimate chains are incorrectly marked animate, providing an additional opportunity for the character model to err. thus, the character model's performance is bounded by that of the animacy model. this dependency is shown in figure 7 , where the character model performed better when we used the human-annotated animacy labels. second, the quality of coreference chains is critical for the character model. we can see from figure 7 : f 1 vs. chain length of the character identification model on ontonotes for both manually corrected and automatically computer animacy markings. third, identifying the character information of short chains is a challenging task because chain length is one of the most effective features of our character model. in the case of short chains, the model only depends on the chain heads, and if a chain head does not carry much meaningful information, then the model can classify that chain incorrectly. we can see the performance improvement of the character model with increasing chain length from figure 7 . solving this problem is critical, but adding more features that carry semantic information of a chain could be helpful. fourth, the data should be balanced to obtain good performance, which is a common requirement for any machine learning model. the performance improvement is shown in finally, one minor source of error for our model is limited foreign words, which is a data specific problem. the extended propplearner data contains numerous russian character names (e.g., parakha, gornya, shabarsha, etc.) that are not commonly found in english training data for ner systems or linguistic resources (wordnet, conceptnet). as a result, our system was sometimes not able to identify these chains as a person, and that affects the model's performance. to address this problem, we could, for example, improve coverage of the ner gazetteers. ## related work prior work on automatic character identification has relied heavily on statistical techniques and linguistic grammar-based techniques. our work is mainly inspired by @xcite who used a support vector machine (svm) classifier to detect sentient actors in spoken stories. the model compares four different ml classifiers with 83 features (including knowledge features extracted from conceptnet) and reports an f 1 of 0.86. it was found that certain speech features enhanced the results for non-named entities. however, the model focuses on animacy detection rather than character identification. a similar line of work by @xcite implemented a case-based approach using the voz system. apart from linguistic features, the most important features were extracted from wordnet and conceptnet. although they reported a 93.49% accuracy for a subset of the proppian folktales, it does not give a concrete definition of a character. they also proposed a similarity measure (continuous jaccard) that compares the entities from the text and case-base of the voz system. valls-vargas (2015) further incorporated a feedback loop into voz; this iterative approach improves co-reference grouping, but there isn't an improvement in character identification. the most recent work on character identification took a supervised ml approach to classify nouns as characters using 24 different linguistic features, including capitalization and possession-based on freeling and javarap @xcite . out of the different classifiers, classificationviaregression, achieved an f 1 of 0.84; however, it only worked for nouns and ignored pronouns. other approaches have used ner systems and domain-specific gazetteers in addition to other techniques such as graphs and verb analysis. @xcite proposed an eight-stage pipeline for identifying characters by building a graph where each name is represented as a node, and the nodes representing the same character are connected with edges. ner and co-reference resolution are used to populate the graph and connect nodes co-occurring in a chain, respectively. the main heuristics used distinguish between distinct characteristics compares genders (by looking at honorifics) and names. the model achieves an average f 1 of 0.58 on two datasets; however, it is limited to a corpus with characters that can be easily recognized by ner. @xcite proposed a ner-based approach to identify the protagonists in fairy tales using wordnet and verb features. they used the stanford parser to extract ne candidates, which is then filtered by verb analysis. they reported an f 1 of 0.67. in further work, @xcite identified the dominant character in fables using the vaha (verbs associated with human activity) architecture @xcite and taking into account quoted speech, achieving an f 1 of 0.76. the same architecture, when applied to news articles, achieves an f 1 of 0.88 @xcite . @xcite has described a modular tool called novel2graph, which generates visual summaries of narrative text. as part of the first module, characters are detected using stanford's ner, which are further filtered using part-of-speech tagging. character aliases are grouped using the dbscan clustering algorithm and stored in a dictionary. they did not report the performance of their approach. lastly, @xcite demonstrated an ontology-based approach for automated character identification in folktales. they compared indefinite noun phrases with ontology labels, and used the matches to propose potential characters. finally, they applied inference rules, and all occurrences of a particular ontology label were marked as references to the same character. the study reports an f 1 of 0.80. although this approach has the closest implicit definition of a character to ours, the ontology is domain-based and is unlikely to generalize well to other domains. ## contributions this paper makes three contributions. first, we proposed a more appropriate definition of character, contrasting with prior computational works which did not provide a theoretically grounded definition. additionally, we reported our findings of a review of the literature that is helpful to delineate and define the concept of character. second, we double annotated 46 russian folktales and singly annotated 94 ontonotes news texts and 30 english novels (one chapter per novel) for character, generating data that will be useful for the community 5 . finally, we have demonstrated a straightforward supervised machine learning classifier for identifying characters, achieving weighted average of 0.90 f 1 , establishing a new standard for this task.