“Politeness, you simpleton!” retorted [ MASK ]: Masked prediction of literary characters (2021)

Abstract: What is the best way to learn embeddings for entities, and what can be learned from them? We consider this question for the case of literary characters. We address the highly challenging task of guessing, from a sentence in the novel, which character is being talked about, and we probe the embeddings to see what information they encode about their literary characters. We find that when continuously trained, entity embeddings do well at the masked entity prediction task, and that they encode considerable information about the traits and characteristics of the entities.

## introduction neural language models have led to huge improvements across many tasks in the last few years @xcite @xcite ). there has been a steady rise in work on entity representations and how they can be combined with language models, for example @xcite ; @xcite ; @xcite ; @xcite .in this paper, we add to the growing literature on neural representations of entities by considering a particularly challenging case: the representations of entities in very long texts, in particular in novels. intriguingly, @xcite recently tested whether literary characters, when represented through distributional vectors trained on the first half of a novel, can be recognized in the second half, and found the task to be near impossible. we take up that same task, but train character embeddings in a masked character prediction task. we ask the following questions. (a) is it possible to use literary character embeddings to do masked character prediction, that is, to guess from a sentence in a novel which character it mentions? (b) if this task is doable, is it doable only locally, or can we train on the first third of a novel and then guess characters towards the end of the novel? (c) what do the resulting embeddings tell us about the literary characters when we probe them? (d) can the embeddings identify a literary character from a short description of their personality? we find that when continuously trained, entity embeddings do well at the masked entity prediction task, and that they encode considerable information about the traits and characteristics of the entities. modeling semantics for natural language understanding is about modeling entities and events, not words. so we view this work as an initial step in the direction of entity modeling over time. ## related work entities have been increasingly common subjects within nlp research. there has been recent work aimed at inducing both characteristics of entities, such as personalities, physical and mental states, and character traits, as well as distributed entity representations, similar to lexical embeddings. ## data in the current paper, we present a model that is able to construct entity representations for characters in classic literary novels. novels are a compelling environment for this exploration as they feature a relatively small number of entities that appear frequently over a long document. to this end, we turn to the novel aficionados dataset introduced by @xcite . the dataset comprises 62 pieces of classic literature, represented as both their original texts (deemed the originalnovels dataset; these texts are distributed by project gutenberg, which maintains a repository of free ebooks of works no longer protected by copyright), and their english wikipedia summaries (the wikinovels dataset). in order to have sufficient description of as many characters as possible, we only utilize the corpus of original novels in training our representations, as this corpus yields significantly more mentions per character. we utilize the wikipedia summaries as a test set to determine how well our entity representations work outside the domain of the novels themselves. the novels are distributed within the dataset in both their original form and having been preprocessed with booknlp @xcite . booknlp is a natural language processing pipeline that extends from stanford corenlp @xcite and is specifically aimed at scaling to books and other long documents. booknlp includes part of speech tagging, dependency parsing, ner, and supersense tagging. most critical to our application, booknlp provides quotation speaker identification, pronominal coreference resolution, and character name clustering. this means that, in addition to standard anaphoric coreference resolution, booknlp can identify different proper names as character aliases (i.e., jane fairfax, a character from jane austen's emma, is referenced throughout the text not only by her full name, but also jane, miss jane fairfax, and miss fairfax; booknlp is able to recognize this and map all of these aliases to a single, unique character id). concerning the quality of the coreference resolution, bamman et al. report average accuracy of 82.7% in a 10-fold cross-validation experiment on predicting the nearest antecedent for a pronominal anaphor. while the accuracy of character clustering was not evaluated, manual inspection of the data revealed it to be very reliable. ## modeling our hypothesis is that it is possible to represent characters in a novel through an embedding in such a way that it is possible for a model to recognize who is who, or, as we call the task here, is this me?. @xcite found that with an approach that treated characters like common nouns, the related doppelgänger task was not feasible. 4 we hypothesize that if embeddings are learned to best facilitate is this me? prediction, the task will be feasible. we further hypothesize that the resulting embeddings can be found to contain information about the characters. in a way, our approach is similar to recent contextualized language models like bert @xcite in that we, too, train on a masked prediction task, and we, too, hope to find the resulting embeddings to be useful beyond the prediction task itself. ## conclusion in the ideal case, an entity embedding would constitute a compact representation of a person, their character traits and life story, and would allow for inferences about that person, including story arcs in which that person is likely to occur. what is the best way to learn embeddings for entities, and what can be learned from them? we have considered this question for the case of literary characters. we have trained entity embeddings through a masked prediction task, reading a collection of novels from begininng to end. we found that when trained continuously, the entity embeddings did well at the is this me? task: given a target entity embedding, and given a sentence of the novel with a masked entity mention, is this a mention of the target entity? the is this me? task becomes much harder when we "skip ahead", training only on the first third of a novel and then evaluating on the middle and end. the task also becomes much harder when applied to wikipedia summaries of novels, which show a marked domain difference from the novels themselves. probing the entity embeddings that result from the masked prediction task, we find that they encode a good amount of information about the novel target candidate result sentence emma emma emma + [mask] the protagonist of the story is beautiful high spirited intelligent and lightly spoiled young woman from the landed gentry. great expectations estella estella -she hates all men and plots to wreak twisted revenge by teaching [mask] to torment and spurn men, including pip who loves her. a tale of two cities miss pross miss pross + [mask] permanently loses her hearing when the fatal pistol shot goes off during her climactic fight with madame defarge. a tale of two cities lucy manette lucy manette -she is the golden thread after whom book the second is named so called because [mask] holds her father and her family lives together and because of her blond hair like her mother. looking ahead, the training regime and trained embeddings allow for many further analyses. we would like to probe further into the "skipping ahead" to better understand why it is so difficult. intuitively, characters that undergo more development across the length of a novel should be more difficult to predict. it is not clear to what extent this is the case with the current model; this needs further study. in addition, we would like to model the change and development of characters more explicitly, for example by representing them as a trajectory over time rather than a single embedding. it would also be beneficial to further explore the ways in which character traits are implicitly present within entity representations learned from the is this me? task. while we attempted to probe this superficially via the evaluation on out-of-domain wikipedia data, this data does not offer the annotation that would be necessary to perform a more in-depth analysis we would also like to extend the model by including additional relevant input. at the moment, we essentially ask the model to bootstrap entity representations from scratch, using only the contextualized sentence representations produced by bert and the current entity representations as input. other useful information such as semantic relations (retrievable via dependency parse) may be useful. we may also consider the kind of events and modifiers that a given entity participates in to be able to exploit patterns across character archetypes (similar to @xcite ). we are also looking to extend the model to directly model relations between characters as relations between entity embeddings, to see whether this would help performance and to see to what extent the interpersonal relations of characters would be encoded in their embeddings. overall, we find the results presented in the current paper to be promising as a first step towards natural language understanding systems that use neural models of entities over time. as we have outlined here, however, there is still much work to be done.