Learning and Evaluating Character Representations in Novels (2022)

Abstract: We address the problem of learning fixed-length vector representations of characters in novels. Recent advances in word embeddings have proven successful in learning entity representations from short texts, but fall short on longer documents because they do not capture full book-level information. To overcome the weakness of such text-based embeddings, we propose two novel methods for representing characters: (i) graph neural network-based embeddings from a full corpus-based character network; and (ii) low-dimensional embeddings constructed from the occurrence pattern of characters in each novel. We test the quality of these character embeddings using a new benchmark suite to evaluate character representations, encompassing 12 different tasks. We show that our representation techniques combined with text-based embeddings lead to the best character representations, outperforming text-based embeddings in four tasks. Our dataset and evaluation script will be made publicly available to stimulate additional work in this area.

## introduction high-quality distributed representations of characters (henceforth, character embeddings) play an important role for the computational analysis of narrative texts @xcite @xcite @xcite @xcite . ideally, characters who share similar properties such as job, gender and a relationship to other characters, should possess similar character embeddings even if they are in different stories (e.g. cinderella and juliet, both young women in forbidden romance situations). this paper aims for learning such fixed-length, distributed representations from novels. the core problem of learning character embeddings is how to aggregate and embed the contextual information of characters into distributed rep- figure 1 : t-sne visualization of our character embeddings for ten characters. each character is sampled from more than 24 different books. the proposed method assigns similar representations to each character even though they exist in different books. the proposed method uses no surface form matching. to overcome the weakness of such text-based embeddings, we propose two novel methods to learn character embeddings using document-level information. first, we propose graph-based embeddings, where we build a full corpus-based character network accompanied with full book-level information and then use a graph neural network to learn character embeddings. second, we propose positional embeddings, where we create lowdimensional embeddings from the occurrence pattern of characters in each novel. to evaluate the quality of character embeddings, we construct a new character embedding benchmark (ceb) consisting of 12 different tasks. at training time, one is allowed to learn fixed-length character embeddings from novels. the learned embeddings are then tested if the important properties of characters such as gender can be recovered solely based on them, similar to recent work on probing pretrained language models @xcite . the contribution of this paper can be summarized as follows: our experiments show that the proposed embedding methods combined with text-based embeddings leads to the best character embeddings, outperforming text-based embeddings in six ceb tasks ( §6.3). fig. 1 shows the key result, indicating that similar character representations are assigned to each cluster of character, even though they exist in different books. ## related work there is a growing interest in computational narrative analysis, ranging from analyzing the structure of narratives @xcite @xcite , identifying important events in stories @xcite @xcite to analyzing the relationship between characters in novels @xcite @xcite @xcite @xcite . the most relevant work to ours is @xcite , who apply word2vec @xcite to learn character embeddings from movie scripts. however, they do not use full document-level information such as the author of documents for learning character embeddings. they also experiment on a small-scale dataset-18 movie scripts, while we experiment on 17k novels. @xcite propose two benchmark tasks for character-centric narrative understanding, namely character identification and character description generation. we extend their benchmark by introducing additional 12 characterrelated tasks. character embeddings are closely related to both static word embeddings such as word2vec @xcite and glove @xcite , and contextualized word embeddings such as dynamic entity embeddings @xcite , elmo @xcite and bert @xcite . as discussed in §1, these methods follow the distributional hypothesis @xcite , encoding the local context of words into distributed representations. we intend to complement this weakness by taking book-level context into account in the graph neural network-based embedding methods. the task setting of ceb shares the similar spirit to a recent paradigm on probing pretrained language models @xcite @xcite . the lama dataset @xcite , for example, creates a sentence with blanks, e.g. ___ was born in, and ask language models to predict words in the blanks solely based on the learned model parameters. our benchmark also follows this task setting, where one learns character embeddings on a particular corpus and is asked to recover information solely based on the learned embeddings in 12 different tasks. ## proposed methods while text-based embeddings introduced in §3 can be expected to capture the local context of characters such as gender, they do not take into account full book-level information, such as the author. intuitively, characters from the same book should have more relatively similar embeddings than those from different books, but the text-based embedding methods cannot use this kind of information. to address this weakness, we propose two methods for character embeddings: (i) gr: we build character network across books and then learn character embeddings using graph neural networks ( §4.1); and ## ceb: character embedding benchmark to test the quality of character embeddings, we construct a new benchmark suite of character embeddings, as summarized in ## evaluation 6.1 setup we follow recent work on probing word embeddings, which report that one should employ less expressive classifiers in order to prevent the classifier itself from learning to solve the probe tasks @xcite . at training time, one has access to all books and learns fixed-length character embeddings of each character. at test time, we freeze the learned character embeddings and train task-specific linear classifiers using the learned embeddings as a feature vector. to solve classification tasks, we train a linear classifier that uses learned character embeddings as a feature vector. for pairwise classification, we merge two character embeddings by element-wise multiplication and absolute element-wise difference, i.e. in our experiments, we employ support vector machines @xcite . to solve multiple-choice tasks with context x and characters {c i } 4 ## qualitative analysis to obtain further insights on the learned character embeddings, we visualize rb_ag+gr+pos by using t-sne (van der maaten and hinton, 2008) with default hyperparameters. ## conclusions we have addressed the problem of learning fixedlength, dense character representations from booklength narrative texts. to overcome the weakness of the text-based embeddings, we have proposed graph-based embeddings and positional embeddings. to test the quality of character embeddings, we have also constructed ceb, a novel benchmark suite for evaluating character embeddings, consisting of 12 different tasks. our experiments have demonstrated that the proposed embeddings combined with text-based embeddings lead to the best character embeddings, outperforming text-based embeddings in four tasks. we also showed that character embeddings capture both character-level and book-level information across books, demonstrating the versatility of the techniques we employed.