Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark (2024)

Abstract: Several systems have been developed to extract information about characters to aid computational analysis of English literature. We propose character similarity grouping as a holistic evaluation task for these pipelines. We present AustenAlike, a benchmark suite of character similarities in Jane Austen’s novels. Our benchmark draws on three notions of character similarity: a structurally defined notion of similarity; a socially defined notion of similarity; and an expert defined set extracted from literary criticism. We use AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP. We build character representations from four kinds of features and compare them to the three AustenAlike benchmarks and to GPT-4 similarity rankings. We find that though computational representations capture some broad similarities based on shared social and narrative roles, the expert pairings in our third benchmark are challenging for all systems, highlighting the subtler aspects of similarity noted by human readers.

## introduction there is growing interest in using computational techniques to analyze works of literary fiction. several systems have been developed to automatically extract information about characters from english literary text @xcite . in this paper, we explore character similarity as a holistic evaluation task for literary pipelines. we use character similarity to explore the information about characters that is captured by the different kinds of features these pipelines extract: their events, utterances, and attributes. because characters can be similar along multiple axes, we construct a multi-part benchmark, austenalike, that uses three different notions of character similarity to group characters in jane austen's novels. the first is a structurally defined james morland from northanger abbey sibling to heroine and single 20-year-old male clergy with income of £400/year social pairings: charles hayter, edward ferrars, robert martin narrative role pairings: isabella knightley, john dashwood, margaret dashwood, susan price, william price, elizabeth elliot, mary musgrove, jane bennet, mary bennet, kitty bennet, lydia bennet expert pairings: edmund bertram, edward ferrars, henry tilney, philip elton figure 1 : example character from austenalike notion of similarity to group austen's characters: characters are similar if they fill similar narrative roles. the second is a socially defined notion of similarity: characters are similar if they share demographic features. the final benchmark takes a wisdom-of-the-crowd approach, but with an expert crowd: we extract comparisons of characters from four decades of persuasions, a journal dedicated to the analysis of austen's work. figure 1 shows an example of how these three views of character similarity can lead to different comparisons. we use austenalike to explore how much information about characters is captured by the different kinds of features that literary pipelines extract. we extract character events, quotes, modifiers, and assertions using the booknlp @xcite and @xcite pipelines. we build character representations using contextualized embeddings of these features, and compare how well these representations align with the three sets of character groupings in the austenalike benchmarks. we also compare a non-feature-based approach by extracting similarity judgments from chatgpt. our results show that event-and assertion-based representations capture more information about character similarity than quote-based representations. overall, however, we show that though computational representations capture some broad social and narratological similarities, there is a wide gap between the similarities they capture and the more nuanced similarities highlighted in our wisdom-of-the-expert-crowd benchmark. the best feature-based representations exhibit only medium correlations with expert rankings of character similarity, and gpt-4 lists the expert-identified most similar character in a top ten similarity list only half of the time. austenalike illustrates how much work remains to achieve nuanced computational representations of literary characters. ## related work there is a growing interest in applying computational methods to analyze literary fiction, both in analyses of large collections (distant reading @xcite ) @xcite @xcite and of individual authors and works @xcite @xcite . though these projects range in scope, they share a foundation of feature extraction: literary evidence must be identified before it can be interpreted. to facilitate computational analysis, a number of pipelines for extracting features from literary text have been developed @xcite @xcite . in this paper, we focus specifically on features related to literary characters. ## a three-part benchmark for evaluating character similarity character similarity is a multi-faceted concept. two characters may play the same role in a narrative or follow the same plot trajectory. they may have similar personality traits or fill similar social roles. austenalike uses a multi-faceted approach to character similarity that explores three aspects of literary characterhood: shared narrative roles, shared social characteristics, and pairwise comparisons from expert analysis. 1 the austenalike benchmark focuses on characters from the six jane austen novels published within or immediately after her lifetime: sense and sensibility, pride and prejudice, mansfield park, emma, persuasion, and northanger abbey. we include all named characters who speak more than once, except those who die in the first chapter. ## building computational representations of character we build computational representations of character from the output of two literary pipelines. we construct representations out of the features they extract: for booknlp, events, quotes, and modifiers; for fanfictionnlp, quotes and assertions. ## evaluating character similarity we have proposed three benchmarks that capture different aspects of character similarity. for the social and narrative roles benchmarks, we are interested in the similarity between characters in the same groupings. for the expert benchmark, we are interested in whether characters are most similar to those they are paired with by experts. ## results we explore how well computational representations of character capture aspects of character similarity using the three-part austenalike benchmark. ## conclusion we present austenalike, a three-part jane austen benchmark for evaluating multiple aspects of character similarity: narrative role similarity, social similarity, and expert judgments of character similarity drawn from prior scholarly analysis. we use austenalike to evaluate five computational representations of character built atop features extracted by pipelines for analyzing english literature. we find that event-and assertion-based representations tend to capture character similarity better than quote-based representations. overall, however, our results show how much work still remains to be done to improve computational representations of character: feature-based representations and gpt-4 alike struggle to place the expertidentified most similar character in their top ten lists of character similarity. we hope that by providing a multi-faceted benchmark with expert judgments, austenalike can guide future work on computational representations of character.