TVS how G uess: Character Comprehension in Stories as Speaker Guessing (2022)

Abstract: We propose a new task for assessing machines’ skills of understanding fictional characters in narrative stories. The task, TVShowGuess, builds on the scripts of TV series and takes the form of guessing the anonymous main characters based on the backgrounds of the scenes and the dialogues. Our human study supports that this form of task covers comprehension of multiple types of character persona, including understanding characters’ personalities, facts and memories of personal experience, which are well aligned with the psychological and literary theories about the theory of mind (ToM) of human beings on understanding fictional characters during reading. We further propose new model architectures to support the contextualized encoding of long scene texts. Experiments show that our proposed approaches significantly outperform baselines, yet still largely lag behind the (nearly perfect) human performance. Our work serves as a first step toward the goal of narrative character comprehension.

## introduction stories have two essential elements, plots and characters @xcite . character comprehension has been widely recognized as key to understanding stories in psychological, literary and educational research @xcite @xcite . when reading stories, humans build mental models for characters based on their personae, which helps people to explain a character's emotional status @xcite , identity, understand future behaviors @xcite , and even make counterfactual inferences for stories about that character @xcite . the ultimate goal of creating a character comprehension system is to equip a machine with practical capabilities that emulate what humans can accomplish. for example, understanding personae can facilitate story memorization and generation of new statements consistent with the story @xcite . such capabilities could be valuable in the construction of dialog engines that help people address practical problems such as those experienced in customer service encounters @xcite @xcite . more importantly, understanding the persona of a particular person can help chatbots to understand the intention behind a human user's language (bender and koller, 2020), which can lead to better services and ultimately give systems the ability to demonstrate behaviors that users interpret as empathetic. for instance, amy's last sentence in figure 1 is a joking braggadocio to remind her boyfriend to value her more. only when sheldon understood the facts of their relationship as a couple and amy's temporary show-off mentality could he see her true intentions. 2 despite the importance of this capability, there has been limited attention to modeling characters in stories in the natural language processing (nlp) community. 3 most existing character-centric prediction tasks have the input sources in expository text such as synopsis (summaries) of stories @xcite or non-narrative dialogues @xcite @xcite . a few exceptions work on stories, but focus on limited aspects of personae, such as facts for coreference resolution @xcite , personality @xcite and character relationships @xcite , with only a few @xcite ; @xcite providing evaluation benchmarks. besides the limited persona aspect coverage, these models also lack the ability to take into account a theory of mind (tom) which is the knowledge of epistemic mental states that humans use to describe, predict, and explain behavior @xcite . in this work, we propose the first task on character comprehension in stories to assess the ability of mental model construction in nlp. a character's words are her direct reflection of the contexts, conditioned on her character model @xcite . our task, tvshowguess (tvsg), aims to guess anonymous speakers using dialogues, scene descriptions and historical scenes, which requires models to interpret the behavior of characters in the form of dialogues, which meets the requirements for the evaluation of toms. through experiments and human studies we found the following results. first, human performance was nearly perfect, while the model performed poorly. second, although our tvsg has a simple task setup, it has a surprisingly wide coverage of persona understanding skills including the linguistic styles, personality types, factoids, personal relations, and the memories of characters' previous experience. third, most of the cases (>60%) require identification and understanding of characters' historical experiences to resolve. among them, many rely on facts of characters that are not explicitly described in texts but need to be inferred from event history. the wide persona coverage and heavy dependency on history present challenges to existing nlp techniques. these challenges lead to more than 20% accuracy gap between our baselines and humans. specifically, we make the following contributions. @xcite we propose the research direction of character comprehension in stories; with an extended survey (section 2 and appendix a) discussing the differences and challenges compared to related work. (2) we propose the first task and dataset for multi-aspect persona (especially tom) understanding in stories (section 3). (3) we propose a new schema to analyze the required evidence for character understanding; and conduct human studies to analyze the required skills of our task (section 4 and appendix c). ( 4 ) we propose new model architectures as the initial step of this direction; and conduct comprehensive experiments to provide insights to future work (section 5 and 6). ## related work we discuss and compare the following related areas: the assessment benchmarks for general narrative comprehension skills and the tasks specifically designed for character-centered predictions over narratives and non-narratives. ## analysis of our benchmark we propose a comprehensive schema of persona types for the machine narrative comprehension. the schema facilitates the analysis of the challenges in our task; and provides insights into the deficiencies in current narrative comprehension models, by allowing a decomposition of model performance to the dimensions of categories (section 6). ## methods inspired by the successes of pre-trained transformers on reading comprehension tasks, we benchmarked our tvshowguess by building baseline solutions on top of these pre-trained models. the key challenge of our task is that the prediction relies on how a character reacts to the scenario with her/his words, therefore the embedding of each utterance should be highly context-aware. this requires handling a whole scene as input, which is usually over the limits of these pre-trained models. we propose two solutions. the first is to benefit from sparse attention, specifically, longformer @xcite . the second is to organize each utterance with its necessary history context as one row, and have a bert model to encode each relatively short row independently. for both models, we finally conduct attentive pooling for each character over the contextualized embeddings of all her utterances for prediction. our baselines simplified the problem by (1) ignoring the historical scenes in eq.( 1 ); and (2) making independent prediction of characters within a scene. the former poses the challenge of handling longer contexts and the latter requires specific predictor design. we believe both are important to handle in future work. ## experiments we evaluate the instance-level accuracy. an instance refers to a masked speaker in a scene. ## conclusion we present the first task and dataset for evaluating machine reading comprehension models for understanding characters in narratives. based on linguistic, educational, and psychological theories, we proposed a new schema and conducted human studies to analyze the types of evidence and reasoning required in understanding characters. we design a new model architecture and conduct comprehensive experiments as a basis for future studies.