TVS HOW G UESS : Character Comprehension in Stories as Speaker Guessing
Abstract
We propose a new task for assessing machines’ skills of understanding fictional characters in narrative stories.
The task, TVS HOW G UESS , builds on the scripts of TV series and takes the form of guessing the anonymous main characters based on the backgrounds of the scenes and the dialogues.
Our human study supports that this form of task covers comprehension of multiple types of character persona, including understanding characters’ personalities, facts and memories of personal experience, which are well aligned with the psychological and literary theories about the theory of mind (ToM) of human beings on understanding fictional characters during reading.
We further propose new model architectures to support the contextualized encoding of long scene texts.
Experiments show that our proposed approaches significantly outperform baselines, yet still largely lag behind the (nearly perfect) human performance.
Our work serves as a first step toward the goal of narrative character comprehension.
Introduction
Stories have two essential elements, plots and characters (McKee, 1997).
Character comprehension has been widely recognized as key to understanding stories in psychological, literary and educational research (Bower and Morrow, 1990;
Kennedy et al., 2013; Currie, 2009; Paris and Paris, 2003).
When reading stories, humans build mental models for characters based on their personae, which helps people to explain a character’s emotional status (Gernsbacher et al., 1998), identity, understand future behaviors (Mead, 1990), and even make counterfactual inferences for stories about that character (Fiske et al., 1979).
The ultimate goal of creating a character comprehension system is to equip a machine with prac- tical capabilities that emulate what humans can accomplish.
For example, understanding personae can facilitate story memorization and generation of new statements consistent with the story (Riedl and
Young, 2010).
Such capabilities could be valuable in the construction of dialog engines that help people address practical problems such as those experienced in customer service encounters (Mairesse and Walker, 2007; Zhang et al., 2018; Urbanek et al., 2019).
More importantly, understanding the persona of a particular person can help chatbots to understand the intention behind a human user’s language (Bender and Koller, 2020), which can lead to better services and ultimately give systems the ability to demonstrate behaviors that users interpret as empathetic.
For instance, Amy ’s last sentence in Figure 1 is a joking braggadocio to remind her boyfriend to value her more.
Only when Sheldon understood the facts of their relationship as a couple and Amy ’s temporary show-off mentality could he see her true intentions.
Despite the importance of this capability, there has been limited attention to modeling characters in stories in the natural language processing (NLP) community.
3 Most existing character-centric prediction tasks have the input sources in expository text such as synopsis (summaries) of stories (Brahman et al., 2021) or non-narrative dialogues (Zhang et al., 2018; Urbanek et al., 2019; Li et al., 2020).
A few exceptions work on stories, but focus on limited aspects of persona, such as facts for coreference resolution (Chen and Choi, 2016), personality (Bamman et al., 2013; Flekova and Gurevych, 2015) and character relationships (Iyyer et al., 2016), with only a few Chen and Choi (2016);
Flekova and Gurevych (2015) providing evaluation benchmarks.
Besides the limited persona aspect coverage, these models also lack the ability to take into account a theory of mind ( ToM ) which is the knowledge of epistemic mental states that humans use to describe, predict, and explain behavior (Baron-Cohen, 1997).
In this paper, we propose the first task focused on character comprehension in stories, to assess the ability of mental model construction in NLP.
A character’s words are her direct reflection of the contexts, conditioned on her character model (Holtgraves, 2010).
Our task,
TVS HOW G UESS (TVSG) , aims to guess anonymous speakers using dialogues, scene descriptions and historical scenes, which requires models to interpret the behavior of characters in the form of dialogues, which meets the requirements for the evaluation of ToMs.
Through experiments and human studies we found the following results: First, human performance was nearly perfect, while the model performed poorly.
Second, although our TVSG has a simple task setup, it has a surprisingly wide coverage of persona understanding skills including the linguistic styles, personality types, factoids, personal relations, and the memories of characters’ previous experience.
Third, most of the cases ( > 60%) require identification and understanding of characters’ historical experiences to resolve.
Among them, many rely on facts of characters that are not explicitly described in texts but need to be inferred from event history.
The wide persona coverage and heavy dependency on history present challenges to existing NLP techniques; and explain the more than 20% accuracy gap between our base- lines and humans.
We make the following contributions:
(1) We propose the research direction of character comprehension in stories; with an extended survey (Section 2 and Appendix A) discussing the differences and challenges compared to related work.
(2) We propose the first task and dataset for multi-aspect persona (especially ToM) understanding in stories (Section 3).
(3) We propose a new schema to analyze the required evidence for character understanding; and conduct human studies to analyze the required skills of our task (Section 4 and Appendix C).
(4) We propose new model architectures as the initial step of this direction; and conduct comprehensive experiments to provide insights to future work (Section 5 and 6).
Related Work
In this section we mainly discuss and compare related areas: the assessment benchmarks for general narrative comprehension skills; and the tasks specifically designed for character-centered predictions over narratives.
Table 1 gives a summary of these narrative comprehension tasks, associated with their required comprehension skills.
We also reviewed studies on character-centered tasks over non-narrative texts like synopses and chit-chat
( i.e. , not story-related) conversations.
Detailed rationales of the required skills for each task are discussed in Appendix A.
Assessment of Narrative Comprehension There are many forms of reading comprehension tasks such as cloze tests (Bajgar et al., 2016; Ma et al., 2018), question answering (Richardson et al., 2013;
Koˇcisk`y et al., 2018; Yang and Choi, 2019; Lal et al., 2021; Xu et al., 2022), and text summarization (Ladhak et al., 2020; Kry´sci´nski et al., 2021; Chen et al., 2021).
Most of these tasks are built on very short stories or can be solved in segments of a story, and therefore present limited challenges to understanding the elements of the story, especially the characters.
The exceptions are NarrativeQA (Koˇcisk`y et al., 2018) and three summarization tasks which are mainly eventcentric tasks focusing on understanding the plot structures in the stories.
The NarrativeQA consists of a small portion of character-related questions according to the human study in (Mou et al., 2021), but mainly about simple facts of characters like age, place of birth and profession. Finally, text
Dataset
TaskFormat
NarrativeType
AssessedNarrativeComprehensionSkills Source
Length
PlotStructures CharacterFacts CharacterToMs
MCTest
Multi-choiceQA
Shortfiction (Childrenstories) ∼ 20 ∗ ✓ BookTest
Clozetest
Literature (Excerpt) – ✓ (Maetal.,2018)
Clozetest
TVshowtranscripts (Scenes) ∼✓ NarrativeQA
GenerativeQA
MovieScripts,Literature (Fullstories) ∼ 11K ∗ ✓ ✓ FriendsQA
ExtractiveQA
TVshowtranscripts [Table] [Table] 20 ∗ ✓ ✓ NovelChapters/
[Table]
Summarization
Literature [Table]
Summarization
[Table] [Table]
CorefResolution TVshowtranscripts
[Table]
Classification
[Table]
games (Hausknecht et al., 2019) have been proposed as a reinforcement learning task that requires understanding of narrative fiction stories.
Studies have been conducted (Guo et al., 2020; Yao et al., 2021) to investigate the roles reading comprehension plays in these games.
Character-Centric Prediction over Narratives
The task of coreference resolution of story characters (Chen and Choi, 2016; Chen et al., 2017a) is most closely related to our TVS
HOW G UESS .
These tasks focus on identifying the characters mentioned in multiparty conversations, which mainly requires the understanding of discourse relations and assessment of personal facts.
However, coreference does not assess the modeling of the character’s theory-of-mind, especially the character’s memories, as there are no predictions of character behaviors involved.
The prediction of fictional characters’ personality types by reading the original stories (Flekova and Gurevych, 2015) is another character-centric task related to the present work.
This work covers only the character’s personality such as the big five and the MBTI types, also a perspective of the persona our work considers.
Character-Centric Prediction over Non-
Narratives Many tasks do not use the original story, but rather a summary of it.
For example, the textual entailment task LiSCU (Brahman et al., 2021) links an anonymous character summary to the name appearing in the story’s summary.
The usage of summaries precludes ToM modeling, as discussed in Appendix A.1.
Personalized dialogue generation (Mairesse and Walker, 2007; Walker et al., 2012; Zhang et al., 2018; Urbanek et al., 2019; Li et al., 2020) benchmarks are based on daily chit-chats.
They usually cover a single facet of a multi-dimensional persona (Moore et al., 2017), e.g. , personal facts (Zhang et al., 2018) or personality types (Mairesse and Walker, 2007; Li et al., 2020).
The LIGHT environment (Urbanek et al., 2019) covers both facts and personalities.
None of the above covers a comprehensive persona like ours, especially how a character’s past experience builds her ToM.
Authorship attribution has a parallel goal to ours, insofar as it aims at guessing author identities from the texts they wrote (Ni et al., 2019; Andrews and
Bishop, 2019; Bevendorff et al., 2020).
These tasks differ from ours in two respects: first, multiple prose examples generated by the same author do not usually form consecutive plot lines, and second, they rarely model the event history of depicted characters.
On this basis, they mainly require understanding authors’ writing styles rather than building mental models of facts, events, and experiences.
Our TVS HOW G UESS Benchmark 3. Task Definition
TVSG adopts a multi-choice setting.
The goal is to guess the anonymous speakers who are the main characters (maximum number of 6 for each show) in the scene.
The models are provided with an anonymous scene that consists of n lines ˜ S ( t ) =
{ ˜ s ( t ) 1 , ˜ s ( t ) 2 , ..., ˜ s ( t ) n } ( t stands for the t -th scene in the entire show).
Each line ˜ s i can be either a dialogue turn or a background description.
When the line is a dialogue turn, it is associated with an anonymous speaker ID (with the form of P x , 1 ≤ x ≤ 6 ) of a main character, or the real name of a supporting character.
Similarly, we introduce the notation of the standard scene
S ( t ) =
{ s ( t ) 1 , s ( t ) 2 , ..., s ( t ) n } , which has the same definition as the anonymous scenes, with the only difference that the dialogue turns always have their real names of speakers associated.
The anonymous scene ˜ S ( t ) is associated with a candidate set C ( t ) =
c ( t ) 1 , ..., c ( t ) k , k ≤ 6 , with each character c ( t ) j is a main character who appears in
S .
The goal is thus predicting each P x ’s actual role c ( t ) j , i.e. , a match π ( · ) from the anonymous IDs to the real characters, conditioned on the scene ˜ S ( t ) and all previous scenes S (1: t − 1) :
[Equation]
3. Dataset Collection
We use community contributed transcripts from
The TV MegaSite (TMS) 4 like (Chen et al., 2021).
Our scenes are from the scripts of five popular TV series: Friends , The Big Bang Theory (TBBT) ,
The
Office , Frasier and Gilmore Girls .
Data Cleaning Our data consists of character dialogues and background descriptions.
The dialogues start with the characters’ names.
One or more turns of dialogue between characters comprise a scene.
Scenes are separated by short background cus that begin with markers such as location (e.g.
“ Howard’s car ”, “ Kingman Police Station ”), special words (e.g., “ Scene ”, “ Cut ”), or symbols (e.g.
We created a rule-based parser which splits the content of an episode into multiple independent scenes using scene separation markers.
Character Recognition and Anonymization We used the main characters’ names to identify their dialogues within each scene and randomly labeled them with speaker IDs (i.e., P0, P1).
Since the different names of the characters, such as nicknames, first names, and last names, mark the dialogue in a mixed way, in order to match the lines to the correct speaker, we first identified the main characters in each TV series by consulting Fandom’s cast list.
Then, we calculated the frequency of speech to find references to the same main character’s name.
Analysis of Our Benchmark
We propose a comprehensive schema of persona types for the machine narrative comprehension.
The schema facilitates the analysis of the challenges in our task; and provides insights into the deficiencies in current narrative comprehension models, by allowing a decomposition of model performance to the dimensions of categories (Section 6).
4. Our Annotation Schema for the Human Study
Two researchers with backgrounds in psychology, linguistics, NLP, and education developed and tested an inductive coding method based on the methods of grounded theory (Glaser and Strauss, 2017).
They conducted three rounds of independent annotation and discussion of the evidence needed to identify the characters, using 10 randomly selected scenes for each round.
After each discussion, they updated the codebook accordingly.
Modifications to the codebook led to the achievement of saturation during the process.
Then the two researchers coded a total of 318 characters from 105 scenes of Friends and The Big Bang Theory.
The annotation interface appears in Appendix B.
This schema categorizes the required evidence to resolve the task into four persona data types: linguistic style , personality , fact , memory .
Tablereports inter-rater reliability calculated by Cohen’s
Kappa (Cohen, 1960).
The kappa values range from 0.76 to 0.87 and would all be considered satisfactory (Viera et al., 2005), reflecting the success of our codebook and process.
We have one additional type, inside-scene , referring to tasks that can be resolved purely within local contexts, thus not requiring persona understanding.
To better depict how these pieces of evidence are used in human rationales, we added two complementary categories: (1) how the task instance relies on the history scenes , and (2) when there are multiple pieces of evidence required, what types of reasoning skills are used to derive the answer from the evidence (see Section C).
Table 6 shows the definitions of each evidence type.
We provide examples of each evidence type in Section B.2.
4.1. Major Evidence Types
Linguistic Style Personalized language patterns that reflect individual differences in self expression and is consistently reliable over time and situations (Pennebaker and King, 1999).
Personality Stable individual characteristics (Vinciarelli and Mohammadi, 2014) that can distinguish “internal properties of the person from overt behaviors” (Matthews et al., 2003).
Memory A character’s episodic memory of events from previous episodes and the semantic memoryinferred from events.
Note here we want to model the memory of a particular character, i.e. , the historical scenes experienced by the particular character
[Table]
instead of from the audiences’ perspective.
A character’s memory is crucial for humans to build her
ToM, but is largely ignored as a part of persona in previous research.
Fact The truth about characters as opposed to interpretation, which can usually be represented as knowledge triples.
• Attribute All explicitly provided factual character identity information in the TV series setting, such as race, occupation, and education level.
• Relationship Relationship includes social relationships (e
g., husband and wife) and dramatic relationships (e
g., arch-enemy).
When talking to people with different relationships, characters change their identity masks by using different words (Gergen, 1972).
Status The emotional or psychological status of a character when facing a specific situation.
Inside-Scene
The textual evidence inside the scene, independent from the character’s persona.
• Background Background introduction and descriptions in other character dialogues.
• Mention
The character’s name or alias is called out by other characters.
Although mention is persona-independent
, it still presents challenging cases.
In a multi-person, multi-round dialog, because anaphora in the current sentence may not be bound to an antecedent on the right frontier of the structure, common sense analysis of conversational coherence is needed to determine which speaker is being referred to.
Exclusion A guessing technique for elimination using a given list of characters which is neither evidence nor inference, but it depends on the character list provided within the scene, so we include it as a subcategory of inside-scene evidence.
4.1. Dependence on History
To understand how much humans rely on memory to identify a character, the annotators coded whether the evidence necessary to solve the task depends directly on historical events or whether it depends indirectly on history by abstracting from historical events.
Direct Dependency Characters that can only be identified through events that are explicitly expressed in previous episodes, 6 for example:
Background: (from TBBT )
[The stairwell] Candidates: {Leonard, Penny} P0: There’s something I wanted to run past you.
P1:
What’s up? P0: Mm, the guys and I were thinking about investing in Stuart’s comic book store.
Is that okay?
P1: Why are you asking me?
Answer
[Table]
Rationale
[Table]
In a previous
[Table]
, Leonard and
[Table]
friends discussed about
[Table]
in Stuart’s
[Table]
, so he is the only one between the two who has this memory.
Indirect Dependency Characters can only be identified using evidence not explicitly expressed in previous episodes, but inducible from previous events.
For example, Personality can be inferred from the character’s previous behavior.
Background: (from Friends )
[Central Perk] Candidates: {Joey, Rachel, Ross} P0: Here you are (Hands Rachel a cup of coffee) P1: Thank you Joey.
You know what?
I’m not even sure I can have caffeine.
P2:
I went thru this with Ben and Carol.
One cup of coffee won’t affect your milk.
P1:
Yeah.
Just to be sure I’m gonna call Dr. Wiener.
Answer: P
[Table]
If the answer can be inferred within the scene, like answering P0 → Joey in the above example.
We have a special rule on the Exclusion evidence type – If a character can only be inferred on the
[Table]
[Table]
basis of other characters being eliminated, it should have dependency type labeled if any of the other character has a history dependency.
In other words, when guessing the identity with Exclusion requires history dependency on another character, the dependency type is transitive.
4. Analysis
Main Statistics Table 3 shows the proportions of the required evidence types and dependency of history.
According to the statistics, history is an important factor in guessing the characters.
46.36% of the examples from Friends and 67.55% examples from the Big Bang Theory require history.
Human Performance Accuracy One annotator (who had not watched the seasons under evaluation) obtained nearly perfect accuracy in guessing the characters in FRIENDS (98.68%), and a lower but still good accuracy in TBBT (89.82%).
A second annotator (who had watched all episodes and thus would be considered an expert) confirmed that most the error cases were unsolvable given only data extracted from scenes.
We list the unsolvable cases and human mistakes in Appendix E.
Correlation between Evidence Types and History Dependence Figure 2 visualizes the flow from evidence types to the dependency of history.
Most of them are correlated.
Personality and history dependency are most closely related.
Methods
Inspired by the successes of pre-trained Transformers on reading comprehension tasks, we benchmarked our TVS HOW G UESS by building baseline solutions on top of these pre-trained models.
The key challenge of our task is that the prediction relies on how a character reacts to the scenario with her/his words, therefore the embedding of each utterance should be highly context-aware .
This requires handling a whole scene as input, which is usually over the limits of these pre-trained models.
We propose two solutions.
The first is to benefit from sparse attention, specifically, Longformer (Beltagy et al., 2020).
The second is to organize each utterance with its necessary history context as one row , and have a BERT model to encode each relatively short row independently.
For both models, we finally conduct attentive pooling for each character over the contextualized embeddings of all her utterances for prediction.
Our baselines simplified the problem by (1) ignoring the historical scenes in Eq. 1; and (2) making independent prediction of characters within a scene.
The former poses the challenge of handling longer contexts and the latter requires specific predictor design.
We believe both are important to handle in future work.
5. Transformers with Character-Pooling
Our first approach (the top in Figure 3) is denoted as Longformer-Pooling (or Longformer-P ).
Scene Encoding
The input ˜ S to the model includes the concatenation of all the utterances in an anonymous scene.
Each utterance is prefixed by a speaker
ID token and suffixed by a separation token, i.e. ,
[Equation]
where U i is the i -th utterance and [P x i ] is its speaker ID (e.g., [P 0 ] and [P 1 ]).
[SPLIT] is a special token.
⊕ denotes concatenation.
We use a
Longformer to encode the whole ˜ S , to make the embedding of each utterance token context-aware , i.e. , H = Longformer ( ˜ S )
∈ R L × D .
Character-Specific Attentive Pooling For each character ID P x , we have a token-level mask M x ∈
R L × 1 such that M x
[ j ]=1 if the j -th word belongs to an utterance of P x and M x
[ j ]=0 otherwise.
For each P x , we then collect the useful information from all her utterances selected by M x as:
[Equation]
where Attention ( · ) is a one-layer feedforward network to compute the token-level attention weights.
The character-specific attention α x is then used to pool the hidden states to summarize a character representation in the input scene ˜ S and make the prediction: P ( P x = c | ˜ S )
=
f k ( H T α x ) .
Here f k : R d × 1 −→ R C × 1 is the character classifier for the k -th TV show.
5. Multi-Row BERT
The second approach (the bottom in Figure 3) is denoted as the multi-row BERT ( MR. BERT ).
We split the long scene ˜ S into multiple segments { ˜ s i } .
Encoding the segments reduces the overall complexity from O ( L 2 ) to O ( RL 2 s ) , where R is the number of segments and
L s
≪ L is the maximum segment length.
To construct each segment E i , we take an utterance T i as in Eq. 2 and concatenate it with the nearest history utterances T i ′
( i ′ <
i ) until arriving the maximum length L s .
The R segments for each instance yield { ˜ s i } as follows:
[Equation]
Then we encode the { ˜ s i } with a BERT encoder:
[Equation]
Different from Longformer-P, we have a segment-level mask M x ∈ R R for each character
ID such that M x [ j ] = 1 if the first utterance in the j -th row (i.e., T t j in Eq. 5) is said by P x .
Applying the same attentive pooling technique to each segment following Eq. 4, we obtain R segment embeddings { E i } R 1 .
We take the concatenation of these embeddings as the new input to the show-specific predictor and calculate the probability distribution of P x being each character, i.e. ,
[Equation]
Compared to Longformer-P, the MR.
BERT model takes a smaller number of R utterances and benefits from their concatenated contextual utterances.
To make the selection of R utterances representative, we applies two tricks: (1) fill-empty , which makes sure each P x has at least one segment selected; (2) the reverse trick , which selects the utterances starting from the end of scene to the start – as the utterances at the end have more histories, they cover more contents from the scene if selected.
Experiments
We mainly evaluate the instance-level accuracy.
An instance refers to a masked speaker in a scene, as defined in Eq.
6. Baselines and Implementation Details
We also compare with the vanilla pre-trained Transformer baseline, Vanilla Longformer Classifier .
The model conducts direct classification over the concatenation of a character’s utterances in the scene.
It can be viewed as a discriminative language model of the characters’ lines.
We include the implementation details of the baseline and our models in Appendix G.
[Table]
6. Results
Overall Results Table 5 compares different models on our TVS HOW G UESS .
The proposed architectures beat our vanilla character classifier with large margins (4-5%).
However, human performance is significantly (21-26%) better than the best models, showing models are still far from reaching human level of character understanding.
Among all the shows, TBBT is the most challenging one, while Frasier and Gilmore Girls are relatively simpler.
Given that there is no correlation between performance and scene lengths (Table 2), this shows that the difficulty of the task mainly comes from the persona modeling, inference and reasoning.
Specifically, the InsideScene evidence requires less persona understanding.
Therefore, the relatively smaller amount of InsideScene cases makes TBBT more difficult.
Also the existing models are not good at resolving the related memory or facts from the history, thus the high ratio of history dependent cases in TBBT also leads to lower performance.
6. Analysis
Learning Curves We plots the learning curves of
Friends and TBBT , with increasing number of shows used as training data (Figure 4).
The curves become flat with all shows added, showing that our task has sufficiently data for training.
Scene-Level Performance Besides the instancelevel accuracy, we further investigated the scenelevel performance, i.e. , the macro-average of instance-level accuracy in all the scenes.
Tableshows the results, together with the decomposed results on scenes containing different numbers of speakers.
The results show the more characters involved, the lower the accuracy is, even though our
[Table] [Table]
model is making independent predictions of each speaker.
One possibility is that there is fewer available utterances per speaker.
In addition, a larger set of speakers may make the logical structure of the conversation more complex.
Character-Level Performance
Next, we examined whether our task is uniformly challenging for different characters, or whether there were certain characters that were more difficult to guess.
Tableshows the results, where the characters are ranked by the accuracy of their guesses.
There are clear discrepancies in accuracy by character.
Impact of the Dependence on History The bar charts in Figure 5 show the performance on different history dependence types.
The performance of cases that require history supports is in general harder for most of the models ( ∼ 20% lower compared to the cases without dependency of history).
The results suggest that a more thorough extraction of historical events associated with each character is needed to make the model an improvement.
This notion aligns with the theories that past experience is an key aspect of building characters’ ToM, showing that our TVS HOW G UESS may serve as a good benchmark for the in-depth study of character comprehension from stories.
Another finding is that the cases requiring indirect history dependence (usually Personality and
Facts ) are even more challenging.
The human mind develops detailed profiles of characters when reading stories.
The neural models represent each character as a single vector ( i.e. , the weight vector in the output layer).
This indicates a promising future direction of constructing structured persona representations ( e.g. , based on our schema of evidence) for more accurate character modeling.
Breakdown to Evidence Types The wind-rose
charts (bottom) in Figure 5 provide performance breakdown onto our evidence categories.
We omit the type of Linguistic style because there are only two cases in Friends so the results are not stable.
As expected, the cases that can be resolved locally without character understanding ( InsideMention ) are relatively easier.
All of Personality ,
Fact and Memory cases have much lower performance as they correspond to heavy dependency on the modeling of history.
The type Exclusion gives the worst overall performance on the two shows.
However, this does not indicate difficulty of character understanding –
According to the definition, these cases cannot be directly resolved with the scene inputs, but require the model to have specific strategy to exclude some incorrect answers first.
It is surprising that the Inside-Background type poses difficulties to our models, because it looks to human annotators mostly like standard textual inference.
We have identified two possible reasons: (1) As discussed in the introduction, some cases require pragmatic understanding from the surface form to intention, only after which textual inference can be performed (2)
The proportion of instances of this type is relatively smaller so the model may fail to recognize the required textual inference skills during training.
Effect of Scene Contexts Finally, the vanilla character classifier behaves differently compared to the other models.
Because it cannot make use of contexts within scenes, there is a performance gap on the Inside-Mention type (hence the drop on the No
Dep type).
However, it does not suffer from significant differential on the other types.
The gap appears because Longformer-P and/or Mr. BERT perform considerably better on this type.
Challenges
of
History Retrieval
Our experiments show that the history dependency presents serious challenges for existing models.
Finding the evidence from history scenes is a retrieval task (but without groundtruth).
We applied a state-of-the-art model to retrieve the history scenes and conducted an additional human study to evaluate the results.
Our study shows that on our identified cases with
Direct Dependency , the top-3 results (from in total 20 candidates) of a state-of-the-art semantic search model only give a recall of 35.5%.
The result confirms that our task requires further advances on semantic retrieval.
The detailed task setting and our discussions can be found in Appendix F.
Conclusion
In this paper, we present the first task and dataset for evaluating machine reading comprehension models for understanding characters in narratives.
Based on linguistic, educational, and psychological theories, we proposed a new schema and conducted two human studies to analyze the types of evidence and reasoning required in understanding characters.
We further design a new model architecture and conduct comprehensive experiments to serving as a testbed for future studies.
A A Detailed Survey of Related Work
In this section, we first give an in-depth analysis on the difference between narrative and synopsis, from both the empirical challenges in NLP studies and the linguistic theory from (Morrow, 1985).
Then we provide detailed discussion on how we summarize related work in Table 8.
A. Background: Narrative versus Synopsis
As our work focuses on narrative comprehension, following setups like (Koˇcisk`y et al., 2018; Kry´sci´nski et al., 2021; Chen et al., 2021), it is necessary to distinguish between comprehension of original narrative stories versus comprehension of their synopses (the human-written plot summaries), e.g. , from the story’s Wikipedia page.
Narrative stories are told by creating scenes, with the goal of helping readers experience events as they occur in the plot, and empathize with the story characters in relation to their own experiences.
To engage readers, story writers often use complex narrative clues ( e.g. , character activities, event development, scenery changes); variable narrative sequences ( e.g. , narrative, flashback, interpolation); and various linguistic expressions ( e.g. , argument, lyricism, narrative, description, illustration).
By comparison, a synopsis is a descriptive summary of the main idea of a story in plain language.
It contains only the main characters, time, place, important plot points, and resolution, rather than allowing the story to unfold through the actions of the characters.
The goal of the synopsis is to inform readers what happened with little or no original material from original story.
Therefore, comprehension of narrative stories requires more sophisticated skills to understand the complex chain of clues and expressions, in order to finally build a complete narrative representation from a sequence of individual scene comprehensions along with a developed understanding of characters’ mental models (Morrow, 1985).
In this light, a synopsis represents "processed results" from the application of these comprehension skills by a (experienced) human reader.
A. Background: Dialogue in Stories vs. Real-Life Conversation
Fictional dialogue canonically serves a purpose in a narrative.
Either i contributes to the develop of a character or advances the plot (McHale, 2004).
Nash proposed three categories of fictional dialogue: (1) confrontational dialogue which “includes challenges, quarrels, disputes, interviews, and any kind of personal encounter in which the participants are in covert or overt opposition to each other”; (2) instructional dialogue, which “conveys information about matters of science, technology, politics, world events, etc, some knowledge of which is essential to understanding the plot”; (3) collaborative dialogue that consists of “a series of exchanges which cumulatively present, for the reader’s benefit, a picture of events, histories, personalities, and relationships” (Nash, 1990).
Authors have total control of the fictional dialogue, and ideally it functions according to the author’s intention at every part of the story (De Haan, 1996).
However, real-life conversation is a joint action and is natural.
There is no individual has whole control and the conversation goal of each party may be very different.
Additionally, real-life conversations are temporally linear, such that communicators cannot revise earlier speech to fit a story.
Moreover, real-life conversations tend to be highly implicit because spoken language derives much of its meaning from context (Warren, 2006).
A. Assessment of Narrative Comprehension
We summarize the related tasks people use for assessment of general narrative comprehension skills.
Cloze Test
Cloze tests take a snippet of the original text with some pieces (usually entities) masked as blanks, with the goal of filling these blanks from a list of candidates.
Cloze tests can be automatically constructed, resulting in an advantage in creation of large scale datasets.
Examples of cloze tests for narrative comprehension assessments are
BookTest (Bajgar et al., 2016) and (Ma et al., 2018).
Both datasets are based on excerpts of books or scenes of TV shows.
As the input consists only of short paragraphs, there is not sufficient information to infer complex character set via reading the stories.
Therefore, these datasets address few questions related to the understanding of characters.
Moreover, when built on short snippets, cloze tests are known to prone to mostly local inference but not much reasoning and commonsense
[Table]
AssessedNarrativeComprehensionSkills
AssessedCommonsenseKnowledge
[Table]
CharacterToMs
[Table] [Table]
StoryFlows
[Table]
Multi
[Table]
choiceQA
Shortfiction
[Table]
Childrenstories
[Table]
∼
[Table]
∗ ✓ ✓ ✓ ✓
[Table]
Clozetest
Literature
[Table]
Excerpt
[Table] [Table]
Maetal
[Table] [Table]
Clozetest
TVshowtranscripts
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
knowledge, as studies in the NLP community suggested (Chen et al., 2016).
On the other hand, although our task also has form similar to cloze, it requires information about the characters from previous scenes, which is not only about understanding the characters, but also requires global inference across features of the story (see Figure 1).
Question Answering
The most popular form of narrative comprehension evaluation is through question answering, starting from the early work of MCTest (Richardson et al., 2013), to the more recent crowd-sourced tasks like NarrativeQA (Koˇcisk`y et al., 2018),
FriendsQA (Yang and Choi, 2019), TellMeWhy (Lal et al., 2021) and
FairytaleQA (Xu et al., 2022).
Among them, the MCTest and TellMeWhy conduct multi-choice question answering on short stories.
As above, te input consists of short paragraphs, so there is not sufficient information to infer complex character factes via reading the stories.
Therefore, these datasets also cover few questions assessing the understanding of characters.
The TellMeWhy has a specific focus on why -questions assessing the causal knowledge between states and events.
The inputs are short stories from the ROCStories dataset (Mostafazadeh et al., 2016).
MCTest covers a much wider set of reading skills, as it is based on complete stories, and generates questions with the goal of assessing children’s reading comprehension over both story plots and commonsense.
[Table]
conduct natural question answering tasks.
NarrativeQA aims to infer free-form answers to questions about a specific book or movie script.
According to the human study from (Mou et al., 2021), the major part of the dataset is event-centric questions, which queries the explicit plots from the original books thus do not require a significant amount of commonsense reasoning.
The study also reveals that NarrativeQA consists of a small portion of character-related questions.
These questions mainly query the simple facts of characters, such as age and profession.
The more complexity character persona types, like personality, emotional/psychological status and history experience studied in our work, are not covered.
Similar to NarrativeQA, FriendsQA is a QA task over TV show scripts.
The dataset consists of six types of questions: who, what, when, where, why , and how .
The who questions target on asking speaker names of utterance contents or participants of events, therefore are mainly assessing understanding of plot structures ( i.e. , participant arguments of events).
Both NarrativeQA and FriendsQA have humanwritten questions with a reference of the plot summary, which require evidence explicitly exists in the original story texts, and thus do not have much requirement of reasoning.
The FriendsQA ques- tions are based on scene summaries, and thus require mostly local evidence; the NarrativeQA questions are based on the book-level summary, and thus sometimes require the ability to bridge the gap between coarse-grained and fine-grained event descriptions ( i.e. , commonsense of sub-events).
Summarization
There is a recent trend to evaluate model’s understanding of stories via summarization, including NovelChapters (Ladhak et al., 2020), BookSum (Kry´sci´nski et al., 2021) and
ScreenSum (Chen et al., 2021).
These works provide a good research opportunity to future story reading research, by showing that book-level or chapter-level summarization is challenging to existing machine reading models.
However, it is more difficulty to identify the specific required reading skills by these tasks, as there exist many factors beyond reading skills to generate a good summary, such as encoding and generating long narrative texts.
Intuitively, story summarization is largely plot-related instead of character-related; and requires the knowledge to understand the story flow.
Interactive Fiction Game Playing Interactive
fiction (IF) games (Hausknecht et al., 2019) have been proposed as a reinforcement learning task that requires understanding of narrative fiction stories as environment observations.
Research work has successfully demonstrated that reading comprehension can provide helpful inductive biases for efficient policy learning (Guo et al., 2020); while Yao et al.
(2021) also reveal the shortcomings of these games.
The debate calls for future investigations to understand the necessary narrative elements and the roles they play in the IF games.
A. Character-Centric Prediction over Narratives
Our task can be seen as a character-centered understanding of the narrative, where the understanding of the character deepens the understanding of the story and makes the narrative engaging.
There are limited studies on understanding characters’ persona from reading stories.
In this section we review some existing character-centric prediction tasks over narrative texts, and discuss the relations and differences.
Character Name Linking
The task of coreference resolution for story characters (Chen and Choi, 2016; Chen et al., 2017b) is closely related to our
TVS HOW G UESS .
These coreference resolution focuses on identifying the characters mentioned in multiparty conversations from TV shows scripts.
The goal of these tasks is to resolve the coreference of pronouns and character-indicating nominals ( e.g. , you and Mom ) in dialogues of the character names that appear in the local context.
It also covers linking a named entity ( e.g. , Ross ) to the character, which is more on name matching instead of character understanding.
The task form of coreference resolution mainly requires the understanding of discourse relations.
It does not assess the modeling of character theoryof-mind, especially the character’s memories, as there are no predictions of character behaviors involved.
The major character persona type it assesses is character facts, since the resolution of nominals requires the understanding of the target characters’ occupations and relationships.
The lack of ToM modeling and complex reasoning of the coreference resolution task also makes it relatively easier – on Friends and The Big
Bang Theory , a CNN model gives a > 90% average accuracy.
By comparison, our task, although solvable by humans with a ∼ 95% accuracy, is challenging to neural models as the best BERT-based model gives a ∼ 65% average accuracy on the same two shows with even smaller candidate sets.
Personality Prediction
Our work is also related to the prediction of fiction characters’ personality types by reading the stories (Flekova and Gurevych, 2015).
Specifically, the tasks require to predict a fiction character’s MBTI personality types (Myers and McCaulley, 1988) rooted in Jung’s theory, based on the character’s verbal and non-verbal narratives in the original stories.
Compared to the aforementioned character-centric prediction tasks, these studies require to read and comprehend the original long stories, but the prediction task are relatively simpler since they only focus on personality which is a single perspective of persona.
A. Character-Centric Prediction over Non-Narratives Character name linking between story synopses
Recently Brahman et al.
(2021) propose the LiSCU, which is a novel textual entailment task linking an anonymous summative descriptions of story character to the name appearing in the story’s plot summary.
Similarly to (Chen and Choi, 2016), the task assess the resolution of names and events instead of the ToM modeling.
This is because the task does not involve much explicit behavior predictions, since the task form is entailment between two given statements rather than predicting the possibility of new contents.
The usage of synopses over original stories reduces the challenges in narrative understanding; and further prevents the character comprehension from stories, as pointed out by (Koˇcisk`y et al., 2018), the summaries themselves are humans’ comprehension results of the stories.
Personalized Dialogue Generation
Our work is also related to personalized dialogue generation, for which datasets (Mairesse and Walker, 2007; Walker et al., 2012;
Zhang et al., 2018;
Li et al., 2020) and models (Li et al., 2016;
Mazaré et al., 2018; Qian et al., 2018; Zheng et al., 2020) are proposed for generating dialogues for speakers with persona features.
These benchmarks usually cover a single aspect of the multidimensional persona (Moore et al., 2017).
For example, PERSONA-CHAT (Zhang et al., 2018) focuses on personal facts such as “ I’m a writer ” and “ I live in Springfield ”; other works mainly focus on learning language styles from speakers’ personality types, such as the Big Five traits of the extraversion personality in PERSONAGE (Mairesse and Walker, 2007), and the personality types derived from TV tropes (e.g. jealous girlfriend , book doom , anti hero ) in ALOHA (Li et al., 2020).
LIGHT (Urbanek et al., 2019) is a crowdsourced dataset for text game adventure research.
It includes natural language descriptions of fantasy locations, objects and their affordances, characters and their personalities, dialogue and actions of the characters.
The biggest difference between ours and LIGHT is that LIGHT is based on the local environment of the conversation, rather than on a story.
Examples from the LIGHT dataset are independent conversations and the context in which they occur.
Crowd workers created the dialogues of characters by a given setting and a persona.
The persona is modeled by the Persona-Chat dataset which is defined as a set of three to five profile sentences describing their personal facts such as “ I am a part of a group of travelers ” and “ I go from town to town selling food to the locals ”.
To the best of our knowledge, none of the existing studies cover a comprehensive multidimensional persona similar to our work, especially with respect to how a character’s past experience builds her ToM.
Authorship Attribution
Finally, authorship attribution has parallel ideas to our task, insofar as it aims at guessing author identities from the texts they wrote (Ni et al., 2019; Andrews and Bishop, 2019; Bevendorff et al., 2020) and thus requires a certain degree of author profiling.
These tasks differ from ours because the reviews, tweets or fandoms under the same authors do not usually form consecutive plotlines.
Therefore, the tasks mainly require to understand the authors’ writing styles rather than building mental models from the their past experiences.
From this perspective, this direction is in fact more closely related to stylistic analysis in narrative understanding (Vishnubhotla et al., 2019), rather than character understanding.
B Supplementary for the Dataset Analysis B. Summary of the Annotation Schema
We include a summary of our annotation schema in Figure 6.
B. Examples of Each Evidence Types Linguistic Style
Background: (from TBBT )
[Amy’s car] Candidates: {Leonard, Penny, Sheldon, Amy} P0:
Whatever.
You can’t even go on a date without check- ing your relationship agreement.
P1:
If you’ve got a problem basing a relationship on a contract, I’d like to tell you about 13 plucky colonies that entered a relationship agreement called the U.S. Constitu- tion.
And it may not be cool to say so, but I think that love affair is still pretty hot today.
Answer
: P
→
Leonard
Rationale: (Shelton’s language is characterized by the use of long, difficult sentences and references to historical stories.
Personality
Background: (from TBBT )
[The cafeteria] Candidates: {Leonard, Howard, Sheldon, Raj} P0: And you love the sound of your own voice.
P1:
Yeah, well
, of course I do.
Listen to it.
It’s like an earful of melted caramel.
Answer: P1 → Sheldon
Rationale: (Sheldon is a self-centered person so he will praise his own voice.)
Memory
Background: (from TBBT )
[The stairwell]
Candidates: {Leonard, Penny}
P0: There’s something I wanted to run past you.
P1:
What’s up? P0: Mm, the guys and I were thinking about investing in Stuart’s comic book store.
Is that okay?
P1: Why are you asking me?
Answer: P0 → Leonard
Rationale:
(In a previous scene, Leonard and his friends discussed about investing in
Stuart’s
store, so he is the only one between the
two
who has this
memory
Fact
[Table]
• Attribute Background: (from TBBT )
[Amy’s lab] Candidates: {Amy, Penny} P0: Hey.
Ready to go to lunch?
P1: Just give me a minute.
I’m stimulating the pleasure cells of this starfish.
I just need to turn it off.
Answer: P1 → Sheldon Rationale: (Sheldon is Amy’s boyfriend.
After identify P0 is Amy, based on the relationship between Amy and Sheldon, P1 can be identified as Sheldon.)
• Relationship Background: (from TBBT )
[Amy’s lab] Candidates: {Amy, Penny, Sheldon} · · · P0: Hey, boyfriend.
P1: Can’t talk.
Spitball.
Probably gonna die.
Answer: P1 → Sheldon Rationale: (Sheldon is Amy’s boyfriend.
After identify P0 is Amy, based on the relationship between Amy and Sheldon, P1 can be identified as Sheldon.)
Status
Background: (from TBBT )
[The pub] P0:
So when do you guys plan on getting married?
P1:
Uh, we’re not sure.
But I want to wait long enough to prove to my mother I’m not pregnant.
P2:
May I have one of your fries?
P1: Of course.
Can I have a bite of your burger?
P2: Absolutely not.
P3: Some perfect couple.
He won’t even share his food with her.
Answer:P3 → Leonard Rationale: (The aforementioned failure to determine Leonard’s marriage led him to ridicule couples in harmonious relationships.)
Inside-Scene
• Background Background: (from TBBT )
[Penny’s apartment] Candidates: {Amy, Penny} Bernadette: Nah, you got this.
Let’s go for a drink.
I’ll call Amy .
P0:
Okay, good.
She seemed like she really wanted to go out tonight.
P1 (phone ringing, running down stairs from outside penny’s door): Hey, girl.
Answer: P1 → Amy Rationale: (Bernadette said she will call Amy and P1 is the person who answers the phone.)
• Mention Background: (from TBBT )
[The apartment] Candidates: {Raj, Leonard, Sheldon, Amy} P0: Mmm, I love how they put a waterfall at centre field.
It really ties the whole stadium together.
P1:
This is fun, huh?
We get to see our friend throw out the first pitch, have a hot dog, watch the game.
P2:
Whoa.
Nobody said anything about watching the game.
P3: Sheldon , what did you expect?
Answer: P2 → Sheldon Rationale: (P3 mentioned the name of the person being questioned which is “Sheldon”)
Exclusion
Background: (from Friends )
[Scene: Outside the Janitor’s Closet, there are people having s*x and
Mr. Geller is trying to give them some pamphlets.]
Candidates: {Monica, Chandler}
Mr. Geller: Kids, I spoke to a doctor and picked up this pamphlets on how to get pregnant.
(He slides them under the door.
P0: (walking by with Chandler.)
Hey dad!
P1: Hey.
Mr. Geller: (pause)
Sorry to bother you again, but could you pass my pamphlets back?
(They do so.)
Thank you.
Answer: P1 → Chandler
Rationale: (Monica is Mr. Geller’s daughter.
P0 called Mr. Geller dad
so she is Monica.
There are only two candidate so the other one is Chandler)
C Extended Study of Required Reasoning Types on our TVS HOW G UESS
This section provides an in-depth analysis of the types of reasoning used to infer evidence when guessing characters.
C. Our Annotation Schema of Reasoning Types
We define the following reasoning types with examples provided.
A summary of our annotation schema of reasoning types can be found in Figure 7.
Multi-hop on Characters
Reasoning on the basis of other characters that have already been guessed.
Using the already guessed character as a bridge, users can employ history event or the relationship between characters to make guesses about the target character.
The difference between multihop character and exclusion is that after identifying the other characters, the exclusion technique relies only on the list of characters provided for guessing, however, multi-hop character reasoning requires additional evidence such as relationship to infer the target character.
Background: (from TBBT )
[Angels Stadium] Candidates: {Raj, Leonard, Sheldon, Amy} P5: Hey, I hear you’re a dermatologist.
Emily: Uh, yeah,
I’m a resident at Huntington Hospital. ...
P5: I have some odd freckles on my buttocks.
Can I make an appointment for you to look at them?
Emily: Um, okay, I guess.
P0: I’m with him three years, nothing.
She’s
[Table]
Multi-hop on Textual Evidence
Some evidences are not directly presented in the scene but can be inferred from the descriptions of context and dialogues.
Using the inferred evidences as bridges people can multihop over personality, or fact, or event inferred from the text to guess the characters.
[Table]
Commonsense of Concepts/Events
Task requires additional commonsense knowledge of attributes of daily concepts or social events, or their relations including causal/effect relations between an event and a social state or social relation.
We restrict this category to be the aforementioned commonsense knowledge types, to distinguish from other relatively under-studied commonsense knowledge, such as the commonsense of dialogue flow required to work with our inside-scene evidence defined in Figure 6.
[Table]
Default Conjunction
A single piece of evidence will not solve this task; a combination between multiple pieces of evidence is needed to identify the person.
C. Analysis of the Human Annotation Correlation between the Human Annotated Schema Categories
Figure 2 visualizes the flow between (a) evidence types and the dependency of history and (b) evidence types and the reasoning types.
Most evidence types correlate with history dependency.
Personality and history dependency are most closely related.
Default conjunction is the reasoning type that accounts for the largest percentage.
C. Experiments: Performance Decomposition on the Reasoning Types
We further studied the impact of the required reasoning types on the performance (the right column
[Table] [Table]
in Figure 9).
In general there is a clear gap (on average ∼ 10%) between cases that require complex reasoning and those that do not.
The MultihopTextual type is most challenging, because it requires both deep understanding of what the texts implies and multihop reasoning.
There is not a clear performance difference between Multihop-Character and Default Conjunction , though the former is conceptually harder.
We surmise this is because both types are beyond the reasoning ability of the model so the predictions largely rely on fuzzy matching of evidence – recall that we predict identities of main characters, so there can be a statistical bias of their context co-occurrence.
The results on the
Commonsense type fluctuate due to the relatively smaller ratio.
[Table]
D Interface for the Human Study
Figure 10 shows the interfaces of the human study.
E Examples of Human Errors
Table 11 provides an example of unsolvable cases and Table 12 provides an example of human mistakes.
The human mislabeled characters are marked as red.
We further provide all the scene IDs on which our human tester makes incorrect predictions in
Table 10.
F Details of Human Study and Discussions on the Challenges of History Retrieval
Our experiments show that the history dependency challenges existing models.
Finding the evidence
[Table]
Next
[Table] [Table]
Did you get a chance to look over the
[Table]
I gave you?
[Table]
: Uh, not yet, but I will.
[Table]
Great.
When?
P5 : I said I’ll get to it.
P0 : I’m sensing awkwardness, am I right?
P3 : Yes. P0 : Swish.
[Table]
but
[Table] [Table]
[Table]
familiar with
[Table] [Table]
so
[Table]
pet
[Table] [Table]
, I don’t know.
Who here has
[Table]
they
[Table]
the teacher’s pet
[Table]
like the rest
[Table]
wanted Ms.
[Table]
to forget the quiz.
[Table] [Table]
[Figure]
history scenes for such cases is essentially a retrieval task (but without groundtruth).
To see how it brings new challenges to existing semantic search, we applied a state-of-the-art model to retrieve the history scenes and conducted an additional human study to evaluate the results.
Task We conduct the study on scenes in our human annotation sets that have the Memory type labeled.
With each scene as a query, we retrieve from a window of 20 previous scenes with a state-of-the-art model
The window size is decided so as to guarantee that at least one required memory appears in the window, according to our human annotation process.
The task of human study is to recognize whether the top-3 returned scenes contain at least one related history scene.
Results
The same annotators working on the study in Section 4 evaluated the retrieved scenes.
Results show that the recall of the top-3 results from this state-of-the-art model is low (35.5%).
This difficulty in scene retrieval may arise from: (1) the queries are scenes with structures, which leads to different query formats from standard IR tasks; (2) many relevant scenes are dissimilar to the query scenes in the semantic space, but associated with the query in specific aspects or even analogous to the query scene; (3) some scenes require multihop retrieval, especially when combined with ToM modeling (reasoning about what others know).
All these challenges are non-trivial, and calls for further studies on semantic search to address.
G Model Checklist
We implement our baselines based on HuggingFace Transformers.
We use the pretrained allenai/longformer-baseand bert-base-uncased models.
We train all the models with the Adam optimizer.
We train our model on a single V100 GPU.
It takes around 1 hour and 40 minutes to train a
Longformer-based model.
It takes around 2 hour and 10 minutes to train a multi-row BERT model.
For all the models, we train in total 40 epochs.
But the models usually converge in less than 20 epochs.
Hyperparameters
We set the number of rows in MR.
BERT to 12, to maximize the usage of
GPU memory.
We set the maximum length of
Longformer to 2000, which can handle the lengths of most of the input scenes.
The window size is set to 256.
We set the learning rate to 2e-5.
We report our result with a single run.
However, for each model we run twice; and we found the average development accuracy varies less than 0.5%.