“where is this relationship going?”: Understanding Relationship Trajectories in Narrative Text (2020)

Abstract: We examine a new commonsense reasoning task: given a narrative describing a social interaction that centers on two protagonists, systems make inferences about the underlying relationship trajectory. Specifically, we propose two evaluation tasks: Relationship Outlook Prediction MCQ and Resolution Prediction MCQ. In Relationship Outlook Prediction, a system maps an interaction to a relationship outlook that captures how the interaction is expected to change the relationship. In Resolution Prediction, a system attributes a given relationship outlook to a particular resolution that explains the outcome. These two tasks parallel two real-life questions that people frequently ponder upon as they navigate different social situations: “where is this relationship going?” and “how did we end up here?”. To facilitate the investigation of human social relationships through these two tasks, we construct a new dataset, Social Narrative Tree, which consists of 1250 stories documenting a variety of daily social interactions. The narratives encode a multitude of social elements that interweave to give rise to rich commonsense knowledge of how relationships evolve with respect to social interactions. We establish baseline performances using language models and the accuracies are significantly lower than human performance. The results demonstrate that models need to look beyond syntactic and semantic signals to comprehend complex human relationships.

## introduction a relationship between two people is constantly being shaped by their social interactions @xcite . for example, if two people have a proper conversation after a heated argument, they may become more intimate as they understand each other this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . in contrast to humans' innate ability to perceive social situations, machines struggle in developing social understanding. for example, machines' performance in reasoning about motivation and emotional reactions given a short context is significantly lower compared to human performance @xcite . moreover, identifying the intents and reactions of characters in narratives poses another challenge for machines @xcite @xcite . reasoning about relationship trajectories adds yet another layer of difficulty as in addition to the challenge of identifying the implicit factors that guide characters' behavior, such as intents, reactions and mental states, systems also need to understand how these elements collectively contribute to the evolution of relationships. this additional layer of understanding is challenging primarily because the impact of a particular event on a relationship is unique to each social scenario. for instance, an argument can help resolve differences or deepen them, depending on the personalities, intents and reactions of the characters involved in it. the resources and methods discussed in current work do not directly address these challenges. a fixed set of mappings from events to relationship impact, in a style similar to @xcite , is not sufficient to capture the uniqueness of each social situation. similarly, socialiqa @xcite inspects various elements individually but does not unify them into a single force that influences a relationship. other current datasets @xcite @xcite cover a broad spectrum of commonsense knowledge, but do not focus on the social aspect. although it can be annotated for such inspection such as in the investigation of intents and reactions @xcite , this method is not pertaining to relationship analyses because a meaningful change may not always be present. for example, here is a description of a social scenario sampled from the rocstories corpus: tina decided on going hiking with her friend tony. they both decided on a difficult path. upon ascension, tina fell and cut her leg. they both decided it was too dangerous for them to continue. tony carried tina to the car and decided on mini-golf instead. the story provides a few facts regarding a social interaction but does not provide sufficient details to draw conclusions about how the relationship between the two protagonists is changed because of the interaction. unanswered questions include "did tina appreciate tony's gesture?" and "did they bond as a result?". all these deficiencies in using current resources and methods to study human relationships call for the need of new resources that encode implicit relationship trajectories that are driven by the interweaving of social elements. in this work, we introduce social narrative tree foot_0 , a corpus of 1250 social narratives documenting a variety of social interactions, each centers on two protagonists. it is built incrementally from ten seeds in five narrative stages and the story branches into five different paths at each stage. this effectively captures the different possibilities which a relationship trajectory can take on at each diverging point. we choose the narrative stages -seed, buildup, climax, resolution and outlook -based on previous narrative analyses @xcite @xcite , ensuring that intensity varies at different points of the narrative which simulates fluctuating intensity levels in real-life social situations and thus provides a natural space for relationships to develop in. using social narrative tree, we set up two evaluation tasks, relationship outlook prediction mcq and resolution prediction mcq. in each task, the text corresponding to relationship outlook or resolution is removed from the stories for prediction based on other stages. the branching of each story at different stages creates a natural notion of similarity among stories, which can be measured by the number of stages two stories share. this facilitates the setup of confounding choices as the wrong candidates are not completely irrelevant in terms of textual information. this forces systems to truly understand the underlying relationship trajectories instead of relying solely on language modelling. the best accuracy achieved by our bert-based model @xcite (˜60%) is significantly lower than human performance (˜80%). these experimental results demonstrate that models need to look beyond syntactic and semantic signals to comprehend complex human relationships. in summary, our contributions are as follows: (1) we introduce social narrative tree, a corpus of social narratives with contextualized social elements contributing to relationship trajectories. (2) we introduce new evaluation frameworks using relationship outlook prediction and resolution prediction, with meaningful confounding candidate choices that force systems not to depend only on textual information. (3) we establish baseline performances using language models and justify the importance of looking beyond textual information in understanding human relationships. ## related work manually constructed scripts are used to represent structured knowledge in 1970-80s @xcite . subsequently, narrative event chains, unsupervised generation of such representations are introduced. narrative event chains are sequences of events revolving around one central protagonist @xcite . stemming from narrative event chains is the narrative cloze task where one event chain is removed and systems are required to fill in the blanks @xcite . the task is further refined to multiple choice form -multiple choice narrative cloze (mcnc) where different choices are randomly sampled from events that do not belong to the chain @xcite . swaf af @xcite and hella swag @xcite are datasets for next event prediction, containing multiple choice questions covering a wide range of grounded situations that are constructed from video captions. furthermore, @xcite create rocstories, a corpus of 50k commonsense stories each of fivesentence long and propose story cloze test where systems are required to select the most plausible ending for an incomplete narrative. in our work, relationship outlook prediction and resolution prediction are multiple-choice tasks of similar motivation with a focus on social implications. various social elements are extensively studied in many works. @xcite present atomic, a knowledge graph of commonsense knowledge with 877k descriptions in free text form and focuses on 'if-then' relationships of causes, effects and attributes. automatic construction of such knowledge base is explored in comet @xcite . with atomic as the foundation, socialiqa, a dataset of 38,000 multiple-choice questions about social situations is constructed @xcite . event2mind is a corpus of phrasal verbs that is constructed to support the examination of the intents and reactions of common situations @xcite . (mohammad, 2018) presents nrc vad lexicon, a corpus of 20,000 english words with human ratings of valence, arousal and dominance while nrc affect intensity lexicon (ail) provides emotional categories and associated real values for approximately 6,000 english words @xcite . socialsent @xcite assigns tokens sentiment scores with contexts taken into consideration. one set of lexicons is constructed from the subreddit r/relationships which we use in our error analyses. other works model relationships, between literary characters @xcite or countries @xcite , described in text using an unsupervised neural model, by mapping them to a latent space. elements of psychology in rocstories are analyzed in @xcite where rocstories are annotated with motivations and emotional reactions of the characters involved. we create social narrative tree with various social and psychological elements embedded in each story and these elements together contribute to the rise and fall of a relationship. ## collaborative construction of social narratives our main contribution in this paper is the construction of a social narrative corpus consisting of short stories each describing the evolution of a relationship between two characters. the corpus is designed to capture how different social behaviors result in different relationship outcomes. to accomplish this goal we follow a fixed narrative structure consisting of five stages: seed (exposi-tion), buildup (rising action), climax, resolution and outlook (denouement). the stages are based on "freytag's pyramid" @xcite and other more recent work analyzing repeating narrative structures @xcite . fundamentally, a minimal story consists of an initial state, a final state and an intermediate state that transitions the initial state to the final state @xcite . more specifically, the intermediate state can be further broken down into individual stages to make a story more appealing and informative. particularly, we adopt the phases of orientation clause, most reportable event (mre) and resolution @xcite to make up the intermediate state, corresponding to the buildup, climax and resolution stages in our framework. the relationship in each story develops as the intensity of the narrative fluctuates. the dataset is constructed in a collaborative way to help capture the impact of different behaviors at each narrative stage. we use mechanical turk, and provide annotators with a partial story created independently, and ask them to complete the next stage. each mturker is limited to provide three responses in total at each stage with all three responses belonging to different prompts. ## summary we introduce social narrative tree, a corpus of 1250 social narratives that encode rich social elements that collectively drive the relationship trajectory in each story. we propose evaluation tasks relationship outlook prediction mcq and resolution prediction mcq using this dataset to measure the extent that systems understand social relationships. the tree structure of the dataset provides convenient and meaningful confounding choices, making the tasks challenging for language models. particularly, the best performance achieved by bert-base is significantly lower than human performance, justifying that relying on textual information is insufficient in understanding social relationships. a possible future work direction is to incorporate various social elements into predicting models so as to capture the nuances of relationship trajectories in different social situations.