Perspective Taking through Generating Responses to Conflict Situations (2024)

Abstract: Although language model performance across diverse tasks continues to improve, these models still struggle to understand and explain the beliefs of other people. This skill requires perspective-taking, the process of conceptualizing the point of view of another person. Perspective taking becomes challenging when the text reflects more personal and potentially more controversial beliefs.We explore this task through natural language generation of responses to conflict situations. We evaluate novel modifications to recent architectures for conditioning generation on an individual’s comments and self-disclosure statements. Our work extends the Social-Chem-101 corpus, using 95k judgements written by 6k authors from English Reddit data, for each of whom we obtained 20-500 self-disclosure statements. Our evaluation methodology borrows ideas from both personalized generation and theory of mind literature. Our proposed perspective-taking models outperform recent work, especially the twin encoder model conditioned on self-disclosures with high similarity to the conflict situation.

## introduction despite the steadily increasing performance language models achieve on a wide variety of tasks, they continue to struggle with theory of mind, or the ability to understand the mental state of others @xcite . a range of theory of mind tasks have been studied in developmental psychology, including diverse desire, diverse belief, knowledge-access, false belief, and hidden emotion @xcite . results consistently show that the hidden emotion, or understanding the difference between felt and displayed emotions, is the most difficult of these tasks @xcite . language assists in the development of theory of mind, as it facilitates the exploration of mental states @xcite . this ability is central to much of human interaction and could provide many benefits for language models as well, as being able to foresee the reactions of others allows us to better decide which action to take next. this could help language models generate responses that are safer @xcite , in particular for healthcare applications @xcite , or more personalized, e.g. to sound more empathetic @xcite , or provide targeted explanations @xcite . in fact, there is a growing interest in a perspectivist approach to many natural language processing (nlp) tasks, which emphasizes that there is no single ground truth @xcite @xcite . this is a more common view in generation tasks, as it is easier to see that multiple translations or continuations of a dialog are correct. however, @xcite emphasized the need to interpret language with it's personal contextual factors to create higher performing personalized systems. @xcite similarly argue that additional contextual information should be incorporated in such models, particularly for natural language generation (nlg). we construct a corpus to study perspective taking through generating responses to conflict situations. an example from our corpus can be found in figure 1 . we see a user asking if they did something wrong in a conversation with their girlfriend about whether or not to terminate a pregnancy. on the right, there are two responses from other users with different judgments of the situation (reasoning and verdict nta/yta). on the left, we see self-descriptive statements of each user. author y appears to be more family-oriented than author x which may impact their judgement of the situation. our first research question is, rq1: how should we evaluate perspective taking through the lens of nlg? we develop a novel evaluation of by asking humans to rank the human response, model output, and a distractor human response, combining approaches from persona consistency @xcite and theory of mind work @xcite . next, we ask rq2: do tailored, usercontextualized architectures outperform large language models (llms) on this task? we design two transformer architectures to embed personal context and find that our twin encoder approach outperforms llms. lastly, we ask rq3: what user information is most useful to model perspective taking? experiments with varied user context showed that self-disclosure statements semantically similar to the conflict situation were most useful. furthermore, by training a generative model to produce perspectives, we outperform recent supervised data perspectivist models also on verdict prediction. our corpus contains 95k judgements of conflict situations written by 6k authors filtered from social-chem-101 and is extended with 20-500 self-disclosure statements per author. we will release our corpus, code, and human evaluations. ## related work perspective-taking is part of the process of understanding others mental states and theory of mind is fundamentally based on the understanding that individuals have different beliefs and emotions. benchmarks for theory of mind in artificial intelligence sometimes take the form of multiple choice questions about false belief @xcite . these are context specific, as a situation and knowledge of involved parties is described, but not unique to individual people. when we look at emotions, the most difficult of theory of mind tasks for children @xcite , different people may feel differently because of their unique perspective and experiences. as such, our work relates to work on personalized understanding and generation. recent models, such as roberta or gpt-3, struggle with solving simple perspective-taking toy tasks @xcite @xcite , including the false belief test @xcite . currently, the best llms (such as gpt-4 or llama2-70b) are to a limited extent capable of perpective-taking-based reasoning and chain-of-thought explanations in social commonsense scenarios @xcite @xcite . however, these attempts has been heavily criticized by psychologists as over amplified by the ai community while very basic in design, likely exhibiting shortcut learning @xcite . children's theory of mind ability appears associated with "exposure to rich discourse about thoughts, feelings, and intentions" @xcite . ## dataset of social situations we used the dataset of @xcite as the foundation of our work, as it addresses the emotional and social aspects of perspective taking, while requiring knowledge of personal beliefs. the authors collected data from reddit, an online platform with many separate, focused communities called subreddits. the data is from the aita subreddit, where users share descriptions of social situations that they are involved in and ask members of the community for their opinions. these members assess if the poster is the wrongdoer in the described situation. they provide a verdict in the form of "you're the asshole" (yta) or "not the asshole" (nta). the dataset was filtered from forbes et al. (2020)'s social-chem-101 corpus but also includes the post title, full text, all comments, and their corresponding authors. we refer to the post title as the situation, as the title is usually a short description of the conflict situation. the comments are preprocessed in order to extract those that contain a verdict of yta or nta, 1 and others were removed. in order to extract verdicts, they manually created a set of keywords for both classes and filtered the comments to remove these expressions. the initial dataset contains 21k posts, and 364k verdicts (254k nta, 110k yta) written by 104k different authors. ## problem formulation our task considers as a data point, a post that contains a summary of the situation description, a comment of the post containing a personal verdict about the situation, and the author of the verdict jointly with the corresponding self-disclosures. therefore, for our generation task, we have three components: (i) the input sequence which corresponds to the main post, (ii) the target output sequence which corresponds to the comment containing the verdict, and (iii) auxiliary information. the auxiliary information can be self-disclosure statements (s) or comments (c), and allows us to address rq3: what information is most useful for perspective taking? in order to tackle this research question, we formalize the following task. for a given situation post s written from a random author, we have a set of comments c @xmath0, c s a 2 , . . . , c s an } written by n different authors. each post describing a situation s contains many comments c s a @xmath1, and an author a has many comments c s i a on different posts s i . hence, as we have different target outputs, for the same input sequence, we need additional information to condition our model. the generation task can be formalized as p(c s a |s, a). for each author a the model can take advantage of p @xmath2, p a 2 , . . . , p a k }, where p a i , denotes the i-th personal context for author a. we describe two different methods to extract a set of k disclosure statements for each user in the dataset. ## methodology after formally describing the base transformer (5.1), we outline the modifications implemented to the encoder-decoder model in order to incorporate auxiliary information. we compare our twin encoder (5.2) and style decoder (5.3) models to large language models to answer rq2: do tailored architectures outperform llms? ## experiments in our experiments, we utilize two base models, that follow an encoder-decoder architecture. we also compare to three recent methods for incorporating personalized information: priming. this method was originally used in recurrent neural networks. it initially passes information about a user through the model, and then the text that needs to be classified @xcite . in our approach, we sample a number of sentences from a user's history that are up to a maximum number of m tokens in order to fit into the context window of the model. then, we concatenate this sampled text for each user at the beginning of the input text for the encoder during training. user id. in this approach, we append a special user token, at the end of the input text for the encoder during training. several methods incorporate the user id to learn user representations in the model @xcite ). however, one drawback of this method is that it cannot generalize to unseen users during test time. paa model. we also adapt the recent paa model @xcite , which has shown superior performance on the personachat task, to run on our dataset and compare with our proposed architectures. for the paa model, we utilize only the self-disclosure sentences as an auxiliary input. ## results and analysis similar self-disclosure statements were most useful for perspective taking. in in the twin encoder architecture performed best for both models. the key difference between the two architectures is that information about the situation and the auxiliary context is combined. in the twin encoder, information is combined before the decoder performs the cross-attention with the encoder states, while in the style decoder, the information is combined after the decoder. hence, in our case, it proved to be more useful to use only one decoder layer and combine the information earlier, as opposed to previous work @xcite . in general, the flant5 variations proved to perform better, which may be attributed to the size difference of the base models (250m vs 140m). in addition, flant5 + te (ps) performs better than the paa model despite having fewer parameters. moreover, flant5 + te (ps), has the most diverse responses, even across situations, with scores close to the original responses on reddit. 2 among prim- llms performed worse than tailored architectures. model fluency ↑ relevance ↑ bart + te (s) 43% 42% flant5 + te (s) 30.6% 25.6% flant5 + sd (c) 41.7% 40% human evaluation confirms superiority of our model, showing gap between human performance. in generated perspectives are competitive with perspectivist classification. as a follow-up analysis, we evaluated our model on the perspective classification task from plepi et al. (2022) by extracting the labels (nta/yta) from the generated comments. we use their splits; the verdict split (default for our generative experiments), situation and author splits, which have disjoint sets of situations and authors respectively, across train, validation, and test. ## conclusions as we make progress on improving perspective taking ability, we will need to have models that take additional contextual information into account, especially personal contextual factors. we examined this through the lens of generating perspectives on conflict situations. first, we studied which information is most useful to perspective taking, finding that self-disclosure statements that were similar to the situation were most useful. second, we compared tailored architectures to llms, including two novel methods, finding that our twin encoder architecture outperformed recent work, flant5 and llama2 models. lastly, we found that previous consistency evaluation metrics were inadequate and proposed a human ranking evaluation that includes similar human responses. additionally, we found that our generation model performed competitively with previous work on perspective classification. we extended an existing corpus with 20-500 selfdisclosure sentences per author and will release our code and corpus upon publication.