Identifying Power Relations in Conversations using Multi-Agent Social Reasoning (2025)

Abstract: Large language models (LLMs) struggle in social science domains, where critical thinking and human-level inference are crucial. In this work, we propose a multi-agent social reasoning framework that leverages the generative and reasoning capabilities of LLMs to generate and evaluate reasons from multiple perspectives grounded in social science theories, and construct a factor graph for inference. Experimental results on understanding power dynamics in conversations show that our method outperforms standard prompting baselines, demonstrating its potential for tackling hard Computational Social Science (CSS) tasks.

## introduction understanding conversational dynamics is a multifaceted problem, which requires situating the interlocutors' utterances in a specific social context such that the intent behind them, and the reaction to them, could be revealed. past social science work has studied the interaction between conversations and social relationships @xcite , language use in different social situations @xcite and social identities @xcite . this paper focuses on the connection between a specific social indicator, power relations, and several aspects of language use, namely style (e.g., apologetic, assertive), content (e.g.,judgments over dialog acts), coordination (e.g., steering and setting the tone) and engagement (e.g., active participation). identifying power relationships in conversations, taking place in different settings such as organizational emails, online forums and chats, has been studied extensively in the nlp literature @xcite @xcite @xcite and was typically formulated as a supervised learning problem focusing on different aspects such as lexical features @xcite , linguistic coordination @xcite or conversational structure @xcite . the recent paradigm shift in nlp, moving away from task-specific supervised learning and towards broader-purpose llms, raises an open question -can llms understand such social dynamics, without dedicated training? initial results for conversation analysis tasks (including power-relation prediction) were mixed @xcite motivating further research in this area. in this paper we argue this question should be studied with more nuance. instead of directly accounting for the complex interactions between social settings and conversational behaviors via llm autoregressive (i.e., greedy) decoding, we argue that llms can demonstrate their ability to understand conversational data by focusing on different aspects of conversational behavior and raising hypotheses on how they provide evidence for the power relationship between interlocutors. specifically, building on prior work in social science, we identify style, content, coordination, and engagement as key aspects that capture the implicit dynamics of conversations, including speakers' social status @xcite , power relations @xcite , and the overall conversation flow @xcite . we formulate the problem as a multi-agent social reasoning task (see @xcite for an overview), in which each interlocutor is associated with an llm-based agent advocating for their high power status in the conversation, by providing aspect-specific reasons and rebuttals in response to the other side's reasons. we define llm-based assessment functions for scoring the strength of these claims (sec. 2.1) and organize them based on their argumentation structure; we then compile this structure into a factor-graph (sec. 2.2) and perform probabilistic reasoning over that structure @xcite to find the most probable power-relation con-sistent with that structure. figure 1 provides an illustration of our overall framework. we conduct our experiments over the icsi meeting corpus @xcite by sampling conversation snippets and applying our multi-agent reasoning architecture over them. 1 these are very challenging settings, as each snippets captures only a handful of relevant behaviors, which are often misleading as the data consists of informal workrelated interactions between students, postdocs and faculty. this is reflected in our experimental results, showing that the performance of both human and direct llm prompting is worse than random. augmenting the llm prompts with the generated reasons and rebuttals leads to even worse performance, as the model is not able to effectively prioritize between them. however, when applying our argumentation-based reasoning framework, the model can detect inconsistencies and prefer reasons that uniquely identify one of the sides, leading to an 8 points improvement compared to human performance and 5 compared to direct llm prompting. ## multi-agent social reasoning our framework incorporates a generation step and an inference step. in the generation step, we use two llm agents to produce reasons and rebuttals for each prediction candidate. for each reason and rebuttal, a separate llm assigns a score indicating its likelihood of validity and grounding. in addition, the llm also evaluates the reasons' relationships with the other generated reasons. based on these scores and interdependencies, we construct and resolve a factor graph to finalize the prediction. ## results ## discussion and summary this paper presents a multi-agent probabilistic reasoning framework for analyzing conversations. we intentionally structure the agents' interactions to create an argumentation structure based on aspectbased reason-rebuttal pairs and capture global consistency between them, using llm judgments. our results demonstrate that each aspect of the model enhances performance, highlighting the potential of llms to transform social analysis tasks-provided they are leveraged through careful, structured problem decomposition. looking forward, we believe that this paper is only a first step in this direction, motivating several future research directions. first, our framework can be generalized to a broader range of social reasoning tasks. second, we aim to explore the connection between our system and formal theories of argumentation (fta) @xcite @xcite . our conjecture is that our structure can be mapped to a subset of fta (i.e., our rules, such as reasonrebuttal, naturally align with the concept of defeaters in fta). this connection has the potential to bridge llm-based reasoning with theoretically grounded argumentation frameworks.