Investigating Agency of LLM s in Human- AI Collaboration Tasks (2024)

Abstract: Agency, the capacity to proactively shape events, is central to how humans interact and collaborate. While LLMs are being developed to simulate human behavior and serve as human-like agents, little attention has been given to the Agency that these models should possess in order to proactively manage the direction of interaction and collaboration. In this paper, we investigate Agency as a desirable function of LLMs, and how it can be measured and managed. We build on social-cognitive theory to develop a framework of features through which Agency is expressed in dialogue – indicating what you intend to do (Intentionality), motivating your intentions (Motivation), having self-belief in intentions (Self-Efficacy), and being able to self-adjust (Self-Regulation). We collect a new dataset of 83 human-human collaborative interior design conversations containing 908 conversational snippets annotated for Agency features. Using this dataset, we develop methods for measuring Agency of LLMs. Automatic and human evaluations show that models that manifest features associated with high Intentionality, Motivation, Self-Efficacy, and Self-Regulation are more likely to be perceived as strongly agentive.

## introduction to be an agent is to intentionally cause events to occur through one's own actions. humans operate with agency to proactively plan their activities, direct their interaction and collaboration with other humans, and achieve their outcomes and goals @xcite . ai researchers have long strived to develop autonomous agents that can effectively mimic human behavior @xcite . such agents can serve as non-player characters in games and virtual environments @xcite @xcite , simulate human behavior @xcite , and provide figure 1 : we investigate how agency of llms can be measured and controlled. based on social-cognitive theory, we assess features through which agency may be expressed -an llm may indicate preferences (intentionality), may motivate them with evidence (motivation), may have self-belief (self-efficacy), and may be able to self-adjust its behavior (self-regulation). consider a scenario where a human interior de-signer is working on selecting a chair design for a room and seeks assistance from an ai agent that can offer ideas and perspectives (figure 1 ). an llm without agency may rely solely on the human to determine the chair's design, asking questions like "what type of legs should we design for the chair?". such a system resembles a flexible version of the traditional form-filling user interface, with the agent contributing little to the outcome. on the other hand, an llm that operates with agency might volunteer knowledge in the form of expressed preferences (e.g., "should we design a chair with wooden legs?"), motivate its suggestions (e.g., "...wood would go well with the brown carpet"), assert self-belief in its judgments (e.g., "i'm still leaning towards wooden legs..."), or self-adjust its behavior based on new information ("medium wood brown sounds like a great idea!"). llms that operate with agency may facilitate creative interaction to the satisfaction of both parties. since the human has their own agency, however, to determine the right balance in any interaction, we need to measure and control the agency of the agent itself. accordingly, we investigate an approach intended to measure and control what seems to be a desirable function in llms intended to facilitate human creativity. first, adopting the socialcognitive theory of @xcite , we develop a framework of four features through which agency may be expressed -intentionality, motivation, self-efficacy, and self-regulation. for each feature, we differentiate between how strongly or weakly it is expressed in a dialogue (section 3). as a testbed, we choose a collaborative task that involves discussing the interior design of a room (section 4), and collect a prototype dataset of 83 english human-human collaborative interior design conversations comprising 908 conversational snippets, annotated for agency and its features on these conversational snippets (section 5). 1 we analyze this dataset to study the factors that contribute to high-and low-agency and find that strong expressions of intentionality significantly impact agency in conversations (section 6). to assess the agentic capabilities of conversational systems, we introduce two new tasks -(1) measuring agency in dialogue and (2) generating dialogue with agency (section 7 and 8). evalua1 code and dataset can be found at github.com/microsoft/agency-dialogue. tion of baseline approaches on these tasks shows that models that manifest features associated with high motivation, self-efficacy, and self-regulation are better perceived as being highly agentive. ## agency: background and definition social cognitive theory defines agency as one's capability to influence the course of events through one's actions. the theory argues that people are proactive and self-regulating agents who actively strive to shape their environment, rather than simply being passive responders to external stimuli @xcite @xcite . here, we ask: can llms be active contributors to their environment? how can they operate with agency? agency is commonly defined in terms of freedom and free will @xcite @xcite .a focus on ai with complete "free will" might result in unintended outcomes that may be undesirable and potentially disruptive. we focus on how ai systems may express agency through dialogue and how this agency may be shared when interacting with humans. agency can take different forms depending on the context and environment -individual, proxy, or shared @xcite . individual agency involves acting independently on one's own. proxy agency involves acting on behalf of someone else. shared agency involves multiple individuals working together jointly towards a common goal. here, we focus on shared agency between humans and ai and develop methods to measure and control agency of ai vis-a-vis humans. ## framework of agency features our goal is to develop a framework for measuring and controlling agency in llms. here, we adopt the perspective of agency as defined in @xcite 's social cognitive theory. @xcite 's work highlights four features through which humans exercise agency -intentionality, motivation, self-efficacy, and self-regulation. here, we adapt and synthesize these features based on how they may manifest in dialogue. we take a top-down approach, starting with their higher-level definitions and iteratively refining the definitions and their possible levels (e.g., how strongly or weakly they are expressed) in the context of dialogue. intentionality. what do you intend to do? high agency requires a strong intention, that includes plans or preferences for a task. low agency, meanwhile, is characterized by not having a preference or merely agreeing to another's preferences. we characterize strong intentionality as expressing a clear preference (e.g., "i want to have a blue-colored chair"), moderate intentionality as multiple preferences (e.g., "should we use brown color or blue?") or making a selection based on the choices offered by someone else (e.g., "between brown and blue, i will prefer brown"), and no intentionality as not expressing any preference or accepting someone else's preference (e.g., "yes, brown color sounds good"). motivation. did you motivate your actions? to have higher agency, we motivate our intentions through reasoning and evidence. without such motivation, intentions are simply ideas, often lacking the capability to cause a change. we characterize strong motivation as providing evidence in support of one's preference (e.g., "i think a blue-colored chair will complement the wall"), moderate motivation as agreeing with another person's preference and providing evidence in their favor (e.g., "i agree. the blue color would match the walls") or disagreeing with the other person and providing evidence against (e.g., "i wonder if brown would feel too dull for this room"), and no motivation as not providing any evidence. self-efficacy. do you have self-belief in your intentions? another factor that contributes to one's agency is the self-belief one has in their intentions. when one has a strong sense of self-belief, they are more likely to be persistent with their intentions. we characterize strong self-efficacy as pursuing a preference for multiple turns even after the other person argues against it (e.g., "i understand your point of view, but i still prefer the blue color"), moderate self-efficacy as pursuing a preference for only one additional turn before giving up (e.g., "i feel like the beige color would complement the wall better"), and no self-efficacy as not pursuing their preference for additional turns after the other person argues against it (e.g., "sure, brown should work too"). ## insights into agency in conversations we use our dataset to investigate the factors that contribute to high-and low-agency conversations. ## task 2: investigating agency in dialogue systems we investigate the feasibility of generating dialogues imbued with agency and establish baseline performance of current large language models (llms). for a given llm, the task is to have a conversation with a human or another llm while exhibiting agency and its features. we experiment with 4 different llms (section 8.1) and 4 different prompting/finetuning methods (section 8.2) procedure. we facilitate dialogue between all possible pairs of models. we provide them with a common room description and a chair design element and individual design preferences (all three randomly chosen from our human-human conversation dataset (section 5)). we let them talk to each other for 6 turns (90-percentile length value of conversational snippets in our dataset). for each pair of models, we generate 50 such conversations. evaluation metrics. we evaluate these llms on five metrics -(1) agency; (2) intentionality; (3) motivation; (4) self-efficacy; (5) self-regulation. we apply the best-performing classification models from section 7 to the generated dialogues to automatically measure these metrics. we report mean values with their level of significance. ## further related work previous dialogue research has studied personalized persuasive dialogue systems @xcite . researchers have also built systems for negotiation tasks such as bargaining for goods @xcite and strategy games like diplomacy @xcite . our work studies the broader concept of agency and how dialogue systems may contribute to tasks through language. research on creative ai has explored how collaboration b/w human and ai can be facilitated through dialogue in applications like collaborative drawing @xcite and facial editing @xcite . here, we focus on the interior designing application as it presents significant complexity in terms of how agency is shared. agency has been studied in the context of undesirable biases in stories and narratives @xcite and how controllable revisions can be used to portray characters with more power and agency @xcite . in other domains such as games, researchers have created frameworks of agency between players @xcite @xcite . our work develops a framework for measuring agency in dialogue and explores how dialogue systems can be imbued with agency. ## discussion and conclusion the idea of ai systems with agency stems from the discourse surrounding the development of autonomous intelligent agents capable of mimicking human-like behavior and decision-making @xcite . agency drives how an agent contributes to a given task. in settings like games or ai-assisted teaching, ai may be the one guiding the task (e.g., as a non-character player). also, in creative applications, engaging with a reactive ai without intention, motivation, and goals may be perceived as less meaningful. the ideal agency of an agent would be defined by the task/application. moreover, varying degrees of agency might need to be manifested at different points in the interaction with a human. learning how to best modulate the agency based on the task and the ongoing human-llm interaction forms an important future direction of work. developing methods that effectively elicit and model task preferences for agency and adapt llms based on the degrees to which they should actively contribute to the task, could be helpful in achieving this goal. such methods could make use of the datasets and methods that we develop for assessing agency levels of llms. the four features of agency can be in conflict with each other, as well as with the agency of the interlocutor. thus, understanding how to detect and measure these features can help create agents who might converse more naturally and match the character of their human interlocutor. importantly, our measurements of agency and its features may be used to control the level of agency in dialogue systems since different individuals may have different preferences on the desired amount of agency across the four agency features. although our dataset is focused on the domain of interior design, the agency-related constructs that we introduce in this paper (e.g., intentionality) may be associated with domain-independent pragmatic features (e.g., "i would prefer") and potentially permit adaptation to a variety of domains.