Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions (2023)

Abstract: A natural way to design a negotiation dialogue system is via self-play RL: train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data. Although this procedure has been adopted in prior work, we find that it results in a fundamentally flawed system that fails to learn the value of compromise in a negotiation, which can often lead to no agreements (i.e., the partner walking away without a deal), ultimately hurting the model’s overall performance. We investigate this observation in the context of DealOrNoDeal task, a multi-issue negotiation over books, hats, and balls. Grounded in negotiation theory from Economics, we modify the training procedure in two novel ways to design agents with diverse personalities and analyze their performance with human partners. We find that although both techniques show promise, a selfish agent, which maximizes its own performance while also avoiding walkaways, performs superior to other variants by implicitly learning to generate value for both itself and the negotiation partner. We discuss the implications of our findings for what it means to be a successful negotiation dialogue system and how these systems should be designed in the future.

## introduction negotiation is a complex mixed-motive interaction, involving motivations for both self-serving as well as cooperative and socialistic behaviors. a successful negotiator must not only learn to extract concessions from the partner but also to make concessions in order to reach an agreement. maintaining this balance between self-interest and the interests of negotiation partners makes it a challenging task for automated dialogue agents. if an agent tries to take too much without any compromise, this can push the partner to walk away without an agreement, hurting the outcomes for both players. one natural way to design such a system is through self-play reinforcement learning (rl). ## related work historically, negotiation has been studied across several disciplines, including game theory @xcite and psychology @xcite . more recently, there has been an increasing interest in human-agent negotiations as well @xcite . extensive research has examined the effects of both agent and human personality in negotiation and related decision-making topics/personality) tasks @xcite @xcite . however, most prior efforts analyze interactions based on structured communication channels such as through a menu of options @xcite . instead, @xcite studied participants' extreme reactions to unfair offers by a selfish chatbot in an ultimatum game. we contribute to this line of research by exploring diverse dialogue agent personalities and studying their impact on negotiation performance. several dialogue datasets @xcite @xcite have fueled research into designing negotiation dialogue systems. rl has been a popular technique of choice in this space @xcite . yang et al. (2021) modeled the personality of the partners by a one-step dialogue-act look ahead in a buyer-seller negotiation domain and found that it leads to a higher agreement rate. complementary to this, our work investigates the impact of diverse agent personalities by modifying both the underlying reward and the partner personality for rl training. in addition, we focus on using selfplay rl directly at the utterance level, which does not need additional annotations or separate parser and generator modules that are relatively difficult to design for general multi-issue negotiation tasks. other recent work has also explored the incorporation of additional annotations such as dialogue acts and strategy labels @xcite . nevertheless, our paper focuses on designing agents for mixed-motive interactions, which is fundamental to any underlying negotiation context and model architecture. ## methodology we focus on bilateral multi-issue negotiations which involve a fixed set of issues (e.g., books, balls, and hats in the dealornodeal dataset @xcite ). each issue has a predefined quantity along with a random value (potentially different) assigned for every player. the players engage in a dialogue to reach an agreement -a possible division of all the available items in which they try to maximize the total value of the items that they get. our goal here is to develop techniques so that the trained dialogue models learn to make concessions (e.g., by offering deals that help the partner) for their partners apart from just learning to extract concessions from them. as discussed earlier, this mixed-motive behavior is a fundamental expectation from a practical negotiation dialogue system. to achieve this, we propose two complementary techniques -first, where we explicitly incorporate the partner's performance into the reward function of the rl agent, and second, where the model implicitly learns to make concessions by interacting with a specific partner during training. we start by describing our base rl framework and then discuss the two proposed techniques. ## experimental design we proposed two ways of training dialogue models that capture the mixed-motive nature of negotiations: 1) explicitly, by varying the reward function for the rl algorithm (section 3.2.1), and 2) implicitly, by varying the partner with which the rl model is trained (section 3.2.2). the primary research question we aim to answer is what variation leads to superior performance with human partners. we first describe the dataset and the study design, followed by results in section 5. dataset: we use the dealornodeal dataset @xcite , which is based on the multi-issue bargaining task @xcite design. the dataset uses a simplistic design involving 3 issues (books, hats, and balls), and has been a popular choice for research in negotiation dialogue systems. it comprises 5808 dialogues in english based on 2236 unique scenarios, where a scenario refers to the available items up for grabs and their corresponding values for the two players. in each scenario, there is a fixed quantity of each issue, and players are randomly assigned a point value before the negotiation for each of the 3 issues. the goal of the dialogue is to reach an agreement on the possible division of all the available items, where each player strives to maximize the total value of the items that they get. the maximum possible value for a player is 10. however, if no agreement is reached, then both players end up with 0 points. nearly 80% of the dialogues end in agreement, with an average of 6.6 turns per dialogue and 7.6 words per turn. we use the same splits as the original dataset paper to train our dialogue agents. study design: we design a 2 x 3 study based on the strategies described in section 3. we use a three-stage process to develop the 6 agent personalities: stage 1: develop a supervised likelihood model, following @xcite . stage 2: train two rl dialogue agents by varying the reward using the selfish and fair utility functions selected from ## results to analyze the overall performance, we conducted 2 (reward r: selfish vs. fair) x 3 (partner p: supervised vs. selfish vs. fair) anovas on the points earned in the negotiation. first, we found no significant differences in the points earned by the dialogue agents. however, the agent reward r significantly affected human points (f(1, 577) = interestingly, human partners earned more points with m @xmath0, which also led to more joint points, bringing it on par with (or even better than) fair agents. a plausible explanation is that since the m @xmath1, it is better exposed to the repercussions of not making concessions for the partner since the agent receives a 0 reward if there is no agreement (within 20 utterances). thus, the agent learns to "give in" in order to avoid no agreements. next, we test this explicitly by analyzing the %age of walkaways for each agent. ## discussion going beyond the typical reward formulations used in the literature, this is the first instance of leveraging prior economics theories to explicitly incorporate the partner performance within the reward of the selfplay rl negotiation agent. our formulation provides a systematic and general way to train mixed-motive agents with diverse personalities ( the overall points scored in we especially note the superior performance of m @xmath3. trained with a simplistic reward that maximizes its own performance, m @xmath4. this observation aligns with the philosophy of the 'invisible hand' in economics by adam smith @xcite , which suggests that self-interested players are implicitly led (as if by an invisible hand) to cooperate and take actions that benefit others. ## conclusion we devised two variations of the standard self-play rl technique to inculcate the mixed-motive nature of negotiation into the dialogue agents. the first approach worked by varying the reward function and thereby, by explicitly pushing the model to take the partner's performance into account. in the second approach, we modified the personality of the partner agent during training, which allowed the rl agent to discover the mixed-motive nature of the task implicitly. we find that both techniques hold promise, with an especially strong performance from the agent that is trained with a selfish reward and a self-interested partner. this agent not only improves on the agreement rate but also learns to discover offers that create value for its partner without hurting its own points significantly.