Improving Dialog Systems for Negotiation with Personality Modeling (2021)

Abstract: In this paper, we explore the ability to model and infer personality types of opponents, predict their responses, and use this information to adapt a dialog agent’s high-level strategy in negotiation tasks. Inspired by the idea of incorporating a theory of mind (ToM) into machines, we introduce a probabilistic formulation to encapsulate the opponent’s personality type during both learning and inference. We test our approach on the CraigslistBargain dataset (He et al. 2018) and show that our method using ToM inference achieves a 20% higher dialog agreement rate compared to baselines on a mixed population of opponents. We also demonstrate that our model displays diverse negotiation behavior with different types of opponents.

## introduction developing dialog systems for negotiation is challenging since the task requires a combination of good communication skills and strategic reasoning capabilities @xcite @xcite . while recent neural models @xcite @xcite have shown that useful dialogue strategies can be learned from offline corpora, they do not explicitly model the mental state of other agents, which can make it challenging to generate tailored strategies and utterances for different types of opponents. in this paper, we introduce a new framework for generating strategic dialog inspired by the idea of theory of mind (tom) from cognitive science @xcite @xcite . when negotiating with others, humans innately infer the intention of the other party, and guess how their own utterances would affect the opponent's mental state. to emulate this capability in machines, we train a firstorder tom model to predict an opponent's response given the current state and the agent's own possible utterances. this first-order tom model can then be incorporated into dialog agents to enable one-step lookaheads during inference. in order to predict future responses, we model the opponent's personality type as a intermediate variable (z), which can be predicted using the dialogue history. we use this predicted personality, along with the previous state and utterance to calculate the likelihood of the opponent's next state for all possible actions that our agent can take in the current state. this allows us to compute an expected value of return for each action, which is subsequently used to produce a policy for our agent. we propose two variants of our tom-based dialog agent -an explicit version that outputs the opponent type as an intermediate prediction, and an implicit version that models the opponent type as a latent variable. both models can be instantiated as end-to-end neural networks and can be trained using reinforcement learning. our approach differs from existing opponent modeling work @xcite @xcite @xcite in three aspects: 1) it provides strategic benefit during inference which leads to more successful negotiations, 2) it can flexibly adjust the degree of dependence on tom predictions by changing a temperature parameter, and 3) it utilizes text utterances to infer types of opponents, thereby capturing side information (e.g., emotion) that is useful yet absent from standard dialog state transitions. we perform experiments on a modified version of the craigslistbargain negotiation task @xcite , where the agent is matched with dif-ferent opponents from diverse populations (e.g., cooperative, competitive, and aggressive negotiators), without being provided information about their identity. empirically, our method outperforms several baselines on the task by completing more deals and achieving higher utility. for instance, our model achieves about 20% higher dialog agreement rate and utility than a baseline dialog manager trained with reinforcement learning. our analysis reveals that the agent demonstrates diverse negotiation behavior and adapts well to different types of opponents. our approach with explicit characteristic modeling is also similar to the tomnet @xcite , which uses a multi-agent reinforcement learning setting to learn identity embeddings of populations from past trajectories, and predict the mental state of an agent using the current trajectory. however, our first-order tom models for negotiation also take utterances into account, which makes improving upon a base rl policy non-trivial. theory of mind in dialog systems. theory of mind for modeling user personality types and predicting responses has been studied in the context of building user simulators @xcite for training rl-based dialog systems, and to make dialog systems explainable @xcite . recent work on dialog policy learning has employed theory of mind with a focus on specific domains. the recursive mental model (rmm) @xcite was proposed for navigation settings, where questions and answers are generated between a navigating agent and a guiding agent. another approach -answerer in questioner's mind (aqm) @xcite -tackled an answer guessing game with information-theoretic methods. in these domains, the opponents are assumed to be cooperative, while our method is applicable for interacting with both cooperative and competitive opponents. recently, @xcite employed bayesianoptimal monte-carlo planning for end-to-end dialog generation at the utterance level. however, their method only models the latent goal of the opponent instead of potential responses like we do. opponent modeling in rl. apart from dialog systems, opponent modeling has been explored in other multi-agent reinforcement learning settings @xcite @xcite @xcite . our approach differs from these works by: 1) providing strategic benefit during real-time inference, 2) adjusting the degree of dependence on the tom predictions through a temperature parameter, and 3) utilizing text utterances in the dialog to infer types of opponents, thereby capturing side information that is useful yet absent from standard state transitions. ## framework task. we consider a task-oriented dialog setting where there are two agents, a buyer and a seller. the buyer's goal is to purchase the listed item with minimum cost, and the seller's goal is to sell the item at a price as high as possible. the item description is public for both agents, while the target prices are private for both buyer and seller. two agents negotiate in alternating turns until they conclude with an agreement or disagreement. ## first-order theory of mind for dialog the goal of the first-order theory of mind is to predict how a dialog act and an utterance generated by us would affect the reaction of the opponent. as illustrated in figure 1 , suppose that our current dialog state is s t-1 , which consists of the history of past dialog acts and the initial information, as well as the current utterance u -i t-1 from the opponent. the tom model simulates the situations where we take dialog act a i t (e.g., propose(@xmath3, and estimates the probability distribution of the opponents response a i t+1 . by combining actions and states by definition, our first-order tom model estimates the transition probability t (s t+1 |u -i t-1 , s t , u i t ). in practice, the opponent may have different language preferences (e.g., using more aggressive or mild words when countering) and strategies (e.g., tend to insist on their target price or agree to a compromise). the first-order tom can either implicitly capture these personalities by learning the transition t (s t+1 |u -i t-1 , s t , u i t ), or explicitly infer the type of the opponent's personalities z -i first, from the past interaction and the opponent's utterance, i.e., learning an identifier z -i t-1 = f (s t-1 , u -i t-1 ), and then learns the transition based on that information, i.e., t (s t+1 |z -i t-1 , s t , u i t ), to make accurate prediction about opponents reaction. ## dialog managers we compare three hybrid dialog managers combining neural networks and rules to control the flow of dialog: an extra lstm model is used to encode u -1 t-1 in both explicit and implicit tom models, and learn the personality z -i t-1 = lstm(u -1 t-1 , s t-1 ) in explicit tom models which encodes a distribution. note that for all three managers, we applied reasonable hand-crafted rules to prevent unreasonable policies. specifically, the agent will never offer a price below its bottom line and will reject the opponent's offer if it is worse than its bottom line. training and fine tuning. we first train the supervised learning (sl) manager to minimize a loss function for the dialog act predictions dialog act definition example greet say hello or chat randomly. "hello i am interested in buying." inquire ask any question about product, year, price, usage, etc. "how long have you had it? does it come with any of the accessories?" inform provide information about the product, year, usage, etc. "it has the capability to offer great support for those over 6 foot." propose(@xmath4. "the list price is 600 but am willing to negotiate." counter(@xmath5. "i'm sorry, i find homeopathy and any other pseudoscience to be profoundly upsetting as well. i'd be willing to go as high as @xmath130 it is!" disagree(@xmath6. "i cannot take @xmath265) accept final acceptance, no utterance accept reject final rejection, no utterance reject quit leave the negotiation, no utterance quit to obtain the 1st-order tom policy for the inference, we approximate the sum (expectation) in π tom by monte carlo sampling with the generator, and discretize the price in a normalized price range. in practice, we found quantizing the price range with 100 units is a good balance between time comsumption and the quality of approximation. ## experimental setup we test our tom negotiation framework on the craigslistbargain @xcite , which contains 6682 human-human dialogs between a buyer and a seller alternately bargaining for the price of an item on craigslist. ontology. we redesign the ontology of the craigslistbargain dataset to support a more diverse dialog act than the original coarse dialog acts @xcite , which can reflect more ways of mental state change in a negotiation. we used the microsoft language understanding intelligent service (luis) to relabel the dataset , and merged some similar label types, such as insist and vagueprice into counter-noprice, and intro and great into greet. all fifteen dialog acts after our modifications are in reward function design. we set the reward r i for the agent i to be a linear function of the final price, such that the buyer achieves maximal reward of 1 at its target price, the seller achieves maximal reward of 1 at the listing price, and both agents receive zero rewards at the midpoint of the listing price and the target price. when there is no deal, both agents receives equivalent penalty. ## results improvement of dialog policy. we evaluate sl+rule, rl, and our tom model on a mixed population for 4352 dialogs, which contains about 630 dialogs for each population. as shown in implicit vs. explicit models. we found that the implicit tom model can also achieve better ag and ut than the baselines for all populations. but the overall performance is slightly worse than the explicit tom model. this can be explained by the fact that the explicit model has more information about the population type during training. one may worry about the potential error cascade issue the explicit tom models, as we see in figure 2 , the top 1 accuracy of the identifier in the explicit model is only 69%, though it is significantly above the chance. our experiment show that even with an imperfect identifier, the explicit model can still outperform an implicit model, which is directly optimized for better performance. population-aware strategies. as when opponents are cooperative and easy to negotiate with, our tom model can achieve much better agent utility by taking advantage of losing some dialog fairness. this implies our tom model is able to utilize different characteristics of the opponents in the strategy generation. we provide some sample dialogs from the explicit tom model in visualization of population embeddings. in figure 3 , we show the pca visualization of the normalized latent variables in both explicit and implicit tom models. the latent variables are extracted from one layer before the output of the identifier or its equivalence in the implicit model. the explicit tom model learns embeddings encoding different opponent populations, as the major variances of variable are captured by the difference of opponent populations. however, without extra supervision, the extraction of the population identity is difficult in the implicit tom model. further analysis shows that the variances of the latent variables in the implicit tom model are mainly explained by intent types. we include more detailed analysis and t-sne visualization in appendix b. ## conclusion in this work, we proposed a novel framework to integrate the concept of theory of mind (tom) into generating task-oriented dialogs. our approach provides the ability to model and infer personality types of opponents, predict changes in their mental state, and use this information to adapt the agent's high-level strategy in negotiation tasks. we in-troduced a probabilistic formulation for first-order tom and introduce two ways to incorporate it into a dialog agent, by 1) explicitly and 2) implicitly modeling the personality of the opponent. we tested our approach on a modified version of the craigslist-bargain dataset @xcite with diverse opponents. our experiments show that our method using tom inference achieves about 20% higher dialog agreement rate and utility compared to baselines on a mixed population of opponents. when negotiating with the cooperative opponents, the improvement of agreement rate is 54%. some directions for future work include developing efficient schemes to approximate the value computation for future states, exploring higher orders of tom, as well as a tighter integration of tom into utterance generation and processing.