Opponent Modeling in Negotiation Dialogues by Related Data Adaptation (2022)

Abstract: Opponent modeling is the task of inferring another party’s mental state within the context of social interactions. In a multi-issue negotiation, it involves inferring the relative importance that the opponent assigns to each issue under discussion, which is crucial for finding high-value deals. A practical model for this task needs to infer these priorities of the opponent on the fly based on partial dialogues as input, without needing additional annotations for training. In this work, we propose a ranker for identifying these priorities from negotiation dialogues. The model takes in a partial dialogue as input and predicts the priority order of the opponent. We further devise ways to adapt related data sources for this task to provide more explicit supervision for incorporating the opponent’s preferences and offers, as a proxy to relying on granular utterance-level annotations. We show the utility of our proposed approach through extensive experiments based on two dialogue datasets. We find that the proposed data adaptations lead to strong performance in zero-shot and few-shot scenarios. Moreover, they allow the model to perform better than baselines while accessing fewer utterances from the opponent. We release our code to support future work in this direction.

## introduction negotiations are key to our everyday interactions such as allocating available resources, salary decisions, business deals, and legal proceedings. the ability to effectively negotiate is also critical for automated systems deployed in complex social scenarios @xcite . this enables these automated systems to engage in strategic conversations @xcite and also assists in pedagogy by making social skills training more accessible @xcite . @xcite . the negotiation involves 3 issues: food, water, and firewood, each with 3 items that must be divided among the two players. from the perspective of player p1, the task of opponent modeling considered in this work involves inferring the priority order of the opponent p2 from the interaction between the two. consider the scenario presented in figure 1 . two participants role-play as campsite neighbors and engage in a multi-issue negotiation @xcite over three issues: food, water, and firewood @xcite . each negotiator has their own priority order depending on the relative importance assigned to each issue. the goal of the negotiation is to divide the available quantities of food, water, and firewood packages, such that each package is assigned to exactly one of the players in the final agreement. the priority order of the opponent is typically unknown to negotiators beforehand, and can only be inferred based on the interaction between the two. prior work argues that understanding what one's opponent wants is one of the key aspects of successful negotiations @xcite ). an accurate model of the opponent can enable a dialogue system to roll out offers that work for both parties, which has implications on both its objective performance such as the final points scored from the agreed deal, and the subjective performance such as opponent's satisfaction and affinity for the dialogue system. this can also aid in pedagogy by allowing the system to provide concrete feedback to students who fail to incorporate the priorities of their opponents @xcite . discovering these priorities from an interaction with an opponent is usually referred to as opponent modeling in the context of multi-issue negotiations. information about an opponent's priorities can primarily be gathered from their preference and offer statements @xcite . sharing preferences by explicitly mentioning 'we need water' or more implicitly -'we like to go on runs' can provide information that water is of high priority to the negotiator. further, offers such as 'i would like two food items and one water' can imply that food is of a higher priority than water. building techniques for opponent modeling that are useful in realistic chat-based negotiations poses several key challenges: 1) it is non-trivial to directly use counting-based methods on these preference and offer statements, which are common in prior work that does not use natural language, such as agent-agent negotiations @xcite and human-agent negotiations based on button clicks @xcite , 2) to alleviate this problem for language-based interactions, prior work has resorted to gathering additional utterance-level annotations to convert the desirable information into a more structured format, that can then be used with counting methods @xcite . however, this approach remains expensive, requires expertise, and hurts generalizability. further, these annotations are unavailable for systems that are deployed to end users, needing a separate nlu module which can potentially lead to error propagation in the downstream dialogue system pipeline, and 3) some real-world applications require the system to guess the opponent's priorities with only partial dialogue so as to inform the future decision process of the system -a scenario which has not been well explored in prior works. to address these challenges, we propose a transformer-based @xcite hierarchical ranker for opponent modeling in negotiation dialogues. our model takes a partial dialogue as input and guesses the opponent's priority order. instead of relying on utterance-level discourse information, we devise simple and effective ways to project related data sources to this task. as opposed to multi-task learning which typically involves task-agnostic and task-specific parameters and back-to-back fine-tuning procedures that suffer from catastrophic forgetting issues, our adaptations augment the training data available to the model, allowing end-to-end joint learning and parameter sharing. we summarize our contributions below: ## problem formulation consider a negotiation c between two parties over m issues. we define the problem from the perspective of a specific negotiator (referred to as self, hereafter), and aim to predict the priority order of the opponent (see figure 1 ). assume that c contains an alternating sequence of n utterances between the negotiator self s and the opponent o. the partial interaction is c k , which is obtained after s observes k utterances from the opponent. 1 the goal is to build the model m , with y @xmath0, where y o is the desired priority order of the opponent. in our experiments, we consider metrics that measure the performance for the complete dialogue and for different values of k (section 5). figure 2: our proposed methodology for opponent modeling in negotiation dialogues. the approach involves three main components: section 3.1 describes our core hierarchical encoder that takes in a partial dialogue and outputs the opponent priority order after seeing each utterance, section 3.2.1 covers the adaptation of an argument-centric dataset (ca data) targeted towards better modeling the preference statements of the opponent, and section 3.2.2 describes the adaptation of an offer-centric dataset (dnd data) targeted towards the offer statements of the opponent. ## methodology we present our approach in figure 2 , which contains three main components: a hierarchical core model that takes in a partial dialogue and outputs the desired ranking order, and two modules for data adaptation that are designed to better model the preference and offer statements of the opponent. we first describe our core model, assuming a general input, and then describe the proposed data augmentation techniques. ## experimental design we address the following questions: q1) how useful is the proposed transformer-based ranker along with data augmentations for opponent modeling in negotiation dialogues? we experiment with two pretrained language models and compare our ranker to standard baselines. to test the data augmentations, we analyze model ablations, including 0-shot and few-shot settings. we also observe if they lead to a better performance with a lower number of utterances. q2) do preferences and offers contribute to the performance? to further shed light on the contributions of these utterances to the final opponent modeling performance, we look at average attention scores on these utterances. further, for a more explicit analysis, we observe whether the performance varies by the integrative potential in the negotiation, which essentially captures how aligned the preferences of the two negotiators are @xcite . the scenarios with low integrative potential are usually associated with a higher expression of preferences and offers. hence, we expected the performance to be higher in the cases with low integrative potential. q3) how does our approach compare to a human expert? we compare our model to a human expert and recognize some of the errors that the model makes, discussing potential directions for future work. datasets: each data point in cd results in two dialogues for our analysis, based on the perspectives of the two negotiators (section 2). we report results on 5-fold cross validation for this dataset. we further leave out 100 dialogues from the training data for hyperparameter tuning, resulting in 1548 dialogues for training, 100 for tuning, and 412 for evaluation -for each cross fold. we extract ca from the metadata corresponding to the training data of cd, leaving out 200 constructed dialogues for validation (following section 3.2.1). for dnd data, we only select the dialogues with at least 4 total utterances and unique priority values for meaningful training. after adaption (following section 3.2.2), we end up with 4074 dialogues for training and 444 for validation. all the models are 5 https://idecisiongames.com/promo-home primarily validated and tested on the corresponding subsets of cd (except for some additional analysis presented in section 5). evaluation metrics: our metrics are inspired by the negotiation literature, along with related research in dialog state tracking (dst) and learning-to-rank(ltr) tasks in nlp. our primary metric is exact match accuracy (ema): the percentage of cases where the predicted priority order is entirely correct. this is analogous to the popular joint goal accuracy in dst which captures the cases where all the slots are correctly identified @xcite . for negotiation tasks, even knowing the topmost priority can be useful. hence, we also report top-1 accuracy: the percentage of cases where the highest priority issue is correctly predicted. finally, we report the normalized discounted cumulative gain (ndcg@3). ndcg has been widely used in ltr tasks with distinct relevance values @xcite , which is also true for the setting that we consider. in our case, we use the relevance values as 5, 4, and 3 for the most, second, and least ranked issues respectively, following the incentive design structure of casino. we compute these metrics for all k from 1 to 5, varying the number of opponent utterances seen by the model. we present the results at @xmath1. to capture the performance with partial dialogues, we report corresponding k-penalty versions that take a weighted average of the performance for different values of k, while giving a linearly higher weight to the performance at a lower k. methods: we call the complete model from figure 2 that combines all the three datasets for training as cd + ca + dnd. we compare it with its ablations, including 0-shot and few-shot scenarios. we further develop two standard baselines. the random baseline chooses the final ranking at random, from all the possible orderings. bow-ranker is based on the bag-of-words paradigm. the input features are based on the normalized frequencies of the 500 most frequent words in the training dataset, except stopwords. instead of contextualized hierarchical representations, this method directly uses a feed-forward network on the input bow features to predict the ranking. the model is trained on partial dialogues using the same margin ranking loss. training details: the embedding dimension throughout is 768 for transformer-based models. these models use base variant of either bert @xcite or roberta @xcite for level i encoder. the level ii encoder uses one transformer layer. the feed-forward network contains two fully connected layers with a final sigmoid activation. we train the model with adam optimizer using a learning rate of 2e -5 for transformer-based methods and 2e -3 for bow-ranker. the margin c is kept as 0.3. we use a dropout of 0.1 to prevent overfitting. we further employ a loss-specific dropout of 0.15, in order to backpropagate the loss from fewer ks simultaneously. the models were trained for 20 epochs with a batch size of 25. we checkpoint after every epoch and the one with the highest ema at @xmath2. we provide the details on the computing infrastructure, hyper-parameter tuning, and validation performance in appendix a. ## related work opponent modeling encompasses several tasks in negotiations such as priority estimation, predicting opponent limits like batna @xcite , and classifying opponents into categories such as based on personality traits @xcite . in this work, we focused only on inferring the opponent's priorities but in a more challenging domain involving chat-based interactions, instead of structured communication channels often used in prior work @xcite @xcite . using a realistic interface like natural language fundamentally alters the negotiation dynamics in terms of the exchange of information, and hence, requires a separate investigation. for chat-based negotiations, nazari et al. (2015) relied on heuristics and utterance-level annotations to infer the opponent's priorities using frequencybased methods. @xcite explored a symbolic rule-based system to parse the utterances collected from a multimodal interaction. instead, our focus is on modeling the priorities directly from partial dialogues as input. research in negotiation dialogue systems has mainly focused on end-to-end modeling of the agent, without any explicit opponent modeling @xcite @xcite @xcite . however, there is evidence that even end-to-end systems can benefit from being more opponent-aware, as seen in recent work that uses dialogue acts to estimate opponent's behavior @xcite . a number of related data augmentation strategies have been explored in computer vision and nlp @xcite . most methods use rules or models to transform the available data or create synthetic data to avoid overfitting while training. this especially helps in low-resource languages @xcite and few-shot scenarios @xcite . ## conclusion we presented and evaluated a transformer-based approach for opponent modeling in negotiation dialogues. our objective was to address the challenges to bridge the gap between existing research and practical applications of opponent modeling techniques. our comparison to baselines and ablations attest to the utility of our method. we found that the proposed data adaptations can be especially beneficial in 0-shot and few-shot scenarios. in the future, we will explore two primary directions: first, improving the model performance on opponent modeling by leveraging other related available datasets and by better incorporating the negotiation dialogue context, and secondly, using effective opponent modeling techniques towards the design of automated negotiation systems for applications in pedagogy and conversational ai.