INA : An Integrative Approach for Enhancing Negotiation Strategies with Reward-Based Dialogue Agent (2023)

Abstract: In this paper, we propose a novel negotiation agent designed for the online marketplace. Our dialogue agent is integrative in nature i.e, it possesses the capability to negotiate on price as well as other factors, such as the addition or removal of items from a deal bundle, thereby offering a more flexible and comprehensive negotiation experience. To enable this functionality, we create a new dataset called Integrative Negotiation Dataset (IND). For this dataset creation, we introduce a new semi-automated data creation method, which combines defining negotiation intents, actions, and intent-action simulation between users and the agent to generate potential dialogue flows. Finally, the prompting of GPT-J, a state-of-the-art language model, is done to generate dialogues for a given intent, with a human-in-the-loop process for post-editing and refining minor errors to ensure high data quality. We first train a maximum likelihood loss based model on IND, and then employ a set of novel rewards specifically tailored for the negotiation task to train our Integrative Negotiation Agent (INA). These rewards incentivize the agent to learn effective negotiation strategies that can adapt to various contextual requirements and price proposals. We train our model and conduct experiments to evaluate the effectiveness of our reward-based dialogue agent for negotiation. Our results demonstrate that the proposed approach and reward functions significantly enhance the negotiation capabilities of the dialogue agent. The INA successfully engages in integrative negotiations, displaying the ability to dynamically adjust prices and negotiate the inclusion or exclusion of items in a deal bundle.

## introduction in an online marketplace, customers and sellers engage in discussions involving product inquiry and bargaining before reaching a common consensus @xcite . in such a setting, negotiation between the customer and the seller is a core facet of discourse that ultimately decides the profit of sale and customer satisfaction. negotiation on the price of a product is very common, however, customers have an open-ended approach to negotiation often also involving negotiation on certain aspects related to the deal. for example, while buying a chair the customer may negotiate a deal without the cushions, or even negotiate between delivery and in-store pick-up. as a result, a dialogue system for negotiation in an online marketplace should be capable of engaging in negotiation on different aspects such as price, product, and delivery. additionally, such a system should also be capable of responding to product inquiries with relevant and knowledge-grounded information. a systemic survey conducted by @xcite discussed various datasets, evaluation metrics, and methodologies in common literature. from this, it can be implied that bargain in the marketplace typically follows a "distributive" strategy where each party involved aims to maximize their gain rather than mutually benefiting outcomes. this strategy follows a win-lose model, where one party can gain only if the other party loses. the craigslistbargains dataset @xcite is the most prominent dataset in the price bargain domain with other datasets having less than 1,000 dialogues. this dataset contains dialogues between two human agents assigned the role of customer and seller negotiating over a product on craigslist, the strategy used in the dialogues are largely distributive in nature. in contrast to a distributive approach, an "integrative" approach to negotiation aims to reach a win-win situation by understanding the other party's needs and reaching a mutually satisfying consensus. it has been shown that an integrative approach to negotiation in retail e-commerce is more effective and leads to better customer satisfaction, than distributive approaches @xcite that typically utilize agents that negotiate only on price. it is common in online marketplaces for products to have several items, such as a "a chair and its cushion", a negotiation agent that is capable of satisfying customers that only want select items from the product such as customers that only want a chair or customers that only want a cushion is beneficial since the agent better understands customer requirements and may lead to win-win outcomes. hence, treating a product as a "bundle" of items that customers can choose is a more integrative approach than treating the product as a single entity. to incorporate this integrative approach, in this paper, we propose a novel dialogue system for negotiation in the online marketplace domain, which can respond to customers' inquiries and engage in negotiation with the customer. unlike existing systems @xcite that primarily focus on negotiation over the price of a product, our system follows a more integrative approach wherein negotiation involves different aspects such as adding or removing products from the aforementioned "bundle" of products, the price of the bundle, and the delivery of the product. datasets for negotiation such as the craigslistbargains dataset do not explicitly model the product as a bundle of smaller items. hence, we construct a dataset (ind) consisting of integrative negotiation dialogues where the deal is modeled as a bundle of products. to avoid complete manual data creation, we design prompts for the gpt-j model @xcite to generate integrative negotiation utterances. to ensure the dataset's quality, we use humans in the loop for minor edits and filtering of the generated dialogues. using the constructed dataset, we build an integrative negotiation-powered dialogue agent (ina) using a supervised learning (sl) + reinforcement learning (rl) approach. to train our system, we leverage a novel reward function and maximize it using ppo loss @xcite to ensure aspects of negotiation consistency, negotiation power, and intent consistency. as per our knowledge, this is the first attempt to build an integrative-negotiation based dialogue system. therefore we present a pioneering effort in developing an integrative-negotiation-based dialogue system, making several key contributions. first, we introduce a new task of integrative negotiation, expanding the scope of dialogue system research. sec-ond, we propose an efficient approach for automatically generating data with minimal manual intervention, addressing the challenge of data scarcity in certain domains. this contribution will drive the development of more robust dialogue systems. third, we create a unique dataset of integrative negotiation dialogues. finally, we leverage the strengths of both supervised and reinforcement learning to construct a powerful dialogue system empowered by integrative negotiation strategies. unlike these previous works, our proposed negotiation agent (ina) is capable of doing integrative negotiation. our agent is not only capable of negotiation with respect to the price of an item but can also modify the deal to better suit the customer's preference. similarly, our agent can also handle the customization of a deal proposed by the customer and decide on accepting or rejecting the deal. these capabilities are currently absent in any negotiation agent. ## dataset creation we construct the ind dataset for the task of integrative negotiation. to save on human effort and resources , we come up with a novel mechanism based on prompting a large language model for dataset creation. we keep human annotators in the loop only for making minor edits and filtering the automatically generated dialogues to ensure the quality of the conversations. the overall process consists of creating a skeleton of dialogues by dynamically deciding the correct intent for any arbitrary conversation. our overall dataset creation process consists of 5 steps: (i). background data creation, (ii). intent definition, (iii). dialogue flow generation, (iv). prompting for dialogue generation, and (v). data correction. ## dataset statistic the statistics of the dataset created are given in ## methodology to force a language model to negotiate with the user while following its own price goal as well as approach, we fine-tune it using a novel-designed reward function in a reinforcement learning setting. here, first, a pre-trained language model (gpt-2-medium) is fine-tuned in a supervised setting using traditional cross-entropy loss between the ground truth and predicted utterances probability distributions. for a supervised dialogue dataset @xmath0, d 1 , .., d n }, where, @xmath1, u 0 , .., a i , u i , .., a t -1 , u t -1 } -a multi-turn dialogue with u i + cxt i (u i -user's utterance at i th turn and cxt @xmath2, u 0 , .., a i-1 }) as input and a i (agent's utterance at i th turn) as output. the supervised learning dialogue model ρ θ (d) can be expressed as: here, normalized rewards are obtained by a novel designed reward function (r) incorporating intent consistency reward (r 1 ), price gap reward (r 2 ), negotiation strategy reward (r 3 ) and interactiveness (r 4 ) in generated responses. r intuitively nudges sldm towards these aspects by providing appropriate respective aspect penalization/reward for generated responses. for example, if the model generates intent inconsistent response then r 3 will penalize the model to discourage it from generating a similar type of content. all five rewards can be written as: intent consistency: in a negotiation system with complex intents there can often be divergence between the predicted intent and the intent of the generated utterance. to enforce this consistency, we propose intent consistency (ic) reward. this reward function is implemented by first training a bert model @xcite on the training set of ind for the task of intent prediction. this task is modeled as a classification task where the input to the bert model is an agent utterance at turn t, u a t , and the expected output is the intent of the utterance ia t . the accuracy of the trained intent classifier is 71.2%. we use the [cls] token for computing the probability distribution of the intent classes. we sample the probability value p it of the intent predicted i by our end-to-end sldm dialogue model and use it as r 1 (eq. 6). price gap reward: the purpose of negotiation is to find a win-win solution for both the customer and the seller. the winning scenario for a seller would be as little reduction in the initially proposed price as possible. in line with this logic, we propose a price gap (pg) reward. this reward is simply the fraction of the initial proposed price by the agent p ai and the final selling price after negotiation p af (eq 7). the higher the final price the greater the reward. negotiation strategy reward: a successful negotiation might not always entail deal acceptance. in cases where the customer wants to go below the minimum selling price of the agent p a-min it would not be judicious for the seller to satisfy the customer. in such situations where the negotiation could result in a win-lose situation, the deal should be rejected. hence, the success criterion of the negotiation lies in not just acceptance of the deal but also the fairness of the deal. to ensure that our negotiation succeeds only in win-win scenarios we design the negotiation strategy (ns) reward. in the above equations, p b is the customer's proposed price, and intent @xmath3, reject} is the final intent in the conversation used to capture the negotiation result. the reward incentivizes acceptance of a deal when the negotiated price is within the limit of a minimum price for the seller, and rejection when the negotiated price is below this minimum price. interactiveness: to ensure interactiveness, repetitions, and conversation blackholes are penalized such that system can engage the user for a longer duration with interactive responses. to penalize the generation of similar utterances for a given intent in the dialogue we use equation 11 . ## experiments 6.1 evaluation metrics to properly assess ina's performance, we perform both automatic and manual evaluations. in automatic evaluation to measure the surface similarity with the gold responses, we compute meteor @xcite . for semantic similarity, we compute bert score (bs-f1) @xcite and word mover distance (wm). we also report the perplexity (ppl) and the average response length (r-len) of the generated responses. human evaluations were conducted by three postgraduate evaluators who possess proficiency in similar tasks. each evaluator interacted with the proposed system 15 times and assessed the conversations based on: (i). negotiation consistency (n-con): it is the measure of consistency (absence of arbitrariness) in the negotiation approach within a dialogue (ii). bargaining efficacy (b-eff): it measures the ability of the negotiation system to present compelling arguments, reasoning, or incentives that influence the other party's decision-making process., (iii). outcome fairness (o-fair): it assesses the fairness or equity of the final outcomes reached during the negotiation process., (iv). dialoguefluency (d-f): it measures the overall grammatical correctness of the generated responses, and (v). dialogue-engagingness (d-e): measures the extent to which a conversation or dialogue is interesting, captivating, and able to hold the attention of the participants. the evaluators assigned scores on a scale of 1 to 3 for each metric (the higher the better). ## results and analysis it can also be inferred that the difference of bs-f1, and wm scores decrease in the following order: @xmath4. this shows the importance of task-specific rewards in our proposed system ina. this indicates that the ina is able to generate longer responses, hence, showcasing more engagingness with the user. it can be due to the incorporation of all four rewards where r 1 , r 2 , and r 3 play the crucial role in handling negotiation and price consistency, and r 4 helps in maintaining the non-repetitiveness, hence, driving the agent to build the rapport with a user as well as be on the goal by generating diverse and interactive negotiation responses. human evaluation: ## conclusion in this paper, we have presented a novel dialogue agent for negotiation (ina) in the online marketplace domain, focusing on an integrative approach that goes beyond price negotiations. our system can respond to customer inquiries and engage in negotiations that encompass various aspects, such as modifying product bundles, adjusting prices, and arranging product delivery. unlike existing systems that mainly concentrate on price negotiations, our approach provides a more comprehensive and versatile solution. to achieve this, we constructed a dataset of negotiation dialogues (ind) where the product is represented as a bundle of smaller items. to minimize manual effort in data creation, we employed prompts for the gpt-j model to generate integrative negotiation utterances. using ind, we developed a ina through a combination of supervised learning and reinforcement learning. our training process incorporated a novel reward function that suits the negotiation task, which we optimized using the proximal policy optimization (ppo) loss. our results show that ina is able to perform integrative negotiations with the customer and enable engaging negotiations that can lead to a win-win deal for the seller and the customer. in the future, it would be interesting to explore the role of the customer persona like age, gender, hobbies, etc. during negotiation.