ACE : A LLM -based Negotiation Coaching System (2024)

Abstract: The growing prominence of LLMs has led to an increase in the development of AI tutoring systems. These systems are crucial in providing underrepresented populations with improved access to valuable education. One important area of education that is unavailable to many learners is strategic bargaining related to negotiation. To address this, we develop a LLM-based Assistant for Coaching nEgotiation (ACE). ACE not only serves as a negotiation partner for users but also provides them with targeted feedback for improvement. To build our system, we collect a dataset of negotiation transcripts between MBA students. These transcripts come from trained negotiators and emulate realistic bargaining scenarios. We use the dataset, along with expert consultations, to design an annotation scheme for detecting negotiation mistakes. ACE employs this scheme to identify mistakes and provide targeted feedback to users. To test the effectiveness of ACE-generated feedback, we conducted a user experiment with two consecutive trials of negotiation and found that it improves negotiation performances significantly compared to a system that doesn’t provide feedback and one which uses an alternative method of providing feedback.

## introduction the rapid progress of llms in recent years has spurred the creation of more sophisticated ai tutoring systems @xcite . these systems give learners easier access to training in areas outside what is typically available for most learners @xcite . one area that falls into this category are social competencies, such as negotiation tactics. negotiation skills are crucial as they help individuals maximize their gains in competitive situations across different areas of their professional * denotes equal contribution. and personal life. however, these skills are hard to learn from a traditional classroom lecture because they involve reflexive behavioral habits. they are typically taught through small seminars centered on role-playing exercises and instructor coaching, which are expensive and limited in access. therefore, effective negotiation training is not available to many populations who lack it yet need it the most, such as women and minorities. these groups particularly lack negotiation skills and are less accustomed to advocating for themselves @xcite which is a significant factor contributing to their relatively lower starting salaries and fewer promotion opportunities @xcite . as such, increased coaching is needed to address gender and ethnic differences in negotiation performance @xcite . recent research also finds that linguistic assertiveness skills, such as those used in negotiations, vary as a function of cultural/ethnic backgrounds, partly account for ethnic differences in promotions to leadership positions, and are amenable to training interventions @xcite . to democratize access to high-quality negotiation coaching, we proposed a llm-based assistant for coaching negotiation (ace). ace is designed to serve as a negotiation partner that provides learners with targeted feedback similar to what they would receive from an instructor in a seminar setting. to provide this feedback, we begin by collecting a dataset of negotiation transcripts between students in an master's of business administration (mba) negotiation class. the scenarios in these negotiations come from a rigorous curriculum and are designed to mirror real-world negotiation settings. furthermore, the participants in these negotiations have been trained in bargaining tactics, unlike previous datasets which rely on crowd-workers @xcite . using this dataset along with expert consulta-tions, we develop an annotation scheme to identify and correct mistakes that users make during their negotiations. our annotations mirror actual error categories that instructors look for in negotiation classes. we then build ace according to this scheme. ace uses the annotation categories to identify users' mistakes and then provides targeted feedback based on the error definitions, along with in-context examples of feedback written by experts. our system also includes a prompt-based negotiation chatbot agent which serves as a simulated negotiation partner for learners. to test the efficacy of ace, particularly the feedback it provides, we recruited a group of 374 users who participated in two trials of negotiation. we found that ace significantly improves learning outcomes compared to users who negotiate with no coaching from our system. our contributions are summarized as follows: ## background negotiation is a general task and there are many ways to formulate a bargaining problem. in this section we give an overview of the the types of negotiation problems we consider when designing our coaching system and annotation scheme. the negotiation settings we consider here are ones which involve one agent selling a single item to another agent. this type of negotiation is known as single-issue distributive bargaining @xcite . the agent selling the item is referred to as the "seller" and the agent buying the item is called the "buyer". both the buyer and seller have access to a "role" (also referred to as a "negotiation scenario") which provides private information about their options and preferences and public information about details of the object for sale and the range of market prices for this item. in a role-play simulation, participants use their role to prepare a strategy, this includes their upper limit or "walkaway" price, their target price, and the opening price that they will mention. an example of buyer and seller roles for a used car negotiation can be found in appendix d. ## dataset our dataset was collected in collaboration with an instructor who teaches a course on negotiation to mba students. during this course, 50 students were randomly assigned to dyads for a negotiation task. they conducted three successive negotiations against three different counterparts, involving different kinds of cars. before negotiating, students answered standard preparation questions and then began the negotiation while recording their conversation audio. we transcribed the conversation audio using openai's whisper api @xcite along with manual edits to construct our dataset. a total of 42 dialogues were collected from this process, of which 40 were usable. a summary of the dataset statistics is in ## annotation scheme we designed a negotiation error annotation scheme to identify and correct user mistakes. our scheme is based on the dataset collected in section 4, expert input, and common distributive bargaining tactics from @xcite we identified eight error categories that can be divided into preparation errors and negotiation errors. all the categories in our scheme are binary true/false labels, where a label of false indicates that a mistake has been made by the negotiator. our annotation scheme is designed from the perspective of the buyer but can be easily adjusted to fit the seller's perspective. the two categories below belong to preparation errors, which are mistakes in users' answers to a set of standard preparation questions before the negotiation. these categories are designed to identify whether the user prepared their negotiation strategy correctly. 1. strategic walk-away point assesses whether the user has properly analyzed the facts of the negotiation scenario to set an appropriate walk-away price. from the buyer's perspective, a walk-away price is the maximum amount they would pay to purchase the item in the negotiation. if the scenario outlines an explicit budget limit, then a strategic walk-away point is one which exactly matches the budget amount. otherwise, we consider any point below the maximum market price to be strategic. 2. strategic target price evaluates the target price that the buyer sets before the negotiation. the buyer's target price is strategic if it falls within the first third of the range between the minimum market price and the buyer's walk-away point. if the buyer sets their target below this range, it is too ambitious to be a realistic outcome to aim for. conversely, if their target is above this range, it is too weak to test how far their opponent can be pushed in the negotiation. the six categories below refer to negotiation errors, which are mistakes that users make "at the role utterance annotation buyer hi, i'm new to california and i'm looking for probably a honda accord with reasonable mileage around maybe @xmath012,000. do you have anything like that? breaking the ice giving the first offer ambitious opening point including rationale seller nice. we have something similar. we have a nice 2013 honda. it does have a little bit more miles than that. it has about 50,000. it doesn't have any rust and it's in great condition. what's the price range you're looking to come out with? buyer probably around @xmath112,000. strong counteroffer including rationale seller ooh, that's kind of rough. our sticker price for this car is closer to @xmath212,500 and i would buy it today? strong counteroffer including rationale seller @xmath313,000? buyer yeah, i could probably do $13,000. strong counteroffer including rationale seller all right. buyer all right. strategic closing seller sounds great. a counter-offer o t is considered strong if it's below the midpoint of the remaining bargaining range: o @xmath4, w ) 2 where o t-1 is the buyer's previous offer, s is the seller's current offer, and w is the buyer's walkaway point. ## approach in this section, we outline our approach to designing ace. we first provide a high-level overview of the system and how users progress through it. we then describe the components of the system, including the negotiation chatbot agent and feedback modules. ## conclusion ai tutoring systems have the potential to democratize high-quality education in key areas such as negotiation. prior work has shown that additional negotiation coaching is needed to correct for systematic gender and ethnic differences in bargaining performance @xcite . to address this need, we built ace to mimic the coaching learners would receive from a professional negotiation instructor in a seminar setting. ace was built based on a dataset and annotation scheme created in collaboration with experienced negotiation instructors. we evaluated our system and confirmed its error identification accuracy and feedback effectiveness. results from a user experiment demonstrated that users who interact with ace improved their negotiation performance significantly compared to those who do not recieve ace-coaching.