Measuring Bargaining Abilities of LLM s: A Benchmark and A Buyer-Enhancement Method (2024)

Abstract: Bargaining is an important and unique part of negotiation between humans. As LLM-driven agents learn to negotiate and act like real humans, how to evaluate agents’ bargaining abilities remains an open problem.For the first time, we formally described the Bargaining task as an asymmetric incomplete information game, defining the gains of the Buyer and Seller in multiple bargaining processes. It allows us to quantitatively assess an agent’s performance in the Bargain task.We collected a real product price dataset, AmazonHistoryPrice, and conducted evaluations of various LLM agents’ bargaining abilities. We find that playing a Buyer is much harder than a Seller, and increasing model size can not effectively improve the Buyer’s performance.To address the challenge, we propose a novel approach called OG-Narrator that integrates a deterministic Offer Generator to control the price range of Buyer’s offers, and an LLM Narrator to create natural language sentences for generated offers.Experimental results show that OG-Narrator improves the buyer’s deal rates from 26.67% to 88.88% and brings a ten times multiplication of profits on all baselines, even a model that has not been aligned.

## introduction the negotiation ability of humans holds paramount significance, serving as a crucial means for humans to resolve conflicts of interest, seek mutually acceptable solutions, and facilitate the exchange of information and resources beneficial to all parties involved. this importance is underscored by various scholars such as @xcite . the ability to engage in bargaining is crucial for the autonomous actions of ai agents @xcite @xcite . independent ai agents might engage in purchasing * co-corresponding authors 1 the dataset amazonhistoryprice and our code are available at https://github.com/tianxiasjtu/amazonpricehistory . ## dataset current research on the communication abilities of llm agents lacks a large real product dataset. in the work by @xcite , a single artificially designed product (i.e., a balloon) was used. au-carena (chen et al., 2023) employed artificially designed products in two categories, i.e., cheap items and expensive items. previous nlp research related to bargaining @xcite introduced the craigslistbargaining dataset, primarily composed of different dialogues for repetitive items. its test set comprises only 161 second-hand items. to address this gap, we collected a dataset, amazonhistoryprice, from the camelcamelcamel website. this dataset includes 930 popular products with their real prices across 18 categories: electronics, books, music, etc., as seen in the left figure of figure 2 . each product's data includes the product name, description, features, lowest and highest price, current price, list price, and an image link, as shown in figure 8 . ## experiments in this section, we report the implementation details and the benchmark performances of many wellknown llms against chatgpt in the bargaining task on our dataset. llms are listed in appendix e. ## discussion in this section, we discuss our findings from the benchmarks for buyers and sellers. playing buyer is more difficult than playing seller. as seen in the difficulties include 1) the buyer should correctly understand the buyer role along with the concept of budget and realize that the deal price should not exceed the budget to avoid loss; 2) the buyer needs to actively persuade the conservative 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10. seller who prefers to use reject actions, as seen in the dialogue from for buyer, different-sized models from the same family have close performance. qwen-7b, qwen-13b, and baichuan2-13b seem not to fully understand the prompt for the buyer. they have a high probability, which is 99% for qwen-7b, 95% for qwen-14b, and 69% for baichuan2-13b, of perceiving themselves as the seller from the beginning of the conversation, as seen in apart from qwen-7b, qwen-13b, and baichuan2-13b, we found that different-sized models of the same series, including llama-2, yi, and mistral, have similar performances as a buyer in most models start from a high bid. in most of the cases we studied, the buyer tends to start with a slightly lower price than the given budget. it heavily affects the final deal price if a deal could be made. as demonstrated in we define first bid ratio (fbr) as the ratio of the buyer's first bid's price to the buyer's budget and then calculate the average first bid ratio in all valid sessions for each buyer. as seen in for seller, model size matters. in contrast to the buyer, the seller's snp is highly related to the valid rate. a high valid rate requires good instruction in the following capabilities. as observed in the only exceptions are mixtral-8x7b and baichuan2-13b-chat. mixtral-8x7b applies the mixture of experts (moe) mechanism on the same size of transformer as mistral-7b @xcite . its extra parameters of expert layers do not improve its valid rate and snp, but increase the share as a seller. baichuan2-13b-chat has a lower valid rate, a lower deal rate, and a lower snp than baichuan2-7b-chat. the reason could be its positional embedding alibi, which is a special kind of relative attention @xcite makes the model ignore the critical information for seller from the long dialogue, such as the cases of in this section, we propose a new method, og-narrator, to enhance an llm agent's bargaining performance. we believe that llms using cot do not fully exploit the potential of the buyer in the bargaining task. and also, currently the buyer's weak performance limits the fair assessment of the chatgpt seller. it is unknown if the chatgpt seller's performance remains good when their opponents are improved. so we design a method og-narrator in section 6 to improve the buyer and re-evaluate the chatgpt seller as well. ## related work in this section, we listed related works from the perspectives of ai agents and bargaining. ## conclusion to the best of our knowledge, we formally described the bargaining task for the first time, defining the evaluation metrics for the buyer and seller to quantitatively assess agent performance. we collected a large-scale real product price dataset, amazonhistoryprice, for the bargaining task. using this dataset, we evaluated the bargaining abilities of various llms. our findings indicate that playing the buyer is more difficult than playing the seller. we found that larger models lead to better seller performance but do not improve buyer performance. we observed that buyers cannot effectively bid low at the beginning of a session. consequently, we proposed the og-narrator to enhance the buyer performance of llm agents in bargaining tasks. we also found that the chatgpt seller is vulnerable when facing a buyer enhanced by og-narrator.