Conversational AI for Positive-sum Retailing under Falsehood Control (2022)

Abstract: Retailing combines complicated communication skills and strategies to reach an agreement between buyer and seller with identical or different goals. In each transaction a good seller finds an optimal solution by considering his/her own profits while simultaneously considering whether the buyer’s needs have been met. In this paper, we manage the retailing problem by mixing cooperation and competition. We present a rich dataset of buyer-seller bargaining in a simulated marketplace in which each agent values goods and utility separately. Various attributes (preference, quality, and profit) are initially hidden from one agent with respect to its role; during the conversation, both sides may reveal, fake, or retain the information uncovered to come to a final decision through natural language. Using this dataset, we leverage transfer learning techniques on a pretrained, end-to-end model and enhance its decision-making ability toward the best choice in terms of utility by means of multi-agent reinforcement learning. An automatic evaluation shows that our approach results in more optimal transactions than human does. We also show that our framework controls the falsehoods generated by seller agents.

## introduction retailing is a mixture of cooperation and competition between buyer and seller. the construction of virtual retailers has received widespread attention due to their broad applications in the e-commerce era. if the focus of the conversational retailer is limited to the buyer's needs, the retailer is actually a conversational recommendation system. however, if the conversational retailer's purpose is to maximize his/her own profit, the retailer is in fact a negotiation system, which typically must use discourse with opponents to perceive their intent and build strategies to achieve the retailer's own goals @xcite . previous nlp research on negotiation concerns closed-domain scenarios in games such as settlers of catan @xcite , goods distribution @xcite , and open-ended settings, for example, price bargaining on a single item in a zero-sum, second-hand market @xcite . however, these scenarios do not attempt to find an optimal solution for both sides, which crucially defines a good retailer who always takes into account future transactions. therefore, inspired by @xcite , we propose a positive-sum setting in this paper: a buyer and a seller negotiate to achieve a transaction, and the seller not only considers his/her profit but also takes into account whether the buyer's needs have been met, thus seeking a mutually optimal solution. to simulate such a real-world vending scenario and provide enough motivation to start a conversation, both buyer and seller are given incomplete information prior to the conversation. the buyer knows what he/she prefers among multiple products but does not know the quality of the product prior to the conversation, and the seller does not know in advance the buyer's preferences but is aware of the quality of the product and its profit. the seller seeks a mutually optimal solution by which to build his/her own reputation for future business while simultaneously making a profit. thus we propose separate utility functions for buyers and sellers. to facilitate end-to-end fine-tuning for this scenario, we collected a large dataset of 4232 dialogues between two people negotiating on goods in a simulated market on amazon mechanical turk (amt). our model is based on the transformer architecture @xcite , which is predominant in recent nlp research, due in part to its inherent parallelism, which facilitates the use of large-scale datasets to train complex models such as gpt2 @xcite , evolved transformer @xcite , and t5 @xcite . further, these complex models are often pre-trained in an unsupervised fashion, yielding powerful performance in downstream tasks in an end-to-end, supervised manner, which lays the foundation for training two acceptable conversational agents to fit the proposed scenario. the supervised fine-tuning maximizes the likelihood of human utterances in the dataset. to maximize agent targets, we leverage reinforcement learning (rl) to direct the finetuning process. in addition, due to the increasing saturation of machine learning algorithms in contemporary society, there has been a surge in interest in building truthful ai, a system that avoids stating falsehoods, thus enhancing transparency and helping to establish trust between system and human @xcite . to achieve such truthful ai, we attempt to reduce a certain type of statement against the ground truth in our negotiation scenario. first, we build a falsehood detector with respect to such statements. second, we formulate a deduction mechanism in the rl stage to decrease the generation of falsehoods. in summary, the contributions of our study are the following: ## data collection in this paper, we discuss the behavior of two conversational agents negotiating given imperfect information. to promote end-to-end training, we collected fruitstand, a dataset of human-human dialogues designed around a novel scenario which simulates a fruit stand at which the negotiation takes place. in fruitstand, one agent plays the role of the buyer and the other that of the seller, communicating in natural language, developing strategies and eventually making a deal. ## analysis and discussion goal-based models are more task-centered. although the fine-tuned t5-base model can generate fluent and reasonable utterances, it tends to output generic responses such as "great!" which poorly reflect the task setting. see behavior control besides falsehood, we also investigated how to control virtual sellers' other behaviors. four different sellers are investigated: balanced seller is the standard seller described all over the paper, which utility is the sum of buyer's utility plus items' profits. win-win seller's utility is based on whether mutual optimality was achieved. recommender's utility is exactly the same as buyer's utility. profit-oriented seller's utility base on only items' profits. appendix c shows their vending results accordingly. we found that in general, balanced seller remains a certain level of profitability and satisfy customers at the same time. actually, the decision of choose what kind of virtual seller to employ in practice would depend on employers' willingness and needs. here we just demonstrate that how virtual sellers can be customised by just adjusting their utility design. deduction mechanism silences all. the falsehood detector is meant to prevent the seller from generating untruthful claims, and ensure that only factual claims are made. however, we find that the deduction mechanism suppresses not only such falseness, but also expressions containing such claims. that is, it prevents the seller from generating any utterances with matching patterns. for example, at some checkpoints, the seller does not even produce the string 'the best', which is clearly not a desired consequence. with the deduction mechanism or its adjustment. each number in a cell (expect for those horizontal to 'checkpoints') shows how many falsehoods found by the detector in each checkpoint. we thus adjust the mechanism using two approaches. first, we retain the -2 deduction on falsehood, but compensate those expressions by +0.5, denoted by rl (dm b ). second, we instead reduce the deduction to -1, a more conservative value corresponding to r (τ ). this path is denoted by rl (dm r ). the results in ## related work during the recent, rapid development of conversational agents, also known as chatbots, various applications have been created. open-domain chatbots such as facebook's bst @xcite ) and google's meena @xcite seek to be more human-like, engaging in conversation on any topic. closed-domain chatbots instead focus on improved task performance, for instance guess-which @xcite , persuasion @xcite , and negotiation @xcite @xcite . to negotiate item distribution (book, hat, ball), @xcite apply a bi-directional gru model to train a language model and use reinforcement learning with self play to develop data-driven strategies. for price bargaining on a single item (e.g., a tv), @xcite use a hybrid approach involving rule-based and lstm models that decouple natural language understanding, dialogue act prediction, and natural language generation to facilitate controllable negotiation strategies. however, these scenarios do not attempt to find an optimal solution for both sides, and do not control the falsehoods generated by sellers. these limitations motivate this work. ## conclusion we introduce a novel negotiation task and present fruitstand, a rich dataset of human-human dialogues, for negotiation dialogue research. we demonstrate the effectiveness of reinforcement learning in guiding the conversational agent toward a specific goal. finally, our experiments in falsehood suppression show the potential of rl for truthful ai. a more robust falsehood detector would be our first future work. in our initial observations, a strong natural language inference (nli) model could play this role. figure 5: seller's interface 32 c behavior control figure 6: rate of buyer's best items being chosen. 'human' stands for the human selections in testing. 'before_rl' stands for the model before reinforcement learning.