Conversational AI for Positive-sum Retailing under Falsehood Control
Abstract
Retailing combines complicated communication skills and strategies to reach an agreement between buyer and seller with identical or different goals.
In each transaction a good seller finds an optimal solution by considering his/her own profits while simultaneously considering whether the buyer’s needs have been met.
In this paper, we manage the retailing problem by mixing cooperation and competition.
We present a rich dataset of buyer-seller bargaining in a simulated marketplace in which each agent values goods and utility separately.
Various attributes (preference, quality, and profit) are initially hidden from one agent with respect to its role; during the conversation, both sides may reveal, fake, or retain the information uncovered to come to a final decision through natural language.
Using this dataset, we leverage transfer learning techniques on a pretrained, end-to-end model and enhance its decisionmaking ability toward the best choice in terms of utility by means of multi-agent reinforcement learning.
An automatic evaluation shows that our approach results in more optimal transactions than human does.
We also show that our framework controls the falsehoods generated by seller agents.
The code and dataset are available on https://github.com/ckiplab/Fruit_Stand.
Introduction
Retailing is a mixture of cooperation and competition between buyer and seller.
The construction of virtual retailers has received widespread attention due to their broad applications in the E-commerce era.
If the focus of the conversational retailer is limited to the buyer’s needs, the retailer is actually a conversational recommendation system.
However, if the conversational retailer’s purpose is to maximize his/her own profit, the retailer is in fact a negotiation system, which typically must use discourse with opponents to perceive their intent and build strategies to achieve the retailer’s own goals (Keizer et al., 2017; Afantenos et al., 2012).
Previous NLP research on negotiation concerns closed-domain scenarios in games such as Settlers of Catan (Asher and Lascarides, 2013), goods distribution (DeVault et al., 2015; Lewis et al., 2017), and open-ended settings, for example, price bargaining on a single item in a zero-sum, second-hand market (He et al., 2018).
However, these scenarios do not attempt to find an optimal solution for both sides, which crucially defines a good retailer who always takes into account future transactions.
Therefore, inspired by Shapiro (1983), we propose a positive-sum setting in this paper: a buyer and a seller negotiate to achieve a transaction, and the seller not only considers his/her profit but also takes into account whether the buyer’s needs have been met, thus seeking a mutually optimal solution.
To simulate such a real-world vending scenario and provide enough motivation to start a conversation, both buyer and seller are given incomplete information prior to the conversation.
The buyer knows what he/she prefers among multiple products but does not know the quality of the product prior to the conversation, and the seller does not know in advance the buyer’s preferences but is aware of the quality of the product and its profit.
The seller seeks a mutually optimal solution by which to build his/her own reputation for future business while simultaneously making a profit.
Thus we propose separate utility functions for buyers and sellers.
To facilitate end-to-end fine-tuning for this scenario, we collected a large dataset of 4232 dialogues between two people negotiating on goods in a simulated market on Amazon Mechanical Turk (AMT).
Our model is based on the Transformer architecture (Vaswani et al., 2017), which is predominant in recent NLP research, due in part to its inherent parallelism, which facilitates the use of large-scale datasets to train complex models such as GPT2 (Radford et al., 2019), evolved Transformer (So et al., 2019), and T5 (Raffel et al., 2020).
Further, these complex models are often pre-trained in an unsupervised fashion, yielding powerful performance in downstream tasks in an end-to-end, supervised manner, which lays the foundation for training two acceptable conversational agents to fit the proposed scenario.
The supervised fine-tuning maximizes the likelihood of human utterances in the dataset.
To maximize agent targets, we leverage reinforcement learning (RL) to direct the finetuning process.
In addition, due to the increasing saturation of machine learning algorithms in contemporary society, there has been a surge in interest in building truthful AI, a system that avoids stating falsehoods, thus enhancing transparency and helping to establish trust between system and human (Evans et al., 2021).
To achieve such truthful AI, we attempt to reduce a certain type of statement against the ground truth in our negotiation scenario.
First, we build a falsehood detector with respect to such statements.
Second, we formulate a deduction mechanism in the RL stage to decrease the generation of falsehoods.
In summary, the contributions of our study are the following:
• We propose a simplified market setting where vendor and purchaser are in a “coopetitive” relation with information asymmetry.
To this purpose we gathered FruitStand, a rich dataset of human-human negotiations under this scenario on Amazon Mechanical Turk (AMT).
• We propose an RL framework by which to cause a virtual retailer to learn how to find optimal solutions under positive-sum situation.
• The experiments demonstrate the effectiveness of reinforcement learning in improving the ability to achieve optimal transactions.
• We analyze the lies in a crowd-sourced dataset and the falsehoods generated by the seller model, based on which we propose an approach to reduce falsehoods.
Data Collection
In this paper, we discuss the behavior of two conversational agents negotiating given imperfect information.
To promote end-to-end training, we collected FruitStand, a dataset of human-human dialogues designed around a novel scenario which simulates a fruit stand at which the negotiation takes place.
In FruitStand, one agent plays the role of the buyer and the other that of the seller, communicating in natural language, developing strategies and eventually making a deal.
2. Task
The scenario simulates two agents transacting at a fruit stand.
In each dialogue, the agents are first assigned a role, either buyer or seller, and the order of turns in which to send natural language messages.
There are 3 item types— apples , bananas , and oranges —each of which has three attributes— preference , quality , and profit — as shown in upper left corner of Fig. 2.
Theseattributes determine the initial condition o .
The buyer and seller each have an individual utility.
The buyer’s utility U b ( item ) to an item is defined as preference( item ) × quality( item ) , following the intuition that the buyer is satisfied by purchasing what he/she likes in excellent condition (e.g., red, sweet, and juicy apples).
Likewise, the seller’s utility U s ( item ) is defined as U b + profit( item ) , taking into account the seller’s current profit and the buyer’s satisfaction for future profit, since the buyer might become a regular if he/she is satisfied.
Each agent’s best option is that which provides the highest utility.
Depending on the best options, the agents’ goals may be identical, or may conflict, which leads to opportunities for cooperation or competition, respectively.
In each dialogue, buyer and seller bargain turn by turn, trying to make a deal on their own best option(s).
Agents possess imperfect information.
Initially, the buyer knows only its preference , and the seller only the quality and profit of an item.
During the conversation, they must estimate the other’s exclusive attributes by skill of speech, all the while not revealing any exact values.
Absolute honesty is not required; agents can be deceptive.
In particular, the seller may mislead the buyer when a given item is more profitable; however, the final decision lies with the buyer.
Each conversation ends when the buyer makes a decision; typically this occurs within 5 to 20 turns.
The design of the utility functions and the right to choose compensates for the buyer’s inferior position in terms of the amount of information.
2. Collection
We collected the FruitStand dataset based on the above task via AMT with the interface shown in
Figures 1 and 3.
Workers were paid per dialogue, with a bonus for achieving the best option in terms of utility.
The starter of a dialogue could be either a seller or buyer, and we kept the number of starters from both sides roughly balanced.
The dataset statistics in Table 1 show that FruitStand has longer and more variant dialogues than DealorNodeal(Lewis et al., 2017).
FruitStand has a total of 4232 dialogues with unique initial conditions, 76.1% of which have mutually optimal solutions (the overlapping best options from two sides), as illustrated in Table 2.
We partitioned 80%/10%/10% of the dialogues for training/validation/testing.
[Figure] [Figure]
[Table] [Table]
Retailer 3. Data Representation
Every turn in a dialogue is transformed into a training pair—input sequence X and label sequence Y — from the perspective of the agent.
For example, as illustrated in Figure 2, the buyer starts the conversation and its preferences and utterance in this turn are converted into the first training pair of the dialogue, ⟨ X B 1 , Y B 1 ⟩ .
Note that
Y B 1 = { y B 11 , y B 12 , ..., y B 1 T } , where y ij is a token and
T is the length of the utterance at this turn.
Next, the seller’s scenario along with the buyer’s previous utterance and its response in this turn become the second training pair, ⟨ X S 1 , Y S 1 ⟩ , the seller’s first.
The process continues until the end of the conversation.
A similar technique has been used, see, e.g.,
Wolf et al.
Note that we take the natural form for the agents’ scenario, o B and o S , instead of merely numbers, to leverage the words’ underlying information from pretrained models.
3. Baseline Models
For the first training stage, we fine-tune a T model (Raffel et al., 2020) pretrained on our FruitStand dataset.
T5 is a standard encoder-decoder
Transformer (Vaswani et al., 2017) which regards all NLP tasks as a text-to-text format.
We leverage its baseline version (T5-base) as described in Raffel et al.
(2020) as our starting point.
T5-base is a composite of 12 Transformer blocks (each block combines self-attention, optional encoder-decoder attention, and a feedforward layer with a hidden size of 3072).
It performs well on downstream tasks as varied as machine translation, document summarization, and sentiment classification.
The pretrained model is then fine-tuned as in supervised learning (SL), i.e., by minimizing the cross-entropy loss between the generated sequence Z and the label sequence Y described in
Sec. 3.1.
We have two transfer paths: one for the buyer and one for the seller.
The buyer path uses labels from the buyer’s perspective, and the seller path uses its part in the dialogue.
The pair of the two picked models, denoted as ⟨ M Bϕ , M Sθ ⟩ , forms the baseline for later evaluation, where ϕ and θ are the learned parameters of the buyer and seller model, respectively.
See Sec.
4.1 for more details.
3. Goal-oriented Reinforcement
The goal of supervised learning is to imitate average human behavior; however, not every person is good at making deals.
We further fine-tune the agents via reinforcement learning to improve the choice of—or the persuasion of the buyer to choose—the best option through a dialogue.
This two-stage learning strategy has been widely used to enhance pretrained models toward a specific goal, e.g., Stiennon et al.
; Lewis et al.
; Li et al.
In reinforcement learning, we utilize self play (Lewis et al., 2017) to enhance our baseline models M Bϕ and M Sθ by making one agent talk to the other turn by turn.
Each turn ends when an agent outputs the END - OF - SENTENCE token, and the dialogue finishes when the buyer outputs the
SELECTION token in a turn, or when the dialogue length limit is reached, as in the human case depicted in Fig.
A buyer’s utterance in the i -th turn of a dialogue is denoted as Z Bi , with Z Si for the seller
’s.
We denote the trajectory τ B or τ S as the sequence of all tokens generated by buyer or seller during a dialogue.
For instance, the buyer’s trajectory is
[Equation]
where || denotes concatenation and N is the number of turns.
After a complete dialogue has been generated, we update the agents’ parameters based on the negotiation results.
Agents get the final reward
R ( τ ) when the dialogue is terminated.
We define
R ( τ ) = 1 if the buyer selects the item with highest utility, R ( τ ) =
if the buyer selects an item other than the best one, and R ( τ ) = − 1 otherwise.
Note that the best item for a buyer is not necessarily the same that for a seller.
Similar to AlphaGO (Silver and Huang et al, 2016), R ( τ ) is then assigned to tokens generated at each previous, non-terminal time step.
We use REINFORCE (Williams, 1992) to optimize the baseline models separately toward the best options.
Given a sampled trajectory τ and the final reward R ( τ ) , let a i be the i -th token generated in a turn; we update the model’s parameters θ by
[Equation]
where η is the learning rate and b is the baseline calculated by the average reward of the previousupdates.
Whereas the canonical Transformer is difficult to optimize in the RL setting, often resulting in performance comparable to a random policy (Parisotto et al., 2020), or leading to meaningless results (Lewis et al., 2017; He et al., 2018), we find the pretrained T5 model works well with parameter updates by policy gradient when we simply set a smaller learning rate.
3. Falsehood Control
One way to increase one’s integrity is to tell no lies.
We follow this notion to build a more trustworthy conversational agent, especially a seller, by decreasing the possibility that an agent produces an untruthful utterance.
In the FruitStand task, the seller might claim that one type of fruit is the best in quality when it really is not, attempting to attract a buyer to choose a more profitable item, and vice versa, to keep a buyer away from a less lucrative one.
Motivated by these observations, we construct a simple rule-based falsehood detector that first
[Table]
SUP are the FRUIT
SUP FRUIT
FRUITs are your SUP
FRUITs are my SUP
FRUITs are the SUP <Ignore>
FRUITs are the best seller
[Table]
Claim a type of fruit is the best or worst but actually not.
parses the claim for two superlatives, as shown in
Table 3, and then determines whether the seller’s claim conflicts with any known fact based on a given scenario o .
We further use this to establish a deduction mechanism D ( τ ) on the final reward in the reinforcement learning stage.
Given a trajectory τ , D ( τ ) =
− 2 if any of the seller’s utterances conflict with the facts about the quality of an item;
D ( τ ) =
if none of this kind of falsehood is detected.
The updated final reward then becomes
R ( τ ) + D ( τ ) ; we term this approach RL (w/DM).
Experiments 4. Training Details
We used PyTorch to implement our models, and used the pretrained T5-base model from Hugging Face.
We added the special tokens BUYER ,
SELLER , and SELECTION as self-defined tokens to T5Tokenizer.
For the baseline models, we fine-tuned the pretrained T5-base for 20 epochs; after each epoch we set a checkpoint, and then picked that with the least perplexity on the validation set: for the buyer this yielded model
M Bϕ and for the seller M Sθ separately.
We used
AdamW (Loshchilov and Hutter, 2019) to optimize models with a learning rate of 6 .
25 × 10 − 5 , β 1 = 0 .
9 , β 2 = 0 .
999 , and a mini-batch size of 4.
We clipped gradients above 1.0.
Supervised finetuning was run on a single RTX 2080Ti GPU.
In reinforcement learning, we estimated the expected final reward under an initial condition by
[Table] [Table]
hello, i am looking for some fresh fruit
[Table]
buy
[Table]
fruit
[Table]
S
[Table]
i have some great oranges.
[Table]
great oranges
[Table]
are they fresh?
[Table]
then
[Table]
great
[Table]
B: hello, i’m looking for some apples.
[Table]
: hello, i am looking for some fresh fruit.
S:
i have some great apples.
what do you have that is fresh?
they are very fresh and juicy.
S: my oranges are the freshest i have today.
B: great!
i love apples!
B: oh great, i love oranges.
how are your apples?
S: great.. how many would you like?
[Table]
my apples are ok B: i will take a dozen please but my oranges are the freshest i have today.
S: great.. here you go.
B: ok.
i will take some oranges then.
thank you.
B: <selection> apples S: great!
i will get them ready for you.
[Table]
> oranges
sampling N turns of utterances from self-play dialogue.
In each turn, at the T5 decoding phase, the next token a t was randomly chosen according to its conditional probability distribution a t ∼ P ( a | a 1: t − 1 ) using top- K sampling (Fan et al., 2018), in which the K most likely next tokens are filtered in and the pmf of the output tokens is redistributed among the K tokens.
We empirically chose N = 32 and
K = 50 for a given o and set the mini-batch size to N .
We also used AdamW for the parameter updates but reduced the learning rate to one-tenth of that used in the supervised fine-tuning.
We chose the number of dialogues in the validation dataset as the amount of dialogues used in an epoch for RL approaches.
We updated the parameter per minibatch for 10 epochs.
This took about 40 hours on a single Quadro RTX 8000.
4. Comparison
We compare the performance of the following models:
• Fine-tuned SL : our baseline models described in Sec. 3.2: a pair of pretrained T models fine-tuned on FruitStand.
Given O
train
′ , the initial conditions of the dialogues randomly picked from the training set to the size of
the
validation set, we evaluated the variants derived
from Sec
• RL (interleaved on O train ′ ) :
Direct optimization of the agent goals via RL
often results in
language that differs from human language
Similar to Lewis et al.
, we
fine-tuned the baseline models with RL followed by SL in each epoch.
The learning rate was one-tenth of that for Fine-tuned SL. • RL (directly on O train ′ )
: Under the same initial conditions, we evaluated the scenario without the following SL part.
The learning rate was one-tenth of that for Fine-tuned SL. • RL (random initialized O ) :
The baseline models self play under randomly initialized scenarios.
Since the outputs of the baseline models diverge from human language during the RL process for unseen initial conditions, we further reduced their learning rate to onehundredth of that for Fine-tuned SL.
4. Evaluation
We evaluated the performance of the proposed approaches on FruitStand by the proportion of the best options being chosen after self play, denoted as the p -score, with respect to the unseen initial conditions in the testing dataset.
Note that in the evaluation stage, for fair competition, we used not top- K sampling but instead greedy search, which simply selects the token with the highest probability as the next token:
[Equation]
For each RL variant described in Sec.
4.2, we first evaluated our models on the validation set, pair by pair at each checkpoint, and chose that pair with the highest average p -score for testing.
The results are shown in Table 4.
The RL approaches considerably enhance the ability to select the best item from the baseline models.
Compared to human-human negotiation in the FruitStand testing set, RL (interleaved on O train ′ ), the best model, achieves even better performance.
This success provides evidence that maximizing the reward outplays average humans and constitutes an acceptable imitation.
For falsehood detection, we compared the number of a typical kind of detected falsehood produced by a seller from dialogues in the testing dataset (Human), the number from baseline models (Baseline models), and the number from the RL (interleaved on O train ′ ) variant, RL (w/o DM).
Checkpoints
RL(w/o DM) RL(w/ DM)
Checkpoints
RL(w/o DM) RL(w/ DM)
Human:
Baseline models:
[Table]
The results are shown in Table 6.
In the crowdsourced testing dataset, the specific type of falsehood exists in 18 out of 423 dialogues.
In the baseline, falsehoods were detected in 32 out ofdialogues.
RL (interleaved) on O train ′ ) performs poorly on falsehood detection with 6 to 41 falsehoods among all the checkpoints.
In contrast, our approach, RL (w/DM), significantly reduces the falsehoods in the pattern.
Analysis and Discussion
Goal-based models are more task-centered.
Although the fine-tuned T5-base model can generate fluent and reasonable utterances, it tends to output generic responses such as “great!”
which poorly reflect the task setting.
See Table 5.
In comparison,
RL approaches generate utterances that better fit the simulated scene.
A general phenomenon is that they generate long utterances, similar to humans, who show their interest in goods by asking more questions, and vendors, who show their passion by promoting their products.
We also find that models learn to compare goods; comparison is an effective way to determine which item to choose.
Behavior Control
Besides falsehood, we also investigated how to control virtual sellers’ other behaviors.
Four different sellers are investigated:
Balanced Seller is the standard seller described all over the paper, which utility is the sum of buyer’s utility plus items’ profits.
Win-win Seller ’s utility is based on whether mutual optimality was achieved.
Recommender ’s utility is exactly the same as buyer’s utility.
Profit-oriented Seller ’s utility base on only items’ profits.
Appendix C shows their vending results accordingly.
We found that in general, Balanced Seller remains a certain level of profitability and satisfy customers at the same time.
Actually, the decision of choose what kind of virtual seller to employ in practice would depend on employers’ willingness and needs.
Here we just demonstrate that how virtual sellers can be customised by just adjusting their utility design.
Deduction mechanism silences all.
The falsehood detector is meant to prevent the seller from generating untruthful claims, and ensure that only factual claims are made.
However, we find that the deduction mechanism suppresses not only such falseness, but also expressions containing such claims.
That is, it prevents the seller from generating any utterances with matching patterns.
For example, at some checkpoints, the seller does not even produce the string ‘the best’, which is clearly not a desired consequence.
Checkpoints
RL(w/o DM) 18/ 8/ 8/
RL(w/ DM) 0/ 0/
RL(DM B ) 6/ 5/ 3/ 0/
RL(DM R )
Checkpoints
RL(w/o DM) 41/ 9/ 6/ 16/
RL(w/ DM) 0/ 0/
RL(DM B ) 0/
RL(DM R ) 0/
Checkpoints
RL(w/o DM) 7/ 11/
RL(w/ DM)
RL(DM B ) 6/ 0/
RL(DM R )
Human: 18/
Baseline models: 32/
Size of testing dataset:
We thus adjust the mechanism using two approaches.
First, we retain the -2 deduction on falsehood, but compensate those expressions by +0.5, denoted by RL (DM B ).
Second, we instead reduce the deduction to -1, a more conservative value corresponding to R ( τ ).
This path is denoted by
RL (DM R ).
The results in Table 7 show that it is difficult to avoid mistakenly silencing non-deceptive utterances.
In the experiment on both paths, at some checkpoints the seller avoids indiscriminate silencing, whereas at other checkpoints falsehoods are generated which still use those combinations of words.
The underlying reasons for such unstable results are poorly understood.
We leave this as future work.
Related Work
During the recent, rapid development of conversational agents, also known as chatbots, various applications have been created.
Open-domain chatbots such as Facebook’s BST (Roller et al., 2021)
and
Google’s Meena (Adiwardana et al., 2020) seek to be more human-like, engaging in conversation on any topic.
Closed-domain chatbots instead focus on improved task performance, for instance
GuessWhich (Das et al., 2017), persuasion (Wang et al., 2019; Shi et al., 2020), and negotiation (Afantenos et al., 2012; Papangelis and Georgila, 2015; Lewis et al., 2017;
He et al., 2018).
To negotiate item distribution (book, hat, ball),
Lewis et al.
(2017) apply a bi-directional GRU model to train a language model and use reinforcement learning with self play to develop data-driven strategies.
For price bargaining on a single item (e.g., a TV), He et al.
(2018) use a hybrid approach involving rule-based and LSTM models that decouple natural language understanding, dialogue act prediction, and natural language generation to facilitate controllable negotiation strategies.
However, these scenarios do not attempt to find an optimal solution for both sides, and do not control the falsehoods generated by sellers.
These limitations motivate this work.
Conclusion
We introduce a novel negotiation task and present
FruitStand, a rich dataset of human-human dialogues, for negotiation dialogue research.
We demonstrate the effectiveness of reinforcement learning in guiding the conversational agent toward a specific goal.
Finally, our experiments in falsehood suppression show the potential of RL for truthful AI.
A more robust falsehood detector would be our first future work.
In our initial observations, a strong Natural Language Inference (NLI) model could play this role.
Acknowledgements
We are grateful for the insightful comments from anonymous reviewers.
This work is supported by the Ministry of Science and Technology of Taiwan under grant numbers MOST111-2634-F-001-001.
B FruitStand Interface (Cont.)
C Behavior Control