Designing an Automatic Agent for Repeated Language–based Persuasion Games (2022)

Abstract: Persuasion games are fundamental in economics and AI research and serve as the basis for important applications. However, work on this setup assumes communication with stylized messages that do not consist of rich human language. In this paper we consider a repeated sender (expert) – receiver (decision maker) game, where the sender is fully informed about the state of the world and aims to persuade the receiver to accept a deal by sending one of several possible natural language reviews. We design an automatic expert that plays this repeated game, aiming to achieve the maximal payoff. Our expert is implemented within the Monte Carlo Tree Search (MCTS) algorithm, with deep learning models that exploit behavioral and linguistic signals in order to predict the next action of the decision maker, and the future payoff of the expert given the state of the game and a candidate review. We demonstrate the superiority of our expert over strong baselines and its adaptability to different decision makers and potential proposed deals.1

## introduction natural language processing (nlp) has made substantial progress in recent years, excelling on text understanding applications such as machine translation @xcite , information extraction @xcite , and question answering @xcite . however, these applications do not assume that language is used for interaction between strategic participants whose objectives overlap only partially. in contrast, in the fields of economics and artificial intelligence (ai), such setups have been widely explored. for example, the settings of personalized advertising and targeted recommendation systems @xcite @xcite suggest personalized services for their customers, and solutions are formed as strategic sender-receiver interactions @xcite . however, this work assumes stylized messaging that does not involve real-world natural language. in this paper we address the setting of senderreceiver interaction, but, in contrast to previous research, we assume natural language interaction between the players in an iterative non zero-sum persuasion game. in our setting the two participants are strategic players with their own private utilities. crucially, the sender has more information about the world than the receiver does. taking the nlp perspective, we are particularly interested in the persuasion game setting, where the sender's objective is to persuade the receiver, using natural language messages, to select an action from a set of alternatives. the receiver, in turn, has different payoffs for the different actions. the receiver's payoff depends on properties of the setup that are unavailable to her, and she has a higher level of uncertainty about the setup than the sender has. our focus is on repeated non-cooperative setups, where the utilities of the players do not fully overlap. consider a repeated persuasion game where the interests of the players are aligned. in such a case, the sender should reveal the complete information she possesses, letting the receiver take an action that maximizes both their payoffs. in a repeated non-cooperative setup, in contrast, the sender opts to reveal a piece of information that should yield her a high payoff but also maintain a trustful relationship with the receiver, in order to avoid damaging her reputation and hence possibly also her future payoff. designing agents to play games is a long standing goal of deep reinforcement learning (rl) research. however, these games are typically zero-sum games, modeled as a utility maximization problem (see, e.g., @xcite and the references within). in contrast, in economic contexts like ours, games are rarely zero-sum. a commerce web site that aims to recommend a hotel cares about the customer choosing the hotel, while the customer cares about the hotel quality; their incentives are non-identical, but are also non-opposite. these games cannot be solved as a maximization problem, and there is in fact no optimal player in such problems @xcite . incontrast to economic games where the communication among agents is typically through formal signals or bids @xcite , we focus on natural language communication, which is very natural to persuasion games. recently, @xcite were the first to adapt the aforementioned setup to natural language messaging. specifically, they designed a repeated persuasion game in which an expert (travel agent) repeatedly interacts with a decision-maker @xcite . at each trial of the interaction the expert observes a hotel alongside its scored textual reviews, and should choose a single review to reveal to the dm, in a hope to convince her to choose the hotel. the dm, in turn, can choose to either accept or reject the hotel, and her payoff stochastically depends on the review score distribution available to the expert only. finally, both players observe their payoffs and proceed to the next, similar, step of the game. while @xcite focus on predicting the dm's actions, we adapt their setting and aim to design an artificial expert (ae) that should take the expert role in a way that maximizes its payoff. our ae is implemented within the monte carlo tree search (mcts) algorithm, which has been extensively used in ai-based game playing ( §4.1). we present language-and behavior-based deep learning models for two crucial components of the mcts: (a) a decision making model (dmm), which predicts the actions taken by the dm given the current state of the game; and (b) a value model (vm), which predicts the future payoff of the ae given the current state of the game and a potential review that can be presented at the current step. we focus on three questions: (1) can our ae achieve a high payoff? (2) does our ae adapt its strategy to different decision maker types? and (3) do our automated ae's strategies resemble those of human aes? we test our ae against various types of artificial dms, compare it to strong alternative experts, and demonstrate its superiority. we further show that our ae is able to adapt its strategy to the dm it faces. we evaluate the impact of proper modeling of the linguistic signal (revealed reviews), comparing a bert-based approach to hand-crafted features, and show that the later are generally better. further, we analyze the reviews chosen by our ae, shedding light on its strategy. lastly, we also test our ae against human dms, comparing its performance to a strong baseline. we provide a detailed analysis of the pros and cons of our ae, and discuss the differences between evaluation with human and simulationbased dms. ## task definition we consider a two-player, travel agent (expert) and customer (decision-maker, dm), repeated persuasion game. the game, first introduced by @xcite , consists of a sequence of ten trials. in each trial, the expert observes seven reviews of a given hotel, alongside their scores, and she then sends the dm one of the reviews, without its score. based on this review, the dm decides between two options: accepting or rejecting the hotel. if the hotel is not accepted by the dm, the payoff of both players is 0. otherwise, the expert's payoff is 1, and the dm's payoff is a score randomly sampled from the seven scores presented to the expert at the beginning of this trial, referred to as the lottery result, minus the constant 8. this constant imposes a zero expected payoff for a dm who chooses to accept the hotel in all the ten trials. 2 a more abstract description of each trial in this multi-stage game would be as follows. every hotel is associated with an unknown distribution over payoffs, corresponding to the distribution over experiences that guests will have at this hotel. the scored reviews are sampled from this distribution, and the dm's reward is another sample from the distribution. because in our setting we do not have access to the real payoff distribution of each hotel, we approximate it using the empirical distribution from the payoffs observed by the expert. formally, denote the suggested hotel at trial t by h t , the dm's decision at this trial by a t , where a @xmath0, and the seven scores attached to the reviews of h t by s t 1 , s t 2 , ..s t 7 , where s t @xmath1, 10]. the players' payoffs are: while the two players would ideally like to gain the highest possible payoff (i.e., this is not a zero-sum game), their strategies are not necessarily coordinated. particularly, while the expert aims to sell as many hotels as possible, the dm aims to accept only hotels that are likely to yield a positive payoff. note that the dm is not fully informed of the hotel state, and should make her decision based on the partial information provided by the expert. the repeated nature of the game adds complexity to the decisions, as the expert's choice in a specific trial affects not only the dm's decision in this trial but also the expert's reputation in the next trials. let us consider the game from the expert's point of view. consider an expert who cares solely about the present and reveals a high-score review in order to tempt the dm to choose the hotel, even if the acceptance decision is likely to yield a negative payoff. this expert is likely to gain a high payoff at the first few rounds. however, as the game proceeds the dm would probably understand that the expert is unreliable. on the other hand, if the expert reveals only reviews that reliably describe the hotel (e.g., the median scoring reviews), the dm is likely not to choose the hotel when she is presented with mediocre reviews. @xcite provide an equilibrium analysis of our game. this is a theoretical analysis, under some constraining assumptions and, as the authors demonstrate, the players do not follow it in practice. this further motivates our work, which aims to design an nlp-based agent of the expert in this game. note, that our approach is different from that of @xcite , who aimed to predict individual decisions of the dm, rather than constructing an artificial dm or expert. data we use the dataset collected by @xcite using amazon mechanical turk. 3 the dataset is composed of 509 ten-trial games. the participants were randomly and anonymously paired, and each of them was randomly selected to be in one of the two roles: dm or expert. the training set consists of 408 games. in these games the same hotels and reviews were used, but the hotels were randomly permuted between the 10 trials. the test set consists of 101 games, played with a different set of hotels and reviews, such that the hotels are again randomly permuted. each participant was allowed to participate in the experiment only once, such that the training and test sets consist of different players. each hotel is accompanied by seven reviews collected from the booking.com web site along with their scores, continuously ranging between 0 and 10 (see an example review in figure 1 ). all the reviews contain at least 100 characters and are separated into positive and negative parts. figure 1 demonstrates a sampled review from the dataset. the order in which each of these parts were presented to the experts was also assigned at random. for more details, see @xcite . ## method we design an ae that aims to maximize its payoff in the persuasion game. ## experiments experimental setting evaluating our ae against humans is highly expensive and time-consuming, and hence infeasible at large scales. we hence start with another, widely used solution: human simulations @xcite @xcite @xcite . in this approach we evaluate the ae against an automatic algorithm that simulates human dms. although this evaluation is not performed against actual humans, it allows us to evaluate the ae against various types of players, by changing the data-driven dm in a controlled manner. we perform 1000 simulated games over the test set per dm simulator, where the order in which the hotels are presented to the ae is randomly permuted at each simulation. we employ two dmms (hc-lstm and bert-lstm) as our basic dm simulators, as they are trained to imitate the human dm's behavior in the game. we further modify the behavior of these ''human like'' dmms, by changing their hotel acceptance probability in a controlled manner. we consider: (a) α-compromised dmms, where the acceptance probability is increased by α = 0.1 or α = 0.2 over the prediction of the basic dmm; and (b) α-inflexible dmms, where the acceptance probability is similarly decreased. baselines we next describe the baselines for the ae and for its components, the dmm variants (hc-lstm and bert-lstm), and the vm variants (hc-lstm and bert-lstm). dmm. the dmm decides in each trial whether to accept a suggested hotel or not. we propose four different dmm variants, differing in their decision strategy, architecture and features: (a) hc-svm -a support vector machine (svm; @xcite based on the hc and sg features. it allows us to evaluate the power feature name feature description feature formulation behavioral features hotelacceptance avg #trials where the hotel was accepted t-1 @xmath2.* t-1 @xmath3.* t-1 @xmath4.* t-1 @xmath5.* t-1 @xmath6.5 was accepted. t-1 @xmath7.5} t-1 ¬excellenthotel acceptance avg #trials where a hotel with average score higher than 9.5 was accepted. t-1 @xmath8.5} t-1 dmpayoff avg dm's payoff pet trial t-1 @xmath9.* t-1 @xmath10.* t-1 @xmath11.* t-1 @xmath12 ## results this section presents our results. we would first like ( §6.1) to evaluate the performance of our dmm and vm models, since they are key elements of our ae. after verifying their quality, we turn to present our main results ( §6.2), comparing our ae to the various baselines. this will allow us to answer our three research questions ( §1), related to the ae performance (q1), its adaptation to different decision maker types (q2), and its strategy compared to humans (q3). f1-score of 73.20. this result reflects the value of the hand-crafted textual features, a pattern that was also reported by @xcite . bert-lstm lags a bit behind (accuracy of 80.80%, macro f1 score of 68.30), demonstrating that clever feature design can outperform this strong language encoder. in general, the svm baselines fall short of the neural networks, whereas the deterministic baselines pd and ewg are not very successful. ## ablation analysis in this section we analyze several aspects of the main results presented above. we start by analyzing the impact of text-based communication on our results, evaluating the performance of our ae when performing numerical communication. then, we analyze different aspects of the observed behavior of our ae in our main text-based communication experiments: the average payoff of the dms (indicating whether our ae facilitates fairness), the decision patterns of our ae when playing against the various dms (shedding more light on q2-does the ae adapt to the dm it plays with), comparing the reviews revealed by our ae to those revealed by the human experts in @xcite (thus shedding more light on q3), and, finally, analyzing the textual properties of the reviews revealed by our ae. ## human experiments finally, we evaluate our ae when playing with human dms. we do believe that simulation-based evaluation is crucial for our setting as it allows us to test our ae against dms with a variety of controlled characteristics at a relatively low-cost (see §5). yet, human-based evaluation, even if it is small-scale due to its high cost, provides important complementary information. following @xcite , our ae plays with 100 different human dms on the amazon mechanical turk (amt) platform, 7 such that no dm competes against more than one expert. 8 we follow the same experimental setting as in our simulations, and particularly use the same test-set hotels. we compare the performance of our ae to those of the strongest alternative: highest, the second best baseline in our simulations (in terms of average performance; the various ae agents are not considered as baselines in this definition). figure 5 (left) illustrates the average expert payoff for hotels with an average review score of at most @xmath13, . . . , 10}. the results suggest that our ae achieves the highest average payoffs for the 4 hotels with the lowest average review score (average score of up to 8), that is, the hotels for which the expected dm payoff is negative. this observation implies that our ae is able to maximize its payoff on the most challenging hotels. the highest agent excels on the other 6 hotels, those with an average review score higher than 8 and hence a positive expected dm payoff. interestingly, 5 of these 6 hotels have a review with the maximal score of 10, which is chosen by highest. we next analyze the scores of the revealed reviews-that is, the reviews that were chosen by the experts and presented to the dms. figure 5 (middle) presents the average expert payoff when its revealed review score is at most 7 https://www.mturk.com . 8 we followed the exact same amt experimental setup as in @xcite . particularly, we filtered the amt workers according to the two attention checks described in section 4.1 of their paper. @xmath14, . . . , 10}. while the highest agent achieves the best payoff when it reveals a review with the maximal score of 10, when moving to lower scores we see that our agent maintains a higher average payoff. for such cases where the hotel does not have any review with the score of 10, the highest agent achieves a low average payoff of 4.2. the final analysis (figure 5 (right)) is similar to first two, except that now we are focusing on the average dm payoff, when the revealed review score is at least @xmath15, . . . , 10}. the leftmost point, corresponding to all experiments, suggests that in total the human dms who played with our ae achieve the lowest average payoff. however, we notice that as the ae chooses to reveal reviews with higher scores the average dm payoff increases and surpasses the average payoff of the dms who played with the highest agent. this is an interesting pattern, given that the ae is trained to maximize its own payoff, and its objective does not take the dm's payoff into account. finally, ## conclusions we consider the problem of automatic expert design for a repeated non-cooperative persuasion game. our ae is based on the mcts search algorithm with deep learning models for dm decision and expert's future payoff predictions. our experiments quantitatively and qualitatively analyze the performance of our ae in comparison to a large variety of alternatives. while our main evaluation is with simulated (automatic) dms, we also examine the generalizability of our results to experiments with human dms. our work relies on the dataset of @xcite for training and testing the various expert models. one limitation of this dataset is its size: it is based on only 10 training and 10 test hotels, each with 7 scored reviews. moreover, the training set, which is used for training our dmm and vm models, consists of only 408 ten-trial games. we aimed to compensate for this by performing a large number of simulations (1000) for each expert/dm pair, and by reporting 95% confidence intervals (cis), demonstrating limited overlap between the 95% ci of our ae and the baselines. yet, richer datasets in terms of the size and diversity of the hotel sets, as well as the richness of interaction between the human players, are required in order to further validate our results. in future we would like to extend our ae in three main directions: (a) designing end-to-end architectures, where the dmm and vm are jointly trained in order to maximize the ae's payoff; (b) letting the ae generate persuasive language rather than choosing from pre-written reviews; and (c) considering other ae strategies such as fair payoff division between the expert and the dm, instead of maximizing the ae's payoff.