Learning to Mitigate AI Collusion on Economic Platforms
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Algorithmic pricing on online e-commerce platforms raises the concern of tacitcollusion, where reinforcement learning algorithms learn to set collusive pricesin a decentralized manner and through nothing more than proﬁt feedback.
Thisraises the question as to whether collusive pricing can be prevented through thedesign of suitable "buy boxes," i.e., through the design of the rules that governthe elements of e-commerce sites that promote particular products and prices toconsumers.
In this paper, we demonstrate that reinforcement learning (RL) can alsobe used by platforms to learn buy box rules that are effective in preventing collusionby RL sellers.
For this, we adopt the methodology of Stackelberg POMDPs , anddemonstrate success in learning robust rules that continue to provide high consumerwelfare together with sellers employing different behavior models or having out-ofdistribution costs for goods.
Introduction
The last decade has witnessed a dramatic shift of trading from retailers to online e-commerce platformssuch as Amazon and Alibaba.
In these platforms, sellers are increasingly using algorithms to set prices.
Algorithmic pricing can be beneﬁcial for market efﬁciency, enabling sellers to quickly react to marketchanges and also in enabling price competition.
At the same time, the U.S. Federal Trade Commission(FTC)
U.S. Federal Trade Commission (2018) and European Commission (The Organisation for
Economic Co-operation
and Development, 2017) have raised concerns that algorithmic pricing mayfacilitate collusive behaviors.
Calvano et al.
(2020a) support these concerns through a study of pricingagents in a simulated platform economy, and show that commonly used reinforcement-learning (RL)algorithms learn to initiate and sustain collusive behaviors.
Assad et al.
(2020) also provide empiricalsupport for algorithmic collusion in a study of Germany’s retail gas stations, showing an associationbetween algorithmic pricing and an increase in price markups.
As highlighted by Calvano et al.(2020b), these kinds of collusive behaviors are unlikely to be a violation of antitrust law, as they arelearned responses to proﬁt signals and not the result of explicit agreements.
One can try to prevent algorithmic collusion by introducing suitable rules by which platforms canchoose which sellers to promote to buyers, thus promoting competition.
Could Amazon’s "buy boxalgorithm," for example, play this role in the future, in determining for a given consumer searchwhich products and prices to highlight to a consumer?
Responding to this, Johnson et al.
(2020)design hand-crafted rules that succeed in hindering collusion between RL algorithms.
At the sametime, their rules introduce the undesirable effect of limiting consumers to a single seller, and thereremains potential for more effective interventions.
In this paper, we demonstrate for the ﬁrst time how RL can also be used defensively by a platformto automatically design rules that promote consumer welfare and prevent collusive pricing.
This isa problem of multi-agent learning, with the interaction between the platform and sellers modeled as a Stackelberg game (Fudenberg and Tirole, 1991).
The leader is the platform designer and setsthe platform rules and the sellers respond, using RL to set prices given these rules.
We introduce theclass of threshold platform rules , and formally show that this class contains rules that approximatelymaximize consumer surplus in a unique subgame perfect equilbrium.
At the same time, this class ofthreshold rules is fragile to unexpected deviations by sellers, for example caused by cost perturbations.
The role of RL on the part of the platform is to learn rules with similar performance that are alsomore robust.
To solve the Stackelberg problem, we make use of the Stackelberg partially observable Markovdecision process (POMDP) framework (Brero et al., 2021a), which deﬁnes an episode structureof a POMDP such that the RL algorithm representing the leader will learn to optimize its reward(here, consumer surplus) given that its rules cause re-equilibration on the part of the followers (here,the sellers who use Q-learning algorithms to set prices).
The Stackelberg POMDP framework iswell formed as long as the re-equilibration behavior of sellers can be modeled through Markoviandynamics, as is the case with Q-learning.
We show successful results in learning effective platform policies that outperform handcraftedrules (Johnson et al., 2020).
This demonstrates how the Stackelberg POMDP framework can besuccessfully applied in settings where followers play repeated games, and their strategies are alsopolicies trained via reinforcement learning algorithms.
We then show how our threshold platformrules allow us to obtain a similar learning performance when training the platform policy "in thewild,” i.e., without accessing the sellers’ private information.
With this, we demonstrate how the
Stackelberg POMDP framework can be applied in more general learning scenarios than the ofﬂinelearning ones for which it was originally designed.
Finally, we show how the platform rules learnedvia our Stackelberg POMDP framework continue to be effective when market conditions change, forexample as the result of a change to the cost structure of sellers.
Further related work.
Zheng et al.
; Tang (2017);
Shen et al.
(2020); Brero et al.
(2021b)make use of RL to optimize different economic systems (including matching markets, internetadvertising, tax policies, and auctions) under strategic agents’ responses.
Unlike our work, thesemethods do not leverage the designer’s commitment power or the Stackelberg structure of the inducedgame.
Brero et al.
(2021a) introduce and study the Stackelberg POMDP framework for a very differentsetting than that of the present paper: the design of sequential price auctions.
1 Abada and Lambin(2020) study collusion by RL pricing in markets for electric power, and use machine learning by aregulator agent for the mitigation of collusion, albeit without a Stackelberg framing (and withoutsuccess, leading to lower welfare than the collusive outcome).
The broader research program ondifferentiable economics uses representation learning for optimal economic design (Duetting et al.
Shen et al., 2019; Kuo et al., 2020;
Tacchetti et al., 2019; Rahme et al., 2021a; Curry et al.
,2022; Rahme et al., 2021b;
Curry et al., 2020; Peri et al., 2021); this work avoids the need for Stackelberg design by emphasizing the use of direct, incentive-compatible mechanisms.
Also relatedis empirical mechanism design (Areyan Viqueira et al., 2019; Vorobeychik et al., 2006; Brinkmanand Wellman, 2017), which applies empirical game theory to search for the equilibria of mechanismswith a set of candidate strategies (Wellman, 2006; Kiekintveld and Wellman, 2008; Jordan et al. ,2010); see also Bünz et al.
(2018) for the design of iterative auctions.
Preliminaries
Seller Competition Model.
There is a set of sellers N = { 1 , . . . , n } , each of whom sells adifferentiated product on an economic platform.
Each seller has the same (private) marginal costc > 0 for producing one unit of its product.
Sellers interact with each other repeatedly over time insetting prices and selling goods.
At each time period, t = 0 , 1 , . . .
, each seller i observes all pastprices, and sets a price p i,t  0 .
We let p t
=
( p 1 ,t , . . .
, p n,t ) denote a generic price proﬁle quoted at time t .
The platform sets the rules of a buy box that governs, in each period t , which set N t ✓ N ofsellers are available.
Consumers can only buy from these sellers and others forfeit sales.
There isalso an outside option, indexed by 0 , which provides each consumer with a fallback choice with zeroutility.
Following Johnson et al.
(2020), competition between sellers for consumer demand is modeledthrough the standard logit model of consumer choice.
For this, seller i has quality index ↵ i > 0 ,this providing horizontal differentiation across products, and the outside good has quality index↵ 0 > 0 .
In the logit model, each consumer samples ⇣ 0 , ⇣ 1 , ... ⇣ n , independently from a type I extremevalue distribution with scale parameter µ > 0 , for each product and the outside option, with utility↵ i + ⇣ i  p i,t for product i , and ↵ 0 + ⇣ 0 for the outside option.
Considering a unit mass of consumers inperiod t , seller i 2 N t receives fractional demand D i ( p t ; N t ) =
exp(( ↵ i  p i,t ) /µ ) /
( p t ; N t ) , where 94  ( p t ; N t ) =
P j 2 N t exp(( ↵ j  p j,t ) /µ )
+exp( ↵ 0 /µ ) , and any seller i / 2 N t has zero demand.
Scaleparameter µ > 0 serves to control the extent of horizontal differentiation, with no differentiation andperfect substitutes obtained as µ ! 0 .
The total consumer surplus is U ( p t ; N t ) = µ · log[  ( p t ; N t )] ,and is maximized with minimum prices and all sellers displayed (so consumers have a full choice ofproducts).
Seller i ’s proﬁt ⇢
i in period t is ⇢ i ( p t ; N t )
=
( p i,t  c ) · D i ( p t ; N t ) , and its per-unit proﬁtmultiplied by demand.
Reinforcement learning by sellers.
In a single-agent Markov decision process (MDP), an agentfaces a sequential decision problem under uncertainty.
At each step t , the agent observes a statevariable s t 2 S and chooses an action a t 2 A .
Upon action a t in state s t , the agent obtainsreward r ( s t , a t ) , and the environment moves to state s t +
according to p ( s t +1 | s t , a t ) .
We let⌧ =
( s 0 , a 0 , ..., s T , a T ) denote a state-action trajectory determined by executing policy policy⇡ : S !
A , and p ⇡ ( ⌧ ) denote the probability of trajectory ⌧ .
The optimal policy ⇡ ⇤ solves⇡ ⇤ 2 argmax ⇡ E ⌧ ⇠ p ⇡ ( ⌧ ) [
P
Tt =
t r ( s t , a t )] , where  2 [0 , 1] is the discount factor and time-horizon
T can be ﬁnite or inﬁnite.
In a partially-observable MDP (POMDP) , the policy ⇡ cannot accessstate s t but only observation o t sampled from p ( o t | s t ) .
A multi-agent MDP (Boutilier, 1996) forn agents has states S common to all agents and a set of actions
A i for each agent i .
When eachagent i picks action a i,t in state s t , the environment moves to state s t +
according to a distributionp ( s t +1 | s t , a 1 ,t , .., a n,t ) and agent i obtains a reward r i ( s t , a t ) that depends on the joint action.
Wefollow Calvano et al.
(2020a)
and Johnson et al.
and adopt decentralized Q-learning by sellersas a positive theory for sellers in regard to their behavior in setting prices on an e-commerce platform(see Appendix A).
Although Q-learners may not converge, we also conﬁrm these earlier studies inshowing convergence in our simulations (deﬁned over a particular time horizon as detailed by Johnsonet al.
The Platform Stackelberg Problem
To formalize the problem facing the platform in mitigating collusive behavior by sellers, we modelthe interaction between the platform, which sets the rules of the buy box, and the sellers as a
Stackelberg game .
The platform designer is the leader, and ﬁxes the platform rules.
The sellers arethe followers, and play an inﬁnitely repeated game according to these rules.
As discussed above, andfollowing Calvano et al.
(2020a)
and Johnson et al.
, we model the sellers’ behavior throughdecentralized Q-learning.
As a result, the problem facing the platform is a behavioral Stackelbergproblem , in that the followers are modeled as Q-learners (and need not, necessarily, be playing anequilibrium of the induced game).
The sellers.
In this model, we ﬁx the states that comprise the MDP of a seller to include the pricesset by all sellers in the last period, i.e., s t = p t  1 .
We initialize s 0 to be a randomly selected priceproﬁle.
The action of a seller is modeled as one of m equally-spaced points in the interval rangingfrom just below the sellers’ cost c to just above the monopoly price p m .
At each step t  0 , eachseller i selects a price p
i,t and is rewarded by its per-period proﬁt ⇢ i ( p t ; N t ) , which depends onp t
= ( p 1 ,t , . .
. , p n,t ) and the choice of which sellers N t are displayed by the platform.
The platform.
To formalize the platform’s problem, let  ⇤ =
(  ⇤ 1 , ..,  ⇤ n ) denote a strategy proﬁleselected by Q-learning on the part of sellers, in response to the platform rule, and in the long run, aftera suitably large number of steps.
We leave implicit here the dependence of seller strategy proﬁle on the platform’s policy.
The platform must decide in each period which sellers to display to consumers.
For this, we denote the platform rule as policy ⇡ , and we adopt for the state of the platform policythe prices quoted by sellers in step t , p t , so that the platform’s policy uses these prices to select aset of agents to display, with N t selected according to ⇡ ( p t ) .
Let p ⇤ t =
⇤ ( s t ) denote a price proﬁlechosen under seller strategies  ⇤ , i.e., in response to the platform rules, and at some large enoughtime step t ⇤ , and let ⌧ ⇤ =
( p ⇤ t ⇤ , p ⇤ t ⇤ +1 , .. ) denote a trajectory of prices forward from t ⇤ .
We denotethe distribution of these trajectories as p ⇡ ( ⌧ ⇤ ) .
As above, the dependence on the platform’s policy isleft implicit in this notation.
The Stackelberg problem facing the platform is to ﬁnd a platform policy ⇡ that maximizes consumersurplus given the effect of this policy on the induced strategy proﬁle of sellers.
Deﬁnition 1 (Behavioral Stackelberg Problem)
The optimal platform policy solves ⇡ ⇤ 2 146 argmax ⇡ CS ( ⇡ ) , where CS ( ⇡ ) is the expected sum consumer surplus when sellers follow strategy  ⇤forward from period t ⇤ , i.e.,
[Equation]
where T ⇤ is suitably chosen horizon and p ⇡ ( ⌧ ⇤ ) denotes the distribution over Q-learning induced,seller pricing trajectories, in response to platform policy ⇡ .
Learning Optimal Platform Rules
In this section, we solve the platform’s problem, in responding to Q-learning sellers, through the
Stackelberg POMDP framework (Brero et al., 2021a).
This creates a suitably deﬁned POMDP inwhich the optimal policy solves the behavioral Stackelberg problem (Deﬁnition 1).
Deﬁnition 2 (Stackelberg POMDP for platform rules)
The Stackelberg POMDP for platformrules is a ﬁnite-horizon POMDP, where each episode has the following two phases:
• An equilibrium phase , consisting of n e  1 steps.
In this phase, each state s t includes the stepcounter t , the sellers’ current Q-matrices, and the prices p t quoted by the agents.
Observationsconsists of the prices quoted by the sellers ( o t = p t ) and policy actions determine the set of agentsdisplayed (in their more general version, a t = N t ).
State transitions are determined by Q-learning,
where each seller i updates its Q-matrix after being rewarded by ⇢ i ( p t ; N t ) .
The policy has zeroreward in every time step ( r ( s t , a t ) = 0 , for t  n e ).
• A reward phase , consisting of n r  1 steps, each with the same actions, states, and observations as
the equilibrium steps.
The reward phase differs in two ways.
First, the Q-matrices of sellers are not
updated, and second, the platform policy now receives a non-zero reward, and this is set in each stepto be equal to the consumer surplusin that step ( r ( s t , a t ) =
U ( p t ; N t ) , for t > n e ).
This Stackelberg POMDP formulation is an adaptation of that provided by Brero et al.
(2021a),who used it to learn sequential price mechanisms (SPMs) in the presence of communication frombidders.
Here, our stage games replace SPMs, and the followers respond through Q-learningdynamics rather than no-regret algorithms.
Following Brero et al.
(2021a), we show the Stackelberg
POMDP formulation is well-founded by showing that an optimal policy will also solve the Behavioral
Stackelberg design problem of Deﬁnition 1.
Speciﬁcally, when the number of reward steps n r islarge enough and when n e  t ⇤ , the optimal policy, denoted ⇡ ⇤ n e ,n r , for the Stackelberg POMDP 173 with n e equilibrium and
n
r reward steps maximizes the objective in Equation (1).
Proposition 1 The optimal Stackelberg POMDP policy ⇡ ⇤ n e ,n r , for an equilibrium phase with n e  1steps and a reward phase with n r  1 steps, maximizes CS ( ⇡ ) , for seller behavior induced after n esteps when n r = T ⇤ .
The proposition follows from the construction of the Stackelberg POMDP, especially the fact the ourpolicy is only rewarded under the response behavior reached after n e steps, in line with the deﬁnitionof CS ( ⇡ ) (see Appendix B for the full proof argument).
Brero et al.
(2021a) use the Stackelberg POMDP framework in an "ofﬂine” environment, i.e., ina simulation that assumes access, at design time, to followers’ internal information.
This allows them to solve their POMDP using the paradigm of centralized training and decentralized execution(Lowe et al., 2017).
The leader policy is trained via an actor-critic deep RL algorithm, and the criticnetwork (which estimates the sum of rewards until the end of the episode) accesses the full stateduring training.
Only the actor network, which represents the policy, is restricted to the partial-stateinformation.
Here, we also study the use of the Stackelberg POMDP framework to train useful leader policies "inthe wild,” where the learning algorithm of the platform can only access the kind of information that aneconomic platform would have based on observations of sellers.
As we will empirically demonstrate,we can successfully relax the ofﬂine learning requirements—i.e., we operate without (1) access tosellers’ private information in regard to Q-matrices and exploration rate, and (2) requiring that
the
Q-matrices of sellers become frozen for the reward phase of the Stackelberg POMDP—withoutaffecting learning performance.
Threshold platform rules.
In our experiments, we consider the class of threshold platform rules .
These rules use the current prices set by sellers to set a price threshold above which a seller will notbe displayed, with the same threshold set for all sellers.
Deﬁnition 3 (Threshold Platform Rule)
A threshold platform rule sets a threshold ⌧ ( p t )  0 , foreach price proﬁle p t , such that N t
= { i 2 { 1 , .., n } : p i,t  ⌧ ( p t ) } , i.e., any seller whose price is no 199 greater than the threshold is displayed to consumers.
This class of threshold rules has a corresponding optimality result: there is a threshold rule that makesthe market competitive, with all sellers displayed and consumer surplus maximized in the uniquesubgame perfect Nash equilibrium (SPE) of the induced continuous pricing game.
Even though thepricing behaviors that arise from Q-learning need not converge to SPEs, we ﬁnd empirical evidence,consistent with Johnson et al.
(2020), that the seller learning dynamics invariably converge to thisequilibrium.
As such, this provides useful theoretical support for adopting the family of thresholdplatform rules by the platform learner.
We have the following result:
Proposition 2 For any ✏ > 0 , there exists a threshold platform rule ⇡ such that CS ( ⇡ )  CS ( ⇡ ⇤ )
P t  t ✏ under the unique subgame perfect Nash equilibrium (SPE) of any ﬁnitely-repeated continuouspricing game induced by platform rule ⇡ .
This proposition follows from a platform rule with a limiting threshold that is arbitrarily close to thesellers’ cost c (see Appendix C for the proof).
Under this rule, sellers are displayed only if their priceis minimal.
At the same time, this particular threshold rule is fragile, and would lead to market failureif seller costs vary in unexpected ways.
By letting the threshold ⌧ also vary with the price proﬁle p t ,as is allowed by the family of threshold platform rules, we seek to learn milder interventions that stillmitigate collusion but remain robust to variations in the costs faced by sellers in the marketplace.
Experimental Results
In this section, we evaluate our learning approach via three main experiments.
We ﬁrst considerperformance in terms of consumer surplus, benchmarking our RL interventions against the onesintroduced by Johnson et al.
We demonstrate the ability to learn optimal leader strategies inthe Stackelberg game with the followers across all the seeds we tested, signiﬁcantly outperformingexisting interventions.
We then train platform rules without access to the sellers’ private information("in the wild"), and show that this is not crucial for our learning performances.
We conclude bytesting the robustness of our interventions, adding price perturbation during training and evaluatingthe effect on the robustness of our learned platform rules in environments where sellers have differentcosts from those assumed during training.
Experimental set-up.
As in Calvano et al.
(2020a)
and Johnson et al.
, we consider settingswith two pricing agents with cost c
= 1 , quality indexes ↵ 1 = ↵ 2 = 2 , and ↵ 0 = 0 , and we setparameter µ = 0 .
25 to control horizontal differentiation.
The seller Q-learning algorithms are also trained using discount factor  = 0 .
95 , exploration rate " t = e   t with  = 1 e  5 , and learningrate ↵ = 0 .
We also include results for variations of this default setting in Appendix D.
We adopt ﬁve prices choices for the action of each seller, these prices ranging from just below thesellers’ cost to the monopoly price.
Earlier work provided sellers with a choice of ﬁfteen differentprices (over a similar range).
We need a smaller grid in order to satisfy our computational constraints;earlier work studied the effect of different, hand-designed platform rules, and did not also use RL forthe automated design of suitable platform rules.
We also follow the choices of earlier work, and studyan economy with two sellers (again, for reasons of computational resources).
This coarsened pricegrid allows us to train a platform policy through Stackelberg POMDP for 50 million steps in 18 hoursusing a single core on a Intel(R) Xeon(R)
Platinum 8268 CPU @
2.90GHz machine.
Learning algorithm.
To train the platform policy, we start from the A2C algorithm provided by
Stable Baselines
(Rafﬁn et al., 2021, MIT License).
Given that our policy is only rewarded atthe end of a Stackelberg POMDP episode, we conﬁgure A2C
so that neural network parametersare only updated after this reward phase.
In this way, we guarantee that policies inducing desiredfollowers’ equilibria are properly rewarded.
Furthermore, to reduce variance in sellers’ responsesdue to non-deterministic policy behavior, we maintain an observation-action map throughout eachepisode.
When a new observation is encountered during the episode, the policy chooses an actionfollowing the default training behavior and stores this new observation-action pair in the map.
Wewill show the importance of this variation via an ablation study that is presented in Appendix E.
Sellers restart the Q-learning process by re-initializing exploration rates every time the platform ruleschange (i.e., at the beginning of every Stackelberg POMDP episode).
In Appendix F, we also showhow the training approach is robust to different sellers’ behavior models, where the sellers restart thelearning rate asynchronously, and not necessarily at the beginning of episodes.
5.1 Platform Learning Performance
In this section, we evaluate the performance of our learned platform policies.
For this, we train ourpolicies for 50 million steps in total.
We set up the Stackelberg POMDP environment using 50kequilibrium steps and 30 reward steps.
In these initial experiments, we train our policies using thecentralized training-decentralized execution paradigm as used for this Stackelberg learning problemby Brero et al.
(2021a), giving the critic network access to the sellers’ learning information (i.e.,
Q-tables and exploration rates).
We relax this below in studying the robustness of the computationalframework to online training ("in the wild.")
We consider the following interventions on behalf of theplatform designer:
• No intervention: Sellers are always displayed, no matter the price they quote.
To derive this baseline,we run our Q learning dynamics until convergence (as described in Johnson et al.
for eachseed
and then average the surplus at ﬁnal strategies.•
PDP: We test price-directed prominence , a platform intervention introduced by Johnson et al.(2020).
Here, the platform only displays the seller who quotes the lower price (breaking ties atrandom), thus enhancing competition.
As for no intervention , we compute the performance of PDPby averaging consumer surplus after Q-learning dynamics converge.•
DPDP:
Dynamic price-directed prominence is another intervention introduced by Johnson et al.(2020), which also conditions the choice of the (unique) displayed seller on past prices.
Under thisintervention, quoting prices equal to cost is a subgame perfect equilibrium of the induced game (undersuitable discount factors).
As for the previous baselines, we compute the performance of DPDP byaveraging consumer surplus after Q-learning dynamics converge.•
No State RL:
Here we use the Stackelberg POMDP methodology to train a platform policy thatdoes not use prices p t to determine the threshold at which to admit each seller to the buy box (thus,"no state").
Here, Q-learning is restarted whenever a Stackelberg POMDP episode begins.•
No Stackelberg No State RL: A variation on "No State RL" that does not use the Stackelberg POMDPmethodology.
Rather, the platform and sellers each follow decentralized learning, and the platform3 See Appendix H for a discussion around parameter selection.
This class of policies already includes the optimal policy described in the proof of Proposition 2.
receives a consumer surplus reward at every step.
Q-learning is restarted after the same number
steps that are used in a Stackelberg POMDP episode.
• State-based RL:
Here we use the Stackelberg POMDP methodology to train a platform policy thatsets a threshold at which to admit each seller as a function of the price proﬁle quoted by the sellers(thus, "state-based").
This is the full class of threshold platform rules.
Here, Q-learning is restartedwhenever a Stackelberg POMDP episodes begins.•
No Stackelberg State-based RL: A variation on "State-based RL" that does not use the StackelbergPOMDP methodology.
Rather, the platform and sellers each follow decentralized learning, and theplatform receives a consumer surplus reward at every step.
Q-learning is restarted after the samenumber of steps that are used in a Stackelberg POMDP episode.
Figure 1 shows the consumer surplus that is realized under these different interventions.
First, weconﬁrm the results of Johnson et al.
(2020), and see consumer surplus improvements from both
PDP and DPDP compared to No intervention , with DPDP outperforming PDP .
At the same time,the no Stackelberg baselines are not able to outperform DPDP, conﬁrming the beneﬁts of usinglearning methodologies that exploit the leader-follower structure of our game.
Indeed, our RLinterventions based on the Stackelberg framework dramatically improve consumer surplus, driving itto (approximately, in the state-based scenario)
its maximal level.
In our setting, this optimal level forsurplus is approximately 0.94.
This is conﬁrmed by the fact that, for both No State and State-based
RL, all sellers are displayed and they invariably quote minimum prices at the end of training.
This isthe optimal (i.e., surplus maximizing) seller behavior, conﬁrming the effectiveness of the Stackelbergbased learning methodology in ﬁnding an optimal leader strategy given the Q-learning behavior ofsellers.
It is easier for No State RL to reach the optimal performance since its class of policies ismuch smaller than the class considered by State-based RL .
However, as we will see in Section 5.3,the state-based policy is more ﬂexible and is robust to the case that the cost basis changes for sellerswhile No State RL is not.
5.2 Learning in the Wild
We now test the performance of the Stackelberg POMDP learning methodology when it has no accessto sellers’ private information during training.
This can potentially create learning instabilities giventhat actor-critic training such as A2C generally require that the environment accessed by their criticnetworks is Markovian (Grondman et al., 2012).
Despite this, we ﬁnd success in this test of "inthe wild" learning.
The results are displayed in Figure 5.2 and show, despite relaxing this Markovassumption, that the A2C algorithm is able to learn optimal policies for both policy classes (No
State and State-based).
For No State RL this comes along with lower variance.
For State-based
RL, the empirical performance is roughly unaffected.
We conjecture that the reason behind this good performance is related to the class of threshold platform policies.
Given a threshold policy, itis possible to predict the overall episode reward based only on the action taken by the policy (thethreshold) and ignoring the part of the state that is internal to the sellers (i.e., the Q-matrix andexploration rate).
Thus, we see empirically that the additional information that relates to sellers’learning can actually make the platform’s learning problem harder.
In Appendix G, we also demonstrate successful experimental results when we replace the use ofconsumer surplus (1) for reward with a reward that corresponds to the number of agents displayedand the sum of the negated prices offered by sellers.
This shows robustness to a possible knowledgegap in knowing the speciﬁc functional form of consumer surplus.
5.3 Robustness of Learned Platform Rules
As observed in our previous experiments, the Stackelberg-based RL algorithm is effective in learninginterventions that maximize consumer surplus for a given economic setting.
However, as they aretailored to the economic setting at hand, these interventions can perform poorly when facing settingsthat are different from those during training.
To learn more robust platform rules, we also trainwith a modiﬁed version of the Stackelberg POMDP: at each reward step, with some random-priceprobability , sellers quote prices sampled uniformly at random from the price grid.
In this way,the platform is rewarded during training for performance that remains robust to prices that are notproduced by the Q-learning equilibrium dynamics (given seller costs at training).
We evaluate the effect of adding this perturbation-based robustness to the training procedure insettings with different seller costs: in addition to the default c = 1 . 0 , we also test with cost c = 1 . 38(between the second and the third price in the grid of prices between 0.95 and 2.1) and cost c = 1 . 67(between
the third and the fourth price in the price grid).
Here, and for additional realism, wealso continue to train the platform rule according to the "in the wild" approach described above, in
Section 5.2.
As we see in Figure 5.3 (right), this training approach (and in particular with probability 0.4 ofrandom-price perturbation) succeeds in making the state-based policy much more robust in the faceof sellers who experience a different cost environment at test time.
The robust, state-based policydisplays sellers with higher prices (due to their higher costs), while continuing to signiﬁcantly mitigatecollusion when seller costs are as they were during training.
This is also conﬁrmed by the policyvisualizations in Figure 5.3 (left), which show how the buy box learned for State-based RL tendsto be much more open under this modiﬁed training regime.
In contrast, the policy learned by No
State RL performs very poorly (zero consumer surplus!) when tested at costs that differ from thoseassumed during training, and even under this modiﬁed training regime.
There is no single thresholdthat provides a good compromise between performance at cost 1 and handling price perturbations.
Conclusion
This work has demonstrated that rules that are effective in preventing collusion by sellers can belearned through a framework that correctly solves the two-level, Stackelberg problem (making use ofthe platform’s commitment power).
Speciﬁcally, we have introduced the class of threshold policiesthat contain policies that optimize consumer surplus and a learning methodology that is effective inlearning optimal leader policies in this class.
The interventions we learned are shown to substantiallyoutperform the hand-designed interventions introduced in prior work when the cost environment attest time is as anticipated during training.
We also showed how our learned platform interventionscan be made more robust when settings are dynamic, with varying seller cost structures, by adoptinga suitably-modiﬁed training methodology.
This also highlights the importance of the state-basedplatform rule relative to a no-state rule.
Interesting future directions include testing our approach in more complex settings, e.g., when sellers’costs vary between training episodes.
In this case, optimal policy actions are based on the pricesquoted during the sellers’ equilibration phase, as these prices may provide useful information aboutthe current underlying costs (intuitively, the quoted prices will be higher under higher costs).
Inthis scenario, it may be necessary to represent our platform policies via recurrent neural networks,keeping a memory of past prices.
Finally, we believe that this approach can also be effective in otherapplications, e.g., to design and understand effective interventions for the electricity markets studiedby Abada and Lambin (2020), a setting where the successful use of RL as a defensive response by aplatform is not yet established.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.1.
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The
paper itself is focused on tackling a potential negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Proofs will beprovided in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will include
it in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] But not in Figure 3 to reduce clutter.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the beginning of Section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We used Stable
Baselines3 , which we cite at the beginning of Section 5.
(b) Did you mention the license of the assets?
[Yes] We mentioned the Stable Baselines
at the beginning of Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]