Learning to Mitigate AI Collusion on Economic Platforms
Abstract
Algorithmic pricing on online e-commerce platforms raises the concern of tacit collusion, where reinforcement learning algorithms learn to set collusive prices in a decentralized manner and through nothing more than proﬁt feedback.
This raises the question as to whether collusive pricing can be prevented through the design of suitable "buy boxes," i.e., through the design of the rules that govern the elements of e-commerce sites that promote particular products and prices to consumers.
In this paper, we demonstrate that reinforcement learning (RL) can also be used by platforms to learn buy box rules that are effective in preventing collusion by RL sellers.
For this, we adopt the methodology of Stackelberg POMDPs , and demonstrate success in learning robust rules that continue to provide high consumer welfare together with sellers employing different behavior models or having out-ofdistribution costs for goods.
Introduction
The last decade has witnessed a dramatic shift of trading from retailers to online e-commerce platforms such as Amazon and Alibaba.
In these platforms, sellers are increasingly using algorithms to set prices.
Algorithmic pricing can be beneﬁcial for market efﬁciency, enabling sellers to quickly react to market changes and also in enabling price competition.
At the same time, the U S. Federal Trade Commission (FTC)
U.S. Federal Trade Commission (2018) and European Commission (The Organisation for
Economic Co-operation
and Development, 2017) have raised concerns that algorithmic pricing may facilitate collusive behaviors.
Calvano et al.
(2020a) support these concerns through a study of pricing agents in a simulated platform economy, and show that commonly used reinforcement-learning (RL) algorithms learn to initiate and sustain collusive behaviors.
1 Assad et al.
(2020) also provide empirical support for algorithmic collusion in a study of Germany’s retail gas stations, showing an association between algorithmic pricing and an increase in price markups.
As highlighted by Calvano et al. (2020b), these kinds of collusive behaviors are unlikely to be a violation of antitrust law, as they are learned responses to proﬁt signals and not the result of explicit agreements.
One can try to prevent algorithmic collusion by introducing suitable rules by which platforms can choose which sellers to promote to buyers, thus promoting competition.
Could Amazon’s "buy box algorithm," for example, play this role in the future, in determining for a given consumer search which products and prices to highlight to a consumer?
Responding to this, Johnson et al. (2021) design hand-crafted rules that succeed in hindering collusion between RL algorithms.
At the same time, their rules introduce the undesirable effect of limiting consumers to a single seller, and there remains potential for more effective interventions.
In this paper, we demonstrate for the ﬁrst time how RL can also be used defensively by a platform to automatically design rules that promote consumer welfare and prevent collusive pricing.
This is a problem of multi-agent learning, with the interaction between the platform and sellers modeled as a Stackelberg game (Fudenberg and Tirole, 1991).
The leader is the platform designer and sets the platform rules and the sellers respond, using RL to set prices given these rules.
We introduce the class of threshold platform rules , and formally show that this class contains rules that approximately maximize consumer surplus in a unique subgame perfect equilibrium (Mas-Colell et al., 1995, chapter 9).
At the same time, this class of threshold rules is fragile to unexpected deviations by sellers, for example caused by cost perturbations.
The role of RL on the part of the platform is to learn rules with similar performance that are also more robust.
To solve the Stackelberg problem, we make use of the Stackelberg partially observable Markov decision process (POMDP) framework (Brero et al., 2022), which deﬁnes an episode structure of a POMDP such that the RL algorithm representing the leader will learn to optimize its reward (here, consumer surplus) given that its rules cause re-equilibration on the part of the followers (here, the sellers who use Q-learning algorithms to set prices).
The Stackelberg POMDP framework is well formed as long as the re-equilibration behavior of the sellers can be modeled through Markovian dynamics, as is the case with Q-learning.
We show successful results in learning effective platform policies that outperform handcrafted rules (Johnson et al., 2021).
This demonstrates how the Stackelberg POMDP framework can be successfully applied in settings where followers play repeated games, and their strategies are also policies trained via reinforcement learning algorithms.
We then show how our threshold platform rules allow us to obtain a similar learning performance when training the platform policy "in the wild," i.e., without accessing the sellers’ private information.
With this, we demonstrate how the
Stackelberg POMDP framework can be applied in more general learning scenarios than the ofﬂine learning ones for which it was originally designed.
Finally, we show how the platform rules learned via our Stackelberg POMDP framework continue to be effective when market conditions change, for example as the result of a change to the cost structure of sellers.
Further related work.
Zheng et al.
; Tang (2017);
Shen et al.
(2020); Brero et al.
(2021) make use of RL to optimize different economic systems (including matching markets, internet advertising, tax policies, and auctions) under strategic agents’ responses.
Unlike our work, these methods do not leverage the designer’s commitment power or the Stackelberg structure of the induced game.
Brero et al.
(2022) introduce and study the Stackelberg POMDP framework for matrix games and simple auctions using regret minimization for followers.
2 Abada and Lambin (2022) study collusion by RL pricing in markets for electric power, and use machine learning by a regulator agent for the mitigation of collusion, albeit without a Stackelberg framing (and without success, leading to lower welfare than the collusive outcome).
The broader research program on differentiable economics uses representation learning for optimal economic design (Duetting et al., 2019;
Shen et al., 2019; Kuo et al., 2020;
Tacchetti et al., 2019; Rahme et al., 2021a; Curry et al., 2022; Rahme et al., 2021b; Curry et al., 2020; Peri et al., 2021); this work avoids the need for Stackelberg design by emphasizing the use of direct, incentive-compatible mechanisms.
Also related is empirical mechanism design (Areyan Viqueira et al., 2019; Vorobeychik et al., 2006; Brinkman and Wellman, 2017), which applies empirical game theory to search for the equilibria of mechanisms with a set of candidate strategies (Wellman, 2006; Kiekintveld and Wellman, 2008; Jordan et al., 2010); see also Bünz et al.
(2018) for the design of iterative auctions.
Preliminaries
Seller Competition Model.
There is a set of sellers N = { 1 , . . . , n } , each of whom sells a differentiated product on an economic platform.
Each seller has the same (private) marginal cost c > 0 for producing one unit of its product.
Sellers interact with each other repeatedly over time in setting prices and selling goods.
At each time period, t = 0 , 1 , . . .
, each seller i observes all past prices, and sets a price p i,t ≥ 0 .
We let p t
=
( p 1 ,t , . . .
, p n,t ) denote a generic price proﬁle quoted at time t .
The platform sets the rules of a buy box that governs, in each period t , which set N t ⊆ N of sellers are available.
Consumers can only buy from these sellers and others forfeit sales.
There is also an outside option, indexed by 0 , which provides each consumer with a fallback choice with zero utility.
Following Johnson et al.
(2021), competition between sellers for consumer demand is modeled through the standard logit model of consumer choice.
For this, seller i has quality index α i > 0 , this providing horizontal differentiation across products, and the outside good has quality index α 0 > 0 .
In the logit model, each consumer samples ζ 0 , ζ 1 , ...ζ n , independently from a type I extreme value distribution with scale parameter µ > 0 , for each product and the outside option, with utility α i + ζ i − p i,t for product i , and {+α 
+} + ζ 0 for the outside option.
Considering a unit mass of consumers in period t , seller i ∈ N t receives fractional demand D i ( p t ; N t ) =
exp(( α i − p i,t ) /µ ) /λ
( p t ; N t ) , where λ ( p t ; N t ) =
(cid:80) j ∈N t exp(( α j − p j,t ) /µ )
+exp( α 0 /µ ) , and any seller i / ∈ N t has zero demand.
Scale parameter µ > 0 serves to control the extent of horizontal differentiation, with no differentiation and perfect substitutes obtained as µ → 0 .
The total consumer surplus is U ( p t ; N t ) = µ · log
Reinforcement learning by sellers.
In a single-agent Markov decision process (MDP), an agent faces a sequential decision problem under uncertainty.
At each step t , the agent observes a state variable s t ∈ S and chooses an action a t ∈ A .
Upon action a t in state s t , the agent obtains reward r ( s t , a t ) , and the environment moves to state s t +
according to q ( s t +1 | s t , a t ) .
We let τ =
( s 0 , a 0 , ..., s T , a T ) denote a state-action trajectory determined by executing policy policy π : S → A , and q π ( τ ) denote the probability of trajectory τ .
The optimal policy π ∗ solves π ∗ ∈ argmax π E τ ∼ q π ( τ ) [ (cid:80) Tt =
δ t r ( s t , a t )] , where δ ∈ [0 , 1] is the discount factor and time-horizon
T can be ﬁnite or inﬁnite.
In a partially-observable MDP (POMDP) , the policy π cannot access state s t but only observation o t sampled from q ( o t | s t ) .
A multi-agent MDP (Boutilier, 1996) for n agents has states S common to all agents and a set of actions
A i for each agent i .
When each agent i picks action a i,t in state s t , the environment moves to state s t +
according to a distribution q ( s t +1 | s t , a 1 ,t , .., a n,t ) and agent i obtains a reward r i ( s t , a t ) that depends on the joint action.
We follow Calvano et al.
(2020a)
and Johnson et al.
and adopt decentralized Q-learning by sellers as a positive theory for sellers in regard to their behavior in setting prices on an e-commerce platform (see Appendix A).
Although Q-learners may not converge, we also conﬁrm these earlier studies in showing convergence in our simulations (deﬁned over a particular time horizon as detailed by Johnson et al.
The Platform Stackelberg Problem
To formalize the problem facing the platform designer in mitigating collusive behavior by sellers, we model the interaction between the platform, which sets the rules of the buy box, and the sellers as a
Stackelberg game .
The platform designer is the leader, and ﬁxes the platform rules.
The sellers are the followers, and play an inﬁnitely repeated game according to these rules.
As discussed above, and following Calvano et al.
(2020a)
and Johnson et al.
(2021), we model the sellers’ behavior through decentralized Q-learning.
As a result, the problem facing the platform is a behavioral Stackelberg problem , in that the followers are modeled as Q-learners (and need not, necessarily, be playing an equilibrium of the induced game).
The sellers.
In this model, we ﬁx the states that comprise the MDP of a seller to include the prices set by all sellers in the last period, i.e., s t = p t − 1 .
We initialize s 0 to be a randomly selected price proﬁle.
The action of a seller is modeled as one of m equally-spaced points in the interval ranging from just below the sellers’ cost c to just above the monopoly price p m .
At each step t ≥ 0 , each seller i selects a price p i,t and is rewarded by its per-period proﬁt ρ i ( p t ; N t ) , which depends on p t = ( p 1 ,t , . . .
, p n,t ) and the choice of which sellers N t are displayed by the platform.
The platform.
To formalize the platform’s problem, let σ ∗ =
( σ ∗ 1 , .., σ ∗ n ) denote a strategy proﬁle selected by Q-learning on the part of sellers, in response to the platform rule, and in the long run, after a suitably large number of steps.
We leave implicit here the dependence of seller strategy proﬁle on the platform’s policy.
The platform must decide in each period which sellers to display to consumers.
For this, we denote the platform rule as policy π , and we adopt for the state of the platform policy the prices quoted by sellers in step t , p t , so that the platform’s policy uses these prices to select a set of agents to display, with N t selected according to π ( p t ) .
Let p ∗ t = σ ∗ ( s t ) denote a price proﬁle chosen under seller strategies σ ∗ , i.e., in response to the platform rules, and at some large enough time step t ∗ , and let τ ∗ =
( p ∗ t ∗ , p ∗ t ∗ +1 , .. ) denote a trajectory of prices forward from t ∗ .
We denote the distribution of these trajectories as q π ( τ ∗ ) .
As above, the dependence on the platform’s policy is left implicit in this notation.
The Stackelberg problem facing the platform is to ﬁnd a platform policy π that maximizes consumer surplus given the effect of this policy on the induced strategy proﬁle of sellers.
Deﬁnition 1 (Behavioral Stackelberg Problem)
The optimal platform policy solves π ∗ ∈ argmax π CS ( π ) , where CS ( π ) is the expected sum consumer surplus when sellers follow strategy σ ∗ forward from period t ∗ , i.e.,
[Equation]
t = t ∗ U ( p ∗ t ; π ( p ∗ t )) (cid:35) , (1) where T ∗ is suitably chosen horizon and q π ( τ ∗ ) denotes the distribution over Q-learning induced, seller pricing trajectories, in response to platform policy π .
Learning Optimal Platform Rules
In this section, we solve the platform’s problem, in responding to Q-learning sellers, through the
Stackelberg POMDP framework (Brero et al., 2022).
This creates a suitably deﬁned POMDP in which the optimal policy solves the behavioral Stackelberg problem (Deﬁnition 1).
Deﬁnition 2 (Stackelberg POMDP for platform rules)
The Stackelberg POMDP for platform rules is a ﬁnite-horizon POMDP, where each episode has the following two phases:
• An equilibrium phase , consisting of n e ≥ 1 steps.
In this phase, each state s t includes the step counter t , the sellers’ current Q-matrices, and the prices p t quoted by the agents.
Observations consists of the prices quoted by the sellers ( o t = p t ) and policy actions determine the set of agents displayed (in their more general version, a t = N t ).
State transitions are determined by Q-learning, where each seller i updates its Q-matrix after being rewarded by ρ i ( p t ; N t ) .
The policy has zero reward in every time step ( r ( s t , a t ) = 0 , for t ≤ n e ).
• A reward phase , consisting of n r ≥ 1 steps, each with the same actions, states, and observations as the equilibrium steps.
The reward phase differs in two ways.
First, the Q-matrices of the sellers are not updated, and second, the platform policy now receives a non-zero reward, and this is set in each step to be equal to the consumer surplus in that step ( r ( s t , a t ) =
U ( p t ; N t ) , for t > n e ).
This Stackelberg POMDP formulation is an adaptation of that provided by Brero et al.
(2022), who used it to learn leader strategies in matrix games and allocation mechanisms.
Following Brero et al.
(2022), we show the Stackelberg POMDP formulation is well-founded by showing that an optimal policy will also solve the Behavioral Stackelberg design problem of Deﬁnition 1.
Speciﬁcally, when the number of reward steps n r is large enough and when n e ≥ t ∗ , the optimal policy, denoted π ∗ n e ,n r , for the Stackelberg POMDP with n e equilibrium and n r reward steps maximizes the objective in
Equation (1).
Proposition 1 The optimal Stackelberg POMDP policy π ∗ n e ,n r , for an equilibrium phase with n e ≥steps and a reward phase with n r ≥ 1 steps, maximizes CS ( π ) , for seller behavior induced after n e steps when n r = T ∗ .
The proposition follows from the construction of the Stackelberg POMDP, especially the fact the our policy is only rewarded under the response behavior reached after n e steps, in line with the deﬁnition of CS ( π ) (see Appendix B for the full proof argument).
Brero et al.
(2022) use the Stackelberg POMDP framework in an "ofﬂine" environment, i.e., in a simulation that assumes access, at design time, to followers’ internal information.
This allows them to solve their POMDP using the paradigm of centralized training and decentralized execution (Lowe et al., 2017).
The leader policy is trained via an actor-critic deep RL algorithm, and the critic network (which estimates the sum of rewards until the end of the episode) accesses the full state during training.
Only the actor network, which represents the policy, is restricted to the partial-state information.
Here, we also study the use of the Stackelberg POMDP framework to train useful leader policies "in the wild," where the learning algorithm of the platform can only access the kind of information that an economic platform would have based on observations of sellers.
As we will empirically demonstrate, we can successfully relax the ofﬂine learning requirements—i.e., we operate without access to sellers’ private information in regard to Q-matrices and exploration rate—without affecting learning performance.
Threshold platform rules.
In our experiments, we consider the class of threshold platform rules .
These rules use the current prices set by sellers to set a price threshold above which a seller will not be displayed, with the same threshold set for all sellers.
Deﬁnition 3 (Threshold Platform Rule)
A threshold platform rule sets a threshold τ ( p t ) ≥ 0 , for each price proﬁle p t , such that N t
= { i ∈ { 1 , .., n } : p i,t ≤ τ ( p t ) } , i.e., any seller whose price is no greater than the threshold is displayed to consumers.
This class of threshold rules has a corresponding optimality result: there is a threshold rule that makes the market competitive, with all sellers displayed and consumer surplus maximized in the unique subgame perfect Nash equilibrium (SPE) of the induced continuous pricing game.
Even though the pricing behaviors that arise from Q-learning need not converge to SPEs, we ﬁnd empirical evidence, consistent with Johnson et al.
(2021), that the seller learning dynamics invariably converge to this equilibrium.
As such, this provides useful theoretical support for adopting the family of threshold platform rules by the platform learner.
We have the following result:
Proposition 2 For any (cid:15) > 0 , there exists a threshold platform rule π such that CS ( π ) ≥ CS ( π ∗ ) − (cid:80)
t δ t (cid:15) under the unique subgame perfect Nash equilibrium (SPE) of any ﬁnitely-repeated continuous pricing game induced by platform rule π .
This proposition follows from a platform rule with a limiting threshold that is arbitrarily close to the sellers’ cost c (see Appendix C for the proof).
Under this rule, sellers are displayed only if their price is minimal.
At the same time, this particular threshold rule is fragile, and would lead to market failure if seller costs vary in unexpected ways.
By letting the threshold τ also vary with the price proﬁle p t , as is allowed by the family of threshold platform rules, we seek to learn milder interventions that still mitigate collusion but remain robust to variations in the costs faced by sellers in the marketplace.
Experimental Results
In this section, we evaluate our learning approach via three main experiments.
We ﬁrst consider performance in terms of consumer surplus, benchmarking our RL interventions against the ones introduced by Johnson et al.
We demonstrate the ability to learn optimal leader strategies in the Stackelberg game with the followers across all the seeds we tested, signiﬁcantly outperforming existing interventions.
We then train platform rules without access to the sellers’ private information ("in the wild"), and show that this is not crucial for our learning performances.
We conclude by testing the robustness of our interventions, adding price perturbation during training and evaluating the effect on the robustness of our learned platform rules in environments where sellers have different costs from those assumed during training.
Experimental set-up.
As in Calvano et al.
(2020a)
and Johnson et al.
(2021), we consider settings with two pricing agents with cost c
= 1 , quality indexes α 1 = α 2 = 2 , and α 0 = 0 , and we set parameter µ = 0 .
25 to control horizontal differentiation.
The seller Q-learning algorithms are also trained using discount factor δ = 0 .
95 , exploration rate ε t = e − βt with β = 1 e − 5 , and learning rate α = 0 .
We also include results for variations of this default setting in Appendix D. A particular concern is that if the exploration rate is still high during the reward phase of the Stackelberg POMDP episode, our policy may be rewarded based on random prices. To address this problem, one could extend the Stackelberg POMDP equilibrium phase to reach a minimal ε t or isolate stages when price proﬁles are more stable to audit rewards. For simplicity (and to satisfy our computational constraints) we pause exploration during the reward phase of the Stackelberg POMDP. As in Johnson et al.
(2021), these prices range from 0.95 (just below the sellers’ cost) to 2.1 (which is above the monopoly price under no intervention).
Earlier work provided sellers with a choice of ﬁfteen different prices (over a similar range).
We need a smaller grid in order to satisfy our computational constraints; earlier work studied the effect of different, hand-designed platform rules, and did not also use RL for the automated design of suitable platform rules.
We also follow the choices of earlier work, and study an economy with two sellers (again, for reasons of computational resources).
This coarsened price grid allows us to train a platform policy through Stackelberg POMDP for 50 million steps in 18 hours using a single core on a Intel(R) Xeon(R)
Platinum 8268 CPU @
2.90GHz machine.
Learning algorithm.
To train the platform policy, we start from the A2C algorithm provided by
Stable Baselines
(Rafﬁn et al., 2021, MIT License).
Given that our policy is only rewarded at the end of a Stackelberg POMDP episode, we conﬁgure A2C
so that neural network parameters are only updated after this reward phase.
In this way, we guarantee that policies inducing desired followers’ equilibria are properly rewarded.
Furthermore, to reduce variance in sellers’ responses due to non-deterministic policy behavior, we maintain an observation-action map throughout each episode.
When a new observation is encountered during the episode, the policy chooses an action following the default training behavior and stores this new observation-action pair in the map.
We will show the importance of this variation via an ablation study that is presented in Appendix E.
Sellers restart the Q-learning process by re-initializing exploration rates every time the platform rules change (i.e., at the beginning of every Stackelberg POMDP episode).
In Appendix F, we also show how the training approach is robust to different sellers’ behavior models, where the sellers restart the learning rate asynchronously, and not necessarily at the beginning of episodes.
5.1 Platform Learning Performance
In this section, we evaluate the performance of our learned platform policies.
For this, we train our policies for 50 million steps in total.
We set up the Stackelberg POMDP environment using 50k equilibrium steps and 30 reward steps.
In these initial experiments, we train our policies using the centralized training-decentralized execution paradigm as used for this Stackelberg learning problem by Brero et al.
, giving the critic network access to the sellers’ learning information (i.e.,
Q-tables and exploration rates).
We relax this below in studying the robustness of the computational framework to online training ("in the wild").
We consider the following interventions on behalf of the platform designer:
• No intervention: Sellers are always displayed, no matter the price they quote.
To derive this baseline, we run our Q learning dynamics until convergence (as described in Johnson et al.
for each seed
and then average the surplus at ﬁnal strategies.
• PDP: We test price-directed prominence , a platform intervention introduced by Johnson et al.
Here, the platform only displays the seller who quotes the lower price (breaking ties at random), thus enhancing competition.
As for no intervention , we compute the performance of PDP by averaging consumer surplus after Q-learning dynamics converge.
• DPDP:
Dynamic price-directed prominence is another intervention introduced by Johnson et al. (2021), which also conditions the choice of the (unique) displayed seller on past prices.
Under this intervention, quoting prices equal to cost is a subgame perfect equilibrium of the induced game (under suitable discount factors).
As for the previous baselines, we compute the performance of DPDP by averaging consumer surplus after Q-learning dynamics converge.
• No State RL:
Here we use the Stackelberg POMDP methodology to train a platform policy does not use prices p t to determine the threshold at which to admit each seller to the buy box (thus, "no state").
Here, Q
is
begins. • No Stackelberg No State RL: A variation on "No State RL" that does not use the Stackelberg POMDP methodology.
Rather, the platform and sellers each follow decentralized learning, and the platform receives a consumer surplus reward at every step.
Q-learning is restarted after the same number of steps that are used in a Stackelberg POMDP episode.
• State-based RL:
Here we use the Stackelberg POMDP methodology to train a platform policy that sets a threshold at which to admit each seller as a function of the price proﬁle quoted by the sellers (thus, "state-based").
This is the full class of threshold platform rules.
Here, Q-learning is restarted whenever a Stackelberg POMDP episodes begins.
• No Stackelberg State-based RL: A variation on "State-based RL" that does not use the Stackelberg POMDP methodology.
Rather, the platform and sellers each follow decentralized learning, and the platform receives a consumer surplus reward at every step.
Q-learning is restarted after the same number of steps that are used in a Stackelberg POMDP episode.
Figure 1 shows the consumer surplus that is realized under these different interventions.
First, we conﬁrm the results of Johnson et al.
and see consumer surplus improvements from both
PDP and DPDP compared to No intervention , with DPDP outperforming PDP .
At the same time, the no Stackelberg baselines are not able to outperform DPDP, conﬁrming the beneﬁts of using learning methodologies that exploit the leader-follower structure of our game.
Indeed, our RL interventions based on the Stackelberg framework dramatically improve consumer surplus, driving it to (approximately, in the state-based scenario)
its maximal level.
In our setting, this optimal level for surplus is approximately 0.94.
This is conﬁrmed by the fact that, for both No State and State-based
RL, all sellers are displayed and they invariably quote minimum prices at the end of training.
This is the optimal (i.e., surplus maximizing) seller behavior, conﬁrming the effectiveness of the Stackelbergbased learning methodology in ﬁnding an optimal leader strategy given the Q-learning behavior of sellers.
It is easier for No State RL to reach the optimal performance since its class of policies is much smaller than the class considered by State-based RL .
However, as we will see in Section 5.3, the state-based policy is more ﬂexible and is robust to the case that the cost basis changes for sellers while No State RL is not.
5.2 Learning in the Wild
We now test the performance of the Stackelberg POMDP learning methodology when it has no access to sellers’ private information during training.
This can potentially create learning instabilities given that actor-critic training such as A2C generally require that the environment accessed by their critic networks is Markovian (Grondman et al., 2012).
Despite this, we ﬁnd success in this test of "in the wild" learning.
The results are displayed in Figure 2 and show, despite relaxing this Markov assumption, that the A2C algorithm is able to learn optimal policies for both policy classes (No
State and State-based).
For No State RL this comes along with lower variance.
For State-based
RL, the empirical performance is roughly unaffected.
We conjecture that the reason behind this good performance is related to the class of threshold platform policies.
Given a threshold policy, it is possible to predict the overall episode reward based only on the action taken by the policy (the threshold) and ignoring the part of the state that is internal to the sellers (i.e., the Q-matrix and exploration rate).
Thus, we see empirically that the additional information that relates to sellers’ learning can actually make the platform’s learning problem harder.
In Appendix G, we also demonstrate successful experimental results when we replace the use of consumer surplus (1) for reward with a reward that corresponds to the number of agents displayed and the sum of the negated prices offered by sellers.
This shows robustness to a possible knowledge gap in knowing the speciﬁc functional form of consumer surplus. Remark.
We note that, despite the positive results of this section, the high number of training steps required by our learning framework can be prohibitive for its online deployment.
One possible solution to this problem is to ﬁrst train the rules of the platform in an ofﬂine environment to calibrate them before deployment and further adaptation.
5.3 Robustness of Learned Platform Rules
As observed in our previous experiments, the Stackelberg-based RL algorithm is effective in learning interventions that maximize consumer surplus for a given economic setting.
However, as they are tailored to the economic setting at hand, these interventions can perform poorly when facing settings that are different from those during training.
To learn more robust platform rules, we also train with a modiﬁed version of the Stackelberg POMDP: at each reward step, with some random-price probability , sellers quote prices sampled uniformly at random from the price grid.
In this way, the platform is rewarded during training for performance that remains robust to prices that are not produced by the Q-learning equilibrium dynamics (given seller costs at training).
We evaluate the effect of adding this perturbation-based robustness to the training procedure in settings with different seller costs: in addition to the default c = 1 . 0 , we also test with cost c = 1 . (between second and
(between the third and the fourth price in the price grid).
, and for additional realism, we also continue to train the platform rule according to the "in the wild" approach described above, in
Section 5.2.
As we see in Figure 3 (right), this training approach (and in particular with probability 0.4 of randomprice perturbation) succeeds in making the state-based policy much more robust in the face of sellers who experience a different cost environment at test time.
The robust, state-based policy displays sellers with higher prices (due to their higher costs), while continuing to signiﬁcantly mitigate collusion when seller costs are as they were during training.
This is also conﬁrmed by the policy visualizations in Figure 3 (left), which show how the buy box learned for State-based RL tends to be much more open under this modiﬁed training regime.
In contrast, the policy learned by No State RL performs very poorly (zero consumer surplus!)
when tested at costs that differ from those assumed during training, and even under this modiﬁed training regime.
There is no single threshold that provides a good compromise between performance at cost 1 and handling price perturbations.
Conclusion
This work has demonstrated that rules that are effective in preventing collusion by sellers can be learned through a framework that correctly solves the two-level, Stackelberg problem (making use of the platform’s commitment power).
Speciﬁcally, we have introduced the class of threshold policies that contain policies that optimize consumer surplus and a learning methodology that is effective in learning optimal leader policies in this class.
The interventions we learned are shown to substantially outperform the hand-designed interventions introduced in prior work when the cost environment at test time is as anticipated during training.
We also showed how our learned platform interventions can be made more robust when settings are dynamic, with varying seller cost structures, by adopting a suitably-modiﬁed training methodology.
This also highlights the importance of the state-based platform rule relative to a no-state rule.
Interesting future directions include testing our approach in more complex settings, e.g., when sellers’ costs vary between training episodes.
In this case, optimal policy actions are based on the prices quoted during the sellers’ equilibration phase, as these prices may provide useful information about the current underlying costs (intuitively, the quoted prices will be higher under higher costs).
In this scenario, it may be necessary to represent our platform policies via recurrent neural networks, keeping a memory of past prices.
Finally, we believe that this approach can also be effective in other applications, e.g., to design and understand effective interventions for the electricity markets studied by Abada and Lambin (2022), a setting where the successful use of RL as a defensive response by a platform is not yet established. Acknowledgements This research is funded in part by Defense Advanced Research Projects Agency under Cooperative Agreement HR00111920029.
The content of the information does not necessarily reﬂect the position or the policy of the Government, and no ofﬁcial endorsement should be inferred. This is approved for public release; distribution is unlimited. The work of G. Brero was also supported by the SNSF (Swiss National Science Foundation) under Fellowship P2ZHP1_191253. We thank Emilio Calvano and Justin Johnson for their availability to answer questions about their work and for guidance in replicating some of their results.
We also thank Alon Eden, Matthias Gerstgrasser, and Alexander
MacKay for for helpful discussions and feedback.
Many computations in this paper were run on the
FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at
Harvard University.
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 5.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The paper itself is focused on tackling a potential negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Proofs are provided in the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will include it in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] But not in Figure 3 to reduce clutter.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the beginning of Section 5. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We used Stable Baselines3 , which we cite at the beginning of Section 5.
(b) Did you mention the license of the assets?
[Yes] We mentioned the Stable Baselines at the beginning of Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Q learning is an RL algorithm that learns an estimate of the action-value function Q ∗ ( s, a ) . This action-value function gives the expected total reward of taking action a at state s and using the optimal policy function in the future. Once Q ∗ ( s, a ) has been learned, the optimal policy is
[Equation] A Q-learning algorithm maintains an | S | × | A | Q-matrix, Q t , representing the estimate of Q ∗ ( s, a ) at step t . Usually, this matrix is randomly initialized. At each step t , the algorithm takes action a t that with probability 1 − ε t is optimal according to its current Q-matrix Q t , and with probability ε t chosen uniformly at random from the set of avalilable actions.
We call ε t the exploration rate .
The entry ( s t , a t ) of Q t is updated based on feedback via a convex combination of its previous value and the reward r ( s t , a t ) attained from the action plus the discounted value of the state s t +1 :
[Equation]
Parameter α ∈ [0 , 1] is the learning rate .
When the environment is stationary and Markovian, and under suitable assumptions on the learning rate and exploration rate, the Q-matrix is guaranteed to converge in the limit to the action-value function Q ∗ ( s, a ) and the policy to the optimal policy.
B Proof of Proposition
Proposition 1 The optimal Stackelberg POMDP policy π ∗ n e ,n r , for an equilibrium phase with n e ≥steps and a reward phase with n r ≥ 1 steps, maximizes CS ( π ) , for seller behavior induced after n e steps when n r = T ∗ .
Proof.
Let τ ∼ q π ( τ ) denote a generic trajectory determined by executing policy π in the Stackelberg
POMDP environment.
We have
[Equation]
recognizing r ( s t , a t )
{+= 
+}
if t ≤ n e .
After replacing r ( s t , a t ) with U ( p t ; π ( p t )) , we can rewrite the objective in (4) as
[Equation]
which is equal to (1) when n r = T ∗ .
C Proof of Proposition
Proposition 2 For any (cid:15) > 0 , there exists a threshold platform rule π such that CS ( π ) ≥ CS ( π ∗ ) − (cid:80)
t δ t (cid:15) under the unique subgame perfect Nash equilibrium (SPE) of any ﬁnitely-repeated continuous pricing game induced by platform rule π .
Proof.
For any η > 0 , we study the stage game with continuous prices induced by the threshold platform rule that sets threshold τ = c + η for each price proﬁle p .
We show that this stage game has a unique Nash equilibrium in which each seller sets a price p i
= p ∗ , for some p ∗ ∈ ( c, c + η ] .
Given this, we have that every seller pricing at p ∗ ∈ ( c, c + η ] in every period is the unique SPE of any ﬁnitely repeated game between sellers, since this is an open-loop Nash equilibrium proﬁle (and thus
SPE by the single-deviation principle).
Moreover (as it is a continuous function over price proﬁles) the consumer surplus comes arbitrarily close, for a small enough η > 0 , to the maximum consumer surplus, which coincides with every seller pricing at cost.
Left to prove is that every seller pricing at p ∗ is a Nash eq (NE) of the stage game.
First, pricing p i > τ ( p t ) =
c + η provides zero proﬁt to a seller because the seller is not part of the displayed set of sellers.
Similarly, pricing p i =
c provides zero proﬁt.
Now dropping the time period t , because we study a generic stage game, and considering prices p
= ( p 1 , . .
. , p n ) ∈
× ni =1 ( c, c + η ] , so that all sellers are displayed, and with seller proﬁt ρ i ( p ; N ) for price proﬁle p , we have
[Equation]
By ﬁrst-order optimality conditions, we have ∂∂p i ρ i ( p ; N ) = 0 , for each i , only when p i
= ˆ p , for some ˆ p > c (Anderson and De Palma, 1992).
Furthermore, ρ i ( p i , p − i ; N ) is concave.
Thus, if c + η ≥ ˆ p , there is only one Nash equilibrium of the stage game, where each seller i sets price p ∗ =
ˆ p ≤ c + η .
On the other hand, if c + η < ˆ p , we have that ρ
i ( p ; N ) is strictly increasing when p ∈ × ni
=1 ( c, c + η ] .
This means that a generic seller i can always increase its utility by quoting p i + ε instead of p
i .
This happens unless p i is equal to c + η : in this case, increasing p i would drive the agent out of the buybox (zeroing its utility).
Thus, there is a unique Nash equilibrium where each seller quotes price p ∗ =
c + η .
D Variations on Setting Parameters
Following Calvano et al.
(2020a)
and Johnson et al.
(2021), we also consider the learning performance of our RL algorithms in settings with different horizontal differentiation parameter µ .
Speciﬁcally,
Figure 4 displays our training curves when the horizontal differentiation is lowered from 0.25 to 0.
and increased from 0.25 to 0.40, and our policies are learned "in the wild.
" We notice that, in both scenarios, our RL policies bring consumer welfare close to its optimal level ( 0 . 8 when µ = 0 . 05 and 1 .
when µ = 0 . 4 ) outperforming all the baselines.
E Deterministic Policies During Stackelberg POMDP Episodes
In this section, we test the learning performance of our RL algorithms when policies are not deterministic during the Stackelberg POMDP episodes.
As we can see from Figure 5, this variation dramatically affects our learning performance, which only slightly improves during training and leads to ﬁnal policies that do not outperform our baselines.
As discussed in Section 5, this poor performance is caused by the sellers not being able to learn optimal response strategies due to the high variance introduced by the non-deterministic behavior of our policies.
F Variations on Sellers’ Learning Behavior
In our previous experiments, we assumed that both sellers restart their learning processes any time the platform rules change.
This is consistent with the original experiments run by Calvano et al.
(2020a), which demonstrated seller collusive behavior.
However, this assumption may not hold in real-world settings, where sellers can restart their learning processes asynchronously and at any time.
This behavior can present new challenges to learning an effective platform policy.
Indeed, in this scenario, changes in the sellers’ behavior may be caused not only by different platform interventions, but also as a result of learning restarts.
In this section, we evaluate the performance of the Stackelberg POMDP framework in scenarios where sellers randomly restart their exploration rate during training.
Speciﬁcally, we assume that, at each step of the platform’s learning process, each seller restarts its exploration rate with some probability.
We set this probability such that, in expectation, each seller restarts its exploration once per Stackelberg POMDP episode (which corresponds to the number of steps between updates).
Note that if restarts occur close to the reward phases, rewards may reﬂect an out-ofequilibrium behavior of the sellers (even if exploration is paused).
To avoid this problem, we generate our plots by logging rewards in an evaluation Stackelberg POMDP episode we run every 100k training steps.
These evaluation episodes use the current platform policy and operate it executing the action with the highest weight given each observation.
In these episodes, the Q learning processes are run as in the previous sections, without intra-episode restarts.
As we can see from Figure 6, the Stackelberg learning framework allows us to derive close-to-optimal policies even under this less stationary behavior.
Given that rewards are collected in evaluation episodes (where policies are operated via highest-weighted actions), the optimal intervention under
No State RL is executed much earlier than in the simulations of Figure 2, reaching the maximum reward after only 15M training steps.
G Variations on Policy Rewards
In our main experiments, we assume that the platform can compute the consumer surplus U ( p t ; N t ) = µ · log[ λ ( p t ; N t )] , where λ ( p t ; N t ) = (cid:80) j ∈N t exp(( α j − p j,t ) /µ )
+ exp( α 0 /µ ) at each step t to reward the policy.
We note that, however, to compute this surplus one needs to access the consumers’ quality indexes α
i s, which may not be available to the platform.
However, we note that, to maximize consumer surplus, we can replace U ( p t ; N t ) with any reward function that increases with respect to the number of agents displayed and decreases as prices increase.
In this section, we will use ˜
U ( p t ; N t ) = µ · log[˜ λ ( p t ; N t )] , where ˜ λ ( p t ; N t ) =
(cid:80) j ∈N t exp( − p j,t /µ ) and show that we can achieve similar learning performance as shown in Figure 7 .
H Selecting Stackelberg POMDP Parameters
The hyperparameters of our Stackelberg POMDP are selected the following criteria: First, we want to make sure that the equilibrium phase of our Stackelberg POMDP is long enough such that sellers’ dynamics produce best responses to platform policies.
At the same time, we want to avoid too long equilibrium phases, as this makes rewards too sparse.
With 50k steps we have a good trade-off between these two desiderata.
Regarding the reward phase, we want to make sure that this reward is representative of the converged policy reached by Q-learners, and their converged behavior is usually a single price proﬁle or a loop of two or three price proﬁles.
We adopt 30 reward steps to be conservative.