T HE G OOD S HEPHERD : A N O RACLE A GENT FOR M ECHANISM D ESIGN
A BSTRACT
From social networks to trafﬁc routing, artiﬁcial learning agents are playing a central role in modern institutions.
We must therefore understand how to leverage these systems to foster outcomes and behaviors that align with our own values and aspirations.
While multiagent learning has received considerable attention in recent years, artiﬁcial agents have been primarily evaluated when interacting with ﬁxed, non-learning co-players.
While this evaluation scheme has merit, it fails to capture the dynamics faced by institutions that must deal with adaptive and continually learning constituents.
Here we address this limitation, and construct agents (“mechanisms”) that perform well when evaluated over the learning trajectory of their adaptive co-players (“participants”).
The algorithm we propose consists of two nested learning loops: an inner loop where participants learn to best respond to ﬁxed mechanisms; and an outer loop where the mechanism agent updates its policy based on experience.
We report the performance of our mechanism agents when paired with both artiﬁcial learning agents and humans as co-players.
Our results show that our mechanisms are able to shepherd the participants strategies towards favorable outcomes, indicating a path for modern institutions to effectively and automatically inﬂuence the strategies and behaviors of their constituents.
I NTRODUCTION
Modern institutions often serve two distinct and equally important roles in our society.
First they mediate and foster economic or social interactions among citizens (e.g. taxation policies ensure governments receive enough funds to build roads and schools).
Second, they foster behaviors that bring us closer to our aspirations as a society (e.g. charitable donations are tax-free).
As artiﬁcial learning agents mediate more and more interactions among humans, ﬁrms, and organization, it is paramount that we study how to construct adaptive systems that can fulﬁl both roles.
However, while multiagent learning has received considerable attention in recent years, the standard evaluation scheme pairs our artiﬁcial agents with other ﬁxed, and potentially adversarial co-players (e.g. exploitability)
(Vinyals et al., 2019; Muller et al., 2019; Goodfellow et al., 2014).
While this evaluation scheme has merit, it fails to capture the dynamics faced by modern institutions that are often paired with learning constituents, and where agents must take into account not only what other agents will do next, but also, in the long run, how they will adapt to the current strategies present in the system.
Here we address this shortcoming and construct low-exploitability agents that do well when paired with learning co-players, in the general-sum setting.
We construct players that, through their behavior, are able to inﬂuence what others will learn to do, and explicitly leverage the link between one agent’s actions and another agent learning trajectory.
In other words, we construct agents (“mechanisms”) that learn to act so as to shepherd participants’ strategies both at equilibrium, and during learning.
Our proposed method takes the form of an inner-outer loop learning process.
In the inner loop, participant agents respond to a ﬁxed mechanism strategy, while in the outer-loop our mechanism agent adapts its policy based on experience.
Unlike previous work, our mechanisms make very few assumptions on the preferred strategies, outcomes, or learning capabilities of the participants, and only have access to the consequences of their own behavior on the learning of others.
M e c h a n i s m
R e t u r n
Assurance
Mechanism
IO-Loop
Pro-Social
Random
Selfish
Tit-for-Tat
Win-Stay-Lose-Shift
LoLa
Battle of the Sexes
Chicken
Coordination
Deadlock
Harmony {+
+}
{+Hero 
+}
Mixed Harmony {+
+}
No Conflict {+
+}
Prisoner Delight {+
+}
Prisoner Dilemma {+
+}
[Figure]
Stag Hunt
We investigate the performance of our mechanism agents with both artiﬁcial and human co-players in simple 2-player 2-strategy repeated games, and in a stylized resource allocation problem.
We study how our method can be adapted when mechanisms are granted access to the inner workings of participants, and when that is not the case.
Our results show that our mechanism agents successfully shepherd the learning of others towards desirable outcomes, and that the direction presented here is promising for agent-agent interactions, and withstand a transfer to the agent-human interaction setting.
In the broader context of AI in modern day institutions, our methods and ideas show that adaptive agents can successfully shepherd the learning of their co-players towards desirable outcomes and behavior, opening the door to learning-based institutions that ﬁne-tune the incentives faced by their constituents in pursuit of group level goals.
R ELATED WORK
The inner-outer loop method that we propose here provides insights into two challenges for multiagent reinforcement learning.
The ﬁrst challenge is the non-stationarity of the environment.
When training multiple policies simultaneously, the environment is non-stationary from the point of view of any agent due to the change in the other players’ policies.
Common approaches to mitigate this non-stationarity involve building populations of agents (Brown, 1951; Muller et al., 2019; Vinyals et al., 2019) or exploiting knowledge of the learning dynamics of others (Balduzzi et al., 2018; Hemmat et al., 2020).
Both these approaches have often focused on competitive zero-sum game.
Here we focus on setting where the environment is always stationary from the point of view of all agents since we don’t update policies concurrently in the same training loop.
Second is the challenge of equilibrium selection .
Generally, and particularly in non-zero-sum games, multiple Nash Equilibria may exist.
This leads to the problem of both ﬁnding and selecting among possibly unequal equilibria, with the goal of a) biasing learning towards outcomes preferred by one agent (shaping), or b) generalizing with unseen co-players.
Progress towards this has been
[Figure]
M e c h a n i s m
R e t u r n
Assurance
Mechanism IO-Loop
Pro-Social
Random
Selfish
Tit-for-Tat
Win-Stay-Lose-Shift
LoLa
Battle of the Sexes
Chicken
Coordination
Deadlock
Harmony
[Figure]
Prisoner Dilemma
[Figure]
made in recent years through centralized learning with decentralized execution , a framework for multiagent RL where agents can exploit privileged knowledge about other agents’ during training, but not at time of deployment.
Centralized learning can be useful for equilibrium selection: for example, access to a centralized value function provides a recipe to construct agents that are able to coordinate at execution in cooperative settings (Sunehag et al., 2017); coupled training of multiple agents can improve learning of communication protocols (Foerster et al., 2016; 2019); learning from interactions with agents at different stages of training can improve generalization at evaluation with human participants (Strouse et al., 2022); and exploiting information about how other agents update their behaviour can be used to shape them, both within an episode (Lerer & Peysakhovich, 2018;
Peysakhovich & Lerer, 2018) and across training (Foerster et al., 2017; Yang et al., 2020).
In our method, we do not separate training from execution.
We make no assumptions about the learning rule of the co-players, or how they will adapt to the strategies currently present in the system.
Instead, we infer the relationship between the mechanism’s actions and the participant’s learning directly from the observed interactions.
M ETHODS
We consider the problem of constructing agents (“mechanisms”) that shepherd the learning of others (“participants”).
We use repeated symmetric two-player, two-strategies games as our initial test-bed as they are easy to analyze and train on.
We then move on to a simple resource allocation game.
We start by assuming that the mechanism agent has access to the inner workings of participant agents in Differentiable mechanism design, and then extend our methods to remove this assumption using
Evolutionary Strategies (Salimans et al., 2017).
3. E NVIRONMENTS
We tackle iterated 2-player, 2-strategies symmetric matrix games, and a simple resource allocation game with one mechanism agent and four participant agents.
3.1. I TERATED MATRIX GAMES
[Figure]
We consider the 12 symmetric 2-player, 2-strategies matrix games identiﬁed in (Wikipedia, 2022;
Robinson & Goforth, 2005), with payouts scaled down by 4 (our payouts are between -3 and 0), and consider iterated interactions between a mechanism (row) player and a participant (column) player with a single memory step.
Our naming convention is most easily followed when focusing on the
Prisoner Dilemma game.
For each game we deﬁne a Markov Decision Process with statespace S =
( s 0 , CC, CD, DC, DD ) , with s 0 being the initial state, and the rest being the state after the joint actions in the previous state (e.g. cooperate, cooperate; cooperate, defect and so on).
Our agents’ one-memory policies can be represented by a 5-tuple θ corresponding to the probability of cooperating on each state.
In these simple repeated games, the transition kernel T takes the form of a matrix whose entries describe the probability of the next state as a function of the previous state, and can be derived analytically given the one-memory policy parameters from both players.
The reward functions are speciﬁed on arrival at each state as r m = (0 , r R , r S , r T , r P ) and r p
= (0 , r R , r T , r S , r P ) , where r P , r R , r S , r T correspond to the punishment, reward, sucker and temptation payoffs respectively.
The returns R m and R p that both the mechanism and participant aim to maximize corresponds to the state value for the initial state s 0 .
3.1. R ESOURCE A LLOCATION GAME
We further consider a modiﬁcation of the classic Public Goods Game (as described in (Koster et al., 2022)).
The game consists of a single interaction between four participants i = 1 , 2 , 3 ,and a single mechanism agent.
Each participant receives an endowment e i and allocates a fraction
ρ i of it to a common investment pool, which is then grown by a ﬁxed constant factor ( 1 . 6 ) and redistributed to participants in full.
The speciﬁc amount received by each participant i is denoted as p i and is determined by the mechanism.
Participants seek to maximize their individual welfare (i.e. R p i
= p i
+ (1 − ρ i ) e i ), while the mechanism seeks to maximize total participants’ welfare (i.e. R m = 1 N (cid:80) Ni R
tp i , with N the number of participants).
When interacting with naive or poorly designed redistribution mechanisms, the incentive to free-ride may tempt each player away from the contributing to the common pool, which in turn decreases total welfare.
3. T HE I NNER OUTER LOOP A LGORITHM
Our learning process takes the form of an inner-outer loop that exposes the mechanism to the consequences of its actions on the learning of others.
In the inner loop, participant agents repeatedly interact with a ﬁxed mechanism, and use independent gradient ascent to improve their own policies.
In the outer loop, mechanism agents update their strategies based on the experience they acquired in the inner loop.
Require:
MDP , T m , T p , R , θ 0 m , γ m , γ p for t m in 0 :: T m do θ
p ∼ R ; ¯ R m ←for t p in 0 :: T p do ( R m , R p ) ← MDP ( θ t m m , θ t p p ) ¯
R m ← ¯ R m + R m θ t p +1 p ← θ t p p + γ p ∇ θ tpp R p end for θ t m +1 m ← m m + γ m ∇ θ tmm ¯ R m end for
3.2. D IFFERENTIABLE MECHANISM DESIGN
We ﬁrst consider the case where the mechanism agents can directly compute the gradients of its return ¯
R m with respect to its policy parameters θ m .
Inspecting Algorithm
we note that, at any given θ t m m update, the mechanism return R m depends on the mechanism parameters θ t m m , as well as on the entire trajectory of participants parameters over the inner loop θ p 0 , . .
, θ T p p .
In Differentiable Mechanism design (Diff-MD), we let the mechanism update have gradient access to the entire trajectory, as well as the environment transition kernel T .
In practice, this can easily be implemented using a tensor library with auto-differentiation (such as JAX (Bradbury et al., 2018)).
3.2. E VOLUTIONARY STRATEGIES FOR NON - DIFFERENTIABLE MECHANISM DESIGN
When the mechanism agent cannot take derivatives through the environment, we used evolutionary strategies (ES).
In this case, the inner loop is repeated N p times so as to form an experience “batch” with mechanism parameters slightly perturbed at the beginning of each inner loop ( t p = 0 ) as θ pm = θ pm + (cid:15) p , where (cid:15)
p ∼ N (0 , σ 2 m ) with σ m a hyper-parameter
( 1 in our experiments).
Given an experience batch the mechanism policy gradient, estimated as ∇ θ m ≈ (cid:80) N p p =1 (cid:15) p ¯ R m N p σ m , moves the mechanism parameters in the direction of those used in episodes that led to positive outcomes (see (Salimans et al., 2017) for details).
We refer to this method as ES-MD.
3. L EARNING WITH O PPONENT -L EARNING A WARENESS (LOLA)
We implemented LOLA (Foerster et al., 2017) as a baseline in our experiments.
In LOLA, the mechanism agent projects the learning of the participants forward in time.
In contrast to the original paper, in which both agents are assumed to be using LOLA, here only let the mechanism agent be learning-aware.
R ESULTS
Here we show how a trained mechanism performs when paired with learning participants.
We report the return collected by a (ﬁxed) mechanism in each episode over the learning trajectory of its co-players.
Figures 1 and 3 show how mechanisms trained with Diff-MD perform in thematrix games and resource allocation game respectively, while Figure 2 shows the performance of a mechanism trained with ES-MD in the 12 matrix games we consider.
4. M ATRIX GAMES
In the matrix games we compare our mechanism (labelled as IO-Loop in the legends) to well known one memory strategies (e.g. Tit-for-Tat) and pure strategies (e.g. Selﬁsh).
Additionally, in DiffMD we also compare a mechanism trained with LOLA.
In all games, and for both Diff-MD and
ES-MD, our IO-loop mechanism achieves the same performance as the best alternative available strategy, with the exception of Prisoner Dilemma where it signiﬁcantly outperforms it.
We used the following hyper-parameters for both experiments: T m = 10000 , T p = 50 , γ m = 0 .
1 , γ p = 10 , θ 0 m =
5] , in the ES-MD experiment we further set N p = 256 and σ m = 1 .
R ESOURCE ALLOCATION GAME
In the resource allocation game we report mechanism performance when paired both with artiﬁcial learning agents and human co-players.
In particular, in Fig.
3, we consider unequal endowments with e i ∈ [0 . 2 , 1 . 0] , and compare our mechanism with four alternative redistribution strategies: Absolute
Proportional and Relative Proportional redistribute funds proportionally to the absolute contribution
ρ i e i or to the fraction of endowment ρ i contributed by each participants, while the Uniform and
Random mechanisms redistributed the funds equally and randomly respectively.
Figure 3 shows that
Diff-MD ﬁnds a mechanism policy that shepherds the participants towards higher welfare outcomes.
In this experiment we represented the participants policy as their propensity to contribute to the public fund: θ p i =
ρ i , and the mechanism policy as a MLP with a single 32-units hidden layer.
We further set T m = 5000 , T p = 10 , γ m = 0 .
01 , γ p = 0 .
1 and θ 0 m the default MLP initialization.
4. E VALUATION WITH HUMAN CO - PLAYERS
Figure 3 (right) shows the performance of our mechanism, and alternative baselines, when paired with human co-players in the stylized resource allocation game outlined above (endowment condition for the 4 players: [1 . 0 , 0 . 5 , 0 . 4 , 0 . 3] ).
We used crowd-sourcing platforms to collect data, and all participants gave informed consent to participate in the experiment.
Participants were organized in groups of 4, and after a tutorial phase, they played the resource allocation game outlined above completing the 10 steps constituting our “inner loop”.
The tutorial round explained the mechanics of the game, instructed participants on how to use the web interface, and outlined how participants would be rewarded in real money: participants received a base compensation for completing the experiment, and a bonus proportional to their aggregate return over the course of the experiment.
After one game with the Uniform mechanism, each group of participants interacted with two mechanisms, either resulting from our training, or with one of the baseline mechanisms outlined above in counterbalanced order.
We collected data from 236 non overlapping groups.
If a participant dropped out during the experiment, their actions were replaced with random actions, which were subsequently removed in the analysis ( 39% of responses were removed this way).
The results presented in the right hand panel in Figure 3 show that our mechanism could withstand a basic transfer to interacting with human co-players, and that its performance remained consistent with what we observed in simulation.
C ONCLUSION
We have shown here that our inner-outer loop algorithm can provide an oracle-style benchmark to test agents’ ability to shepherd the behavior of learning co-players to desired outcomes, and that in simple environments, our agents transfer to human co-players.
As more and more of the systems we use and deploy become adaptive, it becomes increasingly important to 1) construct agents that can plan and act taking into account the fact that others are learning around them, and 2) construct agents that can shape the incentives faced by co-players in pursuit of group-wide objectives.
The ideas and results presented here show that exposing agents to the consequences of their actions on the learning of others is a sensible ﬁrst step toward these goals.
Moreover, the transfer to human co-players we were able to showcase suggests that our method contains the basic elements required to design adaptive institutions can fulﬁll their basic “mechanical” mediation function in society, as well as shepherd their constituents towards more desirable strategies and behaviors.