S TACKELBERG P OLICY G RADIENT : E VALUATING THE P ERFORMANCE OF L EADERS AND F OLLOWERS
A BSTRACT
Hierarchical order of play is an important concept for reinforcement learning to understand better the decisions made by strategic agents in a shared environment.
In this paper, we compare the learning dynamics between Stackelberg and simultaneous reinforcement learning agents.
Agents are trained using their policy gradient and are tested against each other in a tournament.
We compare agent performance in zero-sum and non-zero-sum Markov games.
We show that the
Stackelberg leader performs better in training under the same parameters.
However, under the same parameters in the tournament setting, Stackelberg leaders and followers performed similarly to the simultaneous player.
Analytically, hierarchical training can potentially provide stronger guarantees for policy gradient.
I NTRODUCTION
An emerging field of machine learning pits strategic agents against each other in competitive games.
A popular example of the success of competitive machine learning is Google’s AlphaZero vs. Stockfish in a zero-sum game of chess.
(Dai & Song, 2018; Zhang et al., 2021)
Although Stockfish calculates roughly 1000 times more moves per second than AlphaZero, each agent is evenly matched.
AlphaZero uses reinforcement learning (RL) to determine the best moves to make given the state of the environment.
Motivated by the interest in RL and hierarchical learning, we aim to compare the performance of agents trained via simultaneous and Stackelberg reinforcement learning in both competitive (zero-sum) and general (non-zero-sum) environments.
An important application of RL and game theory involves a hierarchical order of play, which are settings best modeled as Stackelberg games.
The concept of the Stackelberg equilibrium (Von Stackelberg, 2010) generalizes the min-max solution to general-sum games.
In the two-player scenario, one player acts as the leader who selects an action given the knowledge that the other player (follower) plays a best-response.
This viewpoint has long been researched from a control perspective on games (Bas¸ar & Olsder, 1998) and in the bilevel optimization community (Danskin, 1966).
The current machine learning perspective on games with a hierarchical decision-making structure mainly focuses on zero-sum games.
Theoretical work relates that all stable critical points of simultaneous gradient descent with a timescale separation between players approaching infinity satisfy sufficient conditions for a local Stackelberg equilibrium (Jin et al., 2019). 1. R ELATED WORKS
The theoretical results of multi-agent reinforcement learning are reviewed in (Zhang et al., 2021).
The framework of Markov games (Shapley, 1953; Littman, 1994) has been used to develop learning algorithms.
Policy gradient with function approximation (Sutton et al., 1999) is used to implement policies that map states to actions.
It is known that the Nash equilibrium always exists for finitespace infinite-horizon discounted competitive Markov games (Filar & Vrieze, 2012), but may not be unique in general.
Algorithms have been developed for general sum games (Hu & Wellman,
Reinforcement learning in games with asymmetry like hierarchy amongst agents are explored in (K¨on¨onen, 2004; Pereira, 2020).
Our work exists in the context of other research on learning dynamics for multi-agent systems where agents model the learning process of opponents.
These models are used to derive various learning updates using gradients (Foerster et al., 2017).
Games involving a hierarchical order of play are ubiquitous in nature but are a less explored area of RL.
Where previous works focused on implicit learning dynamics for improving the performance of learning algorithms (Fiez et al., 2020; Zheng et al., 2021), we focus on comparing the performance of reinforcement learning agents that adopt a leader or follower role against players trained without hierarchy.
We conducted several experiments with agents trained via simultaneous and Stackelberg policy gradient agents by varying learning rates, batch sizes, and neural network hidden layers in zero-sum and non-zero-sum Markov games to understand hierarchy of play dynamics in game type.
Contributions We show that hierarchical training has a benefit to agent performance.
In a nonzero-sum Markov game, the Stackelberg leader and follower generally performed better than the simultaneous player.
In the zero-sum Markov game, the same trend occurred, with the addition that there was a guarantee that the leader performed better than the follower.
In comparison, the first or second player could interchange in performance with the simultaneous player.
There is also potential to guarantees for policy gradient.
Our work is a preliminary investigation into evaluating the performance of the Stackelberg policy gradient in two-player games.
Future work is proposed in the discussion section.
P RELIMINARIES
We provide standard mathematical preliminaries for multi-agent reinforcement learning.
We consider a two-player fully observable Markov game defined by tuple G =
( S , A 1 , A 2 , P, r 1 , r 2 ) where
S is the state space, s ∈ S is a state, player i ∈ { 1 , 2 } , A i is the player
i ’s action space with a i ∈
A i .
The transition kernel is P : S ×
A 1 ×
A 2 → S such that P ( s ′ | s, a 1 , a 2 ) is the probability of transitioning to state s ′ given that the previous state was s and the agents took actions ( a 1 , a 2 ) simultaneously in state s .
The reward r i : S×A 1 ×B 2 → R is the reward function for player i .
Note that for zero-sum games, r 1 ≡ − r 2 , and for non-zero-sum games, the rewards may be anti-aligned.
Each agent uses a stochastic policy π iθ i parameterized by θ i .
The cumulative rewards of agent i given a trajectory τ = ( s 0 , a 10 , a 20 , . .
. , a 2 T ) is defined as
R ( τ ) =
(cid:80) Tt =
γ t r ( s t , a 1 t , a 2 t ) with discount factor 0 < γ ≤ 1 .
The expected cumulative rewards of policy π = { π 1 θ 1 , π 2 θ 2 } is Q
θ ( s t , a 1 t , a 2 t ) =
E [ (cid:80) Tt ′
=
t γ t ′ − t r ( s t ′ , a 1 t ′ , a 2 t ′ | s t , a 1 t , a 2 t )] .
The objective for each player is the expected return given by J i ( θ ) =
E τ ∼ π θ
[ (cid:80) Tt =
γ t r ( s t , a 1 t , a 2 t )] .
Agents aim to maximize their expected return.
T RAINING A GENTS USING P OLICY G RADIENT
We train agents using policy gradient algorithms based on simultaneous and Stackelberg updates.
In this section, we provide descriptions of the two policy gradient updates.
Algorithm 1 summarizes the training procedures.
3. S IMULTANEOUS POLICY GRADIENT
For the simultaneous policy gradient algorithm ( simgrad ), players compute their gradient without extra information of the other players.
Each player optimizes its objective [Equation] by descending individual policy gradients [Equation]
We direct the readers to the review paper (Zhang et al., 2021) for details about running policy gradient for two players in the mixed (non-zero-sum) setting.
[Figure]
Algorithm 1 Simultaneous or Stackelberg policy gradient ( simgrad or stackgrad )
Require:
Markov game G , parameters T, K, γ 1 , γ 2 > 0 and ( θ 01 , θ 02 ) .
1: for t = 0; t < T do 2:
Sample a batch of K trajectories from G using policies π 1 θ t 1 , π 2 θ t3: if simgrad then 4: θ t +
[Equation]
= θ t
[Equation]
+ γ
[Equation]
∇ θ 1 J 1 ( θ t 1 , θ t 2 ) Equation (1) 5: else if stackgrad then 6: θ t + = θ t 1 + γ 1 ∇ θ 1 J 1 ( θ t 1 , r ∗ ( θ t 1 ))
Equation (2) 7: end if 8: θ t + = θ t 2 + γ 2 ∇ θ 2 J 2 ( θ t
[Equation]
Equation (1) 9: end for
3. S TACKELBERG POLICY GRADIENT
For the Stackelberg policy gradient algorithm ( stackgrad ), the leader of the game knows the other player’s response and incorporates this response in its gradient update.
In a Stackelberg game, the leader and follower aim to solve the following optimization problems
[Equation]
where the leader uses its total derivative
[Equation]
) and the follower has response r ∗ : Θ 1 → Θ 2 and uses its individual policy gradient.
We direct the readers to the papers (Fiez et al., 2020; Yang et al., 2022) for more details on how to compute the leader’s update using the chain rule and compute the inverse operation by conjugate gradient.
3. T RAINING RESULTS
We perform a grid search over learning rates γ 1 , γ∈ { 10 − 1 , 10 − 2 , 10 − 3 } , batch sizes K ∈ { 1000 , 5000 } , and feed-forward neural network hidden layers using one-layer with 32 , 64 , 128 neurons per layer and one three-layer parameter, [128 , 64 , 32] . These parameters total 24 different agent configurations per policy gradient.
We run both policy gradient methods for a total of 100 epochs.
[Figure] The Stackelberg leader performs better in Markov Soccer than the follower during training, as shown in Figure 2(a)C.
The average magnitude of return for the Stackelberg leader was 488.
In contrast, the performance of simgrad seems to oscillate randomly as shown in Figure 2(a)A, with a smaller average magnitude of return of 96.4.
This shows that the Stackelberg leader is capable of anticipating the follower’s response during training to achieve a higher reward.
We see a similar reward and loss behavior in Grid Hunter for both algorithms.
As shown in Figure 2(b).
For simgrad player 1, the average return was 51.1, and for simgrad player 2, the average return was 41.3.
Similarly, for Stackelberg player 1, the average return was 48.5, and for simgrad player 2, the average return was 38.6.
This trend continues with the losses as shown in Figure 2bb and Figure 2bd.
For simgrad player 1, the average loss was 39.7, and for simgrad player 2, the average loss was 39.1.
For Stackelberg player 1, we saw an average loss of 37.4, and for Stackelberg player 2, 38.6.
Because Grid Hunter is a general sum game, there may be no strong guarantees of convergence.
Furthermore, the learning dynamics exhibit some behaviors of cycling.
T ESTING A GENTS USING T OURNAMENTS
We test the agents’ performance against each other by fixing the parameters of the agent’s policies and running a tournament-style competition.
The competition allows us to compare agents with different parameters and neural network models.
It also allows us to compare leaders and followers.
The motivation for running tournaments is to determine how well the agents perform relative to each other.
The resulting “test matrix” can be further analyzed to determine the agent rankings.
In this preliminary work, we report the average results from the tournament.
Future work will involve how to better understand the output of the test matrix.
4. P LAYING AGAINST A LEADER OR FOLLOWER
To test the leader’s performance, we pit player 1 trained with stackgrad against player 2 trained with simgrad .
Futhurmore, to test the follower’s performance, we pit player 2 trained with stackgrad against player 1 trained with simgrad .
Finally, to test all agents’ performance, we pit all agents against an agent performing random actions.
The agents’ relative performance in the game (i.e. which agent achieves a higher reward) is used to evaluate the agent’s performance relative to other agents trained with different parameters or algorithms. [Table]
[Figure]
4. P LAYING AGAINST RANDOM AGENTS
We employed agents performing random moves to test agent performance against a baseline.
Generally, the simgrad and Stackelberg agents performed similarly better than the random agents.
This comparison against a baseline also revealed agent parameters that were not as strong or hyper performers.
As well as this, to test the validity of our environments and sample size
, we competed two random agents.
As shown in Table 1, the results in both environments, the random agents playerhad a 50.0% win percentage, proving symmetry of the environment.
4. T EST RESULTS
Competitive tournaments revealed the relative performance between agents.
As shown in Table 1, simgrad and Stackelberg performance as either player 1 or 2 against random agents was almost identical.
While against each other, player 2 of simgrad and Stackelberg performed similarly better than the player 1.
Interestingly, where simgrad player 1 performed better than the player 2, Stackelberg was the opposite with a better performing follower than the leader.
In Markov Soccer, performance was similar between all agents.
However, one Stackelberg agent did notably worst than the rest (batch size = 5000, learning rate = 0.1, neural net of size [128,64,32]) as shown as the distinct streak in Figure 3a.
The best performing agents were simgrad with batch sizes = 1000 or 5000, learning rate = 0.001 and neural net of size 128.
Similar parameters for the Stackelberg agents were close behind.
In Grid Hunter, agents with either policy gradient with a high batch size of 5000, learning rate of 0.01, and a neural net of size [64] highly performed against all other agents, likely due to the highdimensionality of the environment.
Close behind were agents with batch size of 5000, learning rate of 0.001, and neural net of size [128], as shown as the distinctly bright regions of Figure 3b.
The worst performing agents similarly had a learning rate of 0.1.
D ISCUSSION
In this paper, we study the simultaneous and Stackelberg policy gradient algorithms and the performance of agents trained with them.
Our preliminary experiments show that leaders achieve a larger average reward than followers for zero-sum games during training.
On the other hand, in general sum games, the leader did not show a significant advantage over the follower.
More future work is needed to examine the leader and followers’ learned strategies in general-sum games.
For example, we can perform analyses not only of their average rewards, but also of their trajectories through the environment.
In addition, we can compare these learned strategies to that of a fully-cooperative general-sum game with team based reward setting.
A further investigation into how the Stackelberg learning dynamics affect the quality of the solutions is needed.
Other experiments can be devised to determine whether hierarchical training is beneficial for particular types of environments and agent reward structures.
Although our results found good parameters for each environment, we can extend our work to find more suitable hyper parameters.
In particular, the learning rate ratio γ 2 /γ 1 is a important parameter to tune, as it represents the timescale separation between the agents.
We want the follower to be attracted to its equilibrium at a faster timescale than the leader.
In future work, we will test these learning updates on a wider range of environments.
Currently, our simple experiments serve as a testing ground for the algorithm.
We plan to expand the domain of environments and test our algorithm on other developed multi-agent gym packages.
Finally, we can further quantify the difference between agent performance under different hyper parameters using a method for calcuating ELO scores (Gosiewska, 2019).
Other works in RL has utilized ELO scores in comparing agent models (Wang, 2021) as well as the addition of a Bayesian algorithm to understand opponent behavior more (Yang, 2019).
These extensions could provide a new lens to evaluate the performance of agents trained using the Stackelberg policy gradient algorithm.
C ONCLUSION
Understanding the dynamics of Stackelberg games is an essential step for translating RL to more comprehensive applications.
An important aspect of this understanding is the performance between simultaneous and Stackelberg players.
In training the agents, we saw that the Stackelberg leader rewards tend to be higher than the Stackelberg follower.
Furthermore, in competitive tournaments in zero-sum games, we saw similar performance between simgrad and Stackelberg agents.
In contrast to non-zero-sum games, we saw stronger performing player 1’s than player 2’s.
In conclusion, we show that agents that undergo hierarchical training can be beneficial to its performance when playing against other agents. A CKNOWLEDGMENTS
This research was supported in part by National Science Foundation Awards #1836819, #2045014, #