Influencing Long-Term Behavior in Multiagent Reinforcement Learning
Abstract
The main challenge of multiagent reinforcement learning is the difficulty of learning useful policies in the presence of other simultaneously learning agents whose changing behaviors jointly affect the environment’s transition and reward dynamics.
An effective approach that has recently emerged for addressing this non-stationarity is for each agent to anticipate the learning of other agents and influence the evolution of future policies towards desirable behavior for its own benefit.
Unfortunately, previous approaches for achieving this suffer from myopic evaluation, considering only a finite number of policy updates.
As such, these methods can only influence transient future policies rather than achieving the promise of scalable equilibrium selection approaches that influence the behavior at convergence.
In this paper, we propose a principled framework for considering the limiting policies of other agents as time approaches infinity.
Specifically, we develop a new optimization objective that maximizes each agent’s average reward by directly accounting for the impact of its behavior on the limiting set of policies that other agents will converge to.
Our paper characterizes desirable solution concepts within this problem setting and provides practical approaches for optimizing over possible outcomes.
As a result of our farsighted objective, we demonstrate better long-term performance than state-of-the-art baselines across a suite of diverse multiagent benchmark domains.
Introduction
Learning in multiagent reinforcement learning (MARL) is fundamentally difficult because an agent interacts with other simultaneously learning agents in a shared environment [1].
The joint learning of agents induces non-stationary environment dynamics from the perspective of each agent, requiring an agent to adapt its behavior with respect to potentially unknown changes in the policies of other agents [2].
Notably, non-stationary policies will converge to a recurrent set of joint policies by the end of learning.
In practice, this converged joint policy can correspond to a game-theoretic solution concept, such as a Nash equilibrium [3] or more generally a cyclic correlated equilibrium [4], but multiple equilibria can exist for a single game with some of these Pareto dominating others [5].
Hence, a critical question in addressing this non-stationarity is how individual agents should behave to influence convergence of the recurrent set of policies towards more desirable limiting behaviors.
Our key idea in this work is to consider the limiting policies of other agents as time approaches infinity.
Specifically, the converged behavior of this dynamic multiagent system is not due to some arbitrary stochastic processes, but rather each agent’s underlying learning process that also depends on the behaviors of the other interacting agents.
As such, effective agents should model how their actions can influence the limiting behavior of other agents and leverage those dependencies to shape the convergence process.
This farsighted perspective contrasts with recent work that also considers influencing the learning of other agents [6–11].
While these approaches show improved performance over methods that neglect the learning of other agents entirely [12–14], they suffer from myopic evaluation: only considering a few updates to the policies of other agents or optimizing for the discounted return, which only considers a finite horizon time of 1 / (1 − γ ) for discount factor γ [15].
Our contribution.
With this insight, we make the following primary contributions in this paper:
• Formalization of multiagent non-stationarity (Section 2).
We introduce an active Markov game setting that formalizes MARL with simultaneously learning agents as a directed graphical model and captures the underlying non-stationarity over time.
We detail how such a system eventually converges to a stationary periodic distribution.
As such, the objective is to maximize its long-term rewards over this distribution and, if each agent maximizes this objective, the resulting multiagent system settles into a new and general equilibrium concept that we call an active equilibrium.
• Practical framework for optimizing an active Markov game (Section 3).
We outline a practical approach for optimization in this setting, called FUlly Reinforcing acTive influence witH averagE Reward (FURTHER).
Our approach is based on a policy gradient and Bellman update rule tailored to active Markov games.
Moreover, we show how variational inference can be used to approximate the update function of other agents and support decentralized execution and training.
• Comprehensive evaluation of our approach (Section 4).
We demonstrate that our method consistently converges to a more desirable limiting distribution than baseline methods that either neglect the learning of others [14] or consider their learning with a myopic perspective [8] in various multiagent benchmark domains.
We also demonstrate that FURTHER provides a flexible framework such that it can incorporate recent advances in multiagent learning and improve performance in large-scale settings by leveraging the mean-field method [16].
Problem Statement: Active Markov Game
This work studies a general multiagent learning setting, where each agent interacts with other independently learning agents in a shared environment.
Agents in this setting update their policies based on recent experiences which are affected by the joint actions of all agents.
As such, while an agent cannot directly modify the future policies of other interacting agents, the agent can actively influence them by changing its own actions.
In this section, we first formalize the presence of this causal influence in multiagent interactions by introducing the new framework of an active Markov game.
We then formalize solution concepts and objectives for learning within this framework.
Finally, we discuss dependence on initial states and policies, detailing choices that we can make to minimize the impact of these initial conditions on behavior after convergence.
[Figure]
2.1 Directed Graphical Model of Active Markov Game
We define an active Markov game as a tuple M n = ⟨ I , S , A , T , R , Θ , U ⟩ ; I = { 1 , ... , n } is the set of n agents;
S is the state space; A = × i ∈I
A i is the joint action space; T : S × A (cid:55)→ S is the state transition function; R = ×
i ∈I R i is the joint reward function; Θ = × i ∈I
Θ i is the joint policy parameter space; and U = × i ∈I
U
i is the joint Markovian policy update function.
We typeset sets in bold for clarity.
Compared to the stationary Markov game that effectively represents MARL with wrongly assumed stationary policies in the future, the active Markov game considers how policies change over time (see Figure 1).
Specifically, at each timestep t , each agent i executes an action at a current state s t ∈ S according to its stochastic policy a it ∼ π i ( ·| s t ; θ it ) parameterized by θ it ∈
Θ i .
A joint action a t = { a it , a (cid:57) it } yields a transition from s t to s t +1 with probability T ( s t +1 | s t , a t ) , where the notation (cid:57) i indicates all other agents except agent i .
Each agent i then obtains a reward according to its reward function r it
=
R i ( s t , a t ) and updates its policy parameters according to
U i
( θ it +1 | θ it , τ it ) ,
where τ it = { s t , a t , r it , s t +1 } denotes agent i ’s transition.
This process continues until the convergence of non-stationary policies.
Notably, the joint policy update function U is a function of a it , which affects the state transitions and rewards, so agent i can actively influence future joint policies by changing its own behavior.
Modeling this influence rather than ignoring it is the main advantage of using active Markov games rather than the stationary Markov game formalism.
2.2 Solution Concepts in Active Markov Games
The formalism of active Markov games provides a principled framework for each agent to model the impact of its behavior on joint future policies.
In this section, we study the theoretical convergence properties of an active Markov game and develop relevant terminology that will help us characterize this convergence.
We begin by formalizing the limiting behavior as a stationary periodic distribution.
Definition 1.
(Stationary k -Periodic Distribution).
The limiting behavior of an active Markov game can be represented by a stationary periodic probability distribution over the joint space of states and policies, defined as a stationary conditional distribution with respect to a period of order k : µ
[Equation]
[Equation] [Equation]
where ℓ = t % k with % denoting the modulo operation.
The stationary k -periodic distribution satisfies the following property as its time averaged expectation stays stationary in the limit :
[Equation]
Our notion of a stationary k -periodic distribution provides a flexible representation for characterizing the limiting distribution, generalizing from fully stationary fixed-point convergence (when k =1 ) to the extreme case of totally non-stationary convergence (when k →∞ ).
Having defined the joint convergence behavior of an active Markov game, we can now develop an objective that each agent can optimize to maximize its long-term rewards.
Our key finding is that the average reward formulation, developed for single-agent learning [17, 18], is well suited for studying the limiting behavior of other interacting agents in multiagent learning.
In particular, the average reward formulation maximizes the agent’s average reward per step with equal weight given to immediate and delayed rewards, unlike the discounted return objective.
Once the joint policy converges to the stationary periodic distribution, rewards collected by this recurrent set of policies govern each agent’s average reward as t →∞ .
Thus, optimizing for the average reward in an active Markov game encourages agents to consider how to influence the limiting set of policies after convergence rather than transient policies that are only experienced momentarily.
Definition 2.
(Active Average Reward Objective).
Each agent i in an active Markov game aims to find policy parameters θ i and update function U i that maximize its expected average reward
ρ i
[Equation]
where T denotes the time horizon.
It is important to note that Equation (3) has no preference over the large equivalence class of update functions that eventually converge to an optimal limiting behavior
[Equation]
and we only require finding an update function in this class even if the convergence rate is slow.
This is advantageous for our discussion to come about solution concepts in active Markov games.
However, in our practical approach to optimization, we also optimize over the transient distribution, pushing towards solutions with lower regret by modeling our value function based on the differential returns as in Proposition 2.
We also note that even if a single agent maximizes this objective, agents will not necessarily arrive at any kind of equilibrium.
This is because other agents may have sub-optimal or biased update functions beyond the agent’s control, and a rational agent can potentially seek to converge to an average reward that is better for it than that of any equilibrium as a result.
Additionally, whether an agent just seeks to optimize its policy or maximize its update function as well depends on the kind of solution concept that is desired.
For example, finding a fixed stationary policy equates to using an update function that arrives at a fixed point, whereas we can also optimize over the update function if we seek to find an optimal non-stationary policy as in the meta-learning literature [9, 19].
If all agents maximize the active average reward objective, we arrive at a new and general equilibrium concept that we call an active equilibrium, where no agents can further optimize its average reward:
Definition 3.
(Active Equilibrium).
In an active Markov game, an active equilibrium is joint policy parameters θ ∗ =
{ θ i ∗ , θ (cid:57) i ∗ } with associated joint update function U ∗
=
{U i ∗ , U (cid:57) i ∗ } such that :
[Equation]
where U i denotes the space of agent i ’s update functions.
Our active equilibrium definition is related to non-stationary solution concepts in game theory, such as the non-stationary Nash equilibrium [20], that search for a sequence of best-response joint policies.
However, these non-stationary solutions are generally intractable to compute due to the unconstrained sequence over the infinite horizon and the resulting large policy search space size [21].
By contrast, the active equilibrium provides a more refined and practical notion than these solution concepts by having a constraint on the sequence based on the update functions.
We also note the generality of active equilibrium that it can correspond to other standard solution concepts as we impose restrictions on relevant variables:
Remark 1.
(Connection to Existing Solution Concepts).
Stationary Nash [3] and correlated equilibria [22] are special kinds of active equilibria when k =1 and joint action distributions are independent and correlated, respectively.
Cyclic Nash and cyclic correlated equilibria [4] are also special cases of an active equilibrium if k > 1 , the joint update function is deterministic, and joint action distributions are independent and correlated, respectively.
2.3 Addressing Sensitivity to Initial Conditions
The recurrent set of converged joint policies is generally dependent on initial states and policies, as specified by the conditioned initial variables in Equations (1) to (3).
This initial condition dependence implies that there can be instances of poor convergence performance simply due to undesirable initial states and policies (see Appendix A for an example).
In this paper, we address this sensitivity to initial conditions by considering the stochastically stable periodic distribution, which is a special case of the stationary periodic distribution.
The stochastic distribution describes the limiting joint behavior when each agent has communicating strategies (i.e., for every pair of policy parameters θ i , θ i ′ ∈
Θ i , θ i transitions to θ i ′ in a finite number of steps with non-zero probability and vice versa) by adding noise ϵ to its update function U iϵ , and noise ϵ → 0 over time (i.e., lim t →∞ U iϵ
= U i ).
Importantly, the stochastic distribution provides an important analytical benefit of independent convergence with respect to the initial conditions.
Specifically, assuming communicating state transitions T , if only agent i ’s update function is perturbed, then we arrive at the notion of self-stable periodic distribution:
Definition 4.
(Self-Stable Periodic Distribution).
Given communicating state transition T , if noise ϵ is added only to the agent i ’s update function U iϵ , we achieve the stationary k -periodic distribution independent of the initial state and the agent i ’s initial policy as ϵ → 0 over time :
[Equation]
Similarly, if the full joint update function is perturbed with noise U ϵ , this induces a unique stationary periodic distribution independent of the initial state and initial joint policy:
Definition 5.
(Jointly-Stable Periodic Distribution).
Given communicating state transition T , if noise ϵ is added to the joint update function U ϵ , we achieve the same stationary k -periodic distribution independent of the initial state and the initial policies as ϵ → 0 over time :
[Equation]
Proposition 1.
(Uniqueness of Jointly-Stable Periodic Distribution).
Given communicating state transition T and perturbed joint update function with noise U ϵ , the jointly-stable periodic distribution is unique as ϵ → 0 over time.
Proof.
See Appendix B for details.
□
The jointly-stable periodic distribution is induced in many cases of interest to multiagent learning, including when all policies employ update functions leveraging the Greedy in the
Limit with Infinite Exploration (GLIE) property [18]: 1) all state-action pairs are visited infinitely often and 2) as t → ∞ , the behavior policy converges to the greedy policy.
[Figure]
concepts
In particular, a broad class of action exploration or noisy policy update functions lead to this kind of distribution [23– 27].
Indeed, MARL algorithms generally rely on persistent exploration and thus satisfy GLIE.
Lastly, as demonstrated in Figure 2, although maximizing over the space of jointly-stable periodic distributions confines the search space of stationary periodic distributions, the best possible active equilibria still lie within this smaller space while also allowing for optimization robust to initial conditions.
We focus on designing a learning algorithm that can find an equilibrium in the practical and confined search space of the jointly-stable distributions in the following section.
FURTHER: Practical Method for Solving Active Markov Game
In this section, we develop a practical method, called FURTHER, for learning beneficial policies in the space of the jointly-stable periodic distributions.
We first outline a practical version of the average reward objective and derive its policy gradient.
We then detail our model-free implementation that builds on top of soft actor-critic [28] and variational inference [29] to learn policies that efficiently optimize for the average reward objective in a decentralized manner.
3.1 Formulation and Policy Gradient Theorem of FURTHER
While the objective in Equation (3) ideally maximizes over the space of update functions and learns a non-stationary policy, addressing the computational difficulty of long horizon meta-learning still remains an active area of research [9, 30, 31].
As such, in FURTHER, we take a practical step forward and learn the optimal fixed point policy that influences joint policy behavior to maximize its long-term average reward ρ iθ i ∈ R at a state s ∈S and policy parameters of other agents θ (cid:57) i ∈ Θ (cid:57) i :
[Equation]
where the subscript θ i notation denotes the implicit dependence on the learning of agent i ’s fixed stationary policy.
As discussed in Section 2.3, a useful result under the jointly-stable periodic distribution is that the average reward becomes independent of the initial states and policies:
[Equation]
We now derive the Bellman equation in an active Markov game that defines the relationship between the value function and average reward.
Proposition 2.
(Active Differential Bellman Equation).
The differential value function v iθ i represents the expected total difference between the accumulated rewards from s and θ (cid:57) i and the average reward
ρ iθ i [18].
The differential value function inherently includes the recursive relationship with respect to v iθ i at the next state s ′ and the updated policies of other agents θ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Proof.
See Appendix C for a derivation.
□
Finally, we derive the policy gradient based on the active differential Bellman equation:
Proposition 3.
(Active Average Reward Policy Gradient Theorem).
The gradient of active average reward objective in Equation (7) with respect to agent i ’s policy parameters θ i is :
[Equation]
[Table]
with q iθ i ( s ℓ , θ (cid:57) iℓ , a ℓ )=
(cid:80) s
[Table]
Proof.
See Appendix D for a derivation.
□
3.2 Soft Actor-Critic Implementation with Variational Inference
Algorithm overview.
FURTHER broadly consists of inference and reinforcement learning modules.
In practice, each agent has partial observations about others and cannot directly observe their true policy parameters θ (cid:57) i and policy dynamics U (cid:57) i .
The inference learning module predicts this hidden information about other agents leveraging variational inference [29] modified for sequential prediction.
The inferred information becomes the input to the reinforcement learning module, which extends the policy gradient theorem in Equation (10) and learns active average reward policies sample efficiently by building on the multiagent soft actor-critic (MASAC) framework [14, 28, 32].
We note that each agent interacts and learns these modules by only observing the actions of other agents, so our implementation supports decentralized execution and training.
We provide further details, including implementation for k> 1 and psuedocode, in Appendix E.
For simplicity, we consider the period k
=
and develop corresponding soft reinforcement learning optimizations in Equations (12) to (14).
Inference learning module.
This module aims to infer the current policies of other agents and their learning dynamics based on an approximate variational inference [29].
Specifically, we optimise a tractable evidence lower bound (ELBO), defined together with an encoder p ( ˆ z (cid:57) it +1 | ˆ z (cid:57) it , τ it ; ϕ i enc ) and a decoder p ( a (cid:57) it | s t , ˆ z (cid:57) it ; ϕ i dec ) , parameterised by ϕ i enc and ϕ i dec , respectively:
[Equation]
where latent strategies ˆ z (cid:57) it represents inferred policy parameters of other agents θ (cid:57) it , the encoder represents the policy dynamics of other agents U (cid:57) i with parameters ϕ
i enc , and τ i 0:
t = { τ i 0 , ..., τ it } denotes i ’s transitions up to timestep t .
We refer to Appendix F for a detailed ELBO derivation.
By optimizing the reconstruction term, the encoder aims to infer accurate next latent strategies of other agents.
Also, by imposing the prior through the KL divergence, where we set the prior to the previous posterior with initial prior p ( ˆ z (cid:57) i 0 )=
N (0 , I ) , the inferred policies from the encoder are encouraged to be sequentially consistent across time (i.e., no abrupt changes in policies of others).
Reinforcement learning module.
This module aims to learn a policy that can maximize the agent’s average reward based on the inferred information about other agents.
Each agent maintains its policy π ( ·| s, ˆ z (cid:57) i ; θ i ) parameterized by θ i , two q -functions q iθ i ( s, ˆ z (cid:57) i , a ; ψ i 1 ) and q iθ i ( s, ˆ z (cid:57) i , a ; ψ i 2 )
[Figure]
parameterized by ψ i 1 , ψ i 2 , and learnable average reward ρ iθ i ∈ R .
We train the q-functions and ρ iθ i by minimizing the soft Bellman residual:
[Equation]
where β =1 , 2 , D i denotes i ’s replay buffer, and ¯ ψ iβ denotes the target q -network parameters.
The soft value function v iθ i calculates the state value with the policy entropy H and entropy weight α : v iθ i ( s, ˆ z (cid:57) i ; ψ i )=
(cid:80)
[Equation]
Finally, the policy is trained to maximize:
[Equation]
We note that Equations (13) and (14) are for discrete action space, and we detail optimizations for continuous action space in Appendix E.
Mean-Field FURTHER.
FURTHER provides a flexible framework such that it can easily integrate recent advances in multiagent learning.
For example, by reconstructing and predicting the mean action and latent strategy of neighbor agents in Equation (11), we can incorporate the mean-field framework to improve performance in large-scale learning settings.
Appendix E details the mean-field version of FURTHER with pseudocode.
Evaluation
We demonstrate FURTHER’s efficacy on a diverse suite of multiagent benchmark domains.
We refer to Appendix G for experimental details and hyperparameters.
The code is available at https: //bit.ly/3fXArAo , and video highlights are available at https://bit.ly/37IWeb9 .
The mean and 95% confidence interval computed for 20 seeds are shown in each figure.
Baselines.
We compare FURTHER with the following baselines (see Appendix G.2 for details):
• LILI [8]: An approach that considers the learning dynamics of other agents but suffers from myopic evaluation bias by optimizing the discounted return objective (see Equation (25)).
• MASAC [14]: An approach that extends SAC [28] to a
multiagent learning setting by having centralized critics [12].
This baseline assumes other agents will have stationary policies in the future and thus neglects their learning (see Equation (26)).
We note that these selected baselines are closely related to FURTHER, optimizing different objectives with respect to U (cid:57) i .
In particular, LILI and MASAC optimize the discounted return objective with and without modeling U (cid:57) i , respectively.
As such, our baseline choices enable us to separately analyze the effect of FURTHER’s novel average reward objective.
For completeness, we also consider additional baselines of an opponent modeling method (DRON) and an incentive MARL method (MOA).
These results are shown in Appendix H.
[Figure]
v e r a g
[Figure]
a r d f o r A g e n t i
IBS Comparison against Q-Learner
[Figure]
Question 1.
How do methods perform when playing against a q -learning agent?
We consider playing the iterated Bach or Stravinsky game (IBS; see Figure 3a).
This general-sum game involves conflicting elements with two pure strategy Nash equilibria, where convergence to (B,B) and (S,S) equilibrium are more preferable from agent
i ’s and j ’s perspective, respectively.
Suppose agent i plays against a naive learner j , such as q -learner [34], whose initial q -values are set to prefer action (S).
In this experimental setting, it is ideal for agent i to change j ’s influence behavior to select (B) such that they converge to i ’s optimal pure strategy Nash equilibrium of (B,B).
The average reward performance when an agent i , trained with either FURTHER or the baseline methods, interacts with the q -learner j is shown in Figure 4a.
There are two notable observations.
First, the FURTHER agent i consistently converges to its optimal equilibrium of (B,B), while the
LILI agent often converges to the sub-optimal equilibrium of (S,S).
The FURTHER agent i learns to select (B) while j selects (S), receives the worst rewards of zero, and waits until j ’s q -value for (S) is updated to be lower than the q -value for (B).
With the limiting view, i learns that the waiting process is only temporary, and receiving the eventual rewards of 2 by converging to (B,B) is optimal.
By contrast, LILI suffers from myopic evaluation and shows decreased performance upon convergence because the agent prefers simply converging to the sub-optimal equilibrium rather than waiting indefinitely.
Figure 5a also shows that LILI achieves sub-optimal performance for any value of γ and shows unstable learning as γ → 1 .
Second, FURTHER and LILI outperform the other approach of
MASAC, showing the benefit of considering the active influence on future policies of other agents.
Question 2.
Which equilibrium do methods converge to in a self-play setting?
We experiment with a self-play setting in which both agents learn with the same algorithm in an iterated cooperative (IC) game with identical payoffs (see Figure 3b).
This game has two pure strategy
Nash equilibria of (U,U) and (D,D), in which the (D,D) equilibrium Pareto dominates the other.
Figure 4b shows the average reward performance as the training iteration increases.
First we find that
LILI performs better than MASAC by considering the learning of agents.
However, similar to the
IBS results, we observe that FURTHER consistently converges to the best equilibrium of (D,D) while the baseline methods can converge to the sub-optimal equilibrium of (U,U) due to the myopic view.
Question 3.
How does FURTHER’s limiting optimization perform directly against baselines?
We consider FURTHER agent i directly competing against either LILI or MASAC opponent j in the iterated matching pennies (IMP) game (see Figure 3c).
To show that FURTHER has a long-term perspective and thus can collect more rewards than the opposing method over time, we evaluate using a metric of relative accumulated reward summed up to the current timestep: (cid:80) t r it − r jt .
Figure 4c shows that the relative accumulated reward for FURTHER is positive for both settings, meaning that FURTHER receives higher rewards than LILI and MASAC over time.
This result suggests that FURTHER is more effective than LILI by employing the limiting view via the average reward formulation.
This result also conveys that it is beneficial to consider the underlying learning dynamics rather than ignoring them because FURTHER can more easily exploit the MASAC opponent and achieve higher accumulated rewards than when competing against the LILI opponent.
Question 4.
How does FURTHER scale to a more complex environment?
[Figure]
To answer this question, we use the MuJoCo RoboSumo domain ([19]; see Figure 3d), where two ant robots compete with each other with the objective of pushing the opponent out of the ring.
The reward function consists of a sparse reward of 5 for winning against the opponent and shaped rewards of moving towards the opponent and pushing the opponent further from the center of the ring.
This environment has complex interactions because an agent must learn how to control its joints with continuous action space to move around the ring while learning to push the opponent.
Similar to the setup in Question 3, FURTHER agent i directly competes against either LILI or MASAC opponent
j .
We note that each agent has only partial observations about its opponent.
As such, an agent infers its opponent’s hidden policies and learning dynamics based on partial observations.
We show the RoboSumo results in Figure 5b.
Consistent with our results in the iterated matrix games, we observe that FURTHER gains more rewards than the baselines over time and wins against
MASAC more often than against LILI.
The averaged winning rate across the entire interaction shows that FURTHER wins against LILI and MASAC with 60.6% and 63.9%, respectively.
Therefore,
FURTHER provides a scalable framework that can learn policies in an environment with complex interactions and continuous action space.
Question 5.
How does FURTHER scale to a large number of agents?
Finally, we show the scalability of our method regarding the number of agents using the battle domain ([33]; see Figure 3e).
In this large-scale mixed cooperative-competitive setting, a red team ofagents and a blue team of 25 agents interact in a gridworld, where each agent collaborates with its teammates to eliminate the opponents.
Specifically, we compare when red and blue agents learn with the mean-field version of FURTHER (i.e., FURTHER-MF) and LILI (i.e., LILI-MF), respectively, where they predict the mean actions of neighboring agents.
We note that all 50 agents learn in a decentralized manner without sharing parameters with one another.
It is evident that FURTHER-MF outperforms LILI-MF, which shows the effectiveness of having the limiting perspective.
This result also conveys that FURTHER can easily incorporate other techniques in multiagent learning and show improved performance in large-scale settings.
Related Work
Stationary MARL.
The standard approach for addressing non-stationarity in MARL is to consider information about other agents and reason about joint action effects [35].
Example frameworks include centralized training with decentralized execution, which accounts for the actions of other agents through a centralized critic [12–14, 16, 36, 37].
Other related approaches include opponent modeling frameworks that infer opponent policies and condition an agent’s policy on this inferred information about others [38–41].
While this does alleviate non-stationarity, each agent learns its policy by assuming that other agents will follow the same policy into the future.
This assumption is incorrect because other agents can have different behavior in the future due to their learning [6], resulting in instability with respect to their changing behavior.
In contrast, FURTHER models the learning processes of other agents and considers how to actively influence limiting behavior.
Learning-aware MARL.
Our framework is closely related to prior work that considers the learning of other agents in the environment.
The framework by [42], for instance, learns the best response adaptation to the other agent’s anticipated updated policy.
Notably, LOLA [6] and its more recent improvements [7, 43] study the impact of behavior on one or a few of another agent’s policy updates.
Our work is also related to frameworks that leverage the inferred policy dynamics of other agents to impact their future policies by maximizing the discounted return objective [8, 10].
Meta-learning frameworks are also related that directly account for the non-stationary policy dynamics in multiagent settings based on the inner-loop and outer-loop optimization [9, 11, 19, 44].
Lastly, the field of incentive MARL [45–48] is related, where agents additionally optimize incentive rewards and learn successful policies in solving sequential social dilemma domains [49, 50].
However, all of these approaches only account for a finite number of updates to the policies of other agents, so we observe that these methods can converge to a less desirable solution.
FURTHER addresses this issue by optimizing for the average reward objective in the active Markov game setting.
Game-theoretic MARL.
Another effective approach to addressing the non-stationarity is learning equilibrium policies that correspond to game-theoretic solution concepts [4, 51–54].
These frameworks predict stationary joint action values by the end of learning and can guarantee convergence to
Nash [3] or correlated [22] equilibrium values under certain assumptions.
However, as noted in [55], this convergence is guaranteed only while ignoring the actual learning dynamics of other agents, and each agent assumes all agents will play the same joint equilibrium strategy.
As such, equilibrium learners can fail to learn best-response policies when others choose to play different equilibrium strategies in the future as a result of their learning.
By contrast, FURTHER considers convergence to a recurrent set of joint policies by inferring the true policy dynamics of other agents.
Conclusion
In this paper, we have introduced FURTHER to address non-stationarity by considering each agent’s impact on the converged policies of other agents.
The key idea is to consider the limiting policies of other agents through the average reward formulation for a newly proposed active Markov game framework, and we have developed a practical model-free and decentralized approach in this setting.
We evaluated our method on various multiagent settings and showed that FURTHER consistently converges to more desirable long-term behavior than state-of-the-art baseline approaches.
Acknowledgments
Research funded by IBM (as part of the MIT-IBM Watson AI Lab initiative).
A Example of Initial Condition Sensitivity
[Figure]
i
IPD
[Figure] [Figure]
Consider playing the iterated prisoner’s dilemma (IPD) game (see Table 1), where agent i plays against a q -learning agent j .
We perform a policy iteration analysis [17] with respect to
C
D
C ( (cid:57) 1 , (cid:57) 1) ( (cid:57) 3 , 0)
D (0 , (cid:57) 3) ( (cid:57) 2 , (cid:57) 2)
[Table]
j ’s varying initial q -values for each action θ (cid:57) i 0 .
Figure 6a and Figure 6b show agent
i ’s average reward after convergence with respect to θ (cid:57) i 0 when j trains with a greedy and GLIE algorithm, respectively.
Interestingly, the analysis with the greedy algorithm shows that i ’s average reward depends on θ (cid:57) i 0 in IPD, where there is a set of j ’s initial policies that i can achieve the high average reward, but there is the other set of initial policies that can result in the undesirable average reward of (cid:57) 2 .
By contrast, Figure 6b shows that i ’s average reward is independent of θ (cid:57) iwhen j ’s learning satisfies GLIE, empirically supporting our discussion in Section 2.3.
B Uniqueness of Jointly-Stable Periodic Distribution
Proposition 1.
(Uniqueness of Jointly-Stable Periodic Distribution).
Given communicating state transition T and perturbed joint update function with noise U ϵ , the jointly-stable periodic distribution is unique as ϵ → 0 over time.
Proof.
A perturbed Markov process has a unique stochastically stable distribution as noise ϵ →over time if a perturbed Markov process is regular: the transition matrix corresponding to a stationary policy contains a single recurrent class of states (i.e., states that are visited infinitely often) and a possibly empty set of transient states (i.e., states that are visited only finitely often) [27] (Corollary 4.8, Section 5).
As such, we prove that a Markov process of an active Markov game is regular by contradiction and thus show that the jointly-stable periodic distribution is unique as ϵ → 0 .
Suppose a perturbed Markov process of an active Markov game is irregular (i.e., there is more than one recurrent class), where the corresponding Markov matrix over the joint space of states and policies is defined as p ( s ′ , θ ′ | s, θ )
= (cid:80) a π ( a | s ; θ ) T ( s ′ | s, a )
U
ϵ
( θ ′ | θ , τ ) ∀ s, s
′ ∈ S , θ , θ ′ ∈ Θ .
Because the perturbed joint update function has communicating strategies and thus contains a single recurrent class of policies, the state transition T must have multiple recurrent classes to result in an irregular active
Markov game.
However, T has a single recurrent class only due to the communicating assumption, which is the contradiction.
Therefore, we conclude that a perturbed Markov process of an active game is regular, which has a unique stochastically stable distribution as ϵ → 0 by [27].
□
C Derivation of Active Differential Bellman Equation
Proposition 2.
(Active Differential Bellman Equation).
The differential value function v iθ i represents the expected total difference between the accumulated rewards from s and θ (cid:57) i and the average reward
ρ iθ i [18].
The differential value function inherently includes the recursive relationship with respect to v iθ i at the next state s ′ and the updated policies of other agents θ (cid:57) i ′ :
[Equation]
Proof.
We seek to derive the recursive relationship between v iθ i ( s, θ (cid:57) i ) and v iθ i ( s ′ , θ (cid:57) i ′ ) .
We leverage the general derivation outlined in [18] (page 59) and extend it to our active Markov game formulation:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
) □
D Derivation of Active Average Reward Policy Gradient
Proposition 3.
(Active Average Reward Policy Gradient Theorem).
The gradient of active average reward objective in Equation (7) with respect to agent i ’s policy parameters θ i is :
[Equation]
with
[Equation]
Proof.
We seek to derive an expression for optimizing the average reward objective in Equation (7) with respect to agent i ’s policy parameters θ i .
Our derivation leverages the general policy gradient theorem proof for the continuing case in [18] (page 334).
We begin by expressing the gradient of the differential value function v iθ i ( s, θ (cid:57) i ) for s ∈S and θ (cid:57)
i ∈ Θ (cid:57) i :
[Equation]
We continue to derive the Term A in Equation (16):
[Equation]
We summarize Equation (16) and Equation (17) together and re-arrange terms to obtain:
[Equation]
We define the jointly-stable periodic distribution with respect to the agent i ’s fixed stationary policy:
[Equation]
where θ ℓ = { θ i , θ (cid:57) iℓ } .
We now apply Equation (19) to Equation (18) and derive the final expression for policy gradient by writing ∇ θ i
ρ iθ i as ∇ θ
i J iπ ( θ i ) :
[Equation]
ℓ
[Equation]
Note that the left-hand side ∇ θ i J iπ ( θ i ) does not depend on s ℓ and θ (cid:57) iℓ , so Equation (20) becomes:
[Equation]
E Additional Implementation Details E.1 Network Structure
Our neural networks for the inference learning and reinforcement learning module consist of fullyconnected layers for vector observations (e.g., iterated matrix games, MuJoCo RoboSumo [19]) and additional convolution layers for image observations (e.g., MAgent Battle [33]).
The encoder outputs the mean and standard deviation for the Gaussian distribution of p ( ˆ z (cid:57) it +1 | ˆ z (cid:57) it , τ it ; ϕ i enc ) , where we sample ˆ z (cid:57) it by applying the reparameterization trick [29].
From the sampled ˆ z (cid:57) it , the decoder p ( a (cid:57) it | s t , ˆ z (cid:57) it ; ϕ i dec ) outputs a probability for the categorical distribution (discrete action space) or a mean and variance for the Gaussian distribution (continuous action space).
Similarly, the policy π ( a it | s t , ˆ z (cid:57) it ; θ i ) outputs a probability for the categorical distribution (discrete action space) or a mean and variance for the Gaussian distribution (continuous action space).
Lastly, the critic outputs q -values for all actions for discrete action space (i.e., q iθ i ( a it | s t , ˆ z (cid:57) it , a (cid:57) it ; ψ iβ ) ) by following [32] or outputs a q -value given the joint action for continuous action space (i.e., q iθ i ( s t , ˆ z (cid:57) it , a t ; ψ iβ ) ).
E.2 Optimization
We detail additional notes about our implementation:
• For simplicity, we consider the period k
=
and develop corresponding soft reinforcement learning optimizations in Section 3.2.
The current FURTHER implementation can be extended to settings with k> 1 by sampling k states and policies that are consecutive within each batch.
• For continuous action space, we modify SAC for continuous action space [28] and replace the soft value function v iθ i in Equation (13) with: v iθ i ( s, ˆ z (cid:57) i ; ψ
[Equation] [Equation]
We also replace the policy optimization in Equation (14) with the following:
[Equation]
where a i =
f
θ i
( ϵ ; s, ˆ z (cid:57) i ) denotes the output of the reparameterized i ’s policy [28]
• In practice, we apply a weighting of 0 .
01 on the KL divergence term in Equation (11) for balanced training of the inference learning module.
• Because it is impractical to consider the entire interactions from the beginning of the game in computing Equation (11), we limit τ i 0: t − 1 to be recent interactions specified by a batch size.
Require:
Learning rates α q , α ρ , α π , α ϕ , soft q -target update rate τ q 1: # Agent initialization 2: for Each agent i do 3:
Initialize RL module θ i , ψ i 1 , ψ i 2 , ¯ ψ i 1 , ¯ ψ i 2 , ρ iθ i , D i 4:
Initialize inference module ϕ i enc , ϕ i dec 5:
Initialize other agents’ latent strategies ˆ z (cid:57) i6: end for 7: for Each timestep t do 8: # Decentralized execution 9: for Each agent i do 10:
Select action a it ∼ π ( ·| s t , ˆ z (cid:57) it ; θ i ) 11: end for 12:
Execute joint action a t and receive next state s t +1 and joint rewards r t 13:
# Mean action computation and perform inference 14: for Each agent i do 15: if Apply mean-field then 16:
Compute mean action of its neighborhood ¯ a (cid:57) i t and set a t = { a i t , ¯ a (cid:57) i t } 17: end if 18:
Infer next updated policies of other agents ˆ z (cid:57) it +1 ∼ p ( ·| ˆ z (cid:57) it , τ it ; ϕ i enc ) 19:
Add a transition to its replay memory D
i ←D i ∪{ s t , ˆ z (cid:57) it , a t , r it , s t +1 , ˆ z (cid:57) it +1 } 20: end for 21: # Decentralized training 22: for Each agent i do 23: { ψ iβ , ρ iθ i } ← { ψ iβ , ρ iθ i } − { α q , α ρ } J iq ( ψ iβ , ρ iθ i ) for β = 1 ,24: θ i ← θ i
+ α π J iπ ( θ i ) 25: { ϕ i enc , ϕ
i dec } ← { ϕ i enc , ϕ i dec } − α ϕ J i elbo ( ϕ i enc , ϕ i dec ) 26: ¯ ψ iβ ← τ q ψ iβ + (1 − τ q ) ¯ ψ iβ for β = 1 ,27: end for 28: end for
F ELBO Derivation
We derive our ELBO optimization in Equation (11) for the inference module.
In particular, we follow the ELBO derivation in [56] (Appendix A) and modify it for our multiagent setting:
[Equation]
Finally
[Equation]
we summarize terms to obtain:
[Equation]
G Experimental and Hyperparameter Details G.1 Domain Details
Iterated matrix games.
As in [6], we model the state space in all iterated matrix games as s 0 = ∅ and s t = a t − 1 for t ≥ 1 .
For these simple domains, we empirically observe that training the policy and critics based on the most recent transition improves training performance.
Lastly, in Question 1, we consider agent i playing against a q -learning agent j with a learning rate α q of 0 .
5 , a discount factor γ of 0 .
9 , and a fixed ϵ -exploration of 0 . 05 .
MuJoco RoboSumo.
Each ant robot observes a vector with size 128 , which consists of the position of its own and the opponent’s body, its own joint angles and velocities, and forces exerted on each part of its own body and the opponent’s torso [19].
We note that each agent has partial observations about its opponent and cannot observe the opponent’s velocities and limb positions.
Regarding the action space, each agent has a continuous action space with a dimension of 8 .
Lastly, we use the reward function that consists of a sparse reward of 5 for winning against the opponent and the following shaped rewards:
• Reward for moving towards the opponent proportional to (cid:57) d opp , where d opp denotes the distance between the agent and the opponent.
• Reward for pushing the opponent further from the center of the ring proportional to exp ( (cid:57) d center ) , where d center denotes the distance of the opponent from the center of the ring.
We refer to [19] (Appendix D) for more RoboSumo details.
MAgent Battle.
Each agent receives an observation of a 13 × 13 × 9 image with the following channels: its and opponent’s team presence, its and opponent’s team HP, its and opponent’s team minimap, and its position [33].
The discrete action space has a dimension of 21 for moving around the gridworld and attacking the opponents.
Lastly, reward is given as 5 for killing an opponent, (cid:57) 0 .
005 for every timestep cost, 0 , 2 for attacking an opponent, and (cid:57) 0 .
1 reward for dying.
We refer to [33] for more
MAgent details.
G.2 Baseline Details
• LILI [8] maximizes the discounted return v iθ i in the active Markov game:
[Equation]
We implement LILI by replacing the average reward target y in Equation (12) with the discounted return target:
y =
[Equation]
i
[Equation]
• MASAC [14] maximizes the discounted return v iθ i in the stationary Markov game:
[Equation]
MASAC employs the framework of centralized training with decentralized execution [12] and has access to other agents’ policies to perform optimization during training.
• DRON [57]: An approach that extends DQN [58] with opponent modeling by predicting both Q-values and current strategies of other agents.
This baseline fails to predict future policies of others.
• MOA [45]: An approach that additionally optimizes the influence reward to consider influential actions to other agents.
This baseline also has the discounted return objective.
G.3 Hyperparameter Details
We use an internal cluster equipped with GPUs of RTX 3090 and CPUs of AMD Threadripper 3960X for choosing hyperparameters.
We report the important hyperparameter values that we used for each of the methods in our experiments:
[Table] [Table]
[Table] [Table]
Hyperparameter
Value
Critic learning rate
[Figure]
Gain learning rate
[Figure]
Actor learning rate α
[Figure] [Figure]
Inference learning rate α ϕ
[Figure]
Entropy weight
[Figure]
Dimension of latent space | z
[Figure]
Batch size
H Additional Evaluation
[Figure]
r
[Figure]
t
i
[Figure]
We show additional results in playing the iterated matrix games with DRON and MOA (see Figures 7a to 7c).
Because DRON and MOA also suffer from myopic evaluation, we generally observe the sub-optimal performance of these baselines in our evaluations.
In particular, DRON does not consider the underlying learning of other agents, resulting in the FURTHER agent easily exploiting the DRON opponent in Figure 7c.
We also observe that, while MOA’s optimization of the influence reward can effectively learn coordination in sequential social dilemma domains [49, 45], this influence reward optimization may not be useful in the competitive setting.
I Limitation and Societal Impact
FURTHER has a limitation that the framework does not consider an agent i ’s own non-stationary policy.
As discussed in Section 3, it is ideal to maximize the average reward over the space of joint update functions, including i ’s own update function.
However, it is computationally intractable to solve long horizon meta-learning by considering i ’s own policy dynamics, and this remains an active area of research [9, 30, 31].
Instead, we take a practical approach by assuming i ’s fixed stationary policy.
Taking an agent’s own non-stationary policy into account is one of the future directions.
We also model the period as k = 1 for simplicity in our experiments, and studying how varying k has a potential effect on performance is another future direction.
Regarding the societal impact, while
FURTHER can achieve a better social outcome in cooperative and self-play settings, a FURTHER agent aims to influence other agents to converge to desirable policies from its perspective.
As such, there can be applications, where the framework may lead to negative societal impacts by taking advantage of other agents’ defective decision-making.