In ﬂ uencing Long-Term Behavior in Multiagent Reinforcement Learning
Anonymous Author(s)
Af ﬁ liation
Address
Abstract
The main challenge of multiagent reinforcement learning is the dif ﬁ culty of learning useful policies in the presence of other simultaneously learning agents whosechanging behaviors jointly affect the environment’s transition and reward dynamics.
An effective approach that has recently emerged for addressing this non-stationarityis for each agent to anticipate the learning of other agents and in ﬂ uence the evolution of future policies towards desirable behavior for its own bene ﬁ t.
Unfortunately,previous approaches for achieving this suffer from myopic evaluation, consideringonly a ﬁ nite number of policy updates.
As such, these methods can only in ﬂ uencetransient future policies rather than achieving the promise of scalable equilibriumselection approaches that in ﬂ uence the behavior at convergence.
In this paper, wepropose a principled framework for considering the limiting policies of other agentsas time approaches in ﬁ nity.
Speci ﬁ cally, we develop a new optimization objectivethat maximizes each agent’s average reward by directly accounting for the impactof its behavior on the limiting set of policies that other agents will converge to.
Our paper characterizes desirable solution concepts within this problem setting andprovides practical approaches for optimizing over possible outcomes.
As a resultof our farsighted objective, we demonstrate better long-term performance thanstate-of-the-art baselines across a suite of diverse multiagent benchmark domains.
Introduction
Learning in multiagent reinforcement learning (MARL) is fundamentally dif ﬁ cult because an agentinteracts with other simultaneously learning agents in a shared environment [1].
The joint learning ofagents induces non-stationary environment dynamics from the perspective of each agent, requiringan agent to adapt its behavior with respect to potentially unknown changes in the policies of otheragents [2].
Notably, non-stationary policies will converge to a recurrent set of joint policies by theend of learning.
In practice, this converged joint policy can correspond to a game-theoretic solutionconcept such as a Nash equilibrium [3] or more generally a cyclic correlated equilibrium [4], butmultiple equilibria can exist for a single game with some of these Pareto dominating others [5].
Hence, a critical question in addressing this non-stationarity is how individual agents should behaveto in ﬂ uence convergence of the recurrent set of policies towards more desirable limiting behaviors.
Our key idea in this work is to consider the limiting policies of other agents as time approachesin ﬁ nity.
Speci ﬁ cally, the converged behavior of this dynamic multiagent system is not due to somearbitrary stochastic processes, but rather each agent’s underlying learning process that also dependson the behaviors of the other interacting agents.
As such, effective agents should model how theiractions can in ﬂ uence the limiting behavior of other agents and leverage those dependencies to shapethe convergence process.
This farsighted perspective contrasts with recent work that also considersin ﬂ uencing the learning of other agents [6–11].
While these approaches show improved performance over methods that neglect the learning of other agents entirely [12–14], they suffer from myopicevaluation: only considering a few updates to the policies of other agents or optimizing for thediscounted return, which only considers a ﬁ nite horizon time of 1 / (1 − γ ) for discount factor γ [15].
Our contribution.
With this insight, we make the following primary contributions in this paper:
• Formalization of multiagent non-stationarity (Section 2).
We introduce an active Markov gamesetting that formalizes MARL with simultaneously learning agents as a directed graphical model andcaptures the underlying non-stationarity over time.
We
detail how such a system eventually convergesto a stationary periodic distribution.
As such, the objective is to maximize its long-term rewards overthis distribution and, if each agent maximizes this objective, the resulting multiagent system settlesinto a new and general equilibrium concept that we call an active equilibrium.
• Practical framework for optimizing an active Markov game (Section 3).
We outline a practicalapproach for optimization in this setting, called FUlly Reinforcing acTive in ﬂ uence witH averagEReward (FURTHER).
Our approach is based on a policy gradient and Bellman update rule tailored toactive Markov games.
Moreover, we show how variational inference can be used to approximate theupdate function of other agents and support decentralized execution and training.• Comprehensive evaluation of our approach (Section 4).
We demonstrate that our methodconsistently converges to a more desirable limiting distribution than baseline methods that eitherneglect the learning of others [14] or consider their learning with a myopic perspective [8] in variousmultiagent benchmark domains.
We also demonstrate that FURTHER provides a ﬂ exible framework
such that it can incorporate recent advances in multiagent learning and improve performance inlarge-scale settings by leveraging the mean- ﬁ eld method [16].
Problem Statement: Active Markov Game
This work studies a general multiagent learning setting, where each agent interacts with otherindependently learning agents in a shared environment.
Agents in this setting update their policiesbased on recent experiences which are affected by the joint actions of all agents.
As such, while anagent cannot directly modify the future policies of other interacting agents, the agent can activelyin ﬂ uence them by changing its own actions.
In this section, we ﬁ rst formalize the presence of thiscausal in ﬂ uence in multiagent interactions by introducing the new framework of an active Markovgame.
We then formalize solution concepts and objectives for learning within this framework.
Finally,we discuss dependence on initial states and policies, detailing choices that we can make to minimizethe impact of these initial conditions on behavior after convergence.
[Figure]
2.1 Directed Graphical Model of Active Markov Game
We de ﬁ ne an active Markov game as a tuple M n = ⟨
I , S , A , T , R , Θ , U ⟩ ; I = { 1 , ... , n } is the 69 set of n agents;
S is the state space; A = × i ∈
I
A i is the joint action space; T : S ×
A 7 → S is the 70 state transition function;
R = × i ∈
I
R i is the joint reward function; Θ = × i ∈
I Θ i is the joint policy 71 parameter space; and
U = ×
i ∈
I U
i is the joint Markovian policy update function.
We typeset sets 72 in bold for clarity.
Compared to the stationary Markov game that effectively represents MARLwith stationary policies in the future, the active Markov game considers how policies change overtime (see Figure 1).
Speci ﬁ cally, at each timestep t , each agent i executes an action at a currentstate s t ∈ S according to its stochastic policy a it ∼ π
i ( ·| s t ; θ it ) parameterized by θ it ∈
Θ i .
A joint action a t = { a it , a 9 it } yields a transition from s t to s t +1 with probability T ( s t +1 | s t , a t ) , where the 77 notation 9 i indicates all other agents except agent i .
Each agent i then obtains a reward according toits reward function r it
=
R i ( s t , a t ) and updates its policy parameters according to U
i ( θ it +1 | θ it , τ it ) ,where τ it = { s t , a t , r it , s t +1 } denotes agent i ’s transition.
Notably, the joint policy update function Uis a function of a it , which affects the state transitions and rewards, so agent i can actively in ﬂ uencefuture joint policies by changing its own behavior.
Modeling this in ﬂ uence rather than ignoring it isthe main advantage of using active Markov games rather than the stationary Markov game formalism.
2.2 Solution Concepts in Active Markov Games
The formalism of active Markov games provides a principled framework for each agent to model theimpact of its behavior on joint future policies.
In this section, we study the theoretical convergenceproperties of an active Markov game and develop relevant terminology that will help us characterizethis convergence.
We begin by formalizing the limiting behavior as a stationary periodic distribution.
De ﬁ nition 1.
(Stationary k -Periodic Distribution).
The limiting behavior of an active Markov gamecan be represented by a stationary periodic probability distribution over the joint space of states andpolicies, de ﬁ ned as a stationary conditional distribution with respect to a period of order k :
[Equation]
where ℓ = t % k with % denoting the modulo operation.
The stationary k -periodic distribution satis ﬁ esthe following property as its time averaged expectation stays stationary in the limit :
[Equation]
Our notion of a stationary k -periodic distribution provides a ﬂ exible representation for characterizingthe limiting distribution, generalizing from fully stationary ﬁ xed-point convergence (when k =1 ) tothe extreme case of totally non-stationary convergence (when k →∞ ).
Having de ﬁ ned the joint convergence behavior of an active Markov game, we can now developan objective that each agent can optimize to maximize its long-term rewards.
Our key ﬁ nding isthat the average reward formulation, developed for single-agent learning [17, 18], is well suitedfor studying the limiting behavior of other interacting agents in multiagent learning.
In particular,the average reward formulation maximizes the agent’s average reward per step with equal weightgiven to immediate and delayed rewards, unlike the discounted return objective.
Once the jointpolicy converges to the stationary periodic distribution, rewards collected by this recurrent set ofpolicies govern each agent’s average reward as t →∞ .
Thus, optimizing for the average reward in anactive Markov game encourages agents to consider how to in ﬂ uence the limiting set of policies afterconvergence rather than transient policies that are only experienced momentarily.
De ﬁ nition 2.
(Active Average Reward Objective).
Each agent i in an active Markov game aims toﬁ nd policy parameters θ i and update function U i that maximize its expected average reward ρ i ∈ R :
[Equation]
where T denotes the time horizon.
It is important to note that Equation (3) has no preference over thelarge equivalence class of update functions that eventually converge to an optimal limiting behavior,and we only require ﬁ nding an update function in this class even if the convergence rate is slow.
Thisis advantageous for our discussion to come about solution concepts in active Markov games.
However,in our practical approach to optimization, we also optimize over the transient distribution, pushingtowards solutions with lower regret by modeling our value function based on the differential returnsas in Proposition 2.
We also note that even if a single agent maximizes this objective, agents willnot necessarily arrive at any kind of equilibrium.
This is because other agents may have sub-optimalor biased update functions beyond the agent’s control, and a rational agent can potentially seek toconverge to an average reward that is better for it than that of any equilibrium as a result.
Additionally, whether an agent just seeks to optimize its policy or maximize its update function as well depends onthe kind of solution concept that is desired.
For example, ﬁ nding a ﬁ xed stationary policy equates tousing an update function that arrives at a ﬁ xed point, whereas we can also optimize over the updatefunction if we seek to ﬁ nd an optimal non-stationary policy as in the meta-learning literature [9, 19].
If all agents maximize the active average reward objective, we arrive at a new and general equilibriumconcept that we call an active equilibrium, where no agents can further optimize its average reward:
De ﬁ nition 3.
(Active Equilibrium).
In an active Markov game, an active equilibrium is joint policyparameters θ ∗ =
{ θ i ∗ , θ 9 i ∗ } with associated joint update function U ∗
=
{U i ∗ , U 9 i ∗ } such that :
[Equation]
where U i denotes the space of agent i ’s update functions.
The active equilibrium de ﬁ nition is generaland thus can correspond to other solution concepts as we impose restrictions on relevant variables:
Remark 1.
(Connection to Existing Solution Concepts).
Nash [3] and correlated equilibria [20]are special kinds of active equilibria when k =1 and joint action distributions are independent andcorrelated, respectively.
Cyclic Nash and cyclic correlated equilibria [4] are also special cases of anactive equilibrium if k> 1 and joint action distributions are independent and correlated, respectively.
2.3 Addressing Sensitivity to Initial Conditions
The recurrent set of converged joint policies is generally dependent on initial states and policies, asspeci ﬁ ed by the conditioned initial variables in Equations (1) to (3).
This initial condition dependenceimplies that there can be instances of poor convergence performance simply due to undesirable initialstates and policies (see Appendix A for an example).
In this paper, we address this sensitivity toinitial conditions by considering the stochastically stable periodic distribution, which is a special caseof the stationary periodic distribution.
The stochastic distribution describes the limiting joint behaviorwhen each agent has communicating strategies (i.e., for every pair of policy parameters θ i , θ i ′ ∈
Θ i ,θ i transitions to θ i ′ in a ﬁ nite number of steps with non-zero probability and vice versa) by addingnoise ϵ to its update function U
i ϵ , and noise ϵ → 0 over time (i.e., lim t →∞ U
i
ϵ
= U i ).
Importantly,the stochastic distribution provides an important analytical bene ﬁ t of independent convergence withrespect to the initial conditions.
Speci ﬁ cally, assuming communicating state transitions T , if onlyagent i ’s update function is perturbed, then we arrive at the notion of self-stable periodic distribution:
De ﬁ nition 4.
(Self-Stable Periodic Distribution).
Given communicating state transition T , if noise ϵis added only to the agent i ’s update function U
i ϵ , we achieve the stationary k -periodic distributionindependent of the initial state and the agent
i ’s initial policy as ϵ → 0 over time :
[Equation]
Similarly, if the full joint update function is perturbed with noise U ϵ , this induces a unique stationaryperiodic distribution independent of the initial state and initial joint policy:
De ﬁ nition 5.
(Jointly-Stable Periodic Distribution).
Given communicating state transition T , if noiseϵ is added to the joint update function U ϵ , we achieve the same stationary k -periodic distributionindependent of the initial state and the initial policies as ϵ → 0 over time :
[Equation]
[Equation]
Proposition 1.
(Uniqueness of Jointly-Stable Periodic Distribution).
Given communicating statetransition T and perturbed joint update function with noise U ϵ , the jointly-stable periodic distributionis unique as ϵ → 0 over time.
Proof.
See Appendix B for details.
□
The jointly-stable periodic distribution is induced in many cases of interest to multiagentlearning, including when all policies employ update functions leveraging the Greedy in the
Limit with In ﬁ nite Exploration (GLIE) property [18]: 1) all state-action pairs are visitedin ﬁ nitely often and 2) as t → ∞ , the behavior policy converges to the greedy policy. [Figure]
In particular, a broad class of action exploration or noisypolicy update functions lead to this kind of distribution [21–25].
Indeed, MARL algorithms generally rely on persistent exploration and thus satisfy GLIE.
Lastly, as demonstrated in Figure 2, while maximizing over the space ofjointly-stable periodic distributions con ﬁ nes the searchspace of stationary periodic distributions, this still allowsconvergence to the best possible active equilibria while allowing for optimization that is robust to initial conditions.
We focus on designing a learning algorithm that can ﬁ ndan equilibrium in the practical and con
ﬁ ned search spaceof the jointly-stable distributions in the following section.
FURTHER: Practical Method for Solving Active Markov Game
In this section, we develop a practical method, called FURTHER, for learning bene ﬁ cial policies inthe space of the jointly-stable periodic distributions.
We ﬁ rst outline a practical version of the averagereward objective and derive its policy gradient.
We then detail our model-free implementation thatbuilds on top of soft actor-critic [26] and variational inference [27] to learn policies that ef ﬁ cientlyoptimize for the average reward objective in a decentralized manner.
3.1 Formulation and Policy Gradient Theorem of FURTHER
While the objective in Equation (3) ideally maximizes over the space of update functions and learnsa non-stationary policy, addressing the computational dif ﬁ culty of long horizon meta-learning stillremains an active area of research [9, 28, 29].
As such, in FURTHER, we take a practical stepforward and learn the optimal ﬁ xed point policy that in ﬂ uences joint policy behavior to maximize itslong-term average reward ρ i θ i ∈ R at a state s ∈ S and policy parameters of other agents θ 9 i ∈ Θ 9 i :
[Equation]
where the subscript θ i notation denotes the implicit dependence on the learning of agent i ’s ﬁ xedstationary policy.
As discussed in Section 2.3, a useful result under the jointly-stable periodicdistribution is that the average reward becomes independent of the initial states and policies:
[Equation]
We now derive the Bellman equation in an active Markov game that de ﬁ nes the relationship betweenthe value function and average reward.
Proposition 2.
(Active Differential Bellman Equation).
The differential value function v i θ
i representsthe expected total difference between the accumulated rewards from s and
θ 9 i and the average rewardρ i
θ
i
The differential value function inherently includes the recursive relationship with respect
tov
i θ i at the next state s ′ and the updated policies of other agents θ 9 i ′ :
[Equation]
Proof.
See Appendix C for a derivation.
□
Finally, we derive the policy gradient based on the active differential Bellman equation:
Proposition 3.
(Active Average Reward Policy Gradient Theorem).
The gradient of active averagereward objective in Equation (7) with respect to agent i ’s policy parameters θ i is :
[Equation]
with q i θ i ( s, θ 9 i , a )=
[Equation]
i
[Equation]
Proof.
See Appendix D for a derivation.
□
3.2 Soft Actor-Critic Implementation with Variational Inference
Algorithm overview.
FURTHER broadly consists of inference and reinforcement learning modules.
In practice, each agent has partial observations about others and cannot directly observe their truepolicy parameters θ 9 i and policy dynamics U 9 i .
The inference learning module predicts this hiddeninformation about other agents leveraging variational inference [27] modi ﬁ ed for sequential prediction.
The inferred information becomes the input to the reinforcement learning module, which extendsthe policy gradient theorem in Equation (10) and learns active average reward policies sampleef ﬁ ciently by building on the multiagent soft actor-critic (MASAC) framework for discrete actionspaces [14, 26, 30].
We note that each agent interacts and learns these modules by only observing theactions of other agents, so our implementation supports decentralized execution and training.
Weprovide further implementation details and psuedocodes in Appendix E.
Inference learning module.
This module aims to infer the current policies of other agents and theirlearning dynamics based on an approximate variational inference [27].
Speci ﬁ cally, we optimise atractable evidence lower bound (ELBO), de ﬁ ned together with an encoder [-ˆ U 
 i-]p ( ˆ z 9 it [-+
-]+1 | ˆ
z 9 it , τ it ; ϕ i enc ) anda decoder p ( a 9 it | s t , ˆ z 9 it ; ϕ i dec ) , parameterised by ϕ i enc and ϕ i dec , respectively:
[Equation]
where latent strategies ˆ z 9 it represents inferred policy parameters of other agents θ
9 it and τ i 0: t ={ τ i 0 , ..., τ it } denotes i ’s transitions up to timestep t .
We refer to Appendix F for a detailed ELBOderivation.
By optimizing the reconstruction term, the encoder aims to infer accurate latent strategiesof others.
Also, by imposing the prior through the KL divergence, where we set the prior to theprevious posterior with initial prior p ( ˆ z 9 i 0 ) =
N (0 , I ) , the inferred policies from the encoder areencouraged to be sequentially consistent across time (i.e., no abrupt changes in policies of others).
Reinforcement learning module.
This module aims to learn a policy that can maximize the agent’saverage reward based on the inferred information about other agents.
Each agent maintains itspolicy π ( ·| s, ˆ z 9 i ; θ i ) parameterized by θ i , two q -functions q i θ i ( s, ˆ z 9 i , a ; ψ i 1 ) and
q i θ i ( s, ˆ z 9 i , a ; ψ i 2 )parameterized by ψ i 1 , ψ i 2 , and learnable average reward ρ i θ i ∈ R .
We train the q-functions and ρ i θ i byminimizing the soft Bellman residual:
[Equation]
where β =1 , 2 , D i denotes i ’s replay buffer, and ¯ ψ i β denotes the target q -network parameters.
Thesoft value function v i θ i calculates the state value with the policy entropy H and entropy weight α :
[Equation]
Finally, the policy is trained to maximize:
[Equation]
Mean-Field FURTHER.
FURTHER provides a ﬂ exible framework such that it can easily integraterecent advances in multiagent learning.
For example, by reconstructing and predicting the meanaction and latent strategy of neighbor agents in Equation (11), we can incorporate the mean- ﬁ eldframework to improve performance in large-scale learning settings.
Appendix E details the mean- ﬁ eldversion of FURTHER with pseudocode.
[Figure]
Evaluation
We demonstrate FURTHER’s ef ﬁ cacy on a diverse suite of multiagent benchmark domains.
We referto Appendix G for experimental details and hyperparameters.
The code and video highlights areincluded in the supplementary material.
The mean and 95% con ﬁ dence interval computed for 20seeds are shown in each ﬁ gure.
Baselines.
We compare our method with the following baseline adaptation strategies:
• LILI [8]: An approach that considers the learning dynamics of other agents but suffers frommyopic evaluation bias by optimizing the discounted return objective.• MASAC [14]: An approach that extends SAC [26] to a multiagent learning setting by havingcentralized critics [12].
This baseline assumes other agents will have stationary policies in the futureand thus neglects their learning.
Question 1.
How do methods perform when playing against a q -learning agent?
We consider playing the iterated Bach or Stravinsky game (IBS; see Figure 3a).
This general-sumgame involves con ﬂ icting elements with two pure strategy Nash equilibria, where convergence to(B,B) and (S,S) equilibrium are more preferable from agent
i ’s and j ’s perspective, respectively.
Suppose agent i plays against a naive learner j , such as q -learner [32], whose initial q -values are setto prefer action (S).
In this experimental setting, it is ideal for agent i to change j ’s in ﬂ uence behaviorto select (B) such that they converge to i ’s optimal pure strategy Nash equilibrium of (B,B).
The average reward performance when an agent i , trained with either FURTHER or the baselinemethods, interacts with the q -learner j is shown in Figure 4a.
There are two notable observations.
First, the FURTHER agent i consistently converges to its optimal equilibrium of (B,B), while the
LILI agent often converges to the sub-optimal equilibrium of (S,S).
The FURTHER agent i learns toselect (B) while j selects (S), receives the worst rewards of zero, and waits until j ’s q -value for (S) isupdated to be lower than the q -value for (B).
With the limiting view, i learns that the waiting processis only temporary, and receiving the eventual rewards of 2 by converging to (B,B) is optimal.
Bycontrast, LILI suffers from myopic evaluation and shows decreased performance upon convergencebecause the agent prefers simply converging to the sub-optimal equilibrium rather than waitinginde ﬁ nitely.
Figure 5a also shows that LILI achieves sub-optimal performance for any value of γ andshows unstable learning as γ → 1 .
Second, FURTHER and LILI outperform the other approach of MASAC, showing the bene ﬁ t of considering the active in ﬂ uence on future policies of other agents.
Question 2.
Which equilibrium do methods converge to in a self-play setting?
We experiment with a self-play setting in which both agents learn with the same algorithm in aniterated cooperative (IC) game with identical payoffs (see Figure 3b).
This game has two pure strategy
Nash equilibria of (U,U) and (D,D), in which the (D,D) equilibrium Pareto dominates the other.
Figure 4b shows the average reward performance as the training iteration increases.
First we ﬁ nd that
LILI performs better than MASAC by considering the learning of agents.
However, similar to the
IBS results, we observe that FURTHER consistently converges to the best equilibrium of (D,D) whilethe baseline methods can converge to the sub-optimal equilibrium of (U,U) due to the myopic view.
[Figure]
e r a g
[Figure]
a r d f o r A g e n t i
IBS Comparison against Q-
[Figure]
Question 3.
How does FURTHER’s limiting optimization perform directly against baselines?
We consider FURTHER agent i directly competing against either LILI or MASAC opponent j inthe iterated matching pennies (IMP) game (see Figure 3c).
To show that FURTHER has a long-termperspective and thus can collect more rewards than the opposing method over time, we evaluate usinga metric of relative accumulated reward summed up to the current timestep:
P t r it − r jt .
Figure 4c 276 shows that the relative accumulated reward for FURTHER is positive for both settings, meaningthat FURTHER receives higher rewards than LILI and MASAC over time. This result suggeststhat FURTHER is more effective than LILI by employing the limiting view via the average rewardformulation.
This result also conveys that it is bene ﬁ cial to consider the underlying learning dynamicsrather than ignoring them because FURTHER can more easily exploit the MASAC opponent andachieve higher accumulated rewards than when competing against the LILI opponent.
Question 4.
How does FURTHER scale to a more complex environment?
To answer this question, we use the MuJoCo RoboSumo domain ([19]; see Figure 3d), where twoant robots compete with each other with the objective of pushing the opponent out of the ring.
Thereward function consists of a sparse reward of 5 for winning against the opponent and shaped rewardsof moving towards the opponent and pushing the opponent further from the center of the ring.
Thisenvironment has complex interactions because an agent must learn how to control its joints withcontinuous action space to move around the ring while learning to push the opponent.
Similarto the setup in Question 3, FURTHER agent i directly competes against either LILI or MASACopponent j .
We note that each agent has only partial observations about its opponent.
As such,an agent infers its opponent’s hidden policies and learning dynamics based on partial observations.
We show the RoboSumo results in Figure 5b.
Consistent with our results in the iterated matrixgames, we observe that FURTHER gains more rewards than the baselines over time and wins against
MASAC more often than against LILI.
The averaged winning rate across the entire interaction showsthat FURTHER wins against LILI and MASAC with 60.6% and 63.9%, respectively.
Therefore,
FURTHER provides a scalable framework that can learn policies in an environment with complexinteractions and continuous action space.
Question 5.
How does FURTHER scale to a large number of agents?
Finally, we show the scalability of our method regarding the number of agents using the battle domain([31]; see Figure 3e).
In this large-scale mixed cooperative-competitive setting, a red team of 25agents and a blue team of 25 agents interact in a gridworld, where each agent collaborates with itsteammates to eliminate the opponents.
Speci ﬁ cally, we compare when red and blue agents learn withthe
mean-
ﬁ eld version of FURTHER (i.e., FURTHER-MF) and LILI (i.e., LILI-MF), respectively,where they predict the mean actions of neighboring agents.
We note that all 50 agents learn in adecentralized manner without sharing parameters with one another.
It is evident that FURTHER-MFoutperforms LILI-MF, which shows the effectiveness of having the limiting perspective. This resultalso conveys that FURTHER can easily incorporate other techniques in multiagent learning and showimproved performance in large-scale settings.
[Figure]
Related Work
[Figure]
Stationary MARL.
The standard approach for addressing non-stationarity in MARL is to considerinformation about other agents and reason about joint action effects [33].
Example frameworksinclude centralized training with decentralized execution, which accounts for the actions of otheragents through a centralized critic [12–14, 16, 34, 35].
Other related approaches include opponentmodeling frameworks that infer opponent policies and condition an agent’s policy on this inferredinformation about others [36–39].
While this does alleviate non-stationarity, each agent learns itspolicy by assuming that other agents will follow the same policy into the future.
This assumptionis incorrect because other agents can have different behavior in the future due to their learning [6],resulting in instability with respect to their changing behavior.
In contrast, FURTHER models thelearning processes of other agents and considers how to actively in ﬂ uence limiting behavior.
Learning-aware MARL.
Our framework is closely related to prior work that considers the learningof other agents in the environment.
The framework by [40], for instance, learns the best responseadaptation to the other agent’s anticipated updated policy.
Notably, LOLA [6] and its more recentimprovements [7, 41] study the impact of behavior on one or a few of another agent’s policy updates.
Our work is also related to frameworks that leverage the inferred policy dynamics of other agentsto impact their future policies by maximizing the discounted return objective [8, 10, 42].
Lastly,meta-learning frameworks are related that directly account for the non-stationary policy dynamics inmultiagent settings based on the inner-loop and outer-loop optimization [9, 11, 19, 43].
However, allof these approaches only account for a ﬁ nite number of updates to the policies of other agents, so weobserve that these methods can converge to a less desirable solution.
FURTHER addresses this issueby optimizing for the average reward objective in the active Markov game setting.
Game-theoretic MARL.
Another effective approach to addressing the non-stationarity is learningequilibrium policies that correspond to game-theoretic solution concepts [4, 44–47].
These frameworks predict stationary joint action values by the end of learning and can guarantee convergence to
Nash [3] or correlated [20] equilibrium values under certain assumptions.
However, as noted in [48],this convergence is guaranteed only while ignoring the actual learning dynamics of other agents, andeach agent assumes all agents will play the same joint equilibrium strategy.
As such, equilibriumlearners can fail to learn best-response policies when others choose to play different equilibriumstrategies in the future as a result of their learning.
By contrast, FURTHER considers convergence toa recurrent set of joint policies by inferring the true policy dynamics of other agents.
Conclusion
In this paper, we have introduced FURTHER to address non-stationarity by considering each agent’simpact on the converged policies of other agents.
The key idea is to consider the limiting policiesof other agents through the average reward formulation for a newly proposed active Markov gameframework, and we have developed a practical model-free and decentralized approach in this setting.
We evaluated our method on various multiagent settings and showed that FURTHER consistentlyconverges to more desirable long-term behavior than state-of-the-art baseline approaches.