I2Q: A Fully Decentralized Q-Learning Algorithm
Anonymous Author(s)
AffiliationAddress email
Abstract
Fully decentralized multi-agent reinforcement learning has shown great potentialsfor many real-world cooperative tasks, where the global information, e.g. , the actions of other agents, is not accessible.
Although independent Q-learning is widelyused for decentralized training, the transition probabilities are non-stationary sinceother agents are updating policies simultaneously, which leads to non-guaranteedconvergence of independent Q-learning.
To deal with non-stationarity, we firstintroduce stationary ideal transition probabilities, on which independent Q-learningcould converge to the global optimum.
Further, we propose a fully decentralizedmethod, I2Q, which performs independent Q-learning on the modeled ideal transition function to reach the global optimum.
The modeling of ideal transitionfunction in I2Q is fully decentralized and independent from the learned policiesof other agents, helping I2Q be free from non-stationarity and learn the optimalpolicy.
Empirically, we show that I2Q can achieve remarkable improvement in avariety of cooperative multi-agent tasks.
Introduction
Multi-agent reinforcement learning (MARL) has shown great potentials in real-world applications,including UAV [19], IoT [3] and games [28].
A number of MARL methods have been proposed fortraining agents to cooperatively maximize the cumulative shared reward, most of which follow theparadigm of centralized training and decentralized execution (CTDE) where the information of allagents is collected and used in the training phase.
However, in many industrial applications whereagents may belong to different companies, e g. , autonomous vehicles or robots, the actions of otheragents may not be accessible, so that CTDE methods cannot work.
One way to address this challengeis fully decentralized learning , where the agents only use local experiences without the actions ofother agents in both training and execution.
Independent Q-learning [27, 26] is one of the most straightforward decentralized methods.
However,since other agents are treated as a part of the environment, from the perspective of individual agent,the transition probabilities that also depend on the policies of other agents will change as otheragents are updating their policies simultaneously [7].
Whether the agent updates the individual
Q-values using the off-policy experiences, which are stored in the decentralized replay buffer, orthe on-policy experiences, which are collected under the latest learned policies of all agents, thetransition probabilities are non-stationary, thus the convergence of independent Q-learning is nottheoretically guaranteed.
To tackle the non-stationarity problem, we propose a novel variant of independent Q-learning.
First,we introduce ideal transition probabilities for each agent, which is induced by the optimal conditionaljoint policy of other agents, conditioned on action of this agent.
We theoretically show that if allagents independently perform Q-learning on respective ideal transition probabilities, their policiesconverge to the optimal joint policy when there is only one optimal joint policy.
Certainly, suchideal transition probabilities are unknown in advance, but they can be deliberately learned by each agent in a decentralized way.
We let each agent learn the QSS-value [6], the value of state and nextstate , which will converge on its replay buffer and be equivalent with the optimal joint Q-value indeterministic environments.
Then, for each agent, the next state under ideal transition probabilities,given a state and an action, is the one with the highest QSS-value.
Therefore, each agent can modelan ideal transition function by learning QSS value and perform independent Q-learning on the idealtransition function, which guarantees the convergence to the optimal policy.
The proposed method, ideal independent Q-learning ( I2Q ), is fully decentralized, without theinformation of other agents.
Although the theoretical proof is built on deterministic environments,
I2Q can also be applied in stochastic environments.
We theoretically analyze the value gap andexperimentally demonstrate its effectiveness in stochastic environments.
We evaluate I2Q on a varietyof multi-agent cooperative tasks, i.e. , matrix games, MPE-based differential games [14], Multi-Agent
MuJoCo [5], and SMAC [22], covering fully and partially observable, deterministic and stochastic,discrete and continuous environments.
Empirically, I2Q outperforms baselines, verifying the analyzedconvergence and optimality, and the modeled ideal transition function.
Related Work
CTDE.
Most of recent MARL methods follow the paradigm of centralized training and decentralizedexecution (CTDE).
Policy-based methods [14, 8, 30, 34, 24, 17] extend policy-gradient into multiagent cases by designing different optimization objectives.
Value factorization methods [25, 21,23, 29, 20] decompose the joint value function into individual value functions according to the Individual-Global-Max condition.
In these methods, the information of all agents can be accessed ina centralized way during training, and the convergence can be guaranteed.
Unlike these methods, wefocus on decentralized learning where global information is not available.
DTDE.
When agents cannot obtain the global information, they have to adopt decentralized trainingand decentralized execution (DTDE).
The most straightforward decentralized methods are independent Q-learning [26] and independent PPO [4].
However, since all agents are updating policiessimultaneously, from the perspective of each agent, the transition probabilities are non-stationary, andthus the convergence of these methods may not be guaranteed.
Fingerprints [7] uses iteration numberand exploration rate to alleviate the problem of obsolete experiences in replay buffer for independent Q-learning.
Hysteretic-QL [15] and Lenient-QL [16] let the agents be optimistic and attach lessimportance to the value punishment.
However, none of these variants guarantees the convergenceof independent Q-learning.
Some theoretical methods [2, 35] guarantee the convergence to a Nashequilibrium, but the converged equilibrium may not be the optimal one when there are multipleequilibria [32].
Moreover, there methods cannot use replay buffer and have to re-collect experiencesonce the policies are updated, which is not practical.
Our I2Q is a practical method to learn theoptimal policy using replay buffer.
Some studies [33, 10] consider decentralized learning withcommunication.
However, in our fully decentralized settings, agents cannot share any information.
Method
We first introduce the problem setting and analyze the non-stationarity in independent Q-learning.
Then, we construct ideal transition probabilities, on which the agents are theoretically guaranteedto converge to the optimal joint policy via independent Q-learning.
Finally, we propose I2Q, whichperforms independent Q-learning on the modeled ideal transition function.
3.1 Preliminaries
There are N agents in multi-agent MDP M env = < S , O , A , R, P env , γ > with the state space S andthe joint action space A .
At each timestep, each agent i performs an individual action a i , and theenvironment transitions to the next state s ′ by taking the joint action a with the transition probabilities
P env ( s ′ | s, a ) .
For simplicity of theoretical analysis, we let all agents obtain the state s [12], thoughin practice each agent i may learn based on the observation o i ∈
O .
All agents get a [Table] [Table] [Table]
[Table]
[Table]
[Table]
since each agent only observes its own action a i instead of the joint action a .
From the perspective ofeach agent i , there is an MDP M i = < S ,
A i , R, P i , γ > with the individual action space
A i and the transition probabilities
[Equation]
where a − i and π − i respectively denote the joint action and the joint policy of all agents except agenti .
Each agent performs independent Q-learning:
[Equation]
In this paper, all Q values denote the optimal values Q ∗ , we refer to Q ∗ as Q for simplicity.
Accordingto (1), the transition probabilities depend on the policies of other agents.
Since other agents areupdating their policies continuously, P i becomes non-stationary during training.
Concretely, if eachagent i updates Q i in an off-policy way, using the experiences stored in the replay buffer D i , whichdoes not contain other agents’ actions, the transition probabilities P i in the replay buffer could be seenas the ones depending on the average policy ¯ π − i along the training process, which is non-stationaryand outdated [7].
If each agent learns in an on-policy way, only using the most recent experiences, P icould be seen as the one under the latest learned policy π − i , which are not outdated but still nonstationary.
The convergence of Q-learning on non-stationary transition probabilities is not guaranteed.
How to solve the non-stationarity?
In the next section, we introduce ideal transition probabilities andprove that if all agents perform independent Q-learning on the ideal transition probabilities, they willconverge to the optimal joint policy.
3.2 Ideal Transition Probabilities
Assume there is only one optimal joint policy π ∗ ( s )
= arg max a Q ( s, a ) , where Q is the optimaljoint Q-function,
[Equation]
The unique optimal policy π ∗ ( s ) must be deterministic [18].
Thus, π ∗ can be uniquely factorizedinto deterministic optimal individual policies π ∗ = ⟨ π ∗ 1 , π ∗ 2 , · · · , π ∗ N ⟩ .
For agent i , we build adeterministic optimal joint policy of other agents which is conditioned on a i :
[Equation]
There may exist multiple π ∗− i , and we can choose one arbitrarily.
If other agents act the policyπ ∗− i ( s, a i ) , the transition probabilities viewed by agent i is P i ( s ′ | s, a i ) =
P env
[Equation]
s ′ | s, a i , π ∗− i ( s, a i ) (cid:1) ,which are defined as ideal transition probabilities .
Then we have the following theorem.
Theorem 1.
If agent i performs Q-learning on the ideal transition probabilities P i ( s ′ | s, a i ) =
P env (cid:0) s ′
| s, a i , π ∗− i ( s, a i ) (cid:1) , its policy will converge to the optimal individual policy π ∗ i .
Proof.
According to the fixed-point formulation of Q-learning, we have
[Equation]
where (6) is from taking max a − i on both sides of (3), and (7) is by folding max a − i into P env .
From(5) and (7), we see Q i ( s, a i ) and max a − i Q ( s, a i , a − i ) have the same stationary Bellman operator.
According to the contraction mapping theorem, the Bellman operator converges to a unique fixedpoint.
Thus,
[Equation]
Then, we have:
[Equation]
By contradiction, if an ˜ a i ̸ = π ∗ i ( s ) but satisfies Q
i
( s, ˜ a i ) ≥ Q
i ( s, π ∗ i ( s ))
=
Q ( s, π ∗ ( s )) , it contradicts that π ∗ is the unique optimal joint policy.
Therefore, π ∗ i ( s ) =
arg max Q i ( s, a i ) .
By extending this theorem, if all agents perform Q-learning on such ideal transition probabilitiesunder the conditional optimal joint policy, they will converge to the respective optimal individualpolicies, and arrive at the optimal joint policy ⟨ π ∗ 1 , π ∗ 2 , · · · , π ∗ N ⟩ .
3.3 I2Q
How to obtain the ideal transition probabilities from the non-stationary replay buffer for each agent?
We first introduce the QSS-learning [6].
Following the setting of QSS [6], we assume the environmentis deterministic.
1 Each agent i learns a value function Q ss i ( s, s ′ ) using the experiences in its ownreplay buffer D i :
[Equation]
(9) where N ( s ′ ) is the neighboring state set of the state s ′ (the set of all next states of s ′ ).
The valuefunction Q ss i has several advantages.
First, it has been proven in QSS [6] that Q ss i and Q learnequivalent values, which means
[Equation]
The equivalence shows that agent i can independently infer the next state under the optimal jointpolicy according to arg max s ′ Q ss i ( s, s ′ ) .
Second, as Q ss i ( s, s ′ ) decouples the action a i from thevalue and the implied transition probabilities: P ( s ′ | s, s ′ )
= 1 , ∀ s ′ ∈
N ( s ) , are always stationary,
Q ss i converges to the optimal value even on its own replay buffer, without the information of otheragents.
Third, since all agents act in the same environment, they collect the same states set andthe same next state set in their own replay buffers.
Therefore, they will converge to the same Q ss iindependently, which builds the consensus between the learned policies of agents.
Then, how to build the ideal transition function from Q ss i ?
As the environment P env and the optimalconditional joint policy of other agents π ∗− i ( s, a i ) are deterministc, under the ideal transition probabilities
P env (cid:0) s ′
| s, a i , π ∗− i ( s, a i ) (cid:1) , the environment from the perspective of agent i will deterministicallytransition to the next state that has the highest Q ss i among all neighboring states given a i :
[Equation]
where N ( s, a i ) if the neighboring state set of s given a i .
This is proved in the proof of the followingtheorem.
Theorem 2.
In deterministic environments, all agents will converge to the optimal policies, if eachagent i performs Q-learning on the transition function s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) .
Proof.
The proof is given in Appendix A, which shows that s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ )
isthe ideal transition function.
To model the stationary ideal transition function s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) , we train a neuralnetwork f i ( s, a i ) to predict s ′∗ and update f i ( s, a i ) by maximizing:
[Equation]
The first term enforces that the predicted next state has the highest Q ss i , and the second term constrainsthe predicted next state to be in the set N ( s, a i ) .
The hyperparameter λ is the coefficient.
Based onthe transition model f i ( s, a i ) , Q ss i is updated by minimizing the TD-error:
[Equation]
And Q i is updated by minimizing:
[Equation]
Algorithm 1.
I2Q
¯ Q is the target network of Q .
When updating Q i (14), the target value is not computed on the nextstate sampled from replay buffer D i , but on the one predicted by the modeled ideal transition functionf i ( s, a i ) .
In (14), the reward r is simplified to depending just on s .
When r depends on both s and s ′ ,given s and a i , the next state under the ideal transition function (11) satisfies:
[Equation]
deterministic ideal transition probabilities )
[Equation]
Therefore, we can update Q i by minimizing:
[Equation]
The training procedure of I2Q is summarize in Algorithm 1, where each agent i learns f i , Q ss i ,and Q i .
Although all the modules update simultaneously, as the convergence of Q ss i is guaranteed,the transition function f
i derived from Q ss i will be stationary in the later stage.
Thus Q i will alsoconverge.
I2Q can be applied in environments with both discrete and continuous state-action space.
In continuous action space, we build I2Q on DDPG [13], where a policy network π i ( s ) is trainedby maximizing Q i ( s, π i ( s )) as a substitute of arg max a i Q i ( s, a i ) .
In continuous state space, sincef i ( s, a i ) is differentiable, (12) can be maximized by gradient ascent.
In discrete state space or largestate space, we can map the state space to a continuous embedding space, and apply I2Q on theembedding space.
3.4 Assumptions
We will further discuss the two assumptions.
The prime one is that there is only one optimal jointpolicy, but I2Q can easily solve tasks with multiple optimal joint policies.
With multiple optimalactions (with the max Q i ( s, a i ) ), if each agent arbitrarily selects one of the optimal independentactions, the joint action might not be the optimal.
To address this, we can set a performance toleranceε and introduce a fixed randomly initialized reward function ˆ r ( s, s ′ ) in the range (0 , ˆ r max ] , whereˆ r max =
(1 − γ ) ε .
Then all agents perform I2Q on the shaped reward r + ˆ r and learn the valuefunction ˆ Q i ( s, a i ) in terms of of r + ˆ r .
Since ˆ r is positive, ˆ Q i ( s, a i ) > Q i ( s, a i ) .
In ˆ Q i ( s, a i ) ,the maximal contribution from ˆ r is ˆ r max / (1 − γ ) =
ε , so the minimal contribution from r isˆ Q i ( s, a i ) − ε > Q
i
( s, a i ) − ε , which means that the maximal performance drop is ε when selectingactions according to ˆ Q i .
Moreover, since the reward function ˆ r ( s, s ′ ) is randomly initialized, it is asmall probability event to find multiple optimal joint policies on the reward function r + ˆ r .
Thus, if εis set to be small enough, I2Q could solve the task with multiple optimal joint policies.
The secondary one is deterministic environments.
In stochastic environments, (10) does not hold, and
Q ss i ( s, s ′ ) can be considered the “best possible value” [6].
So s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ )
isthe best possible transition function, rather than the ideal transition function.
The difference betweenthe learned Q i of I2Q and the true value under the ideal transition function Q stoi is guaranteed by thefollowing theorem.
, 𝑠 0 𝑟, 𝑠 11 𝑟, 𝑠 21 𝑟, 𝑠 |  | 1 𝑟 |  |𝐚𝐚𝐚 | 𝐴 | ⋯ , 𝑠 12 𝑟, 𝑠 22 𝑟, 𝑠 |  | 2 𝑟 |  |𝐚𝐚𝐚 | 𝐴 | ⋯ , 𝑠 13 𝑟, 𝑠 23 𝑟, 𝑠 |  | 3 𝑟 |  | [Figure] 𝐚
[Figure]
𝐚 | 𝐴 |
[Figure]
[Figure]
d
[Figure]
[Figure]
[Figure]
Theorem
[Equation]
[Table]
where ∆
[Table]
= r
[Table]
− r [Table]
Proof.
The proof is given in Appendix A.
Theorem 3 shows that Q i of I2Q is closer to the true value if the transition probability of s ′∗ is higherin stochastic environments.
Moreover, we empirically show that I2Q can also obtain remarkableperformance gain in stochastic tasks.
Experiments
In experiments, we first evaluate I2Q on a class of randomly generated matrix games to verify ourtheoretical analysis.
Second, we compare I2Q with a series of Q-learning variants on MPE-based [14]differential games to illustrate the convergence and optimality of I2Q on more complex tasks withcontinuous action.
Third, we test I2Q on two popular MARL benchmarks: Multi-Agent MuJoCo[17] and SMAC [22].
The experiments cover both fully and partially observable, deterministicand stochastic, discrete and continuous environments.
Since we consider the fully decentralizedsettings, I2Q and the baselines do not use parameter-sharing .
The results are presented using meanand standard deviation (std) with different random seeds.
More details about hyperparameters areavailable in Appendix C.
4.1 Matrix Games
To support the theoretical analysis of I2Q, we perform experiments on a class of two-agent matrixgames with L stages.
The action space of each agent is 3 , so the joint action space |A|
= 9 .
Thepayoff tree is shown in Figure 1, where the reward r is randomly generated in the range [ − 1 , 1] andfixed in each matrix game.
Without loss of generality, we randomly generate 100 matrix games foreach stage L , train the agents for four different seeds in each game, and plot the mean normalizedreturn (normalized by the optimal return) and std over the 100 matrix games.
Since the states arecountable, we adopt Q-tables instead of neural networks, where Q i table and Q ss i table are updatingsimultaneously , and use ϵ -greedy policies.
Figure 2 show the learning curves with different stages Lunder ϵ = 0 .
IQL cannot converge to the optimal policies and shows large std, because it is afflictedwith non-stationarity.
I2Q converges to the global optimum on the matrix games, even when the stage L grows, which confirms our theoretical analysis.
In Figure 3, we show the effect of exploration rate ϵ on the matrix games with L = 4 .
As ϵ increases,
I2Q converges faster, while the performance of IQL drops.
In conventional decentralized methodslike IQL, there is a dilemma of exploration and exploitation .
If ϵ is small, the weak exploration causeslow sample efficiency and slow learning.
When ϵ is large, all agents act more randomly.
From theperspective of individual agent, the transition probabilities will be much different from the real ones
[Figure]
Published as a conference paper at ICLR
[Figure]
Proposition 2.
Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
As introduced by Son et al.
(2019) and Wang et al.
(2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.
4 E XPERIMENTS
In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
The completeness of the
IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).
In particular, the second term of Eq.
(11) is the main difference between QPLEX and Qatten.
Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
The implementation details of these algorithms and experimental settings are deferred to Appendix B. We also conduct two ablation studies to study the inﬂuence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred to be discussed in Appendix E. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles over 6 random seeds.
4.1 M ATRIX G AMES
QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
The optimal joint strategy of these two games is to perform action A (1) simultaneously.
To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
As shown in Figure 2b, QPLEX, QTRAN, and WQMIX, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions.
These results are deferred to Appendix C. QTRAN 0(a) case
Published as a conference paper at ICLR
[Figure]
Proposition 2.
Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
As introduced by Son et al.
(2019) and Wang et al.
(2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.
4 E XPERIMENTS
In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
The completeness of the
IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).
In particular, the second term of Eq.
(11) is the main difference between QPLEX and Qatten.
Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
The implementation details of these algorithms and experimental settings are deferred to Appendix B. We also conduct two ablation studies to study the inﬂuence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred to be discussed in Appendix E. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles over 6 random seeds.
4.1 M ATRIX G AMES
QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
The optimal joint strategy of these two games is to perform action A (1) simultaneously.
To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
As shown in Figure 2b, QPLEX, QTRAN, and WQMIX, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions.
These results are deferred to Appendix C. QTRAN
[Figure]
under the learned policies of other agents, which causes bad performance.
That is the reason why
IQL drops when ϵ increases.
I2Q can avoid the dilemma, because Q ss i ( s, s ′ ) is independent fromagent policies and converges faster with larger ϵ as it sees more states.
Thus, I2Q can use large ϵ , e.g. ,0 . 5 or 0 .
7 , to promote sample efficiency.
Moreover, we test I2Q on two one-stage matrix games proposed in QTRAN [23] and QPLEX [29]as shown in Figure 4, which are special cases of Figure 1.
In case 1 , IQL converges to the localoptimum, and in case 2 , IQL does not converge.
Our I2Q can converge to the global optimum easily,in a fully decentralized way.
4.2 MPE
To investigate the effectiveness of I2Q in complex continuous environments with neural networkimplementation, we design a class of MPE-based differential games, where N agents can move inthe range [ − 1 , 1] .
In each timestep, agent i acts the action a i ∈ [ − 1 , 1] , and the position of agent i isupdated as x i
= clip ( x i + 0 .
1 × a i , − 1 , 1) ( i.e. , the updated position is clipped to [ − 1 , 1] ).
The stateis the position vector { x 1 , x 2 , · · · , x N } .
The reward function of each timestep is defined as
[Equation]
The visualization of reward function of two-agent case is shown in Figure 5a.
There is only oneglobal optimum ( l = 0 and r = 1 ) but infinite sub-optimum ( l = 0 .
8 and r = 0 . 3 ), and the regionwith r > 0 .
3 is very narrow and surrounded by the region with r = 0 .
So it is hard to learn theoptimal policies in a decentralized way.
The episode contains 100 timesteps, and the agents’ positionsare randomly initialized at the beginning of each episode.
To verify the scalability, we test I2Q in thesettings with different agent numbers N , and train the agents for eight random seeds in each setting.
The results are shown in Figure 6.
IDDPG always falls into local optimum due to the non-stationarityand outdated transition probabilities.
To be optimistic towards the value punishment is an importanttechnique in decentralized MARL.
Although adopting the optimistic update, Hysteretic IDDPG[15] still falls into local optimum due to the non-stationarity problem.
D3G [6] applies the ideaof QSS in single-agent learning.
To extend D3G into decentralized MARL, we train each agentindependently using D3G, without the information of other agents.
However, D3G cannot learn theoptimal solution either, since the forward model in D3G requires the transition probabilities P i tobe deterministic.
However, when learning on the decentralized replay buffer, P i is the average ofnon-stationary transition probabilities and cannot be deterministic.
Thus, D3G is not applicable todecentralized multi-agent learning.
Learning on the ideal transition function inferred from Q ss i ( s, s ′ ) ,
I2Q can escape from local optimum and significantly outperform others.
To thoroughly demonstratethe effectiveness of I2Q, in Figure 5, we visualize the learned values of agent 1 in the two-agent case.
[Figure] [Figure]
[Figure] [Figure]
In Figure 5b
[Figure]
we let the agent 2 always act the optimal policy (moving to the position x 2 = 0 )
[Figure]
whichmeans the agent 1 performs Q-learning on the true ideal transition probabilities.
Thus, Figure 5bshows the optimal values under ideal transition probabilities.
In Figure 5c, both agents use I2Q. I2Qvalues are very similar to the true ideal values, where the value of the center point ( l = 0 ) is thehighest, which means that the modeled transition function in I2Q is close to the true one.
In Theorem
(see Appendix A), we theoretically analyze that the difference between ideal values (Figure 5b) and
I2Q values (Figure 5c) is bounded by the model error.
In Figure 5d, both agents use IQL, resultingin that the value of center point ( l = 0 ) is the lowest and the suboptimal points ( l = 0 . 8 ) have thehighest values.
To model the ideal transition function, next states predicted by f i should have the highest Q ss i values(optimality) and be in the neighboring state set given state-action pair (referred to as neighborhoodconstraint).
To achieve this objective, λ in (12) controls the balance between optimality and neighborhood constraint.
Figure 7 shows the effect of λ .
When λ is too small, the next state predicted by f icannot maximize Q ss i ( s, s ′ ) , and thus I2Q gets stuck at local optimum.
Especially, when λ = 0 , I2Qdegrades into a popular model-based method MBPO [9].
When λ is too large, e.g. , λ = 10 , payingmuch more attention to maximize Q ss i ( s, s ′ ) , f i would generate out-of-neighborhood states, causingthat I2Q even cannot converge to local optimum.
We notice that λ
= 0 .
5 can generally achieve agood performance in different settings of N .
[Figure] [Figure]
Although we assume deterministic environments in theoretical analysis
[Figure]
I2Q can also obtain performance gain in stochastic environments.
To measure the impact of stochastic environments, we add anoise to the position update: x
i = clip ( x i + 0 .
1 × a i + ξ × z, − 1 , 1) , where z is a uniform randomvariable in [ − 1 , 1] , and ξ is a constant controlling the randomness.
Figure 8 shows the performancewith different ξ in the three-agent case.
In the stochastic environments, IDDPG still converges tolocal optimum, while I2Q always escapes from local optimum when ξ = 0 .
02 and 0 .
03 , free fromthe impact of stochastic environments.
When ξ = 0 . 04 and 0 .
05 , the effect of stochasticity is nearlyhalf of the effect of action.
I2Q cannot always learn optimal policies with the strong stochasticity butstill significantly outperforms IDDPG.
4.3 Multi-Agent MuJoCo and SMAC
To investigate the effectiveness of I2Q in partially observable environments, we perform experimentson Multi-Agent MuJoCo [17], where each agent independently controls one or some joints of therobot and could only observe the state of its own joints and bodies (with the parameter agent_obsk= 0).
The results are shown in Figure 9 with eight random seeds.
I2Q achieves higher rewardthan IDDPG, which indicates that I2Q could be extended to partially observable tasks and obtainperformance gain by performing Q-learning on ideal transition probabilities instead of transitionprobabilities in replay buffer.
In the partially observable setting, we only consider two-agent cases.
When there are more agents, each agent can only control one or two joints, and the observation rangeis too limited to learn strong policies.
In Appendix B.4, we provide the 6 -agent Walker and 8 -agent
Ant experiments with full observation setting to verify the scalability.
We test I2Q on partially observable and stochastic SMAC tasks [22] with the version SC2.4.10,including both easy and hard maps [31].
We adopt the implementation of PyMARL [22] that takes asinput the trajectory of partial observations, which is high-dimensional and not differentiable on somedimensions, meaning that directly optimizing (12) is hard.
So the QSS model Q ss i is not built on thetrajectory of partial observations, but on the hidden state of Q i (the output of RNN layer), which is64 -dimension and differentiable, and the transition function f
i predicts the next hidden state.
Wetrain the agents for four random seeds.
D3G can only be used in continuous action space, so we donot compare with it in SMAC.
As shown in Figure 10, I2Q is capable of handling high-dimensionaland stochastic tasks.
Conclusion
We propose I2Q, a novel variant of independent Q-learning, to deal with the non-stationarity in decentralized MARL.
I2Q models the ideal transition function, which depend on the optimal conditionaljoint policy of other agents and performs independent Q-learning on the modeled ideal transitionfunction.
Theoretically, we prove that I2Q converges to the optimal joint policy in deterministicenvironments.
Empirically, I2Q obtains performance gain in a variety of multi-agent tasks.