I2Q: A Fully Decentralized Q-Learning Algorithm
Abstract
Fully decentralized multi-agent reinforcement learning has shown great potential for many real-world cooperative tasks, where the global information, e.g. , the actions of other agents, is not accessible.
Although independent Q-learning is widely used for decentralized training, the transition probabilities are non-stationary since other agents are updating policies simultaneously, which leads to non-guaranteed convergence of independent Q-learning.
To deal with non-stationarity, we first introduce stationary ideal transition probabilities, on which independent Q-learning could converge to the global optimum.
Further, we propose a fully decentralized method, I2Q, which performs independent Q-learning on the modeled ideal transition function to reach the global optimum.
The modeling of ideal transition function in I2Q is fully decentralized and independent from the learned policies of other agents, helping I2Q be free from non-stationarity and learn the optimal policy.
Empirically, we show that I2Q can achieve remarkable improvement in a variety of cooperative multi-agent tasks.
Introduction
Multi-agent reinforcement learning (MARL) has shown great potential in real-world applications, including UAV [20], IoT [3] and games [29].
A number of MARL methods have been proposed for training agents to cooperatively maximize the cumulative shared reward, most of which follow the paradigm of centralized training and decentralized execution (CTDE), where the information of all agents is collected and used in the training phase.
However, in many industrial applications where agents may belong to different companies, e g. , autonomous vehicles or robots, the actions of other agents may not be accessible, so that CTDE methods cannot work.
One way to address this challenge is fully decentralized learning , where the agents only use local experiences without the actions of other agents in both training and execution.
Independent Q-learning [28, 27] is one of the most straightforward decentralized methods.
However, since other agents are treated as a part of the environment, from the perspective of an individual agent, the transition probabilities that also depend on the policies of other agents will change as other agents are updating their policies simultaneously [7].
Whether the agent updates the individual
Q-values using the off-policy experiences, which are stored in the decentralized replay buffer, or the on-policy experiences, which are collected under the latest learned policies of all agents, the transition probabilities are non-stationary, thus the convergence of independent Q-learning is not theoretically guaranteed.
To tackle the non-stationarity problem, we propose a novel variant of independent Q-learning.
First, we introduce ideal transition probabilities for each agent, which is induced by the optimal conditional joint policy of other agents, conditioned on action of this agent.
We theoretically show that if all agents independently perform Q-learning on respective ideal transition probabilities, their policies
[Table]
converge to the optimal joint policy when there is only one optimal joint policy.
Certainly, such ideal transition probabilities are unknown in advance, but they can be deliberately learned by each agent in a decentralized way.
We let each agent learn the QSS-value [6], the value of state and next state , which will converge on its replay buffer and be equivalent with the optimal joint Q-value in deterministic environments.
Then, for each agent, the next state under ideal transition probabilities, given a state and an action, is the one with the highest QSS-value.
Therefore, each agent can model an ideal transition function by learning QSS value and perform independent Q-learning on the ideal transition function, which guarantees the convergence to the optimal policy.
The proposed method, ideal independent Q-learning ( I2Q ), is fully decentralized, without the information of other agents.
Although the theoretical proof is built on deterministic environments,
I2Q can also be applied in stochastic environments.
We theoretically analyze the value gap and experimentally demonstrate its effectiveness in stochastic environments.
We evaluate I2Q on a variety of multi-agent cooperative tasks, i.e. , matrix games, MPE-based differential games [15], Multi-Agent
MuJoCo [5], and SMAC [23], covering fully and partially observable, deterministic and stochastic, discrete and continuous environments.
Empirically, I2Q outperforms baselines, verifying the analyzed convergence and optimality, and the modeled ideal transition function.
Related Work
CTDE.
Most recent MARL methods follow the paradigm of centralized training and decentralized execution (CTDE).
Policy-based methods [15, 8, 32, 37, 25, 18] extend policy-gradient into multiagent cases by designing different optimization objectives.
Value factorization methods [26, 22, 24, 31, 21] decompose the joint value function into individual value functions according to the
Individual-Global-Max condition.
In these methods, the information of all agents can be accessed in a centralized way during training, and the convergence can be guaranteed.
Unlike these methods, we focus on decentralized learning where global information is not available.
DTDE.
When agents cannot obtain global information, they have to adopt decentralized training and decentralized execution (DTDE).
The most straightforward decentralized methods are independent Qlearning [27] and independent PPO [4].
However, since all agents are updating policies simultaneously, from the perspective of each agent, the transition probabilities are non-stationary, and thus the convergence of these methods may not be guaranteed.
Fingerprints [7] uses iteration number and exploration rate to alleviate the problem of obsolete experiences in replay buffer for independent
Q-learning.
Hysteretic-QL [16] and Lenient-QL [17] let the agents be optimistic and attach less importance to the value punishment.
However, none of these variants guarantees the convergence of independent Q-learning.
Some theoretical methods [2, 38] guarantee the convergence to a Nash equilibrium, but the converged equilibrium may not be the optimal one when there are multiple equilibria [35].
Moreover, these methods cannot use replay buffer and have to re-collect experiences once the policies are updated, which is not practical.
Our I2Q is a practical method to learn the optimal policy using replay buffer.
Some studies [36, 11] consider decentralized learning with communication.
However, in our fully decentralized settings, agents cannot share any information.
Method
We first introduce the problem setting and analyze the non-stationarity in independent Q-learning.
Then, we construct ideal transition probabilities, on which the agents are theoretically guaranteed to converge to the optimal joint policy via independent Q-learning.
Finally, we propose I2Q, which performs independent Q-learning on the modeled ideal transition function.
3.1 Preliminaries
There are N agents in multi-agent MDP M env = < S , O , A , R, P env , γ > with the state space S and the joint action space A .
At each timestep, each agent i performs an individual action a i , and the environment transitions to the next state s ′ by taking the joint action a with the transition probabilities
P env ( s ′ | s, a ) .
For simplicity of theoretical analysis, we let all agents obtain the state s [13], though in practice each agent i may learn based on the observation o i ∈
O .
All agents get a shared reward r = R ( s, s ′ ) and learn to maximize the expected return E (cid:80) ∞ t =
γ t r t , where γ is the discount factor.
We consider the fully decentralized learning, where M env is partially observable to each agent since each agent only observes its own action a i instead of the joint action a .
From the perspective of each agent i , there is an MDP M i = < S ,
A i , R, P i , γ > with the individual action space
A i and the transition probabilities
[Equation]
where a − i and π − i respectively denote the joint action and the joint policy of all agents except agent i .
Each agent performs independent Q-learning:
[Equation]
In this paper, all Q values denote the optimal values Q ∗ , we refer to Q ∗ as Q for simplicity.
According to (1), the transition probabilities depend on the policies of other agents.
Since other agents are updating their policies continuously, P i becomes non-stationary during training.
Concretely, if each agent i updates Q i in an off-policy way, using the experiences stored in the replay buffer D i , which does not contain other agents’ actions, the transition probabilities P i in the replay buffer could be seen as the ones depending on the average policy ¯ π − i along the training process, which is non-stationary and outdated [7].
If each agent learns in an on-policy way, only using the most recent experiences,
P i could be seen as the one under the latest learned policy π − i , which is not outdated but still nonstationary.
The convergence of Q-learning on non-stationary transition probabilities is not guaranteed.
How to solve the non-stationarity?
In the next section, we introduce ideal transition probabilities and prove that if all agents perform independent Q-learning on the ideal transition probabilities, they will converge to the optimal joint policy.
3.2 Ideal Transition Probabilities
Assume there is only one optimal joint policy π ∗ ( s )
= arg max a Q ( s, a ) , where Q is the optimal joint Q-function,
[Equation]
The unique optimal policy π ∗ ( s ) must be deterministic [19].
Thus, π ∗ can be uniquely factorized into deterministic optimal individual policies π ∗ = ⟨ π ∗ 1 , π ∗ 2 , · · · , π ∗ N ⟩ .
For agent i , we build a deterministic optimal joint policy of other agents which is conditioned on a i :
[Equation]
There may exist multiple π ∗− i , and we can choose one arbitrarily.
If other agents act the policy π ∗− i ( s, a i ) , the transition probabilities viewed by agent i is P i ( s ′ | s, a i ) =
P env
[Equation]
s ′ | s, a i , π ∗− i ( s, a i ) (cid:1) , which are defined as ideal transition probabilities .
Then we have the following theorem.
Theorem 1.
If agent i performs Q-learning on the ideal transition probabilities P i ( s ′ | s, a i ) =
P env (cid:0) s ′
| s, a i , π ∗− i ( s, a i ) (cid:1) , its policy will converge to the optimal individual policy π ∗ i .
Proof.
According to the fixed-point formulation of Q-learning, we have
[Equation]
where (6) is from taking max a − i on both sides of (3), and (7) is by folding max a − i into P env .
From (5) and (7), we see Q i ( s, a i ) and max a − i Q ( s, a i , a − i ) have the same stationary Bellman operator.
According to the contraction mapping theorem, the Bellman operator converges to a unique fixed point.
Thus,
[Equation]
Then, we have:
[Equation]
By contradiction, if an ˜ a i ̸ = π ∗ i ( s ) but satisfies Q
i
( s, ˜ a i ) ≥ Q
i ( s, π ∗ i ( s ))
=
Q ( s, π ∗ ( s )) , it contradicts that π ∗ is the unique optimal joint policy.
Therefore, π ∗ i ( s ) =
arg max Q i ( s, a i ) .
By extending this theorem, if all agents perform Q-learning on such ideal transition probabilities under the conditional optimal joint policy, they will converge to the respective optimal individual policies, and arrive at the optimal joint policy ⟨ π ∗ 1 , π ∗ 2 , · · · , π ∗ N ⟩ .
3.3 I2Q
How to obtain the ideal transition probabilities from the non-stationary replay buffer for each agent?
We first introduce the QSS-learning [6].
Following the setting of QSS [6], we assume the environment is deterministic.
1 Each agent i learns a value function Q ss i ( s, s ′ ) using the experiences in its own replay buffer D i :
[Equation]
where N ( s ′ ) is the neighboring state set of the state s ′ (the set of all next states of s ′ ).
The value function Q ss i has several advantages.
First, it has been proven in QSS [6] that Q ss i and Q learn equivalent values, which means
[Equation]
The equivalence shows that agent i can independently infer the next state under the optimal joint policy according to arg max s ′ Q ss i ( s, s ′ ) .
Second, as Q ss i ( s, s ′ ) decouples the action a i from the value and the implied transition probabilities: P ( s ′ | s, s ′ )
= 1 , ∀ s ′ ∈
N ( s ) , are always stationary, Q ss i converges to the optimal value even on its own replay buffer, without the information of other agents.
Third, since all agents act in the same environment, they collect the same state set and the same next state set in their own replay buffers.
Therefore, they will converge to the same Q ss i independently, which builds the consensus between the learned policies of agents.
Then, how to build the ideal transition function from Q ss i ?
As the environment P env and the optimal conditional joint policy of other agents π ∗− i ( s, a i ) are deterministic, under the ideal transition probabilities P env (cid:0) s ′ | s, a i , π ∗− i ( s, a i ) (cid:1) , the environment from the perspective of agent i will deterministically transition to the next state that has the highest Q ss i among all neighboring states given a i :
[Equation]
where N ( s, a i ) is the neighboring state set of s given a i .
This is proved in the proof of the following theorem.
Theorem 2.
In deterministic environments, all agents will converge to the optimal policies, if each agent i performs Q-learning on the transition function s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) .
Proof.
The proof is given in Appendix A, which shows that s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) is the ideal transition function.
To model the stationary ideal transition function s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) , we train a neural network f i ( s, a i ) to predict s ′∗ and update f i ( s, a i ) by maximizing:
[Equation]
The first term enforces that the predicted next state has the highest Q ss i , and the second term constrains the predicted next state to be in the set N ( s, a i ) .
The hyperparameter λ is the coefficient.
Based on the transition model f i ( s, a i ) , Q ss i is updated by minimizing the TD-error:
[Equation] [Equation]
Algorithm 1.
I2Q for each agent
i
And Q i is updated by minimizing:
[Equation]
¯ Q is the target network of Q .
When updating Q i (14), the target value is not computed on the next state sampled from replay buffer D i , but on the one predicted by the modeled ideal transition function
f i ( s, a i ) .
In (14), the reward r is simplified to depending just on s .
When r depends on both s and s ′ , given s and a i , the next state under the ideal transition function (11) satisfies:
[Equation]
Therefore, we can update Q i by minimizing:
[Equation]
The training procedure of I2Q is summarize in Algorithm 1, where each agent i learns f i , Q ss i , and Q i .
Although all the modules update simultaneously, as the convergence of Q ss i is guaranteed, the transition function f
i derived from Q ss i will be stationary in the later stage.
Thus Q i will also converge.
I2Q can be applied in environments with both discrete and continuous state-action space.
In continuous action space, we build I2Q on DDPG [14], where a policy network π i ( s ) is trained by maximizing Q i ( s, π i ( s )) as a substitute of arg max a i Q i ( s, a i ) .
In continuous state space, since f i ( s, a i ) is differentiable, (12) can be maximized by gradient ascent.
In discrete state space or large state space, we can map the state space to a continuous embedding space, and apply I2Q on the embedding space.
We also provide an implementation without forward model
f i in Appendix B.7.
3.4 Assumptions
We will further discuss the two assumptions.
The prime one is that there is only one optimal joint policy, but I2Q can easily solve tasks with multiple optimal joint policies.
With multiple optimal actions (with the max Q i ( s, a i ) ), if each agent arbitrarily selects one of the optimal independent actions, the joint action might not be optimal.
To address this, we can set a performance tolerance ε and introduce a fixed randomly initialized reward function ˆ r ( s, s ′ ) in the range (0 , ˆ r max ] , where ˆ r max = (1 − γ ) ε .
Then all agents perform I2Q on the shaped reward r + ˆ r and learn the value function ˆ Q i ( s, a i ) in terms of r + ˆ r .
Since ˆ r is positive, ˆ Q i ( s, a i ) > Q i ( s, a i ) .
In ˆ Q i ( s, a i ) , the maximal contribution from ˆ r is ˆ r max / (1 − γ ) =
ε , so the minimal contribution from r is ˆ
Q i ( s, a i ) − ε > Q i ( s, a i ) − ε , which means that the maximal performance drop is ε when selecting actions according to ˆ Q i .
Moreover, since the reward function ˆ r ( s, s ′ ) is randomly initialized, it is a small probability event to find multiple optimal joint policies on the reward function r + ˆ r .
Thus, if ε is set to be small enough, I2Q could solve the task with multiple optimal joint policies.
The secondary one is deterministic environments.
In stochastic environments, (10) does not hold, and
Q ss i ( s, s ′ ) can be considered the “best possible value” [6].
So s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) is
, 𝑠 0 𝑟, 𝑠 11 𝑟, 𝑠 21 𝑟, 𝑠 |  | 1 𝑟 |  |𝐚𝐚𝐚 | 𝐴 | ⋯ , 𝑠 12 𝑟, 𝑠 22 𝑟, 𝑠 |  | 2 𝑟 |  |𝐚𝐚𝐚 | 𝐴 | ⋯ , 𝑠 13 𝑟, 𝑠 23 𝑟, 𝑠 |  | 3 𝑟 |  |𝐚
[Figure]
𝐚𝐚 | 𝐴 |
[Figure]
[Figure]
d
[Figure] [Figure]
[Figure] [Figure]
the best possible transition function, rather than the ideal transition function.
The difference between the learned Q i of I2Q and the true value under the ideal transition function Q stoi is guaranteed by the following theorem.
Theorem 3.
[Equation]
Q
[Equation]
, where ∆ r = r
[Equation]
− r
[Equation]
Proof.
The proof is given in Appendix A.
Theorem 3 shows that Q i of I2Q is closer to the true value if the transition probability of s ′∗ is higher in stochastic environments.
Since we sample transitions from D i to update f i by maximizing (12), the second term of (12) makes the predicted next state
f i ( s, a i ) be close to the high-frequency next states in replay buffer D i given s and a i , which means
that the transition probability of the predicted next state P env (cid:0) f i ( s, a i ) | s, a i , π ∗− i ( s, a i ) (cid:1) would not be too small.
Thus, Q i of I2Q will be close to the true value and the worst cases where the predicted next states have very small transition probabilities can be avoided.
That is the reason why I2Q can be successfully applied in stochastic environments
Experiments
In experiments, we first evaluate I2Q on a class of randomly generated matrix games to verify our theoretical analysis.
Second, we compare I2Q with a series of Q-learning variants on MPE-based [15] differential games to illustrate the convergence and optimality of I2Q on more complex tasks with continuous action.
Third, we test I2Q on two popular MARL benchmarks: Multi-Agent MuJoCo [18] and SMAC [23].
The experiments cover both fully and partially observable, deterministic and stochastic, discrete and continuous environments.
Since we consider the fully decentralized settings, I2Q and the baselines do not use parameter-sharing .
The results are presented using mean and standard deviation (std) with different random seeds.
More details about hyperparameters are available in Appendix C.
4.1 Matrix Games
To support the theoretical analysis of I2Q, we perform experiments on a class of two-agent matrix games with L stages.
The action space of each agent is 3 , so the joint action space |A|
= 9 .
The payoff tree is shown in Figure 1, where the reward r is randomly generated in the range [ − 1 , 1] and fixed in each matrix game.
Without loss of generality, we randomly generate 100 matrix games for each stage L , train the agents for four different seeds in each game, and plot the mean normalized return (normalized by the optimal return) and std over the 100 matrix games.
Since the states are countable, we adopt Q-tables instead of neural networks, where Q i table and Q ss i table are updating simultaneously , and use ϵ -greedy policies.
Figure 2 shows the learning curves with different stages L under ϵ = 0 .
IQL cannot converge to the optimal policies and shows large std, because it is afflicted
[Figure]
z
[Figure]
e
[Figure]
Published as a conference paper at ICLR
[Figure]
Proposition 2.
Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
As introduced by Son et al.
(2019) and Wang et al.
(2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.
4 E XPERIMENTS
In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
The completeness of the
IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).
In particular, the second term of Eq.
(11) is the main difference between QPLEX and Qatten.
Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
The implementation details of these algorithms and experimental settings are deferred to Appendix B. We also conduct two ablation studies to study the inﬂuence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred to be discussed in Appendix E. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles over 6 random seeds.
4.1 M ATRIX G AMES
QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
The optimal joint strategy of these two games is to perform action A (1) simultaneously.
To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
As shown in Figure 2b, QPLEX, QTRAN, and WQMIX, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions.
These results are deferred to Appendix C. QTRAN
[Figure]
(a) case
Published as a conference paper at ICLR
[Figure] [Figure]
Proposition 2.
Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
As introduced by Son et al.
(2019) and Wang et al.
(2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.
4 E XPERIMENTS
In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
The completeness of the
IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).
In particular, the second term of Eq.
(11) is the main difference between QPLEX and Qatten.
Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
The implementation details of these algorithms and experimental settings are deferred to Appendix B. We also conduct two ablation studies to study the inﬂuence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred to be discussed in Appendix E. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles over 6 random seeds.
4.1 M ATRIX G AMES
QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
The optimal joint strategy of these two games is to perform action A (1) simultaneously.
To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
As shown in Figure 2b, QPLEX, QTRAN, and WQMIX, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions.
These results are deferred to Appendix C. QTRAN
[Figure]
with non-stationarity.
I2Q converges to the global optimum on the matrix games, even when the stage
L grows, which confirms our theoretical analysis.
In Figure 3, we show the effect of exploration rate ϵ on the matrix games with L = 4 .
As ϵ increases,
I2Q converges faster, while the performance of IQL drops.
In conventional decentralized methods like IQL, there is a dilemma of exploration and exploitation .
If ϵ is small, the weak exploration causes low sample efficiency and slow learning.
When ϵ is large, all agents act more randomly.
From the perspective of an individual agent, the transition probabilities will be much different from the real ones under the learned policies of other agents, which causes bad performance.
That is the reason why IQL drops when ϵ increases.
I2Q can avoid the dilemma, because Q ss i ( s, s ′ ) is independent from agent policies and converges faster with larger ϵ as it sees more states.
Thus, I2Q can use large ϵ , e.g. , 0 . 5 or 0 .
7 , to promote sample efficiency.
Moreover, we test I2Q on two one-stage matrix games proposed in QTRAN [24] and QPLEX [31] as shown in Figure 4, which are special cases of Figure 1.
In case 1 , IQL converges to the local optimum, and in case 2 , IQL does not converge.
Our I2Q can converge to the global optimum easily, in a fully decentralized way.
4.2 MPE
To investigate the effectiveness of I2Q in complex continuous environments with neural network implementation, we design a class of MPE-based differential games, where N agents can move in the range [ − 1 , 1] .
In each timestep, agent i acts the action a i ∈ [ − 1 , 1] , and the position of agent i is updated as x i = clip ( x i + 0 .
1 × a i , − 1 , 1) ( i.e. , the updated position is clipped to [ − 1 , 1] ).
The state is the position vector { x 1 , x 2 , · · · , x N } .
The reward function of each timestep is defined as
[Equation]
The visualization of reward function of two-agent case is shown in Figure 5a.
There is only one global optimum ( l = 0 and r = 1 ) but infinite sub-optima ( l = 0 .
8 and r = 0 . 3 ), and the region with r > 0 .
3 is very narrow and surrounded by the region with r = 0 .
So it is hard to learn the optimal policies in a decentralized way.
The episode contains 100 timesteps, and the agents’ positions are randomly initialized at the beginning of each episode.
To verify the scalability, we test I2Q in the settings with different agent numbers N , and train the agents for eight random seeds in each setting.
The results are shown in Figure 6.
IDDPG always falls into local optimum due to the non-stationarity and outdated transition probabilities.
To be optimistic towards the value punishment is an important technique in decentralized MARL.
Although adopting the optimistic update, Hysteretic IDDPG [16] still falls into local optimum due to the non-stationarity problem.
D3G [6] applies the idea
[Figure] [Figure]
[Figure] [Figure]
of QSS
in single-agent learning.
To extend D3G into decentralized MARL, we train each agent independently using D3G, without the information of other agents.
However, D3G cannot learn the optimal solution either, since the forward model in D3G requires the transition probabilities P i to be deterministic.
However, when learning on the decentralized replay buffer, P i is the average of non-stationary transition probabilities and cannot be deterministic.
Thus, D3G is not applicable to decentralized multi-agent learning.
Learning on the ideal transition function inferred from Q ss i ( s, s ′ ) ,
I2Q can escape from local optimum and significantly outperform others.
To thoroughly demonstrate the effectiveness of I2Q, in Figure 5, we visualize the learned values of agent 1 in the two-agent case.
In Figure 5b, we let the agent 2 always act the optimal policy (moving to the position x 2 = 0 ), which means the agent 1 performs Q-learning on the true ideal transition probabilities.
Thus, Figure 5b shows the optimal values under ideal transition probabilities.
In Figure 5c, both agents use I2Q. I2Q values are very similar to the true ideal values, where the value of the center point ( l = 0 ) is the highest, which means that the modeled transition function in I2Q is close to the true one.
In Theorem 4 (see Appendix A), we theoretically analyze that the difference between ideal values (Figure 5b) and
I2Q values (Figure 5c) is bounded by the model error.
In Figure 5d, both agents use IQL, resulting in that the value of the center point ( l = 0 ) is the lowest and the suboptimal points ( l = 0 . 8 ) have the highest values.
[Figure] [Figure]
To model the ideal transition function, next states predicted by f i should have the highest Q ss i values (optimality) and be in the neighboring state set given state-action pair (referred to as neighborhood constraint).
To achieve this objective, λ in (12) controls the balance between optimality and neighborhood constraint.
Figure 7 shows the effect of λ .
When λ is too small, the next state predicted by f i cannot maximize Q ss i ( s, s ′ ) , and thus I2Q gets stuck at local optimum.
Especially, when λ = 0 , I2Q degrades into a popular model-based method MBPO [9].
When λ is too large, e.g. , λ = 10 , paying much more attention to maximize Q ss i ( s, s ′ ) , f i would generate out-of-neighborhood states, causing that I2Q even cannot converge to local optimum.
We notice that λ
= 0 .
5 can generally achieve a good performance in different settings of N .
Although we assume deterministic environments in theoretical analysis, I2Q can also obtain performance gain in stochastic environments.
To measure the impact of stochastic environments, we add a noise to the position update: x i = clip ( x i + 0 .
1 × a i + ξ × z, − 1 , 1) , where z is a uniform random variable in [ − 1 , 1] , and ξ is a constant controlling the randomness.
Figure 8 shows the performance with different ξ in the three-agent case.
In the stochastic environments, IDDPG still converges to local optimum, while I2Q always escapes from local optimum when ξ = 0 .
02 and 0 .
03 , free from the impact of stochastic environments.
When ξ = 0 . 04 and 0 .
05 , the effect of stochasticity is nearly half of the effect of action.
I2Q cannot always learn optimal policies with the strong stochasticity but still significantly outperforms IDDPG.
4.3 Multi-Agent MuJoCo and SMAC
To investigate the effectiveness of I2Q in partially observable environments, we perform experiments on Multi-Agent MuJoCo [18], where each agent independently controls one or some joints of the robot and could only observe the state of its own joints and bodies (with the parameter agent_obsk = 0).
The results are shown in Figure 9 with eight random seeds.
I2Q achieves higher rewards than IDDPG, which indicates that I2Q could be extended to partially observable tasks and obtain performance gain by performing Q-learning on ideal transition probabilities instead of transition probabilities in replay buffer.
In the partially observable setting, we only consider two-agent cases.
When there are more agents, each agent can only control one or two joints, and the observation range is too limited to learn strong policies.
In Appendix B.4, we provide the 6 -agent Walker and 8 -agent
Ant experiments with the full observation setting to verify the scalability.
We test I2Q on partially observable and stochastic SMAC tasks [23] with the version SC2.4.10, including both easy and hard maps [33].
We adopt the implementation of PyMARL [23] that takes as input the trajectory of partial observations, which is high-dimensional and not differentiable on some dimensions, meaning that directly optimizing (12) is hard.
So the QSS model Q ss i is not built on the trajectory of partial observations, but on the hidden state of Q i (the output of RNN layer), which is 64 -dimension and differentiable, and the transition function f
i predicts the next hidden state.
We train the agents for four random seeds.
D3G can only be used in continuous action space, so we do not compare with it in SMAC.
As shown in Figure 10, I2Q is capable of handling high-dimensional and stochastic tasks.
Closing Remarks
In this paper, we propose I2Q, a novel variant of independent Q-learning, to deal with the nonstationarity in decentralized cooperative MARL.
I2Q models the ideal transition function, which depends on the optimal conditional joint policy of other agents and performs independent Q-learning on the modeled ideal transition function.
Theoretically, we prove that I2Q converges to the optimal joint policy in deterministic environments.
Empirically, I2Q obtains performance gain in a variety of multi-agent tasks.
We focus on the base algorithm for fully decentralized learning in cooperative MARL, as it lay the algorithmic foundation for decentralized learning.
Although we did not consider explicit coordination among agents, in partially observable environments coordination such as communication is desirable to improve empirical performance.
We envision that more methods can be further built on the base algorithm.
Moreover, currently, although fully decentralized learning algorithms may not empirically perform as well as CTDE methods like QMIX [22] in some benchmarks, e.g., hard maps in SMAC, we believe more efforts should be made by the MARL community on decentralized learning, considering its abundant benefits over CTDE.
First, fully decentralized algorithms do not require a centralized learner during training, hence they are more applicable and cleaner, easier to implement [10].
Second, fully decentralized algorithms are more versatile as the individual learners are indifferent to the number of other agents, thus they have better scalability [34].
Third, fully decentralized algorithms make agents more robust to the presence of agents they were not trained with (such as humans)
Therefore, fully decentralized learning is a fundamental problem in cooperative MARL, but remains to be investigated.
Acknowledgments and Disclosure of Funding
This work was supported by NSF China under grant 61872009.
The authors would like to thank the anonymous reviewers for their valuable comments.