I2Q: A Fully Decentralized Q-Learning Algorithm
Abstract
Fully decentralized multi-agent reinforcement learning has shown great potential for many real-world cooperative tasks, where the global information, e.g. , the actions of other agents, is not accessible.
Although independent Q-learning is widely used for decentralized training, the transition probabilities are non-stationary since other agents are updating policies simultaneously, which leads to non-guaranteed convergence of independent Q-learning.
To deal with non-stationarity, we first introduce stationary ideal transition probabilities, on which independent Q-learning could converge to the global optimum.
Further, we propose a fully decentralized method, I2Q, which performs independent Q-learning on the modeled ideal transition function to reach the global optimum.
The modeling of ideal transition function in I2Q is fully decentralized and independent from the learned policies of other agents, helping I2Q be free from non-stationarity and learn the optimal policy.
Empirically, we show that I2Q can achieve remarkable improvement in a variety of cooperative multi-agent tasks.
Introduction
Multi-agent reinforcement learning (MARL) has shown great potential in real-world applications, including UAV [20], IoT [3] and games [29].
A number of MARL methods have been proposed for training agents to cooperatively maximize the cumulative shared reward, most of which follow the paradigm of centralized training and decentralized execution (CTDE), where the information of all agents is collected and used in the training phase.
However, in many industrial applications where agents may belong to different companies, e g. , autonomous vehicles or robots, the actions of other agents may not be accessible, so that CTDE methods cannot work.
One way to address this challenge is fully decentralized learning , where the agents only use local experiences without the actions of other agents in both training and execution.
Independent Q-learning [28, 27] is one of the most straightforward decentralized methods.
However, since other agents are treated as a part of the environment, from the perspective of an individual agent, the transition probabilities that also depend on the policies of other agents will change as other agents are updating their policies simultaneously [7].
Whether the agent updates the individual
Q-values using the off-policy experiences, which are stored in the decentralized replay buffer, or the on-policy experiences, which are collected under the latest learned policies of all agents, the transition probabilities are non-stationary, thus the convergence of independent Q-learning is not theoretically guaranteed.
To tackle the non-stationarity problem, we propose a novel variant of independent Q-learning.
First, we introduce ideal transition probabilities for each agent, which is induced by the optimal conditional joint policy of other agents, conditioned on action of this agent.
We theoretically show that if all agents independently perform Q-learning on respective ideal transition probabilities, their policies
[Table]
converge to the optimal joint policy when there is only one optimal joint policy.
Certainly, such ideal transition probabilities are unknown in advance, but they can be deliberately learned by each agent in a decentralized way.
We let each agent learn the QSS-value [6], the value of state and next state , which will converge on its replay buffer and be equivalent with the optimal joint Q-value in deterministic environments.
Then, for each agent, the next state under ideal transition probabilities, given a state and an action, is the one with the highest QSS-value.
Therefore, each agent can model an ideal transition function by learning QSS value and perform independent Q-learning on the ideal transition function, which guarantees the convergence to the optimal policy.
The proposed method, ideal independent Q-learning ( I2Q ), is fully decentralized, without the information of other agents.
Although the theoretical proof is built on deterministic environments,
I2Q can also be applied in stochastic environments.
We theoretically analyze the value gap and experimentally demonstrate its effectiveness in stochastic environments.
We evaluate I2Q on a variety of multi-agent cooperative tasks, i.e. , matrix games, MPE-based differential games [15], Multi-Agent
MuJoCo [5], and SMAC [23], covering fully and partially observable, deterministic and stochastic, discrete and continuous environments.
Empirically, I2Q outperforms baselines, verifying the analyzed convergence and optimality, and the modeled ideal transition function.
Related Work
CTDE.
Most recent MARL methods follow the paradigm of centralized training and decentralized execution (CTDE).
Policy-based methods [15, 8, 32, 37, 25, 18] extend policy-gradient into multiagent cases by designing different optimization objectives.
Value factorization methods [26, 22, 24, 31, 21] decompose the joint value function into individual value functions according to the
Individual-Global-Max condition.
In these methods, the information of all agents can be accessed in a centralized way during training, and the convergence can be guaranteed.
Unlike these methods, we focus on decentralized learning where global information is not available.
DTDE.
When agents cannot obtain global information, they have to adopt decentralized training and decentralized execution (DTDE).
The most straightforward decentralized methods are independent Qlearning [27] and independent PPO [4].
However, since all agents are updating policies simultaneously, from the perspective of each agent, the transition probabilities are non-stationary, and thus the convergence of these methods may not be guaranteed.
Fingerprints [7] uses iteration number and exploration rate to alleviate the problem of obsolete experiences in replay buffer for independent
Q-learning.
Hysteretic-QL [16] and Lenient-QL [17] let the agents be optimistic and attach less importance to the value punishment.
However, none of these variants guarantees the convergence of independent Q-learning.
Some theoretical methods [2, 38] guarantee the convergence to a Nash equilibrium, but the converged equilibrium may not be the optimal one when there are multiple equilibria [35].
Moreover, these methods cannot use replay buffer and have to re-collect experiences once the policies are updated, which is not practical.
Our I2Q is a practical method to learn the optimal policy using replay buffer.
Some studies [36, 11] consider decentralized learning with communication.
However, in our fully decentralized settings, agents cannot share any information.
Method
We first introduce the problem setting and analyze the non-stationarity in independent Q-learning.
Then, we construct ideal transition probabilities, on which the agents are theoretically guaranteed to converge to the optimal joint policy via independent Q-learning.
Finally, we propose I2Q, which performs independent Q-learning on the modeled ideal transition function.
3.1 Preliminaries
There are N agents in multi-agent MDP M env = < S , O , A , R, P env , γ > with the state space S and the joint action space A .
At each timestep, each agent i performs an individual action a i , and the environment transitions to the next state s ′ by taking the joint action a with the transition probabilities
P env ( s ′ | s, a ) .
For simplicity of theoretical analysis, we let all agents obtain the state s [13], though in practice each agent i may learn based on the observation o i ∈
O .
All agents get a shared reward r = R ( s, s ′ ) and learn to maximize the expected return E (cid:80) ∞ t =
γ t r t , where γ is the discount factor.
We consider the fully decentralized learning, where M env is partially observable to each agent since each agent only observes its own action a i instead of the joint action a .
From the perspective of each agent i , there is an MDP M i = < S ,
A i , R, P i , γ > with the individual action space
A i and the transition probabilities
[Equation]
where a − i and π − i respectively denote the joint action and the joint policy of all agents except agent i .
Each agent performs independent Q-learning:
[Equation]
In this paper, all Q values denote the optimal values Q ∗ , we refer to Q ∗ as Q for simplicity.
According to (1), the transition probabilities depend on the policies of other agents.
Since other agents are updating their policies continuously, P i becomes non-stationary during training.
Concretely, if each agent i updates Q i in an off-policy way, using the experiences stored in the replay buffer D i , which does not contain other agents’ actions, the transition probabilities P i in the replay buffer could be seen as the ones depending on the average policy ¯ π − i along the training process, which is non-stationary and outdated [7].
If each agent learns in an on-policy way, only using the most recent experiences, P i could be seen as the one under the latest learned policy π − i , which is not outdated but still nonstationary.
The convergence of Q-learning on non-stationary transition probabilities is not guaranteed.
How to solve the non-stationarity?
In the next section, we introduce ideal transition probabilities and prove that if all agents perform independent Q-learning on the ideal transition probabilities, they will converge to the optimal joint policy.
3.2 Ideal Transition Probabilities
Assume there is only one optimal joint policy π ∗ ( s )
= arg max a Q ( s, a ) , where Q is the optimal joint Q-function,
[Equation]
The unique optimal policy π ∗ ( s ) must be deterministic [19].
Thus, π ∗ can be uniquely factorized into deterministic optimal individual policies π ∗ = ⟨ π ∗ 1 , π ∗ 2 , · · · , π ∗ N ⟩ .
For agent i , we build a deterministic optimal joint policy of other agents which is conditioned on a i :
[Equation]
There may exist multiple π ∗− i , and we can choose one arbitrarily.
If other agents act the policy π ∗− i ( s, a i ) , the transition probabilities viewed by agent i is P i ( s ′ | s, a i ) =
P env [Equation]
s ′ | s, a i , π ∗− i ( s, a i ) (cid:1) , which are defined as ideal transition probabilities .
Then we have the following theorem.
Theorem 1.
If agent i performs Q-learning on the ideal transition probabilities P i ( s ′ | s, a i ) =
P env (cid:0) s ′
| s, a i , π ∗− i ( s, a i ) (cid:1) , its policy will converge to the optimal individual policy π ∗ i .
Proof.
According to the fixed-point formulation of Q-learning, we have
[Equation]
where (6) is from taking max a − i on both sides of (3), and (7) is by folding max a − i into P env .
From (5) and (7), we see Q i ( s, a i ) and max a − i Q ( s, a i , a − i ) have the same stationary Bellman operator.
According to the contraction mapping theorem, the Bellman operator converges to a unique fixed point.
Thus,
[Equation]
Then, we have:
[Equation]
By contradiction, if an ˜ a i ̸ = π ∗ i ( s ) but satisfies Q
i
( s, ˜ a i ) ≥ Q
i ( s, π ∗ i ( s ))
=
Q ( s, π ∗ ( s )) , it contradicts that π ∗ is the unique optimal joint policy.
Therefore, π ∗ i ( s ) =
arg max Q i ( s, a i ) .
By extending this theorem, if all agents perform Q-learning on such ideal transition probabilities under the conditional optimal joint policy, they will converge to the respective optimal individual policies, and arrive at the optimal joint policy ⟨ π ∗ 1 , π ∗ 2 , · · · , π ∗ N ⟩ .
3.3 I2Q
How to obtain the ideal transition probabilities from the non-stationary replay buffer for each agent?
We first introduce the QSS-learning [6].
Following the setting of QSS [6], we assume the environment is deterministic.
1 Each agent i learns a value function Q ss i ( s, s ′ ) using the experiences in its own replay buffer D i :
[Equation]
where N ( s ′ ) is the neighboring state set of the state s ′ (the set of all next states of s ′ ).
The value function Q ss i has several advantages.
First, it has been proven in QSS [6] that Q ss i and Q learn equivalent values, which means
[Equation]
The equivalence shows that agent i can independently infer the next state under the optimal joint policy according to arg max s ′ Q ss i ( s, s ′ ) .
Second, as Q ss i ( s, s ′ ) decouples the action a i from the value and the implied transition probabilities: P ( s ′ | s, s ′ )
= 1 , ∀ s ′ ∈
N ( s ) , are always stationary, Q ss i converges to the optimal value even on its own replay buffer, without the information of other agents.
Third, since all agents act in the same environment, they collect the same state set and the same next state set in their own replay buffers.
Therefore, they will converge to the same Q ss i independently, which builds the consensus between the learned policies of agents.
Then, how to build the ideal transition function from Q ss i ?
As the environment P env and the optimal conditional joint policy of other agents π ∗− i ( s, a i ) are deterministic, under the ideal transition probabilities P env (cid:0) s ′ | s, a i , π ∗− i ( s, a i ) (cid:1) , the environment from the perspective of agent i will deterministically transition to the next state that has the highest Q ss i among all neighboring states given a i :
[Equation]
where N ( s, a i ) is the neighboring state set of s given a i .
This is proved in the proof of the following theorem.
Theorem 2.
In deterministic environments, all agents will converge to the optimal policies, if each agent i performs Q-learning on the transition function s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) .
Proof.
The proof is given in Appendix A, which shows that s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) is the ideal transition function.
To model the stationary ideal transition function s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) , we train a neural network f i ( s, a i ) to predict s ′∗ and update f i ( s, a i ) by maximizing:
[Equation]
The first term enforces that the predicted next state has the highest Q ss i , and the second term constrains the predicted next state to be in the set N ( s, a i ) .
The hyperparameter λ is the coefficient.
Based on the transition model f i ( s, a i ) , Q ss i is updated by minimizing the TD-error:
[Equation] [Equation]
Algorithm 1.
I2Q for each agent
i
And Q i is updated by minimizing:
[Equation]
¯ Q is the target network of Q .
When updating Q i (14), the target value is not computed on the next state sampled from replay buffer D i , but on the one predicted by the modeled ideal transition function
f i ( s, a i ) .
In (14), the reward r is simplified to depending just on s .
When r depends on both s and s ′ , given s and a i , the next state under the ideal transition function (11) satisfies:
[Equation]
Therefore, we can update Q i by minimizing:
[Equation]
The training procedure of I2Q is summarize in Algorithm 1, where each agent i learns f i , Q ss i , and Q i .
Although all the modules update simultaneously, as the convergence of Q ss i is guaranteed, the transition function f
i derived from Q ss i will be stationary in the later stage.
Thus Q i will also converge.
I2Q can be applied in environments with both discrete and continuous state-action space.
In continuous action space, we build I2Q on DDPG [14], where a policy network π i ( s ) is trained by maximizing Q i ( s, π i ( s )) as a substitute of arg max a i Q i ( s, a i ) .
In continuous state space, since f i ( s, a i ) is differentiable, (12) can be maximized by gradient ascent.
In discrete state space or large state space, we can map the state space to a continuous embedding space, and apply I2Q on the embedding space.
We also provide an implementation without forward model
f i in Appendix B.7.
3.4 Assumptions
We will further discuss the two assumptions.
The prime one is that there is only one optimal joint policy, but I2Q can easily solve tasks with multiple optimal joint policies.
With multiple optimal actions (with the max Q i ( s, a i ) ), if each agent arbitrarily selects one of the optimal independent actions, the joint action might not be optimal.
To address this, we can set a performance tolerance ε and introduce a fixed randomly initialized reward function ˆ r ( s, s ′ ) in the range (0 , ˆ r max ] , where ˆ r max = (1 − γ ) ε .
Then all agents perform I2Q on the shaped reward r + ˆ r and learn the value function ˆ Q i ( s, a i ) in terms of r + ˆ r .
Since ˆ r is positive, ˆ Q i ( s, a i ) > Q i ( s, a i ) .
In ˆ Q i ( s, a i ) , the maximal contribution from ˆ r is ˆ r max / (1 − γ ) =
ε , so the minimal contribution from r is ˆ
Q i ( s, a i ) − ε > Q i ( s, a i ) − ε , which means that the maximal performance drop is ε when selecting actions according to ˆ Q i .
Moreover, since the reward function ˆ r ( s, s ′ ) is randomly initialized, it is a small probability event to find multiple optimal joint policies on the reward function r + ˆ r .
Thus, if ε is set to be small enough, I2Q could solve the task with multiple optimal joint policies.
The secondary one is deterministic environments.
In stochastic environments, (10) does not hold, and
Q ss i ( s, s ′ ) can be considered the “best possible value” [6].
So s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) is
, 𝑠 0 𝑟, 𝑠 11 𝑟, 𝑠 21 𝑟, 𝑠 |  | 1 𝑟 |  |𝐚𝐚𝐚 | 𝐴 | ⋯ , 𝑠 12 𝑟, 𝑠 22 𝑟, 𝑠 |  | 2 𝑟 |  |𝐚𝐚𝐚 | 𝐴 | ⋯ , 𝑠 13 𝑟, 𝑠 23 𝑟, 𝑠 |  | 3 𝑟 |  |𝐚
[Figure]
𝐚𝐚 | 𝐴 |
[Figure]
[Figure]
d
[Figure] [Figure]
[Figure] [Figure]
the best possible transition function, rather than the ideal transition function.
The difference between the learned Q i of I2Q and the true value under the ideal transition function Q stoi is guaranteed by the following theorem.
Theorem 3.
[Equation]
Q
[Equation]
, where ∆ r = r
[Equation]
− r
[Equation]
Proof.
The proof is given in Appendix A.
Theorem 3 shows that Q i of I2Q is closer to the true value if the transition probability of s ′∗ is higher in stochastic environments.
Since we sample transitions from D i to update f i by maximizing (12), the second term of (12) makes the predicted next state
f i ( s, a i ) be close to the high-frequency next states in replay buffer D i given s and a i , which means
that the transition probability of the predicted next state P env (cid:0) f i ( s, a i ) | s, a i , π ∗− i ( s, a i ) (cid:1) would not be too small.
Thus, Q i of I2Q will be close to the true value and the worst cases where the predicted next states have very small transition probabilities can be avoided.
That is the reason why I2Q can be successfully applied in stochastic environments
Experiments
In experiments, we first evaluate I2Q on a class of randomly generated matrix games to verify our theoretical analysis.
Second, we compare I2Q with a series of Q-learning variants on MPE-based [15] differential games to illustrate the convergence and optimality of I2Q on more complex tasks with continuous action.
Third, we test I2Q on two popular MARL benchmarks: Multi-Agent MuJoCo [18] and SMAC [23].
The experiments cover both fully and partially observable, deterministic and stochastic, discrete and continuous environments.
Since we consider the fully decentralized settings, I2Q and the baselines do not use parameter-sharing .
The results are presented using mean and standard deviation (std) with different random seeds.
More details about hyperparameters are available in Appendix C.
4.1 Matrix Games
To support the theoretical analysis of I2Q, we perform experiments on a class of two-agent matrix games with L stages.
The action space of each agent is 3 , so the joint action space |A|
= 9 .
The payoff tree is shown in Figure 1, where the reward r is randomly generated in the range [ − 1 , 1] and fixed in each matrix game.
Without loss of generality, we randomly generate 100 matrix games for each stage L , train the agents for four different seeds in each game, and plot the mean normalized return (normalized by the optimal return) and std over the 100 matrix games.
Since the states are countable, we adopt Q-tables instead of neural networks, where Q i table and Q ss i table are updating simultaneously , and use ϵ -greedy policies.
Figure 2 shows the learning curves with different stages L under ϵ = 0 .
IQL cannot converge to the optimal policies and shows large std, because it is afflicted
[Figure]
z
[Figure]
e
[Figure]
Published as a conference paper at ICLR
[Figure]
Proposition 2.
Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
As introduced by Son et al.
(2019) and Wang et al.
(2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.
4 E XPERIMENTS
In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
The completeness of the
IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).
In particular, the second term of Eq.
(11) is the main difference between QPLEX and Qatten.
Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
The implementation details of these algorithms and experimental settings are deferred to Appendix B. We also conduct two ablation studies to study the inﬂuence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred to be discussed in Appendix E. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles over 6 random seeds.
4.1 M ATRIX G AMES
QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
The optimal joint strategy of these two games is to perform action A (1) simultaneously.
To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
As shown in Figure 2b, QPLEX, QTRAN, and WQMIX, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions.
These results are deferred to Appendix C. QTRAN
[Figure]
(a) case
Published as a conference paper at ICLR
[Figure] [Figure]
Proposition 2.
Given the universal function approximation of neural networks, the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
In practice, QPLEX can utilize common neural network structures (e.g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
As introduced by Son et al.
(2019) and Wang et al.
(2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.
4 E XPERIMENTS
In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
The completeness of the
IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).
In particular, the second term of Eq.
(11) is the main difference between QPLEX and Qatten.
Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
The implementation details of these algorithms and experimental settings are deferred to Appendix B. We also conduct two ablation studies to study the inﬂuence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred to be discussed in Appendix E. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles over 6 random seeds.
4.1 M ATRIX G AMES
QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
The optimal joint strategy of these two games is to perform action A (1) simultaneously.
To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
As shown in Figure 2b, QPLEX, QTRAN, and WQMIX, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions.
These results are deferred to Appendix C. QTRAN
[Figure]
with non-stationarity.
I2Q converges to the global optimum on the matrix games, even when the stage
L grows, which confirms our theoretical analysis.
In Figure 3, we show the effect of exploration rate ϵ on the matrix games with L = 4 .
As ϵ increases,
I2Q converges faster, while the performance of IQL drops.
In conventional decentralized methods like IQL, there is a dilemma of exploration and exploitation .
If ϵ is small, the weak exploration causes low sample efficiency and slow learning.
When ϵ is large, all agents act more randomly.
From the perspective of an individual agent, the transition probabilities will be much different from the real ones under the learned policies of other agents, which causes bad performance.
That is the reason why IQL drops when ϵ increases.
I2Q can avoid the dilemma, because Q ss i ( s, s ′ ) is independent from agent policies and converges faster with larger ϵ as it sees more states.
Thus, I2Q can use large ϵ , e.g. , 0 . 5 or 0 .
7 , to promote sample efficiency.
Moreover, we test I2Q on two one-stage matrix games proposed in QTRAN [24] and QPLEX [31] as shown in Figure 4, which are special cases of Figure 1.
In case 1 , IQL converges to the local optimum, and in case 2 , IQL does not converge.
Our I2Q can converge to the global optimum easily, in a fully decentralized way.
4.2 MPE
To investigate the effectiveness of I2Q in complex continuous environments with neural network implementation, we design a class of MPE-based differential games, where N agents can move in the range [ − 1 , 1] .
In each timestep, agent i acts the action a i ∈ [ − 1 , 1] , and the position of agent i is updated as x i = clip ( x i + 0 .
1 × a i , − 1 , 1) ( i.e. , the updated position is clipped to [ − 1 , 1] ).
The state is the position vector { x 1 , x 2 , · · · , x N } .
The reward function of each timestep is defined as
[Equation]
The visualization of reward function of two-agent case is shown in Figure 5a.
There is only one global optimum ( l = 0 and r = 1 ) but infinite sub-optima ( l = 0 . 8 and r = 0 . 3 ), and the region with r > 0 .
3 is very narrow and surrounded by the region with r = 0 .
So it is hard to learn the optimal policies in a decentralized way.
The episode contains 100 timesteps, and the agents’ positions are randomly initialized at the beginning of each episode.
To verify the scalability, we test I2Q in the settings with different agent numbers N , and train the agents for eight random seeds in each setting.
The results are shown in Figure 6.
IDDPG always falls into local optimum due to the non-stationarity and outdated transition probabilities.
To be optimistic towards the value punishment is an important technique in decentralized MARL.
Although adopting the optimistic update, Hysteretic IDDPG [16] still falls into local optimum due to the non-stationarity problem.
D3G [6] applies the idea
[Figure] [Figure]
[Figure] [Figure]
of QSS
in single-agent learning.
To extend D3G into decentralized MARL, we train each agent independently using D3G, without the information of other agents.
However, D3G cannot learn the optimal solution either, since the forward model in D3G requires the transition probabilities P i to be deterministic.
However, when learning on the decentralized replay buffer, P i is the average of non-stationary transition probabilities and cannot be deterministic.
Thus, D3G is not applicable to decentralized multi-agent learning.
Learning on the ideal transition function inferred from Q ss i ( s, s ′ ) ,
I2Q can escape from local optimum and significantly outperform others.
To thoroughly demonstrate the effectiveness of I2Q, in Figure 5, we visualize the learned values of agent 1 in the two-agent case.
In Figure 5b, we let the agent 2 always act the optimal policy (moving to the position x 2 = 0 ), which means the agent 1 performs Q-learning on the true ideal transition probabilities.
Thus, Figure 5b shows the optimal values under ideal transition probabilities.
In Figure 5c, both agents use I2Q. I2Q values are very similar to the true ideal values, where the value of the center point ( l = 0 ) is the highest, which means that the modeled transition function in I2Q is close to the true one.
In Theorem 4 (see Appendix A), we theoretically analyze that the difference between ideal values (Figure 5b) and
I2Q values (Figure 5c) is bounded by the model error.
In Figure 5d, both agents use IQL, resulting in that the value of the center point ( l = 0 ) is the lowest and the suboptimal points ( l = 0 . 8 ) have the highest values.
[Figure] [Figure]
To model the ideal transition function, next states predicted by f i should have the highest Q ss i values (optimality) and be in the neighboring state set given state-action pair (referred to as neighborhood constraint).
To achieve this objective, λ in (12) controls the balance between optimality and neighborhood constraint.
Figure 7 shows the effect of λ .
When λ is too small, the next state predicted by f i cannot maximize Q ss i ( s, s ′ ) , and thus I2Q gets stuck at local optimum.
Especially, when λ = 0 , I2Q degrades into a popular model-based method MBPO [9].
When λ is too large, e.g. , λ = 10 , paying much more attention to maximize Q ss i ( s, s ′ ) , f i would generate out-of-neighborhood states, causing that I2Q even cannot converge to local optimum.
We notice that λ
= 0 .
5 can generally achieve a good performance in different settings of N .
Although we assume deterministic environments in theoretical analysis, I2Q can also obtain performance gain in stochastic environments.
To measure the impact of stochastic environments, we add a noise to the position update: x i = clip ( x i + 0 .
1 × a i + ξ × z, − 1 , 1) , where z is a uniform random variable in [ − 1 , 1] , and ξ is a constant controlling the randomness.
Figure 8 shows the performance with different ξ in the three-agent case.
In the stochastic environments, IDDPG still converges to local optimum, while I2Q always escapes from local optimum when ξ = 0 .
02 and 0 .
03 , free from the impact of stochastic environments.
When ξ = 0 . 04 and 0 .
05 , the effect of stochasticity is nearly half of the effect of action.
I2Q cannot always learn optimal policies with the strong stochasticity but still significantly outperforms IDDPG.
4.3 Multi-Agent MuJoCo and SMAC
To investigate the effectiveness of I2Q in partially observable environments, we perform experiments on Multi-Agent MuJoCo [18], where each agent independently controls one or some joints of the robot and could only observe the state of its own joints and bodies (with the parameter agent_obsk = 0).
The results are shown in Figure 9 with eight random seeds.
I2Q achieves higher rewards than IDDPG, which indicates that I2Q could be extended to partially observable tasks and obtain performance gain by performing Q-learning on ideal transition probabilities instead of transition probabilities in replay buffer.
In the partially observable setting, we only consider two-agent cases.
When there are more agents, each agent can only control one or two joints, and the observation range is too limited to learn strong policies.
In Appendix B.4, we provide the 6 -agent Walker and 8 -agent
Ant experiments with the full observation setting to verify the scalability.
We test I2Q on partially observable and stochastic SMAC tasks [23] with the version SC2.4.10, including both easy and hard maps [33].
We adopt the implementation of PyMARL [23] that takes as input the trajectory of partial observations, which is high-dimensional and not differentiable on some dimensions, meaning that directly optimizing (12) is hard.
So the QSS model Q ss i is not built on the trajectory of partial observations, but on the hidden state of Q i (the output of RNN layer), which is 64 -dimension and differentiable, and the transition function f
i predicts the next hidden state.
We train the agents for four random seeds.
D3G can only be used in continuous action space, so we do not compare with it in SMAC.
As shown in Figure 10, I2Q is capable of handling high-dimensional and stochastic tasks.
Closing Remarks
In this paper, we propose I2Q, a novel variant of independent Q-learning, to deal with the nonstationarity in decentralized cooperative MARL.
I2Q models the ideal transition function, which depends on the optimal conditional joint policy of other agents and performs independent Q-learning on the modeled ideal transition function.
Theoretically, we prove that I2Q converges to the optimal joint policy in deterministic environments.
Empirically, I2Q obtains performance gain in a variety of multi-agent tasks.
We focus on the base algorithm for fully decentralized learning in cooperative MARL, as it lay the algorithmic foundation for decentralized learning.
Although we did not consider explicit coordination among agents, in partially observable environments coordination such as communication is desirable to improve empirical performance.
We envision that more methods can be further built on the base algorithm.
Moreover, currently, although fully decentralized learning algorithms may not empirically perform as well as CTDE methods like QMIX [22] in some benchmarks, e.g., hard maps in SMAC, we believe more efforts should be made by the MARL community on decentralized learning, considering its abundant benefits over CTDE. First, fully decentralized algorithms do not require a centralized learner during training, hence they are more applicable and cleaner, easier to implement [10]. Second, fully decentralized algorithms are more versatile as the individual learners are indifferent to the number of other agents, thus they have better scalability [34]. Third, fully decentralized algorithms make agents more robust to the presence of agents they were not trained with (such as humans)
Therefore, fully decentralized learning is a fundamental problem in cooperative MARL, but remains to be investigated.
Acknowledgments and Disclosure of Funding
This work was supported by NSF China under grant 61872009.
The authors would like to thank the anonymous reviewers for their valuable comments.
A Proof
Theorem 2.
In deterministic environments, all agents will converge to the optimal policies, if each agent i performs Q-learning on the transition function s
[Equation]
Proof.
According to Theorem 1, we know that if each agent i performs Q-learning on ideal transition function, all agents will converge to the global optimal policies.
Therefore, to prove Theorem 2 is to prove that s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) is a transition function under one of the optimal conditional joint policies of other agents π ∗− i ( s, a i ) .
Since
[Equation]
where s ′ is the deterministic next state of a − i when given s and a i , according to (10), we have
[Equation]
Thus, max s ′ Q ss i ( s, s ′ )
= max a − i Q ( s, a i , a − i ) , which means Q ss i ( s, s ′∗ ) =
Q ( s, a i , π ∗− i ( s, a i )) .
Therefore, when there are multiple s ′∗ = arg max s ′ ∈N ( s,a i ) Q ss i ( s, s ′ ) , any one of them is an ideal transition function under one of the optimal conditional joint policies of other agents, and Theoremholds.
Theorem
[Equation] [Equation]
[Table]
Proof.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Therefore, we have
[Equation]
The bound in Theorem 3 show that Q i of I2Q is closer to the true value if the transition probability of s ′∗ is higher in stochastic environments.
Since we sample transitions from D i to update f i by maximizing (12):
[Equation]
the second term makes the predicted next state
f i ( s, a i ) be close to the high-frequency next states given s and a i in replay buffer D i , which means
that the transition probability of the predicted next state P env (cid:0) f i ( s, a i ) | s, a i , π ∗− i ( s, a i ) (cid:1) would not be too small.
Thus, Q i of I2Q will be close to the true value and the worst cases where the predicted next states have very small transition probabilities can be avoided according to our implementation (12).
That is the reason why I2Q can be successfully applied in stochastic environments , as shown in Figure 8 and Figure 10.
Theorem 4.
Q oi ( s, a i ) is the value function learned on the ideal transition function s ′ o
=
f oi ( s, a i ) , Q i ( s, a i ) is the value function learned on the modeled ideal transition function s ′ =
f i ( s, a i ) , considering the continues state space, ∥ Q oi − Q i ∥ ∞ ≤ T + γG 1 − γ ∥
f oi − f i ∥ ∞ , where
G = max s,a | ∂Q i ( s,a ) ∂s | , T = max s,s ′ | ∂r ( s,s ′ ) ∂s ′ | , when f i ( s, a i ) is close to f oi ( s, a i ) .
Proof.
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Omiting the remainder term, we have
[Equation]
Theorem 4 shows that the difference between ideal values Q oi and learned values
Q
i is bounded by the error of modeled transition function.
As shown in Figure 5, the learned values of I2Q are very close to the ideal values, because I2Q accurately models the ideal transition function.
B Additional Results B.1 Hyperparameter λ
In Section 4.2, we have shown the effectiveness of λ .
Here, we further discuss the cases where λ = 0 .
When λ = 0 , I2Q degenerates into a popular model-based method MBPO [9].
In existing model-based methods, the modeled transition functions follow the transition probabilities in the replay buffer, which are non-stationary and outdated, so existing model-based methods also face the non-stationary problem.
As shown in Figure 7 and Figure 11, I2Q outperforms I2Q with λ = 0 , which verifies the effectiveness of the proposed objective function of the transition function f i in I2Q.
[Figure]
B.2 Two Update Rules
We propose two update rules for Q i : rule 1 (14) and rule 2 (15).
Figure 12 shows that the two rules achieve similar performance, especially in the tasks with long horizons where the reward of one timestep is insignificant compared with the cumulated return value.
We use rule 2 in differential games and use rule 1 in SMAC and Multi-Agent MuJoCo
[Figure]
[Figure]
B.3 Independent PPO
Independent PPO (IPPO)
[4] is a strong on-policy decentralized MARL baseline.
However, it is not fair to compare off-policy algorithms with on-policy algorithms
[Figure]
since on-policy algorithms do not use old data, which makes them weaker on sample efficiency [1]
[Figure]
as shown in Figure
[Figure]
B.4 Scalability
We test I2Q on 6 -agent Walker and 8 -agent Ant with full observation setting, the results are shown in Figure 14.
Taking state information as input, IDDPG is strong enough, but I2Q can still obtain performance gain on 8 -agent Ant.
[Figure]
B.5 Multiple Optimal Joint Policies
In Section 3.4, we have analyzed that I2Q can easily solve the task with multiple optimal joint policies.
Here, we give another way to solve this problem.
When agent i selects one of the optimal independent action, the neighboring state set N ( s, a i ) contains one of the optimal next states, which have the max Q ss i ( s, s ′ ) .
The union of the neighboring state sets ∩ i N ( s, a i ) is a singleton set, which contains the next state of the joint action.
If each N ( s, a i ) contains different optimal next states, the union ∩ i N ( s, a i ) might contain none of the optimal ones.
We let each agent independently and randomly initialize a same fixed noise function η ( s ) , using the same random seed, as a partial order, and makes decisions by the partial order
[Equation]
Thus, each N ( s, a i ) will contain the optimal next state with the max η value, which is the next state of the joint action.
Thus, the agents are coordinated by η .
Since the noise function η is randomly initialized, it is a small probability event to find multiple next states with the same η value.
We could also adopt other partial orders, e.g. , l 1 norm of s ′ .
We test I2Q on a one-stage matrix game with two optimal joint policies (1 , 2) and (2 , 1) , as shown in
Figure 15.
If the agents independently select actions, they might choose the miscoordinated joint policies (1 , 1) and (2 , 2) .
IQL learns the suboptimal policy (1 , 1) , but I2Q agents always select coordinated actions, though the value gap between the optimal policy and suboptimal policy is so small
[Figure]
Published as a conference paper at ICLR
[Figure]
Proposition 2.
Given the universal function approximation of neural networks
[Figure]
the action-value function class that QPLEX can realize is equivalent to what is induced by the IGM principle.
In practice
[Figure]
QPLEX can utilize common neural network structures (e
[Figure]
g., multi-head attention modules) to achieve superior performance by approximating the universal approximation theorem (Csáji et al., 2001).
We will discuss the effects of QPLEX’s duplex dueling network with different conﬁgurations in Section 4.1.
As introduced by Son et al.
(2019) and Wang et al.
(2020a), the completeness of value factorization is very critical for multi-agent Q-learning and we will illustrate the stability and state-of-the-art performance of QPLEX in online and ofﬂine data collections in the next section.
4 E XPERIMENTS
In this section, we ﬁrst study didactic examples proposed by prior work (Son et al., 2019; Wang et al., 2020a) to investigate the effects of QPLEX’s complete IGM expressiveness on learning optimality and stability.
To demonstrate scalability on complex MARL domains, we also evaluate the performance of QPLEX on a range of StarCraft II benchmark tasks (Samvelyan et al., 2019).
The completeness of the
IGM function class can express richer joint action-value function classes induced by large and diverse datasets or training buffers.
This expressiveness can provide QPLEX with higher sample efﬁciency to achieve state-of-the-art performance in online and ofﬂine data collections.
We compare QPLEX with state-of-the-art baselines: QTRAN (Son et al., 2019), QMIX (Rashid et al., 2018), VDN (Sunehag et al., 2018), Qatten (Yang et al., 2020), and WQMIX (OW-QMIX and CW-QMIX; Rashid et al., 2020).
In particular, the second term of Eq.
(11) is the main difference between QPLEX and Qatten.
Thus, Qatten provides a natural ablation baseline of QPLEX to demonstrate the effectiveness of this discrepancy term.
The implementation details of these algorithms and experimental settings are deferred to Appendix B. We also conduct two ablation studies to study the inﬂuence of the attention structure of dueling architecture and the number of parameters on QPLEX, which are deferred to be discussed in Appendix E. Towards fair evaluation, all experimental results are illustrated with the median performance and 25-75% percentiles over 6 random seeds.
4.1 M
[Figure]
G
[Figure]
QTRAN (Son et al., 2019) proposes a hard matrix game, as shown in Table 4a of Appendix C.
In this subsection, we consider a harder matrix game in Table 2a, which also describes a simple cooperative multi-agent task with considerable miscoordination penalties, and its local optimum is more difﬁcult to jump out.
The optimal joint strategy of these two games is to perform action A (1) simultaneously.
To ensure sufﬁcient data collection in the joint action space, we adopt uniform data distribution.
With this ﬁxed dataset, we can study the optimality of multi-agent Q-learning from an optimization perspective, ignoring the challenge of exploration and sample complexity.
As shown in Figure 2b, QPLEX, QTRAN, and WQMIX, which possess a richer expressiveness power of value factorization can achieve optimal performance, while other algorithms with limited expressiveness (e.g., QMIX, VDN, and Qatten) fall into a local optimum induced by miscoordination penalties.
In the original matrix proposed by QTRAN, QPLEX and QTRAN can also successfully converge to optimal joint action-value functions.
These results are deferred to Appendix C. QTRAN
[Figure]
B.6 Discussion on D3G
D3G predicts the optimal next state s ′∗ with the highest QSS value and decodes actions in execution by training a inverse model a i =
g
( s, s ′∗ ) .
Different from D3G, I2Q builds transition function f
i ( s, a i ) using QSS value and trains Q-learning on f i ( s, a i ) .
Several drawbacks limit the performance of D3G in decentralized MARL.
First, over-generalization of g ( s, s ′∗ ) would lead to wrong a i .
Considering two inverse maps ( s, s ′ 1 ) → a 1 and ( s, s ′ 2 )
→ a 2 , where s ′ 1 and s ′ 2 are very similar but a 1 and aare very different, which is common in complex continuous environments, trained by supervised learning, the inverse model g may mistakenly predict g ( s, s ′ 1 ) as the average of a 1 and a 2 , due to the generalization of neural network on state space.
Since a 1 and a 2 are very different, the prediction error is large and leads to poor performance.
Experimentally, we find it is hard to perfectly train the inverse model g ( s, s ′ ) , and the prediction error of actions greatly influences the performance in practice.
I2Q could avoid this problem since it learns accurate values Q i ( s, a 1 ) and Q i ( s, a 2 ) and selects the action with the highest value.
Second, as stated in the experiment section, the implementation of D3G adopts a forward model, which requires the transition probabilities in replay buffer D
i to be deterministic, which is impossible in decentralized MARL.
The motivation and implementation of I2Q are very different from that of D3G, which makes I2Q more suitable and practical in decentralized MARL, as shown in Figure 6 and Figure 9.
The original D3G can only be used in continuous action space, and we extend it to discrete space and compare it in SMAC.
Due to the analyzed drawbacks, discrete
D3G cannot obtain a winning rate in SMAC, as shown in Table 1.
[Table]
B.7 I2Q without Forward Model
For practicability, we design an approximation of I2Q without forward model f i , which is enlightened by implicit Q-learning[12].
Implicit Q-learning learns the optimal value without predicting next actions, similarly, we can obtain QSS value without predicting next states.
Following Implicit
Q-learning, we introduce V i ( s ) and utilize expectile regression.
Specifically, we update Q ss i by minimizing
[Equation]
update V i ( s ′ ) by minimizing expectile loss
[Equation]
Using the approximate QSS value, we have
[Equation]
According to Eq 15 and the mathematical derivation between line 161 and line 162, we update
Q i ( s, a i ) to be Q ss i ( s, s ′∗ ) by minimizing
[Equation]
Although QSS value is a biased estimation in this implementation, the implementation without forward model is practical.
We test I2Q w/o forward model in SMAC.
The results are shown in
Figure 16.
I2Q w/o f shows similar performance with I2Q, and does not need to learn the forward model, thus it would be more practical in complex environments.
[Figure]
C Hyperparameters
In 2 × 3 HalfCheetah, there are two agents and each of them controls 3 joints of HalfCheetah.
In 2 |
HalfCheetah, there are two agents.
One of them controls 2 joints, and one of them controls 4 joints.
And so on.
In MPE-based (MIT license) differential games and Multi-Agent MuJoCo (MIT license), we adopt the implementation of SpinningUp [1] (MIT license), the SOTA implementation of DDPG, and follow all hyperparameters in SpinningUp.
The discount factor γ = 0 .
99 , the learning rate is 0 .
001 with
Adam optimizer, the batch size is 100 , the replay buffer contains 1 × 10 6 transitions, the hidden units are 256 .
In SMAC (MIT license), we adopt the implementation of PyMARL [23] (Apache-2.0 license) and follow all hyperparameters in PyMARL.
The discount factor γ = 0 .
99 , the learning rate is 0 .with
RMSprop optimizer, the batch size is 32 episodes, the replay buffer contains 5000 episodes, the hidden units are 64 .
We adopt the version SC2.4.10.
The hyperparameter λ has been fully tested in MPE-based differential games.
In Multi-Agent
MuJoCo, we set λ = 0 .
01 , and in SMAC, we set λ = 0 .
The experiments are carried out on Intel i7-8700 CPU and NVIDIA GTX 1080Ti
GPU.
The training of each MPE and Multi-Agent MuJoCo task could be finished in 5 hours, and the training of each
SMAC task could be finished in 20 hours.