Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs
Abstract
Covering option discovery has been developed to improve the exploration of RL in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph.
Given that joint state space grows exponentially with the number of agents in multi-agent systems, existing researches still relying on single-agent option discovery either become prohibitive or fail to directly discover joint options that improve the connectivity of the joint state space.
In this paper, we show how to directly compute multi-agent options with collaborative exploratory behaviors while still enjoying the ease of decomposition.
Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector using the Laplacian spectrum of individual agents’ transition graphs.
Further, considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method by estimating eigenfunctions through NN-based representation learning techniques.
The evaluation on multi-agent tasks built with simulators like Mujoco, shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms the state-of-the-art.
Codes are available at: https://github.itap.purdue.edu/Clan-labs/Scalable_MAOD_via_KP.
Introduction
Option discovery [1] enables temporally-abstract actions to be constructed and utilized in RL, which can significantly improve the performance of RL agents.
Among recent developments, Covering
Option Discovery [2, 3] has been shown to be an effective approach to improve the exploration in environments with sparse reward signals.
In particular, it first computes the second smallest eigenvalue λ 2 and the corresponding eigenvector F (i.e., Fiedler vector [4]) of the Laplacian matrix extracted from the state transition process in RL.
Then, options are built to connect the states corresponding to the minimum or maximum in F , which greedily improves the algebraic connectivity of the state space [5] and accelerate exploration within it.
In this paper, we consider constructing and utilizing covering options in MARL.
Due to the exponentially-large state space in multi-agent scenarios, a commonly-adopted way to solve this problem [6–9] is to construct individual options as if in a single-agent environment first, and then learn to collectively leverage these individual options to tackle multi-agent tasks.
This method fails to consider the coordination among agents in the option discovery process, and thus can suffer from poor behavior in multi-agent collaborative tasks.
To this end, we propose a framework that makes novel use of Kronecker graphs to directly construct multi-agent options and adopt them to accelerate the joint exploration of agents in MARL.
In particular, the joint state space is approximated as the Kronecker product of the state spaces of individual agents.
Then, based on the theorem we propose, we can directly estimate the Fiedler vector of the joint state space using the Laplacian spectrum of individual ones.
Next, the joint options can be constructed to explore the joint states corresponding to the minimum or maximum in the Fiedler vector, resulting in a greedy improvement of the joint state space’s connectivity.
However, calculating the Laplacian spectrum can be expensive in a matrix base.
As a scalable extension, we leverage a deep neural network approximation to learn the Laplacian spectrum so that our method can be adopted to tasks with infinite-scale state spaces.
Our main contributions are: (1) We propose Multi-agent
Covering Option Discovery – a decentralized manner to discover multi-agent options which can greatly aid the joint exploration of agents in MARL.
(2) We propose that multi-agent options can be adopted to MARL in either a decentralized or centralized manner, which means that agents can either jointly decide on their options, or choose their options independently and select different options to execute simultaneously.
(3) We scale Multi-agent Covering Option Discovery to infinite-scale state spaces using SOTA representation learning techniques, and propose that the discovered options can be adopted with deep MARL methods for continuous tasks.
(4) We empirically show that agents using our multi-agent options significantly outperform agents with single-agent options or no options.
Related Work
Option Discovery:
The option framework was proposed in [1], which extends the usual notion of actions to include options — the closed-loop policies for taking actions over a period of time.
A lot of option discovery algorithms have been proposed.
Some of them are based on task-related reward signals [10–13].
Specifically, they directly define or learn the options through gradient descent that can lead the agent to the rewarding states, and then utilize these trajectory segments (options) to compose the completed trajectory toward the goal state.
These methods rely on dense reward signals, which are hard to acquire in real-life tasks.
Other works define the sub-goal states (termination states of the options) based on the visitation frequency of the states.
For example, in [14–16], they discover the options by recognizing the bottleneck states in the environment, through which the agent can transfer between the sub-areas that are loosely connected in the state space, and they define these options as betweenness options.
Recently, there are some state-of-the-art option generation methods based on the Laplacian spectrum of the state-transition graph, such as [2, 3, 17–19], since the eigenvectors of the Laplacian of the state space can provide embeddings in lower-dimensional space, based on which we can obtain good measurements of the accessibility from one state to another.
Especially, in [3, 19], they propose covering options and prove that their option generation method has higher exploration speed and better performance compared with the previous option discovery methods.
Note that all the approaches mentioned above are for single-agent scenarios, we will extend the construction and adoption of covering options to MARL in this paper.
Adopting options in multi-agent scenarios: Most of the researches on adopting options in MARL [6–9, 20, 21] try to first learn the options for each agent with the option discovery methods we mentioned above, and then learn to collaboratively utilize these individual options.
Therefore, the options they use are still single-agent options, and the coordination among agents can not be utilized in the option discovery process.
In this paper, we propose directly constructing multi-agent covering options of the joint state transition graphs based on Laplacian spectrum of the individual ones, and explore how to utilize the multi-agent options in MARL tasks effectively.
Further, to scale our algorithm, we extend a tabular algorithm considered in [22], adopt SOTA representation learning techniques to obtain the Laplacian spectrum for option discovery, and integrate the discovered options with deep MARL algorithms, to aid the learning performance in scenarios with infinite-scale state spaces, which is testified on multiple complex Grid and Mujoco tasks.
Background
Kronecker Product of Graphs: Let G 1 =
( V G 1 , E G 1 ) and G 2 =
( V G 2 , E G 2 ) be two state transition graphs (defined in Appendix B), corresponding to the individual state space S 1 and S 2 .
The Kronecker
[Figure]
product of them denoted by G 1 ⊗ G 2 is a graph defined on the set of vertices V G
1 × V G 2 , such that [23]: Two vertices of G 1 ⊗ G 2 , namely ( g, h ) and ( g ′ , h ′ ) , are adjacent if and only if g and g ′ are adjacent in G 1 and h and h ′ are adjacent in G 2 .
Thus, the Kronecker graph can capture the joint transitions of agents in their joint state space very well.
In Section 4.2, we propose to use the
Kronecker graph as an effective approximation of the joint state transition graph, so that we can discover the joint options based on the factor graphs.
Further, A 1 ⊗ A 2 is an |S 1 ||S 2 | × |S 1 ||S 2 | matrix with elements defined by ( A 1 ⊗ A 2 )( I, J ) =
A 1 ( i, j )
A 2 ( k, l ) with Equation (1), where Aand A 2 are the adjacency matrices of G 1 and G 2 , A 1 ( i, j ) is the element lies on the i -th row and j -th column of A 1 (indexed from 1).
[Equation]
Covering Option Discovery: As defined in [1], an option ω consists of three components: an intraoption policy π ω : S x A → [0 , 1] , a termination condition β ω : S → { 0 , 1 } , and an initiation set
I ω ⊆ S .
An option < I ω , π ω , β ω > is available in state s if and only if s ∈ I ω .
If the option ω is taken, actions are selected according to π ω until ω terminates stochastically according to β ω (i.e., β ω = 1 ).
Recently, the authors of [3] proposed Covering Option Discovery – discovering options by minimizing the upper bound of the expected cover time of the state space.
First, they compute the Fiedler vector
F of the Laplacian matrix of the state transition graph.
Then, they collect the states s i and s j with the largest ( F i − F j ) 2 ( F i is the i -th element in F ), based on which they construct two symmetric options: ω ij = <
I ω ij = { s i } , π ω ij , β ω ij =
s j } >, ω ji
= <
I ω ji = { s j } , π ω ji , β ω ji = { s i } > to connect these two subgoal states, where π ω ij is defined as the optimal path from s i to s j .
The intuition of this method is as follows.
The authors of [5] prove that ( F i − F j ) 2 gives the first order approximation of the increase in λ 2 ( L ) (i.e., algebraic connectivity) by connecting ( s i , s j ) , and propose a greedy heuristic to improve the algebraic connectivity of a graph: adding a certain number of edges one at a time, and each time connecting ( s i , s j ) corresponding to the largest ( F i − F j ) 2 .
Further, in [3], they prove that the larger the algebraic connectivity is, the smaller the upper bound of the expected cover time would be and the easier the exploration tends to be.
Therefore, applying this greedy heuristic to the state transition graph can effectively improve the exploration in the state space.
Proposed Algorithm 4.1 System Model
In this paper, we consider to compute covering options in multi-agent scenarios, with n being the number of agents, (cid:101) S =
S 1 × S 2 × · · · × S n being the set of joint states, (cid:101)
A =
A 1 × A 2 × · · · × A n being the set of joint actions, S i and A i being the individual state space and action space of agent i .
The size of the joint state space grows exponentially with the number of agents.
Thus, it is prohibitive to directly compute the covering options based on the joint state transition graph using the approach introduced in Section 3 for a large n .
A natural method to tackle this problem is to compute the options for each agent by considering only its own state transitions, and then learn to collaboratively leverage these individual options.
However, it fails to directly recognize joint options composed of multiple agents’ temporal action sequences for encouraging the joint exploration of all the agents.
The algebraic connectivity of the joint state space may not be improved with these single-agent options.
We illustrate this with a simple example.
Illustrative example: Figure 1(a) shows a joint state transition graph (cid:101) G of two agents, where
S 1 = { 1 , 2 } and S 2 = { 1 , 2 , 3 , 4 } .
In order to compute the individual options, we can restrict our attention to the state transition graph of each agent, namely G 1 and G 2 .
The Fiedler vectors (not
[Figure]
normalized) of G 1 and G 2 are: v G 1 = [ − 1 , 1] , v G 2 =
[ − 1 , −√ 2 + 1 , √ 2 − 1 , 1] .
Then, according to the option discovery approach described in Section 3, we can get the individual options for agent 1 to connect its state 1 (minimum) and state 2 (maximum), and individual options for agent 2 to connect its state 1 and state 4.
With these options, the updated joint state space is Figure 1(b).
The straightforward decomposition of option discovery for MARL fails to create a connected graph.
Thus, utilizing single-agent options alone may not be sufficient for efficient exploration.
Therefore, we propose to build Multi-agent Covering Options to enhance the connectivity of the joint state space.
We can represent it as a tuple: < I ω , π ω , β ω > , where I ω ⊆
(cid:101) S is the set of initiation joint states, β ω : (cid:101) S
→ { 0 , 1 } indicates the joint states to terminate, π ω
=
( π 1 ω , · · · , π nω )
( π iω :
S i ×
A i → [0 , 1]) , is the joint intra-option policy that can lead the agents from the initiation states to the termination states.
The key challenge is to calculate the Fiedler vector of the joint state space according to which we can define < I ω , π ω , β ω > .
Given that | (cid:101) S| grows exponentially with n , we propose to estimate the joint Fiedler vector based on the individual state spaces in the next section.
4.2 Theory results
We propose to use the Kronecker graph to decompose the calculation of the joint Fielder vector to single-agent state spaces, because: (1) the Kronecker product of individual state transition graphs ⊗ ni =1 G i
=
G 1 ⊗ · · · ⊗ G n provides a good approximation of the joint state transition graph (cid:101) G ; (2)
The Fielder vector of ⊗ ni =1 G i can be estimated with its factor graphs G i ( i = 1 , · · · , n ) .
The use of ⊗ ni =1 G i as a factorized approximation of (cid:101) G introduces noise, since (cid:101) G = ⊗ ni =1 G i becomes exact only in the case where agents’ transitions are not influenced by the others.
However, for the purpose of option discovery, we only need to identify areas in the state space with relatively low or high values in the Fielder vector, so an exact calculation of (cid:101) G and its Fiedler vector is not necessary.
Moreover, the state transition influence among agents, e.g., collisions and blocking, would most likely result in local perturbations of the transition graph and thus is inconsequential to global properties of (cid:101) G
(e.g., the algebraic connectivity), especially for large-scale state spaces.
Therefore, approximating (cid:101) G by ⊗ ni =1 G
i allows efficient option discovery.
We empirically show in Figure 8 that superior exploration can still be achieved under such approximation noise, even if the transition influence is heavy (affecting 63% of the episode length), numerically validating the robustness of our proposed approach to the approximation error.
We further provide a quantitative study on the approximation error in Appendix E.4.
Next, we show how to effectively approximate the Fiedler vector of ⊗ ni =1 G i based on the Laplacian spectrum of the factor graphs, to achieve an effective decomposition.
Inspired by [24] that proposed an estimation of the Laplacian spectrum of the Kronecker product of two factor graphs, we prove Theorem 4.1 for an arbitrary number of factor graphs.
Theorem 4.1.
For graph (cid:101) G = ⊗ ni
=1 G i , we can approximate the eigenvalues µ and eigenvectors v of its Laplacian L by:
[Equation]
where λ G i k i and
v G i
k
i are the k
i -th
smallest eigenvalue and corresponding eigenvector of L G i (normalized Laplacian of G i ), d G i
k i is the k
i -th smallest diagonal entry of D G i (degree matrix of G i ).
The proof of Theorem 4.1 is provided in Appendix A. Through enumerating ( k 1 , · · · , k n ) , we can collect the eigenvalues and corresponding eigenvectors of ⊗ ni =1 G i by Equation (2).
Then, the eigenvector v ˆ k 1 , ··· , ˆ k n corresponding to the second smallest eigenvalue µ ˆ k 1 , ··· , ˆ k n is the estimated joint
Fiedler vector F (cid:101) G .
We can define the joint states corresponding to the maximum or minimum in F (cid:101) G as the initiation or termination joint states.
As discussed in Section 3, connecting these two joint states with options can greedily improve the algebraic connectivity of the joint state space.
Illustrative example: Now we reconsider Figure 1(a), where (cid:101) G
=
G 1 ⊗ G 2 .
We approximate F (cid:101) G using Theorem 4.1 and get two approximations (details are in Appendix C):
[Equation]
Based on them and the indexing relationship between (cid:101) G and its factor graphs (Equation (1)), we can get two sets of multi-agent options: { I ω
={(1,2), (1,3), (2,2), (2,3)}, β ω 1 ={(1,1), (1,4), (2,1), (2,4)}}
and { I ω
={(1,2), (2,3)}, β ω 2 ={(1,3), (2,2)}}, where we set the joint states corresponding to the maximum or minimum as the initiation or termination states respectively.
For example, in v 11 , the 7 -th element (indexed from 1) is a maximum, so the 7 -th joint state is in I ω 1 and denoted as (2 , 3) (Equation (1)).
As shown in Figure 2, both of the two options can lead to a connected graph.
Thus, the adoption of multi-agent options has the potential to encourage efficient exploration in the joint state space.
Next, we will formalize our algorithm.
4.3 Scalable Multi-agent Covering Option Discovery
In this section, we first introduce how to construct multi-agent options based on individual state transition graphs of each agent (represented as adjacency matrices A 1: n ), of which the detailed pseudo codes are in Appendix D.
Then, we propose a scalable extension of our algorithm to infinite-scale state space, by which we don’t need to explicitly construct the state transition graph or compute its
Laplacian spectrum through eigendecomposition.
First, we need to calculate the estimation of F (cid:101) G through Theorem 4.1 based on A 1: n , and collect the joint states corresponding to the minimum or maximum in F (cid:101) G as the lists of subgoals MIN and MAX .
Then, we split each joint state into a list of individual states.
For example, we convert ( s min , s max ) into ( ( s 1 min , · · · , s nmin ) , ( s 1 max , · · · , s nmax ) ), so that we can connect ( s min , s max ) in the joint state space by connecting each ( s imin , s imax ) in the corresponding individual space.
After decentralizing the joint states, we can define the multi-agent options as follows: (1) For each option ω , we define I ω as the explored joint states, and β ω as a joint state in MIN ∪ MAX or the unexplored area.
Option ω is available in state s if and only if s ∈ I ω .
Therefore, instead of constructing a point option between ( s min , s max ), e.g., setting { s min } as I ω and { s max } as β ω , we extend I ω to the known area to increase the accessibility of ω .
(2) As for the intra-option policy π ω used for connecting the initiation and termination joint state, we divide it into a list of single-agent policies π iω ( i = 1 , · · · , n ) , where π iω can be trained with any single-agent RL algorithm aiming at leading agent i from its own initiation state to the termination state s imin ( s imax ).
To sum up, the proposed algorithm first discovers the joint states that need to be explored most, then projects each sub-goal into individual state spaces and train the policy for each agent to visit the projection.
Here is the computational complexity analysis: Consider an MDP with n agents and m states for each agent.
To compute the Fiedler vector, which is the computation bottleneck of the algorithm, directly from the joint state transition graph would require time complexity O ( m 3 n ) , since there are m n joint states and the time complexity of eigenvalue decomposition is cubic with the number of joint states.
While, with our approach, we can decompose the original problem into computing eigenvectors of the individual state transition graphs, of which the overall time complexity is O ( nm 3 ) .
Thus, our solution significantly reduces the problem complexity from O ( m 3 n ) to O ( nm 3 ) .
As for problems with large state space (i.e., m is large), our option discovery approach could be directly integrated with sample-based techniques for eigenfunction estimation [25, 26], which makes it highly-scalable.
Extension to tasks with infinite-scale state space.
According to [26], the k smallest eigenvalues λ 1: k and corresponding eigenvectors v 1: k of the normalized Laplacian L can be estimated by:
[Equation]
For the large-scale state space, the eigenvectors can be represented as a neural network that takes a state s as input and outputs a k -dimension vector [ f 1 ( s ) , · · · , f k ( s )] as an estimation of [ v 1 ( s ) , · · · , v k ( s )] .
Accordingly, the objective in Equation (4) can be expressed as: (please refer to [26] for details)
[Equation]
where T is a set of state-transitions collected by interacting with the environment randomly.
Further, the orthonormal constraints in Equation (4) is implemented as a penalty term:
[Equation]
where β is the weight term and ρ is the distribution of states in T .
To sum up, the eigenfunctions can be trained as NN by minimizing the loss function: L ( f 1 , · · · , f k ) =
G ( f 1 , · · ·
[Equation]
f k )+ P ( f 1 , · · ·
[Equation]
f k ) .
After getting estimation of v 1: k , we can estimate the corresponding eigenvalues λ 1: k by:
[Equation]
As for the degree d of each state in the individual state transition graph, we can obtain them through f 1 which is the eigenvector corresponding to the smallest eigenvalue of L and satisfies f 1 ( s 1 ) /f 2 ( s 2 ) =
(cid:112) d ( s 1 ) /
(cid:112) d ( s 2 ) [27].
Therefore, we can collect the k -smallest eigenvalues and corresponding eigenvectors, and the degree list for each agent, after which we can utilize Theorem 4.
to estimate the joint Fielder vector.
Note that since we only care about the second smallest eigenvalue, we don’t need to enumerate all the eigenvalues of the factor graphs, and we only need to know the relative value of the degrees.
After getting the joint Fiedler vector, we define the subgoal joint states corresponding to the minimum or maximum (e.g., s min =( s 1 min , · · · , s nmin )) as the termination set, and the other joint states as the initiation set.
As for the intra-option policy, it can be trained with any deep RL algorithms for each agent i to reach its corresponding individual termination state (e.g., s imin ).
In this way, we can obtain the multi-agent options in infinite-scale state spaces.
4.4 Adopting Multi-agent Options in MARL
To utilize options in MARL, we adopt a hierarchical algorithm framework shown in Figure 3.
Typically, we train a RL agent to select among the primitive actions, aiming to maximize the accumulated reward.
We view this agent as a one-step option – primitive option.
As shown in Figure 3, when getting new observations, the hierarchical agent first decides on which option ω to use with the high-level policy, and then decides on the primitive action to take based on the corresponding intra-option policy π ω .
The agent does not decide on a new option with the high-level policy until the current option terminates.
For a multi-agent option < I ω , π ω =
( π 1 ω , · · · , π n ω ) , β ω =
{ ( s 1 , · · · , s n ) } > , it can be adopted either in a decentralized or centralized manner.
As the purple arrows in Figure 3, the agents choose their own options independently, and they may choose different options to execute in the meantime.
In this case, if agent i selects option ω , it will execute π iω until reaching its termination state s i or exceeding the execution step limit.
On the other hand, we can force the agents to execute the same multi-agent option simultaneously.
As shown by the blue arrows in Figure 3, we view the n agents as a whole, which takes the joint state as the input and chooses the same multi-agent option to execute at a time.
Once a multi-agent option ω is chosen, agents 1 : n will execute π 1: n ω until they reach ( s 1 , · · · , s n ) or exceed the step limit.
The decentralized way is more flexible but has a larger search space.
While, the centralized way fails to consider all the possible solutions but makes it easier for the agents to visit the sub-goal joint states, since the agents select the same joint option which won’t terminate within the time limit until the agents arrive at the sub-goal.
Further, the centralized manner may not be applicable when the number of agents n is large, since the input space will grow exponentially with n .
Thus, we propose to partition the agents into sub-groups, and then learn the joint options within each sub-group.
The intuition is that a multi-agent task can usually be divided into sub-tasks, each of which can be completed by a sub-group of agents.
For each sub-group, we can learn a list of joint options, and then the agents of this group can utilize these
[Figure]
options in a decentralized or centralized way as mentioned above.
Further, if there is no way to divide the agents based on sub-tasks, we can still partition them randomly to some two-agent or three-agent sub-groups.
Agents within the same sub-group will co-explore their joint state space sharing their views.
More intelligent grouping methods, e.g., attention mechanism [28], can be easily integrated into our solution to boost the performance, which is not the focus here (multi-agent option discovery).
Evaluation and Results 5.1 Experiment Setup
As shown in Figure 4, the proposed approach is evaluated on four multi-agent goal-achieving tasks.
(1) For tasks shown as Figure 4(a) and 4(b), n (2 ∼ 8) agents agents (triangles) must reach the goal area (circles) at the same time to complete this task; Or even harder, n × m agents are divided into m groups and each group of agents has a special goal labeled with the same color, the n × m agents should get to their related goals simultaneously to complete this task, without knowing which goal is related to them at first.
Hence, highly coordinated policy is required.
(2) To test the performance of the deep learning extension of our algorithm, we build the tasks shown as Figure 4(c) and 4(d) based on Mujoco [29], where two point agents need to reach the target area (red) simultaneously to complete the task.
Both the state and action spaces of the point agent are continuous and infinite-scale.
Note that these tasks are quite challenging: (1) Agents’ observations do not include any information about the goal area, so they need to fully explore their joint state space to find the rewarding state.
The reward space is highly sparse and delayed: for all the tasks, only when the agents complete the task can they receive a reward signal r = 1 .
(shared by all the agents); otherwise, they will receive r = 0 . 0 .
Hence, agents without highly-efficient exploration strategies like ours cannot complete these tasks.
In Section 5.2, we conduct experiments with tasks of increasing complexity (e.g., Figure 5), showing that the more difficult the task is, the more advantageous our approach becomes.
There are a high-level and low-level policy in the hierarchical framework (Figure 3).
The low-level policy is decomposed into single-agent policies for each agent to reach its individual termination state, so it can be trained with single-agent RL.
(1) For discrete tasks (e.g., Figure 4(a) and 4(b)), we adopt Distributed Q-Learning [30] ( decentralized manner : each agent decides on its own option based on the joint state) or Centralized Q-Learning + Force ( centralized manner : viewing n agents as a whole, adopting Q-Learning to this joint agent and forcing them to choose the same joint option at a time) to train the high-level policy and adopt Value Iteration [31] for the low-level policy training.
(2) For continuous control tasks (e.g., Figure 4(c) and 4(d)), the tabular RL algorithms mentioned above cannot work.
Instead, to improve the scalability of the our method, we adopt MAPPO [32] to train the high-level policy of the joint options, and Soft Actor-Critic [33] to train the low-level policy, which are SOTA deep MARL and RL algorithms respectively.
We compare our approach – agents with multi-agent options, with two baselines: (1) Agents without options: SOTA MARL algorithms are adopted to train the agents without using options.
For discrete tasks, we use Distributed and Centralized Q-Learning.
These tabular Q-learning algorithms which update the Q-value for all the state-action pairs in each training episode, usually outperform the
NN-based algorithms.
We show this through comparison with COMA [34], MAVEN [35], Weighted
QMIX [36].
For continuous tasks, we use MAPPO, MADDPG [37] and MAA2C as comparisons, which have been proven as strong MARL baselines for continuous control [38, 39].
Comparisons with this baseline can show the effectiveness of using options to aid exploration.
(2) Agents with single-agent options: we first construct covering options for each agent based on their individual state
[Figure] [Figure]
spaces, and then utilize these options in MARL, like what they do in [6, 7, 20, 8, 9].
As for the option discovery method, we adopt the SOTA algorithm proposed in [3, 19], which claims to outperform previous option discovery algorithms for sparse reward scenarios, like [17, 40].
Comparisons with this baseline can show the superiority of our approach to directly identify and adopt joint options in multi-agent tasks.
The number of single- and multi-agent options for each agent to select is the same.
Last, regarding the setup of the option discovery phase, we collect 1 × 10 4 transitions (i e., { ( s, a, s ′ ) } ) for the discrete tasks and 5 × 10 4 transitions for the continuous control tasks in order to build the state transition graphs, based on which we can extract single- or multi-agent options.
In discrete tasks, we use a random walk policy to collect the data.
While, in Mujoco tasks, the data collection is in stages.
Specifically, the agents explore the environment through a random walk in the first stage.
Then, in the next stage, the agents explore with their primitive actions and options that are extracted from the samples collected in the first stage.
Our results guarantee that these options can significantly improve the joint exploration in the second stage.
As for the number, we learn 16 multi-agent options in the Grid tasks and 8 multi-agent options in the Mujoco tasks.
5.2 Main Results
For each experiment, we present comparisons among agents with multi-agent options (blue line), single-agent options (red line) and no options.
We use the number of time steps to complete the task as the metric (the lower the better).
The maximum of steps that an agent can take is 200 for the discrete task and 500 for the continuous task.
We run each experiment for five times with different random seeds and plot the mean and standard deviation during the training process.
n -agent Grid Maze task:
In Figure 5(a)-5(c), we test these methods with Distributed Q-learning as the high-level policy.
The performance improvement brought by our approach are more and more
[Figure] [Figure]
significant as the number of agents increases.
When n = 4 , both the baselines fail to complete the task, while agents with four-agent options can converge within ∼ 400 episodes.
While, in Figure 5(d)-5(f), we show the results of using Centralized Q-Learning + Force.
We can see that the centralized manner to utilize n -agent options leads to faster convergence, since the joint output space of the agents is pruned (Figure 3).
The results of the n -agent Grid Room task can be found in Appendix E.1, based on which we can get the same conclusion.
To justify the difficulty of the Grid tasks for evaluation, we provide the performance of COMA, MAVEN, Weighted QMIX on the most challenging 4-agent
Grid Maze in E.1.
None of them can learn to complete the coordinated goal-achieving task due to the highly sparse reward setting in our evaluation tasks.
Still, our method performs much better (Figure 5(c)5(f)).
Given that the neural network training in these baselines requires a longer time, we set the training horizon as 5000 episodes which is five times that of the tabular methods and the networks are trained for ten iterations in each episode, to maintain the fairness.
Grid Maze task with sub-task grouping:
The size of the joint state space grows exponentially with the number of agents, making it infeasible to directly construct n -agent options for a large n .
However, in practice, a multi-agent task can usually be divided into sub-tasks, and the agents can be divided into sub-groups based on the sub-tasks they are responsible for.
Thus, we test our proposed method on the m × n Grid Maze tasks shown as Figure 4(b), where we divide the agents into m sub-groups, each of which contains n agents with the same goal area (labelled with the same color).
Note that, in the 2 × 2 ( 3 × 2 ) task, we use two-agent (pairwise) options rather than four-agent (six-agent) options, and for the high-level policy, we only use the joint state of the two agents as input to decide on their joint option choice.
We can see from Figure 6 that agents with pairwise options can learn to complete the tasks much faster than the baselines (e.g., improved by about two orders of magnitude in Figure 6(c)).
Also, we see that agents trained with Centralized Q-Learning + Force (Figure 6(c)-6(d)) have faster convergence speed and better convergence value.
The results for the m × n Grid Room tasks are in Appendix E.2.
Grid Maze task with random grouping:
Further, we note that our method also works with random grouping when sub-task grouping may not work.
The intuition is that adopting two-agent or threeagent options can encourage the joint exploration of the agents in small sub-groups, which can increase the overall performance compared with only utilizing single-agent explorations.
As shown in Figure 7, we compare the performance of agents using pairwise options with the baselines on the n -agent Grid Maze tasks.
When n = 6 , agents with single-agent options or no options can’t complete this task, while we can get a significant performance improvement with only pairwise options.
On the other hand, agents with pairwise options can’t complete the most challenging six-agent task, when using Distributed Q-Learning (Figure 7(b)).
However, if we adopt Centralized Q-Learning + Force, agents with pairwise options can still complete this challenging task with satisfaction (Figure 7(d)).
The same set of results for the Grid Room task are in Appendix E.3.
Grid Room task with random grouping and dynamic influences among agents: Further, we show that even if in environments where an agent’s state transitions can be strongly influenced by the others, we can still obtain good approximations of the multi-agent options to encourage joint exploration using Theorem 4.1.
For this new setting, different agents cannot share the same grid so that an agent may be blocked (controlled with a probability parameter) by others when moving ahead, and this influence is highly dynamic.
We use the Centralized Q-Learning + Force as the high-level policy, of which the results are shown as Figure 8.
Compared to the results shown in Appendix E.3, this modification leads to slight instability in the training process, but we can still get significant performance improvement with the identified pairwise options.
Especially, in 8(c), our solution still perform well (i.e., reach the target in 60 − 100 steps) in cases where the collision influence is heavy (i.e., collisions occur in up to 63% of the steps).
[Figure]
Mujoco continuous control tasks: Finally, in order to show the effectiveness and generality of the deep learning extension of our approach (introduced in Section 4.3) on tasks with infinite-scale state space, we compare it with the baselines on the Mujoco tasks shown as Figure 4(c) and 4(d), of which the results are shown in Figure 9.
On one hand, the comparison between our method and MARL (including MAPPO,
MADDPG, MAA2C) without using options shows that abstracting some control series into options and integrating them with a hierarchical training framework can reduce the learning difficulty of continuous control tasks.
On the other hand, the comparison with agents using single-agent options shows that discovery and centralized adoption of multi-agent options can aid the exploration in their joint state space, and lead them toward the “bottleneck" joint states which are essential to complete the cooperation tasks.
Note that the options adopted are pre-trained with SAC for 1 × 10 4 episodes, so we have pretrained the baselines:
MAA2C, MADDPG, and MAPPO, for the same number of episodes to maintain fairness.
Conclusion And Future Works
In this paper, we propose to approximate the joint state space in MARL as a Kronecker graph and estimate its Fiedler vector using the Laplacian spectrum of the individual agents’ state transition graphs.
Based on the approximation of the Fiedler vector, multi-agent covering options are constructed, containing multiple agents’ temporal action sequence towards the sub-goal joint states which are usually infrequently visited, so as to accelerate the joint exploration in the environment.
Moreover, we propose a scalable extension of our algorithm so that it can work on tasks with large-scale or continuous state spaces, in order to improve the generality.
Further, we propose algorithms to adopt these multi-agent options in MARL, using centralized, decentralized, and group-based strategies, respectively.
We empirically show that agents with multi-agent options have significantly superior performance than agents relying on single-agent options or no options.
One limitation of our method is that there will be non-negligible differences between our approximation ⊗ ni =1 G i and the true joint state transition graph (cid:101) G , if the state transitions of an agent are hugely influenced by the others.
Therefore, developing a specific approximation error bound as a supplement to Theorem 4.1 can be an interesting future direction.
On the other hand, to utilize the multi-agent options, agents need to share their views (i.e., observations) since the initiation and termination conditions of the multi-agent options are defined based on the joint observations.
Communication among the agents is required in cases where the joint observations are not directly available.
Thus, integrating our method with efficient communication strategies in MARL [41] to mitigate the communication complexity can be meaningful future works.
[-since ⊗ ni =
-]
[-=
-]
[-=
-]
[-=
-]
[-Based on this, we can get that (cid:12)(cid:12) (cid:12) (cid:81) n i =
-]
[-The algebraic connectivity of our estimation ⊗ 2 i =
-]