SA2MA in Deep RL bgu ac
Abstract
WQMIX, QMIX, QTRAN, and VDN are SOTA algorithms for Dec-POMDP.
All of them cannot solve complex agents’ cooperation domains.
We give an algorithm to solve such problems.
In the first stage, we solve a single-agent problem and get a policy.
In the second stage, we solve the multi-agent problem with the single-agent policy.
SA2MA has a clear advantage over all competitors in complex agents’ cooperative domains.
Introduction
RL [ Wikipedia contributors, 2022d ] is an area of ML concerned with how intelligent agents ought to take action in an environment in order to maximize the cumulative reward.
RL [ Kaelbling et al. , 1996 ] does not require labeled data to make learning.
RL algorithms aim to find a balance between exploration and exploitation.
Background 2. MDP
An MDP [ Wikipedia contributors, 2022a ] is a 4-tuple ( S, A, P a , R a ) where S is the state space,
A is the action space, P a is the probability that action a in state s at time t will lead to state s t +1 at time t
+ 1 , R a is the immediate reward received after transforming from state s t to state s t +1 .
A policy function π is a mapping from state space to action space.
2. POMDP
A POMDP [Wikipedia contributors, 2022b] is a 7-tuple ( S, A, T, R, Ω , O, γ ) where S is the set of states, A is the set of actions, T is a set of transition probabilities between states, R is the reward function, Ω is a set of observations, O is a set of observation probabilities, γ ∈ [0 , 1] is the discount factor.
At each time period, the environment is in some state s t .
The agent takes an action a , which causes the environment to transition to state s t +1 with probability T ( s t +1 | s t , a ) .
At the same time, the agent receives an observation o which depends on the new state of the environment, s t +1 , and on the just taken action a , with probability
O ( o | s t , a ) .
Finally, the agent receives a reward r equal to R ( s t , a ) .
2. Dec-POMDP
A Dec-POMDP [Wikipedia contributors, 2020] is a 7-tuple ( S, { A i } , T, R, { Ω i } , O, γ ) where S is the set of states, A i is the set of actions for agent
i , A = × i A
i is the set of joint actions, T is a set of transition probabilities between states, Ω i is a set of observations for agent
i , Ω = × i Ω
i is the set of joint observations, O is a set of observation probabilities, γ ∈ [0 , 1] is the discount factor.
At each time step, each agent takes an action a t , the state updates based on the transition function T , each agent observes an observation based on the observation function O and a reward is generated for the whole team based on the reward function R .
2. Q-learning
Q-learning [ Wikipedia contributors, 2022c ] is a model-free RL algorithm.
For any finite MDP, Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state.
2. VDN
VDN [ Sunehag et al. , 2017 ] is a multi-agent deep RL algorithm for Dec-POMDP.
VDN studies the problem of cooperative MARL with a single joint reward signal.
VDN is built on purely independent DQN-style agents using a deep neural network of LSTM.
The main assumption of VDN is that the joint value function of the system is decomposed into a value function of agents sum.
The multi-agent Q-values formula of VDN is Q (( h 1 , h 2 , ..., h d ) , ( a 1 , a 2 , ..., a d )) ≈ [Equation] Q i
[Equation]
h i , a i ) where each (cid:98) Q i depends on the agent local observations.
2. QMIX
QMIX [Rashid et al. , 2020a] is a multi-agent deep RL algorithm for Dec-POMDP.
QMIX consists of agent networks representing each Q a , and a mixing network that combines them into Q tot in a complex non-linear way that ensures consistency between the centralized and decentralized policies.
QMIX ensures that global argmax performed on Q tot yields the same results as a set of individual argmax operations on each Q a by enforcing a monotonicity constraint.
Weighted QMIX [Rashid et al. , 2020b] is a multi-agent deep
RL algorithm for Dec-POMDP.
Weighted QMIX is based on QMIX, adding weights to the loss.
This weighted projection allows more emphasis to be placed on better joint actions.
CW QMIX down-weight every suboptimal action and OW
QMIX assigns a higher weight to joint actions that are underestimated relative to Q.
2. Curriculum Learning
Curriculum learning [Narvekar et al. , 2020] is a 3-part approach consisting task generation, sequencing, and transfer learning.
Task generation is the problem of creating intermediate tasks to be part of a curriculum.
As sequensing, DQN [Mnih et al. , 2013] uses a reply buffer to store past state action reward experience tuples.
In transfer learning, the agent is first trained on source tasks and transferred the knowledge acquired to aid in solving the target.
Co-learning
Co-learning [ Narvekar et al. , 2020 ] is a multi-agent curriculum learning in which agents interact with each other in the environment to make learning.
AlphaGo [ Silver et al. , 2017 ] uses self-play to speed up the learning process.
AlphaGo is focused on the game of Go but SA2MA focuses on collaborative multi-agent problems.
Robust adversarial RL
Asymmetric self-play [ Sukhbaatar et al. , 2017 ] focuses on a game of hide and seek [ Sukhbaatar et al. , 2017 ] .
The Hiders must hide, while the Seekers must find the Hiders.
Both teams must use the physics rules of the game.
The authors show that as one team converged on a successful strategy, the other team was pressured to learn a counter-strategy.
This process was repeated.
Asymmetric self-play uses a multi-agent policy for the multi-agent training while SA2MA uses a single-agent policy.
Collaborative MARL
CM3 [ Yang et al. , 2018 ] is a two-stage approach.
In the first stage, a single agent is trained without the presence of other agents.
This is done by inducing a new MDP that removes all dependencies on agent interactions and training a network on this subspace.
In the second stage, cooperation is learned by adding the parameters for the other agents into the network.
CM3 chooses a single agent out of all agents for training and then adds the parameters of the other agents into the neural network while SA2MA train single-agent problem for actions choosing in the multi-agent problem.
Algorithm 3. Mapping
A multi-agent problem has multi-agent actions vector, multiagent observations vector and multi-agent task.
In the multiagent problem all agents have the same actions and all agents have the same observation variables.
A team problem is a single-agent problem with a single-agent actions vector, single-agent observations vector and single agent task.
For the mapping, in the single-agent problem the agent actions are the same actions of each agent from the multi-agent problem, and the agent observation variables are the same of each agent from the multi-agent problem. 3. CL
We use curriculum learning for our algorithm.
In task the generation we take the multi-agent problem and make a new single-agent team problem.
In sequencing, we solve singleagent team problem and output a policy π for solving the multi-agent problem.
In transfer learning in the multi-agent problem we chose an action by a policy sample from π in probability α .
3. SA2MA
First solve the single-agent problem using WQMIX output policy π .
Then train the multi-agent problem with the policy π .
3. Run episode
At episode run we choose an action from the single agent policy π and from WQMIX actions-selector.
By probability α we chose actions from the policy π , otherwise from WQMIX actions-selector.
3. Select single-agent actions
We iterate over all agents.
We sample single agent policy β from π and get the corresponding single agent state, observation and action.
Then we save the action.
Finally we return all actions.
Domains 4. Box-pushing
Multiple agents in a grid the state is agents location and the observation is boxes location.
The agent has an action of move left and right, a sense action to get the box location, and a stochastic push action to push the box that can fail.
The agents will succeed pushing the box only if both are located at the box location and both make the push action at the same time.
The target is to push all the boxes to the edge.
Box-pushing collaborative
Box pushing with an additional collaborative constraint for sense: sense action allowed only collaboratively.
Box-pushing stochastic
Box pushing collaborative with an additional noise for sense.
4. Rock-sample
Multiple agents in a grid with a state of agents location and an initial hidden observation of randomized rocks locations and conditions.
The agents have an action of move left and right, a sense action to get the rocks location and condition, and a sample action to sample the rock.
The agents will succeed sampling the rock only if both of them are located at the rock location and both of them make the sample action at the same time step.
The target is to sample all the rocks in good condition.
Rock-sample with an additional noise for sense action.
4. Predator-prey
Multiple agents in a grid with a state of agents location and an initial hidden observation of preys location.
The agents have an action to move left and right, a sense action to get the location of the prey, and a stochastic hunt action to hunt the prey that can fail.
The agents will succeed in hunting the prey only if both of them are located at the prey location and both of them make the hunt action at the same time step.
The target is to hunt all the preys.
Results
Each experiment is composed of 5 different random experiments.
Each graph is composed of solid lines and shaded lines.
A solid line is the experiments mean.
The shaded lines are experiments mean plus standard error and mean minus standard error.
Each experiment has two graphs.
The left graph is a single-agent experiment of Single-agent to multiagent.
The right graph is a multi-agent experiment of all algorithms.
5. Box-pushing Grid size 3, total agents 2, total boxes
The single-agent problem has converged in 80K time step.
In the multi-agent problem SA2MA has been converged at 600K time step, QMIX at 800K, OW QMIX, CW QMIX,
QTRAN, and VDN have been not converged.
Box-pushing collaborative Grid
size total agents total boxes
The single-agent problem has converged at a 50K time step.
In multi-agent problem SA2MA has been converged at 600K time step, OW QMIX, CW QMIX, QMIX, QTRAN, and VDN have not converged. Box-pushing
stochastic
Grid
size , total agents total boxes
The single-agent problem has converged at a 70K time step.
In the multi-agent problem SA2MA has been converged at 600K time step, OW QMIX, CW QMIX, QMIX, QTRAN, and VDN have not converged. Box-pushing
stochastic collaborative Grid size 3, total agents 2, total boxes
The single-agent problem has converged at an 80K time step.
In the multi-agent problem Single-agent to multi-agent has been converged at 600K time step, OW QMIX, CW
QMIX, QMIX, QTRAN, and VDN have not converged.
Rock-sample
Grid size 5, total agents 2, total rocks
The single-agent problem has converged at a 30K time step.
The multi-agent problem has been convergened at a 600K time step from VDN, SA2MA, OW QMIX, and QMIX at 900K. QTRAN has been not converged. Rock-sample stochastic
Grid size 3, total agents 2, total rocks
The single-agent problem has converged at a 70K time stept.
The multi-agent problems have been converted at 300K with QMIX and VDN, 600K with SA2MA and has been not converged with QTRAN, OW QMIX and CW QMIX.
Grid size 4, total agents 2, total rocks
The single-agent problem has converged at a 60K time step.
The multi-agent problem has been converged at 300K with VDN and QMIX, SA2MA with 600K and has not converged with OW QMIX, CW QMIX or QTRAN.
Predator-prey
Grid size 2, total agents 2, total preys
The single-agent problem has converged at a 40K time step.
The multi-agent problem has been converged at 200K steps with OW QMIX, QMIX, VDN, and QTRAN but has been not converged with SA2MA or OW QMIX.
Discussion 6. Average rewards
SA2MA has a clear advantage on box-pushing domains which require complex agents cooperation, but has no advantage on rock-sample or predator-prey which do not.
The optimal policy for box-pushing is both agents sense, then both agents move to the box location, then both agents push the box collaboratively while moving right repeatedly.
The optimal policy of predator-prey is both agents sense, then both agents move to the prey location and hunt it collaboratively.
The optimal policy of rock-sample is both agents sense, then both agents move to the good rocks location and sample it collaboratively.
6. Policies quality
SA2MA has a clear advantage on policy quality on boxpushing domains.
SA2MA recover the optimal policy in all the box-pushing domains.
There is only one case for competitors to recover the optimal policy on the box-pushing domain which is done by QMIX.
SA2MA has no clear advantage on policy quality on rock sample or predator-prey.
At rocksample domains, SA2MA, VDN and QMIX recover the optimal policy.
At the predator-prey domain CW QMIX, QMIX, VDN, and QTRAN recover the optimal policy but SA2MA and OW QMIX do not.
Conclusion
SA2MA has a clear advantage on complex agents collaboration domains over all competitors.
The advantage based on the complexity gap between the single-agent problem to the multi-agent.
Solving a singleagent team problem requires much less resources than solving a multi-agent problem.
Further work
SA2MA has been not tested on domains with heterogeneous agents.
Source code
https://github.com/nitsansoffair/rwqmix/tree/heuristics plus