S INGLE - AGENT TO M ULTI - AGENT IN D EEP RL
A BSTRACT
OW QMIX, CW QMIX, QTRAN, QMIX, and VDN are SOTA algorithms for solving Dec-POMDPs.
OW QMIX, CW QMIX, QTRAN, QMIX, and VDN failed to solve complex agents’ cooperation domains such as box-pushing.
We give a 2stage algorithm to solve such problems.
In the first stage, we solve a single-agent problem (POMDP) and get policy traces.
In the second stage, we solve the multiagent problem (Dec-POMDP) with the single-agent policy traces.
Single-agent to multi-agent has a clear advantage over OW QMIX, CW QMIX, QTRAN, QMIX, and VDN on complex agents’ cooperative domains.
I NTRODUCTION
Reinforcement learning [10] is an area of machine learning concerned with how intelligent agents ought to take action in an environment in order to maximize the notion of cumulative reward.
Reinforcement learning [12] does not require labeled data to make learning.
Reinforcement learning algorithms aims to find a balance between exploration and exploitation.
A Markov-decision-process (MDP) deals with a single agent in a fully-observable world.
A Partially-observable Markov decision process (POMDP) deals with a single agent in a partially-observable world.
A decentralized partially-observable Markov decision process (Dec-POMDP) deals with a multi-agent in a partiallyobservable world.
B ACKGROUND 2. M ARKOV DECISION PROCESS (MDP)
A Markov decision process [14] is a 4-tuple ( S , A , p a , R a ) where
• S is the state space •
A is the action space • P a ( s , s ′ ) =
Pr ( s t +1 = s ′
| s t = s , a t = a ) is the probability
that action
a in state s at time t will lead to state s ′ at time t +• R a ( s , s ′ ) is the immediate reward received after transforming from state s to state s ′ , due to action a
A policy function π is a mapping from state space to action space.
2. P ARTIALLY OBSERVABLE M ARKOV DECISION PROCESS (POMDP)
A discrete-time POMDP [16] models the relationship between an agent and its environment.
Formally, a POMDP is a 7-tuple ( S , A , T , R , Ω , O , γ ) where
• S is the set of states •
A is the set of actions • T is a set of conditional transition probabilities between states • R : S ×
A → R is the reward function
• Ω is a set of observations
O is a set of conditional observation probabilities • γ ∈ [0, 1) is the discount factor
At each time period, the environment is in some state s ∈ S .
The agent takes an action a ∈ A , which causes the environment to transition to state s ′ with probability T ( s ′ | s , a ).
At the same time, the agent receives an observation o ∈ Ω which depends on the new state of the environment, s ′ , and on the just taken action a , with probability
O
( o | s ′ , a ).
Finally, the agent receives a reward r equal to
R ( s , a ).
2. D ECENTRALIZED PARTIALLY OBSERVABLE M ARKOV DECISION PROCESS
(D [Equation] POMDP
[Equation] A Dec-POMDP [15] is a 7-tuple ( S , { A i } , T , R , { Ω i } , O , γ ) where
• S is the set of states •
A
i is the set of actions for agent i , with A = × i
A i is the set of joint actions • T is a set of conditional transition probabilities between states, T ( s , a , s ′ )
=
P ( s ′ | s , a ) • Ω i is a set of observations for agent i , with Ω = ×
i Ω i is the set of joint observations
• O is a set of conditional observation probabilities O ( s ′ , a , o ) =
P ( o | s ′ , a ) • γ ∈ [0, 1] is the discount factor
At each time step, each agent takes an action a t ∈
A t , the state updates based on the transition function T ( s , a , s ′ ), each agent observes an observation based on the observation function O ( s ′ , a , o ) and a reward is generated for the whole team based on the reward function R ( s , a ).
2. Q- LEARNING
Q-learning [11] is a model-free reinforcement learning algorithm.
It does not require a model of the environment, and it can handle problems with stochastic transitions and rewards without requiring adaptations.
For any finite Markov decision process (FMDP), Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state.
Initialize Q ( s , a ), ∀ s ∈ S , a ∈ A ( s ) arbitrarily 2.
Initialize Q ( terminal − state , . )
=3.
For each episode do (a) Initialize S (b) Chose A from S using policy derived from Q (c) Take action A , observe R , S ′
(d) Q ( S , A ) ← Q ( S , A )
+ α [ R + γ max a Q ( S ′ , a ) − Q
( S , A )] (e) S ← S ′ 4.
Until S is terminal
2. V ALUE DECOMPOSITION NETWORK (VDN)
VDN [1] is a multi-agent deep reinforcement-learning algorithm for Dec-POMDP.
VDN studies the problem of cooperative MARL with a single joint reward signal.
VDN is built on purely independent
DQN-style agents using a deep neural network of LSTM.
The main assumption of VDN is that the joint value function of the system is decomposed into a value function agent sum known as additivity.
The multi-agent Q-values formula of VDN is
Q (( h 1 , h 2 , ..., h d ), ( a 1 , a 2 , ..., a d )) ≈ (cid:80) si =1 (cid:98) Q i ( h i , a i )
Where each (cid:98) Q i depends on the agent’s local observations.
2. QMIX
QMIX [2] is a multi-agent deep reinforcement-learning algorithm for Dec-POMDP.
QMIX consists of agent networks representing each Q a , and a mixing network that combines them into Q tot in a complex non-linear way that ensures consistency between the centralized and decentralized policies.
QMIX ensures that global argmax performed on Q tot yields the same results as a set of individual argmax operations on each Q a by enforcing the monotonicity constraint of δ Q tot / δ Q a ≥ 0 ∀ a .
QMIX agent’s network is DRQN-style and receives single observation and action.
The mixing network is a feedforwar neural network that receives agent network output, mixing them monotonically producing Q tot .
QMIX hyper network takes the state as input producing mixing network layer weights, and consists of a single linear layer, and then an absolute activation function.
2. QTRAN
QTRAN [3] is a multi-agent deep reinforcement-learning algorithm for Dec-POMDP.
QTRAN consists of 3 deep neural networks: joint action value, individual action value, and a state value.
QTRAN transforms the original joint action value function Q jt to Q ′ jt to share optimal joint actions with Q jt .
Q ′
jt formula is Q ′ jt ( τ , π )
= (cid:80) N i =
1 Q i ( τ i , u i ).
conditioned on both additivity and monotonicity constraints.
QTRAN state-value network is a dueling network that takes the state as input and produces the state value as output.
QTRAN combines 3 loss functions by weight: : td the TD-error loss, and
L opt and L nopt losses for factoring Q jt by Q i condition.
QTRAN uses the DQN algorithm to update networks.
2. W EIGHTED QMIX
Weighted QMIX [4] is a multi-agent deep reinforcement-learning algorithm for Dec-POMDP.
Weighted QMIX is based on QMIX, adding weights to the loss with 2 different versions.
This weighted projection allows more emphasis to be placed on better joint actions.
The Weighted QMIX projection operator is π w Q : = argmin q ∈ Q mix (cid:80) u ∈ U w ( s , u )( Q ( s , u ) - q ( s , u )) ˆ
Idealized-Centrally Weightening (CW) down-weight every suboptimal action.
Optimistic Weighting (OW) assigns a higher weight to joint actions that are underestimated relative to Q.
2. C URRICULUM L EARNING
Curriculum learning [9] is a 3-part approach consisting of task generation, sequencing, and transfer learning.
Task generation is the problem of creating intermediate tasks to be part of a curriculum.
DQN [5] uses a reply buffer to store past state action reward experience tuples.
In transfer learning, the agent is first trained on source tasks and transferred the knowledge acquired to aid in solving the target.
2.9. C O - LEARNING
Co-learning [9] is a multi-agent curriculum learning in which agents interact with each other in the environment to make learning.
AlphaGo [6] uses self-play to speed up the learning process.
AlphaGo is focused on the game of Go.
Single-agent to multi-agent algorithm focuses on general collaborative multi-agent problems.
2.9. R OBUST A DVERSARIAL R EINFORCEMENT LEARNING (RARL)
Asymmetric self-play [7] focuses on a game of hiding and seek [9].
The Hiders must hide, while the Seekers must find the Hiders.
Both teams must use the game’s physics rules.
The authors show that as one team converged on a successful strategy, the other team was pressured to learn a counter-strategy.
This process was repeated.
Asymmetric self-play uses a multi-agent policy for multi-agent problem training.
Single-agent to multi-agent uses single-agent policy traces for multiagent problem training.
2.9. C OLLABORATIVE M ULTI - AGENT REINFORCEMENT LEARNING (MARL)
CM3 [8] takes a two-stage approach.
In the first stage, a single agent is trained without the presence of other agents.
This is done by inducing a new MDP that removes all dependencies on agent interactions and training a network on this subspace.
Then in the second stage, cooperation is learned by adding the parameters for the other agents into the network.
CM3 chooses a single agent out of all agents for training and then adds the parameters of the other agents into the neural network.
Single-agent to multi-agent uses a single-agent team problem for training and then uses the optimal single-agent team policy traces for the multi-agent problem actions’ choosing.
A LGORITHM
A multi-agent problem which is modeled as Dec-POMDP ρ multi has the following components
• Multi-agent actions vector • Multi-agent observations
vector • Multi-agent task
The multi-agent problem has the following
All agents have the same actions 2.
All agents have the same observation variables
A team problem ρ team is a single-agent problem modeled as POMDP has the following components
• Single-agent actions vector • Single-agent observations vector • Single agent task
The single-agent problem has the following
The agent’s actions are the same actions of each agent of the multi-agent problem 2.
The agent observation variables are the same observation variables of each agent of the multi-agent problem
We use curriculum learning for our algorithm.
As task the generation we take the existing problem of the multi-agent problem defined as Dec-POMDP and make a new problem of single-agent team problem defined as POMDP.
As sequencing, we do the following order
Solve single-agent team problem defined as POMDP and output a policy trace π team 2.
Solve the multi-agent problem defined as Dec-POMDP with policy traces π team .
As transfer learning in the multi-agent problem, we chose an action by a sampled policy from π team policy traces in the probability of α where 0 < α < 1 .
3. S INGLE - AGENT TO MULTI - AGENT
We solve the single-agent problem using Weighted QMIX and get a policy trace π team as output.
We initialize as the α probability of the multi-agent model to choose an action by a sampled policy β from π team policy traces.
We train the multi-agent model with π team policy traces.
π =
( a , s , o , r ) = solve − team () 2.
while steps < total − steps do (a) run − episode ( a single , π ) (b) for index in range ( n − training ) do i. batch = buffer .
sample () ii.
train ( batch )
3. R UN EPISODE
An episode runs until it terminates by getting an indicator from the environment.
At episode running, we choose an action by single agent policy traces pi team and by the model actions-selector.
On probability α we chose actions with π team policy traces, on probability 1 − α
we chose actions with a model actions-selector.
Finally, we return the tuple and the total episode rewards.
3. S ELECT SINGLE - AGENT ACTIONS
We initiate an empty vector of actions actions .
We loop over each agent from the total agents.
The multi-agent state is s multi .
The multi-agent observation is obs multi .
We sample a random single agent policy β from π team .
We get the corresponding single agent state s i for s multi .
We get the corresponding single-agent observation obs i for obs multi .
We get the single agent action a i,single for s i , obs i and β .
We get the corresponding multi-agent action a i,multi for a i,single .
We append a i,multi to actions .
D OMAINS 4. B OX - PUSHING
2-agent in a 1-dimensional grid with a state of agents’ location and an initial hidden observation of a randomized boxes’ location.
The agent has an action of move-left and move-right for moving in the grid, a sense action to get the box’s location, and a push action to push the box left.
The agents will succeed in pushing the box only if both of them are located at the box location and both of them make the push action at the same time step.
The agent’s target is to push all the boxes to the right grid edge.
The agents get a prize of -1 for each action, a penalty of -10 for taking a push action where one of them is not at the box location, and a prize of 10 if the box is reached the right grid edge.
push action is a stochastic action with a probability of success of .8.
The grid size is 3.
The total number of boxes is 1. 4.1. B OX - PUSHING COLLABORATIVE
Identical to box pushing with an additional collaborative action of sense.
4.1. B OX - PUSHING STOCHASTIC
Identical to box pushing with an additional stochastic action of sense with a success probability of
R OCK SAMPLE
2-agent in a 1-dimensional grid with a state of agents’ location and an initial hidden observation of randomized rocks’ locations and conditions.
The agents have an action of move-left and move-right for moving in the grid, a sense action to get the rocks’ location and condition, and a sample action to sample the rock.
The agents will succeed in sampling the rock only if both of them are located at the rock location and both of them make the sample action at the same time step.
The agent’s target is to sample all the rocks in good condition.
The agents get a prize of -1 for each action, a penalty of -10 for taking a sample action where one of them is not at the rock’s location, and a prize offor a good rock sample.
The grid size is 5.
The total of rocks is 2.
4.2. R OCK - SAMPLE STOCHASTIC
Identical to rock-sample with a stochastic sense action.
The grid sizes are 3 and 4.
4. P REDATOR - PREY
2-agent in a 1-dimensional grid with a state of agents’ location and an initial hidden observation of a randomized prey’s location.
The agents have an action of move-left and move-right for moving in the grid, a sense action to get the prey’s location, and a hunt action to hunt the prey.
The agents will succeed in hunting the prey only if both of them are located at the prey location and both of them make the hunt action at the same time step.
The agents’ target is to hunt all the prey.
The agents get a prize of -1 for each action, a penalty of -10 for taking a hunt action where one of them is not at the prey’s location, and a prize of 10 if the prey is hunted.
The hunt action is a stochastic action with a success probability of .8.
The grid size is 2.
The total prey is 1.
R ESULTS
Each experiment is composed of 5 different random experiments.
Each graph is composed of solid lines and shaded lines.
A solid line is a mean.
The shaded lines are mean plus standard error and mean minus standard error.
Each experiment gives 2 different graphs.
The left graphs are single-agent problems of Single-agent to
5. B OX - PUSHING Grid size 3, total agents 2, total boxes
The single-agent problem has converged at an 80K time step from Single-agent to multi-agent.
Multi-agent problem Single-agent to multi-agent has been converged at 600K time step.
Multiagent problem QMIX has been converged at 800K time step.
Multi-agent problems OW QMIX, CW
QMIX, QTRAN, and VDN have not converged.
Single-agent to multi-agent has a clear advantage over competitors.
5.1. B OX - PUSHING COLLABORATIVE Grid size 3, total agents 2, total boxes
The single-agent problem has converged at a 50K time step from Single-agent to multi-agent.
Multiagent problem Single-agent to multi-agent has been converged at 600K time step.
Multi-agent problems OW QMIX, CW QMIX, QMIX, QTRAN, and VDN have not converged.
There is a clear advantage to Single-agent to multi-agent over competitors.
5.1. B OX - PUSHING STOCHASTIC Grid size 3, total agents 2, total boxes
The single-agent problem has converged at a 70K time step from Single-agent to multi-agent.
Multiagent problem Single-agent to multi-agent has been converged at 600K time step.
Multi-agent problems OW QMIX, CW QMIX, QMIX, QTRAN, and VDN have not converged.
There is a clear advantage to Single-agent to multi-agent over competitors.
5.1. B OX - PUSHING STOCHASTIC COLLABORATIVE Grid size 3, total agents 2, total boxes
The single-agent problem has converged at an 80K time step from Single-agent to multi-agent.
Multi-agent problem Single-agent to multi-agent has been converged at 600K time step.
Multiagent problems OW QMIX, CW QMIX, QMIX, QTRAN, and VDN have not converged.
There is a clear advantage to Single-agent to multi-agent over competitors.
5.1. R OCK - SAMPLE Grid size 5, total agents 2, total rocks
The single-agent problem has converged at a 30K time step from Single-agent to multi-agent.
The multi-agent problem has been convergened at a 600K time step from VDN.
Multi-agent problems have converged at a 900K time step from Single-agent to multi-agent, OW QMIX, and QMIX.
The multi-agent problem has not converged from QTRAN.
There is no clear advantage to Single-agent to multi-agent over competitors.
5.1. R OCK - SAMPLE STOCHASTIC Grid size 3, total agents 2, total rocks
The single-agent problem has converged at a 70K time step from Single-agent to multi-agent.
Multiagent problems have been converted at 300K from QMIX, and VDN.
The Multi-agent problem has converged at 600K from Single-agent to multi agent.
The multi-agent problem has not converged from QTRAN, OW QMIX, and CW QMIX.
There is no clear advantage to Single-agent to multiagent over competitors.
5.1. R OCK - SAMPLE STOCHASTIC Grid size
, total agents , total rocks
The single-agent problem has converged at a 60K time step from Single-agent to multi-agent.
The multi-agent problem has been converged at 300K from VDN, and QMIX.
The Multi-agent problem has converged at 600K from Single-agent to multi agent.
The multi-agent problem has not converged from OW QMIX, CW QMIX, and QTRAN.
There is no clear advantage to Single-agent to multiagent.
5.1. P REDATOR - PREY Grid size 2, total agents 2, total preys
The single-agent problem has converged at a 40K time step from Single-agent to multi-agent.
The multi-agent problem has been converged at 200K steps from OW QMIX, QMIX, VDN, and
QTRAN.
The multi-agent problem has not converged from Single-agent to multi-agent, and OW
QMIX.
Single-agent to multi-agent has no clear advantage over competitors.
D ISCUSSION 6. A VERAGE REWARDS
Single-agent to multi-agent has a clear advantage on box-pushing domains.
Single-agent to multiagent has no advantage on rock-sample and predator-prey domains.
Box-pushing domains require complex agents’ cooperation.
The optimal policy is both agents sense, then both agents move to the box location, and then both agents push the box collaboratively and move right repeatedly.
Predatorprey and rock-sample do not require complex agents’ cooperation.
The optimal policy of predatorprey is both agents’ sense, then both agents move to the prey location and hunt it collaboratively.
The optimal policy of rock-sample is for both agents to sense, then both agents move to the good rock location and sample it collaboratively.
6. P OLICIES QUALITY
Single-agent to multi-agent has a clear advantage on policy quality on box-pushing domains.
Singleagent to multi-agent recover the optimal policy on all the box-pushing domains.
There is onlycase for competitors to recover the optimal policy on the box-pushing domain from QMIX.
Singleagent to multi agent has no clear advantage on policy quality on rock sample and predator-prey domains.
At rock-sample domains, all Single-agent to multi-agent, VDN, and QMIX recover the optimal policy.
At the predator-prey domain, CW QMIX, QMIX, VDN, and QTRAN recover a stable optimal policy but Single-agent to multi agent, and OW QMIX recovers a not stable optimal policy.
C ONCLUSION
Single-agent to multi-agent has a clear advantage on complex agents’ collaboration domains over
CW QMIX, OW QMIX, QTRAN, QMIX, and VDN.
The advantage of Single-agent to multi-agent-based on the complexity gap between single-agent problems to multi-agent problems.
Solving a single-agent team problem requires much fewer resources than solving a multi-agent problem.
F URTHER WORK
Single-agent to multi-agent has not been tested on domains with heterogeneous agents.
To give good results we suggest training it on multiple single-agent problems, one problem for each agent, and then applying transfer learning through action choosing.
S OURCE CODE
https://github.com/nitsansoffair/rwqmix/tree/heuristics