V ARIATIONAL A UTOENCODERS FOR O PPONENT M ODELING IN M ULTI -A GENT S YSTEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
Multi
-agent systems exhibit complex behaviors that emanate from the interactions of multiple agents in a shared environment
In this work
, we are interested in controlling one agent in a multi-agent system and successfully learn to interact with the other agents that have ﬁxed policies
Modeling the behavior of other agents (opponents) is essential in understanding the interactions of the agents in the system
By taking advantage of recent advances in unsupervised learning
, we propose modeling opponents using variational autoencoders
Additionally many existing methods in the literature assume that the opponent models have access to opponent’s observations and actions during both training and execution.
To eliminate this assumption, we propose a modiﬁcation that attempts to identify the underlying opponent model, using only local information of our agent such as its observations
, actions, and rewards
The experiments indicate that our opponent modeling methods achieve equal or greater episodic returns in reinforcement learning tasks against another modeling method.
I NTRODUCTION
In recent years, several promising works (Mnih et al., 2015; Schulman et al., 2015a; Mnih et al., 2016) have arisen in deep reinforcement learning (RL), leading to fruitful results in single-agent scenarios.
In this work, we are interested in using single-agent RL in multi-agent systems, where we control one agent and the other agents (opponents) in the environment have ﬁxed policies.
The agent should be able to successfully interact with a diverse set of opponents as well as generalize to new unseen opponents.
One effective way to address this problem is opponent modeling.
The opponent models output speciﬁc characteristics of the opponents based on their trajectories.
By successfully modeling the opponents, the agent can reason about opponents’ behaviors and goals and adjust its policy to achieve the optimal outcome.
There is a rich literature of modeling opponents in the multiagent systems (Albrecht & Stone, 2018).
Several recent works have proposed learning opponent models using deep learning architectures (He et al., 2016; Raileanu et al., 2018;
Grover et al., 2018a; Rabinowitz et al., 2018).
In this work, we focus on learning opponent models using Variational Autoencoders (VAEs) (Kingma & Welling, 2014).
This work is, to the best of our knowledge, the ﬁrst attempt to use VAEs in multi-agent scenarios.
VAE are generative models that are commonly used for learning representations of the data, and various works use them in RL for learning representations of the environment (Igl et al., 2018;
Ha & Schmidhuber, 2018; Zintgraf et al., 2019).
We ﬁrst propose a VAE for learning opponent representations in multi-agent systems based on the opponent trajectories.
A shortcoming of this approach and most opponent modeling methods, as will be presented in Section 2, is that they require access to opponent’s information, such as observations and actions, during training as well as execution.
This assumption is too limiting in the majority of scenarios.
For example, consider Poker, where each agent never has access to the opponent’s observations.
Nevertheless, during Poker, humans can reason about the opponent’s behaviors and goals using only their local observations.
For example, an increase in the table’s pot could mean that the opponent either holds strong cards or is blufﬁng.
Based on the idea that an agent can reason about an opponent’s model using its observations, actions, and rewards in a recurrent fashion, we propose a second VAE-based architecture.
The encoder of the VAE learns to represent opponents’ models conditioned on only local information removing the requirement to access the opponents’ information during execution.
To summarize our contribution, in this work, we explore VAEs for opponent modeling in multiagent systems.
We are not interested in VAEs as generative models but as methods for learning representations.
We evaluate our proposed methodology using a toy example and the commonly used Multi-agent Particle Environment (Mordatch & Abbeel, 2017).
We evaluate the quality of the learned representations, and the episodic returns that RL algorithms can achieve.
The experiments indicate that opponent modeling without opponents’ information can perform the same or even better in RL compared to models that access the opponent’s information.
R ELATED W ORK
Learning Opponent Models.
In this work, we are interested in opponent modeling methods that use neural networks to learn representations of the opponents.
He et al.
(2016) proposed an opponent modeling method that learns a modeling network to reconstruct the opponent’s actions given the opponent observations.
Raileanu et al.
(2018) developed an algorithm for learning to infer opponents’ goals using the policy of the controlled agent.
Grover et al.
(2018a) proposed an encoder-decoder method for modeling the opponent’s policy.
The encoder learns a point-based representation of different opponents’ trajectories, and the decoder learns to reconstruct the’ opponent’s policy given samples from the embedding space.
Additionally, Grover et al.
(2018a) introduce an objective to separate embeddings of different agents into different clusters.
[Equation]
(1) where z + and z are embeddings of the same agent from two different episodes and embedding z − is generated from the episode of a different agent.
Rabinowitz et al.
(2018) proposed the Theory of mind Network (TomNet), which learns embedding-based representations of opponents for metalearning.
Tacchetti et al.
(2018) proposed RFM to model opponents using graph neural networks.
A common assumption among these methods, that this work aims to eliminate, is that access to opponents trajectories is available during execution.
Representation Learning in Reinforcement Learning.
Another topic that has received signiﬁcant attention, recently, is representation learning in RL.
Using unsupervised learning techniques to learn low dimensional representations of the MDP has led to signiﬁcant improvement in RL.
Ha &
Schmidhuber (2018) proposed a VAE-based and a forward model to learn state representations of the environment.
Hausman et al.
(2018) learned tasks embeddings and interpolated them to solve harder tasks.
Igl et al.
(2018) used a VAE for learning representation in partially-observable environments.
Gupta et al.
(2018) proposed MAESN, which learns Gaussian embeddings to represent different tasks during meta-training and manages to quickly adapt to new task during meta-testing.
The work of Zintgraf et al.
(2019) is closely related, where Zintgraf et al.
proposed a recurrent VAE model, which receives as input the observation, action, reward of the agent, and learns a variational distribution of tasks.
Rakelly et al.
(2019) used representations from an encoder for off-policy metaRL.
Note that, all these works have been applied for learning representations of tasks or properties of the environments.
On the contrary, our approach is focused on learning representations of the opponents.
B ACKGROUND 3. R EINFORCEMENT L EARNING
Markov Decision Processes (MDPs) are commonly used to model decision making problems.
An
MDP consists of the set of states S , the set of actions A , the transition function, P ( s (cid:48) | s , a ) , which is the probability of the next state, s (cid:48) , given the current state, s , and the action, a , and the reward function, r ( s (cid:48) , a , s ) , that returns a scalar value conditioned on two consecutive states and the intermediate action.
A policy function is used to choose an action given a state, which can be stochastic a ∼ π ( a | s ) or deterministic a
= µ ( s ) .
Given a policy π , the state value function is deﬁned as V ( s t ) =
E π [ (cid:80) Hi =
t γ i − t r t | s = s t ] and the state-action value (Q-value)
Q ( s t , a t ) =
E π [ (cid:80) Hi =
t γ i − t r t | s = s t , a = a t ] , where 0 ≤ γ ≤ 1 is the discount factor and H is the ﬁnite horizon of the episode.
The goal of RL is to compute the policy that maximizes state value function V , when the transition and the reward functions are unknown.
There is a large number of RL algorithms; however, in this work, we focus on two actor-critic algorithms; the synchronous Advantage Actor-Critic (A2C) (Mnih et al., 2016; Dhariwal et al., 2017) and the Deep Deterministic Policy Gradient (DDPG) (Silver et al., 2014; Lillicrap et al., 2015).
DDPG is an off-policy algorithm, using an experience replay for breaking the correlation between consecutive samples and target networks for stabilizing the training (Mnih et al., 2015).
Given an actor network with parameters θ and a critic network with parameter φ , the gradient updates are performed using the following equations.
[Equation]
On the other hand, A2C is an on-policy actor-critic algorithm, using parallel environments to break the correlation between consecutive samples.
The actor-critic parameters are optimized by:
[Equation] [Equation]
) where the advantage term, ˆ A , can be computed using the Generalized Advantage Estimation (GAE) (Schulman et al., 2015b).
3. V ARIATIONAL A UTOENCODERS
Consider samples from a dataset x ∈ X that are generated from some hidden (latent) random variable z based on a generative distribution p u ( x | z ) with unknown parameter u and a prior distribution on the latent variables, which we assume is a Gaussian with 0 mean and unit variance p ( z ) =
N ( z ; 0 , I ) .
We are interested in approximating the true posterior p ( z | x ) with a variational parametric distribution q w ( z | x ) =
N ( z ; µ , Σ , w ) .
Kingma & Welling (2014) proposed the
Variational Autoencoders (VAE) to learn this distribution.
Starting from the Kullback-Leibler (KL) divergence from the approximate to the true posterior D KL ( q w ( z | x ) (cid:107) p ( z | x )) , the lower bound on the evidence log p ( x ) is derived as:
[Equation]
The architecture consists of an encoder which receives a sample x and generates the Gaussian variational distribution p ( z | x ; w ) .
The decoder receives a sample from the Gaussian variational distribution and reconstructs the initial input x .
The architecture is trained using the reparameterization trick Kingma & Welling (2014).
Higgins et al.
(2017) proposed β -VAE, where a parameter β ≥is used to control the trade-off between the reconstruction loss and the KL-divergence.
[Equation]
A PPROACH 4. P ROBLEM F ORMULATION
We consider a modiﬁed Markov Game (Littman, 1994), which consists of N agents I = { 1 , 2 , ..., N } , the set of states S , the set of actions
A =
A 1 × A − 1 , the transition function
P : S ×
A × S → R and the reward function r : S ×
A × S → R N .
We consider partially observable settings, where each agent i has access only to its local observation
o i and reward r i .
Additionally, two sets of pretrained opponents are provided T
=
{ I − 1 ,m } m =
M m =
and G =
{ I − 1 ,m } m =
M m =1 , which are responsible for providing the joint action A − 1 .
Note that by opponent we refer to I − 1 ,m , which consists of one or more agents, independently from the type of the interactions (cooperative, mixed or competitive).
At the beginning of each episode, we sample a pretrained opponent from the set T during training or from G during testing.
Our goal is to train the agent 1 using RL, to maximize the average return against opponents from the training set, T , and generalize to opponents sampled from the test G .
Note, that when we refer to agent 1 we drop the subscript.
[Equation]
4. V ARIATIONAL A UTOENCODER WITH A CCESS TO O PPONENT ’ S I NFORMATION
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
We assume a number of K provided episode trajectories for each pretrained opponents j ∈ T ,
E ( j ) =
{ τ ( j,k ) − 1 } k
= K − 1 k = , where τ ( j,k ) −= { o − 1 ,t , a − 1 ,t } t =
H t =0 , and o − 1 ,t , a − 1 ,t are the observations and actions of the opponent at the time step t in the trajectory.
These trajectories are generated from the opponents in set T , which are represented in the latent space from the variable z and for which we assume there exists an unknown model p u ( τ − 1 | z ) .
Our goal is to approximate the unknown posterior, p ( z | τ − 1 ) , using a variational Gaussian distribution N ( µ , Σ ; w ) with parameters w .
We consider using a β -VAE for the sequential task:
[Equation]
We can further subtract the discrimination objective (equation 1) that was proposed by Grover et al.
(2018a).
Since the discrimination objective is always non-negative, we derive and optimize a lower bound as:
[Equation]
The discrimination objective receives as input the mean of the variational Gaussian distribution, produced by three different trajectories.
Despite the less tight lower bound, the discrimination objective will separate the opponents in the embedding space, which could potentially lead to higher episodic returns.
At each time step t , the recurrent encoder network generates a latent sample z t , which is conditioned on the opponent’s trajectory τ − 1 , : t , until this time step.
The KL divergence can be written as:
[Equation]
The lower bound consist of the reconstruction loss of the trajectory which involves the observation and actions of the opponent.
The opponent’s action, at each time step depends on its observation and the opponent’s policy, which is represented by the latent variable z .
We use a decoder that consists of fully-connected layers, however, a recurrent network can be used, if we instead assume that the opponent decides its actions based on the history of its observations.
Additionally, the observation at each time step depends only on the dynamics of the environment and the actions of the agents and not on the identity of the opponent.
Therefore, the reconstruction loss factorizes as:
[Equation]
From the equation above, we observe that the loss is the reconstruction of the opponent’s policy given the current observation and a sample from the latent variable.
Overall, our proposed VAE takes the form of a Conditional VAE (Sohn et al., 2015).
Figure
4. V ARIATIONAL A UTOENCODER WITHOUT A CCESS TO O PPONENT ’ S I NFORMATION

[Figure]
ode
[Figure]
t
[Figure]
In Sections 1 and 2, it was noted that most agent modeling methods assume access to opponent’s observations and actions is available both during training and execution.
To eliminate this assumption, we propose a VAE that uses a parametric variational distribution which is conditioned on the observation-action-reward triplet of the agent that we control and a variable d indicating whether the episode has terminated; q w ( z | τ =
( o , a , r, d )) .
More precisely, our goal is to approximate the true posterior that is conditioned on opponent’s information, with a variational distribution that only depends on local information.
The use of this local information in a recurrent fashion has been successfully used in meta-RL settings (Wang et al., 2016; Duan et al., 2016).
We start by computing the KL divergence between the two distributions:
[Equation]
By following the works of Kingma & Welling (2014) and Higgins et al.
and using the Jensen inequality, the VAE objective can be written as:
[Equation]
The reconstruction loss factorizes exactly similar to equation 10.
From equation 12, it can be seen that the variational distribution only depends on locally available information.
Since during execution, only the encoder is required to generate the opponent’s model, this approach removes the assumption that access to the opponent’s observations and actions is available during execution.
Figure 2 presents the diagram of the VAE.
4. R EINFORCEMENT L EARNING T RAINING
We use the latent variable z augmented with the agent’s observation to condition the policy of our agent, which is optimized using RL.
Consider the augmented observation space O (cid:48)
=
O × Z , where O is the original observation space of the our agent in the Markov game, and Z is the representation space of the opponent models.
The advantage of learning the policy on O (cid:48) compared to O is that the policy can adapt to different z ∈ Z .
After training the variational autoencoder that was described in Section 4.2, we use it to train our agent against the opponents in the set T .
We use the DDPG (Lillicrap et al., 2015) algorithm for this task.
We did not manage to optimize the representation jointly with the policy, neither with DDPG or A2C. At the beginning of each episode, we sample an opponent from the set T .
The agent’s input is the local observation and a sample from the variational distribution.
We refer to this as OMDDPG (Opponent Modeling DDPG), and the full pseudocode is provided in Appendix D.
We optimize the second proposed VAE method jointly with the policy of the controlled agent.
We use the A2C algorithm, similarly to the meta-learning algorithm RL 2 (Wang et al., 2016; Duan et al., 2016).
In the rest of this paper, we refer to this as SMA2C (Self Modeling A2C).
The actor’s and the critic’s input is the local observation and a sample from the latent space.
We back-propagate the gradient from both the actor and the critic loss to the parameters of the encoder.
Therefore, the encoder’s parameters are shaped to maximize both the VAE’s objective as well as the discounted sum of rewards.
The full pseudocode is provided in Appendix D.
E XPERIMENTS 5. T OY E XAMPLE
We will ﬁrst provide a toy example to illustrate SMA2C.
We consider the classic repeated game of prisoner’s dilemma with a constant episode length of 25 time steps.
We control one agent, and the other agent is selected randomly between two possible opponent policies.
The ﬁrst opponent always defects, while the second opponent follows a tit-for-tat policy.
At the beginning of the episode, one of the two opponents is randomly selected.
We train SMA2C against the two possible opponents.
The agent that we control has to identify the correct opponent, and the optimal policy, it can achieve, is to defect against opponent one and collaborate with opponent two.
Figure 3 shows the payoff matrix, the embedding space at the last time step of the episode, and the episodic return that SMA2C and A2C achieve during training.
Note that, based on the payoff matrix, the optimal average episodic return that can be achieved is 24 . 5 .
[Figure] [Figure]
[Figure] [Figure]
[Figure]
5. E XPERIMENTAL F RAMEWORK
To evaluate the proposed methods in more complex environments, we used the Multi-agent Particle
Environment (MPE) (Mordatch & Abbeel, 2017), which provides several different multi-agent environments.
The environments have continuous observation, discrete action space, and ﬁxed-length episodes of 25 time steps.
Four environments are used for evaluating the proposed methodology; the speaker-listener, the double-speaker listener, the predator-prey, and the spread.
In Appendix A, descriptions of the different environments are provided.
During the experiments, we evaluated the two proposed algorithms OMDDPG and SMA2C as well as the modeling method of Grover et al.
(2018a)
combined with DDPG (Lillicrap et al., 2015).
In all the environments, we pretrain ten different opponents, where ﬁve are used for training and ﬁve for testing.
In the speaker-listener environment, we control the listener, and we create ten different speakers using different communication messages for different colors.
In the double speakerlistener, which consists of two agents that have to be both listener and speaker simultaneously, we control the ﬁrst agent.
We create a diverse set of opponents that have different communication messages similar to speaker-listener, while they learn to navigate using the MADDPG algorithm (Lowe et al., 2017), with different initial random seeds.
In the predator-prey environment, we control the prey and pretrain the three other agents in the environment using MADDPG with different initial parameters.
Similarly, in spread, we control one of the agents, while the opponents are pretrained using MADDPG.
We use agent generalization graphs (Grover et al., 2018b) to evaluate the generalization of the proposed methods.
We evaluate two types of generalizations in this work.
First, we evaluate the episodic returns against the opponents that are used for training, T , which Grover et al.
(2018b) call ”weak generalization”.
Secondly, we evaluate against unknown opponents from set G , which is called ”strong generalization”.
A ﬁgure of an agent generalization graph is provided in Appendix E.
5. E VALUATION OF THE R EPRESENTATIONS
To evaluate the representations created from our models we will estimate the Mutual Information (MI) between the variational distribution ( q ( z | τ ) or q ( z | τ − 1 ) ) and the prior on the opponents’ identities, which is uniform.
This is a common method to estimate the quality of the representation (Chen et al., 2018; Hjelm et al., 2019).
To estimate the MI, we use the Mutual Information Neural
Estimation (MINE) (Belghazi et al., 2018).
Note that, the upper bound of the MI, the entropy of the uniform distribution, in our experiments is 1 . 61 .
We gather 200 trajectories against each opponent in T , where 80% of them are used for training and the remaining for testing.
The visualization of the embedding space, for the predator-prey environment, is provided in Appendix C.
[Table]
From Table 1, we observe that the method of Grover et al.
(2018a) achieves signiﬁcantly higher values of MI.
We believe that the main reason behind this is the discrimination objective that implicitly increases MI.
This is apparent in the MI values of OMDDPG as well.
SMA2C manages to create opponent representations, based only on the local information of our agent, that have information about the opponent identities.
Additionally, based on Figure 4, we observe that the value of MI is not directly related to the episodic returns in RL tasks.
In Appendix B, we demonstrate that when we detach the encoder’s parameters from the policy optimization, the MI decreases.
5. R EINFORCEMENT L EARNING P ERFORMANCE
We evaluate the proposed opponent modeling methods in RL settings.
In Figure 4, the episodic returns for the three methods in all four environments are presented.
[Figure]
Grover et al.
(2018a)
Every line corresponds to the average return over ﬁve runs with different initial seeds, and the shadowed part represents the 95% conﬁdence interval.
We evaluate the models every 1000 episodes for
100 episodes.
During the evaluation, we sample an embedding from the variational distribution at each time step, and the agent follows the greedy policy.
The hyperparameters for all the experiments in Figure 4 were optimized on weak generalization scenarios, against opponents from set T .
Details about the implementation and hyperparameters that were used for generating the ﬁgures are presented in Appendix D.
OMDDPG is an upper baseline for SMA2C achieving higher returns in all environments during weak generalization.
However, OMDDPG, as well as Grover et al.
(2018a), tend to overﬁt and perform poorly during strong generalization in the speaker-listener and double speaker-listener environment.
SMA2C achieves higher returns that Grover et al.
(2018a) in more than half of the scenarios.
Below, in the Section 5.5, we perform an ablation study on different inputs in the encoder of SMA2C. In Appendix B, we evaluate whether back-propagating the RL loss to the parameters of the encoder, in SMA2C, affects the episodic returns.
5. A BLATION S TUDY ON SMA2C I NPUTS
We perform an ablation study to assess the performance requirements of the SMA2C. Our proposed method utilizes the observation, action, reward, and termination sequence to generate the opponent’s model.
We use different combinations of these elements in the encoder and compare the average episodic returns.
In Figure 5, the average episode return is presented for three different cases; SMA2C with all inputs, SMA2C with only observation and action as inputs and SMA2C with only observation as input; for all four environments. [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
r d
[Figure]
[Figure]
5. A BLATION S TUDY ON THE DISCRIMINATION OBJECTIVE
Another element of this work is the utilization of the discrimination objective of the Grover et al.
(2018a) in the VAE loss.
To better understand how the opponent separation in the embedding space is related to RL performance, below, Figure 6 shows the episodic return during the training for the
OMDDPG with and without the discrimination objective is presented.
Using the discrimination objective has a signiﬁcant impact on the episodic returns in the speaker-listener and the double speaker-listener environment.
[Figure]
001e
[Figure]
[Figure]
[Figure]
R e w
[Figure]
{+0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.001e
+}
{+[Figure] 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.001e
 [Figure] 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.001e+}
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
1e
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] 1e
[Figure]
[Figure]
[Figure] [Figure]  C ONCLUSION To conclude this work, we proposed two methods for opponent modeling in multi-agent systems using variational autoencoders.
First, we proposed OMDDPG a VAE-based method that uses the common assumption that access to opponents’ information is available during execution.
The goal of this work is to motivate opponent modeling without access to opponent’s information.
The core contribution of this work is SMA2C, which learns representations without requiring access to opponent’s information during execution.
We performed a thorough experimental evaluation of the proposed methodology.
We evaluated the quality of the representations produced by our models as well as the episodic return that can achieve in RL tasks.
The experiments conclusively indicate that access to the opponent’s information is not necessary during execution, eliminating a long-standing assumption of the prior work.
Additionally, we provided evidence that the relationship between the MI and the RL performance is not apparent.
In the future, we would like to research how these models can be used for non-stationary opponents. Particularly, there are two scenarios worth investigating; the ﬁrst is multi-agent deep RL, where different agents are learning concurrently leading to non-stationarity in the environment, which prevents the agents from learning optimal policies. Secondly we would like to explore whether the proposed models can deal with opponents that try to deceive it and exploit the controlled agent (Ganzfried & Sandholm, 2011; 2015)
A D ETAILS OF THE E XPERIMENTAL E NVIRONMENT A. S PEAKER -L ISTENER E NVIRONMENT
The speaker-listener environment consists of two agents, called speaker and listener as well as three designated landmarks that each one has a different color, red green or blue.
At the beginning of the episode, the listener is assigned a color, which can be red, green, or blue.
The task of the listener is to navigate to the landmark that has the same color.
However, the color of the listener can only be observed by the speaker.
So the speaker has to learn to communicate the correct color to the listener.
The listener should be able to understand the generated message of the speaker and to navigate to the correct color.
The observation space of the listener has 13 dimensions, which consists of the position of the listener, the positions of the three landmarks in the 2D environment and the 5-dimensional communication message of the speaker and its action space has ﬁve dimensions.
The speaker has a 3-dimensional observation space, which is a vector assigned to the color of the listener, and 5dimensional action space.
We use our method to train the listener to be able to understand a set of different speakers, pretrained speakers that use different communication messages.
Both speaker and listener share the same reward, which is the negative Euclidean distance between the listener and the correct landmark.
In Figure 7a, an instance from the speaker-listener environment is presented.
A. D OUBLE S PEAKER -L ISTENER E NVIRONMENT
The double speaker-listener environment consists of two agents and three designated landmarks that each one has a different color, red green or blue, similarly to the speaker-listener environment.
The only difference is that both agents are simultaneously both speakers and listeners.
Therefore, at the beginning of the episode, each agent has a color that can only be observed by the other agent.
Each agent must learn both to communicate a message to the other agent as well as navigate to the correct landmark.
The observation space of each agent has 16 dimensions, where 13 of them are the same as the listener’s in the previous environment, and the other three are the vector assigned to the color of the opponent, while the action space is 5-dimensional for the navigation actions and 5-dimensional for the communication message as well.
The reward is the average of the negative
Euclidean distances between each agent and the correct landmark.
This environment is signiﬁcantly more difﬁcult compared to the speaker-listener because our agent has to infer both the color that the other agent observes as well as communicate the correct message that the opponent expects.
In
Figure 7b, an instance from the double speaker-listener environment is presented.
A. P REDATOR -P REY ENVIRONMENT
This environment consists of one prey agent and three predator agents.
At the beginning of the episode, the prey and the predators are randomly placed on a 2D map.
The goal of the prey is to avoid being caught by predators.
In the environment, there are additionally two large black obstacles in order to block the agents.
The advantage of the prey compared to the three predators is that it can move faster compared to the three adversaries.
This environment, compared to the previous two, is competitive.
We deliberately chose this environment in order to prove that our method is agnostic to the environment setting.
In our work, we will apply the proposed algorithm in the prey agent in order to examine whether it can avoid a large number of different pretrained predator agents.
The observation of each agent has 14 dimensions representing the agents’ positions as well as the obstacles in the 2D space, while their action space consists of 5 actions.
In Figure 7c, an instance from the predator-prey environment is presented.
A. S PREAD E NVIRONMENT
The spread environment consists of three large agents and three landmarks as well.
At the beginning of the episode, the three agents and the three landmarks are spread randomly in the 2D space.
The goal of the agents is to navigate to three different landmarks without colliding.
The reward is the negative distance of each agent from the landmark.
In the case of collision, there is an additional negative reward.
The reward is the same for all agents.
All the agents have the same observation space, which consists of 18 dimensions, while their action space consists of 5 actions.
In Figure 7d, an instance from the spread environment, is presented.
[Figure]
B A BLATION S TUDY ON RL B ACK -P ROPAGATION
We evaluate SMA2C without back-propagating the gradients of the RL loss to the parameters of the encoder.
Therefore, the encoder is only trained based on ELBO.
[Figure]
Figure 8 veriﬁes that not performing back-propagation does not signiﬁcantly affect the episodic returns that SMA2C achieves during weak generalization.
Additionally, we compute the MI between the embeddings and the opponents’ identities similarly to Section 5.3 for the double speaker-listener environment.
We observe that the MI decreases when we do not perform back-propagation to the embeddings of the encoder.
[Table]
C E MBEDDING V ISUALIZATION
Figure 9 visualizes the embedding space for the three different evaluated algorithms in the predatorprey environment.
Note that the embeddings were generated using interactions between the opponents from the set T and the trained agent that we control.
{+1   4 First Principal Component
 S e c ond P r i n c i pa l C o m ponen t+}
{+Predator-Prey (a) OMDDPG     3 First Principal Component
 S e c ond P r i n c i pa l C o m ponen t+}
{+Predator-Prey (b) SMA2C 6 4 2 0 2 4 6 8 First Principal Component
+}
S e c ond
P r i n c i pa l C o m ponen t Predator-Prey (c) Grover et al. (2018a)
D I MPLEMENTATION D ETAILS
The pseudocode for training the VAE from Section 4.2 is provided below.
We consider thattrajectories are provided for each one of the opponents, which are generated against trained agents.
We train the VAE for 1000 epochs, using Adam (Kingma & Ba, 2014) with 10 − 3 learning rate.
Algorithm for i = 1 : M do
Sample an episode and compute the embedding z ← q (sample( E i )) Sample a different episode and compute the embedding z + ← q (sample( E i )) for j = 1 : M do if i ==
j then continue
Sample an episode and compute the embedding z − ← q (sample( E j ))
Update VAE parameters by maximizing
We train OMDDPG for 2 million steps in all the experiments.
Since the encoder of the VAE has an
LSTM layer and has to be trained on sequential data, we use a modiﬁed experience replay, that enables sampling of whole episodes, which has also been used by Hausknecht & Stone (2015).
DDPG algorithm requires continuous action space.
However, since our experimental environments have discrete action space, we use the Gumbel-Softmax trick (Jang et al., 2016) to create differentiable samples from a discrete distribution.
Additionally, we regularize the actor loss by adding the squared logits in order to prevent them from getting large values.
The pseudocode of OMDDPG is presented below 2.
Algorithm for e = 1 : K episodes do opp ← sample(opponents) while episode is not ﬁshed do
Get the observation of our agent o and the observation of the opponent o −
Compute the action of the opponent a −
Get a sample from the encoder z ← q ( z | o − 1 , a − 1 )
Compute the action a of the agent using exploration
Perform the actions in the environment and get new observations and rewards
Store the sequences of both agents in the experience replay if t % update frequency == 0 then
Sample a batch of sequences from the experience replay
Update the actor-critic parameters using 2 where s ← concat( o, z )
Update the target networks
All neural networks have 2 hidden layers with ReLU (Maas et al., 2013) activation function.
We use the Adam optimizer (Kingma & Ba, 2014) for all experiments.
The target networks are updated with τ = 0 .
The hidden dimensions for all VAE layers is 100 .
The parameter λ in the VAE loss (equation 8) is always 1 .
We perform gradient updates every 50 time steps with a batch ofepisodes.
Table 3 summarizes the rest of the hyperparameters.
[Table]
We train A2C for 2 million steps in the speaker-listener environment, 5 million in the double speakerlistener, 10 million in the predator-prey and 15 million in the spread environment.
A2C as an onpolicy algorithm is signiﬁcantly less sample-efﬁcient compared to DDPG, and as a result, more training steps are required.
The pseudocode of SMA2C is presented below 3.
We subtract the policy entropy from the actor loss (Mnih et al., 2016) to ensure sufﬁcient exploration.
The loss that SMA2C minimizes can be written as:
[Equation]
Create D parallel environments t ←for e = 1 : K episodes do
Sample D opponents opp ← sample(opponents) while episode is not ﬁnished do for every environment in D do
Get the observation of our agent o and the observation of the opponent o −
Get a sample from the encoder z ← q ( z | o, a, r, d )
Compute the action a of the agent using exploration t ← t +
Perform the actions in the environment and get new observations, rewards and done if t % update frequency == 0 then
Gather the sequences from all environments to a single batch
Update the actor-critic and VAE parameters using 13 where s ← concat( o, z )
For the advantage computation, we use the GAE (Schulman et al., 2015b) with λ GAE = 0 .
We create 10 parallel environment to break the correlation between consecutive samples.
The actor and the critic share all hidden layers in all the environments except the double speaker-listener.
We use the Adam optimizer (Kingma & Ba, 2014), and we clip the gradient norm to 0 .
Table 4 summarizes the rest of the hyperparameters.
[Table]
E A GENT G ENERALIZATION G RAPHS
Figure 10 presents the agent generalization graph that was used in all experiments in this paper.
[Figure]