S YMMETRIC M ACHINE T HEORY OF M IND
Anonymous authors
Paper under double-blind review
A BSTRACT
Theory of mind (ToM), the ability to understand others’ thoughts and desires, is a cornerstone of human intelligence.
Because of this, a number of previous works have attempted to measure the ability of machines to develop a theory of mind, with one agent attempting to understand anothers’ internal “mental state”.
However, ToM agents are often tested as passive observers or in tasks with speciﬁc predeﬁned roles, such as speaker-listener scenarios.
In this work, we propose to model machine theory of mind in a more ﬂexible and symmetric scenario; a multiagent environment SymmToM where all agents can speak, listen, see other agents, and move freely through a grid world.
An effective strategy to solve SymmToM requires developing theory of mind to maximize each agent’s rewards.
We show that multi-agent deep reinforcement learning models that model the mental states of other agents achieve signiﬁcant performance improvements over agents with no such ToM model.
At the same time, our best agents fail to achieve performance comparable to agents with access to the gold-standard mental state of other agents, demonstrating that the modeling of theory of mind in multi-agent scenarios is very much an open challenge.
I NTRODUCTION
Human communication is shaped by the desire to efﬁciently cooperate and achieve communicative goals (Tomasello, 2009).
Children learn from a young age that the others they interact with have independent mental states, and therefore communicating is necessary to obtain information from or shape the intentions of those they interact with.
Remembering and reasoning over others’ mental states ensures efﬁcient communication by avoiding having to repeat information, and in cases where cooperation is involved contributes to achieving a common goal with minimal effort.
Because of this, there is growing interest in developing agents that can exhibit this kind of behavior, referred to as Theory of Mind (ToM) by developmental psychologists (Premack & Woodruff, 1978).
Previous work on agents imbued with some capability of ToM has focused mainly on two types of tasks.
The former are tasks where the agent is a passive observer of a scene that has to predict the future by reasoning over others’ mental states.
These tasks may involve natural language (Nematzadeh et al., 2018) or be purely spatial (Gandhi et al., 2021; Rabinowitz et al., 2018; Baker et al., 2011).
The latter are tasks where the ToM agent has a speciﬁc role, such as “the speaker” in speaker-listener scenarios (Zhu et al., 2021).
In contrast, human cooperation and communication is very often multi-party, and rarely assumes that people have pre-ﬁxed roles.
Moreover, human interlocutors are seldom passive observers of a scene but instead proactively interact with their environment.
Since previous domains limited us to research questions where most parties involved did not have an active role, we developed a more ﬂexible environment where we can now study what happens when all participants must act as both speaker and listener.
In this paper, we present SymmToM, a fully symmetric multi-agent environment where all agents can see, hear, speak, and move, and are active players of a simple information-gathering game.
To solve SymmToM, agents need to exhibit different levels of ToM, as well as efﬁciently communicate through a simple channel with a ﬁxed set of symbols.
SymmToM is a partially observable setting for all agents: even when agents have full vision, hearing may be limited.
This also differentiates SymmToM from prior work, as modeling may require probabilistic theory of mind .
In other words, agents need to not only remember and infer other agents’ knowledge based on what they saw, but also estimate the probability that certain events happened.
This estimation may be performed by assuming other agents’ optimal behavior and processing the partial information available.
Despite its simplicity, SymmToM fulﬁlls the properties required for symmetric ToM to arise, which will be discussed in the following section.
We ﬁnd that SymmToM cannot be completely solved neither by using well-known multi-agent deep reinforcement learning (RL) models, nor by tailoring those models to our task.
We show that even maintaining the simple rules of the environment, modifying its parameters results in much more difﬁcult challenges, even for models where we artiﬁcially introduce perfect information.
We discuss examples where different levels of theory of mind are required to solve the task, and possible metrics.
T HEORY - OF -M IND A GENTS
A belief Theory-of-Mind agent can be deﬁned as a modiﬁcation of the standard multi-agent RL paradigm, where the agents’ policies are conditioned on their beliefs about others.
Formally, we deﬁne a reinforcement learning problem M as a tuple of a state space S , action space A , state transition probability function T ∈ S ×
A → R , and reward function R ∈ S ×
A → R , i.e. M :=
(cid:104)S , A , T, R (cid:105) .
In this setting, an agent learns a (possibly probabilistic) policy π : S
→ A that maps from states to actions, with the goal of maximizing reward.
In a multi-agent RL setting each agent can potentially have its own state space, action space, transition probabilities, and reward function, so we can deﬁne an instance of M i
= (cid:104)S i , A i , T i , R i (cid:105) for each agent i .
For convenience, we can also deﬁne a joint state space S
=
(cid:83) i S i that describes the entire world in which all agents are interacting.
Importantly, in this setting each agent will have its own view of the entirety of the world, described by a conditional observation function ω
i : S → Ω i that maps from the state of the entire environment to only the information observable by agent i .
As elaborated above, ToM is the ability to know (and act upon) the knowledge that an agent has.
Agents with no ToM will follow a policy that depends only on their current (potentially partial or noisy) observation of their environment: π i ( a i,t | ω i ( s t )) .
Agents with zeroth order ToM can reason over their own knowledge.
These agents will be stateful, π i ( · | ω i ( s t ) , h ( i ) t ) , where h ( i ) t is i ’s hidden state.
Hidden states are always accessible to their owner, i.e. i has access to h ( i ) t .
Agents with capabilities of reasoning over other agents’ mental states will need to estimate h ( j ) t for j (cid:54) =
i .
We will denote the estimation that i does of j ’s mental state in time t as ˆ h ( i,j ) t : π i ( · | ω i ( s t ) , h ( i ) t , ˆ h ( i, 1) t , . . .
ˆ h ( i,i − 1) t , ˆ h ( i,i +1) t . . . ˆ h ( n ) t )
How do we estimate ˆ h ( i,j ) t ?
As a function of i ’s (the predicting agent) previous hidden state t − 1 , i ’s observation in t − 1 , and i ’s prediction of the hidden states of every agent in the previous turn:
[Equation]
i ’s prediction of other agents’ observation in t − 1 is also crucial, but not explicitly mentioned since it can be computed using ω
i ( s t − 1 ) .
For the initial turn, ˆ h ( i,j )may be initialized differently depending on the problem: if initial knowledge is public, ˆ h
( i,j )is trivial; if not, ˆ h ( i,j )may be estimated.
S YMMETRIC T HEORY - OF -M IND
We deﬁne symmetric theory of mind environments as settings where theory of mind is required to perform a task successfully, and all agents have the same abilities.
There are at least four deﬁning characteristics of symmetric ToM to arise:
Symmetric action space.
In symmetric ToM all agents are required to have the same action space (in contrast to, for example, ToM tasks in speaker-listener settings).
Concretely, A i =
A j (cid:54) = ∅ ∀ i, j .
Imperfect information.
In perfect information scenarios all knowledge is public, making it impossible to have agents with different mental states.
In ToM tasks in general, there could be a subset of agents with perfect information: one example would a passive observer that needs to predict future behavior.
In symmetric ToM, since all agents have the same abilities and roles, all agents must have imperfect information.
More precisely, ω i must not be the identity for any agent i .
Observation of others.
Agents must have at least partial information of another agent to estimate its mental state.
In contrast to passive-observer settings, in symmetric ToM every agent must be able to partially observe all others.
More precisely, ω i must observe at least partial information about s ( j ) t (the subset of s t that refers to agent j ), although we do not require s ( j ) t (cid:54)
= ∅ in every single turn. Moreover, if communication is allowed, it is desirable to partially observe or infer interactions between two or more agents to develop second order ToM (i.e. predicting what an agent thinks about what another agent is thinking) or higher.
Information-seeking behavior.
It should be relevant for successfully performing the task to gather as much information as possible, and this information-gathering should involve some level of reasoning over other agent’s knowledge.
This is true for ﬁrst-order ToM tasks in general, and could be formalized as π ∗ (cid:54) =
π for any zeroth-order ToM policy π i ( · | ω i ( s t ) , h ( i ) t ) .
Furthermore, it would be desirable to design a task with perpetual information seeking behavior, since it would ensure that all agents have an incentive to play efﬁciently even in long episodes.
If one wants to design a task with perpetual information-seeking and ﬁnite knowledge, information must be forgotten eventually.
A forgetting mechanism could be implemented as an explicit loss of knowledge under speciﬁc conditions, or make remembrances less reliable or noisy.
Moreover, this introduces the concept of information staleness .
Since information is not cumulative and the environment is only partially observable, agents will need to estimate whether what they knew to be true still holds in the present.
T HE S YMM T O M E NVIRONMENT
SymmToM is an environment where a agents are placed in a w × w grid world, and attempt to maximize their reward by gathering all the information available in the environment.
There are c available information pieces, that each agent may or may not know initially.
Information pieces known at the start of an episode are referred to as ﬁrst-hand information .
Each turn, agents may move through the grid to one of its four neighboring cells, and may speak exactly one of their currently known information pieces.
More precisely, the action space of agent j is deﬁned as follows: A [Equation]
left , right
[Equation]
up , down
[Equation] no [Equation] When an agent utters an information piece, it is heard by every agent in its hearing range (an h × h grid centered in each agent, with h < 2 w − 1 ).
The agents who heard the utterance will be able [Figure] to share this newly-learned information with others in following turns.
We refer to this as secondhand information , since it is learned –as opposed to ﬁrst-hand information , given at the start of each episode.
The state space is comprised of the position of the agents and their current knowledge:
S = {{ ( p i , k i ) , for i ∈ { 1 , . . . , a }} where p i ∈ { 1 , . . .
, w } × { 1 , . . .
, w } , and k i ∈ { 0 , 1 } c }
Each agent aims to maximize their individual reward R i via information seeking and sharing.
Rewards are earned by hearing a new piece of information, giving someone else a new piece of information, or correctly using recharge bases .
Recharge bases are special cells where agents can reset their knowledge in exchange for a large reward (e.g. ( n − 1)
c times the reward for listening to or sharing new information).
Each agent has its own stationary recharge base during an episode.
To trigger a base, an agent must step into its designated base having acquired all the available pieces of information, causing the agent to lose all the second-hand information it learned.
Recharge bases guarantee that there is always reward to seek in this environment.
Concretely, if s = { ( p i , k i ) , for i ∈ { 1 , . .
. , a }} and a i = ( a dir i , a comm i ) , we can deﬁne the reward as the addition of the reward for hearing new information, giving new information, and using the recharge base:
R i ( s, a i ) = (cid:88) i (cid:54) =
j 1 { [Table] A non-ToM agent can have only limited success in this environment.
Without reasoning about its own knowledge (i.e. without zeroth order ToM ), it does not know when to use a recharge base.
Moreover, without knowledge about other agent’s knowledge (i.e. without ﬁrst order ToM ) it is not possibly to know which agents possess the information pieces it is lacking.
Even if it accidentally hears information, a non-ﬁrst-order ToM agent cannot efﬁciently decide what to utter in response to maximize its reward.
Higher order ToM is also often needed in SymmToM, as we will discuss further in Section 7.3.
Even though we only discussed a collaborative task for SymmToM, it can easily be extended for competitive tasks.
Moreover, all our models are also designed to work under competitive settings.
SymmToM satisﬁes the desiderata we laid out in the previous section, as we will detail below:
Symmetric action space.
As deﬁned in Eq.
1, A
i =
A j for all i, j .
Only a subset may be available at a time since agents cannot step outside the grid, speak a piece they have not heard, or move if they would collide with another agent in the same cell, but they all share the same action space.
Imperfect information.
Messages sent by agents outside of the hearing range will not be heard.
For example, in Fig.
2a green sends a message but it is not heard by anyone, since it is outside of red’s and blue’s range.
Hearing ranges are guaranteed not to cover the whole grid, since h < 2 w − 1 .
Observation of others.
Agents have perfect vision of the grid, even if they cannot hear what was said outside of their hearing range.
Hence, an agent may see that two agents were in range of each other, and thus probably interacted, but not hear what was communicated.
An example of this can be seen in Fig.
2a, where green observes blue and red interacting without hearing what was uttered.
The uncertainty in the observation also differentiates SymmToM from prior work: to solve the task perfectly, an agent needs to assess the probability that other agents outside its hearing range shared a speciﬁc piece of information to avoid repetition.
This estimation may be performed using the knowledge of what each agent knows (ﬁrst order ToM), the perceived knowledge of each of the agents in the interaction (second order ToM), as well as higher order ToM.
Information-seeking behavior Rewards are explicitly given for hearing and sharing novel information, guaranteeing information-seeking is crucial in SymmToM. Recharge bases ensure that the optimal solution is not for all agents to accumulate in the same spot and quickly share all the information available; and that the information tracking required is more complex than accumulating past events.
Conceptually, with recharge bases we introduce an explicit and observable forgetting mechanism.
As discussed in Section 3, this allows for perpetual information seeking and requires information staleness estimation.
An example of successful recharge base use is shown in Fig. 2b.
B ASELINE L EARNING A LGORITHMS AND O THER B OUNDS
To learn a policy for acting in the multi-agent SymmToM environment, it is a good strategy to use a multi-agent reinforcement learning algorithm.
We use MADDPG (Lowe et al., 2017), a wellknown multi-agent actor-critic framework with centralized planning and decentralized execution, to counter the non-stationarity nature of multi-agent settings.
In MADDPG, each actor policy receives its observation space as input, and outputs the probability of taking each action.
Notably, actors in MADDPG have no mechanism for remembering past turns.
This is a critical issue in SymmToM, as agents cannot remember which pieces they currently know, which ones they shared and to whom, and other witnessed interactions.
To mitigate this, it is necessary to add a recurrence mechanism to carry over information from past turns.
One option would be to modify the agent policy using a recurrent network like an LSTM, as RMADDPG (Wang et al., 2020) does.
Perfect Information, Heuristic and Lower Bound Models Performance is difﬁcult to interpret without simpler baselines.
As a lower bound model we use the original MADDPG, that since it does not have recurrence embedded, should perform worse or equal to any of the modiﬁcations described above.
We also include an oracle model ( MADDPG-Oracle ), that does not require theory of mind since it receives the current knowledge K for all agents in its observation space.
The performance of
MADDPG-Oracle may not always be achieved, as there could be unobserved communication with multiple situations happening with equal probability.
Moreover, as the number of agents and size of the grid increases, current reinforcement learning models may not be able to ﬁnd an optimal spatial exploration policy; they may also not be capable of inferring the optimal piece of information to communicate in larger settings.
In these cases, MADDPG-Oracle may not perform optimally, so we also include a baseline with heuristic agents to compare performance.
Heuristic agents will always move to the center of the board and communicate round-robin all the information pieces they know until they have all the available knowledge.
Then, they will move efﬁciently to their recharge base and come back to the center of the grid, where the process restarts.
We must mention that this heuristic is not necessarily the perfect policy, but it will serve as a baseline to note settings where current MARL models fail even with perfect information.
Qualitatively, smaller settings have shown to approximately follow a policy like the heuristic just described.
D IRECT M ODELING OF S YMMETRIC T HEORY OF M IND
In contrast to RMADDPG (Wang et al., 2020), we speciﬁcally design algorithms for our environment to maximize performance.
Intuitively, our model computes a matrix, K ∈ { 0 , 1 } c × a , that reﬂects the information pieces known by each agent from the perspective of the agent being modeled: K ij reﬂects if the agent being modeled believes that agent j knows i .
K is updated every turn and used as input of the following turn of the agent, obtaining the desired recurrent behavior.
K is also concatenated to the usual observation space, to be processed by a two-layer ReLU MLP and obtain the probability distributions for speech and movement, as in the original MADDPG.
There are several ways to approximate K .
It is important to note that each agent can only partially observe communication, and therefore it is impossible to perfectly compute K deterministically.
The current knowledge is comprised of ﬁrst-hand information (the initial knowledge of every agent,
F , publicly available) and second-hand information.
Second-hand information may have been heard this turn ( S , whose computation will be discussed below) or in previous turns (captured in the K received from the previous turn, noted K ( t − 1) ).
Additionally, knowledge may be forgotten when an agent steps on a base having all the information pieces.
To express this, we precompute a vector
B ∈ { 0 , 1 } a that reﬂects whether each agent is currently on its base; and a vector E ∈ { 0 , 1 } a that determines if an agent is entitled to use their recharge base: E j = 1 (cid:80)
i K ij = c for all j ∈ { 1 , . . .
, a } .
We are then able to compute K as follows:
[Equation]
F , K ( t − 1) , and B are given as input, but we have not yet discussed the computation of the secondhand information S .
S often cannot be deterministically computed, since our setting is partially observable.
We will identify three behaviors and then compute S as the sum of the three:
[Equation]
For simplicity, we will assume from now on that we are modeling agent k .
S [0] will symbolize the implications of the information spoken by agent k
: if agent k speaks a piece of information, they thus know that every agent in its hearing range must have heard it (ﬁrst order ToM).
S [1] will symbolize the implications of information heard by k : this includes updating k ’s known information (zeroth order ToM) and the information of every agent that is also in hearing range of the speaker heard by k . S [2] will symbolize the estimation of information pieces communicated between agents that are out of k ’s hearing range.
Since we assume perfect vision, k will be able to see if two agents are in range of each other, but not hear what they communicate (if they do at all).
S [0] and S [1] can be deterministically computed.
To do so, it is key to note that every actor knows the set of actions A ∈ { 0 , 1 } c × a performed by each agent last turn, given that those actions were performed in their hearing range.
Moreover, each agent knows which agents are in its range, as they all have perfect vision.
We precompute H ∈ { 0 , 1 } a × a to denote if two given agents are in range.
Then, S [0] ij = 1 if and only if information piece i was said by k , and agents k and j are in hearing range of each other.
More formally,
[Equation]
S [1] ij = 1 if and only if agent k (the actor we are modeling) heard some agent (cid:96) speaking information piece i , and agent j is also in range of agent (cid:96) .
Note that agent k does not need to be in hearing range of agent j .
More precisely,
[Equation]
S [2] –the interactions between agents not in hearing range of the agent we are modeling– can be estimated in different ways.
A conservative approach would be to not estimate interactions we do not witness ( S [2] = 0 , which we will call MADDPG-ConservativeEncounter (MADDPG-CE) ); and another approach would be to assume that every interaction we do not witness results in sharing a piece of information that will maximize the rewards in that immediate turn.
We will call this last approach MADDPG-GreedyEncounter (MADDPG-GE) .
MADDPG-GE assumes agents play optimally, but does not necessarily know all the known information and that could lead to a wrong prediction.
This is particularly true during training, as agents may not behave optimally.
The computation of S [2] for MADDPG-GE is as follows.
First, we predict the information piece U (cid:96) that agent (cid:96) uttered.
MADDPG-GE predicts U (cid:96) will be the piece that the least number of agents in range know, as it will maximize immediate reward:
[Equation]
With this prediction, agent j will know information i if at least one agent in its range said it:
S [2] ij = 1 if exists (cid:96) (cid:54) =
k such that U (cid:96) =
j and H j(cid:96) and j (cid:54)
=
k else
MADDPG-EstimatedEncounter (MADDPG-EE) MADDPG-CE and MADDPG-GE are two paths to information sharing estimation, but in none of them do we estimate the probability of an agent knowing a speciﬁc piece of information.
In MADDPG-EstimatedEncounter (MADDPG-EE) , known information of other agents is not binary, i.e. K ij ∈ [0 , 1] .
This added ﬂexibility can avoid making predictions of shared information based upon unreliable information.
MADDPG-EE estimates the probability that an agent j uttered each piece of information ( U j ∈ R c ) by providing the current information of all agents in its range to an MLP:
[Equation]
all (cid:96) where H j(cid:96) } )) , with f an MLP
Then, the probability of having heard a speciﬁc piece of information will be the complement of not having heard it, which in turn means that none of the agents in range said it.
More formally,
[Equation]
Since MADDPG-EE requires functions to be differentiable, we use a differential approximation of
Eq.
A pseudocode of MADDPG-EE’s implementation can be found in Section A.4.
MADDPGEE solely focuses on ﬁrst order ToM, and we leave to future work modeling with second order ToM.
The structure of the model would be similar but with an order of magnitude more parameters.
E XPERIMENTS 7. E XPERIMENTAL S ETTINGS
In this section, we compare the different algorithms explained in the previous section.
The observation space will be constituted of a processed version of the last turn in the episode, to keep the input size controlled.
More precisely, the observation space is composed of: the position of all agents, all recharge bases, the current direction each agent is moving towards to and what they communicated in the last turn, the presence of a wall in each of the immediate surroundings, and every agents’ ﬁrst-hand information.
First-hand information is publicly available in our experiments to moderate the difﬁculty of the setup, but this constraint could also be removed.
We use the reward as our main evaluation metric.
This metric indirectly evaluates ToM capabilities, since information-seeking is at the core of SymmToM. We train through 60000 episodes, and withrandom seeds to account for high variances in the rewards obtained.
Our policies are parametrized by a two-layer ReLU MLP with 64 units per layer, as in the original MADDPG (Lowe et al., 2017).
MADDPG-EE’s function f is also a two-layer ReLU MLP with 64 units per layer.
We test two board sizes ( w = 6 , and w = 12 ), two numbers of agents ( a = 3 and a = 4 ), and three quantities of information pieces ( c = a , c = 2 a c = 3 a ).
The length of each episode is set to 5 w .
More detail about design decisions can be found in Section A.5.
7. M AIN R ESULTS
As we can observe in Table 1, there is a signiﬁcant difference in performance between MADDPGOracle and MADDPG (MADDPG-Oracle is 123% better on average): this conﬁrms that developing
ToM and recurrence is vital to perform successfully in SymmToM. MADDPG-Oracle is often not an upper bound: when c > a , the heuristic performs better (101% on average).
This shows that even with perfect information, it can be difﬁcult to learn the optimal policy using MADDPG.
Moreover, models with a recurrence mechanism perform signiﬁcantly better than MADDPG (61% better on average), also showing that remembering past information gives a notable advantage.
As expected, having recurrent models tailored to our problem resulted in better performance than a general LSTM recurrence (RMADDPG).
The performance of the best of the tailored models (MADDPGCE, MADDPG-GE, MADDPG-EE) was 42% better on average than plain RMADDPG. LSTM was able to surpass the best of the tailored models only for a = 3 , w = 12 , c = 3 a .
Increasing c generally decreases global rewards for learned agents (on average, rewards for c = 2 a are 75.54% of those for c
= a , and rewards for c = 3 a are 75.66% of those for c = a ).
This suggests that probabilistic decisions are harder to learn, or impossible to successfully navigate [Table] when several events are equally likely.
MADDPG-EE did not show improvements over the other learned agents [Table] and in some cases decreased its performance more heavily than other learned agents (e.g. w = 6 , c
= 3 a ).
MADDPG-EE uses an MLP in its deﬁnition of S [2] [Table] which gives more ﬂexibility but also implies a more complex function to learn
We leave to future work to explore other probabilistic agents
[Table]
but the signiﬁcant difference in performance between all of the learned models and the highest performing ones (MADDPG-Oracle and the heuristic) shows there is ample space for improvement in this task, and hence proves SymmToM to be a simple yet unsolved benchmark.
Increasing a results in a 10% reduction of performance on average for learned models.
Nonetheless, the heuristic improved its rewards by an average of 46%, given the larger opportunities for rewards when including an additional listener.
Overall, this implies that increasing a also makes the setup signiﬁcantly more difﬁcult.
Finally, increasing w did not have a conclusive result: for a = 4 it consistently decreased performance in ∼ 17%, but for a = 3 we saw an improvement of 16% and 61% for c = a and c = 2 a respectively.
In sum, modifying c and a provides an easy way of making a setting more difﬁcult without introducing additional rules.
7. D ISCUSSION
In the past section we analyzed results using the metric of episode rewards.
Although the rewards in SymmToM are designed to correlate with information seeking and knowledge state of the agent themselves and others, they do not show explicitly if agents are exhibiting theory of mind.
To do so more directly, we develop two categories of possible analyses: scenarios speciﬁcally designed to test theory of mind, and post-hoc analyses of episodes.
A classic example of a scenario speciﬁcally designed to test ToM behavior is the Sally-Anne task (Wimmer & Perner, 1983).
This false belief task, originally designed for children, aims to test if a passive observer can answer questions about the beliefs of another person, in situations where that belief may not match reality.
If we were to use it for machine ToM, we could repeat the experiment and ask an agent to predict the position of an object varying the underlying conditions.
This test is feasible because there is only one agent with freedom of action, which ensures that desired conditions are met every time.
Testing becomes unfeasible when giving multiple agents freedom of action, as constraints planned in test design may be broken by a collective drift from the strategy thought by the designer.
Testing becomes easier if we allow for controlling all agents but one, as shown in Fig.
Other tests besides the ones shown may be designed.
In particular, in Fig.
3d we show an example of probabilistic ToM where two communicative events are equally likely, but one could modify this scenario to have different probabilities and test the expected value of the turns until red successfully shares an information piece.
One could also design retroactive deduction tests: for example, in Fig.
3d if red communicates and receives no reward, it can deduce that green had received that information from blue.
If there had been another agent (let’s say, a yellow agent) in [Figure] should be range of blue when it spoke to green, the red agent could also update its knowledge about yellow.
Results for the tests proposed in Figure 3 are detailed in Appendix A.1.
Post-hoc analysis also has its challenges in multi-agent settings, even in the most direct cases.
Thanks to our reward shaping, using recharge bases is always the optimal move when an agent has all the information available: an agent will have a reward of ( n − 1) c for using the base, whereas it can only gain up to n − 1 + c − 1 per turn if it decides not to use it.
Even in this case, small delays in using the base may occur, for example if the agent can gather additional rewards on its path to the base.
More generally, having multiple agents makes a speciﬁc behaviors attributable to any of the several events happening at once, or a combination of them.
Even though it may be difﬁcult to establish causality when observing single episodes, we developed metrics that comparatively show which models are using speciﬁc features of the environment better than others.
Examples of metrics are unsuccessful recharge base usage count; number of times an agent shares an information piece everyone in its range already knows when having better alternatives; number of times an agent moves away from every agent when not having all the information pieces available; among others. See Appendix A.2 for detailed description and results.
Reward can also be understood as a post-hoc metric with a more indirect intepretation.
Post-hoc analyses of single episodes can also be blurred by emergent communication.
Because agents were trained together, they may develop special meaning assignment to particular physical movements or messages.
Even though qualitatively this does not seem to be the case for the models presented in the paper, tests should also account for future developments.
This also implies that one should not overinterpret small differences in the metrics described in the paragraph above.
C ONCLUSIONS AND F UTURE W ORK
We deﬁned a framework to analyze machine theory of mind in a multi-agent symmetric setting, a more realistic setup than the tasks currently used in the community.
Based on the four properties needed for symmetric theory of mind to arise, we provided a simpliﬁed setup on which to test the problem, and we showed we can easily increase difﬁculty by growing the number of agents or communication pieces.
Our main goal in this work was not to solve symmetric theory of mind, but rather to give a starting point to explore more complex models in this area.
We showed that even with this minimal set of rules, SymmToM proves algorithmically difﬁcult for current multi-agent deep reinforcement learning models, even when tailoring them to our speciﬁc task.
We leave to future work to develop models that handle second-order theory of mind and beyond, and models that periodically reevaluate past turns to make new deductions with information gained a posteriori (i.e., models that pass retroactive deduction tests).
Another interesting direction would be to replace the information pieces with constrained natural language: communication sharing in our task is binary, whereas in language there is ﬂexibility to communicate different subsets of a knowledge base using a single sentence. We will make our codebase public upon publication, that also includes additional observation space restrictions to increase difﬁculty.
E THICS AND R EPRODUCIBILITY S TATEMENT
Theory of mind research at its core deals with understanding the mental states of other individuals.
In the present work we focused on collaborative machine theory of mind, which entails interactions only between artiﬁcial agents, and only in scenarios where every party involved has the same incentive structure.
This design decision is intentional.
Other approaches to theory of mind research could include scenarios with human-agent interaction, which could potentially lead to agents learning to model human players’ mental states.
This is not concerning per se, but in scenarios where players do not have the same incentive structure, it could lead to agents learning to deceive other players (potentially human players).
The state of the art in machine theory of mind is still far away from these capabilities, but we believe that experiment design choices should always take this matter into account.
Regarding reproducibility, we will make all code public upon acceptance, including the environment and the models’ code.
The exact set of parameters used for training will also be shared.
Multi-agent reinforcement learning models do have high variance, so models should be run several times to see similar conﬁdence intervals as the ones shown in Fig.
5 and Fig.
A A PPENDIX A. A D -H OC T HEORY OF M IND TESTS
We test on the four examples shown in Figure 3, adapting the examples to ﬁt one of the grid sizes we already experimented on.
For the tests described in Figure 3a and Figure 3b, we test two different grid sizes: w = 6 and w = 12 .
For the tests described in Figure 3c and Figure 3d we only test w = 12 and w = 6 respectively.
Image depictions of the exact test conﬁgurations can be seen in
Figure 4.
We measure three metrics: average success rate , average failure rate , and ratio of average turns to succeed vs. optimum (RATSO) .
Note that Average Success Rate and Average Failure Rate do not necessarily sum 1 since these two metrics only include trials where the agent reached any of the two proposed outcomes.
If, for example, the agent never moved from the starting point, the trial would not be counted positively towards Avg.
Success Rate or Avg.
Failure Rate.
In addition, ratio of average turns to succeed vs. optimum (RATSO) is the ratio between the average turns it took to succeed in successful trials, and the optimum number of turns to succeed in a speciﬁc trial.
For the tests in Figure 4a, 4b, 4d, and 4e, the trial ends when the red agent reaches the hearing range of one of the two possible target agents.
The test depicted in Figure 4f is a pass/fail test: if red moves suboptimally at any point before meeting blue, the trial is declared as failed.
This makes it a particularly difﬁcult test to pass at random.
Because of the nature of this second order ToM test, we only report the average success rate.
Finally, for the probabilistic ToM test we want to measure how fast can red communicate all the information it has to green.
The optimal number of turns is 1.5 (as discribed in Figure 3), and because of the nature of this test we will only report RATSO.
All results can be found in Table 2.
As expected, a larger average success rate correlates with higher reward models (MADDPG-CE and MADDPG-GE are the best models), suggesting that the
[Table]
[Table]
SR
FR
[Table]
SR
[Table]
FR
RATSO
SR
FR 1-SR-FR
RATSO
SR
MADDPG 17% 75% 9% 3.
[Table]
[Table]
[Table]
[Table]
[Table]
CE 32% 49% 19%
[Table]
[Table]
[Table]
[Table]
GE 32% 40% 28%
[Table]
[Table]
[Table]
[Table]
EE 16% 54% 29%
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
SR
FR 1-SR
[Table]
FR
RATSO
SR
FR 1-SR-FR
RATSO
RATSO
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
40 RMADDPG 18% 74%
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
02 MADDPG
[Table]
CE 16% 74% 10%
[Table]
[Table]
[Table]
7.12 MADDPG
[Table]
GE 27% 72% 1% 3.
[Table]
[Table]
[Table]
[Table]
EE 14% 81% 5% 3.
[Table]
reward is a valuable overall metric.
The low average success rates across all tests show there is signiﬁcant room to improve in this benchmark.
Success rate drops sharply when increasing the grid size, suggesting larger grids impose more difﬁcult training settings.
In this analysis, we used models that were trained speciﬁcally for each parameter combination.
Results for Oracle were omitted since some tests assumed no knowledge about communication (e.g. even though agents in Figure 4b do not communicate during the test, the test was designed to test if the red agent assumed them to be).
As we emphasized in the main text, many more tests can be proposed.
The code base we will release allows for easily adding new tests to the suite.
A. P OST - HOC ANALYSES
We developed several post hoc metrics to analyze speciﬁc aspects of our models.
• Unsuccessful recharge base usage rate: Average times per episode an agent steps on its recharge base without having all the information available (i.e. wrong usage of the recharge base).
Note that an agent may step on its base just because it is on the shortest path to another cell.
Therefore, a perfect theory of mind agent will likely not have zero on this score; but generally, lower is better.
See results in Table 3.
• Wrong communication piece selection count: Average times per episode an agent attempted to say an information they currently do not possess.
In these cases, no communication happens.
Lower is better.
See results in Table 4.
• Useless communication piece selection count: Average times per episode an agent communicated an information piece that everyone in its hearing range already knew, when having a piece of information that at least one agent in its range did not know.
Lower is better.
See results in Table 5.
• Useless movement: Average times per episode an agent moves away from every agent that does not have the exact same information it has, given that the agent does not currently possess all the information available.
This means that the agent is moving away from any possible valuable interaction.
Lower is better.
See results in Table 6.
All metrics are normalized by number of agents (i.e., they show the score for a single agent).
This allows for better comparison between a = 3 and a = 4 settings.
RMADDPG had the worst scores for unsuccessful recharge base use rate and useless communication piece selection count.
RMADDPG scored 43% more than Oracle for unsuccessful base usage on average, and 63% more than Oracle on average for usage of a useless communication piece.
The best tailored models (MADDPG-CE and MADDPG-GE) performed similarly to Oracle on average for these two metrics.
In contrast, MADDPG-CE and MADDPG-GE performed signiﬁcantly worse than Oracle for the wrong communication piece selection count (48% and 56% more than Oracle on average).
This suggests that all models may be making wrong decisions, but RMADDPG is biased towards communicating redundant information whereas MADDPG-CE and MADDPG-EE tend towards not communicating at all (the true effect of trying to communicate something they are not allowed).
Further analysis is needed to truly understand if these apparently wrong behaviors were done in turns where the agent had all the information available to make a better move, or if this is their default when they believe they have nothing of value to communicate.
A priori RMADDPG bias seems more principled, but it still showed worse performance overall.
No learned model performed particularly better in the useless movement metric (average differences in performance were less than 10%), suggesting that they perform pointless movements in similar frequencies.
It is important not to overinterpret small differences in these metrics.
For example, a useless movement may be a signal of emergent communication.
Furthermore, an agent may communicate something suboptimal for its immediate reward but this move may not affect its expected reward for the trial.
[Table]
[Table]
pieces
[Table]
a 2 a 3 a a 2 a 3 a a 2 a 3 a a 2 a 3 a
MADDPG
[Table]
Oracle 3.
MMADPG 4.
0.6 RMADDPG 5.
0.5 MADDPG-CE 4.
0.5 MADDPG-GE 4.
[Table]
.6 MADDPG-EE 4.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Figure]
[Figure]
A. P SEUDOCODE OF MADDPG-EE
Input: observation , A ∈ { 0 , 1 } c × a , agent idx ∈ 0 , . . .
a − 1 , F ∈ { 0 , 1 } c × a , K ∈ { 0 , 1 } c × a ,
B ∈ { 0 , 1 } a , H ∈ { 0 , 1 } a × a
Make agents not be in their own hearing range, to avoid talking to themselves from the previous turn.
This would be problematic when using recharge bases.
H =
H −
[Equation]
× a Compute S [0] , all the heard information spoken by agent idx :
S [0] = copy a times A [: , agent idx ] (cid:12) copy c times H [ agent idx , :]
Compute S [1] , all heard information by agent idx , spoken by all agents:
S [1] =
( A (cid:12) copy
C times H [ agent idx , :]) · H
Compute S [2] , an estimation of information pieces communicated between agents that were out of agent idx ’s hearing range:
U j = softmax ( f 1 ( K 1 j , . . .
, K cj , { K 1 (cid:96) , . . .
, K c(cid:96) for all (cid:96) where H j(cid:96) } )) , with f 1 an MLP
S [2] ij = 1 − (cid:81) (cid:96),H j(cid:96) =
1 1 − U (cid:96) for all i ∈ { 0 , . . .
, c − 1 } and j ∈ { 0 , . . .
, a − 1 }
S = S [0] + S [1] + S [2]
E i = 1 sum ( K [: ,i ])=
c ∈ { 0 , 1 } a for all i ∈ { 0 , . . .
, c − 1 }
K = step (cid:0) F · 100 + K + S − 2 · copy c times ( B (cid:12) E ) (cid:1)
A. E XPERIMENT D ESIGN D ECISIONS AND C ONSIDERATIONS A BOUT R ESULT P RESENTATION
All our experiments are with h = 1 : only the immediate neighbors of an agent will hear what they communicate.
We tested with a = 3 and a = 4 and not larger numbers of agents, as the training time increases quadratically with a ; also, the intrinsic difﬁculty of larger setup –even with perfect information– would possibly degrade performance to the point of making it impossible to compare models.
Running experiments with the same number of turns for every setting would imply that agents can move less in combinations with larger values of w , hence the need of making it proportional to the size of the grid.
Since the duration of the experiment is directly proportional to the length of the episodes, we settled on a small multiplier.
5 w allows agents to move to each edge of the grid and back to the center.
A similar decision is required when choosing c : having a constant number of information pieces when increasing the number of agents would make the problem easier, as each agent would have fewer options of ﬁrst-hand information pieces.
We decided to evaluate running additional episodes over the best checkpoints of each model because there was high variance for some runs, and drops in performance after achieving the highest rewards.
Those results are the base of the discussion and can be seen in Table 1.
Still, we share the training curves so that the reader can observe these behaviors in Fig.
5 and Fig.
[Table]