Original: L EARNING A LTRUISTIC B EHAVIOURS IN R EINFORCEMENT L EARNING WITHOUT E XTERNAL R EWARDS
Revised: L EARNING A LTRUISTIC B EHAVIOURS IN R EINFORCEMENT L EARNING WITHOUT E XTERNAL R EWARDS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Can artiﬁcial agents learn to assist others in achieving their goals without knowing what those goals are?
Revised: Can artiﬁcial agents learn to assist others in achieving their goals without knowing what those goals are?

Original: Generic reinforcement learning agents could be trained to behave altruistically towards others by rewarding them for altruistic behaviour, i.e., rewarding them for beneﬁting other agents in a given situation.
Revised: Generic reinforcement learning agents could be trained to behave altruistically towards others by rewarding them for altruistic behaviour, i.e., rewarding them for beneﬁting other agents in a given situation.

Original: Such an approach assumes that other agents’ goals are known so that the altruistic agent can cooperate in achieving those goals.
Revised: Such an approach assumes that other agents’ goals are known so that the altruistic agent can cooperate in achieving those goals.

Original: However, explicit knowledge of other agents’ goals is often difﬁcult to acquire.
Revised: However, explicit knowledge of other agents’ goals is often difﬁcult to acquire.

Original: In the case of human agents, their goals and preferences may be difﬁcult to express fully, may be ambiguous or even contradictory.
Revised: In the case of human agents, their goals and preferences may be difﬁcult to express fully, may be ambiguous or even contradictory.

Original: Thus, it is beneﬁcial to develop agents that do not depend on external supervision and can learn altruistic behaviour in a task-agnostic manner.
Revised: Thus, it is beneﬁcial to develop agents that do not depend on external supervision and can learn altruistic behaviour in a task-agnostic manner.

Original: We propose to act altruistically towards other agents by giving them more choice and thereby allowing them to better achieve their goals.
Revised: We propose to act altruistically towards other agents by giving them more choice and thereby allowing them to better achieve their goals.

Original: Some concrete examples include opening a door for others or safeguarding them to pursue their objectives without interference.
Revised: Some concrete examples include opening a door for others or safeguarding them to pursue their objectives without interference.

Original: We formalize this concept and propose an altruistic agent that learns to increase the choices another agent has by preferring to maximize the number of states that the other agent can reach in its future.
Revised: We formalize this concept and propose an altruistic agent that learns to increase the choices another agent has by preferring to maximize the number of states that the other agent can reach in its future.

Original: We evaluate our approach on three different multi-agent environments where another agent’s success depends on the altruistic agent’s behaviour.
Revised: We evaluate our approach on three different multi-agent environments where another agent’s success depends on the altruistic agent’s behaviour.

Original: Finally, we show that our unsupervised agents can perform comparably to agents explicitly trained to work cooperatively, in some cases even outperforming them.
Revised: Finally, we show that our unsupervised agents can perform comparably to agents explicitly trained to work cooperatively, in some cases even outperforming them.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Altruistic behaviour is often described as behaviour that is intended to beneﬁt others, sometimes at a cost for the actor (Dowding and Monroe, 1997; Fehr and Fischbacher, 2003).
Revised: Altruistic behaviour is often described as behaviour that is intended to beneﬁt others, sometimes at a cost for the actor (Dowding and Monroe, 1997; Fehr and Fischbacher, 2003).

Original: Such behaviour is a desirable trait when integrating artiﬁcial intelligence into various aspects of human life and society – such as personal artiﬁcial assistants, house or warehouse robots, autonomous vehicles, and even recommender systems for news and entertainment.
Revised: Such behaviour is a desirable trait when integrating artiﬁcial intelligence into various aspects of human life and society – such as personal artiﬁcial assistants, house or warehouse robots, autonomous vehicles, and even recommender systems for news and entertainment.

Original: By observing and interacting with us, we may expect that artiﬁcial agents could adapt to our behaviour and objectives, and learn to act helpfully and selﬂessly.
Revised: By observing and interacting with us, we may expect that artiﬁcial agents could adapt to our behaviour and objectives, and learn to act helpfully and selﬂessly.

Original: Altruistic behaviour could be a step towards value alignment (Allen et al., 2005;
Revised: Altruistic behaviour could be a step towards value alignment (Allen et al., 2005;

Original: Gabriel, 2020), which aims to incorporate common-sense human values into artiﬁcial agents.
Revised: Gabriel, 2020), which aims to incorporate common-sense human values into artiﬁcial agents.

Original: Typically, we could achieve such an altruistic behaviour through various forms of supervision such as providing ground-truth actions at each time step, training agents with reinforcement learning (RL) and suitable rewards, or through imitation learning (Song et al., 2018).
Revised: Typically, we could achieve such an altruistic behaviour through various forms of supervision such as providing ground-truth actions at each time step, training agents with reinforcement learning (RL) and suitable rewards, or through imitation learning (Song et al., 2018).

Original: However, none of the approaches above scale up easily.
Revised: However, none of the approaches above scale up easily.

Original: They either require a large amount of supervision or carefully crafted rewards that can easily be misstated, leading to unwanted behaviour (Russell, 2019, ch. 1).
Revised: They either require a large amount of supervision or carefully crafted rewards that can easily be misstated, leading to unwanted behaviour (Russell, 2019, ch. 1).

Original: How can one agent support another agent without knowing its goals?
Revised: How can one agent support another agent without knowing its goals?

Original: One clue might be the instrumental convergence hypothesis (Bostrom, 2017; Omohundro, 2008; Russell, 2019), which states that intelligent agents with varied goals are likely to pursue common subgoals which are generally useful (instrumental).
Revised: One clue might be the instrumental convergence hypothesis (Bostrom, 2017; Omohundro, 2008; Russell, 2019), which states that intelligent agents with varied goals are likely to pursue common subgoals which are generally useful (instrumental).

Original: Some examples are resource acquisition, cognitive enhancement or self-preservation, which all increase an agent’s chance of achieving almost arbitrary ﬁnal goals.
Revised: Some examples are resource acquisition, cognitive enhancement or self-preservation, which all increase an agent’s chance of achieving almost arbitrary ﬁnal goals.

Original: This hypothesis has been validated theoretically under many models, including resource games (BensonTilsen and Soares, 2016) and large classes of policies in discrete MDPs (Turner et al., 2019).
Revised: This hypothesis has been validated theoretically under many models, including resource games (BensonTilsen and Soares, 2016) and large classes of policies in discrete MDPs (Turner et al., 2019).

Original: While instrumental convergence is central to the discussion of value alignment and safe
Revised: While instrumental convergence is central to the discussion of value alignment and safe

Original: AI (Bostrom, 2017), since many instrumental subgoals have harmful effects, we believe that it is also a key to supporting agents with ill-deﬁned goals and values, such as humans.
Revised: AI (Bostrom, 2017), since many instrumental subgoals have harmful effects, we believe that it is also a key to supporting agents with ill-deﬁned goals and values, such as humans.

Original: The reason is that enabling instrumental subgoals for other agents (or not impeding them) can be beneﬁcial, for a wide variety of goals and preferences.
Revised: The reason is that enabling instrumental subgoals for other agents (or not impeding them) can be beneﬁcial, for a wide variety of goals and preferences.

Original: Since these subgoals occur frequently for rational agents, enabling them has the highest chance of success in the absence of more information about the other agent’s preferences, even if it is not guaranteed in the worst case.
Revised: Since these subgoals occur frequently for rational agents, enabling them has the highest chance of success in the absence of more information about the other agent’s preferences, even if it is not guaranteed in the worst case.

Original: We speculate that having the ability to reach many future states is one of the most general convergent subgoals.
Revised: We speculate that having the ability to reach many future states is one of the most general convergent subgoals.

Original: It subsumes self-preservation (avoiding absorbent states), resource acquisition (if they are prerequisites to some actions), and generally maintaining the ability to pursue many goals.
Revised: It subsumes self-preservation (avoiding absorbent states), resource acquisition (if they are prerequisites to some actions), and generally maintaining the ability to pursue many goals.

Original: There is theoretical evidence that many optimal agents pursue this subgoal (Turner et al., 2019)
Revised: There is theoretical evidence that many optimal agents pursue this subgoal (Turner et al., 2019)

Original: (see sec. 3.2).
Revised: (see sec. 3.2).

Original: Thus, we propose to train agents to support other agents by maximizing their choice (future state availability).
Revised: Thus, we propose to train agents to support other agents by maximizing their choice (future state availability).

Original: This unsupervised approach learns altruistic behaviour without any extrinsic supervision such as rewards or expert demonstrations.
Revised: This unsupervised approach learns altruistic behaviour without any extrinsic supervision such as rewards or expert demonstrations.

Original: We evaluate our methods in three diverse multi-agent environments.
Revised: We evaluate our methods in three diverse multi-agent environments.

Original: We always assume there are at least two agents: the leader agent that executes its own policy and can be trained using standard supervised methods, and an altruistic agent whose role is to help the leader.
Revised: We always assume there are at least two agents: the leader agent that executes its own policy and can be trained using standard supervised methods, and an altruistic agent whose role is to help the leader.

Original: The performance of the altruistic agent is thus deﬁned as the reward (success) achieved by the leader agent.
Revised: The performance of the altruistic agent is thus deﬁned as the reward (success) achieved by the leader agent.

Original: In all our environments, the overall success of the leader agent depends on the altruistic agents’ behaviour.
Revised: In all our environments, the overall success of the leader agent depends on the altruistic agents’ behaviour.

Original: We show that our unsupervised approach outperforms unsupervised baselines by a large margin and, in some cases, also outperforms the supervised ones.
Revised: We show that our unsupervised approach outperforms unsupervised baselines by a large margin and, in some cases, also outperforms the supervised ones.

Original: Finally, we demonstrate possible failure cases of our approach where maximising the leader agent’s choice can lead to suboptimal behaviour.
Revised: Finally, we demonstrate possible failure cases of our approach where maximising the leader agent’s choice can lead to suboptimal behaviour.

Original: Our work makes the following three contributions:
Revised: Our work makes the following three contributions:

Original: • We devise a multi-agent RL framework for intrinsically motivated artiﬁcial agents that act altruistically by maximising the choice of others.
Revised: • We devise a multi-agent RL framework for intrinsically motivated artiﬁcial agents that act altruistically by maximising the choice of others.

Original: • We deﬁne and evaluate three task-agnostic methods to estimate the choice that an agent has in a given situation, which are all related to the variety in states it can reach.
Revised: • We deﬁne and evaluate three task-agnostic methods to estimate the choice that an agent has in a given situation, which are all related to the variety in states it can reach.

Original: • We experimentally evaluate our unsupervised approach in three multi-agent environments and are able to match and, in some cases, outperform supervised baselines.
Revised: • We experimentally evaluate our unsupervised approach in three multi-agent environments and are able to match and, in some cases, outperform supervised baselines.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: To the best of our knowledge, we are the ﬁrst to experimentally evaluate unsupervised agents with purely altruistic objectives.
Revised: To the best of our knowledge, we are the ﬁrst to experimentally evaluate unsupervised agents with purely altruistic objectives.

Original: However, there are many related concepts in the literature.
Revised: However, there are many related concepts in the literature.

Original: In human robot cooperation , a robotic agent aids a human agent in achieving its goals (P´erezD’Arpino and Shah, 2015; Hadﬁeld-Menell et al., 2016; Baker et al., 2006; Dragan and Srinivasa, 2013; Fisac et al., 2017; 2020; Javdani et al., 2015; Dragan and Srinivasa, 2013;
Revised: In human robot cooperation , a robotic agent aids a human agent in achieving its goals (P´erezD’Arpino and Shah, 2015; Hadﬁeld-Menell et al., 2016; Baker et al., 2006; Dragan and Srinivasa, 2013; Fisac et al., 2017; 2020; Javdani et al., 2015; Dragan and Srinivasa, 2013;

Original: Macindoe et al., 2012; Pellegrinelli et al., 2016).
Revised: Macindoe et al., 2012; Pellegrinelli et al., 2016).

Original: Methods from Inverse RL (IRL) are often employed to infer human goals, which are then utilized by the robot agent to support the human.
Revised: Methods from Inverse RL (IRL) are often employed to infer human goals, which are then utilized by the robot agent to support the human.

Original: IRL itself aims to learn objectives from observations and can be used in single-agent (Fu et al., 2017) and multi-agent scenarios (Song et al., 2018; Yu et al., 2019; Jeon et al., 2020).
Revised: IRL itself aims to learn objectives from observations and can be used in single-agent (Fu et al., 2017) and multi-agent scenarios (Song et al., 2018; Yu et al., 2019; Jeon et al., 2020).

Original: However, IRL relies on the existence of expert demonstrations, which are often difﬁcult to get at scale.
Revised: However, IRL relies on the existence of expert demonstrations, which are often difﬁcult to get at scale.

Original: In complex environments, it also often suffers from ambiguity of solutions (Arora and Doshi, 2021).
Revised: In complex environments, it also often suffers from ambiguity of solutions (Arora and Doshi, 2021).

Original: In single-agent reinforcement learning, empowerment – which measures an agent’s capacity to affect its environment (Klyubin et al., 2005; 2008) – is used to enable intrinsically-motivated exploration (Gregor et al., 2016; Volpi and Polani, 2020).
Revised: In single-agent reinforcement learning, empowerment – which measures an agent’s capacity to affect its environment (Klyubin et al., 2005; 2008) – is used to enable intrinsically-motivated exploration (Gregor et al., 2016; Volpi and Polani, 2020).

Original: Empowerment is also used for multiagent cooperation (Guckelsberger et al., 2016; Du et al., 2020).
Revised: Empowerment is also used for multiagent cooperation (Guckelsberger et al., 2016; Du et al., 2020).

Original: Du et al.
Revised: Du et al.

Original: (2020) use empowerment to develop a helper agent that assists a (simulated) human agent by maximizing the human’s empowerment, constituting the research work most similar to ours.
Revised: (2020) use empowerment to develop a helper agent that assists a (simulated) human agent by maximizing the human’s empowerment, constituting the research work most similar to ours.

Original: In contrast to our approach, it requires privileged access to an environment simulator and therefore does not allow to learn helpful or altruistic behaviour only from observation.
Revised: In contrast to our approach, it requires privileged access to an environment simulator and therefore does not allow to learn helpful or altruistic behaviour only from observation.

Original: Furthermore, the approach is not unsupervised.
Revised: Furthermore, the approach is not unsupervised.

Original: There are also mathematical formalizations of instrumental convergence (Bostrom, 2017).
Revised: There are also mathematical formalizations of instrumental convergence (Bostrom, 2017).

Original: BensonTilsen and Soares (2016) analyze a MDP that makes ﬁnite resource allocation explicit, and ﬁnd that optimal agents with arbitrary reward functions tend to deplete available resources.
Revised: BensonTilsen and Soares (2016) analyze a MDP that makes ﬁnite resource allocation explicit, and ﬁnd that optimal agents with arbitrary reward functions tend to deplete available resources.

Original: Turner et al.
Revised: Turner et al.

Original: (2019) propose “power” as a convergent subgoal, which they deﬁne as the average difference between the state value of an optimal policy and the reward in the same state.
Revised: (2019) propose “power” as a convergent subgoal, which they deﬁne as the average difference between the state value of an optimal policy and the reward in the same state.

Original: They show that, for environments with certain symmetries, a larger proportion of optimal agents prefer states with higher power.
Revised: They show that, for environments with certain symmetries, a larger proportion of optimal agents prefer states with higher power.

Original: In sec.
Revised: In sec.

Original: 3.2 we will describe these symmetries and relate the result to our method.
Revised: 3.2 we will describe these symmetries and relate the result to our method.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we formalize our framework.
Revised: In this section, we formalize our framework.

Original: We start with the generic deﬁnition describing multiagent setting.
Revised: We start with the generic deﬁnition describing multiagent setting.

Original: Next, we describe our framework where we show various approaches to estimate choice for a single agent, and how it can be applied to a two-agents Markov Game.
Revised: Next, we describe our framework where we show various approaches to estimate choice for a single agent, and how it can be applied to a two-agents Markov Game.

Original: Markov Game.
Revised: Markov Game.

Original: We consider a Markov Game (Littman, 1994), which generalizes a Markov
Revised: We consider a Markov Game (Littman, 1994), which generalizes a Markov

Original: Decision Process (MDP) to a multi-agent scenario.
Revised: Decision Process (MDP) to a multi-agent scenario.

Original: In a Markov Game, agents interact in the same environment.
Revised: In a Markov Game, agents interact in the same environment.

Original: At time step t , each agent (the i th of a total of N agents) takes the action a ti , receives a reward r ti , and ﬁnally the environment transitions from state s t to s t +1 .
Revised: At time step t , each agent (the i th of a total of N agents) takes the action a ti , receives a reward r ti , and ﬁnally the environment transitions from state s t to s t +1 .

Original: A Markov Game is then deﬁned by a state space S ( s t ∈ S ), a distribution of initial states η , the action space
Revised: A Markov Game is then deﬁned by a state space S ( s t ∈ S ), a distribution of initial states η , the action space

Original: A i ( a ti ∈
Revised: A i ( a ti ∈

Original: A i ) and reward function r
Revised: A i ) and reward function r

Original: i ( s, a 1 , . . . , a N ) of each agent i , an environment state transition probability
Revised: i ( s, a 1 , . . . , a N ) of each agent i , an environment state transition probability

Original: P ( s t +1 | s t , a 1 , . .
Revised: P ( s t +1 | s t , a 1 , . .

Original: . , a N ) , and ﬁnally the agents’ discount factors γ i .
Revised: . , a N ) , and ﬁnally the agents’ discount factors γ i .

Original: 3. E STIMATING CHOICE FOR A SINGLE AGENT
Revised: 3. E STIMATING CHOICE FOR A SINGLE AGENT

Original: We ﬁrst consider a single-agent scenario, i.e. N = 1 , where only a leader agent, indicated by the subscript L , interacts with the environment through its pretrained stochastic policy π L .
Revised: We ﬁrst consider a single-agent scenario, i.e. N = 1 , where only a leader agent, indicated by the subscript L , interacts with the environment through its pretrained stochastic policy π L .

Original: We assume that the leader acts Boltzmann-rationally, i.e. that it chooses high-value actions with higher probability.
Revised: We assume that the leader acts Boltzmann-rationally, i.e. that it chooses high-value actions with higher probability.

Original: We believe this to be a reasonable assumption, as, in comparison to deterministic policies, stochastic policies are more robust (Zhang et al., 2020), and often achieve better results in real-world-alike partially observable stochastic domains (Kaelbling et al., 1998).
Revised: We believe this to be a reasonable assumption, as, in comparison to deterministic policies, stochastic policies are more robust (Zhang et al., 2020), and often achieve better results in real-world-alike partially observable stochastic domains (Kaelbling et al., 1998).

Original: We denote the leader agent’s generic choice in a given state s as C L ( s ) , for which we propose concrete realizations below.
Revised: We denote the leader agent’s generic choice in a given state s as C L ( s ) , for which we propose concrete realizations below.

Original: Each method relies on the random variable S t + n , with values s t + n ∈ S , which refers to the leader agent’s state after n environment transitions from a starting state s t .
Revised: Each method relies on the random variable S t + n , with values s t + n ∈ S , which refers to the leader agent’s state after n environment transitions from a starting state s t .

Original: Its probability mass function is deﬁned as the n -step state distribution of the underlying single-agent
Revised: Its probability mass function is deﬁned as the n -step state distribution of the underlying single-agent

Original: MDP, conditioned on the current state: p ( s t + n | s t ) =
Revised: MDP, conditioned on the current state: p ( s t + n | s t ) =

Original: P ( S t + n = s | π L , s t ) .
Revised: P ( S t + n = s | π L , s t ) .

Original: Discrete choice.
Revised: Discrete choice.

Original: Our ﬁrst derived method simply deﬁnes the choice of the leader agent in state s t as the number of states that it can reach within n transitions, which we refer to as its discrete choice :
Revised: Our ﬁrst derived method simply deﬁnes the choice of the leader agent in state s t as the number of states that it can reach within n transitions, which we refer to as its discrete choice :

Original: [Equation]
Revised: [Equation]

Original: where range( X ) is the set of all values that a random variable X takes on with positive probability and | · | measures the size of that set.
Revised: where range( X ) is the set of all values that a random variable X takes on with positive probability and | · | measures the size of that set.

Original: While this count-based estimator of choice is intuitive and easily interpretable, it can hardly be estimated practically in large or continuous state spaces.
Revised: While this count-based estimator of choice is intuitive and easily interpretable, it can hardly be estimated practically in large or continuous state spaces.

Original: Entropic choice.
Revised: Entropic choice.

Original: It can be shown that the entropy of a random variable X acts as a lower bound for the size of the set of values that X takes on with positive probability (Galvin, 2014, Property 2.6), i.e. H ( X ) ≤ log | range( X ) | .
Revised: It can be shown that the entropy of a random variable X acts as a lower bound for the size of the set of values that X takes on with positive probability (Galvin, 2014, Property 2.6), i.e. H ( X ) ≤ log | range( X ) | .

Original: We deﬁne a lower bound of the discrete choice by computing the
Revised: We deﬁne a lower bound of the discrete choice by computing the

Original: Shannon entropy of the n -step state distribution, which we refer to as the agent’s entropic choice :
Revised: Shannon entropy of the n -step state distribution, which we refer to as the agent’s entropic choice :

Original: [Equation]
Revised: [Equation]

Original: which estimates the agent’s choice as the variety in its state after n transitions.
Revised: which estimates the agent’s choice as the variety in its state after n transitions.

Original: Unlike eq. 1, EC nL can be computed in continuous state spaces or efﬁciently estimated by Monte Carlo sampling.
Revised: Unlike eq. 1, EC nL can be computed in continuous state spaces or efﬁciently estimated by Monte Carlo sampling.

Original: Immediate choice.
Revised: Immediate choice.

Original: To further simplify entropic choice and reduce its computational complexity, we may limit the look-ahead horizon to n = 1 and assume an injective relationship from actions to states, i.e. no two actions taken at s t lead to the equivalent state s t +1 .
Revised: To further simplify entropic choice and reduce its computational complexity, we may limit the look-ahead horizon to n = 1 and assume an injective relationship from actions to states, i.e. no two actions taken at s t lead to the equivalent state s t +1 .

Original: This assumption is often true in navigation environments, where different step-actions result in different states.
Revised: This assumption is often true in navigation environments, where different step-actions result in different states.

Original: We can then simplify the one-step state distribution of the leader agent to p ( s t + n | s t ) =
Revised: We can then simplify the one-step state distribution of the leader agent to p ( s t + n | s t ) =

Original: P ( S t +1 = s
Revised: P ( S t +1 = s

Original: | π L , s t ) =
Revised: | π L , s t ) =

Original: π ( a tL = a | s t ) , and compute a simpliﬁed, short-horizon entropic choice, the immediate choice :
Revised: π ( a tL = a | s t ) , and compute a simpliﬁed, short-horizon entropic choice, the immediate choice :

Original: [Equation]
Revised: [Equation]

Original: Immediate choice ( IC ) can be easily computed as the entropy over its policy conditioned on the current state.
Revised: Immediate choice ( IC ) can be easily computed as the entropy over its policy conditioned on the current state.

Original: Even though the assumptions made for immediate choice often do not hold in complex or real-world environments, we found empirically that this objective can yield good results.
Revised: Even though the assumptions made for immediate choice often do not hold in complex or real-world environments, we found empirically that this objective can yield good results.

Original: 3. O PTIMALITY OF CHOICE AS AN INSTRUMENTAL CONVERGENT SUBGOAL
Revised: 3. O PTIMALITY OF CHOICE AS AN INSTRUMENTAL CONVERGENT SUBGOAL

Original: Turner et al.
Revised: Turner et al.

Original: (2019) analyze the instrumental convergence of optimal agents on power-seeking subgoals and show that optimal policies tend to keep their options open (Prop. 6.9).
Revised: (2019) analyze the instrumental convergence of optimal agents on power-seeking subgoals and show that optimal policies tend to keep their options open (Prop. 6.9).

Original: They consider two distinct actions a and a (cid:48) taken at a state s (cid:48) , leading into two sets of possible future states (for an inﬁnite horizon) Reach( a ) and Reach( a (cid:48) ) respectively.
Revised: They consider two distinct actions a and a (cid:48) taken at a state s (cid:48) , leading into two sets of possible future states (for an inﬁnite horizon) Reach( a ) and Reach( a (cid:48) ) respectively.

Original: They also assume that the states in Reach( a ) ∪ Reach( a (cid:48) ) can only be reached from s (cid:48) by taking actions a or a (cid:48) .
Revised: They also assume that the states in Reach( a ) ∪ Reach( a (cid:48) ) can only be reached from s (cid:48) by taking actions a or a (cid:48) .

Original: Then they prove that, if Reach( a ) is “similar” to a subset of Reach( a (cid:48) ) , in the sense that they have the same transition function (up to arbitrary swapping of pairs of states), the probability of a being optimal is higher than the probability of a (cid:48) being optimal (for most reward function distributions).
Revised: Then they prove that, if Reach( a ) is “similar” to a subset of Reach( a (cid:48) ) , in the sense that they have the same transition function (up to arbitrary swapping of pairs of states), the probability of a being optimal is higher than the probability of a (cid:48) being optimal (for most reward function distributions).

Original: Note that, if Reach( a ) is similar to a strict subset of Reach( a (cid:48) ) , then Reach( a ) contains more states than Reach( a (cid:48) ) .
Revised: Note that, if Reach( a ) is similar to a strict subset of Reach( a (cid:48) ) , then Reach( a ) contains more states than Reach( a (cid:48) ) .

Original: Turner et al.
Revised: Turner et al.

Original: (2019) therefore lend theoretical support to our proposal: while there is no guarantee that any one optimal policy (corresponding to a rational agent with arbitrary reward function) pursues higher choice, in expectation (over a bounded space of reward functions) most policies do choose actions that lead to higher choice, all else being equal.
Revised: (2019) therefore lend theoretical support to our proposal: while there is no guarantee that any one optimal policy (corresponding to a rational agent with arbitrary reward function) pursues higher choice, in expectation (over a bounded space of reward functions) most policies do choose actions that lead to higher choice, all else being equal.

Original: As such, while we may not know a rational agent’s concrete goals, there is a high chance that choice works as an instrumental subgoal.
Revised: As such, while we may not know a rational agent’s concrete goals, there is a high chance that choice works as an instrumental subgoal.

Original: 3. C OMPARISON BETWEEN CHOICE AND EMPOWERMENT
Revised: 3. C OMPARISON BETWEEN CHOICE AND EMPOWERMENT

Original: The empowerment (Klyubin et al., 2005) of a leader agent in a given state s t and for horizon n is
Revised: The empowerment (Klyubin et al., 2005) of a leader agent in a given state s t and for horizon n is

Original: [Equation]
Revised: [Equation]

Original: with a n as a sequence of n actions of the leader agent and ω as a probing distribution over its nstep action sequences.
Revised: with a n as a sequence of n actions of the leader agent and ω as a probing distribution over its nstep action sequences.

Original: When setting the probing distribution ω equal to the leader agent’s policy, equation 4 simpliﬁes to E nL ( s t )
Revised: When setting the probing distribution ω equal to the leader agent’s policy, equation 4 simpliﬁes to E nL ( s t )

Original: = EC nL ( s t ) − H ( S t + n | A t + n , s t ) , with EC nL ( s t ) as the entropic choice of the leader agent introduced in equation 2.
Revised: = EC nL ( s t ) − H ( S t + n | A t + n , s t ) , with EC nL ( s t ) as the entropic choice of the leader agent introduced in equation 2.

Original: If we further assume deterministic environment transitions, then empowerment becomes equal to entropic choice, i.e. E nL ( s t )
Revised: If we further assume deterministic environment transitions, then empowerment becomes equal to entropic choice, i.e. E nL ( s t )

Original: = EC nL ( s t ) .
Revised: = EC nL ( s t ) .

Original: In contrast to the previously introduced methods to estimate choice of another agent, empowerment of another agent cannot be estimated from observations of the environment transitions.
Revised: In contrast to the previously introduced methods to estimate choice of another agent, empowerment of another agent cannot be estimated from observations of the environment transitions.

Original: To estimate another agent’s empowerment in a given state ( E nL ( s t ) ), access to its action space as well as privileged access to an environment simulator are be required, which violates the main assumption of our research work, i.e. learning to assist others only from observations of the environment transitions.
Revised: To estimate another agent’s empowerment in a given state ( E nL ( s t ) ), access to its action space as well as privileged access to an environment simulator are be required, which violates the main assumption of our research work, i.e. learning to assist others only from observations of the environment transitions.

Original: Even when assuming privileged access, computing empowerment in large or continuousstate environments often remains infeasible (Mohamed and Rezende, 2015; Gregor et al., 2016;
Revised: Even when assuming privileged access, computing empowerment in large or continuousstate environments often remains infeasible (Mohamed and Rezende, 2015; Gregor et al., 2016;

Original: Zhao et al., 2020), as it requires maximizing over all possible probing distributions ω of the leader agent.
Revised: Zhao et al., 2020), as it requires maximizing over all possible probing distributions ω of the leader agent.

Original: In contrast, estimating state entropy, as needed for the computation of the metrics introduced in this work, is feasible in large and continuous environments (Seo et al., 2021; Mutti et al., 2020).
Revised: In contrast, estimating state entropy, as needed for the computation of the metrics introduced in this work, is feasible in large and continuous environments (Seo et al., 2021; Mutti et al., 2020).

Original: 3. B EHAVING ALTRUISTICALLY BY MAXIMIZING ANOTHER AGENT ’ S CHOICE
Revised: 3. B EHAVING ALTRUISTICALLY BY MAXIMIZING ANOTHER AGENT ’ S CHOICE

Original: Having considered three methods to estimate an agent’s choice (eq. 1-3) we now apply them to a
Revised: Having considered three methods to estimate an agent’s choice (eq. 1-3) we now apply them to a

Original: Markov Game of two agents.
Revised: Markov Game of two agents.

Original: The main hypothesis is that maximizing the choice of another agent is likely to allow it to reach more favourable regions of the state-space (for many possible policies of the agent), thus supporting it without a task-speciﬁc reward signal.
Revised: The main hypothesis is that maximizing the choice of another agent is likely to allow it to reach more favourable regions of the state-space (for many possible policies of the agent), thus supporting it without a task-speciﬁc reward signal.

Original: Altruistic agent’s policy deﬁnition.
Revised: Altruistic agent’s policy deﬁnition.

Original: In this Markov Game, one agent is the leader, with the subscript L , and another one is the altruistic agent, with the subscript A .
Revised: In this Markov Game, one agent is the leader, with the subscript L , and another one is the altruistic agent, with the subscript A .

Original: We deﬁne the optimal policy of the altruistic agent as the one that maximizes the future discounted choice of the leader agent,
Revised: We deﬁne the optimal policy of the altruistic agent as the one that maximizes the future discounted choice of the leader agent,

Original: [Equation]
Revised: [Equation]

Original: where the generic choice C L ( s t ) can be estimated by one of several methods: discrete choice
Revised: where the generic choice C L ( s t ) can be estimated by one of several methods: discrete choice

Original: DC n L ( s t ) , entropic choice EC n L ( s t ) or immediate choice IC L ( s t ) .
Revised: DC n L ( s t ) , entropic choice EC n L ( s t ) or immediate choice IC L ( s t ) .

Original: Conditional estimates of choice.
Revised: Conditional estimates of choice.

Original: As the agents interact in the same environment, they both have inﬂuence over the system state s , which contains the state of both agents.
Revised: As the agents interact in the same environment, they both have inﬂuence over the system state s , which contains the state of both agents.

Original: This makes applying single-agent objectives based on the state distribution (such as eq. 1 and 2) difﬁcult to translate to a multi-agent setting, since the states of both agents are intermingled.
Revised: This makes applying single-agent objectives based on the state distribution (such as eq. 1 and 2) difﬁcult to translate to a multi-agent setting, since the states of both agents are intermingled.

Original: For example, an altruistic agent that maximizes entropic choice naively (eq. 2) will maximize both the state availability of the leader agent (which mirrors the single-agent entropic choice) and its own state availability (which does not contribute towards the altruism goal).
Revised: For example, an altruistic agent that maximizes entropic choice naively (eq. 2) will maximize both the state availability of the leader agent (which mirrors the single-agent entropic choice) and its own state availability (which does not contribute towards the altruism goal).

Original: To maximize entropic choice without also increasing the entropy of the altruistic agent’s actions, we propose to condition the choice estimate on the altruistic agent’s actions over the same time horizon, denoted by the random variable A t : t + n − 1 A
Revised: To maximize entropic choice without also increasing the entropy of the altruistic agent’s actions, we propose to condition the choice estimate on the altruistic agent’s actions over the same time horizon, denoted by the random variable A t : t + n − 1 A

Original: [Equation]
Revised: [Equation]

Original: In order to better understand eq.
Revised: In order to better understand eq.

Original: 6, we can use the chain rule of conditional entropy (Cover and
Revised: 6, we can use the chain rule of conditional entropy (Cover and

Original: Thomas, 2005, ch. 2) to decompose it into two terms: EC nL
Revised: Thomas, 2005, ch. 2) to decompose it into two terms: EC nL

Original: [Equation]
Revised: [Equation]

Original: H ( A t : t + n − 1 A | π L , s t ) , respectively the joint entropy of the states and actions, and the entropy of the actions.
Revised: H ( A t : t + n − 1 A | π L , s t ) , respectively the joint entropy of the states and actions, and the entropy of the actions.

Original: Therefore, we can interpret this objective as the altruistic agent maximizing the variety of states and actions, but subtracting the variety of its own actions, which is the undesired quantity.
Revised: Therefore, we can interpret this objective as the altruistic agent maximizing the variety of states and actions, but subtracting the variety of its own actions, which is the undesired quantity.

Original: We can also relate eq.
Revised: We can also relate eq.

Original: 6 to discrete choice (eq. 1).
Revised: 6 to discrete choice (eq. 1).

Original: Using the fact that H ( X | E ) ≤ log | range( P ( X | E )) | for a random variable X and event E (Galvin, 2014, Property 2.12), we see that eq.
Revised: Using the fact that H ( X | E ) ≤ log | range( P ( X | E )) | for a random variable X and event E (Galvin, 2014, Property 2.12), we see that eq.

Original: 6 is a lower bound for a count-based choice estimate (analogous to eq. 1), also conditioned on the altruistic agent’s actions: EC nL ( s t ) ≤ log DC nL ( s t ) = log
Revised: 6 is a lower bound for a count-based choice estimate (analogous to eq. 1), also conditioned on the altruistic agent’s actions: EC nL ( s t ) ≤ log DC nL ( s t ) = log

Original: | range (cid:0) S t + n | A t : t + n − 1 A , π L , s t (cid:1) | .
Revised: | range (cid:0) S t + n | A t : t + n − 1 A , π L , s t (cid:1) | .

Original: However, assuming simultaneous actions, the immediate choice estimate (eq. 3) stays unchanged, i.e. IC L ( s t )
Revised: However, assuming simultaneous actions, the immediate choice estimate (eq. 3) stays unchanged, i.e. IC L ( s t )

Original: =
Revised: =

Original: H ( π tL ( a | s t ) | a tA )
Revised: H ( π tL ( a | s t ) | a tA )

Original: =
Revised: =

Original: H ( π tL ( a | s t )) .
Revised: H ( π tL ( a | s t )) .

Original: The technical details of how these estimates can be computed from observations of the environment transitions are given in Appendix A.
Revised: The technical details of how these estimates can be computed from observations of the environment transitions are given in Appendix A.

Original:  E XPERIMENTAL EVALUATION
Revised:  E XPERIMENTAL EVALUATION

Original: We introduce three multi-agent environments of increasing complexity, in which the success of a leader agent depends on the behaviour of one or more additional agents.
Revised: We introduce three multi-agent environments of increasing complexity, in which the success of a leader agent depends on the behaviour of one or more additional agents.

Original: In each environment, we ﬁrst evaluate a subset of the proposed methods for choice estimation ( DC nL , EC nL and IC L ) by comparing the estimated choice of the leader agent in minimalistic scenarios.
Revised: In each environment, we ﬁrst evaluate a subset of the proposed methods for choice estimation ( DC nL , EC nL and IC L ) by comparing the estimated choice of the leader agent in minimalistic scenarios.

Original: We then evaluate our approach of behaving altruistically towards others by maximizing their choice (section 3.4) and measure performance of our approach as the reward achieved by the leader agent.
Revised: We then evaluate our approach of behaving altruistically towards others by maximizing their choice (section 3.4) and measure performance of our approach as the reward achieved by the leader agent.

Original: We provide videos of the emergent behaviours in the supp.
Revised: We provide videos of the emergent behaviours in the supp.

Original: mat.
Revised: mat.

Original: (see appendix F).
Revised: (see appendix F).

Original: We compare our method to both an unsupervised and a supervised approach.
Revised: We compare our method to both an unsupervised and a supervised approach.

Original: Note that the supervised approach has stronger assumptions, as it requires direct access to the leader agent’s reward function.
Revised: Note that the supervised approach has stronger assumptions, as it requires direct access to the leader agent’s reward function.

Original: We do not consider inverse RL (IRL) as a relevant baseline, as it would rely on demonstrations of expert behaviour, which we do not assume.
Revised: We do not consider inverse RL (IRL) as a relevant baseline, as it would rely on demonstrations of expert behaviour, which we do not assume.

Original: Even if perfect knowledge of the state transition probabilities is assumed, this does not allow generating expert demonstrations of the leader agent’s policy, as its expert policy would in turn depend on the policy of the altruistic agent, which is yet to be found by IRL.
Revised: Even if perfect knowledge of the state transition probabilities is assumed, this does not allow generating expert demonstrations of the leader agent’s policy, as its expert policy would in turn depend on the policy of the altruistic agent, which is yet to be found by IRL.

Original: 4. D ISCRETE ENVIRONMENTS WITH CONTROLLABLE GATES
Revised: 4. D ISCRETE ENVIRONMENTS WITH CONTROLLABLE GATES

Original: We start by considering three different scenarios on a grid, illustrated in Fig.
Revised: We start by considering three different scenarios on a grid, illustrated in Fig.

Original: 1 (top row), with the starting positions of the leader (green) and an additional agent (blue) shown in faded colors, obstacles are gray, and agents may move in one of the four cardinal directions or stay still.
Revised: 1 (top row), with the starting positions of the leader (green) and an additional agent (blue) shown in faded colors, obstacles are gray, and agents may move in one of the four cardinal directions or stay still.

Original: Choice estimate analysis.
Revised: Choice estimate analysis.

Original: We ﬁrst verify whether the estimated choice for each state (agent position) correctly maps to our intuitive understanding of choice (that is, the diversity of actions that can be taken).
Revised: We ﬁrst verify whether the estimated choice for each state (agent position) correctly maps to our intuitive understanding of choice (that is, the diversity of actions that can be taken).

Original: Therefore, we conducted an analysis of the estimated choice of the leader agent using a simpliﬁed version of the environment (Fig. 1, top left), in which only the leader agent is present and selects actions uniformly at random.
Revised: Therefore, we conducted an analysis of the estimated choice of the leader agent using a simpliﬁed version of the environment (Fig. 1, top left), in which only the leader agent is present and selects actions uniformly at random.

Original: Fig.
Revised: Fig.

Original: 1 (bottom row) shows the three different methods of estimating choice evaluated for each possible cell position of the leader agent.
Revised: 1 (bottom row) shows the three different methods of estimating choice evaluated for each possible cell position of the leader agent.

Original: We can observe that states in less conﬁned areas, e.g. further away from walls, generally feature higher choice estimates, with the least choice being afforded by the dead end at the right.
Revised: We can observe that states in less conﬁned areas, e.g. further away from walls, generally feature higher choice estimates, with the least choice being afforded by the dead end at the right.

Original: All three method’s estimates are qualitatively similar, which validates the chosen approximations.
Revised: All three method’s estimates are qualitatively similar, which validates the chosen approximations.

Original: In line with the simpliﬁcations made, the immediate choice (IC) estimates tend to be more local, as can be observed when comparing the estimates for the cell at row 2, column 4.
Revised: In line with the simpliﬁcations made, the immediate choice (IC) estimates tend to be more local, as can be observed when comparing the estimates for the cell at row 2, column 4.

Original: In conclusion, these results qualitatively agree with an intuitive understanding of choice of an agent in a grid environment.
Revised: In conclusion, these results qualitatively agree with an intuitive understanding of choice of an agent in a grid environment.

Original: Environment setup.
Revised: Environment setup.

Original: In the Door Scenario (Fig. 1, top center), the door switch (row 1, col. 8) can only be operated by the altruistic agent.
Revised: In the Door Scenario (Fig. 1, top center), the door switch (row 1, col. 8) can only be operated by the altruistic agent.

Original: The door (row 2, col. 4) remains open as long as the altruistic agent is on the switch cell and is closed otherwise.
Revised: The door (row 2, col. 4) remains open as long as the altruistic agent is on the switch cell and is closed otherwise.

Original: As the leader agent always starts to the left of the door and the altruistic agent to the right, the leader agent can only attain its goal, the apple (row 2, col. 6), if the altruistic agent uses the door switch to enable the leader agent to pass through the door.
Revised: As the leader agent always starts to the left of the door and the altruistic agent to the right, the leader agent can only attain its goal, the apple (row 2, col. 6), if the altruistic agent uses the door switch to enable the leader agent to pass through the door.

Original: In the Dead End Scenario (Fig. 1, top right), the door is always open, and the leader agent’s target object (green apple) is moved to the top right cell.
Revised: In the Dead End Scenario (Fig. 1, top right), the door is always open, and the leader agent’s target object (green apple) is moved to the top right cell.

Original: Hence, the leader agent can obtain the apple without additional help from the altruistic agent.
Revised: Hence, the leader agent can obtain the apple without additional help from the altruistic agent.

Original: However, the altruistic agent could potentially block the path by positioning itself at the entry to the dead end.
Revised: However, the altruistic agent could potentially block the path by positioning itself at the entry to the dead end.

Original: This situation would be the opposite of altruistic behaviour and is, of course, undesired.
Revised: This situation would be the opposite of altruistic behaviour and is, of course, undesired.

Original: We compare to a supervised approach, to Assistance via Empowerment (AvE, (Du et al., 2020)) and a random-policy baseline.
Revised: We compare to a supervised approach, to Assistance via Empowerment (AvE, (Du et al., 2020)) and a random-policy baseline.

Original: Assistance via Empowerment baseline.
Revised: Assistance via Empowerment baseline.

Original: We compare with the recently-proposed AvE, which has a similar goal (Du et al., 2020).
Revised: We compare with the recently-proposed AvE, which has a similar goal (Du et al., 2020).

Original: There are two major differences: AvE is not unsupervised, and it requires privileged access to an environment simulator to produce estimates.
Revised: There are two major differences: AvE is not unsupervised, and it requires privileged access to an environment simulator to produce estimates.

Original: Hence, its use in real or black-box environments is limited.
Revised: Hence, its use in real or black-box environments is limited.

Original: We used the authors’ implementation with ﬁxed hyperparameters, except for the crucial horizon n , for which we present a sweep in app.
Revised: We used the authors’ implementation with ﬁxed hyperparameters, except for the crucial horizon n , for which we present a sweep in app.

Original: B.
Revised: B.

Original: Training.
Revised: Training.

Original: We start by pretraining the leader agent with Q-Learning (Watkins and Dayan, 1992), with the altruistic agent executing a random policy.
Revised: We start by pretraining the leader agent with Q-Learning (Watkins and Dayan, 1992), with the altruistic agent executing a random policy.

Original: Hence, after convergence, the leader agent’s policy targets the green apple.
Revised: Hence, after convergence, the leader agent’s policy targets the green apple.

Original: Appendix B lists all details and parameters.
Revised: Appendix B lists all details and parameters.

Original: Afterwards, the leader agent’s learning is frozen and the altruistic agent is trained; it always observes the position of the leader agent s L , its own position s A , and the environment state s env , which is composed of the door state (open, closed) and the food state (present, eaten).
Revised: Afterwards, the leader agent’s learning is frozen and the altruistic agent is trained; it always observes the position of the leader agent s L , its own position s A , and the environment state s env , which is composed of the door state (open, closed) and the food state (present, eaten).

Original: The altruistic agent is trained with
Revised: The altruistic agent is trained with

Original: Q-Learning to maximize the discounted future choice of the leader agent (see eq.. 5.
Revised: Q-Learning to maximize the discounted future choice of the leader agent (see eq.. 5.

Original: For that, it uses one of the three proposed methods such as eq. 3, eq. 2 or eq.
Revised: For that, it uses one of the three proposed methods such as eq. 3, eq. 2 or eq.

Original: 1, as detailed in appendix A.1.
Revised: 1, as detailed in appendix A.1.

Original: Results.
Revised: Results.

Original: We investigate the developed behaviour of the altruistic agent after convergence for different choices of the hyperparameters – look-ahead horizon n ∈ { 1 , 3 , 12 } (which determines the scale at which choices are considered) and discount factor γ a ∈ { 0 . 1 , 0 . 7 } (which deﬁnes whether the altruistic agent gives higher importance to the short-term or long-term choice of the leader agent).
Revised: We investigate the developed behaviour of the altruistic agent after convergence for different choices of the hyperparameters – look-ahead horizon n ∈ { 1 , 3 , 12 } (which determines the scale at which choices are considered) and discount factor γ a ∈ { 0 . 1 , 0 . 7 } (which deﬁnes whether the altruistic agent gives higher importance to the short-term or long-term choice of the leader agent).

Original: Success is binary: either the leader agent attains its goal (green apple), or not.
Revised: Success is binary: either the leader agent attains its goal (green apple), or not.

Original: In the Door Scenario (Fig. 1, top center), we found that, for longer horizons n and higher discount factors γ a , the altruistic agent opens the door to allow the leader agent to reach its target, by occupying the switch position (square outline; row 1, col. 8).
Revised: In the Door Scenario (Fig. 1, top center), we found that, for longer horizons n and higher discount factors γ a , the altruistic agent opens the door to allow the leader agent to reach its target, by occupying the switch position (square outline; row 1, col. 8).

Original: For smaller n and lower γ a , the altruistic agent does not execute any coordinated policy and the leader does not succeed.
Revised: For smaller n and lower γ a , the altruistic agent does not execute any coordinated policy and the leader does not succeed.

Original: Using the
Revised: Using the

Original: AvE method, we ﬁnd that it only opens the door for n = 3 , but fails to do so for n = 1 and n = 12 .
Revised: AvE method, we ﬁnd that it only opens the door for n = 3 , but fails to do so for n = 1 and n = 12 .

Original: In the Dead End Scenario (Fig. 1, top right), we observe that, for longer horizons n and large discount factors γ a , the altruistic agent stays out of the leader agent’s way by occupying a far-away cell (square outline; row 1, col. 6).
Revised: In the Dead End Scenario (Fig. 1, top right), we observe that, for longer horizons n and large discount factors γ a , the altruistic agent stays out of the leader agent’s way by occupying a far-away cell (square outline; row 1, col. 6).

Original: For short horizons n and high discount factors γ a , the altruistic
Revised: For short horizons n and high discount factors γ a , the altruistic

Original: [Table]
Revised: [Table]

Original: agent actively blocks the entry to the hallway that contains the target (circle outline; row 3, col. 7), to prohibit the leader agent from entering this region of low estimated choice (recall that the choice for each cell is visualized in Fig.
Revised: agent actively blocks the entry to the hallway that contains the target (circle outline; row 3, col. 7), to prohibit the leader agent from entering this region of low estimated choice (recall that the choice for each cell is visualized in Fig.

Original: 1, bottom right).
Revised: 1, bottom right).

Original: This failure case can be prevented by having a large enough horizon n and discount factor γ a , analogously to the selection of the temperature hyperparameter in maximum entropy single-agent RL (Haarnoja and Abbeel, 2018).
Revised: This failure case can be prevented by having a large enough horizon n and discount factor γ a , analogously to the selection of the temperature hyperparameter in maximum entropy single-agent RL (Haarnoja and Abbeel, 2018).

Original: We ﬁnd that this conﬁguration performs consistently better than others in both scenarios, and hence is more preferred.
Revised: We ﬁnd that this conﬁguration performs consistently better than others in both scenarios, and hence is more preferred.

Original: On the other hand, the AvE method does not block the path of the leader agent for n = 1 , but blocks its path for n = 3 and n = 12 .
Revised: On the other hand, the AvE method does not block the path of the leader agent for n = 1 , but blocks its path for n = 3 and n = 12 .

Original: We found that the resulting behaviour of our approach is independent of the used method for choice estimation, i.e. either discrete choice (eq. 1) or entropic choice (eq. 2) yield the same outcome, with immediate choice (eq. 3) being a special case of entropic choice.
Revised: We found that the resulting behaviour of our approach is independent of the used method for choice estimation, i.e. either discrete choice (eq. 1) or entropic choice (eq. 2) yield the same outcome, with immediate choice (eq. 3) being a special case of entropic choice.

Original: As for the AvE baseline, we hypothesize that the variance of results is due to the nature of the proxy used in practice, which includes components of empowerment from both agents (sec. 3.4).
Revised: As for the AvE baseline, we hypothesize that the variance of results is due to the nature of the proxy used in practice, which includes components of empowerment from both agents (sec. 3.4).

Original: The binary outcomes for all hyperparameter combinations are given in appendix
Revised: The binary outcomes for all hyperparameter combinations are given in appendix

Original: B.
Revised: B.

Original: We also compare to a supervised baseline (receiving a reward when the leader obtains the apple), in which case the leader always succeeds.
Revised: We also compare to a supervised baseline (receiving a reward when the leader obtains the apple), in which case the leader always succeeds.

Original: 4. L EVEL - BASED FORAGING EXPERIMENTS Computational efﬁciency.
Revised: 4. L EVEL - BASED FORAGING EXPERIMENTS Computational efﬁciency.

Original: Due to the computational complexity resulting from the need to estimate a long-term distribution of states ( p ( s t + n | s t ), we focus on immediate choice (IC) to estimate the leader agent’s choice in the remaining sections.
Revised: Due to the computational complexity resulting from the need to estimate a long-term distribution of states ( p ( s t + n | s t ), we focus on immediate choice (IC) to estimate the leader agent’s choice in the remaining sections.

Original: Furthermore, in rare state-action sequences, the assumptions made for IC, i.e. deterministic environment transitions and an injective relationship from actions to states, may not hold.
Revised: Furthermore, in rare state-action sequences, the assumptions made for IC, i.e. deterministic environment transitions and an injective relationship from actions to states, may not hold.

Original: Nonetheless, we did not ﬁnd this to adversely affect the results.
Revised: Nonetheless, we did not ﬁnd this to adversely affect the results.

Original: Due to its dependence on access to the environment simulator and its computational complexity, we do not consider the AvE baseline for the remainder of experiments.
Revised: Due to its dependence on access to the environment simulator and its computational complexity, we do not consider the AvE baseline for the remainder of experiments.

Original: Setup.
Revised: Setup.

Original: We use a fully-observable multi-agent environment that enables us to assess the level of cooperation among agents (level-based foraging, LBF, Christianos et al.
Revised: We use a fully-observable multi-agent environment that enables us to assess the level of cooperation among agents (level-based foraging, LBF, Christianos et al.

Original: to evaluate the performance of altruistic agents in more complex environments with discrete state spaces.
Revised: to evaluate the performance of altruistic agents in more complex environments with discrete state spaces.

Original: We compare our method to a maximum-entropy approach from single-agent RL (Mutti et al., 2020) and a random-policy baseline.
Revised: We compare our method to a maximum-entropy approach from single-agent RL (Mutti et al., 2020) and a random-policy baseline.

Original: A visualization of the environment is depicted in Fig.
Revised: A visualization of the environment is depicted in Fig.

Original: 2 (left).
Revised: 2 (left).

Original: The two agents can forage apples by simultaneously taking positions at different sides of a targeted apple, yielding a ﬁxed reward.
Revised: The two agents can forage apples by simultaneously taking positions at different sides of a targeted apple, yielding a ﬁxed reward.

Original: We ﬁrst train two agents – which receive an equal reward for foraging – using Deep Q-Learning (DQL, Van Hasselt et al.
Revised: We ﬁrst train two agents – which receive an equal reward for foraging – using Deep Q-Learning (DQL, Van Hasselt et al.

Original: , corresponding to fully-supervised sharedreward in multi-agent reinforcement learning (MARL).
Revised: , corresponding to fully-supervised sharedreward in multi-agent reinforcement learning (MARL).

Original: We then take one of these pretrained agents that has learned to forage apples when accompanied by a cooperating agent, freeze its policy, and place it as the leader agent (green) into the test scenario (additional details are provided in app.
Revised: We then take one of these pretrained agents that has learned to forage apples when accompanied by a cooperating agent, freeze its policy, and place it as the leader agent (green) into the test scenario (additional details are provided in app.

Original: C).
Revised: C).

Original: Choice estimate analysis.
Revised: Choice estimate analysis.

Original: We ﬁrst qualitatively evaluate IC as an estimator for choice in Fig.
Revised: We ﬁrst qualitatively evaluate IC as an estimator for choice in Fig.

Original: 3, by comparing representative scenarios.
Revised: 3, by comparing representative scenarios.

Original: To quantitatively analyse IC as an estimator for the leader agent’s choice, we compare the leader agent’s average IC (over 100 episodes) in two scenarios, one in which it can acquire many rewards, i.e. the other agent acts cooperatively, and one where it can acquire only few rewards, i.e. the other agent takes random actions.
Revised: To quantitatively analyse IC as an estimator for the leader agent’s choice, we compare the leader agent’s average IC (over 100 episodes) in two scenarios, one in which it can acquire many rewards, i.e. the other agent acts cooperatively, and one where it can acquire only few rewards, i.e. the other agent takes random actions.

Original: We show the results in Table
Revised: We show the results in Table

Original: We observe that the leader agent’s estimated choice is substantially higher when it is able to acquire high rewards.
Revised: We observe that the leader agent’s estimated choice is substantially higher when it is able to acquire high rewards.

Original: Note that the IC estimate does not have privileged access to the reward function of the leader agent and so this experiment evaluates its worth as a generic proxy for the leader’s reward.
Revised: Note that the IC estimate does not have privileged access to the reward function of the leader agent and so this experiment evaluates its worth as a generic proxy for the leader’s reward.

Original: Assuming that an agent is able to acquire higher rewards when having more choice, these results indicate that IC is a reasonable estimator for the leader agent’s choice in LBF.
Revised: Assuming that an agent is able to acquire higher rewards when having more choice, these results indicate that IC is a reasonable estimator for the leader agent’s choice in LBF.

Original: Training.
Revised: Training.

Original: We now consider an environment that consists of the previously pretrained leader and an additional altruistic agent, which is trained from scratch and does not receive a reward for foraging apples, but is rewarded according to the leader agent’s choice.
Revised: We now consider an environment that consists of the previously pretrained leader and an additional altruistic agent, which is trained from scratch and does not receive a reward for foraging apples, but is rewarded according to the leader agent’s choice.

Original: Its reward is given as the current estimate of the leader agent’s IC (eq. 3) and it is trained using DQL.
Revised: Its reward is given as the current estimate of the leader agent’s IC (eq. 3) and it is trained using DQL.

Original: To compute its internal reward signal, the altruistic agent would therefore need to estimate the state transition probabilities, as detailed in A.2.
Revised: To compute its internal reward signal, the altruistic agent would therefore need to estimate the state transition probabilities, as detailed in A.2.

Original: To decouple our approach’s performance from that of the state transition estimator, we instead directly compute the altruistic agent’s reward using the leader agent’s policy.
Revised: To decouple our approach’s performance from that of the state transition estimator, we instead directly compute the altruistic agent’s reward using the leader agent’s policy.

Original: Results.
Revised: Results.

Original: We deﬁne the performance of the altruistic agent not as its achieved internal reward but as the reward achieved by the leader agent, i.e. its performance in enabling the leader agent to forage apples.
Revised: We deﬁne the performance of the altruistic agent not as its achieved internal reward but as the reward achieved by the leader agent, i.e. its performance in enabling the leader agent to forage apples.

Original: Fig.
Revised: Fig.

Original: 4 shows a comparison of the altruistic agent’s performance to that achieved bybaselines (two unsupervised and one supervised), averaged over 5 random seeds, with the standard deviation as the shaded area.
Revised: 4 shows a comparison of the altruistic agent’s performance to that achieved bybaselines (two unsupervised and one supervised), averaged over 5 random seeds, with the standard deviation as the shaded area.

Original: It can be observed that the performance of the altruistic agent converges to a similar performance to that of the supervised agent, and outperforms the baseline approaches by a large margin.
Revised: It can be observed that the performance of the altruistic agent converges to a similar performance to that of the supervised agent, and outperforms the baseline approaches by a large margin.

Original: Furthermore, the IC improvement of the leader agent is correlated with its reward improvement, which supports using IC as a reasonable proxy for the choice of the leader agent.
Revised: Furthermore, the IC improvement of the leader agent is correlated with its reward improvement, which supports using IC as a reasonable proxy for the choice of the leader agent.

Original: 4. M ULTI - AGENT TAG GAME WITH PROTECTIVE AGENTS
Revised: 4. M ULTI - AGENT TAG GAME WITH PROTECTIVE AGENTS

Original: Setup.
Revised: Setup.

Original: We use a multi-agent tag environment (Tag, Mordatch and Abbeel (2018); Lowe et al.
Revised: We use a multi-agent tag environment (Tag, Mordatch and Abbeel (2018); Lowe et al.

Original: (2017); Terry et al.
Revised: (2017); Terry et al.

Original: , illustrated in Fig.
Revised: , illustrated in Fig.

Original: 2 (right), to evaluate the capabilities of altruistic agents in complex environments with continuous state spaces.
Revised: 2 (right), to evaluate the capabilities of altruistic agents in complex environments with continuous state spaces.

Original: Adversaries are rewarded for catching the leader, which in turn receives a negative reward for being caught or crossing the environment boundaries.
Revised: Adversaries are rewarded for catching the leader, which in turn receives a negative reward for being caught or crossing the environment boundaries.

Original: To speed up training, altruistic agents additionally receive a small negative reward for violating the environment boundaries.
Revised: To speed up training, altruistic agents additionally receive a small negative reward for violating the environment boundaries.

Original: We pretrain the adversaries and the leader (without the presence of altruistic agents) using MADDPG (Lowe et al., 2017) and DDPG (Lillicrap et al., 2016) respectively.
Revised: We pretrain the adversaries and the leader (without the presence of altruistic agents) using MADDPG (Lowe et al., 2017) and DDPG (Lillicrap et al., 2016) respectively.

Original: After pretraining, the adversary agents have learned to cooperatively chase the leader agent, which in turn has learned to ﬂee from the adversaries.
Revised: After pretraining, the adversary agents have learned to cooperatively chase the leader agent, which in turn has learned to ﬂee from the adversaries.

Original: Exact setup speciﬁcations and all parameters are given in appendix D.
Revised: Exact setup speciﬁcations and all parameters are given in appendix D.

Original: Choice estimate analysis.
Revised: Choice estimate analysis.

Original: As done for LBF, we evaluate the IC of the leader agent in representative scenarios in Fig.
Revised: As done for LBF, we evaluate the IC of the leader agent in representative scenarios in Fig.

Original: We also quantitatively evaluate IC as an estimator for the leader agent’s choice, by comparing the leader agent’s IC per timestep for a scenario in which it receives high rewards to
Revised: We also quantitatively evaluate IC as an estimator for the leader agent’s choice, by comparing the leader agent’s IC per timestep for a scenario in which it receives high rewards to

Original: [Table] [Table]
Revised: [Table] [Table]

Original: one where it receives low rewards.
Revised: one where it receives low rewards.

Original: We again hypothesize that the leader agent is able to acquire higher rewards when having more choice.
Revised: We again hypothesize that the leader agent is able to acquire higher rewards when having more choice.

Original: Table 1 shows that the estimated choice is substantially higher in the high-success scenario, indicating that IC is a reasonable estimator also in Tag.
Revised: Table 1 shows that the estimated choice is substantially higher in the high-success scenario, indicating that IC is a reasonable estimator also in Tag.

Original: Training.
Revised: Training.

Original: We freeze the pretrained policies of the adversary agents and the leader agent and insert three additional altruistic agents which observe all agents but are not observed themselves.
Revised: We freeze the pretrained policies of the adversary agents and the leader agent and insert three additional altruistic agents which observe all agents but are not observed themselves.

Original: Each additional altruistic agent’s internal reward signal is given as the IC of the leader agent (equation 3), which is directly computed as done in LBF (see 4.2).
Revised: Each additional altruistic agent’s internal reward signal is given as the IC of the leader agent (equation 3), which is directly computed as done in LBF (see 4.2).

Original: Results.
Revised: Results.

Original: Performance of the altruistic agents is deﬁned as the times per episode that the leader agent is caught by the adversaries, i.e. the lower the better.
Revised: Performance of the altruistic agents is deﬁned as the times per episode that the leader agent is caught by the adversaries, i.e. the lower the better.

Original: In Table 2, the performance of the team of three altruistically trained agents (ours) is compared to three relevant baselines, with the altruistic agents either removed (None), acting randomly (random), or solely receiving a small negative reward for violating the environment boundaries (cage).
Revised: In Table 2, the performance of the team of three altruistically trained agents (ours) is compared to three relevant baselines, with the altruistic agents either removed (None), acting randomly (random), or solely receiving a small negative reward for violating the environment boundaries (cage).

Original: In contrast to LBF, we do not compare to an unsupervised exploration approach, as we are not aware of such an implementation for cooperative MARL.
Revised: In contrast to LBF, we do not compare to an unsupervised exploration approach, as we are not aware of such an implementation for cooperative MARL.

Original: Additionally, we report results for the case in which the altruistic agents receive the same reward as the leader agent (supervised), possibly appended by a negative reward for violating the environment boundaries (supervised + cage).
Revised: Additionally, we report results for the case in which the altruistic agents receive the same reward as the leader agent (supervised), possibly appended by a negative reward for violating the environment boundaries (supervised + cage).

Original: It can be observed that our approach outperforms all relevant baselines by a substantial margin and also outperforms the supervised approach.
Revised: It can be observed that our approach outperforms all relevant baselines by a substantial margin and also outperforms the supervised approach.

Original: We hypothesize this to be due to the dense internal reward signal that our approach provides, as compared to the sparse rewards in the supervised scenario: recall that in the supervised scenario the additional altruistic agents receive a large negative reward only when the leader agent is caught by the adversaries, whereas our approach provides a dense reward signal that corresponds to the current estimate of the leader agent’s choice.
Revised: We hypothesize this to be due to the dense internal reward signal that our approach provides, as compared to the sparse rewards in the supervised scenario: recall that in the supervised scenario the additional altruistic agents receive a large negative reward only when the leader agent is caught by the adversaries, whereas our approach provides a dense reward signal that corresponds to the current estimate of the leader agent’s choice.

Original: Fig.
Revised: Fig.

Original: 5 displays the emerging protective behaviour of altruistic agents trained with our approach.
Revised: 5 displays the emerging protective behaviour of altruistic agents trained with our approach.

Original: Results videos are found in the supplemental material.
Revised: Results videos are found in the supplemental material.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We lay out some initial steps into developing artiﬁcial agents that learn altruistic behaviour from observations and interactions with other agents.
Revised: We lay out some initial steps into developing artiﬁcial agents that learn altruistic behaviour from observations and interactions with other agents.

Original: Our experimental results demonstrate that artiﬁcial agents can behave altruistically towards other agents without knowledge of their objective or any external supervision, by actively maximizing their choice.
Revised: Our experimental results demonstrate that artiﬁcial agents can behave altruistically towards other agents without knowledge of their objective or any external supervision, by actively maximizing their choice.

Original: This objective is justiﬁed by theoretical work on instrumental convergence, which shows that for a large proportion of rational agents this will be a useful subgoal, and thus can be leveraged to design generally altruistic agents.
Revised: This objective is justiﬁed by theoretical work on instrumental convergence, which shows that for a large proportion of rational agents this will be a useful subgoal, and thus can be leveraged to design generally altruistic agents.

Original: This work was motivated by a desire to address the potential negative outcomes of deploying agents that are oblivious to the values and objectives of others into the real world.
Revised: This work was motivated by a desire to address the potential negative outcomes of deploying agents that are oblivious to the values and objectives of others into the real world.

Original: As such, we hope that our work serves both as a baseline and facilitator for future research into value alignment in simulation settings, and as a complementary objective to standard RL that biases the behaviour towards more altruistic policies.
Revised: As such, we hope that our work serves both as a baseline and facilitator for future research into value alignment in simulation settings, and as a complementary objective to standard RL that biases the behaviour towards more altruistic policies.

Original: In addition to the positive impacts of deployed altruistic agents outside of simulation, we remark that altruistic proxy objectives do not yet come with strict guarantees of optimizing for other agents’ rewards, and identify failure modes (sec. 4.1) which are hyperparameter-dependent, and which we hope provide interesting starting points for future work.
Revised: In addition to the positive impacts of deployed altruistic agents outside of simulation, we remark that altruistic proxy objectives do not yet come with strict guarantees of optimizing for other agents’ rewards, and identify failure modes (sec. 4.1) which are hyperparameter-dependent, and which we hope provide interesting starting points for future work.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: We addressed the relevant aspects in our conclusion and have no conﬂicts of interest to declare.
Revised: We addressed the relevant aspects in our conclusion and have no conﬂicts of interest to declare.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We provide detailed descriptions of our experiments in the appendix and list all relevant parameters in table 4.
Revised: We provide detailed descriptions of our experiments in the appendix and list all relevant parameters in table 4.

Original: All experiments were run on single cores of Intel Xeon E7-8867v3 processors (2.5 GHz).
Revised: All experiments were run on single cores of Intel Xeon E7-8867v3 processors (2.5 GHz).

Original: Training times are given in the respective sections in the appendix.
Revised: Training times are given in the respective sections in the appendix.

Original: For the LBF and Tag experiments, we report mean and standard deviation over ﬁve different random seeds.
Revised: For the LBF and Tag experiments, we report mean and standard deviation over ﬁve different random seeds.

Original: The Gridworld experiments yield deterministic results.
Revised: The Gridworld experiments yield deterministic results.

Original: We will provide the source code for all experiments conducted with the ﬁnal version of this publication.
Revised: We will provide the source code for all experiments conducted with the ﬁnal version of this publication.

Original: We created detailed instructions on how to run the code in order to replicate the experimental outcomes presented in this work.
Revised: We created detailed instructions on how to run the code in order to replicate the experimental outcomes presented in this work.

Original: A E STIMATION OF LEADER AGENT ’ S CHOICE FROM OBSERVATION A. M ODEL - BASED ESTIMATION OF CHOICE FROM OBSERVATIONS
Revised: A E STIMATION OF LEADER AGENT ’ S CHOICE FROM OBSERVATION A. M ODEL - BASED ESTIMATION OF CHOICE FROM OBSERVATIONS

Original: We introduce a model-based estimator of choice that is suitable for small-scale discrete-state environments, having the advantage that it is easily interpretable.
Revised: We introduce a model-based estimator of choice that is suitable for small-scale discrete-state environments, having the advantage that it is easily interpretable.

Original: Recalling how we compute the discrete choice and entropic choice estimates for the leader agent, an estimate of the n -step state distribution conditioned on the altruistic agent’s actions is needed, i.e. P ( s t + n | π L , a t : t + n − 1 A , s t ) .
Revised: Recalling how we compute the discrete choice and entropic choice estimates for the leader agent, an estimate of the n -step state distribution conditioned on the altruistic agent’s actions is needed, i.e. P ( s t + n | π L , a t : t + n − 1 A , s t ) .

Original: To simplify this computation, we assume the altruistic agent’s action to equal hold for the next n steps.
Revised: To simplify this computation, we assume the altruistic agent’s action to equal hold for the next n steps.

Original: More speciﬁcally, we assume that the altruistic agent’s state is unchanged for the next n steps.
Revised: More speciﬁcally, we assume that the altruistic agent’s state is unchanged for the next n steps.

Original: Furthermore assuming that both the state and the action space are discrete, we compute
Revised: Furthermore assuming that both the state and the action space are discrete, we compute

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: where the state transition matrix T ( s A ) holds the transition probabilities between all possible states, as a function of the state of the altruistic agent s A .
Revised: where the state transition matrix T ( s A ) holds the transition probabilities between all possible states, as a function of the state of the altruistic agent s A .

Original: To compute T ( s A ) , the system state is encoded into a one-hot vector s 1 .
Revised: To compute T ( s A ) , the system state is encoded into a one-hot vector s 1 .

Original: The n -step discrete choice of the leader agent can then be computed as
Revised: The n -step discrete choice of the leader agent can then be computed as

Original: [Equation]
Revised: [Equation]

Original: its n -step entropic choice as
Revised: its n -step entropic choice as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and its immediate choice as
Revised: and its immediate choice as

Original: [Equation]
Revised: [Equation]

Original: In environments with a discrete state and action space, the altruistic agent can hence use an estimate of the state transition matrix T to estimate the choice of the leader agent using either of the proposed methods, i.e. DC , EC or IC .
Revised: In environments with a discrete state and action space, the altruistic agent can hence use an estimate of the state transition matrix T to estimate the choice of the leader agent using either of the proposed methods, i.e. DC , EC or IC .

Original: An estimate of T can be built over time, by observing the environment transitions and computing the transition probabilities as relative frequencies of observed transitions.
Revised: An estimate of T can be built over time, by observing the environment transitions and computing the transition probabilities as relative frequencies of observed transitions.

Original: A. M ODEL - FREE ESTIMATION OF CHOICE FROM OBSERVATIONS
Revised: A. M ODEL - FREE ESTIMATION OF CHOICE FROM OBSERVATIONS

Original: To limit the computational complexity, which is important for environments with large or continuous state spaces, we also consider immediate choice as an estimator for the leader agent’s choice ( IC L ( s t )
Revised: To limit the computational complexity, which is important for environments with large or continuous state spaces, we also consider immediate choice as an estimator for the leader agent’s choice ( IC L ( s t )

Original: =
Revised: =

Original: H ( S t +1 | s t )) .
Revised: H ( S t +1 | s t )) .

Original: As shown in section 3.1, this estimate can be simpliﬁed to
Revised: As shown in section 3.1, this estimate can be simpliﬁed to

Original: H ( S t +1 | s t ))
Revised: H ( S t +1 | s t ))

Original: =
Revised: =

Original: H ( π tL ( a | s t )) , under the named assumptions.
Revised: H ( π tL ( a | s t )) , under the named assumptions.

Original: Hence, to compute the immediate choice of the leader, the altruistic agent requires an estimate of the leader agent’s policy entropy, which can be learned from observation using a policy estimation network (Hong et al., 2018;
Revised: Hence, to compute the immediate choice of the leader, the altruistic agent requires an estimate of the leader agent’s policy entropy, which can be learned from observation using a policy estimation network (Hong et al., 2018;

Original: Papoudakis et al., 2020; Mao et al., 2019; Grover et al., 2018).
Revised: Papoudakis et al., 2020; Mao et al., 2019; Grover et al., 2018).

Original: B G RIDWORLD EXPERIMENTS B. T RAINING PROCEDURE B.1. S ETUP
Revised: B G RIDWORLD EXPERIMENTS B. T RAINING PROCEDURE B.1. S ETUP

Original: The environment setup is described and displayed in section 4.1.
Revised: The environment setup is described and displayed in section 4.1.

Original: AvE baseline.
Revised: AvE baseline.

Original: We evaluate the AvE baseline for different horizons n .
Revised: We evaluate the AvE baseline for different horizons n .

Original: For each horizon, we tested the AvE baseline as implemented in the provided source code 1 , using the hyper-parameters suggested by the authors.
Revised: For each horizon, we tested the AvE baseline as implemented in the provided source code 1 , using the hyper-parameters suggested by the authors.

Original: The original implementation uses a look-ahead horizon n = 10 .
Revised: The original implementation uses a look-ahead horizon n = 10 .

Original: We found
Revised: We found

Original: [Table]
Revised: [Table]

Original: that results are equal for both n = 10 and n = 12 ,
Revised: that results are equal for both n = 10 and n = 12 ,

Original: [Table]
Revised: [Table]

Original: is why we only display results for n = 12 .
Revised: is why we only display results for n = 12 .

Original: We further evaluated the AvE baseline for n between 1 and 12.
Revised: We further evaluated the AvE baseline for n between 1 and 12.

Original: For the Opens door task, we found that AvE yields success for n = 2 , 3 , 4 , 5 and failing for the remaining.
Revised: For the Opens door task, we found that AvE yields success for n = 2 , 3 , 4 , 5 and failing for the remaining.

Original: For the Non blocking task, we found that AvE yields success for n = 1 , 2 and failing for the remaining
Revised: For the Non blocking task, we found that AvE yields success for n = 1 , 2 and failing for the remaining

Original: . B.1. P RETRAINING
Revised: . B.1. P RETRAINING

Original: We ﬁrst pretrain the leader agent using tabular Q-Learning, with learning parameters given in Table 4.
Revised: We ﬁrst pretrain the leader agent using tabular Q-Learning, with learning parameters given in Table 4.

Original: During this pretraining, the altruistic agent takes random actions.
Revised: During this pretraining, the altruistic agent takes random actions.

Original: We train until all Q-Values are fully converged, i.e. training runs for 300000 environment steps.
Revised: We train until all Q-Values are fully converged, i.e. training runs for 300000 environment steps.

Original: B.1. R EWARD COMPUTATION FOR ALTRUISTIC AGENTS
Revised: B.1. R EWARD COMPUTATION FOR ALTRUISTIC AGENTS

Original: The altruistic agent is then also trained using tabular Q-Learning, and its internal reward signal is given as the choice estimate of the leader agent, i.e. either DC nL ( s t ) , EC nL ( s t ) or IC L ( s t ) , which is computed with the model based-estimation introduced in appendix A.1.
Revised: The altruistic agent is then also trained using tabular Q-Learning, and its internal reward signal is given as the choice estimate of the leader agent, i.e. either DC nL ( s t ) , EC nL ( s t ) or IC L ( s t ) , which is computed with the model based-estimation introduced in appendix A.1.

Original: The altruistic agent records all environment transitions and frequently updates its estimate of the state transition matrix T ( s A ) , which is needed to compute the internal reward signal for the altruistic agent.
Revised: The altruistic agent records all environment transitions and frequently updates its estimate of the state transition matrix T ( s A ) , which is needed to compute the internal reward signal for the altruistic agent.

Original: All training parameters can be found in Table 4.
Revised: All training parameters can be found in Table 4.

Original: Training time is about 15 minutes per experiment.
Revised: Training time is about 15 minutes per experiment.

Original: B. P ERFORMANCE EVALUATION
Revised: B. P ERFORMANCE EVALUATION

Original: Performance of the altruistic agent is reported for two different categories, as shown in Table 3.
Revised: Performance of the altruistic agent is reported for two different categories, as shown in Table 3.

Original: For each category, we report success or failure for choice estimate look-ahead horizons n ∈ { 1 , 3 , 12 } and discount factors of the altruistic agent γ a ∈ { 0 . 1 , 0 . 7 } .
Revised: For each category, we report success or failure for choice estimate look-ahead horizons n ∈ { 1 , 3 , 12 } and discount factors of the altruistic agent γ a ∈ { 0 . 1 , 0 . 7 } .

Original: Success or failure was always deterministic, conditioned on the experiment setup, i.e. 10 simulations were run for each setup which always yielded the same outcome.
Revised: Success or failure was always deterministic, conditioned on the experiment setup, i.e. 10 simulations were run for each setup which always yielded the same outcome.

Original: To estimate the leader agent’s choice, the altruistic agent uses either discrete choice ( D , equations 1 and 9) or entropic choice ( E , equations 2 and 10).
Revised: To estimate the leader agent’s choice, the altruistic agent uses either discrete choice ( D , equations 1 and 9) or entropic choice ( E , equations 2 and 10).

Original: It must be noted that horizon n = 12 is equivalent to an inﬁnite horizon look-ahead for the given environment size and that entropic choice is equivalent to immediate choice (equations 3 and 11) at horizon n = 1 , as the environment satisﬁes the necessary conditions listed for equation 3.
Revised: It must be noted that horizon n = 12 is equivalent to an inﬁnite horizon look-ahead for the given environment size and that entropic choice is equivalent to immediate choice (equations 3 and 11) at horizon n = 1 , as the environment satisﬁes the necessary conditions listed for equation 3.

Original: Table 3 displays the results of this experiment.
Revised: Table 3 displays the results of this experiment.

Original: In the ﬁrst row, it is evaluated whether the altruistic agent opens the door at all times, such that the leader agent can eat the green apple.
Revised: In the ﬁrst row, it is evaluated whether the altruistic agent opens the door at all times, such that the leader agent can eat the green apple.

Original: It can be observed that the altruistic agent only opens the door for longer horizons n , respectively higher discount factors γ a .
Revised: It can be observed that the altruistic agent only opens the door for longer horizons n , respectively higher discount factors γ a .

Original: Given the deﬁnitions of discrete choice (Equation 1) and entropic choice (Equation 2), it can be assumed that the choice horizon n determines the locality for which choice is considered and that the discount factor γ a deﬁnes whether the altruistic agent gives higher importance to the short-term or long-term choice of the leader agent.
Revised: Given the deﬁnitions of discrete choice (Equation 1) and entropic choice (Equation 2), it can be assumed that the choice horizon n determines the locality for which choice is considered and that the discount factor γ a deﬁnes whether the altruistic agent gives higher importance to the short-term or long-term choice of the leader agent.

Original: This is in line with the observed results for the ﬁrst category (Opens door).
Revised: This is in line with the observed results for the ﬁrst category (Opens door).

Original: It can be assumed that, for short horizons n , the altruistic agent does not open the door, as it does not estimate that this would lead to an increase in the leader agent’s choice.
Revised: It can be assumed that, for short horizons n , the altruistic agent does not open the door, as it does not estimate that this would lead to an increase in the leader agent’s choice.

Original: A similar argumentation follows for low discount factors γ a .
Revised: A similar argumentation follows for low discount factors γ a .

Original: The bottom-row category evaluates whether the altruistic agent does not block the hallway that leads up to the leader agent’s target apple in the top right environment cell.
Revised: The bottom-row category evaluates whether the altruistic agent does not block the hallway that leads up to the leader agent’s target apple in the top right environment cell.

Original: This category demonstrates a possible failure case of the proposed approach of maximizing another agent’s choice.
Revised: This category demonstrates a possible failure case of the proposed approach of maximizing another agent’s choice.

Original: For short horizons n and high discount factors γ a , the altruistic agent actively blocks the entry to the lowentropy hallway towards the top right cell – by constantly occupying cell (2, 6) – to prohibit the leader agent from entering this region of low estimated choice.
Revised: For short horizons n and high discount factors γ a , the altruistic agent actively blocks the entry to the lowentropy hallway towards the top right cell – by constantly occupying cell (2, 6) – to prohibit the leader agent from entering this region of low estimated choice.

Original: This failure case can be prevented by an appropriate selection of the hyperparameters – horizon n and discount factor γ a .
Revised: This failure case can be prevented by an appropriate selection of the hyperparameters – horizon n and discount factor γ a .

Original: It is related to the selection of the temperature hyperparameter in maximum entropy single-agent RL (Haarnoja and Abbeel, 2018); if chosen incorrectly, the agent does not foster environment rewards in lowentropy regions.
Revised: It is related to the selection of the temperature hyperparameter in maximum entropy single-agent RL (Haarnoja and Abbeel, 2018); if chosen incorrectly, the agent does not foster environment rewards in lowentropy regions.

Original: A possible solution to this problem would be to deﬁne a constrained optimization problem, as shown by Haarnoja and Abbeel (2018).
Revised: A possible solution to this problem would be to deﬁne a constrained optimization problem, as shown by Haarnoja and Abbeel (2018).

Original: B. A BLATION STUDY ON JOINT LEARNING
Revised: B. A BLATION STUDY ON JOINT LEARNING

Original: Training.
Revised: Training.

Original: To investigate the effects of joint learning of the leader agent’s and the altruistic agent’s policy, we adapted the training process described in section 4.1 for the Gridworld experiments as following.
Revised: To investigate the effects of joint learning of the leader agent’s and the altruistic agent’s policy, we adapted the training process described in section 4.1 for the Gridworld experiments as following.

Original: Instead of ﬁrst learning the policy of the leader agent while the altruistic agent takes random actions, we initialized both policies from scratch and trained both agents simultaneously with the parameters given in Table 4.
Revised: Instead of ﬁrst learning the policy of the leader agent while the altruistic agent takes random actions, we initialized both policies from scratch and trained both agents simultaneously with the parameters given in Table 4.

Original: Results.
Revised: Results.

Original: We evaluated the outcome for the same scenarios, i.e the scenarios described in section 4.1.
Revised: We evaluated the outcome for the same scenarios, i.e the scenarios described in section 4.1.

Original: We found that the results for the individual test cases were equivalent to those achieved when training the leader and the altruistic agent subsequently, i.e. the results are equivalent to those displayed in
Revised: We found that the results for the individual test cases were equivalent to those achieved when training the leader and the altruistic agent subsequently, i.e. the results are equivalent to those displayed in

Original: Table 3.
Revised: Table 3.

Original: C L EVEL B ASED F ORAGING EXPERIMENTS C. T RAINING PROCEDURE C.1. S ETUP
Revised: C L EVEL B ASED F ORAGING EXPERIMENTS C. T RAINING PROCEDURE C.1. S ETUP

Original: We adopted the Level Based Foraging 2 environment as given in Christianos et al.
Revised: We adopted the Level Based Foraging 2 environment as given in Christianos et al.

Original: We only focus on two-agent scenarios and only consider the subset of possible environments that require full cooperation among agents, i.e. those where food can only be foraged by two agents cooperatively.
Revised: We only focus on two-agent scenarios and only consider the subset of possible environments that require full cooperation among agents, i.e. those where food can only be foraged by two agents cooperatively.

Original: We therefore only consider environments where both agents are at level one, and all present food is at level two.
Revised: We therefore only consider environments where both agents are at level one, and all present food is at level two.

Original: In the original implementation, both agents have to simultaneously select the eat action while docking at different sides of a food object to forage the object and receive the reward.
Revised: In the original implementation, both agents have to simultaneously select the eat action while docking at different sides of a food object to forage the object and receive the reward.

Original: To reduce training time, we simplify this setup by reducing the action space to up, down, left, right, stay , i.e. we remove the eat action and enable agents to forage food by being simultaneously at different sides of a food object, with no further action required.
Revised: To reduce training time, we simplify this setup by reducing the action space to up, down, left, right, stay , i.e. we remove the eat action and enable agents to forage food by being simultaneously at different sides of a food object, with no further action required.

Original: C.1. P RETRAINING
Revised: C.1. P RETRAINING

Original: To obtain a pretrained leader agent, we ﬁrst train two agents in the environment that are equally rewarded for foraging food.
Revised: To obtain a pretrained leader agent, we ﬁrst train two agents in the environment that are equally rewarded for foraging food.

Original: This setup corresponds to shared-reward cooperative MARL (Tan, 1993).
Revised: This setup corresponds to shared-reward cooperative MARL (Tan, 1993).

Original: Both agents are trained using Deep Q Learning (DQL, (Van Hasselt et al., 2015)), using a fully connected neural network with two hidden layers and ﬁve output values, resembling the Q values of the ﬁve possible actions.
Revised: Both agents are trained using Deep Q Learning (DQL, (Van Hasselt et al., 2015)), using a fully connected neural network with two hidden layers and ﬁve output values, resembling the Q values of the ﬁve possible actions.

Original: The exact training parameters are listed in Table 4.
Revised: The exact training parameters are listed in Table 4.

Original: We then take either one of the two agents and set it as the pretrained leader agent for the subsequent evaluation of the altruistic agent.
Revised: We then take either one of the two agents and set it as the pretrained leader agent for the subsequent evaluation of the altruistic agent.

Original: C.1. T RAINING OF ADDITIONAL AGENTS
Revised: C.1. T RAINING OF ADDITIONAL AGENTS

Original: We then insert an additional agent into the environment that shall act altruistically towards the leader agent.
Revised: We then insert an additional agent into the environment that shall act altruistically towards the leader agent.

Original: This additional agent is trained in the same fashion and with the same parameters as the previously trained leader agents.
Revised: This additional agent is trained in the same fashion and with the same parameters as the previously trained leader agents.

Original: Only its reward signal is different, as laid out in the next section.
Revised: Only its reward signal is different, as laid out in the next section.

Original: C.1. R EWARD COMPUTATION FOR ADDITIONAL AGENTS
Revised: C.1. R EWARD COMPUTATION FOR ADDITIONAL AGENTS

Original: We compare four different approaches for how the reward of the additional agent is deﬁned, respectively how it behaves.
Revised: We compare four different approaches for how the reward of the additional agent is deﬁned, respectively how it behaves.

Original: Random:
Revised: Random:

Original: The agent takes random actions.
Revised: The agent takes random actions.

Original: Supervised: The agent receives the same reward as the leader agent, i.e. a shared reward as in cooperative MARL.
Revised: Supervised: The agent receives the same reward as the leader agent, i.e. a shared reward as in cooperative MARL.

Original: Ours: semitable lb-foraging
Revised: Ours: semitable lb-foraging

Original: The reward of the additional agent is deﬁned as the immediate choice of the leader agent, as detailed in equation 3.
Revised: The reward of the additional agent is deﬁned as the immediate choice of the leader agent, as detailed in equation 3.

Original: We compute the leader agent’s policy entropy by computing the entropy of the softmax of the leader agent’s Q values in the given state.
Revised: We compute the leader agent’s policy entropy by computing the entropy of the softmax of the leader agent’s Q values in the given state.

Original: We further consider an unsupervised baseline, as detailed in the next paragraph.
Revised: We further consider an unsupervised baseline, as detailed in the next paragraph.

Original: Unsupervised baseline (MaxEnt).
Revised: Unsupervised baseline (MaxEnt).

Original: As an unsupervised baseline, we implemented the MEPOL approach of Mutti et al.
Revised: As an unsupervised baseline, we implemented the MEPOL approach of Mutti et al.

Original: Their task-agnostic unsupervised exploration approach maximizes the entropy over the state distribution of trajectory rollouts.
Revised: Their task-agnostic unsupervised exploration approach maximizes the entropy over the state distribution of trajectory rollouts.

Original: For this baseline, the additional agent is trained with the implementation given by the authors 3 , which itself builds on TRPO (Schulman et al., 2015).
Revised: For this baseline, the additional agent is trained with the implementation given by the authors 3 , which itself builds on TRPO (Schulman et al., 2015).

Original: We leave all parameters unchanged but evaluate different learning rates; lr ∈ { 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 } .
Revised: We leave all parameters unchanged but evaluate different learning rates; lr ∈ { 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 } .

Original: Best results were achieved for a learning rate of 1 e − 5 , which was hence picked as the relevant baseline.
Revised: Best results were achieved for a learning rate of 1 e − 5 , which was hence picked as the relevant baseline.

Original: C. P ERFORMANCE EVALUATION
Revised: C. P ERFORMANCE EVALUATION

Original: Each experiment was run for 5 different random seeds and mean and standard deviation are reported.
Revised: Each experiment was run for 5 different random seeds and mean and standard deviation are reported.

Original: Training progress is shown in Figure 4.
Revised: Training progress is shown in Figure 4.

Original: Evaluations are computed every 10000 environment steps for 200 episodes, with the exploration set to zero.
Revised: Evaluations are computed every 10000 environment steps for 200 episodes, with the exploration set to zero.

Original: Training time was about 14 hours for each run.
Revised: Training time was about 14 hours for each run.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: D T AG EXPERIMENTS D. T RAINING PROCEDURE D.1. P RETRAINING
Revised: D T AG EXPERIMENTS D. T RAINING PROCEDURE D.1. P RETRAINING

Original: We use the Simple Tag (Tag) implementation by Terry et al.
Revised: We use the Simple Tag (Tag) implementation by Terry et al.

Original: (2020) 4 which is unchanged as compared to the original implementation of Mordatch and Abbeel (2018) 5 , only ﬁxing minor errors.
Revised: (2020) 4 which is unchanged as compared to the original implementation of Mordatch and Abbeel (2018) 5 , only ﬁxing minor errors.

Original: We ﬁrst adopt the original conﬁguration and pretrain three adversaries and one good agent (leader agent) using the parameters listed in Table 4.
Revised: We ﬁrst adopt the original conﬁguration and pretrain three adversaries and one good agent (leader agent) using the parameters listed in Table 4.

Original: We use MADDPG (Lowe et al., 2017) 6 to train adversary agents, and modify the framework as follows.
Revised: We use MADDPG (Lowe et al., 2017) 6 to train adversary agents, and modify the framework as follows.

Original: The last layer of each agent’s actor-network outputs one value for each of the environment’s ﬁve possible actions, over which the softmax is computed.
Revised: The last layer of each agent’s actor-network outputs one value for each of the environment’s ﬁve possible actions, over which the softmax is computed.

Original: We then sample the agent’s action from the output softmax vector, which corresponds to the probabilities with which the agent takes a speciﬁc action in a given state.
Revised: We then sample the agent’s action from the output softmax vector, which corresponds to the probabilities with which the agent takes a speciﬁc action in a given state.

Original: We train the leader agent with DDPG (Lillicrap et al., 2016), 6 where we equally modify the output layer.
Revised: We train the leader agent with DDPG (Lillicrap et al., 2016), 6 where we equally modify the output layer.

Original: Each actor and critic network is implemented as a fully-connected neural network with two hidden layers, with layer sizes as given in Table 4.
Revised: Each actor and critic network is implemented as a fully-connected neural network with two hidden layers, with layer sizes as given in Table 4.

Original: To make the environment more challenging for the leader agent, we decrease its maximum speed and acceleration to 70% of the original value.
Revised: To make the environment more challenging for the leader agent, we decrease its maximum speed and acceleration to 70% of the original value.

Original: We next insert three additional agents into the environment whose observations include all agents and objects.
Revised: We next insert three additional agents into the environment whose observations include all agents and objects.

Original: These additional agents are not observed by adversary agents or the leader agent.
Revised: These additional agents are not observed by adversary agents or the leader agent.

Original: The additional agents are of the same size as the adversary agents, and their acceleration and maximum velocity are equal to that of the leader agent.
Revised: The additional agents are of the same size as the adversary agents, and their acceleration and maximum velocity are equal to that of the leader agent.

Original: To speed up training, we made the following changes to the environment, which are applied to our approach as well as to all baselines.
Revised: To speed up training, we made the following changes to the environment, which are applied to our approach as well as to all baselines.

Original: First, we spawn the three additional agents in the vicinity of the leader agent, which itself is spawned at a random position.
Revised: First, we spawn the three additional agents in the vicinity of the leader agent, which itself is spawned at a random position.

Original: Furthermore, we randomly pick two out of the three adversary agents and decrease their maximum acceleration and maximum speed by 50%.
Revised: Furthermore, we randomly pick two out of the three adversary agents and decrease their maximum acceleration and maximum speed by 50%.

Original: We made these changes to be able to observe substantial differences between the different approaches after a training time of less than 24h.
Revised: We made these changes to be able to observe substantial differences between the different approaches after a training time of less than 24h.

Original: D.1. T RAINING OF ADDITIONAL AGENTS
Revised: D.1. T RAINING OF ADDITIONAL AGENTS

Original: We train these three additionally inserted agents with the previously described modiﬁed version of
Revised: We train these three additionally inserted agents with the previously described modiﬁed version of

Original: MADDPG.
Revised: MADDPG.

Original: The reward for each agent is deﬁned either according to our developed approach, or any of the given baselines, as detailed in the next section.
Revised: The reward for each agent is deﬁned either according to our developed approach, or any of the given baselines, as detailed in the next section.

Original: [Table]
Revised: [Table]

Original: D.1. R EWARD COMPUTATION FOR ADDITIONAL AGENTS FOR DIFFERENT BASELINES
Revised: D.1. R EWARD COMPUTATION FOR ADDITIONAL AGENTS FOR DIFFERENT BASELINES

Original: We consider the following implementations for the reward computation of the additional agents, respectively different environment conﬁgurations.
Revised: We consider the following implementations for the reward computation of the additional agents, respectively different environment conﬁgurations.

Original: None: For this scenario, the additional agents are removed from the environment.
Revised: None: For this scenario, the additional agents are removed from the environment.

Original: The remaining approaches purely differ in the way that the reward of the additional agents is computed.
Revised: The remaining approaches purely differ in the way that the reward of the additional agents is computed.

Original: No other changes are made.
Revised: No other changes are made.

Original: Random:
Revised: Random:

Original: The additional agents take random actions.
Revised: The additional agents take random actions.

Original: Cage:
Revised: Cage:

Original: The additional agents receive a negative reward for violating the environment boundaries, which is equal to the negative reward that the leader agent receives for itself violating the environment boundaries (part of the original Tag implementation).
Revised: The additional agents receive a negative reward for violating the environment boundaries, which is equal to the negative reward that the leader agent receives for itself violating the environment boundaries (part of the original Tag implementation).

Original: Supervised: The additional agents receive the same reward as the leader agent.
Revised: Supervised: The additional agents receive the same reward as the leader agent.

Original: That is, they receive a reward of -10 if the leader agent is caught by the adversaries and a small negative reward if the leader agent violates
Revised: That is, they receive a reward of -10 if the leader agent is caught by the adversaries and a small negative reward if the leader agent violates

Original: the environment boundaries.
Revised: the environment boundaries.

Original: Supervised + Cage: The additional agents receive the same reward as the leader agent, and an additional small negative reward if they themselves violate the environment boundaries.
Revised: Supervised + Cage: The additional agents receive the same reward as the leader agent, and an additional small negative reward if they themselves violate the environment boundaries.

Original: Ours: The reward of the additional agents is deﬁned as the immediate choice of the leader agent, as detailed in eq. 3.
Revised: Ours: The reward of the additional agents is deﬁned as the immediate choice of the leader agent, as detailed in eq. 3.

Original: To reduce the variance in the estimate of the leader agent’s immediate choice, we implement an ensemble of ﬁve pretrained actor-networks for the leader agent, evaluate the policy entropy of each network, and take the median of the achieved values as the reward for the altruistic agents.
Revised: To reduce the variance in the estimate of the leader agent’s immediate choice, we implement an ensemble of ﬁve pretrained actor-networks for the leader agent, evaluate the policy entropy of each network, and take the median of the achieved values as the reward for the altruistic agents.

Original: Furthermore, the additional agents receive a small negative reward for themselves violating the environment boundaries.
Revised: Furthermore, the additional agents receive a small negative reward for themselves violating the environment boundaries.

Original: D. P ERFORMANCE EVALUATION
Revised: D. P ERFORMANCE EVALUATION

Original: We train Cage, Supervised, Supervised + Cage and Ours for ﬁve different random seeds with parameters as detailed in Table 4.
Revised: We train Cage, Supervised, Supervised + Cage and Ours for ﬁve different random seeds with parameters as detailed in Table 4.

Original: We then compute the results listed in Table 2 by freezing all weights across all networks, setting the exploration noise to zero and computing the average and standard deviation over 500 rollout episodes.
Revised: We then compute the results listed in Table 2 by freezing all weights across all networks, setting the exploration noise to zero and computing the average and standard deviation over 500 rollout episodes.

Original: [Table]
Revised: [Table]

Original: E R ESOURCE E NVIRONMENT E.0. M OTIVATION AND O VERVIEW
Revised: E R ESOURCE E NVIRONMENT E.0. M OTIVATION AND O VERVIEW

Original: This environment is a special case of the general resource-based MDP proposed by Benson-Tilsen and Soares (2016), which they used to show that intelligent agents pursue instrumentally useful subgoals.
Revised: This environment is a special case of the general resource-based MDP proposed by Benson-Tilsen and Soares (2016), which they used to show that intelligent agents pursue instrumentally useful subgoals.

Original: The motivation behind the choice for this environment is to evaluate our proposal in non-spatial and non-navigation environments.
Revised: The motivation behind the choice for this environment is to evaluate our proposal in non-spatial and non-navigation environments.

Original: In the environment, there are 3 resource types, which two “consumer” agents may consume as an action.
Revised: In the environment, there are 3 resource types, which two “consumer” agents may consume as an action.

Original: Each consumer has different preferences (reward function), and so will only consume 2 of the resource types.
Revised: Each consumer has different preferences (reward function), and so will only consume 2 of the resource types.

Original: A third, altruistic agent, receives one resource unit of each type to distribute among the consumers, and its goal is to satisfy the preferences of the consumers without knowing their reward function.
Revised: A third, altruistic agent, receives one resource unit of each type to distribute among the consumers, and its goal is to satisfy the preferences of the consumers without knowing their reward function.

Original: We deﬁne its performance as the average number of times that the consumers fail to consume their preferred resource (so lower is better).
Revised: We deﬁne its performance as the average number of times that the consumers fail to consume their preferred resource (so lower is better).

Original: We compare our method to a supervised agent that is explicitly trained with the consumers’ reward function, as well as to an agent that assigns the resources randomly.
Revised: We compare our method to a supervised agent that is explicitly trained with the consumers’ reward function, as well as to an agent that assigns the resources randomly.

Original: E.0. E NVIRONMENT D ESCRIPTION
Revised: E.0. E NVIRONMENT D ESCRIPTION

Original: The environment is expressed as a Markov Game (see section 3).
Revised: The environment is expressed as a Markov Game (see section 3).

Original: The Markov game is composed of two human-inspired consumers with subscript C 1 , C 2 and an altruistic agent with subscript A .
Revised: The Markov game is composed of two human-inspired consumers with subscript C 1 , C 2 and an altruistic agent with subscript A .

Original: Three types of resources exist, R X , R Y and R Z .
Revised: Three types of resources exist, R X , R Y and R Z .

Original: The environment state s is given by the number of resources of each type available to each of the consumers.
Revised: The environment state s is given by the number of resources of each type available to each of the consumers.

Original: For example, s =
Revised: For example, s =

Original: [(1 , 0 , 1) , (0 , 1 , 0)] means that agent C 1 has one resource each of type X and Y available, while agent C 2 only has one resource of type Z available.
Revised: [(1 , 0 , 1) , (0 , 1 , 0)] means that agent C 1 has one resource each of type X and Y available, while agent C 2 only has one resource of type Z available.

Original: At the beginning of each time step, the altruistic agent is provided with one resource per category, i.e. R X , R Y and R Z .
Revised: At the beginning of each time step, the altruistic agent is provided with one resource per category, i.e. R X , R Y and R Z .

Original: The altruistic agent can assign each resource individually to any agent or discard the resource.
Revised: The altruistic agent can assign each resource individually to any agent or discard the resource.

Original: The altruistic agent’s action space is hence deﬁned by one sub-action per resource, i.e. a A = ( a XA , a YA , a ZA ) .
Revised: The altruistic agent’s action space is hence deﬁned by one sub-action per resource, i.e. a A = ( a XA , a YA , a ZA ) .

Original: Each sub-action assigns the resource either to one of the consumers or discards it.
Revised: Each sub-action assigns the resource either to one of the consumers or discards it.

Original: The resources are then distributed according to the action taken by the altruistic agent and the environment state is updated.
Revised: The resources are then distributed according to the action taken by the altruistic agent and the environment state is updated.

Original: Resources cannot be stacked, which means that each agent can only have one resource per category available at a time.
Revised: Resources cannot be stacked, which means that each agent can only have one resource per category available at a time.

Original: Next, the consumers attempt to consume one resource each, according to their preference.
Revised: Next, the consumers attempt to consume one resource each, according to their preference.

Original: Agent C 1 dislikes resource R Z , hence it chooses R X or R Y with equal probability. Agent C 1 dislikes resource R X , hence it chooses R Y or R Z with equal probability.
Revised: Agent C 1 dislikes resource R Z , hence it chooses R X or R Y with equal probability. Agent C 2 dislikes resource R X , hence it chooses R Y or R Z with equal probability.

Original: The actions of agents C 1 and C 2 are sampled accordingly and the environment state is updated.
Revised: The actions of agents C 1 and C 2 are sampled accordingly and the environment state is updated.

Original: For each round, we record how many agents failed to consume a resource that was not available.
Revised: For each round, we record how many agents failed to consume a resource that was not available.

Original: E. T RAINING
Revised: E. T RAINING

Original: The altruistic agent is trained with Q-Learning (Watkins and Dayan, 1992) to maximize the discounted future choice of the consumers (see eq. 5).
Revised: The altruistic agent is trained with Q-Learning (Watkins and Dayan, 1992) to maximize the discounted future choice of the consumers (see eq. 5).

Original: For that, it uses one of the three proposed objectives, namely IC (eq. 3), EC (eq. 2) or DC (eq. 1), which it estimates as detailed in appendix
Revised: For that, it uses one of the three proposed objectives, namely IC (eq. 3), EC (eq. 2) or DC (eq. 1), which it estimates as detailed in appendix

Original: A.1.
Revised: A.1.

Original: The exact hyper-parameters are given in Table 4.
Revised: The exact hyper-parameters are given in Table 4.

Original: We compare the performance of the altruistic agent that maximizes the choice of the consumers to that of a supervised agent.
Revised: We compare the performance of the altruistic agent that maximizes the choice of the consumers to that of a supervised agent.

Original: The reward of the supervised agent is the negative of the number of consumers that attempted to consume a resource, in that time step, and failed.
Revised: The reward of the supervised agent is the negative of the number of consumers that attempted to consume a resource, in that time step, and failed.

Original: Further, we compare to a random-policy baseline that distributes the resources randomly but does not discard any resources.
Revised: Further, we compare to a random-policy baseline that distributes the resources randomly but does not discard any resources.

Original: E. R ESULTS
Revised: E. R ESULTS

Original: Table 5 shows that the results achieved by the altruistic agent trained with choice are equivalent to those achieved by the supervised agent.
Revised: Table 5 shows that the results achieved by the altruistic agent trained with choice are equivalent to those achieved by the supervised agent.

Original: Furthermore, they are signiﬁcantly better than those achieved by an agent with a random policy.
Revised: Furthermore, they are signiﬁcantly better than those achieved by an agent with a random policy.

Original: F V IDEOS OF BEHAVIOUR OF ALTRUISTIC AGENT
Revised: F V IDEOS OF BEHAVIOUR OF ALTRUISTIC AGENT

Original: We provide videos for the most relevant outcomes of our experiments in the supplementary material.
Revised: We provide videos for the most relevant outcomes of our experiments in the supplementary material.

Original: F. V IDEOS FOR RESULTS OF G RIDWORLD EXPERIMENTS (S ECTION 4.1) F.1. D OOR S CENARIO IN F IG . 1 TOP CENTER
Revised: F. V IDEOS FOR RESULTS OF G RIDWORLD EXPERIMENTS (S ECTION 4.1) F.1. D OOR S CENARIO IN F IG . 1 TOP CENTER

Original: Altruistic agent opens door for leader agent :
Revised: Altruistic agent opens door for leader agent :

Original: It
Revised: It

Original: can
Revised: can

Original: be
Revised: be

Original: observed
Revised: observed

Original: that the altruistic agent has learned to operate the door switch
Revised: that the altruistic agent has learned to operate the door switch

Original: to
Revised: to

Original: enable the leader agent to pass through the door
Revised: enable the leader agent to pass through the door

Original: and reach
Revised: and reach

Original: its target
Revised: its target

Original: on the other side
Revised: on the other side

Original: 02 Altruistic agent
Revised: 02 Altruistic agent

Original: does
Revised: does

Original: not
Revised: not

Original: open door
Revised: open door

Original: for
Revised: for

Original: leader agent
Revised: leader agent

Original: failure case
Revised: failure case

Original: ) : It can be observed that for an unfavourable choice of
Revised: ) : It can be observed that for an unfavourable choice of

Original: hyperparameters
Revised: hyperparameters

Original: the altruistic agent
Revised: the altruistic agent

Original: does
Revised: does

Original: not open the door
Revised: not open the door

Original: F.1. D EAD E ND SCENARIO IN F IG . 1 TOP RIGHT
Revised: F.1. D EAD E ND SCENARIO IN F IG . 1 TOP RIGHT

Original: 03 Altruistic agent gives way to leader agent : It can be observed that the altruistic agent does not get into the way of the leader agent, which is hence able to reach its target in the top right cell.
Revised: 03 Altruistic agent gives way to leader agent : It can be observed that the altruistic agent does not get into the way of the leader agent, which is hence able to reach its target in the top right cell.

Original: 04 Altruistic agent blocks path of leader agent (failure case) : It can be observed that for an unfavourable choice of hyperparameters, the altruistic agent blocks the entry to the hallway towards the right side of the environment such that the leader agent cannot reach its target at the top right cell.
Revised: 04 Altruistic agent blocks path of leader agent (failure case) : It can be observed that for an unfavourable choice of hyperparameters, the altruistic agent blocks the entry to the hallway towards the right side of the environment such that the leader agent cannot reach its target at the top right cell.

Original: This happens as the altruistic agent forcefully maximizes the estimated choice of the leader agent by hindering it from entering the hallway, which is a region of fewer estimated choice.
Revised: This happens as the altruistic agent forcefully maximizes the estimated choice of the leader agent by hindering it from entering the hallway, which is a region of fewer estimated choice.

Original: F. V IDEO FOR RESULTS OF L EVEL B ASED F ORAGING (S ECTION 4.2)
Revised: F. V IDEO FOR RESULTS OF L EVEL B ASED F ORAGING (S ECTION 4.2)

Original: 05 Altruistic agent enables leader to forage apples : It can be observed how the altruistic agent (blue) learned to coordinate its movements with the leader agent (green), to enable the leader agent to forage apples.
Revised: 05 Altruistic agent enables leader to forage apples : It can be observed how the altruistic agent (blue) learned to coordinate its movements with the leader agent (green), to enable the leader agent to forage apples.

Original: It has learned this behaviour purely through optimizing for the leader agents choice and is itself not rewarded for foraging apples.
Revised: It has learned this behaviour purely through optimizing for the leader agents choice and is itself not rewarded for foraging apples.

Original: F. V IDEO FOR RESULTS OF T AG (S ECTION 4.3)
Revised: F. V IDEO FOR RESULTS OF T AG (S ECTION 4.3)

Original: 06 Altruistic agents protect leader from adversaries : It can be observed how the altruistic agents (blue colors) learned to coordinate their movements to protect the leader agent (green) from its adversaries.
Revised: 06 Altruistic agents protect leader from adversaries : It can be observed how the altruistic agents (blue colors) learned to coordinate their movements to protect the leader agent (green) from its adversaries.

Original: The adversaries (red colors) try to catch the leader, which in turn tries to ﬂee from them.
Revised: The adversaries (red colors) try to catch the leader, which in turn tries to ﬂee from them.

Original: The altruistic agents protect the leader by actively intercepting the paths of the adversaries.
Revised: The altruistic agents protect the leader by actively intercepting the paths of the adversaries.

Original: They have learned this behaviour purely through optimizing for the leader agents choice.
Revised: They have learned this behaviour purely through optimizing for the leader agents choice.