ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward
Abstract
Modern multi-agent reinforcement learning frameworks rely on centralized training and reward shaping to perform well.
However, centralized training and dense rewards are not readily available in the real world.
Current multi-agent algorithms struggle to learn in the alternative setup of decentralized training or sparse rewards.
To address these issues, we propose a self-supervised intrinsic reward ELIGNexpectation alignment - inspired by the self-organization principle in Zoology.
Similar to how animals collaborate in a decentralized manner with those in their vicinity, agents trained with expectation alignment learn behaviors that match their neighbors’ expectations.
This allows the agents to learn collaborative behaviors without any external reward or centralized training.
We demonstrate the efﬁcacy of our approach across 6 tasks in the multi-agent particle and the complex Google Research football environments, comparing ELIGN to sparse and curiosity-based intrinsic rewards.
When the number of agents increases, ELIGN scales well in all multi-agent tasks except for one where agents have different capabilities.
We show that agent coordination improves through expectation alignment because agents learn to divide tasks amongst themselves, break coordination symmetries, and confuse adversaries.
These results identify tasks where expectation alignment is a more useful strategy than curiosity-driven exploration for multi-agent coordination, enabling agents to do zero-shot coordination.
Introduction
Many real world AI applications can be formulated as multi-agent systems, including autonomous vehicles (Cao et al., 2012), resource management (Ying & Dayong, 2005), trafﬁc control (Sunehag et al., 2017), robot swarms (Swamy et al., 2020) and multi-player video games (Berner et al., 2019).
Agents must adapt their behaviors to each other in order to coordinate successfully in these systems.
However, adaptive coordination algorithms are challenging to develop because each agent is not privy to other agents’ intentions and their future behaviors (Foerster et al., 2017).
These challenges are more acute in decentralized training under partial observability than centralized training or full observability.
In the real world, agents act under partial observability and learn in a decentralized manner: they do not learn collaborative behaviors with a single centralized algorithm with a complete knowledge of the environment (Iqbal & Sha, 2019; Liu et al., 2020).
Unfortunately, the most successful multi-agent algorithms train agents with a centralized critic, assuming access to all agents’ observations and actions (Foerster et al., 2018;
Rashid et al., 2018; Sunehag et al., 2017;
Lowe et al., 2017).
The most successful multi-agent algorithms for decentralized training and partial observability assume task-speciﬁc reward shaping (Jain et al., 2020; Iqbal & Sha, 2019), which is expensive to generate.
These algorithms struggle to learn with sparse reward structure.
Consider a cooperative navigation task, where N agents aim to simultaneously occupy N goal locations.
A centralized algorithm with full observability is capable of optimally assigning the nearest
→ → goal location to each respective agent.
However, with partial observability, agents can see only a handful of goal locations and other agents.
They are unaware of others’ observations, actions, and intentions with decentralized training.
We observe that agents simultaneously occupy the same goal; they fail to collaborate because they do not predict which goal each agent is expected to occupy.
To overcome instances of miscoordination, decentralized algorithms have adapted single-agent curiositybased intrinsic rewards (Pathak et al., 2017; Stadie et al., 2015).
Multi-agent curiosity-based rewards incentivize agents to explore novel states (Iqbal & Sha, 2020).
Although curiosity helps agents discover new goal locations, it doesn’t solve the challenge of coordination, such as assigning goals to each agent.
Only a few attempts explore other forms of multi-agent intrinsic rewards (Iqbal & Sha, 2020; Böhmer et al., 2019; Schafer, 2019).
In this work, we propose ELIGN as a novel multi-agent self-supervised intrinsic reward, enabling decentralized training under partial observability.
Intuitively, expectation alignment encourages agents to elicit behaviors that decrease future uncertainty for their team: it encourages each agent to choose actions that match their teammates’ expectations.
Going back to the cooperative navigation task, expectation alignment encourages each agent to move to goals others expect it to occupy, like goals that are either closest to the agent or goals that other agents aren’t moving towards (Figure 1).
We take inspiration from the self-organization principle in Zoology (Couzin, 2007).
This principle hypothesizes that collective animal intelligence emerges because groups synchronize their behaviors using only their local environment; they do not rely on complete information about other agents and can coordinate successfully by predicting the dynamics of agents within their ﬁeld-of-view (Collett et al., 1998; Theraulaz & Bonabeau, 1995; Ben-Jacob et al., 1994; Buhl et al., 2006).
Similarly, expectation alignment as an intrinsic reward is calculated based on the agent’s local observations and its approximation of neighboring agents’ expectations.
It does not require a centralized controller nor full observability.
ELIGN is task-agnostic and we apply it to both collaborative and competitive multi-agent tasks.
We demonstrate the efﬁcacy of our approach in the multi-agent particle and Google Research football environments, two popular benchmarks for multi-agent reinforcement learning (Lowe et al., 2017;
Kurach et al., 2019).
We evaluate ELIGN under partial and full observability, with decentralized and centralized training, and in terms of scalability.
We observe that expectation alignment outperforms sparse and curiosity-based intrinsic rewards (Ndousse et al., 2021; Stadie et al., 2015; Iqbal & Sha, 2020), especially under partial observability with decentralized training.
We additionally test expectation alignment as a way to perform zero-shot coordination with new agent partners, and investigate why ELIGN improves coordination.
We show that agent coordination improves through expectation alignment because agents learn to divide tasks amongst themselves and break coordination symmetries (Hu et al., 2020).
Related Work
Our formulation of expectation alignment, a task-agnostic intrinsic reward for multi-agent training, draws inspiration from the self-organization principle in Zoology, which posits that synchronized group behavior is mediated by local behavioral rules (Couzin, 2007) and not by a centralized controller (Camazine et al., 2020).
Group cohesion emerges by predicting and adjusting one’s behavior to that of near neighbors (Buhl et al., 2006).
This principle underlies the coordination found in multi-cellular organisms (Camazine et al., 2020), the migration of wingless locusts (Collett et al., 1998), the collective swarms of bacteria (Ben-Jacob et al., 1994), the construction of bridge structures by ants (Theraulaz & Bonabeau, 1995), and some human navigation behaviors (Couzin, 2007).
Intrinsic motivation for single agents.
Although we draw inspiration from Zoology for formalizing expectation alignment as an intrinsic reward, there is a rich body of work on intrinsic rewards within the single-agent reinforcement learning community.
To incentivize exploration, even when non-optimal successful trajectories are uncovered ﬁrst, scholars have argued for the use of intrinsic motivation (Schmidhuber, 1991).
Single-agent intrinsic motivation has focused on exploring previously unencountered states (Pathak et al., 2017; Burda et al., 2018a), which works particularly well in discrete domains.
In continuous domains, identifying unseen states requires keeping track of an intractable number of visited states; instead, literature has recommended learning a forward dynamics model to predict future states and identify novel states using the uncertainty of this model (Achiam &
Sastry, 2017).
Other formulations encourage re-visiting states where the dynamics model’s prediction of future states errs (Stadie et al., 2015; Pathak et al., 2017).
Follow up papers have improved how uncertainty (Kim et al., 2020) and model errors (Burda et al., 2018b; Sekar et al., 2020) are calculated.
Intrinsic motivation for multiple agents.
Most multi-agent intrinsic rewards have been adapted from single-agent curiosity-based incentives ( ?
Böhmer et al., 2019; Schafer, 2019) and have primarily focused on cooperative tasks.
They propose intrinsic rewards to improve either coordination, collaboration, or deception: These rewards either maximize information conveyed by an agent’s actions ( ?
Chitnis et al., 2020; Wang et al., 2019), shape the inﬂuence of an agent (Jaques et al., 2019;
Foerster et al., 2017), incentivize agents to hide intentions (Strouse et al., 2018), build accurate models of other agents’ policies (Hernandez-Leal et al., 2019; Jaques et al., 2019), or break extrinsic rewards for better credit assignment (Du et al., 2019).
Several multi-agent intrinsic rewards (Hernandez-Leal et al., 2019; Jaques et al., 2019), including ours, rely on the ability to model others’ dynamics in a shared environment.
This ability is a key component to coordination, closely related to Theory of Mind (Tomasello et al., 2005).
Our work can be interpreted as using a Theory of Mind model of others’ behaviors to calculate an intrinsic motivation loss.
Unlike existing Theory of Mind methods that learn a model per collaborator (Roy et al., 2020), we learn a single dynamics model, allowing our method to scale as the number of agents increase.
Our proposal is related to model-based reinforcement learning (Jaderberg et al., 2016; Wang et al., 2020a); however, instead of learning a dynamics model for control, we learn a dynamics model as a source of reward.
Our work is closely related to a recently proposed auxiliary loss on predicting an agent’s own future states (Ndousse et al., 2021).
However, there are three key differences.
First, their work predicts ego-agent observations, whereas our work additionally predicts future observations from the other agents’ point of view.
Second, their loss optimizes state embeddings while ours optimizes agents’ policies.
Third, their work focuses on cooperative tasks whereas ours applies to both cooperative and competitive domains.
Multi-agent reinforcement learning algorithms.
Today, the predominant deep multi-agent framework uses actor-critic methods with a centralized critic and decentralized execution (Lowe et al., 2017;
Foerster et al., 2018; Iqbal & Sha, 2019; Liu et al., 2020; Rashid et al., 2018).
This framework allows a critic to access the observations and actions of all agents to ease training.
However, there are several situations where centralized training may not be desirable or possible.
Examples include low bandwidth communication restrictions or human-robot tasks where observations cannot be easily shared between agents (Ying & Dayong, 2005; Cao et al., 2012; Huang et al., 2015).
Decentralized training is therefore the most practical training paradigm but it suffers from unstable training: the environment is nonstationary from a single-agent’s perspective (Lowe et al., 2017).
Our work uses a decentralized training framework and tackles the nonstationarity challenge with an intrinsic reward designed to improve an agent’s ability to model others.
We also apply expectation alignment to centralized training and observe that it still aids cooperative and some competitive tasks.
Background
We formulate our setting as a partially observable Markov game ( S , O , A , T , r ex , N ) (Littman, 1994).
A Markov game for N agents is deﬁned by a state space
S describing the possible conﬁgurations of the environment.
The observation space for agents is
O =
( O 1 , . . .
, O N )
and the action space is
A =
( A 1 , . . .
, A N ) .
Each agent i observes o i ∈
O i , a private partial view of the state, and performs actions
a i ∈
A i .
Using the observation, each agent uses a stochastic policy π θ i : O i × A i
→ [0 , 1] , where θ i parameterizes the policy.
The environment changes according to the state transition function which transitions to the next state using the current state and each agent’s actions, T : S ×
A → S .
The team of agents obtains a shared extrinsic reward as a function of the environment state, r ex :
S ×
A → R .
The team’s goal is to maximize the total expected return: R = (cid:80)
Tt =
γ t r t ex where≤ γ ≤ 1 is the discount factor, t is the time step, and T is the time horizon.
The environment may also contain adversarial agents who have their own reward structure.
Expectation Alignment
To understand expectation alignment intuitively, let’s revisit the cooperative navigation task, where N agents are rewarded for simultaneously occupying as many goal locations as possible.
In Figure 1, agent i has a dynamics model trained on its past experiences.
It predicts how future states will evolve from the point of view of agent j , who is within i ’s view.
In this example, j will expect i to move towards the goal since i is closer to it.
ELIGN encourages i to pursue the action that j expects (Figure 1(a)).
In turn, j can now assume that the observed goal location will eventually be occupied by i and should therefore explore to ﬁnd another goal.
By aligning shared expectations, agent behaviors become more predictable.
Conversely, when neighbors behave opposite to an agent’s predictions, the agent can infer about the environment outside of its own receptive ﬁeld (Krause et al., 2002).
For example, in Figure 1 (b), if agent j observes i running away from a goal, this surprising behavior might indicate the existence of an adversary outside j ’s receptive ﬁeld.
Our training algorithm consists of three interwoven phases of learning a dynamics model, calculating the ELIGN reward, and optimizing the agent’s policy (Algorithm 1).
4.1 Training the dynamics model
Similar to prior work (Wang et al., 2018; Kidambi et al., 2020), each agent i learns a dynamics model f
θ i to predict the next observation ˆ o (cid:48)
i given its current observation and action o i , a i , i.e, ˆ o (cid:48)
i =
f θ i
( o i , a i ) .
We use a three-layer Multi-Layer Perceptron with ReLU non-linearities as the dynamics model.
We minimize the mean squared error between its prediction and ground truth next observation o (cid:48) i .
4.2 Calculating intrinsic reward
The intrinsic reward captures how well agent i aligns to its neighbors’ (e.g., agent j ’s) expectations on its next state.
Calculating this reward requires j to accurately predict i ’s behavior, simulating a Theory of Mind (Tomasello et al., 2005).
As suggested by the self-organization principle, i must learn to align to j ’s predictions.
Ideally, the ELIGN intrinsic reward is calculated as:
[Equation]
where N ( i ) is the set of neighbors within i ’s receptive ﬁeld, including i itself.
The ELIGN reward is high when the average L 2 loss is small, i.e, when i ’s actual next observation is close to agent j ’s predicted observation of i for all j in its neighbors.
In that case, i has chosen an action that aligns with j ’s expectations of how i should act.
In a decentralized training setup, however, i doesn’t have access to j ’s dynamics model f θ j , so i approximates j ’s dynamic model with a proxy: its own dynamics model f
θ i and the knowledge of agent j ’s observation radius.
Such an approximation is ecologically valid since we often approximate others’ behaviors using a second-order cognitive Theory of Mind (Morin, 2006).
Additionally, i doesn’t have access to j ’s entire observation; so, we restrict the future prediction from j ’s point of view by using the portion of j ’s observation i can see: o i ∩ j = o i (cid:12) o j .
Agent i ’s decentralized intrinsic reward then becomes:
[Equation]
We found that the approximation of f θ j using f θ i works well empirically.
Dynamics model losses for all agents quickly decrease within 5-10 training epochs.
we validate its applicability in smallscale heterogeneous multi-agent tasks where agents have variable capabilities, although we ﬁnd the methods perform similarly when more heterogeneous agents are added.
4.3 Policy learning
1: Initialize do 4: do 5:
Populate policies
) 6: end for 7: // T MODEL 8: for agent i = N do 9: Sample transitions: { ( o i , a i , r ex , o i ) } ∼ D i 10:
Predict ˆ o (cid:48) i
=
f θ i
( o i , a i ) 11:
Update dynamics θ i using o (cid:48) i .
12: end for 13: // C ALCULATE
REWARD 14: for agent i = 1 . . .
N do 15:
Sample
B transitions: { ( o i , a i , r ex , o (cid:48) i ) } ∼ D i 16:
Compute intrinsic rewards
, a i ) 17:
Add { (
, a ex + βr in , o (cid:48) i ) } to D (cid:48)
i 18: end for 19: //
P OLICY LEARNING 20:
Update all θ i s using transitions from D (cid:48) 21: end while
Once the ELIGN rewards are calculated, the total rewards at each step for each agent i is: r i = r ex + βr in ( o i , a i ) where r ex is the extrinsic reward provided by the environment and β is a hyperparameter for weighing the intrinsic reward in the agent’s overall reward calculation.
In practice, we set β to be 1 |O i | where |O i | is the observation dimension; we ﬁnd this scale generalizes well across tasks.
Since our contribution is agnostic to any particular multi-agent training algorithm, the team of agents can now be trained using any multi-agent training algorithm to maximize returns R =
(cid:80) Tt =
γ t r .
Both centralized and decentralized training algorithms can make use of these rewards.
We primarily use the multi-agent decentralized variant of the soft-actor critic algorithm in our experiments (Haarnoja et al., 2018; Iqbal & Sha, 2019).
Compared to centralized joint-action training, whose action space grows exponentially in N agents, our decentralized method has linear space complexity.
Further, decentralized training can parallelize training time to be less than linear with respect to N .
Although we present results with one centralized training framework, studying the impact of expectation alignment with all the centralized critic frameworks is out of scope for this paper.
4.4 Extending expectation alignment to competitive tasks
We extend the ELIGN formulation to competitive tasks where a team of agents compete against adversaries.
In this case, agents are encouraged to misalign with their adversaries’ expectations, i.e, agents are incentivized to be unpredictable to their adversaries within its receptive ﬁeld ( N adv ( i ) ): r in = |N adv ( i ) | (cid:88)
[Table]
Experiments
Our experiments explore the utility of using expectation alignment as an intrinsic reward compared to sparse and curiosity-based intrinsic rewards.
We primarily focus on decentralized training under partial observability.
However, we also demonstrate that ELIGN can easily augment centralized methods and assist in fully observable tasks.
We vary the number of agents in the multi-agent particle tasks to test scalability.
We end by investigating how and why ELIGN improves coordination by designing three evaluation conditions. First, does expectation alignment improve coordination by [Table] [Figure]
occupancy/collision count per step of decentralized algorithms under partially observable multi-agent particle tasks.
On these metrics, ELIGN self and ELIGN adv perform the best on cooperative and competitive tasks respectively. breaking symmetries (Hu et al., 2020; Wang et al., 2020b)?
Second, does ELIGN enable zero-shot generalization to new partners?
Lastly, is the dynamics model critical in aligning agent behaviors?
5.1 Environments
We evaluate ELIGN across both cooperative and competitive tasks in the multi-agent particle environment (Mordatch & Abbeel, 2017; Lowe et al., 2017) and the Google Research football environment (Kurach et al., 2019).
State and action space
The multi-agent particle environment is a two-dimensional world.
The
Google Research football environment is a three-dimensional world.
Both environments have continuous state spaces and discrete action spaces.
Particle agents observe all agents’ positions and velocities.
They can “stay” or change their velocity in one of the four cardinal directions.
Each football agent controls one player.
Players observe the ball, other players’ positions and directions.
They can apply one of ten actions from “top_left”, “top”, “top_right”, “right”, “bottom_right”, “bottom”, “bottom_left”, “sprint”, and “dribble”.
Observability
The original environments assume full observability, where each agent can observe the position p =
( x, y ) and velocity v = (∆ x, ∆ y ) of all agents; each agent’s observation vector is thus o i, full
=
[ p 1 , . .
. , p N , v 1 , . .
, v N ] .
We extend these environments to be partially observable, where agent i observes only the portion within its receptive ﬁeld; like prior work with partial observability (Corder et al., 2019), we hide the position and velocity information of any agent j outside of agent i ’s receptive ﬁeld; i.e, if the Euclidean distance between agent i and j surpasses a vicinity threshold τ , then p j and v j are 0 in o i, partial .
We set τ = 0 .
5 for partially observable and ∞ in the original fully observable case, where the world’s width and height are 2 .
in the multi-agent particle environment and 0 . 84 : 2 .
in the Google Research football environment.
Both environments also contain objects such as obstacles, goals and a ball; they are similarly hidden if out of sight.
Partial observability is a more ecologically valid training condition since most agents in real-world tasks can only observe a small portion of their environment at a given time.
5.2 Tasks
Multi-agent particle environment We use the following tasks from the multi-agent particle environment (Lowe et al., 2017; Liu et al., 2020).
We choose N based on prior work. [Table]
Cooperative navigation: N agents must cooperate to reach a set of N goal locations.
Agents are collectively rewarded based on the occupancy of any agent on any goal location.
Heterogeneous navigation: N agents must reach N goals but they differ in speeds and sizes.
Nagents are slow and big, and the other N 2 agents are fast and small.
Physical deception: N agents cooperate to reach a single goal location and are rewarded if any one occupies the goal.
However, they are penalized when any of N 2 adversaries occupies the goal and gets rewarded.
The adversaries do not know which landmark is the goal and must infer it from the agents’ behavior.
The agents should learn to deceive the adversaries by covering all the landmarks.
Keep-away: There are N landmarks, one of which is the goal and known to N agents.
Agents are rewarded for occupying it and preventing M adversaries from reaching it.
Adversaries are rewarded for pushing the agents away from the goal, but they can only infer which landmark is the goal.
Predator-prey: N slow adversaries chase and capture N fast cooperating agents around a randomly generated obstacle-ﬁlled environment.
Each time an adversary catches an agent, the agent is penalized and the adversary is rewarded.
Google Research football We use the Academy 3vs1 with Keeper competitive task from the Google
Research football environment (Kurach et al., 2019).
In this task, three agents try to score from the edge of the penalty box, one on each side, and the other at the center.
This task is initialized with the centered agent having the ball and facing the defender.
There is an adversary who plays the keeper.
5.3 Training and evaluation
We train all algorithms with 5 random seeds.
All the hyperparameters used in the training can be found in the Appendix.
For the Multi-agent particle environment, each experiment uses one Tesla
K40 GPU to train until convergence, i.e. the best evaluation episode reward hasn’t changed for 100 epochs.
Each epoch equates to 800 K episodes of 25 timesteps.
We evaluate the algorithms by running 1 K test episodes of 25 timesteps and mainly report the mean average test episode reward and standard error across the random seeds.
We also evaluate on task-speciﬁc metrics, including agent-goal occupancy/agent-adversary collision count, and agent-goal/agent-adversary distance.
For
Google Research football, each experiment trains for 5 M timesteps.
We evaluate on and report the mean average episode rewards and the standard errors across the seeds.
5.4 Baselines
All algorithms are trained using the same agent architecture and optimization algorithm.
They vary in task-speciﬁc extrinsic rewards and intrinsic rewards.
We use two versions of the soft actor-critic algorithm Haarnoja et al.
(2018): a decentralized one that trains each agent individually without access to other agents’ observations and actions (ie.
the original soft-actor critic algorithm) and a centralized one with a critic that has access to other agents’ observations and actions (Iqbal & Sha, 2019).
Note, our intrinsic reward can also be added to non-actor-critic methods, such as COMA (Foerster et al., 2018) and VDN (Sunehag et al., 2017).
We leave this to future work to avoid conﬂating the effects of expectation alignment with COMA’s counterfactual reasoning and VDN’s value decomposition.
We use SPARSE (Lowe et al., 2017; Kurach et al., 2019), CURIO self (Stadie et al., 2015),
CURIO team (Iqbal & Sha, 2020), and variations of our ELIGN rewards.
S PARSE rewards agents only when they reach a goal state.
CURIO team is a curiosity-based multi-agent intrinsic reward which maximizes the average L 2 loss (instead of minimizing it in ELIGN ).
It rewards agents for exploring novel states (Iqbal & Sha, 2020).
CURIO self also maximizes the L 2 loss but only using agent i ’s own observation (Stadie et al., 2015).
We experiment with three variants of ELIGN: ELIGN self , incentivizing alignment to one’s own expectation ELIGN team incentivizing agents to align to their team, and ELIGN adv incentivizing misalignment to adversaries’ expectations.
Note that ELIGN self is similar to the auxiliary loss in Ndousse et al.
but we use it for policy optimization, rather than for training a state encoder.
We also add hand-crafted dense rewards to provide oracle performance for all tasks.
5.5 Results in partially observable environments with decentralized training
ELIGN outperforms baselines across cooperative and competitive tasks in the multi-agent particle environment.
Table 1 demonstrates that both ELIGN self and ELIGN team outperform all
SPARSE and CURIO self/team baseline rewards in cooperative tasks.
While not all ELIGN variants surpass the baselines in competitive tasks, we ﬁnd that ELIGN team achieves the highest reward in
Physical deception , and ELIGN adv performs the best in Predator-prey and Keep-away .
Similarly,
Figure 2 shows that ELIGN self achieves the highest per-step occupancy count in both cooperative tasks, and ELIGN adv does the best in all competitive tasks.
ELIGN outperforms baselines in the complex Google Research football environment.
As shown in Table 1, ELIGN adv achieves the best mean average episode reward in the competitive Academy 3vs1 with keeper task.
Collectively, these results provide empirical evidence that the self-organizing principle improves coordination under partial information, a setting that is most realistic to real world multi-agent systems.
In competitive tasks, agents beneﬁt more from being misaligned to adversaries than being aligned to their team members.
Among the four competitive tasks in the multi-agent particle and football environments, we ﬁnd that ELIGN adv outperforms all SPARSE and CURIO self/team baselines and other variants of ELIGN in Predator-prey , Keep-away and Academy 3vs1 with keeper .
This suggests that being misaligned to adversaries, ie taking surprising actions that conﬂict with the adversary’s expectations, might be a more useful strategy in multi-agent competitive tasks.
[Figure]
When the number of agents increases, ELIGN scales well in all multi-agent particle tasks except for Heterogenous navigation
Table 2 shows that our ELIGN intrinsic reward still largely achieves the best performance when more agents are added to cooperative and competitive tasks.
The only exception is the Heterogenous navigation task, where both ELIGN self and ELIGN team outperform SPARSE but not CURIO self/team .
We hypothesize that it is more difﬁcult for agents to predict their neighbors’ future states accurately when there are more agents with different sizes and speeds, and errors in dynamics prediction could lead to misleading alignment signals.
Further, we see a consistent increase in ELIGN team ’s performance compared against SPARSE even when the number of agents scales to ten in Cooperative navigation (Figure 3).
5.6 Results with full observability and centralized ELIGN
We further test the utility of decentralized ELIGN in fully observable environments and centralized ELIGN under partial observability.
We ﬁnd that decentralized expectation alignment helps in fully observable Cooperative navigation , Heterogenous navigation , and Predator-prey , tasks where expectation alignment has been observed in nature.
Similarly, centralized ELIGN also improves agents’ performance compared against SPARSE and CURIO self/team rewards in the same tasks with partial observability.
These results can be found in Tables 3 and 4 in the Appendix.
As full observability and centralized training are our main focus, we leave it to future work to investigate why expectation alignment beneﬁts these tasks but not others.
5.7 Investigating how the ELIGN reward helps
S P
A R S E c u r
i o s e l f c u r i
o t e a m e l i g n s e l f e l i g n t e a m 0 .0 .
O cc up a n c y
C o un t
Coop nav (3v0) ↑
S P
A R S E c u r
i o s e l f c u r i
o t e a m e l i g n s e l f e l i g n t e a m 0 .0 .
O cc up a n c y
C o un t
Hetero nav (4v0) ↑
S P
A R S E c u r
i o s e l f c u r i
o t e a m e l i g n s e l f e l i g n t e a m e l i g n a d v 0 .0 .
O cc up a n c y
C o un t
Phy decep (2v1) ↑
S P
A R S E c u r
i o s e l f c u r i
o t e a m e l i g n s e l f e l i g n t e a m e l i g n a d v 0 .0 .
C o lli s i o n C o un t
Predator-prey (2v2) ↓
S P
A R S E c u r
i o s e l f c u r i
o t e a m e l i g n s e l f e l i g n t e a m e l i g n a d v 0 .0 .
O cc up a n c y
C o un t
[Figure]
-away (2v2)
[Figure]
[Figure]
[Figure]
We further investigate how expectation alignment improves through ELIGN helps agents divide subtasks. A core challenge in multiagent collaboration is efﬁcient task division (Wang et al., 2020b). Here, we test whether expectation alignment improves sub-task allocation.
We initialize agents in states without an optimal sub-task allocation, necessitating symmetry-breaking (Hu et al., 2020).
Figure 4 illustrates the symmetry-breaking setups: in cooperative navigation, when agents are initialized equidistant to all the goal locations, there isn’t an optimal allocation of agents to goals.
We ﬁnd that ELIGN self achieves the best performance in both cooperative tasks, while ELIGN team and ELIGN adv are the best strategies in Physical deception and Predator-prey respectively (Figure 5).
Upon a qualitative evaluation of Cooperative navigation , we observe that agents with expectation alignment are able to predict which goals will be covered by their collaborators and move towards their allocated one.
Without expectation alignment, agents move towards the same goal.
ELIGN helps agents generalize to new partners.
Another core challenge in multi-agent collaboration is zero-shot coordination, where agents are tested to collaborate with new partners they haven’t been trained with.
We study whether expectation alignment enables better zero-shot coordination.
New partners are sampled from other training runs with different seeds and the team is evaluated using the same metrics as before.
We observe that ELIGN strategies enable better performance than SPARSE on average, and one of ELIGN self,team,adv performs the best in Heterogenous navigation , Physical deception and Keep away .
(Figure 7).
These results suggest that ELIGN results in better zero-shot coordination with new partners sampled from separate training runs.
Accuracy of the dynamics model affects ELIGN.
We investigate the accuracy of the dynamics model in calculating useful intrinsic rewards.
Since ELIGN uses a dynamics model to calculate
[Figure] rewards, we test whether an inaccurate model misleads agents towards unaligned behaviors. We train agents on noisy dynamics models by adding Gaussian noise (cid:15) ∼ N (0 , σ ) to the predictions made by the dynamics model. We run experiments with multiple σ values to study how performance changes as the dynamics model becomes more noisy: σ ∈ [0 .
Our experiments cover one cooperative and one competitive task.
Figure 6 plots the ﬁnal dynamics loss against the reward change from the noiseless run.
As the dynamics model degrades, we observe that the task performance also drops.
This study identiﬁes the importance of an accurate dynamics model, suggesting that expectation alignment should used in environments where an accurate dynamics model can be learned.
Discussion
Limitations and future work.
While curiosity has proven useful for exploration in single-agent tasks, we ﬁnd that expectation alignment—which mathematically encourages agents to be more predictable instead of ﬁnding novelty—outperforms curiosity in numerous multi-agent tasks. We hypothesize that our results arise because today’s multi-agent task state space requires signiﬁcantly less exploration than those used for single-agent (e.g. Atari games). Our ﬁndings are limited to the multi-agent particle and Google Research football environments, which have a smaller action space than most ecologically valid scenarios. Language, motion, and human gesture are all combinatorially vast; in such action spaces, expectation alignment might develop social dynamics that hinder non-optimal multi-agent behaviors.
Similarly, photorealistic environments have a larger state space, where teams perform common household activities (e.g., cooking, cleaning, etc.) or drive together in crowded cities (Srivastava et al., 2021). Future work should develop new multi-agent environments that demand exploration complexity and where both curiosity and expectation alignment would be necessary for collaboration. For example, in a search and rescue task where a single agent is unable to carry the injured, curiosity would encourage “search” while ELIGN would speed up “rescue”. In the end, we envision that both these forms of rewards would be necessary for successful collaboration.
However, choosing when to encourage curiosity versus expectation alignment is an open research problem.
Additionally, enabling stable multi-agent training without centralized training could open up future opportunities for legible (Dragan et al., 2013) agents in human environments.
Agents with interpretable actions can induce more faithful human mental models, improving human-AI interaction; however, predictability does not imply legibility.
Future work could explore the role of legibility in designing intrinsic rewards.
Future work should also explore the use of expectation alignment in massive collaboration settings with hundreds of agents.
Drawing on Zoology research Couzin (2007) expectation alignment should scale in such settings if agents align their behaviors only to their nearest neighbors and not to the entire cohort.
Conclusion.
Inspired by the self-organizing principle in Zoology, we introduce ELIGN, i.e, expectation alignment, a simple, task-agnostic, and self-supervised intrinsic reward for multi-agent systems.
ELIGN rewards agents when they act predictably to their teammates and unpredictably to their adversaries.
ELIGN improves multi-agent performance across six cooperative and competitive tasks in the multi-agent particle and Google Research football environments, especially for decentralized training under partial observability.
It also scales well, helps agents break symmetries, and generalize to new partners.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Section 5 for the results mentioned in the abstract.
(b) Did you describe the limitations of your work?
[Yes] See Section 6 for a discussion on the limitations.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the ethics review guidelines.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the Appendix (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the Experiments 5 and Appendix sections.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See the Experiments 5 section.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the Experiments 5 and Appendix sections.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See the Alignmentand Experiments 5 sections.
(b) Did you mention the license of the assets?
[Yes] See the Appendix (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See the Appendix (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
We did not collect any data.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
We did not collect any data.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not crowdsource any data.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
There were no human subjects experiments in our work.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
No human subjects were hired.
A Appendix A.1 Code
We upload our code for training and evaluating agents with and without expectation alignment in both the multi-agent particle and Google Research football environments here: https://github. com/StanfordVL/alignment .
A.2 Symmetry-breaking initializations
We create a symmetry-breaking version of each task for evaluation by initializing the environment in the following ways:
Cooperative Navigation and Heterogenous Navigation: All agents are initialized at the origin (i
e. center of the world), and target landmarks are placed randomly on a circle perimeter with the maximum radius (i e. world radius - the greatest landmark size) so that each agent is equidistant from each target landmark. Physical Deception: Both agents and adversaries start at the origin. All the landmarks, including the goal, are randomly initialized on a circle perimeter.
Predator-prey: The collaborative agents are initialized at the center while the adversaries are placed randomly on a circle perimeter.
All the landmarks are randomly initialized in the world.
Keep-away: All the cooperative agents are placed at the origin.
Adversaries and landmarks, including the goal, are randomly initialized on a circle perimeter.
In this task setup, we do not initialize the adversaries at the center because they are awarded for colliding with the cooperative agents.
A.3 Learning curves
Compared to the best baseline’s highest performance at the 100th epoch, we ﬁnd that it only takes the best ELIGN variant 31 (Coop nav), 74 (Hetero nav), 30 (Physical dec), 97 (Predator-prey) and 92 (Keep-away) epochs respectively in the ﬁve multi-agent particle tasks (8).
This means that on average the best ELIGN variant requires 35 ± 32 fewer training steps to reach the same performance as curiosity or spare methods.
A.4 Assets and licenses
We use four assets in total, two of which are existing multi-agent reinforcement learning environments, and the other two are libraries for training reinforcement learning algorithms.
We conduct our evaluation on the multi-agent particle (Lowe et al., 2017) and Google Research football (Kurach et al., 2019) environments, which are under the MIT license and Apache-2.0 license respectively.
We adapt the tianshou (Weng et al., 2021) and rllib (part of the ray package) (Liang et al.)
libraries to our experiments, and they are under the MIT license and Apache-2.0 license respectively.
A.5 Societal impacts
While developing new intrinsic rewards to improve decentralized multi-agent training can help develop and deploy agents in a variety of applications, we foresee no immediate societal consequences of this work.
However, our experiments thus far have not studied the possible degradation of behaviors when agents align to malicious teammates.
We have also not tested how emergent properties promote better or worse human collaborators.
[Table]
[Table]
A.6 Additional tables
We include 20 tables of additional results that quantify the agents’ performance under fully observable environments, with centralized training, and beyond extrinsic reward.
Table 3 reports the test episode rewards of decentralized methods under the fully observable multiagent particle environment.
Table 4 reports the test episode rewards of centralized methods under partial observability.
Table 5 and 6 report two sets of metrics of decentralized methods trained with different intrinsic rewards in both partially and fully observable settings.
Table 5 reports the average number of agent-target occupancies per step (or, we can understand it as: on average, the total number of goals occupied by the agents at any given timestep throughout an episode) and agent-adversary collisions in Predator-prey .
Higher scores are better for the occupancy metric, and lower scores are better for collision.
Table 6 reports the average minimum agent-to-target distance and agent-to-adversary distance.
Agent-to-target distances measure the closest distance an agent achieves to the target location; lower scores are better on this metric.
Agent-to-adversary distances measure the closest distance an adversary gets to a good agent; higher scores are better on this metric.
Note that these distance-based metrics are not included in the reward functions, and should mainly be used to make comparisons in the case where primary metrics (i.e, reward and occupancy/collision count) have the same values.
Table 7 and 8 report the same metrics as 5 and 6 respectively, but in scaled environments with more agents. Tables 9, 10 and 11 report the test episode reward and additional metrics of decentralized algorithms in the symmetry-breaking experiments conducted under “Investigating how alignment reward helps”. Table 12, 13 and 14 report the same set of metrics but from experiments conducted in scaled and symmetry-breaking environments. Table 15 reports the test mean episode rewards of centralized algorithms with different intrinsic rewards under full observability.
Table 16 and 17 show the other two sets of metrics (i.e, occupancy/collision count and agent-target/agent-adversary distance) of centralized algorithms.
Table 18, 19, and 20 contain the same metrics as 15, 16 and 17 respectively, but in scaled environments.
Finally, Tables 21 and 22 report the test episode reward values and secondary distance-based metrics for the zero-shot generalization experiments conducted under “Investigating how ELIGN reward helps”.
These experiments measure how well agents trained on different seeds generalized to new partners trained on other seeds.
[Table] [Table]
Table 23 presents the model architecture and hyperparameters used to train the algorithms in the multi-agent particle and Google Research football environments. [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]