Alignment as a Multi-agent Intrinsic Reward
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Modern multi-agent reinforcement learning frameworks rely on centralized training and reward shaping to perform well.
However, centralized training and denserewards are not readily available in the real world.
Current multi-agent algorithmsstruggle to learn in the alternative setup of decentralized training or sparse rewards.
To address these issues, we propose a self-supervised intrinsic reward called alignment inspired by the self-organization principle in Zoology.
Similar to how animalscollaborate in a decentralized manner with those in their vicinity, agents trainedwith alignment learn behaviors that match their neighbors’ expectations.
Thisallows the agents to learn collaborative behaviors without any external reward orcentralized training.
We demonstrate the efﬁcacy of our approach across 6 tasks inthe multi-agent particle and the complex Google Research football environments,comparing alignment to sparse and curiosity-based intrinsic rewards.
When thenumber of agents increases, alignment scales well in all multi-agent tasks except forone where agents have different capabilities.
We show that agent coordination improves through alignment because agents learn to divide tasks amongst themselves,break coordination symmetries, and confuse adversaries.
These results identifytasks where alignment is a more useful strategy than curiosity-driven explorationfor multi-agent coordination, enabling agents to do zero-shot coordination.
Introduction
Many real world AI applications can be formulated as multi-agent systems, including autonomousvehicles (Cao et al., 2012), resource management (Ying & Dayong, 2005), trafﬁc control (Sunehaget al., 2017), robot swarms (Swamy et al., 2020) and multi-player video games (Berner et al., 2019).
Agents must adapt their behaviors to each other in order to coordinate successfully in these systems.
However, adaptive coordination algorithms are challenging to develop because each agent is not privyto other agents’ intentions and their future behaviors (Foerster et al., 2017).
These challenges are more acute in decentralized training under partial observability than centralizedtraining or full observability.
In the real world, agents act under partial observability and learn in adecentralized manner: they do not learn collaborative behaviors with a single centralized algorithmwith a complete knowledge of the environment (Iqbal & Sha, 2019; Liu et al., 2020).
Unfortunately,the most successful multi-agent algorithms train agents with a centralized critic, assuming access toall agents’ observations and actions (Foerster et al., 2018;
Rashid et al., 2018; Sunehag et al., 2017;
Lowe et al., 2017).
The most successful multi-agent algorithms for decentralized training and partialobservability assume task-speciﬁc reward shaping (Jain et al., 2020; Iqbal & Sha, 2019), which isexpensive to generate.
These algorithms struggle to learn with sparse reward structure.
Consider a cooperative navigation task, where N agents aim to simultaneously occupy N goallocations.
A centralized algorithm with full observability is capable of optimally assigning the nearestgoal location to each respective agent.
However, with partial observability, agents can see only ahandful of goal locations and other agents.
They are unaware of others’ observations, actions, and
( a ) A li g n e d
→ h i g h ( b ) M i s a li g n e d → l o w
Predicted next observation:
A g e n t j ’ s d y n a m i
c s p
r e d i c t i o n o f i
Goal
Agent i
Agent j
Receptive ﬁeld
Observation at t+1: Observation at t: I predict that agent j expects me to move up intentions with decentralized training.
We observe that agents simultaneously occupy the same goal;they fail to collaborate because they do not predict which goal each agent is expected to occupy.
Toovercome instances of miscoordination, decentralized algorithms have adapted single-agent curiositybased intrinsic rewards (Pathak et al., 2017; Stadie et al., 2015).
Multi-agent curiosity-based rewardsincentivize agents to explore novel states (Iqbal & Sha, 2020).
Although curiosity helps agentsdiscover new goal locations, it doesn’t solve the challenge of coordination, such as assigning goals toeach agent.
Only a few attempts explore other forms of multi-agent intrinsic rewards (Iqbal & Sha,2020; Böhmer et al., 2019; Schafer, 2019).
In this work, we propose alignment as a novel multi-agent self-supervised intrinsic reward, enablingdecentralized training under partial observability.
Intuitively, alignment encourages agents to elicitbehaviors that decrease future uncertainty for their team: it encourages each agent to choose actionsthat match their teammates’ expectations.
Going back to the cooperative navigation task, alignmentencourages each agent to move to goals others expect it to occupy, like goals that are either closestto the agent or goals that other agents aren’t moving towards (Figure 1).
We take inspirationfrom the self-organization principle in Zoology (Couzin, 2007).
This principle hypothesizes thatcollective animal intelligence emerges because groups synchronize their behaviors using only theirlocal environment; they do not rely on complete information about other agents and can coordinatesuccessfully by predicting the dynamics of agents within their ﬁeld-of-view (Collett et al., 1998;
Theraulaz & Bonabeau, 1995; Ben-Jacob et al., 1994; Buhl et al., 2006).
Similarly, alignment asan intrinsic reward is calculated based on the agent’s local observations and its approximation ofneighboring agents’ expectations.
It does not require a centralized controller nor full observability.
Alignment is task-agnostic and we apply it to both collaborative and competitive multi-agent tasks.
We demonstrate the efﬁcacy of our approach in the multi-agent particle and Google Research footballenvironments, two popular benchmarks for multi-agent reinforcement learning (Lowe et al., 2017;
Kurach et al., 2019).
We evaluate alignment under partial and full observability, with decentralizedand centralized training, and in terms of scalability.
We observe that alignment outperforms sparseand curiosity-based intrinsic rewards (Ndousse et al., 2021; Stadie et al., 2015; Iqbal & Sha, 2020),especially under partial observability with decentralized training.
We additionally test alignment asa way to perform zero-shot coordination with new agent partners, and investigate why alignmentimproves coordination.
We show that agent coordination improves through alignment because agentslearn to divide tasks amongst themselves and break coordination symmetries (Hu et al., 2020).
Related Work
Our formulation of alignment, a task-agnostic intrinsic reward for multi-agent training, draws inspiration from the self-organization principle in Zoology, which posits that synchronized group behavioris mediated by local behavioral rules (Couzin, 2007) and not by a centralized controller (Camazineet al., 2020).
Group cohesion emerges by predicting and adjusting one’s behavior to that of nearneighbors (Buhl et al., 2006).
This principle underlies the coordination found in multi-cellular organ- isms (Camazine et al., 2020), the migration of wingless locusts (Collett et al., 1998), the collectiveswarms of bacteria (Ben-Jacob et al., 1994), the construction of bridge structures by ants (Theraulaz& Bonabeau, 1995), and some human navigation behaviors (Couzin, 2007).
Intrinsic motivation for single agents.
Although we draw inspiration from Zoology for formalizing alignment as an intrinsic reward, there is a rich body of work on intrinsic rewards within thesingle-agent reinforcement learning community.
To incentivize exploration, even when non-optimalsuccessful trajectories are uncovered ﬁrst, scholars have argued for the use of intrinsic motivation (Schmidhuber, 1991).
Single-agent intrinsic motivation has focused on exploring previouslyunencountered states (Pathak et al., 2017; Burda et al., 2018a), which works particularly well indiscrete domains.
In continuous domains, identifying unseen states requires keeping track of anintractable number of visited states; instead, literature has recommended learning a forward dynamicsmodel to predict future states and identify novel states using the uncertainty of this model (Achiam &
Sastry, 2017).
Other formulations encourage re-visiting states where the dynamics model’s predictionof future states errs (Stadie et al., 2015; Pathak et al., 2017).
Follow up papers have improved howuncertainty (Kim et al., 2020) and model errors (Burda et al., 2018b; Sekar et al., 2020) are calculated.
Intrinsic motivation for multiple agents.
Most multi-agent intrinsic rewards have been adaptedfrom single-agent curiosity-based incentives ( ? Böhmer et al., 2019; Schafer, 2019) and have primarily focused on cooperative tasks.
They propose intrinsic rewards to improve either coordination,collaboration, or deception: These rewards either maximize information conveyed by an agent’sactions ( ? Chitnis et al., 2020; Wang et al., 2019), shape the inﬂuence of an agent (Jaques et al. ,2019; Foerster et al., 2017), incentivize agents to hide intentions (Strouse et al., 2018), build accuratemodels of other agents’ policies (Hernandez-Leal et al., 2019; Jaques et al., 2019), or break extrinsicrewards for better credit assignment (Du et al., 2019).
Several multi-agent intrinsic rewards (Hernandez-Leal et al., 2019; Jaques et al., 2019), includingours, rely on the ability to model others’ dynamics in a shared environment.
This ability is a keycomponent to coordination, closely related to Theory of Mind (Tomasello et al., 2005).
Our workcan be interpreted as using a Theory of Mind model of others’ behaviors to calculate an intrinsicmotivation loss.
Unlike existing Theory of Mind methods that learn a model per collaborator (Royet al., 2020), we learn a single dynamics model, allowing our method to scale as the number ofagents increase.
Our proposal is related to model-based reinforcement learning (Jaderberg et al.
,2016; Wang et al., 2020a); however, instead of learning a dynamics model for control, we learn adynamics model as a source of reward.
Our work is closely related to a recently proposed auxiliaryloss on predicting an agent’s own future states (Ndousse et al., 2021).
However, there are threekey differences.
First, their work predicts ego-agent observations, whereas our work additionallypredicts future observations from the other agents’ point of view.
Second, their loss optimizes stateembeddings while ours optimizes agents’ policies.
Third, their work focuses on cooperative taskswhereas ours applies to both cooperative and competitive domains.
Multi-agent reinforcement learning algorithms.
Today, the predominant deep multi-agent framework uses actor-critic methods with a centralized critic and decentralized execution (Lowe et al.
Foerster et al., 2018; Iqbal & Sha, 2019; Liu et al., 2020; Rashid et al., 2018).
This frameworkallows a critic to access the observations and actions of all agents to ease training.
However, there areseveral situations where centralized training may not be desirable or possible.
Examples include lowbandwidth communication restrictions or human-robot tasks where observations cannot be easilyshared between agents (Ying & Dayong, 2005; Cao et al., 2012; Huang et al., 2015).
Decentralizedtraining is therefore the most practical training paradigm but it suffers from unstable training: theenvironment is nonstationary from a single-agent’s perspective (Lowe et al., 2017).
Our work uses adecentralized training framework and tackles the nonstationarity challenge with an intrinsic rewarddesigned to improve an agent’s ability to model others.
We also apply alignment to centralizedtraining and observe that it still aids cooperative and some competitive tasks.
Background
We formulate our setting as a partially observable Markov game ( S , O , A , T , r ex , N ) (Littman, 1994).
A Markov game for N agents is deﬁned by a state space
S describing the possible conﬁgurations 127 of the environment.
The observation space for agents is
O =
( O 1 , . . .
, O N )
and the action space is
A =
( A 1 , . . .
, A N ) .
Each agent i observes o i ∈
O i , a private partial view of the state, and performs actions a i ∈
A i .
Using the observation, each agent uses a stochastic policy π θ i : O i × A i
→ [0 , 1] , 130 where θ i parameterizes the policy.
The environment changes according to the state transition functionwhich transitions to the next state using the current state and each agent’s actions, T : S ×
A → S .
The team of agents obtains a shared extrinsic reward as a function of the environment state, r ex :
S ×
A → R .
The team’s goal is to maximize the total expected return: R = (cid:80)
Tt =
γ t r t ex where 134≤ γ ≤ 1 is the discount factor, t is the time step, and T is the time horizon.
The environment mayalso contain adversarial agents who have their own reward structure.
Alignment
To understand alignment intuitively, let’s revisit the cooperative navigation task, where N agents arerewarded for simultaneously occupying as many goal locations as possible.
In Figure 1, agent i has adynamics model trained on its past experiences.
It predicts how future states will evolve from thepoint of view of agent j , who is within i ’s view.
In this example, j will expect i to move towards thegoal since i is closer to it.
Alignment encourages i to pursue the action that j expects (Figure 1(a)).
In turn, j can now assume that the observed goal location will eventually be occupied by i and shouldtherefore explore to ﬁnd another goal.
By aligning shared expectations, agent behaviors become morepredictable.
Conversely, when neighbors behave opposite to an agent’s predictions, the agent caninfer about the environment outside of its own receptive ﬁeld (Krause et al., 2002).
For example, in
Figure 1 (b), if agent j observes i running away from a goal, this surprising behavior might indicatethe existence of an adversary outside j ’s receptive ﬁeld.
Our training algorithm consists of three interwoven phases of learning a dynamics model, calculatingthe alignment reward, and optimizing the agent’s policy (Algorithm 1).
4.1 Training the dynamics model
2: Initialize N agents with random θ i : i ∈ [1 , N ] 3: while not converged do 4: for b = 1 . . .
B do 5:
Populate buffer D with episode using policies ( π θ 1 , . . .
, π θ N ) 6: end for 7: //
T RAIN DYNAMICS MODEL 8: for agent i = 1 . . .
N do 9:
Sample transitions: { ( o i , a i , r ex , o (cid:48) i ) } ∼ D i 10:
Predict ˆ o (cid:48) i
=
f θ i
( o i , a i ) 11:
Update dynamics θ i using o (cid:48) i .
12: end for 13: //
C ALCULATE ALIGNMENT REWARD 14: for agent i = 1 . . .
N do 15:
Sample
B transitions: { ( o i , a i , r ex , o (cid:48) i ) } ∼ D i 16:
Compute intrinsic rewards r in ( o i , a i ) 17:
Add { ( o i , a i , r ex + βr in , o (cid:48) i ) } to D (cid:48) i 18: end for 19: //
P OLICY LEARNING 20:
Update all θ i s using transitions from D (cid:48) 21: end while
Similar to prior work (Wang et al., 2018; Kidambi et al., 2020), each agent i learns a dynamics model f
θ i to predict the next observation ˆ o (cid:48)
i given its current observation and action o i , a i , i.e, ˆ o (cid:48)
i =
f θ i
( o i , a i ) .
We use a three-layer Multi-Layer Perceptron with
ReLU non-linearities as the dynamics model.
Weminimize the mean squared error between its prediction and ground truth next observation o (cid:48) i .
4.2 Calculating intrinsic reward
The intrinsic reward captures how well agent i aligns to its neighbors’ (e.g., agent j ’s) expectations on its next state.
Calculating this reward requires j to accurately predict i ’s behavior, simulating a Theory of Mind (Tomasello et al., 2005).
As suggested by the self-organization principle, i must learn to align to j ’s predictions.
Ideally, the alignment intrinsic reward is calculated as:
[Equation]
where N ( i ) is the set of neighbors within i ’s receptive ﬁeld, including i itself.
The alignment rewardis high when the average L 2 loss is small, i.e, when i ’s actual next observation is close to agent j ’spredicted observation of i for all j in its neighbors.
In that case, i has chosen an action that alignswith j ’s expectations of how i should act.
In a decentralized training setup, however, i doesn’t have access to j ’s dynamics model f θ j , so i approximates j ’s dynamic model with a proxy: its own dynamics model f θ i and the knowledge of agent j ’s observation radius.
Such an approximation is ecologically valid since we often approximate others’ behaviors using a second-order cognitive Theory of Mind (Morin, 2006).
Additionally, i doesn’t have access to j ’s entire observation; so, we restrict the future prediction from j ’s point of view by using the portion of j ’s observation i can see: o i ∩ j = o i (cid:12) o j .
Agent i ’s decentralized intrinsic reward then becomes:
[Equation]
We found that the approximation of f θ j using f θ i works well empirically.
Dynamics model lossesfor all agents quickly decrease within 5-10 training epochs.
we validate its applicability in smallscale heterogeneous multi-agent tasks where agents have variable capabilities, although we ﬁnd themethods perform similarly when more heterogeneous agents are added.
4.3 Policy learning
Once the alignment rewards are calculated, the total rewards at each step for each agent i is: r i = r ex +βr in ( o i , a i ) where r ex is the extrinsic reward provided by the environment and β is a hyperparameterfor weighing the intrinsic reward in the agent’s overall reward calculation.
In practice, we set β to be1 |O i | where |O i | is the observation dimension; we ﬁnd this scale generalizes well across tasks.
Sinceour contribution is agnostic to any particular multi-agent training algorithm, the team of agents cannow be trained using any multi-agent training algorithm to maximize returns R
= (cid:80) Tt =
γ t r .
Both centralized and decentralized training algorithms can make use of these rewards.
We primarily use the multi-agent decentralized variant of the soft-actor critic algorithm in our experiments (Haarnoja et al., 2018; Iqbal & Sha, 2019).
Compared to centralized joint-action training,whose action space grows exponentially in N agents, our decentralized method has linear spacecomplexity.
Further, decentralized training can parallelize training time to be less than linear withrespect to N .
Although we present results with one centralized training framework, studying theimpact of alignment with all the centralized critic frameworks is out of scope for this paper.
4.4 Extending alignment to competitive tasks
We extend the alignment formulation to competitive tasks where a team of agents compete againstadversaries.
In this case, agents are encouraged to misalign with their adversaries’ expectations, i.e,agents are incentivized to be unpredictable to their adversaries within its receptive ﬁeld ( N adv ( i ) ):
[Equation]
Experiments
Our experiments explore the utility of using alignment as an intrinsic reward compared to sparseand curiosity-based intrinsic rewards.
We primarily focus on decentralized training under partialobservability.
However, we also demonstrate that alignment can easily augment centralized methodsand assist in fully observable tasks.
We vary the number of agents in the multi-agent particle tasks totest scalability.
We end by investigating how and why alignment improves coordination by designingthree evaluation conditions.
First, does alignment improve coordination by breaking symmetries (Huet al., 2020; Wang et al., 2020b)?
Second, does alignment enable zero-shot generalization to newpartners?
Lastly, is the dynamics model critical in aligning agent behaviors?
5.1 Environments
We evaluate alignment across both cooperative and competitive tasks in the multi-agent particleenvironment (Mordatch & Abbeel, 2017; Lowe et al., 2017) and the Google Research footballenvironment (Kurach et al., 2019).
State and action space
The multi-agent particle environment is a two-dimensional world.
The
Google Research football environment is a three-dimensional world.
Both environments have [Table] [Figure] /collision count per step of decentralized algorithms under partially observable multi-agent particle tasks.
On these metrics, ALIGN self and ALIGN adv perform the best on cooperative and competitive tasks respectively.
continuous state spaces and discrete action spaces.
Particle agents observe all agents’ positions andvelocities.
They can “stay” or change their velocity in one of the four cardinal directions. Eachfootball agent controls one player.
Players observe the ball, other players’ positions and directions.
They can apply one of ten actions from “top_left”, “top”, “top_right”, “right”, “bottom_right”,“bottom”, “bottom_left”, “sprint”, and “dribble”.
Observability
The original environments assume full observability, where each agent can observethe position p
=
( x, y ) and velocity v = (∆ x, ∆ y ) of all agents; each agent’s observation vectoris
thus o i, full =
[ p 1 , . .
. , p N , v 1 , . .
, v N ] .
We extend these environments to be partially observable,where agent i observes only the portion within its receptive ﬁeld; like prior work with partialobservability (Corder et al., 2019), we hide the position and velocity information of any agent joutside of agent i ’s receptive ﬁeld; i.e, if the Euclidean distance between agent i and j surpassesa vicinity threshold τ , then p j and v j are 0 in o i, partial .
We set τ = 0 .
5 for partially observableand ∞ in the original fully observable case, where the world’s width and height are 2 .
in themulti-agent particle environment and 0 . 84 : 2 .
in the Google Research football environment.
Bothenvironments also contain objects such as obstacles, goals and a ball; they are similarly hidden ifout of sight.
Partial observability is a more ecologically valid training condition since most agents inreal-world tasks can only observe a small portion of their environment at a given time.
5.2 Tasks
Multi-agent particle environment We use the following tasks from the multi-agent particle environment (Lowe et al., 2017; Liu et al., 2020).
We choose N based on prior work.
Cooperative navigation: N agents must cooperate to reach a set of N goal locations.
Agents arecollectively rewarded based on the occupancy of any agent on any goal location.
Heterogeneous navigation: N agents must reach N goals but they differ in speeds and sizes.
N 2agents are slow and big, and the other N 2 agents are fast and small.
Physical deception: N agents cooperate to reach a single goal location and are rewarded if any oneoccupies the goal.
However, they are penalized when any of N 2 adversaries occupies the goal andgets rewarded.
The adversaries do not know which landmark is the goal and must infer it from theagents’ behavior.
The agents should learn to deceive the adversaries by covering all the landmarks.
[Table]
Keep-away: There are N landmarks one of which is the goal and known to N agents.
Agents are [Table] rewarded for occupying it and preventing M adversaries from reaching it.
Adversaries are rewarded
[Table]
for pushing the agents away from the goal
[Table]
but they infer landmark goal.
[Table]
Predator-prey: N slow adversaries chase and capture N fast cooperating agents around a randomly
[Table]
generated obstacle-ﬁlled environment.
Each time an adversary catches an agent, the agent is penalizedand the adversary is rewarded.
Google Research football We use the Academy 3vs1 with Keeper competitive task from the Google
Research football environment (Kurach et al., 2019).
In this task, three agents try to score from theedge of the penalty box, one on each side, and the other at the center.
This task is initialized with thecentered agent having the ball and facing the defender.
There is an adversary who plays the keeper.
5.3 Training and evaluation
We train all algorithms with 5 random seeds.
All the hyperparameters used in the training can befound in the Appendix.
For the Multi-agent particle environment, each experiment uses one Tesla
K40 GPU to train until convergence, i.e. the best evaluation episode reward hasn’t changed for100 epochs.
Each epoch equates to 800 K episodes of 25 timesteps.
We evaluate the algorithms byrunning 1 K test episodes of 25 timesteps and mainly report the mean average test episode rewardand standard error across the random seeds.
We also evaluate on task-speciﬁc metrics, includingagent-goal occupancy/agent-adversary collision count, and agent-goal/agent-adversary distance.
For
Google Research football, each experiment trains for 5 M timesteps.
We evaluate on and report themean average episode rewards and the standard errors across the seeds.
5.4 Baselines
All algorithms are trained using the same agent architecture and optimization algorithm.
They varyin task-speciﬁc extrinsic rewards and intrinsic rewards.
We use two versions of the soft actor-criticalgorithm Haarnoja et al.
(2018): a decentralized one that trains each agent individually without accessto other agents’ observations and actions (ie.
the original soft-actor critic algorithm) and a centralizedone with a critic that has access to other agents’ observations and actions (Iqbal & Sha, 2019).
Note,our intrinsic reward can also be added to non-actor-critic methods, such as COMA (Foerster et al.
and VDN (Sunehag et al., 2017).
We leave this to future work to avoid conﬂating the effects ofalignment with COMA’s counterfactual reasoning and VDN’s value decomposition.
We use SPARSE (Lowe et al., 2017; Kurach et al., 2019), CURIO self (Stadie et al., 2015),
CURIO team (Iqbal & Sha, 2020), and variations of our ALIGN rewards.
S PARSE rewards agentsonly when they reach a goal state.
CURIO team is a curiosity-based multi-agent intrinsic reward whichmaximizes the average L 2 loss (instead of minimizing it in ALIGN ).
It rewards agents for exploringnovel states (Iqbal & Sha, 2020).
CURIO self also maximizes the L 2 loss but only using agent i ’sown observation (Stadie et al., 2015).
We experiment with three variants of alignment: ALIGN self ,incentivizing self-alignment ALIGN team incentivizing agents to align to their team, and ALIGN advincentivizing misalignment to adversaries.
Note that ALIGN self is similar to the auxiliary loss in
Ndousse et al.
but we use it for policy optimization, rather than for training a state encoder.
We also add hand-crafted dense rewards to provide oracle performance for all tasks.
5.5 Results in partially observable environments with decentralized training
Alignment outperforms baselines across cooperative and competitive tasks in the multi-agentparticle environment.
Table 1 demonstrates that both A LIGN self and A LIGN team outperform all
SPARSE and CURIO self/team baseline rewards in cooperative tasks.
While not all A LIGN variantssurpass the baselines in competitive tasks, we ﬁnd that A LIGN team achieves the highest reward in
Physical deception , and A LIGN adv performs the best in Predator-prey and Keep-away .
Similarly,
Figure 2 shows that A LIGN self achieves the highest per-step occupancy count in both cooperativetasks, and A LIGN adv does the best in all competitive tasks.
Alignment outperforms baselines in the complex Google Research football environment.
Asshown in Table 1, ALIGN adv achieves the best mean average episode reward in the competitive
Academy 3vs1 with keeper task.
Collectively, these results provide empirical evidence that the selforganizing principle improves coordination under partial information, a setting that is most realisticto real world multi-agent systems.
In competitive tasks, agents beneﬁt more from being misaligned to adversaries than beingaligned to their team members.
Among the four competitive tasks in the multi-agent particle andfootball environments, we ﬁnd that ALIGN adv outperforms all SPARSE and CURIO self/team baselines andother variants of ALIGN in Predator-prey , Keep-away and Academy 3vs1 with keeper .
This suggeststhat being misaligned to adversaries, ie taking surprising actions that conﬂict with the adversary’sexpectations, might be a more useful strategy in multi-agent competitive tasks.
[Figure]
When the number of agents increases, alignment scales well inall multi-agent particle tasks except for Heterogenous navigation .
Table 2 shows that our ALIGN intrinsic reward still largelyachieves the best performance when more agents are added to cooperative and competitive tasks.
The only exception is the Heterogenousnavigation task, where both ALIGN self and ALIGN team outperform
SPARSE but not CURIO self/team .
We hypothesize that it is more difﬁcult for agents to predict their neighbors’ future states accuratelywhen there are more agents with different sizes and speeds, anderrors in dynamics prediction could lead to misleading alignmentsignals.
Further, we see a consistent increase in ALIGN team ’s performance compared against SPARSE even when the number of agentsscales to ten in Cooperative navigation (Figure 3).
5.6 Results with full observability and centralized alignment
We further test the utility of decentralized alignment in fully observable environments and centralizedalignment under partial observability.
We ﬁnd that decentralized alignment helps in fully observable
Cooperative navigation , Heterogenous navigation , and Predator-prey , tasks where alignment has beenobserved in nature.
Similarly, centralized alignment also improves agents’ performance comparedagainst SPARSE and CURIO self/team rewards in the same tasks with partial observability.
These resultscan be found in Tables 3 and 4 in the Appendix.
As full observability and centralized training are ourmain focus, we leave it to future work to investigate why alignment beneﬁts these tasks but not others.
5.7 Investigating how alignment reward helps
[Figure]
alignment improves coordination three evaluation setups.
Alignment helps agents divide sub- tasks.
A core challenge in multiagent collaboration is efﬁcient task division (Wang et al., 2020b).
Here, wetest whether alignment improves subtask allocation.
We initialize agentsin states without an optimal sub-taskallocation, necessitating symmetry-breaking (Hu et al., 2020).
Figure 4 illustrates the symmetry-
S P
A R S E c u r
i o s e l f c u r i
o t e a m a l i g n s e l f a l i g n t e a m 0 .0 .
O cc up a n c y
C o un t
Coop nav (3v0) ↑
S P
A R S E c u r
i o s e l f c u r i
o t e a m a l i g n s e l f a l i g n t e a m 0 .0 .
O cc up a n c y
C o un t
Hetero nav (4v0) ↑
S P
A R S E c u r
i o s e l f c u r i
o t e a m a l i g n s e l f a l i g n t e a m a l i g n a d v 0 .0 .
O cc up a n c y
C o un t
Phy decep (2v1) ↑
S P
A R S E c u r
i o s e l f c u r i
o t e a m a l i g n s e l f a l i g n t e a m a l i g n a d v 0 .0 .
C o lli s i o n C o un t
Predator-prey (2v2) ↓
S P
A R S E c u r
i o s e l f c u r i
o t e a m a l i g n s e l f a l i g n t e a m a l i g n a d v 0 .0 .
O cc up a n c y
C o un t
[Figure]
[Figure]
e l f
[Figure]
o
[Figure]
breaking setups: in cooperative navigation, when agents are initialized equidistant to all the goal locations, there isn’t an optimal allocation of agents to goals.
We ﬁnd that ALIGN self achieves the best performance in both cooperative tasks, while ALIGN team and ALIGN adv are the best strategiesin Physical deception and Predator-prey respectively (Figure 5).
Upon a qualitative evaluation of
Cooperative navigation , we observe that agents with alignment are able to predict which goals willbe covered by their collaborators and move towards their allocated one.
Without alignment, agentsmove towards the same goal.
Alignment helps agents generalize to new partners.
Another core challenge in multi-agent collaboration is zero-shot coordination, where agents are tested to collaborate with new partners theyhaven’t been trained with.
We study whether alignment enables better zero-shot coordination.
Newpartners are sampled from other training runs with different seeds and the team is evaluated using thesame metrics as before.
We observe that ALIGN strategies enable better performance than SPARSEon average, and one of ALIGN self,team,adv performs the best in Heterogenous navigation , Physicaldeception and Keep away . (Figure 6).
These results suggest that alignment results in better zero-shotcoordination with new partners sampled from separate training runs.
Accuracy of the dynamics model affects alignment.
We train agents on noisy dynamics models byadding Gaussian noise to the dynamics model outputs.
We observe that the task performance dropsas the dynamics model degrades.
This suggests that agents do rely on accurate predictions of otheragents’ expectations in order for alignment to be effective.
Details can be found in the Appendix.
Discussion
Limitations.
Although our results indicate that aligned agents generalize better to new partners, thisﬁnding is limited to the multi-agent particle and Google Research football environments, which havea smaller action space than most ecologically valid scenarios.
Future work should also explore theuse of alignment in massive collaboration settings with hundreds of agents. Drawing on Zoologyresearch Couzin (2007) alignment should scale in such settings if agents align their behaviors only totheir nearest neighbors and not to the entire cohort.
Conclusion.
Inspired by the self-organizing principle in Zoology, we introduce alignment, a simple,task-agnostic, and self-supervised intrinsic reward for multi-agent systems.
Alignment rewards agentswhen they act predictably to their teammates and unpredictably to their adversaries.
Alignmentimproves multi-agent performance across six cooperative and competitive tasks in the multi-agentparticle and Google Research football environments, especially for decentralized training underpartial observability.
It also scales well, helps agents break symmetries, and generalize to newpartners.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Yes ] See Section 5 for the results mentioned in the abstract.
(b) Did you describe the limitations of your work?
[Yes] See Section 6 for a discussion on
the limitations.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have read the ethics review guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
N/A](b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the
Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See the Experiments 5 and Appendix sections.
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See the Experiments 5 section.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] See the Experiments 5 and
Appendix sections.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See the Alignment 4and Experiments 5 sections.
[Table]
(b) Did you mention the license of the assets?
[Yes] See the Appendix
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
See the Appendix
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
We did not collect any data.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
We did not collect any data.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
We did not crowdsource any data.
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
There were no human subjects experiments
[Table]
in our work.
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
No human subjects were hired.
[Table]
6.1 Code
[Table]
We upload our code for training and evaluating agents with and without alignment in both themulti-agent particle and Google Research football environments here: https://anonymous.4open.science/r/alignment-9F11/ .