Original: Beyond Rewards: a Hierarchical Perspective on Ofﬂine Multiagent Behavioral Analysis
Revised: Beyond Rewards: a Hierarchical Perspective on Ofﬂine Multiagent Behavioral Analysis

Original: Abstract 
Revised: Abstract [Note: revisions shown in blue.]

Original: Each year, expert-level performance is attained in increasingly-complex multiagentdomains, notable examples including Go, Poker, and StarCraft II.
Revised: Each year, expert-level performance is attainedin increasingly-complex multiagent domains, notable examples including Go, Poker, and StarCraft II.

Original: This rapidprogression is accompanied by a commensurate need to better understand howsuch agents attain this performance, to enable their safe deployment, identifylimitations, and reveal potential means of improving them.
Revised: This rapid progression is accompanied by a commensurateneed to better understand how such agents attain this performance, to enable theirsafe deployment, identify limitations, and reveal potential means of improving them.

Original: In this paper we takea step back from performance-focused multiagent learning, and instead turn ourattention towards agent behavior analysis.
Revised: In this paper we take a step back from performance-focused multiagent learning,and instead turn our attention towards agent behavior analysis.

Original: We introduce a model-agnostic methodfor discovery of behavior clusters in multiagent domains, using variational inferenceto learn a hierarchy of behaviors at the joint and local agent levels.
Revised: We introduce amodel-agnostic method for discovery of behavior clusters in multiagent domains,using variational inference to learn a hierarchy of behaviors at the joint and localagent levels.

Original: Our frameworkmakes no assumption about agents’ underlying learning algorithms, does notrequire access to their latent states or  policies, and is  trained  using  only ofﬂineobservational data.
Revised: Our framework makes no assumption about agents’ underlyinglearning algorithms, does not require access to their latent states or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies, (cid:58)(cid:58)(cid:58) and(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58) ofﬂine observational data.

Original: We illustrate the effectiveness of our method for enabling thecoupled understanding of behaviors at the joint and local agent level, detection ofbehavior changepoints throughout training, discovery of core behavioral concepts,demonstrate the approach’s scalability to a high-dimensional multiagent MuJoCocontrol domain,  and  also illustrate that the approach can disentangle previouslytrained policies in OpenAI’s hide-and-seek domain.
Revised: We illustrate the effectiveness ofour method for enabling the coupled understanding of behaviors at the joint andlocal agent level, detection of behavior changepoints throughout training, discoveryof core behavioral concepts (cid:58) , demonstrate the approach’s scalability to a highdimensional multiagent MuJoCo control domain, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) show (cid:58)(cid:58)(cid:58) [Table]

Original:  Introduction
Revised:  Introduction

Original: Multiagent approaches have driven numerous advances in artiﬁcial intelligence research, with seminalexamples including TD-gammon [1], DeepBlue [2], AlphaGo [3], AlphaZero [4], Libratus [5],
Revised: Multiagent approaches have driven numerous advances in artiﬁcial intelligence research, with seminalexamples including TD-gammon [1], DeepBlue [2], AlphaGo [3], AlphaZero [4], Libratus [5],

Original: AlphaStar [6] OpenAI Five [7]  and Pluribus [8].
Revised: AlphaStar [6], OpenAI Five [7] [Table] and Pluribus [8].

Original: During training, many of these approaches seek topush the performance of agents as measured by a reward signal  or derivatives thereof.
Revised: During training, many of these approaches seek topush the performance of agents as measured by a reward signal [Table] or derivatives thereof.

Original: Despite this, post-hoc methods that seek to understand agent interactions often use less reward-centrictechniques.
Revised: Despite this, post-hoc methods that seek to understand agent interactions often use less reward-centrictechniques.

Original: Instead, insights are drawn from behavioral analysis to identify unique or interestingagent strategies.
Revised: Instead, insights are drawn from behavioral analysis to identify unique or interestingagent strategies.

Original: Examples include clustering-based analysis of neuron activations and trajectoriesin capture-the-ﬂag [9], inspection of trajectories in a hide-and-seek domain to detect interestingbehaviors such as agents that learn to exploit the underlying physics engine [10], monitoring ofstatistics such as pass ranges and frequencies in humanoid football [11], and analysis of AlphaZero’sacquisition of chess knowledge [12].
Revised: Examples include clustering-based analysis of neuron activations and trajectoriesin capture-the-ﬂag [9], inspection of trajectories in a hide-and-seek domain to detect interestingbehaviors such as agents that learn to exploit the underlying physics engine [10], monitoring ofstatistics such as pass ranges and frequencies in humanoid football [11], and analysis of AlphaZero’sacquisition of chess knowledge [12].

Original: Crucially, such insights are often drawn via manual analysisand detection of behavioral clusters, or use of statistics associated with certain behaviors as deﬁnedby humans experts (e.g., various skills and relevant metrics in humanoid football).
Revised: Crucially, such insights are often drawn via manual analysisand detection of behavioral clusters, or use of statistics associated with certain behaviors as deﬁnedby humans experts (e.g., various skills and relevant metrics in humanoid football).

Original: As evident above, understanding emergent multiagent behaviors is enriched by techniques beyondpure reward-based analysis, as behavioral signiﬁers are not always discernible via rewards.
Revised: As evident above, understanding emergent multiagent behaviors is enriched by techniques beyondpure reward-based analysis, as behavioral signiﬁers are not always discernible via rewards.

Original: Figure 1aprovides intuition on this notion, illustrating returns (sum of rewards) throughout training for amultiagent hill-climbing domain (later described in detail).
Revised: Figure 1aprovides intuition on this notion, illustrating returns (sum of rewards) throughout training for amultiagent hill-climbing domain (later described in detail).

Original: We highlight two independent trainingtrials (A and B) with similar ﬁnal returns.
Revised: We highlight two independent trainingtrials (A and B) with similar ﬁnal returns.

Original: Despite this, comparing the trajectories generated via the
Revised: Despite this, comparing the trajectories generated via the

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: d
Revised: d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: agents’ deterministic policies at the end of each trial’s training (Fig. 1b) reveals entirely differentbehaviors.
Revised: agents’ deterministic policies at the end of each trial’s training (Fig. 1b) reveals entirely differentbehaviors.

Original: Analogous examples are evident in the above works (e.g., Fig.
Revised: Analogous examples are evident in the above works (e.g., Fig.

Original: 1 of Baker et al.
Revised: 1 of Baker et al.

Original: [10], wheresubstantial behavior changes occur in multiagent hide-and-seek despite a smooth reward curve).
Revised: [10], wheresubstantial behavior changes occur in multiagent hide-and-seek despite a smooth reward curve).

Original: This paper formalizes the problem of ofﬂine multiagent behavior analysis.
Revised: This paper formalizes the problem of ofﬂine multiagent behavior analysis.

Original: Our proposed algorithm,
Revised: Our proposed algorithm,

Original: Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), learns a hierarchical latent space thatsimultaneously reveals behavior clusters at the joint level (i.e., interactions between agents) and locallevel (i.e., behaviors of individual agents).
Revised: Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), learns a hierarchical latent space thatsimultaneously reveals behavior clusters at the joint level (i [Table] e., interactions between agents) and locallevel (i

Revised: [Table]

Revised: e., behaviors of individual agents).

Original: Our method is agnostic of the underlying algorithm usedto generate agents’ behaviors, requires no access or control of the underlying environment, does notassume availability of a reward signal, and does not require access to  agents’ models or internal states.
Revised: Our method is agnostic of the underlying algorithm used [Table] to generate agents’ behaviors [Table]

Revised: agents’ models

Original:  Our  experiments  investigate the structure of the learned behavior space, which goes beyond priorworks on latent-clustering by identifying relationships between individual agent and joint behaviors.
Revised: [Table] Our [Table] experiments (cid:58) investigate the structure of the learned behavior space, which goes beyond priorworks on latent-clustering by identifying relationships between individual agent and joint behaviors.

Original: We illustrate that clusters identiﬁed by MOHBA are useful for highlighting similarities and differencesin behaviors throughout training.
Revised: We illustrate that clusters identiﬁed by MOHBA are useful for highlighting similarities and differencesin behaviors throughout training.

Original: We also quantitatively analyze the completeness of discoveredbehavior clusters by adopting a modiﬁed version of the concept-discovery framework of Yeh et al.[13] to identify interesting behavior concepts in our multiagent setting.
Revised: We also quantitatively analyze the completeness of discoveredbehavior clusters by adopting a modiﬁed version of the concept-discovery framework of Yeh et al.[13] to identify interesting behavior concepts in our multiagent setting.

Original:  We  then  test the scalabilityof our approach by using it for behavioral analysis of  several high-dimensional multiagent MuJoCoenvironments [14]. 
Revised: (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58) then (cid:58) test the scalabilityof our approach by using it for behavioral analysis of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) several high-dimensional multiagent MuJoCo(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments [14]. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Finally,  we  evaluate  the  approach  on  the  open-sourced  OpenAI hide-and-seekpolicy checkpoints [10], conﬁrming that the behavioral clusters detected by MOHBA closelymatchthose of the human-expert annotated labels provided in their policy checkpoints.
Revised: Finally, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) evaluate (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) approach (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) open-sourced (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) OpenAI (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hide-and-seek(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] closely

Original:   Related Work
Revised: [Table] [Table]  Related Work

Original: Signiﬁcant research has been conducted in single-agent skill discovery, which seeks to learn reusablepolicies useful for downstream tasks [15–28].
Revised: Signiﬁcant research has been conducted in single-agent skill discovery, which seeks to learn reusablepolicies useful for downstream tasks [15–28].

Original: Related approaches discover motor primitives to expresslonger-horizon policies [29], including use of ofﬂine reinforcement learning (RL) for learning usefulbehaviors [30].
Revised: Related approaches discover motor primitives toexpress longer-horizon policies [29], including use of ofﬂine reinforcement learning (RL) for learning useful behaviors [30].

Original: Option discovery methods  learn temporally-abstracted actions (i.e., options [31]),chained together to form cohesive skills [32–37].
Revised: Option discovery methods (cid:58)(cid:58)(cid:58)(cid:58) learn temporally-abstracted actions (i.e., options [31]), chained together to form cohesive skills [32–37].

Original: In contrast to our work, these approaches focuson maximizing performance in single-agent settings.
Revised: In contrast to our work, these approachesfocus on maximizing performance in single-agent settings [Table] There [Table]

Original: There also exists a related line of  work forlearning diverse  policies  in  RL  settings  [38–46];  despite  their  focus  on  policy  diversity,  several ofthese approaches make much stronger assumptions than ours (e.g., access to the policies generatingagent trajectories, or unique identiﬁers of the policy that generated each trajectory).
Revised: also [Table] of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) work (cid:58)(cid:58)(cid:58) for(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diverse (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) RL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) settings (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [38–46]; (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) despite (cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) focus (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diversity, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) several (cid:58)(cid:58) of(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original:  By  contrast, ourwork assumes  no  access  to  any  of  the  underlying raw policies, or even labels of which policies agenttrajectories were obtained from.
Revised: [Table] identiﬁers (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generated (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory). (cid:58)(cid:58)(cid:58) By (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) contrast, (cid:58)(cid:58)(cid:58)(cid:58) our(cid:58)(cid:58)(cid:58)(cid:58) work (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) assumes (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) access (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) any (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) underlying (cid:58)(cid:58)(cid:58) [Table]

Original:  Recent works have also focused on hierarchical skill learning inmultiagent reinforcement learning (MARL).
Revised: [Table] from. [Table] Recent works have also focused on hierarchical skill learning inmultiagent reinforcement learning (MARL).

Original: Lee et al.
Revised: Lee et al.

Original: [47] use the mutual information maximizationobjective introduced in Eysenbach et al.
Revised: [47] use the mutual information maximizationobjective introduced in Eysenbach et al.

Original: [16] to learn multiagent policies.
Revised: [16] to learn multiagent policies.

Original: Yang et al.
Revised: Yang et al.

Original: [48] use abi-level policy to learn agent skills: a high-level policy ﬁrst generates latent vectors for each agent,
Revised: [48] use abi-level policy to learn agent skills: a high-level policy ﬁrst generates latent vectors for each agent,

Original: A c t i o n s
Revised: A c t i o n s

Original: S t a t e s
Revised: S t a t e s

Original: P o l i
Revised: P o l i

Original: c i e s
Revised: c i e s

Original: L o c a l L a t e n t
Revised: L o c a l L a t e n t

Original: J o i n t L a t e n t (a) 
Revised: J o i n t L a t e n t (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Joint KL Term (Eq. 7)
Revised: Joint KL Term (Eq. 7)

Original: Local KL Term (Eq. 6)
Revised: Local KL Term (Eq. 6)

Original: Reconstruction Term (Eq. 5)
Revised: Reconstruction Term (Eq. 5)

Original: Joint prior
Revised: Joint prior

Original: Joint encoder
Revised: Joint encoder

Original: Local encoders
Revised: Local encoders

Original: Policies
Revised: Policies

Original: Offline Data 
Revised: Offline Data Local priors (

Revised: [Figure]

Original: Local priors (b) and conditions behaviors perform distinct ‘roles’ are learned for agents to enable decomposition investigateuse of  consistent latent cognition variables in agent neighborhoods to induce increased cooperation.
Revised: [Figure] Mao et al. [Figure] 50] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) investigate(cid:58)(cid:58)(cid:58) use (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consistent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original:  Many MARL approaches have noted the emergence of interesting behaviors in multiagent systems inspeciﬁc domains of interest [51–55].
Revised: [Figure] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] 73 Many MARL approaches have noted the emergence of interesting behaviors in multiagent systems inspeciﬁc domains of interest [51–55].

Original: In contrast to our focus, these approaches use RL to maximizeagent performance, rather than understand arbitrary behaviors via ofﬂine analysis.
Revised: In contrast to our focus, these approaches use RL to maximizeagent performance, rather than understand arbitrary behaviors via ofﬂine analysis.

Original: RL interpretability methods primarily focus on single-agent settings and either modify the RL algorithm itself to increase transparency, or conduct post-hoc explainability [56].
Revised: RL interpretability methods primarily focus on single-agent settings and either modify the RL algorithm itself to increase transparency [Table] or conduct post-hoc explainability [56].

Original: These approachesrepresent agent policies as programming languages [57], extract visual summaries of behaviors using‘interestingness’ statistics such as uncertainty in selected actions or the value of state-transitions [58],or combine agent neuron activations with gradient information to construct behavioral embeddings [59].
Revised: These approachesrepresent agent policies as programming languages [57], extract visual summaries of behaviors using‘interestingness’ statistics such as uncertainty in selected actions or the value of state-transitions [58],or combine agent neuron activations with gradient information to construct behavioral embeddings [59].

Original: Behavioral clusters in our work share similarities with concept-based explanationapproaches in non-RL domains.
Revised: Behavioral clusters in our work share similarities with concept-based explanationapproaches in non-RL domains.

Original: Detecting ‘concepts’ in pre-trained models has been explored invision [60–62], discrete games [12], and language [63, 64].
Revised: Detecting ‘concepts’ in pre-trained models has been explored invision [60–62], discrete games [12], and language [63, 64].

Original: In vision, clustering-based approachesdescribe discovered concepts using examples [60] or use generative modeling to create new datato describe concepts [62].
Revised: In vision, clustering-based approachesdescribe discovered concepts using examples [60] or use generative modeling to create new datato describe concepts [62].

Original: Ghorbani et al.
Revised: Ghorbani et al.

Original: [60] uses a vision-speciﬁc method (i.e., superpixels) tosub-divide input before conducing clustering; while sub-division is less natural in RL, our LSTM and
Revised: [60] uses a vision-speciﬁc method (i.e., superpixels) tosub-divide input before conducing clustering; while sub-division is less natural in RL, our LSTM and

Original: VAE baselines serve as an RL-adopted counterpart of such works.
Revised: VAE baselines serve as an RL-adopted counterpart of such works.

Original: Works using latent-clustering and analysis are also related to ours.
Revised: Works using latent-clustering and analysis are also related to ours.

Original: These include approaches usingmulti-level variational autoencoders [65–68] to learn compositional latent spaces, although not indecision-making domains such as ours.
Revised: These include approaches usingmulti-level variational autoencoders [65–68] to learn compositional latent spaces, although not indecision-making domains such as ours.

Original: Hierarchical latent approaches have been used in single-agent
Revised: Hierarchical latent approaches have been used in single-agent

Original: RL [69, 70].
Revised: RL [69, 70].

Original: Behavior analysis has also been conducted by embedding agent neuron activations intoa low-dimensional space, using a (non-hierarchical) variational approach [9].
Revised: Behavior analysis has also been conducted by embedding agent neuron activations intoa low-dimensional space, using a (non-hierarchical) variational approach [9].

Original: The representationpower of agents’ internal states has also been gauged by predicting future events [11].
Revised: The representationpower of agents’ internal states has also been gauged by predicting future events [11].

Original: Overall, thekey difference between the above works and ours is that our method combined hierarchical learningwith behavior analysis and applies it to the multiagent setting.
Revised: Overall, thekey difference between the above works and ours is that our method combined hierarchical learningwith behavior analysis and applies it to the multiagent setting.

Original:  Ofﬂine Analysis of Multiagent Behaviors
Revised:  Ofﬂine Analysis of Multiagent Behaviors

Original: This section introduces the ofﬂine multiagent behavioral analysis problem and our proposed algorithm.
Revised: This section introduces the ofﬂine multiagent behavioral analysis problem and our proposed algorithm.

Original: Preliminaries.
Revised: Preliminaries.

Original: We ﬁrst formalize the problem of ofﬂine multiagent behavioral analysis.
Revised: We ﬁrst formalize the problem of ofﬂine multiagent behavioral analysis.

Original: Considera rewardless multiagent Markov Decision Process (MA-MDP), deﬁned by tuple ( I , S , A , T ) , where
Revised: Considera rewardless multiagent Markov Decision Process (MA-MDP), deﬁned by tuple ( I , S , A , T ) , where

Original: I = { 1 , ..., N } is the set of N agents, S is the state space,
Revised: I = { 1 , ..., N } is the set of N agents, S is the state space,

Original: A is the action space, and T denotes thestate transition probability function.
Revised: A is the action space, and T denotes thestate transition probability function.

Original: By not relying on the presence of rewards, behaviors generatedeven without reliance on a reward function (e.g., human interactions, or agents using curiosity-basedexploration) can be considered.
Revised: By not relying on the presence of rewards, behaviors generatedeven without reliance on a reward function (e.g., human interactions, or agents using curiosity-basedexploration) can be considered.

Original: We use the term ‘local’ for elements associated with individual agents,and ‘joint’ for those associated with the entire system.
Revised: We use the term ‘local’ for elements associated with individual agents,and ‘joint’ for those associated with the entire system.

Original: At each timestep t ∈ { 0 , . . .
Revised: At each timestep t ∈ { 0 , . . .

Original: , T − 1 } , the agentsexecute joint action a t ∈ A in state s t using joint policy π ( a t | s t ) , causing the state to transition tos t +1 with probability p ( s t +1 | s t , a t ) =
Revised: , T − 1 } , the agentsexecute joint action a t ∈ A in state s t using joint policy π ( a t | s t ) , causing the state to transition tos t +1 with probability p ( s t +1 | s t , a t ) =

Original: T ( s t +1 , a t , s t ) 
Revised: T ( s t +1 , a t , s t ) .

Original: As standard in multiagent frameworks [71], we assume the joint action space factorizes as A = ×
Revised: As standard in multiagent frameworks [71], we assume the joint action space factorizes as A = ×

Original: i
Revised: i

Original: A
Revised: A

Original: i , such that a t = ( a 1 t , . .
Revised: i , such that a t = ( a 1 t , . .

Original: . , a Nt ) , where i ∈ Iand a it ∈
Revised: . , a Nt ) , where i ∈ Iand a it ∈

Original: A i .
Revised: A i .

Original: Similarly, π i ( a it | s t ) is the local policy for agent i , and π = (cid:81)
Revised: Similarly, π i ( a it | s t ) is the local policy for agent i , and π = (cid:81)

Original: i π
Revised: i π

Original: i is the joint policy.
Revised: i is the joint policy.

Original: Let τ = (cid:0) s 0 , ( a i 0 ) i ∈I , . . .
Revised: Let τ = (cid:0) s 0 , ( a i 0 ) i ∈I , . . .

Original: , ( a iT − 1 ) i ∈I , s T (cid:1) denote a trajectory induced by this process, and D ={ τ 1 , . . .
Revised: , ( a iT − 1 ) i ∈I , s T (cid:1) denote a trajectory induced by this process, and D ={ τ 1 , . . .

Original: , τ K } denote a dataset of K such trajectories.
Revised: , τ K } denote a dataset of K such trajectories.

Original: This dataset may consist of trajectories frommultiple training runs, including variations over agent algorithms, hyperparameters, random seeds,or other factors inﬂuencing emergent behaviors.
Revised: This dataset may consist of trajectories frommultiple training runs, including variations over agent algorithms, hyperparameters, random seeds,or other factors inﬂuencing emergent behaviors.

Original: Given dataset D , the ofﬂine multiagent analysisproblem seeks to uncover potential clusters of agent behaviors.
Revised: Given dataset D , the ofﬂine multiagent analysisproblem seeks to uncover potential clusters of agent behaviors.

Original: Approach.
Revised: Approach.

Original: Our approach, called the Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA),uses ofﬂine trajectory data to discover behaviors exhibited by agents at the local and joint level.
Revised: Our approach, called the Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA),uses ofﬂine trajectory data to discover behaviors exhibited by agents at the local and joint level.

Original: We ﬁrst use Fig.
Revised: We ﬁrst use Fig.

Original: 2a to build intuition before discussing technical details.
Revised: 2a to build intuition before discussing technical details.

Original: Let the agent interactionsexhibited in a trajectory τ be encoded by a latent variable, z ω ∈ R D ω , capturing their joint behavior.
Revised: Let the agent interactionsexhibited in a trajectory τ be encoded by a latent variable, z ω ∈ R D ω , capturing their joint behavior.

Original: For example, z ω may encode (at a high level) whether agents were cooperating or competing ina given trajectory; conditioned on joint signal z ω , each agent then exhibits its own local behavior.
Revised: For example, z ω may encode (at a high level) whether agents were cooperating or competing ina given trajectory; conditioned on joint signal z ω , each agent then exhibits its own local behavior.

Original: Let local latent vectors z α = ( z 1 α , . . .
Revised: Let local latent vectors z α = ( z 1 α , . . .

Original: , z Nα ) encode individual agent behaviors, where z iα ∈ R D iα .
Revised: , z Nα ) encode individual agent behaviors, where z iα ∈ R D iα .

Original: Conditioned on the local behavior vector z iα , each agent then executes actions using a behaviorconditioned policy π i ( a i | s, z iα ) .
Revised: Conditioned on the local behavior vector z iα , each agent then executes actions using a behaviorconditioned policy π i ( a i | s, z iα ) .

Original: Given trajectory dataset D , we seek to learn the latent-conditionedpolicies and distributions over latent vectors, such that we can reconstruct any behaviors exhibited bythe agents in D .
Revised: Given trajectory dataset D , we seek to learn the latent-conditionedpolicies and distributions over latent vectors, such that we can reconstruct any behaviors exhibited bythe agents in D .

Original: Thus, latent vectors z ω and z α will encode the agents’ behavioral spaces and, ideally,identify behavioral clusters in the dataset.
Revised: Thus, latent vectors z ω and z α will encode the agents’ behavioral spaces and, ideally,identify behavioral clusters in the dataset.

Original: Given this framework, the joint policy is decomposed,
Revised: Given this framework, the joint policy is decomposed,

Original: [Equation]
Revised: [Equation]

Original: where in (1) we have assumed that each agent’s latent-conditioned policy is conditionally-independentof the high-level latent behavior z ω given its low-level latent z iα  (see Appendix A.1.1  for  discussion).
Revised: where in (1) we have assumed that each agent’s latent-conditioned policy is conditionally-independentof the high-level latent behavior z ω given its low-level latent z iα (cid:58)(cid:58)(cid:58) (see Appendix A.1.1 (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) discussion).

Original: Next, given initial state distribution p ( s 0 ) and latent behavior spaces, the probability of a trajectory τunder joint policy π ( · ) is as follows:
Revised: Next, given initial state distribution p ( s 0 ) and latent behavior spaces, the probability of a trajectory τunder joint policy π ( · ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: We seek to learn the distributions over variables
Revised: We seek to learn the distributions over variables

Original: z α and z ω , alongside the latent-conditioned policiesπ i ( a i | s t , z iα ) , which maximize trajectory probabilities (4).
Revised: z α and z ω , alongside the latent-conditioned policiesπ i ( a i | s t , z iα ) , which maximize trajectory probabilities (4).

Original: In Appendix A.1, we derive the followingvariational lower bound, enabling approximation of these components using parametric models:
Revised: In Appendix A.1, we derive the followingvariational lower bound, enabling approximation of these components using parametric models:

Original: [Equation]
Revised: [Equation]

Original: where q φ ( z iα | τ ) and p θ ( z iα | z ω ) are, respectively, learned encoder (posterior) and prior distributionsover the local behavior latents z iα ; likewise, q φ ( z ω | τ ) and p θ ( z ω ) are, respectively, learned encoderand prior distributions over the joint behavior latent z ω ; β is a KL-weighting term as in β -VAEs [72].
Revised: where q φ ( z iα | τ ) and p θ ( z iα | z ω ) are, respectively, learned encoder (posterior) and prior distributionsover the local behavior latents z iα ; likewise, q φ ( z ω | τ ) and p θ ( z ω ) are, respectively, learned encoderand prior distributions over the joint behavior latent z ω ; β is a KL-weighting term as in β -VAEs [72].

Original: Figure 2b illustrates MOHBA’s model architecture, which is informed by the three bound components (5) to (7).
Revised: Figure 2b illustrates MOHBA’s model architecture, which is informed by the three bound components (5) to (7).

Original: During training, each trajectory τ ∼ D is simultaneously passed through the jointand local encoders, which respectively produce parameters for distributions over z ω and z α (e.g.,parameters of Gaussian distributions).
Revised: During training, each trajectory τ ∼ D is simultaneously passed through the jointand local encoders, which respectively produce parameters for distributions over z ω and z α (e.g.,parameters of Gaussian distributions).

Original: Samples of low-level latent vectors z α are passed to the
Revised: Samples of low-level latent vectors z α are passed to the

Original: [Figure]
Revised: [Figure]

Original: reconstructed agent policies π iθ ( a it | s t , z iα ) , which are trained via the reconstruction component (5).
Revised: reconstructed agent policies π iθ ( a it | s t , z iα ) , which are trained via the reconstruction component (5).

Original: The local KL-divergence component (6) induces the local encoder distribution (which is conditioneddirectly on τ ) to be similar to the local prior distribution (which is conditioned only on samplesz ω ), thus enabling meaningful correlations between the encoded local and joint latent space, aslater shown.
Revised: The local KL-divergence component (6) induces the local encoder distribution (which is conditioneddirectly on τ ) to be similar to the local prior distribution (which is conditioned only on samplesz ω ), thus enabling meaningful correlations between the encoded local and joint latent space, aslater shown.

Original: Finally, the joint KL-divergence component (7) is akin to that in a standard variationalautoencoder [73, 74].
Revised: Finally, the joint KL-divergence component (7) is akin to that in a standard variationalautoencoder [73, 74].

Original: Overall, MOHBA enables learning of a hierarchical behavioral space (at thejoint and local agent levels, z ω and z α , respectively) that exposes interesting behavioral clusters.
Revised: Overall, MOHBA enables learning of a hierarchical behavioral space (at thejoint and local agent levels, z ω and z α , respectively) that exposes interesting behavioral clusters.

Original:  Experiments
Revised:  Experiments

Original: Our experiments showcase a variety of use-cases for MOHBA, which enrich the typical reward-centricperformance monitoring used during training.
Revised: Our experiments showcase a variety of use-cases for MOHBA, which enrich the typical reward-centricperformance monitoring used during training.

Original: Data generation, training, networks, computationresources, and hyperparameter details are provided in Appendix A.2 of the supplementary materials.
Revised: Data generation, training, networks, computationresources, and hyperparameter details are provided in Appendix A.2 of the supplementary materials.

Original: MARL data generation.
Revised: MARL data generation.

Original: Multiagent trajectory data is generated for each considered domain viathe Acme RL library [75], using the TD3 algorithm [76] in a decentralized fashion, with datasetmanagement handled using RLDS [77].
Revised: Multiagent trajectory data is generated for each considered domain viathe Acme RL library [75], using the TD3 algorithm [76] in a decentralized fashion, with datasetmanagement handled using RLDS [77].

Original: Trajectories are collected at constant intervals throughouttraining, which also enables analysis of behavioral emergence.
Revised: Trajectories are collected at constant intervals throughouttraining, which also enables analysis of behavioral emergence.

Original: We conduct a wide sweep over randomseeds for data generation, yielding a diverse set of trajectories to analyze using MOHBA.
Revised: We conduct a wide sweep over randomseeds for data generation, yielding a diverse set of trajectories to analyze using MOHBA.

Original: MOHBA setup.
Revised: MOHBA setup.

Original: To analyze the above data using MOHBA, we use a GMM (Gaussian Mixture
Revised: To analyze the above data using MOHBA, we use a GMM (Gaussian Mixture

Original: Model) for the joint prior, a bidirectional LSTM (long short-term memory network) with GMM headfor the joint encoder, an MLP (multi-layer perceptron) with Gaussian head for the local priors, abidirectional LSTM with Gaussian head for the local encoder, and an MLP for reconstructed policies.
Revised: Model) for the joint prior, a bidirectional LSTM (long short-term memory network) with GMM headfor the joint encoder, an MLP (multi-layer perceptron) with Gaussian head for the local priors, abidirectional LSTM with Gaussian head for the local encoder, and an MLP for reconstructed policies.

Original: GMMs are used for the joint prior and encoder as they produce discernible joint behavior clusters [78],whereas the conditioning of the local prior on z ω yields such clusters at the local level with a standard
Revised: GMMs are used for the joint prior and encoder as they produce discernible joint behavior clusters [78],whereas the conditioning of the local prior on z ω yields such clusters at the local level with a standard

Original: Gaussian head.
Revised: Gaussian head.

Original: We use parameter-sharing across local priors, local encoders, and reconstructedpolicies, as common in multiagent setups [79], with a unique one-hot vector identiﬁer appended toagent-speciﬁc network inputs to enable heterogeneity in model outputs.
Revised: We use parameter-sharing across local priors, local encoders, and reconstructedpolicies, as common in multiagent setups [79], with a unique one-hot vector identiﬁer appended toagent-speciﬁc network inputs to enable heterogeneity in model outputs.

Original: We conduct a wide sweep of
Revised: We conduct a wide sweep of

Original: MOHBA hypereparameters, including the latent space dimensionality ( 4 and 8 -dimensional), MLPhidden sizes ( 64 and 128 units), LSTM hidden sizes ( 64 and 128 units), Adam optimizer [80] learningrates ( 0 . 001 and 0 . 0001 ), with 3 seeds per parameter set (see appendix for full details).
Revised: MOHBA hypereparameters, including the latent space dimensionality ( 4 and 8 -dimensional), MLPhidden sizes ( 64 and 128 units), LSTM hidden sizes ( 64 and 128 units), Adam optimizer [80] learningrates ( 0 . 001 and 0 . 0001 ), with 3 seeds per parameter set (see appendix for full details).

Original: Independent analysis of joint ( z ω ) and local ( z α )  behaviors.
Revised: Independent analysis of joint ( z ω ) and local ( z α ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors.

Original: We here analyze the hierarchicallatent structure learned by MOHBA.
Revised: We here analyze the hierarchicallatent structure learned by MOHBA.

Original: Speciﬁcally, we highlight differences in agent behaviors asidentiﬁed at the joint and local levels, respectively, by z ω and z α .
Revised: Speciﬁcally, we highlight differences in agent behaviors asidentiﬁed at the joint and local levels, respectively, by z ω and z α .

Original: We ﬁrst consider trajectoriesgenerated in a 3-agent hill-climbing domain (our earlier example in Fig. 1b).
Revised: We ﬁrst consider trajectoriesgenerated in a 3-agent hill-climbing domain (our earlier example in Fig. 1b).

Original: States and actionscorrespond, respectively, to 2D positions ( x, y ) and forces (∆ x, ∆ y ) imparted by each agent formovement.
Revised: States and actionscorrespond, respectively, to 2D positions ( x, y ) and forces (∆ x, ∆ y ) imparted by each agent formovement.

Original: Agents spawn at the origin and are each rewarded for climbing any of three hills within
Revised: Agents spawn at the origin and are each rewarded for climbing any of three hills within

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: the domain in episodes of 50 timesteps (with rewards at each position indicated in Fig. 1b).
Revised: the domain in episodes of 50 timesteps (with rewards at each position indicated in Fig. 1b).

Original: Wegenerate a dataset using 50 independent MARL training trials, each conducted for 1 e 5 environmentsteps; trajectories are saved every 200 steps, yielding 1 .
Revised: Wegenerate a dataset using 50 independent MARL training trials, each conducted for 1 e 5 environmentsteps; trajectories are saved every 200 steps, yielding 1 .

Original: 25 e 6 frames of data ( 25000 trajectories).
Revised: 25 e 6 frames of data ( 25000 trajectories).

Original: Figure 3a highlights several example clusters of joint and local behaviors identiﬁed by MOHBA,using samples of z ω and z α from the joint and local encoders.
Revised: Figure 3a highlights several example clusters of joint and local behaviors identiﬁed by MOHBA,using samples of z ω and z α from the joint and local encoders.

Original: We use Euclidean distances in theoriginal latent spaces to identify nearby vectors, then visualize their 2D projection using principalcomponent analysis.
Revised: We use Euclidean distances in theoriginal latent spaces to identify nearby vectors, then visualize their 2D projection using principalcomponent analysis.

Original: For each cluster, we visualize all associated trajectories in Fig.
Revised: For each cluster, we visualize all associated trajectories in Fig.

Original: 3b, thus enablinganalysis of behaviors captured in latent space.
Revised: 3b, thus enablinganalysis of behaviors captured in latent space.

Original: Clusters A, B, and C correspond to examples of jointbehaviors ( z ω ).
Revised: Clusters A, B, and C correspond to examples of jointbehaviors ( z ω ).

Original: Cluster A contains trajectories early in training, where agents behave nearly randomlyand have not learned to converge to a particular rewarding hill in the domain.
Revised: Cluster A contains trajectories early in training, where agents behave nearly randomlyand have not learned to converge to a particular rewarding hill in the domain.

Original: In Cluster B, all threeagents have learned to navigate towards the top-left hill.
Revised: In Cluster B, all threeagents have learned to navigate towards the top-left hill.

Original: In Cluster C, agents 0 and 2 show preferencefor the bottom-left hill, whereas agent 1 prefers the right hill.
Revised: In Cluster C, agents 0 and 2 show preferencefor the bottom-left hill, whereas agent 1 prefers the right hill.

Original: Overall, the behavioral clusters at thejoint level capture the collective behaviors of agents in a meaningful manner.
Revised: Overall, the behavioral clusters at thejoint level capture the collective behaviors of agents in a meaningful manner.

Original: Next, we analyze individual agent clusters.
Revised: Next, we analyze individual agent clusters.

Original: Note that for all agents, three prominent clusters areapparent in their respective z α spaces, as each training trial in the data generation process can lead tovarious agent-wise hill preferences.
Revised: Note that for all agents, three prominent clusters areapparent in their respective z α spaces, as each training trial in the data generation process can lead tovarious agent-wise hill preferences.

Original: We compare two such clusters, D and E, for agent 1, observingin the trajectory plots that these corresponds to this agent preferring the top-left and bottom-left hills,respectively.
Revised: We compare two such clusters, D and E, for agent 1, observingin the trajectory plots that these corresponds to this agent preferring the top-left and bottom-left hills,respectively.

Original: Similarly, Cluster F corresponds to trajectories where agent 2 prefers the bottom-left hill.
Revised: Similarly, Cluster F corresponds to trajectories where agent 2 prefers the bottom-left hill.

Original: These results illustrate that the local latents reasonably disentangle each agent’s observed behaviors.[Table] [Table] C
Revised: These results illustrate that the local latents reasonably disentangle each agent’s observed behaviors.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] Coupled analysis of joint ( z ω ) and local ( z α )  behaviors.
Revised: Coupled analysis of joint ( z ω ) and local ( z α ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors.

Original: The above experiments independently analyzed the joint andlocal latent spaces.
Revised: The above experiments independently analyzed the joint andlocal latent spaces.

Original: We can also concurrently analyze them tobetter understand local agent contributions to joint behaviors.
Revised: We can also concurrently analyze them tobetter understand local agent contributions to joint behaviors.

Original: Consider a two-agent domain with close inter-agent coordination (visualized in the top of Fig. 4a), with state and actionsspaces similar to the hill domain and episodes consisting of50 timesteps.
Revised: Consider a two-agent domain with close inter-agent coordination (visualized in the top of Fig. 4a), with state and actionsspaces similar to the hill domain and episodes consisting of50 timesteps.

Original: Three regions are deﬁned in this domain: A and B (circular regions), and C (regionexterior to the circles).
Revised: Three regions are deﬁned in this domain: A and B (circular regions), and C (regionexterior to the circles).

Original: When an agent enters a given region, it ‘activates’ the corresponding strategyin Table 1, with agents receiving rewards at each timestep according to the joint strategy they haveactivated.
Revised: When an agent enters a given region, it ‘activates’ the corresponding strategyin Table 1, with agents receiving rewards at each timestep according to the joint strategy they haveactivated.

Original: For example, if both agents enter region A , they each receive a reward of 1 , whereas ifone agent enters A while the other is in exterior region C , neither receives a reward.
Revised: For example, if both agents enter region A , they each receive a reward of 1 , whereas ifone agent enters A while the other is in exterior region C , neither receives a reward.

Original: This domaininvolves a signiﬁcant degree of coordination as agents must discover the rewarding regions, whilereceiving a sparse reward signal until a valid combination of strategies is discovered.
Revised: This domaininvolves a signiﬁcant degree of coordination as agents must discover the rewarding regions, whilereceiving a sparse reward signal until a valid combination of strategies is discovered.

Original: There is also potential for miscoordination: navigating to region B is rewarding assuming the other agent navigatesto A , but yields 0 reward if the other agent instead navigates to B (potentially destabilizing training).
Revised: There is also potential for miscoordination: navigating to region B is rewarding assuming the other agent navigatesto A , but yields 0 reward if the other agent instead navigates to B (potentially destabilizing training).

Original: We run 50 independent MARL sweeps in this domain, yielding 1 e 6 data frames ( 20000 trajectories).
Revised: We run 50 independent MARL sweeps in this domain, yielding 1 e 6 data frames ( 20000 trajectories).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, MOHBA discovers three dominant joint behavior clusters z ω here.
Revised: 4, MOHBA discovers three dominant joint behavior clusters z ω here.

Original: In each of
Revised: In each of

Original: Figs. 4a to 4c, we highlight one of these clusters and its corresponding local behavior latents z α foreach agent.
Revised: Figs. 4a to 4c, we highlight one of these clusters and its corresponding local behavior latents z α foreach agent.

Original: MOHBA reveals that across the dataset, the agents have learned to cover all 3 optimaljoint behaviors in Table 1: ( A, A ) , ( A, B ) , and ( B, A ) .
Revised: MOHBA reveals that across the dataset, the agents have learned to cover all 3 optimaljoint behaviors in Table 1: ( A, A ) , ( A, B ) , and ( B, A ) .

Original: Moreover, despite each agent discovering twolocal behaviors ( A and B ), z ω highlights only the 3 observed joint behaviors (i.e., does not simplyhighlight 4 clusters consisting of the Cartesian product of individual agents’ behavior spaces).
Revised: Moreover, despite each agent discovering twolocal behaviors ( A and B ), z ω highlights only the 3 observed joint behaviors (i.e., does not simplyhighlight 4 clusters consisting of the Cartesian product of individual agents’ behavior spaces).

Original: Behavior emergence throughout MARL training.
Revised: Behavior emergence throughout MARL training.

Original: We next use MOHBA’s latent space to inspectbehavior emergence during MARL training.
Revised: We next use MOHBA’s latent space to inspectbehavior emergence during MARL training.

Original: Figure 4d visualizes the training progression of two
Revised: Figure 4d visualizes the training progression of two

Original: MARL runs used for data generation, tracking behavior changes throughout.
Revised: MARL runs used for data generation, tracking behavior changes throughout.

Original: In Training Run 1 (toppanel of Fig. 4d), the agents converge to and maintain a ﬁxed joint behavior throughout training.
Revised: In Training Run 1 (toppanel of Fig. 4d), the agents converge to and maintain a ﬁxed joint behavior throughout training.

Original: Bycontrast, Training Run 2 (bottom panel) has numerous behavior changepoints, where agents ﬂip backand forth between preferring one of two clusters in z ω .
Revised: Bycontrast, Training Run 2 (bottom panel) has numerous behavior changepoints, where agents ﬂip backand forth between preferring one of two clusters in z ω .

Original: Concurrently inspecting the z α space for
Revised: Concurrently inspecting the z α space for

Original: Training Run 2, we observe that agent 0 converges to and maintains a consistent behavior, whereasagent 1 changes its preference sporadically, also explaining the detected changes in joint behaviors.
Revised: Training Run 2, we observe that agent 0 converges to and maintains a consistent behavior, whereasagent 1 changes its preference sporadically, also explaining the detected changes in joint behaviors.

Original: Baseline comparisons.
Revised: Baseline comparisons.

Original: We next compare against baselines previously used for multiagent behavioral analysis in the literature [9, 11].
Revised: We next compare against baselines previously used for multiagent behavioral analysis in the literature [9, 11].

Original: An LSTM baseline conducts next-action prediction at eachstep in a trajectory τ , using an action-prediction loss (APL), deﬁned as the L 2 loss over predictedvs.
Revised: An LSTM baseline conducts next-action prediction at eachstep in a trajectory τ , using an action-prediction loss (APL), deﬁned as the L 2 loss over predictedvs.

Original: ground truth actions. 
Revised: ground truth actions. [Table]

Original: This baseline targets using the LSTM hidden states, rather than a learneddistribution over latent variables, for understanding and clustering agent behaviors (akin to theanalysis in [9, 11]).
Revised: This (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] in [Table]

Original:  A ﬂat-VAE baseline provides a non-hierarchical ablation of MOHBA that simplyfeeds the joint latent z ω (rather than z iα ) to reconstructed agent policies (i.e., π iθ ( a i | s, z ω ) ); we useour usual loss (with the local KL term (6) removed) to train this VAE.
Revised: [Table] A ﬂat-VAE baseline provides a non-hierarchical ablation of MOHBA that simply feedsthe joint latent z ω (rather than z iα ) to reconstructed agent policies (i [Table] e., π iθ ( a i | s, z ω ) ); we use ourusual loss (with the local KL term (6) removed) to train this VAE.

Original: We conduct a hyperparametersweep for the baselines (see Appendix A.2), reporting the best results averaged over 3 random seeds.
Revised: We conduct a hyperparametersweep for the baselines (see Appendix A.2), reporting the best results averaged over 3 random seeds

Original: [Table] 
Revised: [Table] [Table]

Original: Comparisons are conductedat the joint behavior level:for the LSTM, we use the ﬁnal hidden state as an encoding of each trajectory; forour method and the VAE, weuse z ω directly.
Revised: Comparisons are conductedat the joint behavior level:for the LSTM, we use the ﬁnal hidden state as an encoding of each trajectory; forour method and the VAE, weuse z ω directly.

Original: Table 2 compares the methods in the hillclimbing and coordination game domains.
Revised: Table 2 compares the methods in the hillclimbing and coordination game domains.

Original: Here we note the APL, which measures how well eachmethod reconstructs ground truth policies.
Revised: Here we note the APL, which measures how well eachmethod reconstructs ground truth policies.

Original: Additionally, we use K-means (sweeping over the # ofclusters) to identify behavior clusters for each method, then report the intra-cluster trajectory distance(ICTD), deﬁned as the average distance of all trajectories in a cluster from the mean trajectory insaid cluster (akin to intra-cluster point scatter, a common cluster analysis statistic [81]) .
Revised: Additionally, we use K-means (sweeping over the # ofclusters) to identify behavior clusters for each method, then report the intra-cluster trajectory distance(ICTD), deﬁned as the average distance of all trajectories in a cluster from the mean trajectory insaid cluster (akin to intra-cluster point scatter, a common cluster analysis statistic [81]) .

Original: Combined,these measures provide a proxy for evaluating the latent representations in terms of enabling accuratereconstructions (APL) while clustering similar trajectories together (ICTD).
Revised: Combined,these measures provide a proxy for evaluating the latent representations in terms of enabling accuratereconstructions (APL) while clustering similar trajectories together (ICTD).

Original: MOHBA signiﬁcantlyoutperforms the LSTM and VAE baselines in terms of APL, while also clustering similar trajectoriesat the z ω level in terms of ICTD.
Revised: MOHBA signiﬁcantlyoutperforms the LSTM and VAE baselines in terms of APL, while also clustering similar trajectoriesat the z ω level in terms of ICTD.

Original: We provide additional results including APL throughout training,full ICTD sweeps, and visualizations of the latent spaces for the baselines in Appendix A.3.2.
Revised: We provide additional results including APL throughout training,full ICTD sweeps, and visualizations of the latent spaces for the baselines in Appendix A.3.2.

Original: Behavior concept discovery. 
Revised: [Table] discovery. Next, [Table]

Original: Next, we test the representation power of the latent spaces learnedby MOHBA, and illustrate a means of discovering ‘behavior concepts’ in the latent space.
Revised: we [Table] the [Table]

Original:  We adoptthe completeness-aware  concept  explanations framework of Yeh et al.
Revised: space. We adopt 253 the completeness-aware (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) concept (cid:58) explanations framework of Yeh et al.

Original: [13], with slight changes tomake it amenable to this setting (see Appendix A.2.5).
Revised: [13], with slight changes tomake it amenable to this setting (see Appendix A.2.5).

Original: At a high-level, given a set of inputs, a set ofconcept vectors C in the same space as inputs, and prediction targets (e.g.,  domain  characteristicsof interest)
Revised: At a high-level, given a set of inputs, a set ofconcept vectors C in the same space as inputs, and prediction targets (e.g., (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain (cid:58) characteristicsof interest)

Original: , Yeh et al.
Revised: , Yeh et al.

Original: [13] deﬁne a framework to compute class-conditioned Shapley values, called
Revised: [13] deﬁne a framework to compute class-conditioned Shapley values, called

Original: ConceptSHAP.
Revised: ConceptSHAP.

Original: To do so, every input is projected onto each of the concept vectors, yielding a vectorof ‘concept scores’, which is then passed to a simple prediction head g
Revised: To do so, every input is projected onto each of the concept vectors, yielding a vectorof ‘concept scores’, which is then passed to a simple prediction head g

Original: (e.g., small MLP or linearmodel) to predict the targets and compute the ConceptSHAP.
Revised: (e.g., small MLP or linearmodel) to predict the targets and compute the ConceptSHAP.

Original: ConceptSHAP provides numericalscores interpreted as the importance of each concept in C for predicting a given target class, which isuseful for identifying key concepts (and nearby inputs) associated with certain domain characteristics.
Revised: ConceptSHAP provides numericalscores interpreted as the importance of each concept in C for predicting a given target class, which isuseful for identifying key concepts (and nearby inputs) associated with certain domain characteristics.

Original: The concept explanation framework of Yeh et al.
Revised: The concept explanation framework of Yeh et al.

Original: [13] is then used to ﬁrst predict the correct classes (using only z ω , rather than the trajectory τ , as input), then identify core concepts related to each class.
Revised: [13] is then used to ﬁrst predict the correct classes (using only z ω , rather than the trajectory τ , as input), then identify core concepts related to each class.

Original: [Table] 
Revised: [Table] [Table]

Original: In Table 3, we use this technique to identify behavioral concepts in z ω space associated with severalcharacteristics of interest in the hill-climbing domain.
Revised: In Table 3, we use this technique to identify behavioral concepts in z ω space associated with severalcharacteristics of interest in the hill-climbing domain.

Original: The set of concepts C
Revised: The set of concepts C

Original: we consider aregenerated using K-means in the z ω space (see Appendix A.2.5 for details), and a simple 2-layer MLPwith 8-hidden units for prediction head g .
Revised: we consider aregenerated using K-means in the z ω space (see Appendix A.2.5 for details), and a simple 2-layer MLPwith 8-hidden units for prediction head g .

Original: The ﬁrst row of Table 3 shows classes corresponding toincreasing levels of agent dispersion in the domain (with dispersion deﬁned as the sum of L 2 -distancesof agents from their centroid at the ﬁnal timestep of the episode).
Revised: The ﬁrst row of Table 3 shows classes corresponding toincreasing levels of agent dispersion in the domain (with dispersion deﬁned as the sum of L 2 -distancesof agents from their centroid at the ﬁnal timestep of the episode).

Original: For each trajectory τ , we computethe associated dispersion, creating 5 classes of equal-sized bins (with labels 0 through 4 mapping tobins of increasing dispersion).
Revised: For each trajectory τ , we computethe associated dispersion, creating 5 classes of equal-sized bins (with labels 0 through 4 mapping tobins of increasing dispersion).

Original: We create a train-validation split of 80-20, then train the MLP g via asoftmax-cross entropy loss to predict the correct classes using only z ω as input (rather than the actualtrajectory τ ).
Revised: We create a train-validation split of 80-20, then train the MLP g via asoftmax-cross entropy loss to predict the correct classes using only z ω as input (rather than the actualtrajectory τ ).

Original: We attain a validation set accuracy of 54.27%, signifying the predictive capabilities ofz ω .
Revised: We attain a validation set accuracy of 54.27%, signifying the predictive capabilities ofz ω .

Original: Next, we compute the class-conditioned ConceptSHAP, thus identifying the top-scoring conceptvector for each class.
Revised: Next, we compute the class-conditioned ConceptSHAP, thus identifying the top-scoring conceptvector for each class.

Original: In Table 3, we visualize the 20 trajectories with the closest z ω to the top-scoringconcept for each class (with the mean agent dispersion across these trajectories listed under eachimage).
Revised: In Table 3, we visualize the 20 trajectories with the closest z ω to the top-scoringconcept for each class (with the mean agent dispersion across these trajectories listed under eachimage).

Original: Intuitively, the identiﬁed concepts involve agents becoming increasingly dispersed, withagents ﬁrst nearly stationary at the center (class 0), then all converging to the same hill (class 1),spreading across two hills (class 2), and ﬁnally covering all hills (classes 3 and 4).
Revised: Intuitively, the identiﬁed concepts involve agents becoming increasingly dispersed, withagents ﬁrst nearly stationary at the center (class 0), then all converging to the same hill (class 1),spreading across two hills (class 2), and ﬁnally covering all hills (classes 3 and 4).

Original: The second row of Table 3 repeats the same experiment, now using classes associated with the sumof returns of agents.
Revised: The second row of Table 3 repeats the same experiment, now using classes associated with the sumof returns of agents.

Original: We attain a validation set accuracy of 60.98%, and similarly observe behavioraltrajectories associated with generally increasing reward by identifying top clusters for each class(with some overlaps, e.g., classes 1 and 2 with high standard deviation in returns).
Revised: We attain a validation set accuracy of 60.98%, and similarly observe behavioraltrajectories associated with generally increasing reward by identifying top clusters for each class(with some overlaps, e.g., classes 1 and 2 with high standard deviation in returns).

Original: Overall, this lineof experiments provides both a means of quantifying the representational capacity of z ω , and also ofidentifying clusters of concepts associated with certain behavioral characteristics.
Revised: Overall, this lineof experiments provides both a means of quantifying the representational capacity of z ω , and also ofidentifying clusters of [Table] certain [Table]

Original: Scalability to high-dimensional domains.
Revised: Scalability to high-dimensional domains.

Original: We next test MOHBA’s scalability to high-dimensionaldomains using the 2-Agent HalfCheetah multiagent MuJoCo domain [14], where agents 0 and 1,respectively, control the back and front limbs of a cheetah to coordinate movement.
Revised: We next test MOHBA’s scalability to high-dimensionaldomains using the 2-Agent HalfCheetah multiagent MuJoCo domain [14], where agents 0 and 1,respectively, control the back and front limbs of a cheetah to coordinate movement.

Original: Each agent’s stateconsists of a 6D vector summarizing velocity and position information for the 3 joints it controls,with its 3D action space corresponding to motor torques applied to said joints.
Revised: Each agent’s stateconsists of a 6D vector summarizing velocity and position information for the 3 joints it controls,with its 3D action space corresponding to motor torques applied to said joints.

Original: The agents coordinateto maximize the forward-speed of the cheetah over episodes of 200 timesteps each.
Revised: The agents coordinateto maximize the forward-speed of the cheetah over episodes of 200 timesteps each.

Original: We generate datausing 30 independent trials of MARL training, collecting 1 e 5 total trajectories to train MOHBA.
Revised: We generate datausing 30 independent trials of MARL training, collecting 1 e 5 total trajectories to train MOHBA.

Original: Figure 5a visualizes the behavior space learned by MOHBA, wherein we observe several clusters.
Revised: Figure 5a visualizes the behavior space learned by MOHBA, wherein we observe several clusters.

Original: We highlight three example trajectories stemming from distinct clusters in the joint behavior spacez ω , showing the corresponding HalfCheetah behaviors in Fig.
Revised: We highlight three example trajectories stemming from distinct clusters in the joint behavior spacez ω , showing the corresponding HalfCheetah behaviors in Fig.

Original: 5b.
Revised: 5b.

Original: The left panel of each exampleprovides a view of the cheetah’s overall movement, where we observe key behavior differences: in
Revised: The left panel of each exampleprovides a view of the cheetah’s overall movement, where we observe key behavior differences: in

Original: Example A, the cheetah runs forward using subtle vibration of its limbs; in Example B, it bouncesforward with its torso arched up due to its front limb (agent 1) being more extended than its back limb(agent 0); in Example C, it moves closer to the ground with its torso arched down.
Revised: Example A, the cheetah runs forward using subtle vibration of its limbs; in Example B, it bouncesforward with its torso arched up due to its front limb (agent 1) being more extended than its back limb(agent 0); in Example C, it moves closer to the ground with its torso arched down.

Original: In the latent space(Fig.
Revised: In the latent space(Fig.

Original: 5a), we observe that for Examples B and C, the z 0 α
Revised: 5a), we observe that for Examples B and C, the z 0 α

Original: (back limb) latents are close to one another,while the z 1 α (front limb) latents are far apart.
Revised: (back limb) latents are close to one another,while the z 1 α (front limb) latents are far apart.

Original: To investigate these local agent behavioral differences,
Revised: To investigate these local agent behavioral differences,

Original: [Figure]
Revised: [Figure]

Original: Example A (see animated version here) 
Revised: Example A (see animated version here) Example B (see animated version here)

Original: Example B (see animated version here)
Revised: [Figure]

Original: Example C (see animated version here)
Revised: (see animated version here

Original: [Figure]
Revised: [Figure]

Original: the right panel of Fig.
Revised: the right panel of Fig.

Original: 5b provides an overlaid view of the same frames, with  torso aligned (making it easier to discern the behaviors of agent 0 vs. agent 1).
Revised: 5b provides an overlaid view of the same frames, with the torso now aligned(making it easier to discern the behaviors of agent 0 vs. agent 1).

Original: Here  observe that the back limb (agent 0) behaves similarly across both examples, while the front limb (agent 1) stays much closer to the head for Example C, in contrast to Example B, which coincides with the ﬁndings in the 
Revised: Here we observe that the back limb(agent 0) behaves similarly across both examples, while the front limb (agent 1) stays much closer tothe head for Example C, in contrast to Example B, which coincides with the ﬁndings in the z α space.

Original:  Discussion
Revised:  Discussion

Revised: [Figure]

Original: Our proposed method, MOHBA, leverages trajectory data to better understand multiagent behaviorsduring and after training.
Revised: Our proposed method, MOHBA, leverages trajectory data to better understand (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] [Figure] behaviorsduring and after training.

Original: MOHBA assumes no knowledge of agents’ underlying training algorithms,does not require access to their hidden states or internal models, and applies even to reward-free settings.
Revised: MOHBA assumes no knowledge of agents’ underlying training algorithms,does not require access to their hidden states or internal models, and applies even to reward-free settings.

Original: Our experiments showcased a variety of applications of MOHBA, including the analysis of jointand local agent behaviors, monitoring of behavior emergence throughout training, discovery of behavioral concepts associated with certain  domain  criteria,  and  disentanglement  of  behaviors stemmingfrom externally-available  open-source  policies  such  as  those  for OpenAI hide-and-seek [10].
Revised: Our experiments showcased a variety of applications of MOHBA, including the analysis of jointand local agent behaviors, monitoring of behavior emergence throughout training, discovery of behavioral concepts associated with certain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) criteria, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) disentanglement (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stemming(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) externally-available (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) open-source (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) those (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: While we believe our approach is an important step in terms of increasing the understanding ofmultiagent systems, there are several limitations and potential societal impacts of note. 
Revised: While we believe our approach is an important step towards increasing the understanding of multiagent [Table] systems, there are several limitations and potential societal impacts of note. [Table]

Original: One limitationis related to the collection of a dataset of agent behaviors.
Revised: One (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) limitation is relatedto the collection of a dataset of agent behaviors.

Original: By collecting this dataset throughout agenttraining, we ensure that the most prominent behaviors that can occur in a given environment can beidentiﬁed using our method.
Revised: By collecting this dataset throughout agent training,we ensure that the most prominent behaviors that can occur in a given environment can be identiﬁedusing our method.

Original: However, this requires both storage of larger datasets than usual agenttraining requires, and also the use of a large enough sweep of agent training to capture key behaviors.
Revised: However, this requires both storage of larger datasets than usual agent training [Table] requires, and also the use of a large enough sweep of agent training to capture key behaviors.

Original: It may be interesting to consider follow-ups to our model that learn behavioral clusters in a streamingfashion, thus building knowledge of behaviors over time and permitting older data to be discardedthroughout agent training. 
Revised: Itmay be interesting to consider follow-ups to our model that learn behavioral clusters in a streamingfashion, thus building knowledge of behaviors over time and permitting older data to be discardedthroughout agent training. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Moreover, (cid:58)(cid:58) in [Table] our

Revised: [Table]

Revised: datasets,

Original: Moreover, in our datasets, we  collected  trajectories  at  uniform intervalsthroughout original  MARL  training.
Revised: [Table] we [Table] collected (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) uniform (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intervals(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) original (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: However,  it  might  be  interesting  to  consider  a  scheme  that doesso in a non-uniform manner; this can be done in a variety of ways (e.g., using a pre-trained MOHBAmodel to detect divergences from previously-seen behaviors during future MARL training runs, andcollecting trajectories only when such divergences occur).
Revised: However, (cid:58)(cid:58) it (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) might (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interesting (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consider (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) scheme [Table] that [Table]

Original: Finally, it might be interesting to enablethe agents to ﬁrst learn joint behaviors / ‘skills’ using the proposed method, and leverage them totransfer to related downstream tasks (potentially more sample-efﬁciently than training from scratch).
Revised: does [Table] so [Table]

Revised: MOHBA

Revised: [Table]

Original: Moreover, the study of behavioral interactions in multiagent systems can potentially be used forboth positive and negative societal applications.
Revised: to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) detect (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) divergences (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) previously-seen (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) during (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) future (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58) runs, (cid:58)(cid:58)(cid:58) and(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collecting (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) when (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) divergences (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) occur). (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Finally, (cid:58)(cid:58) it (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) might (cid:58)(cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interesting (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) enable(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) ﬁrst (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learn (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58) / (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘skills’ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) proposed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) method, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) leverage (cid:58)(cid:58)(cid:58)(cid:58) them (cid:58)(cid:58) to(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) transfer (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) related (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) downstream (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) tasks (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (potentially (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sample-efﬁciently (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training

Original:  For example, such an approach could be used toprevent certain undesirable behaviors by agents that interact with humans (e.g, self-driving cars), butpotentially also used by adversaries to predict and exploit other behaviors (e.g., exploiting certainhuman preferences for harm), or even inadvertently cause harm due to misinterpretation of certainbehavioral modes.
Revised: [Table] from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) scratch). Moreover, the study of behavioral interactions in multiagent systems can potentially be used forboth positive and negative societal applications. For example, such an approach could be used toprevent certain undesirable behaviors by agents that interact with humans (e.g, self-driving cars), butpotentially also used by adversaries to predict and exploit other behaviors (e.g., exploiting certainhuman preferences for harm), or even inadvertently cause harm due to misinterpretation of certainbehavioral modes.

Original: As such, further research and evaluation will be required prior to deploymentof this and related behavioral analysis approaches to human-facing domains.
Revised: As such, further research and evaluation will be required prior to deploymentof this and related behavioral analysis approaches to human-facing domains.

Original: Nonetheless, as agentcapabilities continue to grow, our view is that behavioral analysis of multiagent systems will becomeincreasingly important and should complement traditional reward-based performance monitoring.
Revised: Nonetheless, as agentcapabilities continue to grow, our view is that behavioral analysis of multiagent systems will becomeincreasingly important and should complement traditional reward-based performance monitoring.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We include these in the Discus-
Revised: [Yes] We include these in the Discus-

Original: sions section.
Revised: sions section.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We include
Revised: [Yes] We include

Original: these in the discussions section.
Revised: these in the discussions section.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] These are
Revised: [Yes] These are

Original: provided in the main paper and the proofs included in the appendix.
Revised: provided in the main paper and the proofs included in the appendix.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Proofs are provided in
Revised: [Yes] Proofs are provided in

Original: the appendix.
Revised: the appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] We provide
Revised: [No] We provide

Original: details for experiment reproducibility in Appendix A.2.
Revised: details for experiment reproducibility in Appendix A.2.

Original: We intend to include code for the model upon paper acceptance.
Revised: We intend to include code forthe model upon paper acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] We provide all training details in Appendix A.2.
Revised: [Yes] We provide all training details in Appendix A.2.(c)

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] We provide all computationaldetails in Appendix A.2.
Revised: [Yes] We provide all computationaldetails in Appendix A.2.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: Our work does notuse existing assets.
Revised: Our work does notuse existing assets.(b)

Original: (b) Did you mention the license of the assets?
Revised: Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.5.
Revised: Our work does not use existing assets.5.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not crowdsourced or human subject data.
Revised: Our work does not crowdsourced or human subject data.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not crowdsourced or
Revised: Our work does not crowdsourced or

Original: [Table]
Revised: [Table]

Original: human subject data.
Revised: human subject data.

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Our work does not crowdsourced or human
Revised: Our work does not crowdsourced or human

Original: [Table]
Revised: [Table]

Original: subject data.
Revised: subject data.

Original: [Table]
Revised: [Table]

Original: AAppendix for
Revised: A

Original:  ‘Beyond Rewards: a Hierarchical Perspective on Ofﬂine
Revised: Appendix for ‘Beyond Rewards: a Hierarchical Perspective on Ofﬂine

Original: Multiagent Behavioral Analysis’A.1 Derivations
Revised: Multiagent Behavioral Analysis’

Original: [-A.1.
-]
Revised: [-A.1.
-][Table]

Original: On the Conditional Independence of π i from z ω given z iα In (1), we assumed that each agent’s latent-conditioned policy is conditionally-independent of thehigh-level latent behavior z ω given its low-level latent z iα . This section provides some justiﬁcation ofthis assumption.
Revised: each (cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ultimately (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) informed (cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) rather (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) global (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (joint) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information.

Original: One of the prototypical paradigms in multiagent (MARL) training is that of ‘centralized training,decentralized execution’. In this regime, each agent’s decision-making policy is conditioned only onits available local information / local observations during execution; thus, the behaviors exhibited byeach agent are ultimately informed by their local information, rather than global (joint) information.
Revised: (cid:58)(cid:58)

Original: In  our  setting,  z iα serves to provide this local behavior context for each agent.
Revised: In (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) setting, (cid:58)(cid:58)(cid:58) z iα (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Thus, the assumptionof the policy being conditionally-independent of z ω given z iα corresponds well to the assumptionof agents only using local information (rather than joint information) in MARL to inform theirpolicy/decision-making. Nonetheless, we also note that there is a strong relationship between local and joint behaviors in oursetting, which provides a coordination signal to the policy implicitly through z iα .
Revised: [Table] [Table]

Original: Speciﬁcally, ourderivation results in a local behavior prior term, p ( z iα | z ω ) ; using this term, the joint behavior latentz ω is able toinﬂuence the learned space over local behavior latents z iα .
Revised: between (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) our(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) setting, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provides (cid:58) a [Table] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) signal (cid:58)(cid:58)(cid:58) to

Original: Thus, the information-ﬂow inour framework can be more intuitively described as follows:
Revised: [Table] the [Table]

Original: The joint behavior latent observes a trajectory and summarizes the joint behavior exhibitedin it (e.g., team-wide cooperation, competition, etc.) in z ω2.
Revised: policy [Table] implicitly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) through (cid:58)(cid:58)(cid:58)(cid:58) z iα . (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Subsequently, this joint latent z ω affects the space  of  z iα ,  which  are  then  sampled  and usedto inform  each  agent  how  to  behave  locally  in  order  to  achieve the joint behavior observed.
Revised: [Table] space (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) z iα , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sampled (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) used(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) inform (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58) how (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behave (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) locally (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) order (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) achieve (cid:58)(cid:58)(cid:58)

Original: Overall, the above
Revised: the [Table]

Original: relationship between the two latent spaces implies makes the conditionalindependence of the policy from z ω a reasonable simplifying assumption.
Revised: joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observed.

Original: A.1.2 Derivation of the Variational Lower Bound
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Overall, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) relationship (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) between (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) two (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) implies (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) makes (cid:58)(cid:58)(cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: This section details the derivation of the variational lower bound described in Section 3 of the maintext, which is used for training MOHBA.
Revised: This section details the derivation of the variational lower bound described in Section 3 of the maintext, which is used for training MOHBA.

Original: Similar to traditional variational autoencoder approaches [73, 74], we approximate the maximizationof the latent-conditioned trajectory probability (4) in the main text using the evidence lower bound
Revised: Similar to traditional variational autoencoder approaches [73, 74], we approximate the maximizationof the latent-conditioned trajectory probability (4) in the main text using the evidence lower bound

Original: [Equation]
Revised: [Equation]

Original: where q φ ( ·| τ ) and p θ ( · ) are, respectively, learned posterior and prior distributions.
Revised: where q φ ( ·| τ ) and p θ ( · ) are, respectively, learned posterior and prior distributions.

Original: Using (4) from the main text to expand this expression yields
Revised: Using (4) from the main text to expand this expression yields

Original: [Equation]
Revised: [Equation]

Original: Note that the terms p ( s 0 ) and p ( s t +1 | s t , a t ) stem from the underlying MA-MDP environment, andthus cannot be optimized via parameters φ and θ .
Revised: Note that the terms p ( s 0 ) and p ( s t +1 | s t , a t ) stem from the underlying MA-MDP environment, andthus cannot be optimized via parameters φ and θ .

Original: Dropping these extraneous terms yields,
Revised: Dropping these extraneous terms yields,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term simply relates to trajectory reconstruction, inducing our agent-wise policies π iθ tobehave similarly to the observed trajectories in the dataset.
Revised: The ﬁrst term simply relates to trajectory reconstruction, inducing our agent-wise policies π iθ tobehave similarly to the observed trajectories in the dataset.

Original: The second term is a regularization term involving our two latent parameters, which we further simplify using assumptions used in previousmulti-level VAE models [67].First [Equation]
Revised: The second term is a regularization terminvolving our two latent parameters, which we further simplify using assumptions used in previousmulti-level VAE models [67].

Original:  we assume that the posterior distribution is factorizable whenconditioned on a particular trajectory, as follows,
Revised: First, we assume that the posterior distribution is factorizable whenconditioned on a particular trajectory, as follows,

Original: [Equation]
Revised: [Equation]

Original: and similarly for the prior,
Revised: and similarly for the prior,

Original: [Equation]
Revised: [Equation]

Original: We next simplify the KL-divergence component of (10) as follows by combining it with (12) and (14):
Revised: We next simplify the KL-divergence component of (10) as follows by combining it with (12) and (14):

Original: [Equation]
Revised: [Equation]

Revised: (cid:34)(cid:88)

Original:  [Equation]
Revised: i D [Equation]

Original: Combining this result with (10) and additionally modulating the KL-terms in the lower bound as inβ -VAEs [72], our overall objective simpliﬁes to
Revised: Combining this result with (10) and additionally modulating the KL-terms in the lower bound as inβ -VAEs [72], our overall objective simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: A.2 Experiment Details and Hyperparameters
Revised: A.2 Experiment Details and Hyperparameters

Original: This section provides an overview of the details for our experiments.
Revised: This section provides an overview of the details for our experiments.

Original: A.2.1 Data Generation Details
Revised: A.2.1 Data Generation Details

Original: The multiagent trajectory data analyzed in this paper is generated using the Acme RL library [75],using the TD3 algorithm [76] in a decentralized fashion, with dataset management handled using
Revised: The multiagent trajectory data analyzed in this paper is generated using the Acme RL library [75],using the TD3 algorithm [76] in a decentralized fashion, with dataset management handled using

Original: RLDS [77].
Revised: RLDS [77].

Original: Each agent uses a 2-layer MLP (256 hidden units for each layer) for its TD3 network(using the vanilla TD3 network speciﬁcations in Acme), with the hyperparameters used for datageneration summarized for each domain in Table 4.
Revised: Each agent uses a 2-layer MLP (256 hidden units for each layer) for its TD3 network(using the vanilla TD3 network speciﬁcations in Acme), with the hyperparameters used for datageneration summarized for each domain in Table 4.

Original: Note that the large number of training trials/seedsused in this data generation pipeline serve to produce a wide variety of agent behaviors in the datasetsubsequently analyzed by MOHBA
Revised: Note that the large number of training trials/seedsused in this data generation pipeline serve to produce a wide variety of agent behaviors in the datasetsubsequently analyzed by MOHBA.

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: A.2.2 MOHBA Hyperparameters
Revised: A.2.2 MOHBA Hyperparameters

Original: We conduct a wide hyperparameter sweep for training MOHBA itself, summarized in Table 5.
Revised: We conduct a wide hyperparameter sweep for training MOHBA itself, summarized in Table 5.

Original: Ourjoint and local encoder models each consist of a bidirectional LSTM, followed by a 2-layer MLPhead for mapping to latent distribution parameters; a GMM head is used for the joint encoder, and a
Revised: Ourjoint and local encoder models each consist of a bidirectional LSTM, followed by a 2-layer MLPhead for mapping to latent distribution parameters; a GMM head is used for the joint encoder, and a

Original: Gaussian for the local encoder.
Revised: Gaussian for the local encoder.

Original: Our local prior model consists of a 2-layer MLP with Gaussian head,with our joint prior model simply being the learned parameters of a GMM.
Revised: Our local prior model consists of a 2-layer MLP with Gaussian head,with our joint prior model simply being the learned parameters of a GMM.

Original: All models use ReLU forintermediate layer activations.
Revised: All models use ReLU forintermediate layer activations.

Original: Note that we found that cyclically-annealing [82] the β term in ourvariational lower bound from 0 to the values speciﬁed in Table 5 to help avoid KL-vanishing.
Revised: Note that we found that cyclically-annealing [82] the β term in ourvariational lower bound from 0 to the values speciﬁed in Table 5 to help avoid KL-vanishing.

Original: All
Revised: All

Original: MOHBA and baseline training is conducted using the Adam optimizer [80], with Adam parametersβ 1 = 0 .
Revised: MOHBA and baseline training is conducted using the Adam optimizer [80], with Adam parametersβ 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , gradient clipping using a max global norm threshold of 10 . 0 , and learningrates swept over as indicated in the tables.[Table] [Table]
Revised: 999 , gradient clipping using a max global norm threshold of 10 . 0 , and learningrates swept over as indicated in the tables.

Original: A.2.3 Baseline Model Hyperparameters
Revised: A.2.3 Baseline Model Hyperparameters

Original: We similarly conduct a wide sweep over hyperparameters for the considered baselines, summarizedin Table 6.[Table] {1e-3, 1e-4} {1e-3, 1e-4} [Table]
Revised: We similarly conduct a wide sweep over hyperparameters for the considered baselines, summarizedin Table 6.

Original: N/A {1e-

Original: [Table]
Revised: [Table]

Original: 1e-2}

Original: [Table]N/A {5e [Table] [Table]
Revised: [Table]

Original: A.2.4 Computational Details
Revised: A.2.4 Computational Details

Original: For MARL trajectory data generation, we used an internal CPU cluster for both the 3-agent hillclimbing and 2-agent coordination domains, using TPUs for only the multiagent MuJoCo datageneration.
Revised: For MARL trajectory data generation, we used an internal CPU cluster for both the 3-agent hillclimbing and 2-agent coordination domains, using TPUs for only the multiagent MuJoCo datageneration.

Original: For training MOHBA itself, we used TPUs for the 3-agent hill-climbing and 2-agentcoordination domains, and a Tesla V100 GPU cluster for the MuJoCo environment.
Revised: For training MOHBA itself, we used TPUs for the 3-agent hill-climbing and 2-agentcoordination domains, and a Tesla V100 GPU cluster for the MuJoCo environment.

Original: For training the
Revised: For training the

Original: LSTM and VAE baselines, we used Tesla P100 GPU clusters.
Revised: LSTM and VAE baselines, we used Tesla P100 GPU clusters.

Original: A.2.5 Concept Discovery Framework
Revised: A.2. Concept Discovery Framework

Original: This section describes details of how we apply the completeness-aware concept-based explanationframework of Yeh et al.
Revised: This section describes details of how we apply the completeness-aware concept-based explanationframework of Yeh et al.

Original: [13] to discover interesting concepts in our setting.
Revised: [13] to discover interesting concepts in our setting.

Revised: Given a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) characteristic (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interest

Revised: [Table]

Original: Given a characteristic of interest (e.g., the level of dispersion of agents), we deﬁne a training setconsisting of joint latents z ω and class labels y
Revised: [Table] [Table] agents) , we deﬁne a training set 711 consisting of joint latents z ω and class labels y

Original: (e.g., classes corresponding to different intervalsof team returns).
Revised: (e.g., classes corresponding to different intervalsof team returns).

Original: Yeh et al.
Revised: Yeh et al.

Original: [13] seek to identify a set of concept vectors that are sufﬁcient forpredicting the labels given the inputs.
Revised: [13] seek to identify a set of concept vectors that are sufﬁcient forpredicting the labels given the inputs.

Original: Speciﬁcally, they let C =
Revised: Speciﬁcally, they let C =

Original: { c j } mj
Revised: { c j } mj

Original: =1 denote the set of concepts,which are unique vectors where c j ∈ R D ω ∀ j .
Revised: =1 denote the set of concepts,which are unique vectors where c j ∈ R D ω ∀ j .

Original: Normalizing z ω , we can then use the inner product(cid:104) z ω , c j (cid:105) as a similarity measure between z ω and concept c j .
Revised: Normalizing z ω , we can then use the inner product(cid:104) z ω , c j (cid:105) as a similarity measure between z ω and concept c j .

Original: The concept product is deﬁned ν c ( z ω ) =
Revised: The concept product is deﬁned ν c ( z ω ) =

Original: TH ( (cid:104) z ω , c j (cid:105) , κ ) ∈ R m , where TH ( · , κ ) clips values less than κ to 0 .
Revised: TH ( (cid:104) z ω , c j (cid:105) , κ ) ∈ R m , where TH ( · , κ ) clips values less than κ to 0 .

Original: Normalizing the concept productyields the concept score ˆ ν c ( z ω ) =
Revised: Normalizing the concept productyields the concept score ˆ ν c ( z ω ) =

Original: ν c ( z ω ) /
Revised: ν c ( z ω ) /

Original: || ν c
Revised: || ν c

Original: ( z ω )
Revised: ( z ω )

Original: || 2 ∈ R m , where elements provide a measure ofsimilarity of the input z ω to each of the m concept vectors.
Revised: || 2 ∈ R m , where elements provide a measure ofsimilarity of the input z ω to each of the m concept vectors.

Original: Using these deﬁnitions, we can gauge the representational power of z ω by learning a mapping g : ˆ ν c ( z ω )
Revised: Using these deﬁnitions, we can gauge the representational power of z ω by learning a mapping g : ˆ ν c ( z ω )

Original: →
Revised: →

Original: y .
Revised: y .

Original: In practice, g is a simple model (e.g., shallow network or linearprojection) so as to gauge the expressivity of the latent space. 
Revised: In practice, g is a simple model (e.g., shallow network or linearprojection) so as to gauge the expressivity of the latent space. Given a mapping g , we usethe classiﬁcation accuracy as a ‘completeness score’ [13] for the set of concepts C , deﬁned [Equation]

Original: Given a mapping g , we usethe classiﬁcation accuracy as a ‘completeness score’ [13] for the set of concepts C , deﬁnedη = sup g P z ω ,y ∼ V (cid:2)
Revised: = [Equation] [Equation] arg

Original: y = arg max y (cid:48) g ( ν c ( z ω )) (cid:3) , where V is a validation set.
Revised: [Equation] g [Equation] ν c ( z ω )) (cid:3) , where V is a validation set.

Original: Importantly, thisapproach permits us to compute class-conditioned Shapley values, called ConceptSHAP in theframework of [13].
Revised: Importantly, this approach permits us to compute class-conditioned Shapley values, called ConceptSHAP in theframework of [13].

Original: Speciﬁcally, for a given class k , the class-conditioned ConceptSHAP value foreach concept c j is deﬁned,
Revised: Speciﬁcally, for a given class k , the class-conditioned ConceptSHAP value foreach concept c j is deﬁned,

Original: [Equation]
Revised: [Equation]

Original: [Equation] where η k is the completeness score for class k (computed simply as the classiﬁcation accuracy ofthe model for the subset of validation points with ground truth label k ).
Revised: where η k is the completeness score for class k (computed simply as the classiﬁcation accuracy ofthe model for the subset of validation points with ground truth label k ).

Original: ConceptSHAP provides ameasure of importance of each concept c j for predicting the outcomes associated with a given class,which we can then use to identify the trajectories.
Revised: ConceptSHAP provides ameasure of importance of each concept c j for predicting the outcomes associated with a given class,which we can then use to identify the trajectories.

Original: The experiments in our main paper use the above setup to both quantitatively evaluate z ω in terms of representation power, and also reveal interestingconcepts associated with  relevant characteristics in the domain.
Revised: The experiments in our main paper use the abovesetup to both quantitatively evaluate z ω in terms of representation power, and also reveal interestingconcepts associated with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) relevant characteristics in the domain.

Original: For each of the prediction problems considered in the main paper (agent dispersion and agent returns),we create an 80-20 training-validation data split (over 5 classes).
Revised: For each of the prediction problems considered in the main paper (agent dispersion and agent returns),we create an 80-20 training-validation data split (over 5 classes).

Original: For concept generation we use
Revised: For concept generation we use

Original: K-means with 16 and 24 clusters, a 2-layer MLP (with 8 hidden units per layer) for the predictionhead g , training with a batch size of 64 , and a concept threshold κ of 0.0 and 0.3 for the dispersionand agent return experiments, respectively, and train the prediction head for 1 e 4 steps.
Revised: K-means with 16 and 24 clusters, a 2-layer MLP (with 8 hidden units per layer) for the predictionhead g , training with a batch size of 64 , and a concept threshold κ of 0.0 and 0.3 for the dispersionand agent return experiments, respectively, and train the prediction head for 1 e 4 steps.

Original: As in Yehet al.
Revised: As in Yehet al.

Original: [13], we use KernelSHAP [83] to approximate ConceptSHAP efﬁciently.
Revised: [13], we use KernelSHAP [83] to approximate ConceptSHAP efﬁciently.

Original: A.3 Additional Results
Revised: A.3 Additional Results

Revised: A.3.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: A.3.1 Additional Large-scale Experiments
Revised: [Table] (Fig. 6a (cid:58)(cid:58) ). (cid:58)(cid:58)(cid:58) To (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collect (cid:58)(cid:58)(cid:58) data (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain, (cid:58)(cid:58)(cid:58)

Original: MultiAgent MuJoCo
Revised: we [Table] wider [Table]

Original: AntWalker Domain Results.
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58) both [Table] the [Table]

Original: We also consider the MultiAgent MuJoCo
Revised: and [Table] SAC [Table]

Original: AntWalker domain, wherein 4 agents each control one of 4 ant legs to coordinate movement towardsthe + x direction (Fig. 6a). To collect data for this domain, we conduct a wider MARL parametersweep using both the TD3 and SAC algorithms,  with  varying  training  batch  sizes  and learning rates togather a  widely-varying  dataset  of  behaviors. 
Revised: algorithms, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) varying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) batch (cid:58)(cid:58)(cid:58)(cid:58) sizes (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learning(cid:58)(cid:58)(cid:58)(cid:58) rates (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) gather (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) widely-varying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors. (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subsequently (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) train MOHBA (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) data, [Table]

Original: We subsequently train MOHBA on this data, visualizingthe learned  behavior clusters in Fig.
Revised: visualizing (cid:58)(cid:58)(cid:58) the [Table] learned (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior [Table]

Original: 6b.
Revised: clusters (cid:58)(cid:58) in [Table] Fig. 6b (cid:58) .

Original:  We  observe  several  behavior  clusters  of interest; notably, alarge joint cluster exists in the top-right region of the joint and local returns, which, upon inspection ofunderlying trajectory  videos,  corresponds  to  cases  where  agents  attain  extremely low return. 
Revised: (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) several (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interest [Table] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) underlying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) videos, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corresponds (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) cases (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) extremely (cid:58)(cid:58)(cid:58)(cid:58) low(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Similarly,there exists  a  smaller cluster in the lower-left region of the joint returns that also attains extremelylow performance.
Revised: Similarly, (cid:58)(cid:58)(cid:58)(cid:58) there (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exists (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) smaller (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: The remaining clusters correspond  to  the  agents displaying various  ant-poses, andmoving only  incrementally. 
Revised: correspond (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) displaying(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) various (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ant-poses, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) moving (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) incrementally. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) One (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exceptional (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) cluster (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exists (cid:58)(cid:58) in [Table] the

Original: One exceptional cluster also exists in the left  regionof the joint behaviorspace, which attains medium-level return (points that are primarily red in color).
Revised: [Table] left [Table] region

Original: On closer inspection,the AntWalker behavior in this cluster corresponds to one of  theagents learning a reasonably goodwalking gait, while the remaining three agents remain stationary.
Revised: [Table] of [Table] the

Original: Application to externally-trained policies: team-based hide-and-seek game.
Revised: [Table] joint [Table] space,

Original: We next considerthe application of MOHBA to pre-existing policies trained by external teams.
Revised: [Table] walking (cid:58)(cid:58)(cid:58)(cid:58) gait, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) while (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) remaining (cid:58)(cid:58)(cid:58)(cid:58) three (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) remain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stationary. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Application (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) externally-trained (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: Speciﬁcally, we considerthe aforementioned  OpenAI  hide-and-seek  environment,  wherein  two  teams of two  agents  (hiders andseekers) compete  against  one  another  in  a  rich, high-dimensional environment  with  various dynamicobjects (boxes and ramps) that the agents can interact with.
Revised: Speciﬁcally,(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consider (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) aforementioned (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) OpenAI (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hide-and-seek (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) wherein (cid:58)(cid:58)(cid:58)(cid:58) two (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) teams (cid:58)(cid:58) of(cid:58)(cid:58)(cid:58) two (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (hiders (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seekers) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) compete (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) against (cid:58)(cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) another (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) rich, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-dimensional(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) various (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: OpenAI has open-sourced policy checkpoints manually  annotated  by  human  experts  as exhibiting various  distinctive  multiagent  behaviors atkey stages  of  training  in github.com/openai/multi-agent-emergence-environments .
Revised: [Table] manually (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) annotated (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experts (cid:58)(cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exhibiting(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) various (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinctive (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) multiagent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58)(cid:58) at (cid:58)(cid:58)(cid:58)(cid:58) key (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stages (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58) in (cid:58) github.com/openai (cid:58)(cid:58)(cid:58)(cid:58)

Original: Weconsider four  distinctive  policy  checkpoints that each  correspond  to  the  following human-annotatedlabels: A)  ‘running  and  chasing’,  B) ‘fort building’,  C)  ‘ramp  use’,  and  D)  ‘ramp 
Revised: We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consider (cid:58)(cid:58)(cid:58)(cid:58) four (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinctive (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58) that(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) following (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human-annotated (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels: (cid:58)(cid:58)(cid:58) A) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘running (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) chasing’, (cid:58)(cid:58)(cid:58) B) (cid:58)(cid:58)(cid:58)(cid:58) ‘fort(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) building’, (cid:58)(cid:58) C) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘ramp (cid:58)(cid:58)(cid:58)(cid:58) use’, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) D) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘ramp (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) defense’. (cid:58)(cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state-space (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)

Original: defense’
Revised: [Table]

Original: The state-space for each of the agents at each timestep of the trajectory is 100-dimensional, consistingof its own state (position, rotation, and velocity information), states of the other 3 agents, and thestates of 3 boxes and one ramp in the environment (position,  velocity,  and box size information);each agent’s  action  space  consists  of  a  3-dimensional  force  vector,  and  a ‘glue’ and  ‘lock’  action forinteracting with  other  objects.Overall, this yields a state-action space with dimensionality ( 4 agents× 105 state-actions), signiﬁcantly larger than the ( 2 agents × 9 state-actions) of the HalfCheetahenvironment above. We collect 100 trajectories per policy checkpoint, with each trajectory being 200decision-steps long.
Revised: (position, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) velocity, (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) box(cid:58)(cid:58)(cid:58) size (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information); (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent’s (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) action (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consists (cid:58)(cid:58) of (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 3-dimensional (cid:58)(cid:58)(cid:58)(cid:58) force (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) vector, (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘glue’(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘lock’ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) action (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interacting (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) other (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) objects.

Original: Note that these trajectories have a wide distribution of behaviors, as agent andobject initializations are also random in each episode. We then mix the trajectories collected from all of the above policy checkpoints together, then train
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Overall, (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) yields (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state-action (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58)(cid:58) with(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] (cid:58)(cid:58)

Original: MOHBA on this shufﬂed dataset.
Revised: [Table] environment (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above.

Original: As usual, note that we provide no reward or return informationto MOHBA. Figure 7 visualizes the behavior spaces discovered by MOHBA in the hide-and-seekdomain.
Revised: (cid:58)(cid:58)(cid:58)(cid:58)

Original: Agents 0 and 1 in this ﬁgure correspond to ‘hiders’, whereas agents 2 and 3 are ‘seekers’. We label each of the trajectories in this ﬁgure withthe human-annotated behavior labels provided by
Revised: We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collect (cid:58)(cid:58)(cid:58)(cid:58) 100 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58) per [Table] policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoint, (cid:58)(cid:58)(cid:58)(cid:58) with

Original: OpenAI.
Revised: [Table]

Original: In Fig.
Revised: each [Table]

Original: 7, we observe the presence of clear behavior clusters that correspond well to the expert policycheckpoint labels, both at the joint and local agent levels.
Revised: being [Table] [Table]

Original: Interestingly, for the seekers (agents 2 and3), policy A (‘running and chasing’) is highly distinctive and well-separated from the other behaviors.
Revised: steps (cid:58)(cid:58)(cid:58)(cid:58) long [Table] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Note [Table]

Original: Interestingly, despite the order of emergent behaviors in the original hide-and-seek MARL trainingbeing A → B → C → D, policies B (‘fort building’) and D (‘ramp defense’) appear to be behaviorally
Revised: that [Table] episode.

Original: [Figure] 
Revised: [Figure] g e n t

Revised: R e t u r n

Original: [Figure]
Revised: [Figure]

Original: A g e n t R e t u r n 
Revised: S u m o f a g e n t - w i s e r e t u r n s (b)

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Figure]
Revised: [Table]

Original: S u m o f a g e n t - w i s e r e t u r n s (b) slightly closer to one another than C (‘ramp use’) and D, namely in the joint space and also in that ofthe seeker (agent 2 and 3) latent spaces.
Revised: space (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58) by (cid:58) MOHBA (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) AntWalker (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain. (cid:58) (cid:58)(cid:58)(cid:58)

Original: One hypothesis is that this could perhaps be due to both the‘fort building’ and ‘ramp defense’ policies being associated with situations where the seekers cannoteasily ﬁnd the hiders, due to  the hiders using obstacles to block entrances (B) and moving ramps toprevent their effective use (D).
Revised: We (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58) mix (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collected (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) together, (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58) train 776 MOHBA (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shufﬂed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset. (cid:58)(cid:58)(cid:58) As (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) usual, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) note (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provide (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reward (cid:58)(cid:58) or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information(cid:58)(cid:58) to MOHBA. (cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) discovered (cid:58)(cid:58)(cid:58) by (cid:58) MOHBA (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hide-and-seek(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain.

Original: {+(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Agents (cid:58)(cid:58) 
+}
Revised: {+(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Agents (cid:58)(cid:58) 
+}

Original: {+(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 1 (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgure (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘hiders’, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) whereas (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58) 
+}
Revised: {+(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 1 (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgure (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘hiders’, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) whereas (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58) 
+}

Revised: (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 3 (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘seekers’.(cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58) label (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgure (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human-annotated (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provided(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) OpenAI.

Revised: (cid:58)(cid:58)(cid:58)

Revised: In Fig.

Revised: 7, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) presence (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) clear (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58)(cid:58)(cid:58) well (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) expert (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels, (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) levels.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: Interestingly, (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seekers (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: (agents (cid:58) 2 (cid:58)(cid:58)(cid:58) and(cid:58)(cid:58) 3), (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)

Revised: A

Revised: [Table]

Revised: ‘running (cid:58)(cid:58)(cid:58)(cid:58) and

Revised: [Table]

Revised: chasing’)

Revised: [Table]

Revised: is

Revised: [Table]

Revised: highly

Revised: [Table]

Revised: distinctive

Revised: [Table]

Revised: and

Revised: [Table]

Revised: well-separated

Revised: [Table]

Revised: from

Revised: [Table]

Revised: the

Revised: [Table]

Revised: other

Revised: [Table]

Revised: .(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: Interestingly, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) despite (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)

Revised: [Table]

Revised: [Table]

Revised: building’)

Revised: (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) D (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (‘ramp (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) defense’) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) appear (cid:58)(cid:58) to (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviorally

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: ramps) (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58) can (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interact (cid:58)(cid:58)(cid:58)(cid:58) with.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)

Revised: We (cid:58)(cid:58)(cid:58) mix (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collected (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) together, (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58) train (cid:58) MOHBA (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shufﬂed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Original: Overall, these experiments help to validate MOHBA’s learned latentspaces using policy labels manually annotated by human experts, and also highlight the applicabilityof our  algorithm in domains with large state spaces.
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Using (cid:58) MOHBA (cid:58) , (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) presence (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clear (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58)(cid:58)(cid:58) well (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) expert (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) both (cid:58) at (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) levels, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) despite (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) algorithm (cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) having (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) access (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) explicit (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) during (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Domain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) screenshots (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reproduced (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) permission (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Baker et al. (cid:58) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) closer (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) another (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58) C (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: A.3.2 Additional Baseline Results
Revised: (‘ramp (cid:58)(cid:58)(cid:58)(cid:58) use’) (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) D, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) namely (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58) of (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seeker (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (agent (cid:58)(cid:58) 2 (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 3) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces. (cid:58)(cid:58)(cid:58)(cid:58) One (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hypothesis (cid:58) is (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) could (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) perhaps (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58) due

Original: Figures 8 and 9 provide additional results for the baseline comparisons conducted in the main paper.
Revised: (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58) ‘fort (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) building’ (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘ramp (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) defense’ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) being (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) associated (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) situations (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seekers (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) cannot(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) easily (cid:58)(cid:58)(cid:58) ﬁnd (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hiders, (cid:58)(cid:58)(cid:58)(cid:58) due (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hiders (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) obstacles (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) block (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) entrances (cid:58)(cid:58)(cid:58) (B) (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) moving (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ramps (cid:58)(cid:58) to(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) prevent (cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) effective (cid:58)(cid:58)(cid:58)(cid:58) use (cid:58)(cid:58)(cid:58) (D). (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Overall, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiments (cid:58)(cid:58)(cid:58)(cid:58) help (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) validate MOHBA (cid:58) ’s

Original: Speciﬁcally, (a) in each ﬁgure provides an expanded comparison of ICTD (with a sweep over the #
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58)(cid:58)(cid:58) latent(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) manually (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) annotated (cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experts, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) highlight (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) applicability (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) algorithm (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] Figures 8 and 9 provide additional results for the baseline comparisons conducted in the main paper

Original: [Figure]
Revised: [Table]

Original:  of clusters in K-means). In (b) of each ﬁgure, we visualize the convergence of the APL throughouttraining for MOHBA and the baselines.
Revised: Speciﬁcally, (a) in each ﬁgure provides an expanded comparison of ICTD (with a sweep over the # of clusters in K-means). In (b) of each ﬁgure, we visualize the convergence of the APL throughouttraining for MOHBA and the baselines.

Original: Subﬁgures (c) and (d), respectively, show a PCA and UMAPprojection of the latent space discovered by the LSTM baseline (recall from the main paper that ‘jointembeddings’ used for the LSTM are simply the ﬁnal hidden state).
Revised: Subﬁgures (c) and (d), respectively, show a PCA and UMAPprojection of the latent space discovered by the LSTM baseline (recall from the main paper that ‘jointembeddings’ used for the LSTM are simply the ﬁnal hidden state).

Original: Likewise, subﬁgures (e) and (f)show the same projections for the VAE baseline.
Revised: Likewise, subﬁgures (e) and (f)show the same projections for the VAE baseline.

Original: As especially evident in the coordination domain (Fig. 9), the joint LSTM baseline embeddings(whether using PCA or UMAP projections, Figs.
Revised: As especially evident in the coordination domain (Fig. 9), the joint LSTM baseline embeddings(whether using PCA or UMAP projections, Figs.

Original: 9c and 9d, respectively) do not reveal to the fairlyinterpretable 3 clusters of coordinated behaviors discussed in the main paper.
Revised: 9c and 9d, respectively) do not reveal to the fairlyinterpretable 3 clusters of coordinated behaviors discussed in the main paper.

Original: By contrast, the VAEbaseline (in both the PCA and UMAP case, Figs. 9e and 9f, respectively) does produce these clusters;despite this, note that the key limitation of this baseline is its non-hierarchical nature, which makes identiﬁcation of the local-to-joint behavioral correspondences noted in the main paper signiﬁcantlymore difﬁcult.[Figure] ICTD K=
Revised: By contrast, the VAEbaseline (in both the PCA and UMAP case, Figs. 9e and 9f, respectively) does produce these clusters;despite this, note that the key limitation of this baseline is its non-hierarchical nature, which makesidentiﬁcation of the local-to-joint behavioral correspondences noted in the main paper signiﬁcantlymore difﬁcult.

Original: K= K= LSTM 4 . 8 ± 0 .0 .
Revised: [Table]

Original: 60 ± 0 .0 . 51 ± 0 .0 . 47 ± 0 . VAE 6 .
Revised: [Table]

Original: 9 ± 0 .0 .

Original: 44 ± 0 .0 .

Original: 30 ± 0 .0 . 18 ± 0 . MOHBA 1 . 3 ± 0 .0 .
Revised: [Table]

Original: 46 ± 0 .0 .

Original: 31 ± 0 .0 .

Original: 17 ± 0 .

Original: [Figure]
Revised: [Figure]

Original: A.4 Hyperparameter Ablations and Effects on Latent Spaces

Original: This section conducts an analysis of the effect of key hyperparameters on the latent spaces learned by

Original: MOHBA.

Original: We focus our analysis on the KL-loss β term in the training objective, alongside the latentz ω and z α dimensionality.

Original: We both compare the policy reconstruction loss (5) across ablations toattain a quantitative comparison of the quality of the latent encodings, and additionally compare thestructure of the learned latent spaces themselves as a function of these parameters for a qualitativeunderstanding.

Original: APL
Revised: APL

Original: ICTD
Revised: ICTD

Original: K=
Revised: K=

Original: K= K=
Revised: K= K=

Original: LSTM 2 .
Revised: LSTM 2 .

Original: 9 ± 0 .0 .
Revised: 9 ± 0 .0 .

Original: 41 ± 0 .0 .
Revised: 41 ± 0 .0 .

Original: 40 ± 0 .0 .
Revised: 40 ± 0 .0 .

Original: 34 ± 0 .
Revised: 34 ± 0 .

Original: VAE 5 . 0 ± 0 .0 .
Revised: VAE 5 . 0 ± 0 .0 .

Original: 18 ± 0 .0 .
Revised: 18 ± 0 .0 .

Original: 17 ± 0 .0 .
Revised: 17 ± 0 .0 .

Original: 16 ± 0 .
Revised: 16 ± 0 .

Original: MOHBA 2 .
Revised: MOHBA 2 .

Original: 4 ± 0 .0 .
Revised: 4 ± 0 .0 .

Original: 18 ± 0 .0 .
Revised: 18 ± 0 .0 .

Original: 17 ± 0 .0 .
Revised: 17 ± 0 .0 .

Original: 16 ± 0 .
Revised: 16 ± 0 .

Original: [Figure]
Revised: [Figure]

Original: LSTM Baseline
Revised: LSTM Baseline

Original: VAE Baseline
Revised: VAE Baseline

Original: [Figure]
Revised: [Figure]

Original: First, in terms of the raw policy reconstruction loss, we ﬁnd in Figs. 10a, 11a and 12a that the latentdimensionality has a higher impact on policy reconstruction performance than the weighing term β ;intuitively, increasing latent dimensionality leads to better reconstruction (lower loss) due to the latentspace being able to encode more behavioral information about the agent trajectories.
Revised: A.

Original: By contrast, theeffects of the KL weighing term are more negligible in terms of reconstruction loss.
Revised: [Figure]

Original: Second, we inspect the effects of these hyperparameter sweeps on the learned latent distributionsthemselves. Speciﬁcally, the respective panels (b)-(e) of each of Fig.
Revised: Hyperparameter (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Ablations (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Effects (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: 10, Fig.
Revised: [Table]

Original: 11, and Fig.
Revised: Spaces [Table]

Original: 12visualize the change in latent space structure as a function of the latent dimensionality and β . At ahigh-level, prominent behavioral clusters mentioned in  the main text are re-discovered throughoutthese parameter sweeps (e.g., the three local clusters for the hill climbing environment, the jointand local clusters for the coordination environment, and the various walking gait clusters for the
Revised: (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) focus (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) analysis (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) KL-loss (cid:58)(cid:58) β (cid:58)(cid:58)(cid:58)(cid:58) term (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) objective, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) alongside (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) latent(cid:58)(cid:58) z ω (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) z α (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dimensionality.

Original: HalfCheetah environment).
Revised: (cid:58)(cid:58)(cid:58)(cid:58)

Original: For the more complex latent spaces (e.g., HalfCheetah), increasing the dimensionality of z tends to increase the number of joint clusters
Revised: We (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) compare (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58) loss (5) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) across (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ablations (cid:58)(cid:58) to(cid:58)(cid:58)(cid:58)(cid:58) attain (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) quantitative (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) comparison (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) quality (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) encodings, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) additionally (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) compare (cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) structure (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) themselves (cid:58)(cid:58) as (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) function (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) parameters (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) qualitative(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) understanding.(cid:58)(cid:58)(cid:58)(cid:58) First, (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) terms (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) raw (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58) loss, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58) ﬁnd (cid:58)(cid:58)(cid:58) in Figs. 10a, 11a and 12a (cid:58)(cid:58)(cid:58)

Original: identiﬁed
Revised: that (cid:58)(cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dimensionality (cid:58)(cid:58)(cid:58) has (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) higher (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) impact (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) performance (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) weighing(cid:58)(cid:58)(cid:58)(cid:58) term (cid:58)(cid:58) β ; (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intuitively, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increasing (cid:58)(cid:58)(cid:58)(cid:58) [Table] [Table] [Table]

Original:  (e.g., compare Fig.
Revised: [Figure] [Table] and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly (cid:58)(cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinguishable (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (e.g., (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) comparing (cid:58)(cid:58) z ω(cid:58)(cid:58) in Fig.

Original: 12cversus Fig. 12e); this is intuitive as a larger latent space results in a richer encoding, capable ofdistinguishing more nuanced behaviors.
Revised: 12c (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) versus (cid:58) Fig. 12d (cid:58) ); (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) aligns (cid:58)(cid:58)(cid:58)(cid:58) well (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intuition, (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increasing (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58) β (cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: Increasing the KL β term from 1 e − 4 to 1 e − 2 tends toresult in clusters that overlap more / are ‘softer’ and slightly less distinguishable (e.g., comparing z ωin Fig. 12c versus Fig. 12d); this also aligns well with intuition, as increasing the β term prioritizesthe KL divergence between the posterior and prior, which deprioritizes disentangling the behaviorsfor the  policy  reconstruction  term  (5). 
Revised: [Table] [Table] the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58) term (cid:58)(cid:58)(cid:58) (5). (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Overall,  these  results  provide  us  intuition  in  terms  of  the  role  of  these hyperparameters in the behaviorclusters learned.
Revised: Overall, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) results (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provide (cid:58)(cid:58)(cid:58) us (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intuition (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) terms (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) role (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] A. [Table]

Revised: (cid:58)(cid:58)(cid:58)(cid:58)

Original: At a high level, it appears that the sensitivity of the results to these hyperparametersis fairly low (in the range of values explored in these experiments), thus allowing the high levelbehaviors discovered to remain reasonably distinctive across the various sweeps.
Revised: [Table] [Table] from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) numerous (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) underlying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58) runs (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain. (cid:58)(cid:58)

Original: A.5 Additional Dataset Details / Reward Distributions
Revised: To (cid:58)(cid:58)(cid:58)(cid:58) help (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ensure (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collection (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) analysis (cid:58)(cid:58) of (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diverse (cid:58)(cid:58) set (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) including (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sub-optimal (cid:58)(cid:58)(cid:58)(cid:58) ones, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collect (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) end (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training, (cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) rather(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout (cid:58)(cid:58) all (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training. (cid:58)(cid:58)(cid:58) All (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) including (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) those (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stemming(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) randomly-initialized (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] at

Original: This section provides additional details and statistics regarding the trajectory datasets used forevaluation.
Revised: [Table] the [Table] beginning

Original: Speciﬁcally, one of the core properties of datasets typically used for ofﬂine RL is theinclusion of a large number of sub-optimal and random trajectories [84].
Revised: [Table] of [Table] are

Original: As noted earlier, thetrajectory data for the agents in our experiments is collected from numerous underlying MARLtraining runs for each domain.
Revised: [Table] and [Table] used

Original: To help ensure the collection and analysis of a diverse set of behaviorsincluding sub-optimal ones, we not only collect trajectories at the end of MARL training, but ratherfrom policy checkpoints throughout all of training.
Revised: [Table] to [Table] [Table]

Original: All such trajectories, including those stemmingfrom randomly-initialized agent policies at the beginning of training, are shufﬂed together and usedto simultaneously train MOHBA.
Revised: [Table] for (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) datasets (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) analyzed. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Domain (cid:58)(cid:58) % (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) total (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) < 0 . 5 × (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (maximum (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table] 
Revised: [Table] [Table]

Original: Coordination game
Revised: of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) considered [Table] that

Original: [Table]
Revised: [Table]

Original: Here we more closely inspect our datasets to better understand the distribution of agent behaviors.
Revised: attain [Table] than

Original: At a high level, we ﬁrst note that our datasets consist of a signiﬁcant proportion of sub-optimaltrajectories.
Revised: [Table] [Table] of [Table]

Original: Speciﬁcally, in Table 7, we note the percent of trajectories in each of the considereddatasets that attain less than 50% of maximum  observed return.
Revised: maximum [Table] observed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return [Table]

Original: Note  that in the high-dimensional
Revised: Note [Table] that [Table]

Original:  MultiAgent  MuJoCo  environments  especially,  a  large  proportion  of  low-reward  trajectories  exist inthe dataset, with the HalfCheetah dataset being a particularly notable one where 92.69% of datasetsattain less than 50% of max observed return.
Revised: in [Table] the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-dimensional(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MultiAgent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MuJoCo (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) especially, (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) large (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) proportion (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) low-reward (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) exist (cid:58)(cid:58) in(cid:58)(cid:58)

Original:  We  additionally  provide  a  more  detailed  overview  of  trajectory  returns  in  each  of Figs. 13a, 14a, 15aand 16a.
Revised: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) HalfCheetah (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) being (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) particularly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) notable (cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 92.69% (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) datasets(cid:58)(cid:58)(cid:58)(cid:58) attain (cid:58)(cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58) 50% (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) max (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return.(cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) additionally (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provide (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) detailed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) overview (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58) of Figs. 13a, 14a, 856 15a and 16a (cid:58) .

Original:  These  ﬁgures  visualize  the  distribution  of  individual  agent  returns  in  each  of the consideredenvironments (respectively, hill-climbing, coordination game, 2-agent HalfCheetah, and 4-agent
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) These (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgures (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualize (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) individual (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) considered (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (respectively, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: AntWalker).
Revised: [Table] [Table]

Original: Notably, many of the trajectories in the HalfCheetah domain are skewed towards mediumand low-return behaviors, with very few high-return trajectories present (Fig. 13a).
Revised: trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) present(cid:58) (Fig. 13a (cid:58) ). (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Interestingly, (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)

Original: Interestingly, inthe AntWalker  dataset  we  observe  a  bimodal  return distribution (seemingly  consisting  of  very lowreturn and  very  high  return  trajectories,  as  seen  in  Fig. 14a).
Revised: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) AntWalker (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) bimodal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (seemingly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consisting (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) very (cid:58)(cid:58)(cid:58) low (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) very (cid:58)(cid:58)(cid:58)(cid:58) high (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories, (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seen (cid:58)(cid:58) in (cid:58) Fig. 14a (cid:58) ).(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Closer  inspection  of  this  domain revealsthat a large number of training runs result in highly sub-optimal behaviors with very low returns; in
Revised: Closer (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) inspection (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reveals (cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: Fig.
Revised: [Figure]

Original: 14b, we visualize the distribution of returns with these low-return trajectories excluded, observinga similar distribution to the HalfCheetah dataset (consisting primarily of medium-return trajectories). Similarly, a large proportion of trajectories with extremely low returns exist in the hill-climbing andcoordination game domains (Figs. 15a and 16a); we veriﬁed that these low-return trajectories consistprimarily of random agent behaviors collected early in training.
Revised: [Figure] [Table]

Original: Overall, this analysis is helpful indetermining that the analyzed datasets exhibit a wide range of trajectories (in terms of agent returns),covering random behaviors, sub-optimal behaviors, and high-return behaviors, similar to typicalofﬂine RL datasets [84]. As part of this investigation, we also analyze the discovered behavior spaces (both at joint and localagent levels) with respect to the return distributions.
Revised: [Table] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original:  Speciﬁcally, Figs.
Revised: [Table] Speciﬁcally, [Table]

Original: 13b, 14c, 15b and 16b visualize the behavior spaces  learned by MOHBA, which are then labeled by the ground truth return corresponding to each trajectory.
Revised: 13b, 14c, 15b and 16b [Table] spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned [Table]

Original: Interestingly, despite MOHBA not using any reward or return informationin learning agent behavior spaces, clear clusters corresponding  to low-return (random/early-trainingtrajectories), medium-return trajectories, and high-return trajectories are automatically discovered inall of the considered domains.
Revised: by MOHBA [Table] [Table] which (cid:58)(cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ground (cid:58)(cid:58)(cid:58)(cid:58) truth(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corresponding (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory.

Original: For example, in the hill-climbing environment, many of the trajectoriescorresponding to low returns are clustered in the center of the joint latent space in Fig.
Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Interestingly, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] low-return(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (random/early-training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories), (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) medium-return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58) are(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: 15b; in the
Revised: [Table]

Original: HalfCheetah environment, a prominent cluster of high-return trajectories is visible in the top-leftand bottom-left local latent spaces of agents 0 and 1, respectively (Fig. 14c); in the AntWalkerenvironment, the lowest-return trajectories  are  clustered  in distinctive regions both at the  joint andlocal level, with a medium-return trajectory set also visible in  the latent spaces (Fig. 14c).
Revised: trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corresponding (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) low (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clustered (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) center (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58) in (cid:58) Fig. 15b; (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) HalfCheetah (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]

Original: Overall,these results indicate the richness of the learned latent spaces in terms of not only raw trajectories butalso their capacity to cluster trajectories that exhibit similar performance, even without observing thereward function.
Revised: [Table] [Table] [Table] distinctive

Original: A.6 Visualizing z ω and z α
Revised: [Table] [Table] [Table] A.

Original:  This  section  summarizes  the  procedure  used  to  generate  the  behavior  space  ﬁgures  associated withz ω  and  z α .
Revised: [Table] and (cid:58)(cid:58) z α(cid:58)(cid:58)(cid:58)(cid:58) This (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) section (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) summarizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) procedure (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) used (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generate (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgures (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) associated (cid:58)(cid:58)(cid:58)(cid:58) with(cid:58)(cid:58) z ω (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) z α .

Original:  The visualization procedure we use is as follows:
Revised: (cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] [Table] [Table]

Original: We train MOHBA using the ofﬂine behavior datasets of interest.2.
Revised: [Table]

Original: Following training of MOHBA, we pass each trajectory τ through both the joint encoderq φ ( z ω | τ ) and the agent-wise local encoders q φ ( z iα | τ ) ,  sampling  a  joint  latent  z ω  and  a set oflocal latents  z iα  for  all  N  agents  accordingly. 
Revised: , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sampling (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58) z ω (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) a (cid:58)(cid:58) set(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latents (cid:58)(cid:58) z iα (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58) N (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accordingly. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Thus,  for  K  such  trajectories  we  obtain a setof latent  parameters  { ( z ω , z
Revised: Thus, (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) K (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) obtain (cid:58) a(cid:58)(cid:58) set (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) parameters (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) { ( z ω , z

Original: 1 α , · · · , z Nα ) k } Kk =1 .3. 
Revised: 1 α , · · · , z Nα ) k } Kk =1 .3. (cid:58)(cid:58)

Original: As  the  latent  parameters  are  high-dimensional  in  nature,  we  visualize  their  2D projection(e.g.
Revised: As (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) parameters (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-dimensional (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) nature, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualize (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58) 2D (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) projection(cid:58)(cid:58)(cid:58)(cid:58) (e.g., (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Principal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Component (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Analysis), (cid:58)(cid:58)(cid:58)(cid:58) thus (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) yielding (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizations(cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) those (cid:58)(cid:58) in Fig. 3a. (cid:58)

Original: , using Principal Component Analysis), thus yielding the behavior space visualizationssuch as those in Fig. 3a.
Revised: [Table] [Figure] [Table] [Table]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Revised: Ablations (cid:58)(cid:58)(cid:58)(cid:58) over (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hyperparameters (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 2-agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) coordination (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) game.

Revised: (cid:58)(cid:58) (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58) loss (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout MOHBA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training.

Revised: (b)-(e) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualize (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) change (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) ﬁnal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MOHBA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) models (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58)(cid:58)(cid:58) over (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ablations.

Original:  [Figure]
Revised: (cid:58) [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: Ablations (cid:58)(cid:58)(cid:58)(cid:58) over (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hyperparameters (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Multiagent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MuJoCo (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) HalfCheetah.

Revised: (cid:58) (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58) loss (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout MOHBA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Figure]
Revised: [Table] visualize [Table]

Revised: the

Revised: [Table]

Revised: change (cid:58)(cid:58)(cid:58) in

Revised: [Table]

Revised: space

Revised: [Table]

Revised: for

Revised: [Table]

Revised: each

Revised: [Table]

Revised: of

Revised: [Table]

Revised: the

Revised: [Table]

Revised: ﬁnal

Revised: [Table]

Revised: MOHBA

Revised: [Table]

Revised: models (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned

Revised: [Table]

Revised: over

Revised: [Table]

Revised: these

Revised: [Table]

Revised: ablations.

Revised: [Table]

Revised: Increasing (cid:58)(cid:58)(cid:58)

Revised: [Table]

Revised: term (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 e − 4 (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 e − 2 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) tends (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) result (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) overlap (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58) /

Revised: (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘softer’ (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly (cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinguishable (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Figure] 
Revised: [Figure] A g e n t

Revised: R e t u r n

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: statistics (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Multiagent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MuJoCo (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 2-agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: HalfCheetah (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment.

Revised: (cid:58) (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Revised: (cid:58) (b) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Original:  [Figure]
Revised: (cid:58) [Figure] [Table] [Figure]

Revised: (anomalous (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58) 0 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) removed).

Original: [Figure] 
Revised: [Figure] A g e n t

Revised: R e t u r n

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: S u m o f a g e n t - w i s e r e t u r n s

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: the

Revised: [Table]

Revised: distribution

Revised: [Table]

Revised: of

Revised: [Table]

Revised: agent

Revised: [Table]

Original: n
Revised: in

Revised: [Table]

Revised: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Revised: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: Interestingly, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) bimodal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (seemingly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consisting (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) low (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) higher (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: [Table]

Revised: excluded, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) better (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) illustrates (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) medium-high (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reward (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Revised: (cid:58) (c) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Revised: labeled (cid:58)(cid:58) by (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Original: [Figure] 
Revised: [Figure] [Table]

Revised: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Revised: (cid:58)

Revised: [Figure]

Revised: g e n t

Revised: R e t u r n

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Revised: (cid:58)