Original: Beyond Rewards: a Hierarchical Perspective on Ofﬂine Multiagent Behavioral Analysis
Revised: Beyond Rewards: a Hierarchical Perspective on Ofﬂine Multiagent Behavioral Analysis

Original: Abstract[Note: revisions shown in blue.]
Revised: Abstract

Original: Each year, expert-level performance is attainedin increasingly-complex multiagent domains, notable examples including Go, Poker, and StarCraft II.
Revised: Each year, expert-level performance is attained in increasingly-complex multiagentdomains, notable examples including Go, Poker, and StarCraft II.

Original: This rapid progression is accompanied by a commensurateneed to better understand how such agents attain this performance, to enable theirsafe deployment, identify limitations, and reveal potential means of improving them.
Revised: This rapidprogression is accompanied by a commensurate need to better understand howsuch agents attain this performance, to enable their safe deployment, identifylimitations, and reveal potential means of improving them.

Original: In this paper we take a step back from performance-focused multiagent learning,and instead turn our attention towards agent behavior analysis.
Revised: In this paper wetake a step back from performance-focused multiagent learning, and instead turnour attention towards agent behavior analysis.

Original: We introduce amodel-agnostic method for discovery of behavior clusters in multiagent domains,using variational inference to learn a hierarchy of behaviors at the joint and localagent levels.
Revised: We introduce a model-agnosticmethod for discovery of behavior clusters in multiagent domains, using variationalinference to learn a hierarchy of behaviors at the joint and local agent levels.

Original: Our framework makes no assumption about agents’ underlyinglearning algorithms, does not require access to their latent states or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies, (cid:58)(cid:58)(cid:58) and(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58) ofﬂine observational data.
Revised: Ourframework makes no assumption about agents’ underlying learning algorithms,does not require access to their latent states or models, and can be trained usingentirely ofﬂine observational data.

Original: We illustrate the effectiveness ofour method for enabling the coupled understanding of behaviors at the joint andlocal agent level, detection of behavior changepoints throughout training, discoveryof core behavioral concepts (cid:58) , demonstrate the approach’s scalability to a highdimensional multiagent MuJoCo control domain, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) show (cid:58)(cid:58)(cid:58) [Table]
Revised: We illustrate the effectiveness of our methodfor enabling the coupled understanding of behaviors at the joint and local agentlevel, detection of behavior changepoints throughout training, discovery of corebehavioral concepts (e.g., those that facilitate higher returns), and demonstrate theapproach’s scalability to a high-dimensional multiagent MuJoCo control domain.

Original:  Introduction
Revised:  Introduction

Original: Multiagent approaches have driven numerous advances in artiﬁcial intelligence research, with seminalexamples including TD-gammon [1], DeepBlue [2], AlphaGo [3], AlphaZero [4], Libratus [5],
Revised: Multiagent approaches have driven numerous advances in artiﬁcial intelligence research, with seminalexamples including TD-gammon [1], DeepBlue [2], AlphaGo [3], AlphaZero [4], Libratus [5],

Original: AlphaStar [6], OpenAI Five [7][Table] and Pluribus [8].
Revised: AlphaStar [6], OpenAI Five [7] and Pluribus [8].

Original: During training, many of these approaches seek topush the performance of agents as measured by a reward signal[Table] or derivatives thereof.
Revised: During training, many of these approaches seek topush the performance of agents as measured by a reward signal or derivatives thereof.

Original: Despite this, post-hoc methods that seek to understand agent interactions often use less reward-centrictechniques.
Revised: Despite this, post-hoc methods that seek to understand agent interactions often use less reward-centrictechniques.

Original: Instead, insights are drawn from behavioral analysis to identify unique or interestingagent strategies.
Revised: Instead, insights are drawn from behavioral analysis to identify unique or interestingagent strategies.

Original: Examples include clustering-based analysis of neuron activations and trajectoriesin capture-the-ﬂag [9], inspection of trajectories in a hide-and-seek domain to detect interestingbehaviors such as agents that learn to exploit the underlying physics engine [10], monitoring ofstatistics such as pass ranges and frequencies in humanoid football [11], and analysis of AlphaZero’sacquisition of chess knowledge [12].
Revised: Examples include clustering-based analysis of neuron activations and trajectoriesin capture-the-ﬂag [9], inspection of trajectories in a hide-and-seek domain to detect interestingbehaviors such as agents that learn to exploit the underlying physics engine [10], monitoring ofstatistics such as pass ranges and frequencies in humanoid football [11], and analysis of AlphaZero’sacquisition of chess knowledge [12].

Original: Crucially, such insights are often drawn via manual analysisand detection of behavioral clusters, or use of statistics associated with certain behaviors as deﬁnedby humans experts (e.g., various skills and relevant metrics in humanoid football).
Revised: Crucially, such insights are often drawn via manual analysisand detection of behavioral clusters, or use of statistics associated with certain behaviors as deﬁnedby humans experts (e.g., various skills and relevant metrics in humanoid football).

Original: As evident above, understanding emergent multiagent behaviors is enriched by techniques beyondpure reward-based analysis, as behavioral signiﬁers are not always discernible via rewards.
Revised: As evident above, understanding emergent multiagent behaviors is enriched by techniques beyondpure reward-based analysis, as behavioral signiﬁers are not always discernible via rewards.

Original: Figure 1aprovides intuition on this notion, illustrating returns (sum of rewards) throughout training for amultiagent hill-climbing domain (later described in detail).
Revised: Figure 1aprovides intuition on this notion, illustrating returns (sum of rewards) throughout training for amultiagent hill-climbing domain (later described in detail).

Original: We highlight two independent trainingtrials (A and B) with similar ﬁnal returns.Despite this, comparing the trajectories generated via the [Figure] [Figure]
Revised: We highlight two independent trainingtrials (A and B) with similar ﬁnal returns.

Original: [Figure] d
Revised: Despite this, comparing the trajectories generated via theagents’ deterministic policies at the end of each trial’s training (Fig. 1b) reveals entirely different

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: agents’ deterministic policies at the end of each trial’s training (Fig. 1b) reveals entirely differentbehaviors. Analogous examples are evident in the above works (e.g., Fig.
Revised: behaviors. Analogous examples are evident in the above works (e.g., Fig.

Original: 1 of Baker et al.
Revised: 1 of Baker et al.

Original: [10], wheresubstantial behavior changes occur in multiagent hide-and-seek despite a smooth reward curve).
Revised: [10], wheresubstantial behavior changes occur in multiagent hide-and-seek despite a smooth reward curve).

Original: This paper formalizes the problem of ofﬂine multiagent behavior analysis.
Revised: This paper formalizes the problem of ofﬂine multiagent behavior analysis.

Original: Our proposed algorithm,
Revised: Our proposed algorithm,

Original: Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), learns a hierarchical latent space thatsimultaneously reveals behavior clusters at the joint level (i [Table] e., interactions between agents) and locallevel (i
Revised: Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), learns a hierarchical latent space thatsimultaneously reveals behavior clusters at the joint level (i.e., interactions between agents) and locallevel (i.e., behaviors of individual agents).

Original: [Table] e., behaviors of individual agents). Our method is agnostic of the underlying algorithm used [Table]
Revised: Our method is agnostic of the underlying algorithm usedto generate agents’ behaviors, does not assume availability of a reward signal, and does not requireaccess to agents’ models or internal states.

Original: to generate agents’ behaviors [Table] agents’ models [Table]
Revised: We conduct experiments that investigate the structureof the learned behavior space, which goes beyond prior works on latent-clustering by identifyingrelationships between individual agent and joint behaviors.

Original: Our [Table] experiments (cid:58) investigate the structure of the learned behavior space, which goes beyond priorworks on latent-clustering by identifying relationships between individual agent and joint behaviors. We illustrate that clusters identiﬁed by MOHBA are  useful for highlighting similarities and differencesin behaviors throughout training.
Revised: We illustrate that clusters identiﬁedby MOHBA are meaningful and useful for highlighting similarities and differences in behaviorsthroughout training.

Original: We also quantitatively analyze the completeness of discoveredbehavior clusters by adopting a modiﬁed version of the concept-discovery framework of Yeh et al.[13] to identify interesting behavior concepts in our multiagent setting.(cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58) then (cid:58) test the scalabilityof our approach by using it for behavioral analysis of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) several high-dimensional multiagent MuJoCo(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments [14]. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Revised: We also quantitatively analyze the completeness of discovered behavior clustersby adopting a modiﬁed version of the concept-discovery framework of Yeh et al. [13] to identifyinteresting behavior concepts in our multiagent setting.

Original: Finally,(cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) evaluate (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) approach (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) open-sourced (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) OpenAI (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hide-and-seek(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]
Revised: Finally, we test the scalability of our approachby using it for behavioral analysis of a high-dimensional multiagent MuJoCo environment [14].

Original: closely [Table] [Table]  Related Work
Revised:  Related Work

Original: Signiﬁcant research has been conducted in single-agent skill discovery, which seeks to learn reusablepolicies useful for downstream tasks [15–28].Related approaches discover motor primitives toexpress longer-horizon policies [29], including use of ofﬂine reinforcement learning (RL) for learning useful behaviors [30]. Option discovery methods (cid:58)(cid:58)(cid:58)(cid:58) learn temporally-abstracted actions (i.e., options [31]), chained together to form cohesive skills [32–37]. In contrast to our work, these approachesfocus on maximizing performance in single-agent settings
Revised: Signiﬁcant research has been conducted in single-agent skill discovery, which seeks to learn reusablepolicies useful for downstream tasks [15–28].

Original: [Table] There [Table] also
Revised: Related approaches seek to discover motor primitivesto express longer-horizon policies [29], including use of ofﬂine reinforcement learning (RL) forlearning useful behaviors [30].

Original: [Table] of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) work (cid:58)(cid:58)(cid:58) for(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diverse (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) RL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) settings (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [38–46]; (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) despite (cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) focus(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diversity, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) several (cid:58)(cid:58) of(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] identiﬁers (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generated (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory).
Revised: Option discovery methods focus on learning temporally-abstractedactions (i.e., options [31]), chained together to form cohesive skills [32–37].

Original: (cid:58)(cid:58)(cid:58) By (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) contrast, (cid:58)(cid:58)(cid:58)(cid:58) our(cid:58)(cid:58)(cid:58)(cid:58) work (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) assumes (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) access (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) any (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) underlying (cid:58)(cid:58)(cid:58) [Table]
Revised: In contrast to our work,these approaches focus on maximizing performance in single-agent settings.

Original: [Table] from. [Table] Recent works have also focused on hierarchical skill learning inmultiagent reinforcement learning (MARL).
Revised: Recent works havealso focused on hierarchical skill learning in multiagent reinforcement learning (MARL).

Original: Lee et al. [47] use the mutual information maximizationobjective introduced in Eysenbach et al.
Revised: Lee et al.[38] use the mutual information maximization objective introduced in Eysenbach et al.

Original: [16] to learn multiagent policies.
Revised: [16] to learnmultiagent policies.

Original: Yang et al.
Revised: Yang et al.

Original: [48] use abi-level policy to learn agent skills: a high-level policy ﬁrst generates latent vectors for each agent, 
Revised: [39] use a bi-level policy to learn agent skills: a high-level policy ﬁrstgenerates latent vectors for each agent, and a low-level policy conditions behaviors on said vector toperform the task.

Original: A c t i o n s
Revised: In Wang et al.

Original: S t a t e s
Revised: [40], distinct ‘roles’ are learned for agents to enable decompositionof tasks. Many MARL approaches have noted the emergence of interesting behaviors in multiagentsystems in speciﬁc domains of interest [41–45]. In contrast to our focus, these approaches use RL tomaximize agent performance, rather than understand arbitrary behaviors via ofﬂine analysis.

Original: P o l i
Revised: RL interpretability methods primarily focus on single-agent settings and either modify the RL algorithm itself to increase transparency, or conduct post-hoc explainability [46]. These approaches

Original:  c  ie s L oc a l L a t e n t
Revised: A c t i o n s

Original: J o i n tL a t e n t (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Joint KL Term (Eq. 7)
Revised: S t a t e s

Original: Local KL Term (Eq. 6)
Revised: P o l i

Original: Reconstruction Term (Eq. 5) Joint prior Joint encoder Local encoders
Revised: c i e s

Original: Policies Offline Data Local priors (
Revised: L o c a l L a t e n t J o i n t L a t e n t

Original: [Figure][Figure]
Revised: [Figure]

Original: Mao et al.

Original: [Figure] 50] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) investigate(cid:58)(cid:58)(cid:58) use (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consistent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Figure] (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Revised: represent agent policies as programming languages [47], extract visual summaries of behaviors using‘interestingness’ statistics such as uncertainty in selected actions or the value of state-transitions [48],or combine agent neuron activations with gradient information to construct behavioral embeddings [49].

Original: [Table] 73 Many MARL approaches have noted the emergence of interesting behaviors inmultiagent systems inspeciﬁc domains of interest [51–55]. In contrast to our focus, these approaches use RL to maximizeagent performance, rather than understand arbitrary behaviors via ofﬂine analysis. RL interpretability methods primarily focus on single-agent settings and either modify the RL algorithm itself to increase transparency
Revised: Behavioral clusters in our work share similarities with concept-based explanationapproaches in non-RL domains.

Original: [Table] or conduct post-hoc explainability [56]. These approachesrepresent agent policies as programming languages [57], extract visual summaries of behaviors using‘interestingness’ statistics such as uncertainty in selected actions or the value of state-transitions [58],or combine agent neuron activations with gradient information to construct behavioral embeddings [59].
Revised: Detecting ‘concepts’ in pre-trained models has been explored invision [50–52], discrete games [12], and language [53, 54].

Original: Behavioral clusters in our work share similarities with concept-based explanationapproaches in non-RL domains. Detecting ‘concepts’ in pre-trained models has been explored invision [60–62], discrete games [12], and language [63, 64]. In vision, clustering-based approachesdescribe discovered concepts using examples [60] or use generative modeling to create new datato describe concepts [62].
Revised: In vision, clustering-based approachesdescribe discovered concepts using examples [50] or use generative modeling to create new datato describe concepts [52].

Original: Ghorbani et al.
Revised: Ghorbani et al.

Original: [60] uses a vision-speciﬁc method (i.e., superpixels) tosub-divide input before conducing clustering; while sub-division is less natural in RL, our LSTM and
Revised: [50] uses a vision-speciﬁc method (i.e., superpixels) tosub-divide input before conducing clustering; while sub-division is less natural in RL, our LSTM and

Original: VAE baselines serve as an RL-adopted counterpart of such works.
Revised: VAE baselines serve as an RL-adopted counterpart of such works.

Original: Works using latent-clustering and analysis are also related to ours.
Revised: Works using latent-clustering and analysis are also related to ours.

Original: These include approaches usingmulti-level variational autoencoders [65–68] to learn compositional latent spaces, although not indecision-making domains such as ours.
Revised: These include approaches usingmulti-level variational autoencoders [55–58] to learn compositional latent spaces, although not indecision-making domains such as ours.

Original: Hierarchical latent approaches have been used in single-agent
Revised: Hierarchical latent approaches have been used in single-agent

Original: RL [69, 70].
Revised: RL [59, 60].

Original: Behavior analysis has also been conducted by embedding agent neuron activations intoa low-dimensional space, using a (non-hierarchical) variational approach [9].
Revised: Behavior analysis has also been conducted by embedding agent neuron activations intoa low-dimensional space, using a (non-hierarchical) variational approach [9].

Original: The representationpower of agents’ internal states has also been gauged by predicting future events [11].
Revised: The representationpower of agents’ internal states has also been gauged by predicting future events [11].

Original: Overall, thekey difference between the above works and ours is that our method combined hierarchical learningwith behavior analysis and applies it to the multiagent setting.
Revised: Overall, thekey difference between the above works and ours is that our method combined hierarchical learningwith behavior analysis and applies it to the multiagent setting.

Original:  Ofﬂine Analysis of Multiagent Behaviors
Revised:  Ofﬂine Analysis of Multiagent Behaviors

Original: This section introduces the ofﬂine multiagent behavioral analysis problem and our proposed algorithm.
Revised: This section introduces the ofﬂine multiagent behavioral analysis problem and our proposed algorithm.

Original: Preliminaries.
Revised: Preliminaries.

Original: We ﬁrst formalize the problem of ofﬂine multiagent behavioral analysis.
Revised: We ﬁrst formalize the problem of ofﬂine multiagent behavioral analysis.

Original: Considera rewardless multiagent Markov Decision Process (MA-MDP), deﬁned by tuple ( I , S , A , T ) , where
Revised: Considera rewardless multiagent Markov Decision Process (MA-MDP), deﬁned by tuple ( I , S , A , T ) , where

Original: I = { 1 , ..., N } is the set of N agents, S is the state space,
Revised: I = { 1 , ..., N } is the set of N agents, S is the state space,

Original: A is the action space, and T denotes thestate transition probability function.
Revised: A is the action space, and T denotes thestate transition probability function.

Original: By not relying on the presence of rewards, behaviors generatedeven without reliance on a reward function (e.g., human interactions, or agents using curiosity-basedexploration) can be considered.
Revised: By not relying on the presence of rewards, behaviors generatedeven without reliance on a reward function (e.g., human interactions, or agents using curiosity-basedexploration) can be considered.

Original: We use the term ‘local’ for elements associated with individual agents,and ‘joint’ for those associated with the entire system.
Revised: We use the term ‘local’ for elements associated with individual agents,and ‘joint’ for those associated with the entire system.

Original: At each timestep t ∈ { 0 , . . .
Revised: At each timestep t ∈ { 0 , . . .

Original: , T − 1 } , the agentsexecute joint action a t ∈ A in state s t using joint policy π ( a t | s t ) , causing the state to transition tos t +1 with probability p ( s t +1 | s t , a t ) =
Revised: , T − 1 } , the agentsexecute joint action a t ∈ A in state s t using joint policy π ( a t | s t ) , causing the state to transition tos t +1 with probability p ( s t +1 | s t , a t ) =

Original: T ( s t +1 , a t , s t ) .
Revised: T ( s t +1 , a t , s t ) .

Original: As standard in multiagent frameworks [71], we assume the joint action space factorizes as A = × i
Revised: As standard in multiagent frameworks [61],we assume the joint action space factorizes as A = × i

Original: A i , such that a t = ( a 1 t , . .
Revised: A i , such that a t = ( a 1 t , . .

Original: . , a Nt ) , where i ∈ Iand a it ∈
Revised: . , a Nt ) , where i ∈ Iand a it ∈

Original: A i .
Revised: A i .

Original: Similarly, π i ( a it | s t ) is the local policy for agent i , and π = (cid:81)
Revised: Similarly, π i ( a it | s t ) is the local policy for agent i , and π = (cid:81)

Original: i π
Revised: i π

Original: i is the joint policy.
Revised: i is the joint policy.

Original: Let τ = (cid:0) s 0 , ( a i 0 ) i ∈I , . . .
Revised: Let τ = (cid:0) s 0 , ( a i 0 ) i ∈I , . . .

Original: , ( a iT − 1 ) i ∈I , s T (cid:1) denote a trajectory induced by this process, and D ={ τ 1 , . . .
Revised: , ( a iT − 1 ) i ∈I , s T (cid:1) denote a trajectory induced by this process, and D ={ τ 1 , . . .

Original: , τ K } denote a dataset of K such trajectories.
Revised: , τ K } denote a dataset of K such trajectories.

Original: This dataset may consist of trajectories frommultiple training runs, including variations over agent algorithms, hyperparameters, random seeds,or other factors inﬂuencing emergent behaviors.
Revised: This dataset may consist of trajectories from multiple training runs, including variations over agent algorithms, hyperparameters, random seeds,or other factors inﬂuencing emergent behaviors.

Original: Given dataset D , the ofﬂine multiagent analysisproblem seeks to uncover potential clusters of agent behaviors.
Revised: Given dataset D , the ofﬂine multiagent analysisproblem seeks to uncover potential clusters of agent behaviors.

Original: Approach.
Revised: Approach.

Original: Our approach, called the Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA),uses ofﬂine trajectory data to discover behaviors exhibited by agents at the local and joint level.
Revised: Our approach, called the Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA),uses ofﬂine trajectory data to discover behaviors exhibited by agents at the local and joint level.

Original: We ﬁrst use Fig.
Revised: We ﬁrst use Fig.

Original: 2a to build intuition before discussing technical details.
Revised: 2a to build intuition before discussing technical details.

Original: Let the agent interactionsexhibited in a trajectory τ be encoded by a latent variable, z ω ∈ R D ω , capturing their joint behavior.
Revised: Let the agent interactionsexhibited in a trajectory τ be encoded by a latent variable, z ω ∈ R D ω , capturing their joint behavior.

Original: For example, z ω may encode (at a high level) whether agents were cooperating or competing ina given trajectory; conditioned on joint signal z ω , each agent then exhibits its own local behavior.
Revised: For example, z ω may encode (at a high level) whether agents were cooperating or competing ina given trajectory; conditioned on joint signal z ω , each agent then exhibits its own local behavior.

Original: Let local latent vectors z α = ( z 1 α , . . .
Revised: Let local latent vectors z α = ( z 1 α , . . .

Original: , z Nα ) encode individual agent behaviors, where z iα ∈ R D iα .
Revised: , z Nα ) encode individual agent behaviors, where z iα ∈ R D iα .

Original: Conditioned on the local behavior vector z iα , each agent then executes actions using a behaviorconditioned policy π i ( a i | s, z iα ) .
Revised: Conditioned on the local behavior vector z iα , each agent then executes actions using a behaviorconditioned policy π i ( a i | s, z iα ) .

Original: Given trajectory dataset D , we seek to learn the latent-conditionedpolicies and distributions over latent vectors, such that we can reconstruct any behaviors exhibited bythe agents in D .
Revised: Given trajectory dataset D , we seek to learn the latent-conditionedpolicies and distributions over latent vectors, such that we can reconstruct any behaviors exhibited bythe agents in D .

Original: Thus, latent vectors z ω and z α will encode the agents’ behavioral spaces and, ideally,identify behavioral clusters in the dataset.
Revised: Thus, latent vectors z ω and z α will encode the agents’ behavioral spaces and, ideally,identify behavioral clusters in the dataset.

Original: Given this framework, the joint policy is decomposed,
Revised: Given this framework, the joint policy is decomposed,

Original: [Equation]
Revised: [Equation]

Original: where in (1) we have assumed that each agent’s latent-conditioned policy is conditionally-independentof the high-level latent behavior z ω given its low-level latent z iα (cid:58)(cid:58)(cid:58) (see Appendix A.1.1 (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) discussion).
Revised: where in (1) we have assumed that each agent’s latent-conditioned policy is conditionally-independentof the high-level latent behavior z ω given its low-level latent z iα .

Original: Next, given initial state distribution p ( s 0 ) and latent behavior spaces, the probability of a trajectory τunder joint policy π ( · ) is as follows:
Revised: Next, given initial state distributionp ( s 0 ) and latent behavior spaces, the probability of a trajectory τ under joint policy π ( · ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: We seek to learn the distributions over variables
Revised: We seek to learn the distributions over variables

Original: z α and z ω , alongside the latent-conditioned policiesπ i ( a i | s t , z iα ) , which maximize trajectory probabilities (4).
Revised: z α and z ω , alongside the latent-conditioned policiesπ i ( a i | s t , z iα ) , which maximize trajectory probabilities (4).

Original: In Appendix A.1, we derive the followingvariational lower bound, enabling approximation of these components using parametric models:
Revised: In Appendix A.1, we derive the followingvariational lower bound, enabling approximation of these components using parametric models:

Original: [Equation]
Revised: [Equation]

Original: where q φ ( z iα | τ ) and p θ ( z iα | z ω ) are, respectively, learned encoder (posterior) and prior distributionsover the local behavior latents z iα ; likewise, q φ ( z ω | τ ) and p θ ( z ω ) are, respectively, learned encoderand prior distributions over the joint behavior latent z ω ; β is a KL-weighting term as in β -VAEs [72].
Revised: where q φ ( z iα | τ ) and p θ ( z iα | z ω ) are, respectively, learned encoder (posterior) and prior distributionsover the local behavior latents z iα ; likewise, q φ ( z ω | τ ) and p θ ( z ω ) are, respectively, learned encoderand prior distributions over the joint behavior latent z ω ; β is a KL-weighting term as in β -VAEs [62].

Original: Figure 2b illustrates MOHBA’s model architecture, which is informed by the three bound components (5) to (7).
Revised: Figure 2b illustrates MOHBA’s model architecture, which is informed by the three bound components (5) to (7).

Original: During training, each trajectory τ ∼ D is simultaneously passed through the jointand local encoders, which respectively produce parameters for distributions over z ω and z α (e.g.,parameters of Gaussian distributions).
Revised: During training, each trajectory τ ∼ D is simultaneously passed through the jointand local encoders, which respectively produce parameters for distributions over z ω and z α (e.g.,parameters of Gaussian distributions).

Original: Samples of low-level latent vectors z α are passed to the [Figure] reconstructed agent policies π iθ ( a it | s t , z iα ) , which are trained via the reconstruction component (5).
Revised: Samples of low-level latent vectors z α are passed to thereconstructed agent policies π iθ ( a it | s t , z iα ) , which are trained via the reconstruction component (5).

Original: The local KL-divergence component (6) induces the local encoder distribution (which is conditioneddirectly on τ ) to be similar to the local prior distribution (which is conditioned only on samplesz ω ), thus enabling meaningful correlations between the encoded local and joint latent space, aslater shown.
Revised: The local KL-divergence component (6) induces the local encoder distribution (which is conditioneddirectly on τ ) to be similar to the local prior distribution (which is conditioned only on samplesz ω ), thus enabling meaningful correlations between the encoded local and joint latent space, aslater shown.

Original: Finally, the joint KL-divergence component (7) is akin to that in a standard variationalautoencoder [73, 74].
Revised: Finally, the joint KL-divergence component (7) is akin to that in a standard variationalautoencoder [63, 64].

Original: Overall, MOHBA enables learning of a hierarchical behavioral space (at thejoint and local agent levels, z ω and z α , respectively) that exposes interesting behavioral clusters. 
Revised: Overall, MOHBA enables learning of a hierarchical behavioral space (at thejoint and local agent levels, z ω and z α , respectively) that exposes interesting behavioral clusters. [Figure]

Original:  Experiments
Revised:  Experiments

Original: Our experiments showcase a variety of use-cases for MOHBA, which enrich the typical reward-centricperformance monitoring used during training.
Revised: Our experiments showcase a variety of use-cases for MOHBA, which enrich the typical reward-centricperformance monitoring used during training.

Original: Data generation, training, networks, computationresources, and hyperparameter details are provided in Appendix A.2 of the supplementary materials.
Revised: Data generation, training, networks, computationresources, and hyperparameter details are provided in Appendix A.2 of the supplementary materials.

Original: MARL data generation.
Revised: MARL data generation.

Original: Multiagent trajectory data is generated for each considered domain viathe Acme RL library [75], using the TD3 algorithm [76] in a decentralized fashion, with datasetmanagement handled using RLDS [77].
Revised: Multiagent trajectory data is generated for each considered domain viathe Acme RL library [65], using the TD3 algorithm [66] in a decentralized fashion, with datasetmanagement handled using RLDS [67].

Original: Trajectories are collected at constant intervals throughouttraining, which also enables analysis of behavioral emergence.
Revised: Trajectories are collected at constant intervals throughouttraining, which also enables analysis of behavioral emergence.

Original: We conduct a wide sweep over randomseeds for data generation, yielding a diverse set of trajectories to analyze using MOHBA.
Revised: We conduct a wide sweep over randomseeds for data generation, yielding a diverse set of trajectories to analyze using MOHBA.

Original: MOHBA setup.
Revised: MOHBA setup.

Original: To analyze the above data using MOHBA, we use a GMM (Gaussian Mixture
Revised: To analyze the above data using MOHBA, we use a GMM (Gaussian Mixture

Original: Model) for the joint prior, a bidirectional LSTM (long short-term memory network) with GMM headfor the joint encoder, an MLP (multi-layer perceptron) with Gaussian head for the local priors, abidirectional LSTM with Gaussian head for the local encoder, and an MLP for reconstructed policies.
Revised: Model) for the joint prior, a bidirectional LSTM (long short-term memory network) with GMM headfor the joint encoder, an MLP (multi-layer perceptron) with Gaussian head for the local priors, abidirectional LSTM with Gaussian head for the local encoder, and an MLP for reconstructed policies.

Original: GMMs are used for the joint prior and encoder as they produce discernible joint behavior clusters [78],whereas the conditioning of the local prior on z ω yields such clusters at the local level with a standard
Revised: GMMs are used for the joint prior and encoder as they produce discernible joint behavior clusters [68],whereas the conditioning of the local prior on z ω yields such clusters at the local level with a standard

Original: Gaussian head.
Revised: Gaussian head.

Original: We use parameter-sharing across local priors, local encoders, and reconstructedpolicies, as common in multiagent setups [79], with a unique one-hot vector identiﬁer appended toagent-speciﬁc network inputs to enable heterogeneity in model outputs.
Revised: We use parameter-sharing across local priors, local encoders, and reconstructedpolicies, as common in multiagent setups [69], with a unique one-hot vector identiﬁer appended toagent-speciﬁc network inputs to enable heterogeneity in model outputs.

Original: We conduct a wide sweep of
Revised: We conduct a wide sweep of

Original: MOHBA hypereparameters, including the latent space dimensionality ( 4 and 8 -dimensional), MLPhidden sizes ( 64 and 128 units), LSTM hidden sizes ( 64 and 128 units), Adam optimizer [80] learningrates ( 0 . 001 and 0 . 0001 ), with 3 seeds per parameter set (see appendix for full details).
Revised: MOHBA hypereparameters, including the latent space dimensionality ( 4 and 8 -dimensional), MLPhidden sizes ( 64 and 128 units), LSTM hidden sizes ( 64 and 128 units), Adam optimizer [70] learningrates ( 0 . 001 and 0 . 0001 ), with 3 seeds per parameter set (see appendix for full details).

Original: Independent analysis of joint ( z ω ) and local ( z α ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors.
Revised: Independent analysis of joint ( z ω ) and local ( z α ) behavioral clusters.

Original: We here analyze the hierarchicallatent structure learned by MOHBA.
Revised: We here analyze thehierarchical latent structure learned by MOHBA.

Original: Speciﬁcally, we highlight differences in agent behaviors asidentiﬁed at the joint and local levels, respectively, by z ω and z α .
Revised: Speciﬁcally, we highlight differences in agentbehaviors as identiﬁed at the joint and local levels, respectively, by z ω and z α .

Original: We ﬁrst consider trajectoriesgenerated in a 3-agent hill-climbing domain (our earlier example in Fig. 1b).
Revised: We ﬁrst considertrajectories generated in a 3-agent hill-climbing domain (our earlier example in Fig. 1b).

Original: States and actionscorrespond, respectively, to 2D positions ( x, y ) and forces (∆ x, ∆ y ) imparted by each agent formovement.
Revised: States andactions correspond, respectively, to 2D positions ( x, y ) and forces (∆ x, ∆ y ) imparted by each agentfor movement.

Original: Agents spawn at the origin and are each rewarded for climbing any of three hills within
Revised: Agents spawn at the origin and are each rewarded for climbing any of three hillswithin the domain in episodes of 50 timesteps (with rewards at each position indicated in Fig. 1b). Wegenerate a dataset using 50 independent MARL training trials, each conducted for 1 e 5 environmentsteps; trajectories are saved every 200 steps, yielding 1 . 25 e 6 frames of data ( 25000 trajectories). A

Original:  [Figure]
Revised: B C [Figure]

Original: d
Revised: A B C

Original: [Figure]
Revised: [Figure]

Original: the domain in episodes of 50 timesteps (with rewards at each position indicated in Fig. 1b). Wegenerate a dataset using 50 independent MARL training trials, each conducted for 1 e 5 environmentsteps; trajectories are saved every 200 steps, yielding 1 .
Revised: A B

Original: 25 e 6 frames of data ( 25000 trajectories).
Revised: C [Figure] (d)

Original: Figure 3a highlights several example clusters of joint and local behaviors identiﬁed by MOHBA,using samples of z ω and z α from the joint and local encoders.
Revised: Figure 3a highlights several example clusters of joint and local behaviors identiﬁed by MOHBA,using samples of z ω and z α from the joint and local encoders.

Original: We use Euclidean distances in theoriginal latent spaces to identify nearby vectors, then visualize their 2D projection using principalcomponent analysis.
Revised: We use Euclidean distances in theoriginal latent spaces to identify nearby vectors, then visualize their 2D projection using principalcomponent analysis.

Original: For each cluster, we visualize all associated trajectories in Fig.
Revised: For each cluster, we visualize all associated trajectories in Fig.

Original: 3b, thus enablinganalysis of behaviors captured in latent space.
Revised: 3b, thus enablinganalysis of behaviors captured in latent space.

Original: Clusters A, B, and C correspond to examples of jointbehaviors ( z ω ).
Revised: Clusters A, B, and C correspond to examples of jointbehaviors ( z ω ).

Original: Cluster A contains trajectories early in training, where agents behave nearly randomlyand have not learned to converge to a particular rewarding hill in the domain.
Revised: Cluster A contains trajectories early in training, where agents behave nearly randomlyand have not learned to converge to a particular rewarding hill in the domain.

Original: In Cluster B, all threeagents have learned to navigate towards the top-left hill.
Revised: In Cluster B, all threeagents have learned to navigate towards the top-left hill.

Original: In Cluster C, agents 0 and 2 show preferencefor the bottom-left hill, whereas agent 1 prefers the right hill.
Revised: In Cluster C, agents 0 and 2 show preferencefor the bottom-left hill, whereas agent 1 prefers the right hill.

Original: Overall, the behavioral clusters at thejoint level capture the collective behaviors of agents in a meaningful manner.
Revised: Overall, the behavioral clusters at thejoint level capture the collective behaviors of agents in a meaningful manner.

Original: Next, we analyze individual agent clusters.
Revised: Next, we analyze individual agent clusters.

Original: Note that for all agents, three prominent clusters areapparent in their respective z α spaces, as each training trial in the data generation process can lead tovarious agent-wise hill preferences.
Revised: Note that for all agents, three prominent clusters areapparent in their respective z α spaces, as each training trial in the data generation process can lead tovarious agent-wise hill preferences.

Original: We compare two such clusters, D and E, for agent 1, observingin the trajectory plots that these corresponds to this agent preferring the top-left and bottom-left hills,respectively.
Revised: We compare two such clusters, D and E, for agent 1, observingin the trajectory plots that these corresponds to this agent preferring the top-left and bottom-left hills,respectively.

Original: Similarly, Cluster F corresponds to trajectories where agent 2 prefers the bottom-left hill.
Revised: Similarly, Cluster F corresponds to trajectories where agent 2 prefers the bottom-left hill.

Original: These results illustrate that the local latents reasonably disentangle each agent’s observed behaviors.
Revised: These results illustrate that the local latents reasonably disentangle each agent’s observed behaviors.

Original: [Table]
Revised: [Table]

Original: Coupled analysis of joint ( z ω ) and local ( z α ) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors.
Revised: Coupled analysis of joint ( z ω ) and local ( z α ) behavioralclusters.

Original: The above experiments independently analyzed the joint andlocal latent spaces.
Revised: The above experiments independently analyzed thejoint and local latent spaces.

Original: We can also concurrently analyze them tobetter understand local agent contributions to joint behaviors.
Revised: We can also concurrently analyzethem to better understand local agent contributions to jointbehaviors.

Original: Consider a two-agent domain with close inter-agent coordination (visualized in the top of Fig. 4a), with state and actionsspaces similar to the hill domain and episodes consisting of50 timesteps.
Revised: Consider a two-agent domain with close interagent coordination (visualized in the top of Fig. 4a), with stateand actions-spaces similar to the hill domain and episodesconsisting of 50 timesteps.

Original: Three regions are deﬁned in this domain: A and B (circular regions), and C (regionexterior to the circles).
Revised: Three regions are deﬁned in this domain: A and B (circular regions), and C (region exterior to the circles).

Original: When an agent enters a given region, it ‘activates’ the corresponding strategyin Table 1, with agents receiving rewards at each timestep according to the joint strategy they haveactivated.
Revised: When an agent enters a given region, it ‘activates’ the correspondingstrategy in Table 1, with agents receiving rewards at each timestep according to the joint strategy theyhave activated.

Original: For example, if both agents enter region A , they each receive a reward of 1 , whereas ifone agent enters A while the other is in exterior region C , neither receives a reward.
Revised: For example, if both agents enter region A , they each receive a reward of 1 , whereas if one agent enters A while the other is in exterior region C , neither receives a reward.

Original: This domaininvolves a signiﬁcant degree of coordination as agents must discover the rewarding regions, whilereceiving a sparse reward signal until a valid combination of strategies is discovered.
Revised: This domaininvolves a signiﬁcant degree of coordination as agents must discover the rewarding regions, whilereceiving a sparse reward signal until a valid combination of strategies is discovered.

Original: There is also potential for miscoordination: navigating to region B is rewarding assuming the other agent navigatesto A , but yields 0 reward if the other agent instead navigates to B (potentially destabilizing training).
Revised: There is alsopotential for miscoordination: navigating to region B is rewarding assuming the other agent navigatesto A , but yields 0 reward if the other agent instead navigates to B (potentially destabilizing training).

Original: We run 50 independent MARL sweeps in this domain, yielding 1 e 6 data frames ( 20000 trajectories).
Revised: We run 50 independent MARL sweeps in this domain, yielding 1 e 6 data frames ( 20000 trajectories).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, MOHBA discovers three dominant joint behavior clusters z ω here.
Revised: 4, MOHBA discovers three dominant joint behavior clusters z ω here.

Original: In each of
Revised: In each of

Original: Figs. 4a to 4c, we highlight one of these clusters and its corresponding local behavior latents z α foreach agent.
Revised: Figs. 4a to 4c, we highlight one of these clusters and its corresponding local behavior latents z α foreach agent.

Original: MOHBA reveals that across the dataset, the agents have learned to cover all 3 optimaljoint behaviors in Table 1: ( A, A ) , ( A, B ) , and ( B, A ) .
Revised: MOHBA reveals that across the dataset, the agents have learned to cover all 3 optimaljoint behaviors in Table 1: ( A, A ) , ( A, B ) , and ( B, A ) .

Original: Moreover, despite each agent discovering twolocal behaviors ( A and B ), z ω highlights only the 3 observed joint behaviors (i.e., does not simplyhighlight 4 clusters consisting of the Cartesian product of individual agents’ behavior spaces).
Revised: Moreover, despite each agent discovering twolocal behaviors ( A and B ), z ω highlights only the 3 observed joint behaviors (i.e., does not simplyhighlight 4 clusters consisting of the Cartesian product of individual agents’ behavior spaces).

Original: Behavior emergence throughout MARL training.
Revised: Behavior emergence throughout MARL training.

Original: We next use MOHBA’s latent space to inspectbehavior emergence during MARL training.
Revised: We next use MOHBA’s latent space to inspectbehavior emergence during MARL training.

Original: Figure 4d visualizes the training progression of two
Revised: Figure 4d visualizes the training progression of two

Original: MARL runs used for data generation, tracking behavior changes throughout.
Revised: MARL runs used for data generation, tracking behavior changes throughout.

Original: In Training Run 1 (toppanel of Fig. 4d), the agents converge to and maintain a ﬁxed joint behavior throughout training.
Revised: In Training Run 1 (toppanel of Fig. 4d), the agents converge to and maintain a ﬁxed joint behavior throughout training.

Original: Bycontrast, Training Run 2 (bottom panel) has numerous behavior changepoints, where agents ﬂip backand forth between preferring one of two clusters in z ω .
Revised: Bycontrast, Training Run 2 (bottom panel) has numerous behavior changepoints, where agents ﬂip backand forth between preferring one of two clusters in z ω .

Original: Concurrently inspecting the z α space for
Revised: Concurrently inspecting the z α space for

Original: Training Run 2, we observe that agent 0 converges to and maintains a consistent behavior, whereasagent 1 changes its preference sporadically, also explaining the detected changes in joint behaviors.
Revised: Training Run 2, we observe that agent 0 converges to and maintains a consistent behavior, whereasagent 1 changes its preference sporadically, also explaining the detected changes in joint behaviors.

Original: Baseline comparisons.
Revised: Baseline comparisons.

Original: We next compare against baselines previously used for multiagent behavioral analysis in the literature [9, 11].
Revised: We next compare against baselines previously used for multiagent behavioral analysis in the literature [9, 11].

Original: An LSTM baseline conducts next-action prediction at eachstep in a trajectory τ , using an action-prediction loss (APL), deﬁned as the L 2 loss over predictedvs. ground truth actions.[Table] This (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Revised: An LSTM baseline conducts next-action prediction at eachstep in a trajectory τ , using an action-prediction loss (APL), deﬁned as the L 2 loss over predicted vs.ground truth actions.

Original: [Table]

Original: in [Table] [Table] A ﬂat-VAE baseline provides a non-hierarchical ablation of MOHBA that simply feedsthe joint latent z ω (rather than z iα ) to reconstructed agent policies (i
Revised: A ﬂat-VAE baseline provides a non-hierarchical ablation of MOHBA that simplyfeeds the joint latent z ω (rather than z iα ) to reconstructed agent policies (i.e., π iθ ( a i | s, z ω ) ); we useour usual loss (with the local KL term (6) removed) to train this VAE.

Original: [Table] e., π iθ ( a i | s, z ω ) ); we use ourusual loss (with the local KL term (6) removed) to train this VAE. We conduct a hyperparametersweep for the baselines (see Appendix A.2), reporting the best results averaged over 3 random seeds
Revised: We conduct a hyperparametersweep for the baselines (see Appendix A.2), reporting the best results averaged over 3 random seeds.

Original: [Table][Table]
Revised: [Table]

Original: Comparisons are conductedat the joint behavior level:for the LSTM, we use the ﬁnal hidden state as an encoding of each trajectory; forour method and the VAE, weuse z ω directly.
Revised: Comparisons are conductedat the joint behavior level:for the LSTM, we use the ﬁnal hidden state as an encoding of each trajectory; forour method and the VAE, weuse z ω directly.

Original: Table 2 compares the methods in the hillclimbing and coordination game domains.
Revised: Table 2 compares the methods in the hillclimbing and coordination game domains.

Original: Here we note the APL, which measures how well eachmethod reconstructs ground truth policies.
Revised: Here we note the APL, which measures how well eachmethod reconstructs ground truth policies.

Original: Additionally, we use K-means (sweeping over the # ofclusters) to identify behavior clusters for each method, then report the intra-cluster trajectory distance(ICTD), deﬁned as the average distance of all trajectories in a cluster from the mean trajectory insaid cluster (akin to intra-cluster point scatter, a common cluster analysis statistic [81]) .
Revised: Additionally, we use K-means (sweeping over the # ofclusters) to identify behavior clusters for each method, then report the intra-cluster trajectory distance(ICTD), deﬁned as the average distance of all trajectories in a cluster from the mean trajectory insaid cluster (akin to intra-cluster point scatter, a common cluster analysis statistic [71]) .

Original: Combined,these measures provide a proxy for evaluating the latent representations in terms of enabling accuratereconstructions (APL) while clustering similar trajectories together (ICTD).
Revised: Combined,these measures provide a proxy for evaluating the latent representations in terms of enabling accuratereconstructions (APL) while clustering similar trajectories together (ICTD).

Original: MOHBA signiﬁcantlyoutperforms the LSTM and VAE baselines in terms of APL, while also clustering similar trajectoriesat the z ω level in terms of ICTD.
Revised: MOHBA signiﬁcantlyoutperforms the LSTM and VAE baselines in terms of APL, while also clustering similar trajectoriesat the z ω level in terms of ICTD.

Original: We provide additional results including APL throughout training,full ICTD sweeps, and visualizations of the latent spaces for the baselines in Appendix A.3.2. [Table]
Revised: We provide additional results including APL throughout training,full ICTD sweeps, and visualizations of the latent spaces for the baselines in Appendix A.3.

Original: discovery.
Revised: Discovering behavioral concepts for user-desired characteristics.

Original: Next, [Table] we [Table]
Revised: Next, consider a scenariowhere a user wishes to identify behavioral concepts associated with multiagent characteristics ofinterest

Original: the [Table] space.
Revised: (e.g., concepts that yield high agent returns).

Original: We adopt253 the completeness-aware (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) concept (cid:58) explanations framework of Yeh et al.
Revised: We adopt the completeness-aware concept-basedexplanations framework of Yeh et al.

Original: [13], with slight changes tomake it amenable to this setting (see Appendix A.2.5).
Revised: [13], with slight changes to make it amenable to this setting (see Appendix A.2.5 for details).

Original: At a high-level, given a set of inputs, a set ofconcept vectors C in the same space as inputs, and prediction targets (e.g., (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain (cid:58) characteristicsof interest) , Yeh et al.
Revised: At a high-level, given a set of inputs, a set of concept vectors C in thesame space as inputs, and prediction targets (e.g., characteristics of interest, such as agent returns inour domain), Yeh et al.

Original: [13] deﬁne a framework to compute class-conditioned Shapley values, called
Revised: [13] deﬁne a framework to compute class-conditioned Shapley values, called

Original: ConceptSHAP.
Revised: ConceptSHAP.

Original: To do so, every input is projected onto each of the concept vectors, yielding a vectorof ‘concept scores’, which is then passed to a simple prediction head g
Revised: To do so, every input is projected onto each of the concept vectors, yielding a vectorof ‘concept scores’, which is then passed to a simple prediction head g

Original: (e.g., small MLP or linearmodel) to predict the targets and compute the ConceptSHAP. ConceptSHAP provides numericalscores interpreted as the importance of each concept in C for predicting a given target class, which isuseful for identifying key concepts (and nearby inputs) associated with certain domain characteristics.
Revised: (e.g., small MLP or linearmodel) to predict the targets and compute the ConceptSHAP. ConceptSHAP provides numerical

Original: The concept explanation framework of Yeh et al.
Revised: The concept explanation framework of Yeh et al.

Original: [13] is then used to ﬁrst predict the correct classes (using only z ω , rather than the trajectory τ , as input), then identify core concepts related to each class. 
Revised: [13] is then used to ﬁrst predict the correct classes (using only z ω , rather than the trajectory τ , as input), then identify core concepts related to each class. [Table]

Original: [Table][Table]
Revised: [Table]

Revised: scores interpreted as the importance of each concept in C for predicting a given target class, which isuseful for identifying key concepts (and nearby inputs) associated with certain domain characteristics.

Original: In Table 3, we use this technique to identify behavioral concepts in z ω space associated with severalcharacteristics of interest in the hill-climbing domain.
Revised: In Table 3, we use this technique to identify behavioral concepts in z ω space associated with severalcharacteristics of interest in the hill-climbing domain.

Original: The set of concepts C
Revised: The set of concepts C

Original: we consider aregenerated using K-means in the z ω space (see Appendix A.2.5 for details), and a simple 2-layer MLPwith 8-hidden units for prediction head g .
Revised: we consider aregenerated using K-means in the z ω space (see Appendix A.2.5 for details), and a simple 2-layer MLPwith 8-hidden units for prediction head g .

Original: The ﬁrst row of Table 3 shows classes corresponding toincreasing levels of agent dispersion in the domain (with dispersion deﬁned as the sum of L 2 -distancesof agents from their centroid at the ﬁnal timestep of the episode).
Revised: The ﬁrst row of Table 3 shows classes corresponding toincreasing levels of agent dispersion in the domain (with dispersion deﬁned as the sum of L 2 -distancesof agents from their centroid at the ﬁnal timestep of the episode).

Original: For each trajectory τ , we computethe associated dispersion, creating 5 classes of equal-sized bins (with labels 0 through 4 mapping tobins of increasing dispersion).
Revised: For each trajectory τ , we computethe associated dispersion, creating 5 classes of equal-sized bins (with labels 0 through 4 mapping tobins of increasing dispersion).

Original: We create a train-validation split of 80-20, then train the MLP g via asoftmax-cross entropy loss to predict the correct classes using only z ω as input (rather than the actualtrajectory τ ).
Revised: We create a train-validation split of 80-20, then train the MLP g via asoftmax-cross entropy loss to predict the correct classes using only z ω as input (rather than the actualtrajectory τ ).

Original: We attain a validation set accuracy of 54.27%, signifying the predictive capabilities ofz ω .
Revised: We attain a validation set accuracy of 54.27%, signifying the predictive capabilities ofz ω .

Original: Next, we compute the class-conditioned ConceptSHAP, thus identifying the top-scoring conceptvector for each class.
Revised: Next, we compute the class-conditioned ConceptSHAP, thus identifying the top-scoring conceptvector for each class.

Original: In Table 3, we visualize the 20 trajectories with the closest z ω to the top-scoringconcept for each class (with the mean agent dispersion across these trajectories listed under eachimage).
Revised: In Table 3, we visualize the 20 trajectories with the closest z ω to the top-scoringconcept for each class (with the mean agent dispersion across these trajectories listed under eachimage).

Original: Intuitively, the identiﬁed concepts involve agents becoming increasingly dispersed, withagents ﬁrst nearly stationary at the center (class 0), then all converging to the same hill (class 1),spreading across two hills (class 2), and ﬁnally covering all hills (classes 3 and 4).
Revised: Intuitively, the identiﬁed concepts involve agents becoming increasingly dispersed, withagents ﬁrst nearly stationary at the center (class 0), then all converging to the same hill (class 1),spreading across two hills (class 2), and ﬁnally covering all hills (classes 3 and 4).

Original: The second row of Table 3 repeats the same experiment, now using classes associated with the sumof returns of agents.
Revised: The second row of Table 3 repeats the same experiment, now using classes associated with the sumof returns of agents.

Original: We attain a validation set accuracy of 60.98%, and similarly observe behavioraltrajectories associated with generally increasing reward by identifying top clusters for each class(with some overlaps, e.g., classes 1 and 2 with high standard deviation in returns).
Revised: We attain a validation set accuracy of 60.98%, and similarly observe behavioraltrajectories associated with generally increasing reward by identifying top clusters for each class(with some overlaps, e.g., classes 1 and 2 with high standard deviation in returns).

Original: Overall, this lineof experiments provides both a means of quantifying the representational capacity of z ω , and also ofidentifying clusters of [Table] certain [Table]
Revised: Overall, this lineof experiments provides both a means of quantifying the representational capacity of z ω , and also ofidentifying clusters of behavioral concepts that summarize key user-deﬁned characteristics.

Original: Scalability to high-dimensional domains.
Revised: Scalability to high-dimensional domains.

Original: We next test MOHBA’s scalability to high-dimensionaldomains using the 2-Agent HalfCheetah multiagent MuJoCo domain [14], where agents 0 and 1,respectively, control the back and front limbs of a cheetah to coordinate movement.
Revised: We next test MOHBA’s scalability to high-dimensionaldomains using the 2-Agent HalfCheetah multiagent MuJoCo domain [14], where agents 0 and 1,respectively, control the back and front limbs of a cheetah to coordinate movement.

Original: Each agent’s stateconsists of a 6D vector summarizing velocity and position information for the 3 joints it controls,with its 3D action space corresponding to motor torques applied to said joints.
Revised: Each agent’s stateconsists of a 6D vector summarizing velocity and position information for the 3 joints it controls,with its 3D action space corresponding to motor torques applied to said joints.

Original: The agents coordinateto maximize the forward-speed of the cheetah over episodes of 200 timesteps each.
Revised: The agents coordinateto maximize the forward-speed of the cheetah over episodes of 200 timesteps each.

Original: We generate datausing 30 independent trials of MARL training, collecting 1 e 5 total trajectories to train MOHBA.
Revised: We generate datausing 30 independent trials of MARL training, collecting 1 e 5 total trajectories to train MOHBA.

Original: Figure 5a visualizes the behavior space learned by MOHBA, wherein we observe several clusters.
Revised: Figure 5a visualizes the behavior space learned by MOHBA, wherein we observe several clusters.

Original: We highlight three example trajectories stemming from distinct clusters in the joint behavior spacez ω , showing the corresponding HalfCheetah behaviors in Fig.
Revised: We highlight three example trajectories stemming from distinct clusters in the joint behavior spacez ω , showing the corresponding HalfCheetah behaviors in Fig.

Original: 5b.
Revised: 5b.

Original: The left panel of each exampleprovides a view of the cheetah’s overall movement, where we observe key behavior differences: in 
Revised: The left panel of each exampleprovides a view of the cheetah’s overall movement, where we observe key behavior differences: in [Figure] Example A (see animated version here)

Original: Example  A, the cheetah runs forward using subtle vibration of its limbs; in Example B, it bouncesforward with its torso arched up due to its front limb (agent 1) being more extended than its back limb(agent 0); in Example C, it moves closer to the ground with its torso arched down.
Revised: Example B (see animated version here) Example C (see animated version here) [Figure] Example A, the cheetah runs forward using subtle vibration of its limbs; in Example B, it bouncesforward with its torso arched up due to its front limb (agent 1) being more extended than its back limb(agent 0); in Example C, it moves closer to the ground with its torso arched down.

Original: In the latent space(Fig.
Revised: In the latent space(Fig.

Original: 5a), we observe that for Examples B and C, the z 0 α
Revised: 5a), we observe that for Examples B and C, the z 0 α

Original: (back limb) latents are close to one another,while the z 1 α (front limb) latents are far apart.
Revised: (back limb) latents are close to one another,while the z 1 α (front limb) latents are far apart.

Original: To investigate these local agent behavioral differences, [Figure] Example A (see animated version here) Example B (see animated version here)
Revised: To investigate these local agent behavioral differences,the right panel of Fig.

Original: [Figure]

Original: (see animated version here [Figure] the right panel of Fig. 5b provides an overlaid view of the same frames, with the torso now aligned(making it easier to discern the behaviors of agent 0 vs. agent 1).
Revised: 5b provides an overlaid view of the same frames, with the torso now aligned(making it easier to discern the behaviors of agent 0 vs. agent 1).

Original: Here we observe that the back limb(agent 0) behaves similarly across both examples, while the front limb (agent 1) stays much closer tothe head for Example C, in contrast to Example B, which coincides with the ﬁndings in the z α space.
Revised: Here we observe that the back limb(agent 0) behaves similarly across both examples, while the front limb (agent 1) stays much closer tothe head for Example C, in contrast to Example B, which coincides with the ﬁndings in the z α space.

Original:  Discussion
Revised:  Discussion

Original: [Figure]

Original: Our proposed method, MOHBA, leverages  trajectory data to better understand (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table] [Figure] behaviorsduring and after training.
Revised: Our proposed method, MOHBA, leverages multiagent trajectory data to better understand agentbehaviors, both during and after training.

Original: MOHBA assumes no knowledge of  agents’ underlying training algorithms,does not require access to their hidden states or internal models, and applies even to reward-free settings.
Revised: MOHBA assumes no knowledge of the agents’ underlyingtraining algorithms, does not require access to their hidden states or internal models, and applies evento reward-free settings.

Original: Our experiments showcased a variety of applications of MOHBA, including the analysis of jointand local agent behaviors, monitoring of behavior emergence throughout training, discovery of behavioral concepts associated with certain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) criteria, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) disentanglement (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stemming(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) externally-available (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) open-source (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) those (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]
Revised: Our experiments showcased a variety of applications of MOHBA, includingthe analysis of joint and local agent behaviors, monitoring of behavior emergence throughout training,and discovery of behavioral concepts associated with certain user-speciﬁed criteria.

Original: While we believe our approach is an important step towards increasing the understanding of multiagent [Table] systems, there are several limitations and potential societal impacts of note.[Table]
Revised: While we believe our approach is an important step in terms of increasing the understanding ofmultiagent systems, there are several limitations and potential societal impacts of note.

Original: One (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) limitation is relatedto the collection of a dataset of agent behaviors.By collecting this dataset throughout agent training,we ensure that the most prominent behaviors that can occur in a given environment can be identiﬁedusing our method. However, this requires both storage of larger datasets than usual agent training
Revised: A keylimitation of our approach is related to the collection of a dataset of agent behaviors.

Original: [Table]

Original: requires, and also the use of a large enough sweep of agent training to capture key behaviors. Itmay be interesting to consider follow-ups to our model that learn behavioral clusters in a streamingfashion, thus building knowledge of behaviors over time and permitting older data to be discardedthroughout agent training. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Moreover, (cid:58)(cid:58) in [Table]
Revised: By collectingthis dataset throughout agent training, we ensure that the most prominent behaviors that can occur ina given environment can be identiﬁed using our method.

Original: our [Table] datasets, [Table]
Revised: However, this requires both storage of largerdatasets than usual agent training requires, and also the use of a large enough sweep of agent trainingto sufﬁciently capture key behaviors.

Original: we [Table] collected (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) uniform (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intervals(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) original (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training.(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Revised: It may be interesting to consider follow-ups to our model thatlearn behavioral clusters in a streaming fashion, thus building knowledge of behaviors over time andpermitting older data to be discarded throughout agent training.

Original: However, (cid:58)(cid:58) it (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) might (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interesting (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consider (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) scheme [Table] that [Table]
Revised: Moreover, the study of behavioral interactions in multiagent systems can potentially be used forboth positive and negative societal applications.

Original: does

Original: [Table]

Original: so

Original: [Table] MOHBA [Table] to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) detect (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) divergences (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) previously-seen (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) during (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) future (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58) runs, (cid:58)(cid:58)(cid:58) and(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collecting (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) when (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) divergences (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) occur).
Revised: For example, such an approach could be used toprevent certain undesirable behaviors by agents that interact with humans (e.g, self-driving cars), butpotentially also used by adversaries to predict and exploit other behaviors (e.g., exploiting certainhuman preferences for harm), or even inadvertently cause harm due to misinterpretation of certainbehavioral modes.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Finally, (cid:58)(cid:58) it (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) might (cid:58)(cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interesting (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) enable(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) ﬁrst (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learn (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58) / (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘skills’ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) proposed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) method, (cid:58)(cid:58)(cid:58) and(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) leverage (cid:58)(cid:58)(cid:58)(cid:58) them (cid:58)(cid:58) to(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) transfer (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) related (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) downstream (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) tasks (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (potentially (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sample-efﬁciently (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training [Table]
Revised: As such, further research and evaluation will be required prior to deploymentof this and related behavioral analysis approaches to human-facing domains.

Original: from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) scratch).

Original: Moreover, the study of behavioral interactions in multiagent systems can potentially be used forboth positive and negative societal applications. For example, such an approach could be used toprevent certain undesirable behaviors by agents that interact with humans (e.g, self-driving cars), butpotentially also used by adversaries to predict and exploit other behaviors (e.g., exploiting certainhuman preferences for harm), or even inadvertently cause harm due to misinterpretation of certainbehavioral modes. As such, further research and evaluation will be required prior to deploymentof this and related behavioral analysis approaches to human-facing domains. Nonetheless, as agentcapabilities continue to grow, our view is that behavioral analysis of multiagent systems will becomeincreasingly important and should complement traditional reward-based performance monitoring.
Revised: Nonetheless, as agentcapabilities continue to grow, our view is that behavioral analysis of multiagent systems will becomeincreasingly important and should complement traditional reward-based performance monitoring.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We include these in the Discus-
Revised: [Yes] We include these in the Discus-

Original: sions section.
Revised: sions section.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We include
Revised: [Yes] We include

Original: these in the discussions section.
Revised: these in the discussions section.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] These are
Revised: [Yes] These are

Original: provided in the main paper and the proofs included in the appendix.
Revised: provided in the main paper and the proofs included in the appendix.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Proofs are provided in
Revised: [Yes] Proofs are provided in

Original: the appendix.
Revised: the appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?

Original: [No] We provide
Revised: [No] We provide

Original: details for experiment reproducibility in Appendix A.2.
Revised: details for experiment reproducibility in Appendix A.2.

Original: We intend to include code forthe model upon paper acceptance.
Revised: We intend to include code for the model upon paper acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] We provide all training details in Appendix A.2.(c)
Revised: [Yes] We provide all training details in Appendix A.2.

Original:  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] We provide all computationaldetails in Appendix A.2.
Revised: [Yes] We provide all computational details in Appendix A.2.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: Our work does notuse existing assets.(b)
Revised: Our work does not use existing assets.

Original:  Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.5.
Revised: Our work does not use existing assets.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A] Our work does not crowdsourced or human subject data. (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table] Board (IRB) approvals, if applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not crowdsourced or[Table] human subject data.[Table]
Revised: Our work does not crowdsourced or human subject data.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table] spent on participant compensation?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not crowdsourced orhuman [Table] subject data. [Table]
Revised: Our work does not crowdsourced or

Original: A

Original: Appendix for ‘Beyond Rewards: a Hierarchical Perspective on Ofﬂine

Original: Multiagent Behavioral Analysis’

Original: [Table]

Original: each (cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ultimately (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) informed (cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) rather (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) global (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (joint) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information.

Original: (cid:58)(cid:58)

Original: In (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) setting, (cid:58)(cid:58)(cid:58) z iα (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: between (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) our(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) setting, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provides (cid:58) a

Original: [Table]

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) signal (cid:58)(cid:58)(cid:58)

Original: to

Original: [Table]

Original: the

Original: [Table]

Original: policy

Original: [Table]

Original: implicitly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) through (cid:58)(cid:58)(cid:58)(cid:58) z iα .

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: space (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) z iα , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sampled (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) used(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) inform (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58) how (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behave (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) locally (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) order (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) achieve (cid:58)(cid:58)(cid:58)

Original: the

Original: [Table]

Original: joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observed.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Overall, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) relationship (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) between (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) two (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) implies (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) makes (cid:58)(cid:58)(cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: This section details the derivation of the variational lower bound described in Section 3 of the maintext, which is used for training MOHBA.

Original: Similar to traditional variational autoencoder approaches [73, 74], we approximate the maximizationof the latent-conditioned trajectory probability (4) in the main text using the evidence lower bound

Original: [Equation]

Original: where q φ ( ·| τ ) and p θ ( · ) are, respectively, learned posterior and prior distributions.

Original: Using (4) from the main text to expand this expression yields

Original: [Equation]

Original: Note that the terms p ( s 0 ) and p ( s t +1 | s t , a t ) stem from the underlying MA-MDP environment, andthus cannot be optimized via parameters φ and θ .

Original: Dropping these extraneous terms yields,

Original: [Equation]

Original: The ﬁrst term simply relates to trajectory reconstruction, inducing our agent-wise policies π iθ tobehave similarly to the observed trajectories in the dataset.

Original: The second term is a regularization terminvolving our two latent parameters, which we further simplify using assumptions used in previousmulti-level VAE models [67].

Original: First, we assume that the posterior distribution is factorizable whenconditioned on a particular trajectory, as follows,

Original: [Equation]

Original: and similarly for the prior,

Original: [Equation]

Original: We next simplify the KL-divergence component of (10) as follows by combining it with (12) and (14):

Original: [Equation]

Original: (cid:34)(cid:88)

Original: i

Original: D

Original: [Equation]

Original: Combining this result with (10) and additionally modulating the KL-terms in the lower bound as inβ -VAEs [72], our overall objective simpliﬁes to

Original: [Equation] A.2 Experiment Details and Hyperparameters This section provides an overview of the details for our experiments. A.2.1 Data Generation Details
Revised: human subject data.

Original: The multiagent trajectory data analyzed in this paper is generated using the Acme RL library [75],using the TD3 algorithm [76] in a decentralized fashion, with dataset management handled using

Original: RLDS [77].

Original: Each agent uses a 2-layer MLP (256 hidden units for each layer) for its TD3 network(using the vanilla TD3 network speciﬁcations in Acme), with the hyperparameters used for datageneration summarized for each domain in Table 4.

Original: Note that the large number of training trials/seedsused in this data generation pipeline serve to produce a wide variety of agent behaviors in the datasetsubsequently analyzed by MOHBA.

Original: [Table]

Original: [Table]

Original: [Table]

Original: A.2.2 MOHBA Hyperparameters

Original: We conduct a wide hyperparameter sweep for training MOHBA itself, summarized in Table 5.

Original: Ourjoint and local encoder models each consist of a bidirectional LSTM, followed by a 2-layer MLPhead for mapping to latent distribution parameters; a GMM head is used for the joint encoder, and a

Original: Gaussian for the local encoder.

Original: Our local prior model consists of a 2-layer MLP with Gaussian head,with our joint prior model simply being the learned parameters of a GMM.

Original: All models use ReLU forintermediate layer activations.

Original: Note that we found that cyclically-annealing [82] the β term in ourvariational lower bound from 0 to the values speciﬁed in Table 5 to help avoid KL-vanishing.

Original: All

Original: MOHBA and baseline training is conducted using the Adam optimizer [80], with Adam parametersβ 1 = 0 .

Original: 9 , β 2 = 0 .

Original: 999 , gradient clipping using a max global norm threshold of 10 . 0 , and learningrates swept over as indicated in the tables.

Original: A.2.3 Baseline Model Hyperparameters

Original: We similarly conduct a wide sweep over hyperparameters for the considered baselines, summarizedin Table 6.

Original: [Table]

Original: [Table]

Original: A.2.4 Computational Details

Original: For MARL trajectory data generation, we used an internal CPU cluster for both the 3-agent hillclimbing and 2-agent coordination domains, using TPUs for only the multiagent MuJoCo datageneration.

Original: For training MOHBA itself, we used TPUs for the 3-agent hill-climbing and 2-agentcoordination domains, and a Tesla V100 GPU cluster for the MuJoCo environment.

Original: For training the

Original: LSTM and VAE baselines, we used Tesla P100 GPU clusters.

Original: A.2. Concept Discovery Framework

Original: This section describes details of how we apply the completeness-aware concept-based explanationframework of Yeh et al.

Original: [13] to discover interesting concepts in our setting.

Original: Given a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) characteristic (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interest

Original: [Table]

Original: [Table]

Original: [Table]

Original: agents)

Original: , we deﬁne a training set 711 consisting of joint latents z ω and class labels y

Original: (e.g., classes corresponding to different intervalsof team returns).

Original: Yeh et al.

Original: [13] seek to identify a set of concept vectors that are sufﬁcient forpredicting the labels given the inputs.

Original: Speciﬁcally, they let C =

Original: { c j } mj

Original: =1 denote the set of concepts,which are unique vectors where c j ∈ R D ω ∀ j .

Original: Normalizing z ω , we can then use the inner product(cid:104) z ω , c j (cid:105) as a similarity measure between z ω and concept c j .

Original: The concept product is deﬁned ν c ( z ω ) =

Original: TH ( (cid:104) z ω , c j (cid:105) , κ ) ∈ R m , where TH ( · , κ ) clips values less than κ to 0 .

Original: Normalizing the concept productyields the concept score ˆ ν c ( z ω ) =

Original: ν c ( z ω ) /

Original: || ν c

Original: ( z ω )

Original: || 2 ∈ R m , where elements provide a measure ofsimilarity of the input z ω to each of the m concept vectors.

Original: Using these deﬁnitions, we can gauge the representational power of z ω by learning a mapping g : ˆ ν c ( z ω )

Original: →

Original: y .

Original: In practice, g is a simple model (e.g., shallow network or linearprojection) so as to gauge the expressivity of the latent space.

Original: Given a mapping g , we usethe classiﬁcation accuracy as a ‘completeness score’ [13] for the set of concepts C , deﬁned

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: arg

Original: [Equation]

Original: g

Original: [Equation] ν c ( z ω )) (cid:3) , where V is a validation set. Importantly, this approach permits us to compute class-conditioned Shapley values, called ConceptSHAP in theframework of [13]. Speciﬁcally, for a given class k , the class-conditioned ConceptSHAP value foreach concept c j is deﬁned,
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Equation]

Original: where η k is the completeness score for class k (computed simply as the classiﬁcation accuracy ofthe model for the subset of validation points with ground truth label k ).

Original: ConceptSHAP provides ameasure of importance of each concept c j for predicting the outcomes associated with a given class,which we can then use to identify the trajectories.

Original: The experiments in our main paper use the abovesetup to both quantitatively evaluate z ω in terms of representation power, and also reveal interestingconcepts associated with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) relevant characteristics in the domain.

Original: For each of the prediction problems considered in the main paper (agent dispersion and agent returns),we create an 80-20 training-validation data split (over 5 classes).

Original: For concept generation we use

Original: K-means with 16 and 24 clusters, a 2-layer MLP (with 8 hidden units per layer) for the predictionhead g , training with a batch size of 64 , and a concept threshold κ of 0.0 and 0.3 for the dispersionand agent return experiments, respectively, and train the prediction head for 1 e 4 steps.

Original: As in Yehet al.

Original: [13], we use KernelSHAP [83] to approximate ConceptSHAP efﬁciently.

Original: A.3 Additional Results

Original: A.3.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: (Fig.

Original: 6a (cid:58)(cid:58) ).

Original: (cid:58)(cid:58)(cid:58) To (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collect (cid:58)(cid:58)(cid:58) data (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain, (cid:58)(cid:58)(cid:58)

Original: we

Original: [Table]

Original: wider

Original: [Table]

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58) both

Original: [Table]

Original: the

Original: [Table]

Original: and

Original: [Table]

Original: SAC

Original: [Table]

Original: algorithms, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) varying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) batch (cid:58)(cid:58)(cid:58)(cid:58) sizes (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learning(cid:58)(cid:58)(cid:58)(cid:58) rates (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) gather (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) widely-varying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors.

Original: (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) subsequently (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) train MOHBA (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) data,

Original: [Table]

Original: visualizing (cid:58)(cid:58)(cid:58) the

Original: [Table]

Original: learned (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior [Table] clusters (cid:58)(cid:58) in [Table]
Revised: spent on participant compensation?

Original: Fig.

Original: 6b (cid:58) .

Original: (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) several (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interest

Original: [Table]

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) underlying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) videos, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corresponds (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) cases (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) extremely (cid:58)(cid:58)(cid:58)(cid:58) low(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Similarly, (cid:58)(cid:58)(cid:58)(cid:58) there (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exists (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) smaller (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: correspond (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) displaying(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) various (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ant-poses, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) moving (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) incrementally.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) One (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exceptional (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) cluster (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exists (cid:58)(cid:58) in

Original: [Table]

Original: the

Original: [Table]

Original: left

Original: [Table]

Original: region

Original: [Table]

Original: of

Original: [Table]

Original: the

Original: [Table]

Original: joint

Original: [Table]

Original: space,

Original: [Table]

Original: walking (cid:58)(cid:58)(cid:58)(cid:58) gait, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) while (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) remaining (cid:58)(cid:58)(cid:58)(cid:58) three (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) remain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stationary.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Application (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) externally-trained (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: Speciﬁcally,(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consider (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: aforementioned (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) OpenAI (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hide-and-seek (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) wherein (cid:58)(cid:58)(cid:58)(cid:58)

Original: two (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) teams (cid:58)(cid:58) of(cid:58)(cid:58)(cid:58) two (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: (hiders (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seekers) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) compete (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) against (cid:58)(cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) another (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) rich, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-dimensional(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) various (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: manually (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) annotated (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experts (cid:58)(cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) exhibiting(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) various (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinctive (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) multiagent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors (cid:58)(cid:58)(cid:58) at (cid:58)(cid:58)(cid:58)(cid:58) key (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stages (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58) in (cid:58) github.com/openai

Original: (cid:58)(cid:58)(cid:58)(cid:58)

Original: We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consider (cid:58)(cid:58)(cid:58)(cid:58) four (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinctive (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58) that(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) following (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human-annotated (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels: (cid:58)(cid:58)(cid:58) A) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘running (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) chasing’, (cid:58)(cid:58)(cid:58) B) (cid:58)(cid:58)(cid:58)(cid:58) ‘fort(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) building’, (cid:58)(cid:58) C) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘ramp (cid:58)(cid:58)(cid:58)(cid:58) use’, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) D) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘ramp (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) defense’.

Original: (cid:58)(cid:58)(cid:58)(cid:58)

Original: The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state-space (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: (position, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) velocity, (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) box(cid:58)(cid:58)(cid:58) size (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information); (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent’s (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) action (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consists (cid:58)(cid:58) of (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 3-dimensional (cid:58)(cid:58)(cid:58)(cid:58) force (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) vector, (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘glue’(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘lock’ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) action (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interacting (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) other (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) objects.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Overall, (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) yields (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) state-action (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58)(cid:58) with(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: (cid:58)(cid:58)

Original: [Table]

Original: environment (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above.

Original: (cid:58)(cid:58)(cid:58)(cid:58)

Original: We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collect (cid:58)(cid:58)(cid:58)(cid:58) 100 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58) per

Original: [Table]

Original: policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoint, (cid:58)(cid:58)(cid:58)(cid:58) with

Original: [Table]

Original: each

Original: [Table]

Original: being

Original: [Table]

Original: [Table]

Original: steps (cid:58)(cid:58)(cid:58)(cid:58) long

Original: [Table]

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Note

Original: [Table]

Original: that

Original: [Table]

Original: episode.

Original: [Figure]

Original: g e n t

Original: R e t u r n

Original: [Figure]

Original: S u m o f a g e n t - w i s e r e t u r n s (b)

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: space (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58) by (cid:58) MOHBA (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: AntWalker (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain.

Original: (cid:58) (cid:58)(cid:58)(cid:58)

Original: We (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58) mix (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collected (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) together, (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58) train 776 MOHBA (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shufﬂed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Original: (cid:58)(cid:58)(cid:58)

Original: As (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) usual, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) note (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provide (cid:58)(cid:58)(cid:58) no (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reward (cid:58)(cid:58) or (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) information(cid:58)(cid:58) to MOHBA.

Original: (cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) discovered (cid:58)(cid:58)(cid:58) by (cid:58) MOHBA (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hide-and-seek(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain.

Original: [-(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Agents (cid:58)(cid:58) 
-]
Revised: [-(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Agents (cid:58)(cid:58) 
-]

Original: [-(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 1 (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgure (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘hiders’, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) whereas (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58) 
-]
Revised: [-(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 1 (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgure (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘hiders’, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) whereas (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58) 
-]

Original: (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 3 (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘seekers’.(cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58) label (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgure (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human-annotated (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provided(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) OpenAI.

Original: (cid:58)(cid:58)(cid:58)

Original: In Fig.

Original: 7, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) presence (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) clear (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58)(cid:58)(cid:58) well (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) expert (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels, (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) levels.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Interestingly, (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seekers (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: (agents (cid:58) 2 (cid:58)(cid:58)(cid:58) and(cid:58)(cid:58) 3), (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)

Original: A

Original: [Table]

Original: ‘running (cid:58)(cid:58)(cid:58)(cid:58) and

Original: [Table]

Original: chasing’)

Original: [Table]

Original: is

Original: [Table]

Original: highly

Original: [Table]

Original: distinctive

Original: [Table]

Original: and

Original: [Table]

Original: well-separated

Original: [Table]

Original: from

Original: [Table]

Original: the

Original: [Table]

Original: other

Original: [Table]

Original: .(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Interestingly, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) despite (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: building’)

Original: (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) D (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (‘ramp (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) defense’) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) appear (cid:58)(cid:58) to (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviorally

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: ramps) (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58) can (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) interact (cid:58)(cid:58)(cid:58)(cid:58) with.

Original: (cid:58)(cid:58)(cid:58)(cid:58)

Original: We (cid:58)(cid:58)(cid:58) mix (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collected (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) above (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) together, (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58) train (cid:58) MOHBA (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) shufﬂed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Using (cid:58) MOHBA (cid:58) , (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) presence (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clear (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58)(cid:58)(cid:58) well (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) expert (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) both (cid:58) at (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) levels, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) despite (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) algorithm (cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) having (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) access (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) explicit (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) during (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Domain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) screenshots (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reproduced (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) permission (cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Baker et al.

Original: (cid:58) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) closer (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) another (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58) C (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: (‘ramp (cid:58)(cid:58)(cid:58)(cid:58) use’)

Original: (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) D, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) namely (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) that (cid:58)(cid:58) of

Original: (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seeker (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (agent (cid:58)(cid:58) 2 (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58) 3) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces.

Original: (cid:58)(cid:58)(cid:58)(cid:58) One (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hypothesis (cid:58) is (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) could (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) perhaps (cid:58)(cid:58) be (cid:58)(cid:58)(cid:58)(cid:58) due

Original: (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58) ‘fort (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) building’ (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘ramp (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) defense’ (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policies (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: being (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) associated (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) situations (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seekers (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) cannot(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) easily (cid:58)(cid:58)(cid:58) ﬁnd (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hiders, (cid:58)(cid:58)(cid:58)(cid:58) due (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hiders (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) obstacles (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) block (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) entrances (cid:58)(cid:58)(cid:58) (B) (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) moving (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ramps (cid:58)(cid:58) to(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) prevent (cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) effective (cid:58)(cid:58)(cid:58)(cid:58) use (cid:58)(cid:58)(cid:58) (D).

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Overall, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experiments (cid:58)(cid:58)(cid:58)(cid:58) help (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) validate MOHBA (cid:58) ’s

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58)(cid:58)(cid:58) latent(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labels (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) manually (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) annotated (cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) human (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) experts, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) highlight (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) applicability

Original: (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) algorithm (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: Figures 8 and 9 provide additional results for the baseline comparisons conducted in the main paper

Original: [Table]

Original: Speciﬁcally, (a) in each ﬁgure provides an expanded comparison of ICTD (with a sweep over the # of clusters in K-means).

Original: In (b) of each ﬁgure, we visualize the convergence of the APL throughouttraining for MOHBA and the baselines.

Original: Subﬁgures (c) and (d), respectively, show a PCA and UMAPprojection of the latent space discovered by the LSTM baseline (recall from the main paper that ‘jointembeddings’ used for the LSTM are simply the ﬁnal hidden state).

Original: Likewise, subﬁgures (e) and (f)show the same projections for the VAE baseline.

Original: As especially evident in the coordination domain (Fig. 9), the joint LSTM baseline embeddings(whether using PCA or UMAP projections, Figs.

Original: 9c and 9d, respectively) do not reveal to the fairlyinterpretable 3 clusters of coordinated behaviors discussed in the main paper.

Original: By contrast, the VAEbaseline (in both the PCA and UMAP case, Figs. 9e and 9f, respectively) does produce these clusters;despite this, note that the key limitation of this baseline is its non-hierarchical nature, which makesidentiﬁcation of the local-to-joint behavioral correspondences noted in the main paper signiﬁcantlymore difﬁcult.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: APL

Original: ICTD

Original: K=

Original: K=

Original: K=

Original: LSTM 2 .

Original: 9 ± 0 .0 .

Original: 41 ± 0 .0 .

Original: 40 ± 0 .0 .

Original: 34 ± 0 .

Original: VAE 5 . 0 ± 0 .0 .

Original: 18 ± 0 .0 .

Original: 17 ± 0 .0 .

Original: 16 ± 0 .

Original: MOHBA 2 .

Original: 4 ± 0 .0 .

Original: 18 ± 0 .0 .

Original: 17 ± 0 .0 .

Original: 16 ± 0 .

Original: [Figure]

Original: LSTM Baseline

Original: VAE Baseline

Original: [Figure]

Original: A.

Original: [Figure]

Original: Hyperparameter (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Ablations (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Effects (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: Spaces

Original: [Table]

Original: (cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) focus (cid:58)(cid:58)(cid:58) our (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) analysis (cid:58)(cid:58) on (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) KL-loss (cid:58)(cid:58) β (cid:58)(cid:58)(cid:58)(cid:58) term (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) objective, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) alongside (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) latent(cid:58)(cid:58) z ω (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)

Original: z α (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dimensionality.

Original: (cid:58)(cid:58)(cid:58)(cid:58)

Original: We (cid:58)(cid:58)(cid:58)(cid:58) both (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) compare (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58) loss (5) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) across (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ablations (cid:58)(cid:58) to(cid:58)(cid:58)(cid:58)(cid:58)

Original: attain (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) quantitative (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) comparison (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) quality (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) encodings, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) additionally (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: compare (cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) structure (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) themselves (cid:58)(cid:58) as (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) function (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) parameters (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) qualitative(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: understanding.(cid:58)(cid:58)(cid:58)(cid:58) First, (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) terms (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) raw (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58) loss, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58) ﬁnd (cid:58)(cid:58)(cid:58) in Figs. 10a, 11a and 12a (cid:58)(cid:58)(cid:58)

Original: that (cid:58)(cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dimensionality (cid:58)(cid:58)(cid:58) has (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) higher (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) impact (cid:58)(cid:58)(cid:58) on (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) performance (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) weighing(cid:58)(cid:58)(cid:58)(cid:58) term (cid:58)(cid:58) β ; (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intuitively, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increasing (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly (cid:58)(cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinguishable (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (e.g., (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) comparing (cid:58)(cid:58)

Original: z ω(cid:58)(cid:58) in Fig.

Original: 12c (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) versus (cid:58) Fig.

Original: 12d (cid:58) ); (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58) also (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) aligns (cid:58)(cid:58)(cid:58)(cid:58) well (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intuition, (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increasing (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58) β

Original: (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: [Table]

Original: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58) term (cid:58)(cid:58)(cid:58) (5).

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Overall, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) results (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provide (cid:58)(cid:58)(cid:58) us (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) intuition (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) terms (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) role (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table] A. [Table]
Revised: [N/A]

Original: (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) numerous (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) underlying (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58) runs (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain.

Original: (cid:58)(cid:58)

Original: To (cid:58)(cid:58)(cid:58)(cid:58) help (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ensure (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collection (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) analysis (cid:58)(cid:58) of (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) diverse (cid:58)(cid:58) set (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behaviors(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) including (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sub-optimal (cid:58)(cid:58)(cid:58)(cid:58) ones, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58) not (cid:58)(cid:58)(cid:58)(cid:58) only (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) collect (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) end (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MARL (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training, (cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) rather(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) checkpoints (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout (cid:58)(cid:58) all (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training.

Original: (cid:58)(cid:58)(cid:58) All (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) including (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) those (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) stemming(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) randomly-initialized (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: at

Original: [Table]

Original: the

Original: [Table]

Original: beginning

Original: [Table]

Original: of

Original: [Table]

Original: are

Original: [Table]

Original: and

Original: [Table]

Original: used

Original: [Table]

Original: to

Original: [Table]

Original: [Table]

Original: [Table]

Original: for (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) datasets (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) analyzed.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Domain (cid:58)(cid:58) % (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) total (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) < 0 .

Original: 5 × (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (maximum (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: considered [Table] that [Table]
Revised: Our work does

Original: attain

Original: [Table]

Original: than

Original: [Table]

Original: [Table]

Original: of

Original: [Table]

Original: maximum

Original: [Table]

Original: observed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return

Original: [Table]

Original: Note

Original: [Table]

Original: that

Original: [Table]

Original: in

Original: [Table]

Original: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-dimensional(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: MultiAgent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MuJoCo (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) especially, (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) large (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) proportion (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) low-reward (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) exist (cid:58)(cid:58) in(cid:58)(cid:58)

Original: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: HalfCheetah (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) being (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) particularly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) notable (cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) where (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 92.69%

Original: (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) datasets(cid:58)(cid:58)(cid:58)(cid:58) attain (cid:58)(cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58)(cid:58)(cid:58)(cid:58) 50% (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) max (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return.(cid:58)(cid:58)(cid:58)

Original: We (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) additionally (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) provide (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) detailed (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) overview (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58) of Figs. 13a, 14a, 856 15a and 16a (cid:58) .

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: These (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgures (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualize (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) individual (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) considered (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environments (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: (respectively, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) present(cid:58) (Fig. 13a (cid:58) ).

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Interestingly, (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)

Original: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) AntWalker (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset (cid:58)(cid:58)(cid:58) we

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) bimodal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (seemingly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consisting (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) very (cid:58)(cid:58)(cid:58) low (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) very (cid:58)(cid:58)(cid:58)(cid:58) high (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories, (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) seen (cid:58)(cid:58) in (cid:58) Fig.

Original: 14a (cid:58) ).(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Closer (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) inspection (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) domain (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reveals (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: Speciﬁcally,

Original: [Table]

Original: 13b, 14c, 15b and 16b

Original: [Table]

Original: spaces (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned

Original: [Table]

Original: by MOHBA

Original: [Table]

Original: [Table]

Original: which (cid:58)(cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ground (cid:58)(cid:58)(cid:58)(cid:58) truth(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corresponding (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectory.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Interestingly, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: low-return(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (random/early-training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories), (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) medium-return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories, (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58) are(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) corresponding (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) low (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clustered (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) center (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58) the(cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58) in (cid:58) Fig.

Original: 15b; (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: HalfCheetah (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: distinctive

Original: [Table]

Original: [Table]

Original: [Table]

Original: A.

Original: [Table]

Original: and (cid:58)(cid:58) z α(cid:58)(cid:58)(cid:58)(cid:58)

Original: This (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) section (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) summarizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) procedure (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) used (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) generate (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ﬁgures (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) associated (cid:58)(cid:58)(cid:58)(cid:58) with(cid:58)(cid:58) z ω (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)

Original: z α .

Original: (cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: , (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) sampling (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58) joint (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58) z ω (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)

Original: a (cid:58)(cid:58) set(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) local (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latents (cid:58)(cid:58)

Original: z iα (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58) N (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agents (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accordingly.

Original: (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: Thus, (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) K (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)

Original: we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) obtain (cid:58) a(cid:58)(cid:58) set (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) parameters (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) { ( z ω , z

Original: 1 α , · · · , z Nα ) k } Kk =1 .3.

Original: (cid:58)(cid:58)

Original: As (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) parameters (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) high-dimensional (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) nature, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: visualize (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) their (cid:58)(cid:58)(cid:58) 2D (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) projection(cid:58)(cid:58)(cid:58)(cid:58)

Original: (e.g., (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Principal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Component (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Analysis), (cid:58)(cid:58)(cid:58)(cid:58) thus (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) yielding (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizations(cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58) as (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) those (cid:58)(cid:58) in Fig. 3a.

Original: (cid:58)

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: Ablations (cid:58)(cid:58)(cid:58)(cid:58) over (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hyperparameters (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 2-agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) coordination (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) game.

Original: (cid:58)(cid:58) (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58) loss (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout MOHBA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training.

Original: (b)-(e) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualize (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) change (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent (cid:58)(cid:58)(cid:58)(cid:58) space (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58) each (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) ﬁnal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MOHBA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) models (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned (cid:58)(cid:58)(cid:58)(cid:58) over (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ablations.

Original: (cid:58)

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: Ablations (cid:58)(cid:58)(cid:58)(cid:58) over (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hyperparameters (cid:58)(cid:58) for (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Multiagent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MuJoCo (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) HalfCheetah.

Original: (cid:58) (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reconstruction (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) policy (cid:58)(cid:58)(cid:58)(cid:58) loss (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) throughout MOHBA (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: visualize

Original: [Table]

Original: the

Original: [Table]

Original: change (cid:58)(cid:58)(cid:58) in

Original: [Table]

Original: space

Original: [Table]

Original: for

Original: [Table]

Original: each

Original: [Table]

Original: of

Original: [Table]

Original: the

Original: [Table]

Original: ﬁnal

Original: [Table]

Original: MOHBA

Original: [Table]

Original: models (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learned

Original: [Table]

Original: over

Original: [Table]

Original: these

Original: [Table]

Original: ablations.

Original: [Table]

Original: Increasing (cid:58)(cid:58)(cid:58)

Original: [Table]

Original: term (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 e − 4 (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1 e − 2 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) tends (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) result (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) clusters (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) overlap (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) more (cid:58) /

Original: (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ‘softer’ (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) slightly (cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distinguishable (cid:58)(cid:58)(cid:58)(cid:58)

Original: [Figure]

Original: A g e n t

Original: R e t u r n

Original: [Figure]

Original: [Figure]

Original: statistics (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Multiagent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MuJoCo (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 2-agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: HalfCheetah (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) environment.

Original: (cid:58) (a) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Original: (cid:58) (b) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Original: (cid:58)

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: (anomalous (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) returns (cid:58)(cid:58)(cid:58) less (cid:58)(cid:58)(cid:58)(cid:58) than (cid:58) 0 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) removed).

Original: [Figure]

Original: A g e n t

Original: R e t u r n

Original: [Figure]

Original: [Figure]

Original: S u m o f a g e n t - w i s e r e t u r n s

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: the

Original: [Table]

Original: distribution

Original: [Table]

Original: of

Original: [Table]

Original: agent

Original: [Table]

Original: in [Table] the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Revised: not crowdsourced or humansubject data.

Original: Interestingly, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) bimodal (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) (seemingly (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) consisting (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) low (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) higher (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: [Table]

Original: excluded, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) which (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) better (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) illustrates (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) distribution (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) medium-high (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reward (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset.

Original: (cid:58) (c) (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) visualizes (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)

Original: labeled (cid:58)(cid:58) by (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Original: [Figure]

Original: [Table]

Original: the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) agent (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Original: (cid:58)

Original: [Figure]

Original: g e n t

Original: R e t u r n

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: behavior (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) spaces, (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trajectories (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) labeled (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) return (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) attained.

Original: (cid:58)