Original: Beyond Rewards: a Hierarchical Perspective on Ofﬂine Multiagent Behavioral Analysis
Revised: Beyond Rewards: a Hierarchical Perspective on Ofﬂine Multiagent Behavioral Analysis

Original: Abstract
Revised: Abstract

Original: Each year, expert-level performance is attained in increasingly-complex multiagent domains, where notable examples include Go, Poker, and StarCraft II.
Revised: Each year, expert-level performance is attained in increasingly-complex multiagent domains, where notable examples include Go, Poker, and StarCraft II.

Original: This rapid progression is accompanied by a commensurate need to better understand how such agents attain this performance, to enable their safe deployment, identify limitations, and reveal potential means of improving them.
Revised: This rapid progression is accompanied by a commensurate need to better understand how such agents attain this performance, to enable their safe deployment, identify limitations, and reveal potential means of improving them.

Original: In this paper we take a step back from performance-focused multiagent learning, and instead turn our attention towards agent behavior analysis.
Revised: In this paper we take a step back from performance-focused multiagent learning, and instead turn our attention towards agent behavior analysis.

Original: We introduce a model-agnostic method for discovery of behavior clusters in multiagent domains, using variational inference to learn a hierarchy of behaviors at the joint and local agent levels.
Revised: We introduce a model-agnostic method for discovery of behavior clusters in multiagent domains, using variational inference to learn a hierarchy of behaviors at the joint and local agent levels.

Original: Our framework makes no assumption about agents’ underlying learning algorithms, does not require access to their latent states or policies, and is trained using only ofﬂine observational data.
Revised: Our framework makes no assumption about agents’ underlying learning algorithms, does not require access to their latent states or policies, and is trained using only ofﬂine observational data.

Original: We illustrate the effectiveness of our method for enabling the coupled understanding of behaviors at the joint and local agent level, detection of behavior changepoints throughout training, discovery of core behavioral concepts, demonstrate the approach’s scalability to a high-dimensional multiagent MuJoCo control domain, and also illustrate that the approach can disentangle previouslytrained policies in OpenAI’s hide-and-seek domain.
Revised: We illustrate the effectiveness of our method for enabling the coupled understanding of behaviors at the joint and local agent level, detection of behavior changepoints throughout training, discovery of core behavioral concepts, demonstrate the approach’s scalability to a high-dimensional multiagent MuJoCo control domain, and also illustrate that the approach can disentangle previouslytrained policies in OpenAI’s hide-and-seek domain.

Original:  Introduction
Revised:  Introduction

Original: Multiagent approaches have driven numerous advances in artiﬁcial intelligence research, with seminal examples including TD-gammon [1], DeepBlue [2], AlphaGo [3], AlphaZero [4], Libratus [5],
Revised: Multiagent approaches have driven numerous advances in artiﬁcial intelligence research, with seminal examples including TD-gammon [1], DeepBlue [2], AlphaGo [3], AlphaZero [4], Libratus [5],

Original: AlphaStar [6], OpenAI Five [7] and Pluribus [8].
Revised: AlphaStar [6], OpenAI Five [7] and Pluribus [8].

Original: During training, many of these approaches seek to push the performance of agents as measured by a reward signal or derivatives thereof.
Revised: During training, many of these approaches seek to push the performance of agents as measured by a reward signal or derivatives thereof.

Original: Despite this, post-hoc methods that seek to understand agent interactions often use less reward-centric techniques.
Revised: Despite this, post-hoc methods that seek to understand agent interactions often use less reward-centric techniques.

Original: Instead, insights are drawn from behavioral analysis to identify unique or interesting agent strategies.
Revised: Instead, insights are drawn from behavioral analysis to identify unique or interesting agent strategies.

Original: Examples include clustering-based analysis of neuron activations and trajectories in capture-the-ﬂag [9], inspection of trajectories in a hide-and-seek domain to detect interesting behaviors such as agents that learn to exploit the underlying physics engine [10], monitoring of statistics such as pass ranges and frequencies in humanoid football [11], and analysis of AlphaZero’s acquisition of chess knowledge [12].
Revised: Examples include clustering-based analysis of neuron activations and trajectories in capture-the-ﬂag [9], inspection of trajectories in a hide-and-seek domain to detect interesting behaviors such as agents that learn to exploit the underlying physics engine [10], monitoring of statistics such as pass ranges and frequencies in humanoid football [11], and analysis of AlphaZero’s acquisition of chess knowledge [12].

Original: Crucially, such insights are often drawn via manual analysis and detection of behavioral clusters, or use of statistics associated with certain behaviors as deﬁned by humans experts (e.g., various skills and relevant metrics in humanoid football).
Revised: Crucially, such insights are often drawn via manual analysis and detection of behavioral clusters, or use of statistics associated with certain behaviors as deﬁned by humans experts (e.g., various skills and relevant metrics in humanoid football).

Original: [Figure]
Revised: [Figure]

Original: As evident above, understanding emergent multiagent behaviors is enriched by techniques beyond pure reward-based analysis, as behavioral signiﬁers are not always discernible via rewards.
Revised: As evident above, understanding emergent multiagent behaviors is enriched by techniques beyond pure reward-based analysis, as behavioral signiﬁers are not always discernible via rewards.

Original: Figure 1a provides intuition on this notion, illustrating returns (sum of rewards) throughout training for a multiagent hill-climbing domain (later described in detail).
Revised: Figure 1a provides intuition on this notion, illustrating returns (sum of rewards) throughout training for a multiagent hill-climbing domain (later described in detail).

Original: We highlight two independent training trials (A and B) with similar ﬁnal returns.
Revised: We highlight two independent training trials (A and B) with similar ﬁnal returns.

Original: Despite similar returns, comparing the trajectories generated via the agents’ deterministic policies following each trial’s training (Fig. 1b) reveals entirely different behaviors.
Revised: Despite similar returns, comparing the trajectories generated via the agents’ deterministic policies following each trial’s training (Fig. 1b) reveals entirely different behaviors.

Original: Analogous examples are evident in the above works (e.g., Fig.
Revised: Analogous examples are evident in the above works (e.g., Fig.

Original: 1 of Baker et al.
Revised: 1 of Baker et al.

Original: [10], where substantial behavior changes occur in multiagent hide-and-seek despite a smooth reward curve).
Revised: [10], where substantial behavior changes occur in multiagent hide-and-seek despite a smooth reward curve).

Original: This paper formalizes the problem of ofﬂine multiagent behavior analysis.
Revised: This paper formalizes the problem of ofﬂine multiagent behavior analysis.

Original: Our proposed algorithm,
Revised: Our proposed algorithm,

Original: Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), learns a hierarchical latent space that simultaneously reveals behavior clusters at the joint level (i.e., interactions between agents) and local level (i.e., behaviors of individual agents).
Revised: Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), learns a hierarchical latent space that simultaneously reveals behavior clusters at the joint level (i.e., interactions between agents) and local level (i.e., behaviors of individual agents).

Original: Our method is agnostic of the underlying algorithm used to generate agents’ behaviors, requires no access or control of the underlying environment, does not assume availability of a reward signal, and does not require access to agents’ models or internal states.
Revised: Our method is agnostic of the underlying algorithm used to generate agents’ behaviors, requires no access or control of the underlying environment, does not assume availability of a reward signal, and does not require access to agents’ models or internal states.

Original: Our experiments investigate the structure of the learned behavior space, which goes beyond prior works on latent-clustering by identifying relationships between individual agent and joint behaviors.
Revised: Our experiments investigate the structure of the learned behavior space, which goes beyond prior works on latent-clustering by identifying relationships between individual agent and joint behaviors.

Original: We illustrate that clusters identiﬁed by MOHBA are useful for highlighting similarities and differences in behaviors throughout training.
Revised: We illustrate that clusters identiﬁed by MOHBA are useful for highlighting similarities and differences in behaviors throughout training.

Original: We also quantitatively analyze the completeness of discovered behavior clusters by adopting a modiﬁed version of the concept-discovery framework of Yeh et al.
Revised: We also quantitatively analyze the completeness of discovered behavior clusters by adopting a modiﬁed version of the concept-discovery framework of Yeh et al.

Original: [13] to identify interesting behavior concepts in our multiagent setting.
Revised: [13] to identify interesting behavior concepts in our multiagent setting.

Original: We then test the scalability of our approach by using it for behavioral analysis of several high-dimensional multiagent MuJoCo environments [14].
Revised: We then test the scalability of our approach by using it for behavioral analysis of several high-dimensional multiagent MuJoCo environments [14].

Original: Finally, we evaluate the approach on the open-sourced OpenAI hide-and-seek policy checkpoints [10], conﬁrming that the behavioral clusters detected by MOHBA closely match those of the human-expert annotated labels provided in their policy checkpoints.
Revised: Finally, we evaluate the approach on the open-sourced OpenAI hide-and-seek policy checkpoints [10], conﬁrming that the behavioral clusters detected by MOHBA closely match those of the human-expert annotated labels provided in their policy checkpoints.

Original:  Related Work
Revised:  Related Work

Original: Signiﬁcant research has been conducted in single-agent skill discovery, which seeks to learn reusable policies useful for downstream tasks [15–28].
Revised: Signiﬁcant research has been conducted in single-agent skill discovery, which seeks to learn reusable policies useful for downstream tasks [15–28].

Original: Related approaches discover motor primitives to express longer-horizon policies [29], including use of ofﬂine reinforcement learning (RL) for learning useful behaviors [30].
Revised: Related approaches discover motor primitives to express longer-horizon policies [29], including use of ofﬂine reinforcement learning (RL) for learning useful behaviors [30].

Original: Option discovery methods learn temporally-abstracted actions (i.e., options [31]), chained together to form cohesive skills [32–37].
Revised: Option discovery methods learn temporally-abstracted actions (i.e., options [31]), chained together to form cohesive skills [32–37].

Original: In contrast to our work, these approaches focus on maximizing performance in single-agent settings.
Revised: In contrast to our work, these approaches focus on maximizing performance in single-agent settings.

Original: There also exists a related line of work for learning diverse policies in RL settings [38–46]; despite their focus on policy diversity, several of these approaches make much stronger assumptions than ours (e.g., access to the policies generating agent trajectories, or unique identiﬁers of the policy that generated each trajectory).
Revised: There also exists a related line of work for learning diverse policies in RL settings [38–46]; despite their focus on policy diversity, several of these approaches make much stronger assumptions than ours (e.g., access to the policies generating agent trajectories, or unique identiﬁers of the policy that generated each trajectory).

Original: By contrast, our
Revised: By contrast, our

Original: A c t i o n s
Revised: A c t i o n s

Original: S t a t e s
Revised: S t a t e s

Original: P o l i
Revised: P o l i

Original: c i e s
Revised: c i e s

Original: L o c a l L a t e n t
Revised: L o c a l L a t e n t

Original: J o i n t L a t e n t (a)
Revised: J o i n t L a t e n t (a)

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: Reconstruction
Revised: Reconstruction

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: prior
Revised: prior

Original: [Figure]
Revised: [Figure]

Original: Policies
Revised: Policies

Original: Offline Data
Revised: Offline Data

Original: [Figure]
Revised: [Figure]

Original: work assumes no access to any of the underlying raw policies, or even labels of which policies agent trajectories were obtained from.
Revised: work assumes no access to any of the underlying raw policies, or even labels of which policies agent trajectories were obtained from.

Original: Recent works have also focused on hierarchical skill learning in multiagent reinforcement learning (MARL).
Revised: Recent works have also focused on hierarchical skill learning in multiagent reinforcement learning (MARL).

Original: Lee et al.
Revised: Lee et al.

Original: [47] use the mutual information maximization objective introduced in Eysenbach et al.
Revised: [47] use the mutual information maximization objective introduced in Eysenbach et al.

Original: [16] to learn multiagent policies.
Revised: [16] to learn multiagent policies.

Original: Yang et al.
Revised: Yang et al.

Original: [48] use a bi-level policy to learn agent skills: a high-level policy ﬁrst generates latent vectors for each agent, and a low-level policy conditions behaviors on said vector to perform the task.
Revised: [48] use a bi-level policy to learn agent skills: a high-level policy ﬁrst generates latent vectors for each agent, and a low-level policy conditions behaviors on said vector to perform the task.

Original: In Wang et al.
Revised: In Wang et al.

Original: [49], distinct ‘roles’ are learned for agents to enable decomposition of tasks.
Revised: [49], distinct ‘roles’ are learned for agents to enable decomposition of tasks.

Original: Mao et al.
Revised: Mao et al.

Original: [50] investigate use of consistent latent cognition variables in agent neighborhoods to induce increased cooperation.
Revised: [50] investigate use of consistent latent cognition variables in agent neighborhoods to induce increased cooperation.

Original: Many MARL approaches have noted the emergence of interesting behaviors in multiagent systems in speciﬁc domains of interest [51–55].
Revised: Many MARL approaches have noted the emergence of interesting behaviors in multiagent systems in speciﬁc domains of interest [51–55].

Original: In contrast to our focus, these approaches use RL to maximize agent performance, rather than understand arbitrary behaviors via ofﬂine analysis.
Revised: In contrast to our focus, these approaches use RL to maximize agent performance, rather than understand arbitrary behaviors via ofﬂine analysis.

Original: RL interpretability methods primarily focus on single-agent settings and either modify the RL algorithm itself to increase transparency, or conduct post-hoc explainability [56].
Revised: RL interpretability methods primarily focus on single-agent settings and either modify the RL algorithm itself to increase transparency, or conduct post-hoc explainability [56].

Original: These approaches represent agent policies as programming languages [57], extract visual summaries of behaviors using ‘interestingness’ statistics such as uncertainty in selected actions or the value of state-transitions [58], or combine agent neuron activations with gradient information to construct behavioral embeddings [59].
Revised: These approaches represent agent policies as programming languages [57], extract visual summaries of behaviors using ‘interestingness’ statistics such as uncertainty in selected actions or the value of state-transitions [58], or combine agent neuron activations with gradient information to construct behavioral embeddings [59].

Original: Behavioral clusters in our work share similarities with concept-based explanation approaches in non-RL domains.
Revised: Behavioral clusters in our work share similarities with concept-based explanation approaches in non-RL domains.

Original: Detecting ‘concepts’ in pre-trained models has been explored in vision [60–62], discrete games [12], and language [63, 64].
Revised: Detecting ‘concepts’ in pre-trained models has been explored in vision [60–62], discrete games [12], and language [63, 64].

Original: In vision, clustering-based approaches describe discovered concepts using examples [60] or use generative modeling to create new data to describe concepts [62].
Revised: In vision, clustering-based approaches describe discovered concepts using examples [60] or use generative modeling to create new data to describe concepts [62].

Original: Ghorbani et al.
Revised: Ghorbani et al.

Original: [60] uses a vision-speciﬁc method (i.e., superpixels) to sub-divide input before conducing clustering; while sub-division is less natural in RL, our LSTM and
Revised: [60] uses a vision-speciﬁc method (i.e., superpixels) to sub-divide input before conducing clustering; while sub-division is less natural in RL, our LSTM and

Original: VAE baselines serve as an RL-adopted counterpart of such works.
Revised: VAE baselines serve as an RL-adopted counterpart of such works.

Original: Works using latent-clustering and analysis are also related to ours.
Revised: Works using latent-clustering and analysis are also related to ours.

Original: These include approaches using multi-level variational autoencoders [65–68] to learn compositional latent spaces, although not in decision-making domains such as ours.
Revised: These include approaches using multi-level variational autoencoders [65–68] to learn compositional latent spaces, although not in decision-making domains such as ours.

Original: Hierarchical latent approaches have been used in single-agent
Revised: Hierarchical latent approaches have been used in single-agent

Original: RL [69, 70].
Revised: RL [69, 70].

Original: Behavior analysis has also been conducted by embedding agent neuron activations into a low-dimensional space, using a (non-hierarchical) variational approach [9].
Revised: Behavior analysis has also been conducted by embedding agent neuron activations into a low-dimensional space, using a (non-hierarchical) variational approach [9].

Original: The representation power of agents’ internal states has also been gauged by predicting future events [11].
Revised: The representation power of agents’ internal states has also been gauged by predicting future events [11].

Original: Overall, the key difference between the above works and ours is that our method combined hierarchical learning with behavior analysis and applies it to the multiagent setting.
Revised: Overall, the key difference between the above works and ours is that our method combined hierarchical learning with behavior analysis and applies it to the multiagent setting.

Original:  Ofﬂine Analysis of Multiagent Behaviors
Revised:  Ofﬂine Analysis of Multiagent Behaviors

Original: This section introduces the ofﬂine multiagent behavioral analysis problem and our proposed algorithm.
Revised: This section introduces the ofﬂine multiagent behavioral analysis problem and our proposed algorithm.

Original: Preliminaries.
Revised: Preliminaries.

Original: We ﬁrst formalize the problem of ofﬂine multiagent behavioral analysis.
Revised: We ﬁrst formalize the problem of ofﬂine multiagent behavioral analysis.

Original: Consider a rewardless multiagent Markov Decision Process (MA-MDP), deﬁned by tuple ( I , S , A , T ) , where
Revised: Consider a rewardless multiagent Markov Decision Process (MA-MDP), deﬁned by tuple ( I , S , A , T ) , where

Original: I = { 1 , ..., N } is the set of N agents, S is the state space,
Revised: I = { 1 , ..., N } is the set of N agents, S is the state space,

Original: A is the action space, and T denotes the state transition probability function.
Revised: A is the action space, and T denotes the state transition probability function.

Original: By not relying on the presence of rewards, behaviors generated even without reliance on a reward function (e.g., human interactions, or agents using curiosity-based exploration) can be considered.
Revised: By not relying on the presence of rewards, behaviors generated even without reliance on a reward function (e.g., human interactions, or agents using curiosity-based exploration) can be considered.

Original: We use the term ‘local’ for elements associated with individual agents, and ‘joint’ for those associated with the entire system.
Revised: We use the term ‘local’ for elements associated with individual agents, and ‘joint’ for those associated with the entire system.

Original: At each timestep t ∈ { 0 , . . .
Revised: At each timestep t ∈ { 0 , . . .

Original: , T − 1 } , the agents execute joint action a t ∈ A in state s t using joint policy π ( a t | s t ) , causing the state to transition to s t +1 with probability p ( s t +1 | s t , a t ) =
Revised: , T − 1 } , the agents execute joint action a t ∈ A in state s t using joint policy π ( a t | s t ) , causing the state to transition to s t +1 with probability p ( s t +1 | s t , a t ) =

Original: T ( s t +1 , a t , s t ) .
Revised: T ( s t +1 , a t , s t ) .

Original: As standard in multiagent frameworks [71], we assume the joint action space factorizes as A = ×
Revised: As standard in multiagent frameworks [71], we assume the joint action space factorizes as A = ×

Original: i
Revised: i

Original: A
Revised: A

Original: i , such that a t = ( a 1 t , . .
Revised: i , such that a t = ( a 1 t , . .

Original: . , a Nt ) , where i ∈ I and a it
Revised: . , a Nt ) , where i ∈ I and a it

Original: ∈
Revised: ∈

Original: A i .
Revised: A i .

Original: Similarly, π i ( a it | s t ) is the local policy for agent i , and π = (cid:81)
Revised: Similarly, π i ( a it | s t ) is the local policy for agent i , and π = (cid:81)

Original: i π
Revised: i π

Original: i is the joint policy.
Revised: i is the joint policy.

Original: Let τ = (cid:0) s 0 , ( a i 0 ) i ∈I , . . .
Revised: Let τ = (cid:0) s 0 , ( a i 0 ) i ∈I , . . .

Original: , s T − 1 , ( a iT − 1 ) i ∈I , s T (cid:1) denote a trajectory induced by this process, and
Revised: , s T − 1 , ( a iT − 1 ) i ∈I , s T (cid:1) denote a trajectory induced by this process, and

Original: D = { τ 1 , . .
Revised: D = { τ 1 , . .

Original: , τ K } denote a dataset of K such trajectories.
Revised: , τ K } denote a dataset of K such trajectories.

Original: This dataset may consist of trajectories from multiple training runs, including variations over agent algorithms, hyperparameters, random seeds, or other factors inﬂuencing emergent behaviors.
Revised: This dataset may consist of trajectories from multiple training runs, including variations over agent algorithms, hyperparameters, random seeds, or other factors inﬂuencing emergent behaviors.

Original: Given dataset D , the ofﬂine multiagent analysis problem seeks to uncover potential clusters of agent behaviors.
Revised: Given dataset D , the ofﬂine multiagent analysis problem seeks to uncover potential clusters of agent behaviors.

Original: Approach.
Revised: Approach.

Original: Our approach, called the Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), uses ofﬂine trajectory data to discover behaviors exhibited by agents at the local and joint level.
Revised: Our approach, called the Multiagent Ofﬂine Hierarchical Behavior Analyzer (MOHBA), uses ofﬂine trajectory data to discover behaviors exhibited by agents at the local and joint level.

Original: We ﬁrst use Fig.
Revised: We ﬁrst use Fig.

Original: 2a to build intuition before discussing technical details.
Revised: 2a to build intuition before discussing technical details.

Original: Let the agent interactions exhibited in a trajectory τ be encoded by a latent variable, z ω ∈ R D ω , capturing their joint behavior.
Revised: Let the agent interactions exhibited in a trajectory τ be encoded by a latent variable, z ω ∈ R D ω , capturing their joint behavior.

Original: For example, z ω may encode (at a high level) whether agents were cooperating or competing in a given trajectory; conditioned on joint signal z ω , each agent then exhibits its own local behavior.
Revised: For example, z ω may encode (at a high level) whether agents were cooperating or competing in a given trajectory; conditioned on joint signal z ω , each agent then exhibits its own local behavior.

Original: Let local latent vectors z α = ( z 1 α , . . .
Revised: Let local latent vectors z α = ( z 1 α , . . .

Original: , z Nα ) encode individual agent behaviors, where z iα ∈ R D iα .
Revised: , z Nα ) encode individual agent behaviors, where z iα ∈ R D iα .

Original: Conditioned on the local behavior vector z iα , each agent then executes actions using a behaviorconditioned policy π i ( a i | s, z iα ) .
Revised: Conditioned on the local behavior vector z iα , each agent then executes actions using a behaviorconditioned policy π i ( a i | s, z iα ) .

Original: Given trajectory dataset D , we seek to learn the latent-conditioned policies and distributions over latent vectors, such that we can reconstruct any behaviors exhibited by the agents in D .
Revised: Given trajectory dataset D , we seek to learn the latent-conditioned policies and distributions over latent vectors, such that we can reconstruct any behaviors exhibited by the agents in D .

Original: Thus, latent vectors z ω and z α will encode the agents’ behavioral spaces and, ideally, identify behavioral clusters in the dataset.
Revised: Thus, latent vectors z ω and z α will encode the agents’ behavioral spaces and, ideally, identify behavioral clusters in the dataset.

Original: Given this framework, the joint policy is decomposed,
Revised: Given this framework, the joint policy is decomposed,

Original: [Equation]
Revised: [Equation]

Original: where in (1) we have assumed that each agent’s latent-conditioned policy is conditionally-independent of the high-level latent behavior z ω given its low-level latent z iα (see Appendix A.1.1 for discussion).
Revised: where in (1) we have assumed that each agent’s latent-conditioned policy is conditionally-independent of the high-level latent behavior z ω given its low-level latent z iα (see Appendix A.1.1 for discussion).

Original: Next, given initial state distribution p ( s 0 ) and latent behavior spaces, the probability of a trajectory τ under joint policy π ( · ) is as follows:
Revised: Next, given initial state distribution p ( s 0 ) and latent behavior spaces, the probability of a trajectory τ under joint policy π ( · ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: We seek to learn the distributions over variables
Revised: We seek to learn the distributions over variables

Original: z α and z ω , alongside the latent-conditioned policies π i ( a i | s t , z iα ) , which maximize trajectory probabilities (4).
Revised: z α and z ω , alongside the latent-conditioned policies π i ( a i | s t , z iα ) , which maximize trajectory probabilities (4).

Original: In Appendix A.1, we derive the following variational lower bound, enabling approximation of these components using parametric models:
Revised: In Appendix A.1, we derive the following variational lower bound, enabling approximation of these components using parametric models:

Original: [Equation]
Revised: [Equation]

Original: where q φ ( z iα | τ ) and p θ ( z iα | z ω ) are, respectively, learned encoder (posterior) and prior distributions over the local behavior latents z iα ; likewise, q φ ( z ω | τ ) and p θ ( z ω ) are, respectively, learned encoder and prior distributions over the joint behavior latent z ω ; β is a KL-weighting term as in β -VAEs [72].
Revised: where q φ ( z iα | τ ) and p θ ( z iα | z ω ) are, respectively, learned encoder (posterior) and prior distributions over the local behavior latents z iα ; likewise, q φ ( z ω | τ ) and p θ ( z ω ) are, respectively, learned encoder and prior distributions over the joint behavior latent z ω ; β is a KL-weighting term as in β -VAEs [72].

Original: [Figure]
Revised: [Figure]

Original: l u
Revised: l u

Original: [Figure]
Revised: [Figure]

Original: Cluster A
Revised: Cluster A

Original: Cluster B
Revised: Cluster B

Original: Cluster C
Revised: Cluster C

Original: [Figure]
Revised: [Figure]

Original: D
Revised: D

Original: Cluster E
Revised: Cluster E

Original: Cluster
Revised: Cluster

Original: [Figure]
Revised: [Figure]

Original: Figure 2b illustrates MOHBA’s model architecture, which is informed by the three bound components (5) to (7).
Revised: Figure 2b illustrates MOHBA’s model architecture, which is informed by the three bound components (5) to (7).

Original: During training, each trajectory τ ∼ D is simultaneously passed through the joint and local encoders, which respectively produce parameters for distributions over z ω and z α (e.g., parameters of Gaussian distributions).
Revised: During training, each trajectory τ ∼ D is simultaneously passed through the joint and local encoders, which respectively produce parameters for distributions over z ω and z α (e.g., parameters of Gaussian distributions).

Original: Samples of low-level latent vectors z α are passed to the reconstructed agent policies π iθ ( a it | s t , z iα ) , which are trained via the reconstruction component (5).
Revised: Samples of low-level latent vectors z α are passed to the reconstructed agent policies π iθ ( a it | s t , z iα ) , which are trained via the reconstruction component (5).

Original: The local KL-divergence component (6) induces the local encoder distribution (which is conditioned directly on τ ) to be similar to the local prior distribution (which is conditioned only on samples z ω ), thus enabling meaningful correlations between the encoded local and joint latent space, as later shown.
Revised: The local KL-divergence component (6) induces the local encoder distribution (which is conditioned directly on τ ) to be similar to the local prior distribution (which is conditioned only on samples z ω ), thus enabling meaningful correlations between the encoded local and joint latent space, as later shown.

Original: Finally, the joint KL-divergence component (7) is akin to that in a standard variational autoencoder [73, 74].
Revised: Finally, the joint KL-divergence component (7) is akin to that in a standard variational autoencoder [73, 74].

Original: Overall, MOHBA enables learning of a hierarchical behavioral space (at the joint and local agent levels, z ω and z α , respectively) that exposes interesting behavioral clusters.
Revised: Overall, MOHBA enables learning of a hierarchical behavioral space (at the joint and local agent levels, z ω and z α , respectively) that exposes interesting behavioral clusters.

Original:  Experiments
Revised:  Experiments

Original: We showcase various use-cases for MOHBA in a range of domains including continuous coordination games, multiagent MuJoCo [14], and OpenAI hide-and-seek [10].
Revised: We showcase various use-cases for MOHBA in a range of domains including continuous coordination games, multiagent MuJoCo [14], and OpenAI hide-and-seek [10].

Original: Appendix A.2 provides data generation, networks, computation, and hyperparameter details.
Revised: Appendix A.2 provides data generation, networks, computation, and hyperparameter details.

Original: Appendix A.7 provides pseudocode.
Revised: Appendix A.7 provides pseudocode.

Original: Data generation.
Revised: Data generation.

Original: Multiagent trajectory data is generated for each domain via the Acme RL library [75], using the TD3 algorithm [76] in a decentralized MARL fashion, with datasets managed using RLDS [77].
Revised: Multiagent trajectory data is generated for each domain via the Acme RL library [75], using the TD3 algorithm [76] in a decentralized MARL fashion, with datasets managed using RLDS [77].

Original: Trajectories are collected at constant intervals throughout training, which also enables analysis of behavioral emergence.
Revised: Trajectories are collected at constant intervals throughout training, which also enables analysis of behavioral emergence.

Original: We conduct a wide sweep over random seeds for data generation, yielding a diverse trajectory dataset (see Appendix A.5 for dataset details and statistics).
Revised: We conduct a wide sweep over random seeds for data generation, yielding a diverse trajectory dataset (see Appendix A.5 for dataset details and statistics).

Original: MOHBA setup.
Revised: MOHBA setup.

Original: To analyze the above data using MOHBA, we use a GMM (Gaussian Mixture
Revised: To analyze the above data using MOHBA, we use a GMM (Gaussian Mixture

Original: Model) for the joint prior, a bidirectional LSTM (long short-term memory network) with GMM head for the joint encoder, an MLP (multi-layer perceptron) with Gaussian head for the local priors, a bidirectional LSTM with Gaussian head for the local encoder, and an MLP for reconstructed policies.
Revised: Model) for the joint prior, a bidirectional LSTM (long short-term memory network) with GMM head for the joint encoder, an MLP (multi-layer perceptron) with Gaussian head for the local priors, a bidirectional LSTM with Gaussian head for the local encoder, and an MLP for reconstructed policies.

Original: GMMs are used for the joint prior and encoder as they produce discernible joint behavior clusters [78], whereas the conditioning of the local prior on z ω yields such clusters at the local level with a standard
Revised: GMMs are used for the joint prior and encoder as they produce discernible joint behavior clusters [78], whereas the conditioning of the local prior on z ω yields such clusters at the local level with a standard

Original: Gaussian head.
Revised: Gaussian head.

Original: We use parameter-sharing across local priors, local encoders, and reconstructed policies, as common in multiagent setups [79], with a unique one-hot vector identiﬁer appended to agent-speciﬁc network inputs to enable heterogeneity in model outputs.
Revised: We use parameter-sharing across local priors, local encoders, and reconstructed policies, as common in multiagent setups [79], with a unique one-hot vector identiﬁer appended to agent-speciﬁc network inputs to enable heterogeneity in model outputs.

Original: Independent analysis of joint ( z ω ) and local ( z α ) behaviors.
Revised: Independent analysis of joint ( z ω ) and local ( z α ) behaviors.

Original: We here analyze the hierarchical latent structure learned by MOHBA.
Revised: We here analyze the hierarchical latent structure learned by MOHBA.

Original: Speciﬁcally, we highlight differences in agent behaviors as identiﬁed at the joint and local levels, respectively, by z ω and z α .
Revised: Speciﬁcally, we highlight differences in agent behaviors as identiﬁed at the joint and local levels, respectively, by z ω and z α .

Original: We ﬁrst conduct a simple sanity check in a 3-agent hill-climbing domain (our earlier example in Fig. 1b).
Revised: We ﬁrst conduct a simple sanity check in a 3-agent hill-climbing domain (our earlier example in Fig. 1b).

Original: States and actions correspond, respectively, to 2D positions ( x, y ) and forces (∆ x, ∆ y ) imparted by each agent for
Revised: States and actions correspond, respectively, to 2D positions ( x, y ) and forces (∆ x, ∆ y ) imparted by each agent for

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: (d) movement.
Revised: (d) movement.

Original: Agents spawn at the origin and are each rewarded for climbing any of three hills (shown in grey in the domain ﬁgures) within the domain in episodes of 50 timesteps.
Revised: Agents spawn at the origin and are each rewarded for climbing any of three hills (shown in grey in the domain ﬁgures) within the domain in episodes of 50 timesteps.

Original: We generate a dataset using 50 independent MARL training trials, each conducted for 1 e 5 environment steps; trajectories are saved every 200 steps, yielding 1 .
Revised: We generate a dataset using 50 independent MARL training trials, each conducted for 1 e 5 environment steps; trajectories are saved every 200 steps, yielding 1 .

Original: 25 e 6 frames of data ( 25000 trajectories
Revised: 25 e 6 frames of data ( 25000 trajectories

Original: Figure 3a highlights several example clusters of joint and local behaviors identiﬁed by MOHBA, using samples of z ω and z α from the joint and local encoders.
Revised: Figure 3a highlights several example clusters of joint and local behaviors identiﬁed by MOHBA, using samples of z ω and z α from the joint and local encoders.

Original: We use Euclidean distances in the original latent spaces to identify nearby vectors, then visualize their 2D projection using principal component analysis.
Revised: We use Euclidean distances in the original latent spaces to identify nearby vectors, then visualize their 2D projection using principal component analysis.

Original: For each cluster, we visualize all associated trajectories in Fig.
Revised: For each cluster, we visualize all associated trajectories in Fig.

Original: 3b, thus enabling analysis of behaviors captured in latent space.
Revised: 3b, thus enabling analysis of behaviors captured in latent space.

Original: Clusters A, B, and C correspond to examples of joint behaviors ( z ω ).
Revised: Clusters A, B, and C correspond to examples of joint behaviors ( z ω ).

Original: Cluster A contains trajectories early in training, where agents have not learned to converge to a particular hill in the domain.
Revised: Cluster A contains trajectories early in training, where agents have not learned to converge to a particular hill in the domain.

Original: In Cluster B, all agents have learned to navigate towards the top-left hill.
Revised: In Cluster B, all agents have learned to navigate towards the top-left hill.

Original: In Cluster C, agents 0 and 2 prefer the bottom-left hill, whereas agent 1 prefers the right hill.
Revised: In Cluster C, agents 0 and 2 prefer the bottom-left hill, whereas agent 1 prefers the right hill.

Original: Overall, clusters at the joint level capture meaningful collective behaviors of the agents.
Revised: Overall, clusters at the joint level capture meaningful collective behaviors of the agents.

Original: Next, we analyze individual agent clusters.
Revised: Next, we analyze individual agent clusters.

Original: Note that for all agents, three prominent clusters are apparent in their respective z α spaces, as each training trial in the data generation process can lead to various agent-wise hill preferences.
Revised: Note that for all agents, three prominent clusters are apparent in their respective z α spaces, as each training trial in the data generation process can lead to various agent-wise hill preferences.

Original: We compare two such clusters, D and E, for agent 1, observing in the trajectory plots that these corresponds to this agent preferring the top-left and bottom-left hills, respectively.
Revised: We compare two such clusters, D and E, for agent 1, observing in the trajectory plots that these corresponds to this agent preferring the top-left and bottom-left hills, respectively.

Original: Similarly, Cluster F corresponds to trajectories where agent 2 prefers the bottom-left hill.
Revised: Similarly, Cluster F corresponds to trajectories where agent 2 prefers the bottom-left hill.

Original: These results illustrate that the local latents reasonably disentangle each agent’s observed behaviors.
Revised: These results illustrate that the local latents reasonably disentangle each agent’s observed behaviors.

Original: [Table]
Revised: [Table]

Original: Coupled analysis of joint ( z ω ) and local ( z α ) behaviors.
Revised: Coupled analysis of joint ( z ω ) and local ( z α ) behaviors.

Original: The above experiments independently analyzed the joint and local latent spaces.
Revised: The above experiments independently analyzed the joint and local latent spaces.

Original: We can also concurrently analyze them to better understand local agent contributions to joint behaviors.
Revised: We can also concurrently analyze them to better understand local agent contributions to joint behaviors.

Original: Consider a two-agent domain with close inter-agent coordination (visualized in the top of Fig. 4a), with state and actionsspaces similar to the hill domain and episodes consisting of 50 timesteps.
Revised: Consider a two-agent domain with close inter-agent coordination (visualized in the top of Fig. 4a), with state and actionsspaces similar to the hill domain and episodes consisting of 50 timesteps.

Original: Three regions are deﬁned in this domain: A and B (circular regions), and C (region exterior to the circles).
Revised: Three regions are deﬁned in this domain: A and B (circular regions), and C (region exterior to the circles).

Original: When an agent enters a given region, it ‘activates’ the corresponding strategy in Table 1, with agents receiving rewards at each timestep according to the joint strategy they have activated.
Revised: When an agent enters a given region, it ‘activates’ the corresponding strategy in Table 1, with agents receiving rewards at each timestep according to the joint strategy they have activated.

Original: For example, if both agents enter region A , they each receive a reward of 1 , whereas if one agent enters A while the other is in exterior region C , neither receives a reward.
Revised: For example, if both agents enter region A , they each receive a reward of 1 , whereas if one agent enters A while the other is in exterior region C , neither receives a reward.

Original: This domain involves a signiﬁcant degree of coordination as agents must discover the rewarding regions, while receiving a sparse reward signal until a valid combination of strategies is discovered.
Revised: This domain involves a signiﬁcant degree of coordination as agents must discover the rewarding regions, while receiving a sparse reward signal until a valid combination of strategies is discovered.

Original: There is also potential for miscoordination: navigating to region B is rewarding assuming the other agent navigates to A , but yields 0 reward if the other agent instead navigates to B (potentially destabilizing training).
Revised: There is also potential for miscoordination: navigating to region B is rewarding assuming the other agent navigates to A , but yields 0 reward if the other agent instead navigates to B (potentially destabilizing training).

Original: We run 50 independent MARL sweeps in this domain, yielding 1 e 6 data frames ( 20000 trajectories).
Revised: We run 50 independent MARL sweeps in this domain, yielding 1 e 6 data frames ( 20000 trajectories).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, MOHBA discovers three dominant joint behavior clusters z ω here.
Revised: 4, MOHBA discovers three dominant joint behavior clusters z ω here.

Original: In each of
Revised: In each of

Original: Figs. 4a to 4c, we highlight one of these clusters and its corresponding local behavior latents
Revised: Figs. 4a to 4c, we highlight one of these clusters and its corresponding local behavior latents

Original: z α for each agent.
Revised: z α for each agent.

Original: MOHBA reveals that across the dataset, the agents have learned to cover all 3 optimal joint behaviors in Table 1: ( A, A ) , ( A, B ) , and ( B, A ) .
Revised: MOHBA reveals that across the dataset, the agents have learned to cover all 3 optimal joint behaviors in Table 1: ( A, A ) , ( A, B ) , and ( B, A ) .

Original: Moreover, despite each agent discovering two local behaviors ( A and B ), z ω highlights only the 3 observed joint behaviors (i.e., does not simply highlight 4 clusters consisting of the Cartesian product of individual agents’ behavior spaces).
Revised: Moreover, despite each agent discovering two local behaviors ( A and B ), z ω highlights only the 3 observed joint behaviors (i.e., does not simply highlight 4 clusters consisting of the Cartesian product of individual agents’ behavior spaces).

Original: Behavior emergence throughout MARL training.
Revised: Behavior emergence throughout MARL training.

Original: We next use MOHBA’s latent space to inspect behavior emergence during MARL training.
Revised: We next use MOHBA’s latent space to inspect behavior emergence during MARL training.

Original: Figure 4d visualizes the training progression of two
Revised: Figure 4d visualizes the training progression of two

Original: MARL runs used for data generation, tracking behavior changes throughout.
Revised: MARL runs used for data generation, tracking behavior changes throughout.

Original: In Training Run 1 (top panel of Fig.
Revised: In Training Run 1 (top panel of Fig.

Original: 4d), the agents converge to and maintain a ﬁxed joint behavior throughout training.
Revised: 4d), the agents converge to and maintain a ﬁxed joint behavior throughout training.

Original: By contrast, Training Run 2 (bottom panel) has numerous behavior changepoints, where agents ﬂip back and forth between preferring one of two clusters in z ω .
Revised: By contrast, Training Run 2 (bottom panel) has numerous behavior changepoints, where agents ﬂip back and forth between preferring one of two clusters in z ω .

Original: Concurrently inspecting the z α space for
Revised: Concurrently inspecting the z α space for

Original: Training Run 2, we observe that agent 0 converges to and maintains a consistent behavior, whereas agent 1 changes its preference sporadically, also explaining the detected changes in joint behaviors.
Revised: Training Run 2, we observe that agent 0 converges to and maintains a consistent behavior, whereas agent 1 changes its preference sporadically, also explaining the detected changes in joint behaviors.

Original: Baseline comparisons.
Revised: Baseline comparisons.

Original: We next compare against baselines previously used for multiagent behavioral analysis in the literature [9, 11].
Revised: We next compare against baselines previously used for multiagent behavioral analysis in the literature [9, 11].

Original: An LSTM baseline conducts next-action prediction at each step in a trajectory τ , using an action-prediction loss (APL), deﬁned as the L 2 loss over predicted vs. ground truth actions.
Revised: An LSTM baseline conducts next-action prediction at each step in a trajectory τ , using an action-prediction loss (APL), deﬁned as the L 2 loss over predicted vs. ground truth actions.

Original: This baseline targets using the LSTM hidden states, rather than a learned distribution over latent variables, for understanding and clustering agent behaviors (akin to the analysis in [9, 11]).
Revised: This baseline targets using the LSTM hidden states, rather than a learned distribution over latent variables, for understanding and clustering agent behaviors (akin to the analysis in [9, 11]).

Original: A ﬂat-VAE baseline provides a non-hierarchical ablation of MOHBA that simply feeds the joint latent z ω (rather than z iα ) to reconstructed agent policies (i.e., π iθ ( a i | s, z ω ) ); we use our usual loss (with the local KL term (6) removed) to train this VAE.
Revised: A ﬂat-VAE baseline provides a non-hierarchical ablation of MOHBA that simply feeds the joint latent z ω (rather than z iα ) to reconstructed agent policies (i.e., π iθ ( a i | s, z ω ) ); we use our usual loss (with the local KL term (6) removed) to train this VAE.

Original: We conduct a hyperparameter sweep for the baselines (see Appendix A.2), reporting the best results averaged over 3 random seeds.
Revised: We conduct a hyperparameter sweep for the baselines (see Appendix A.2), reporting the best results averaged over 3 random seeds.

Original: [Table]
Revised: [Table]

Original: Comparisons are conducted at the joint behavior level: for the LSTM, we use the ﬁnal hidden state as an encoding of each trajectory; for our method and the VAE, we use z ω directly.
Revised: Comparisons are conducted at the joint behavior level: for the LSTM, we use the ﬁnal hidden state as an encoding of each trajectory; for our method and the VAE, we use z ω directly.

Original: Table 2 compares the methods in the hillclimbing and coordination game domains.
Revised: Table 2 compares the methods in the hillclimbing and coordination game domains.

Original: Here we note the APL, which measures how well each method reconstructs ground truth policies.
Revised: Here we note the APL, which measures how well each method reconstructs ground truth policies.

Original: Additionally, we use K-means (sweeping over the # of clusters) to identify behavior clusters for each method, then report the intra-cluster trajectory distance (ICTD), deﬁned as the average distance of all trajectories in a cluster from the mean trajectory in said cluster (akin to intra-cluster point scatter, a common cluster analysis statistic [80]) .
Revised: Additionally, we use K-means (sweeping over the # of clusters) to identify behavior clusters for each method, then report the intra-cluster trajectory distance (ICTD), deﬁned as the average distance of all trajectories in a cluster from the mean trajectory in said cluster (akin to intra-cluster point scatter, a common cluster analysis statistic [80]) .

Original: Combined, these measures provide a proxy for evaluating the latent representations in terms of enabling accurate reconstructions (APL) while clustering similar trajectories together (ICTD).
Revised: Combined, these measures provide a proxy for evaluating the latent representations in terms of enabling accurate reconstructions (APL) while clustering similar trajectories together (ICTD).

Original: MOHBA signiﬁcantly outperforms the LSTM and VAE baselines in terms of APL, while also clustering similar trajectories at the z ω level in terms of ICTD.
Revised: MOHBA signiﬁcantly outperforms the LSTM and VAE baselines in terms of APL, while also clustering similar trajectories at the z ω level in terms of ICTD.

Original: We provide additional results including APL throughout training, full ICTD sweeps, and visualizations of the latent spaces for the baselines in Appendix A.3.2.
Revised: We provide additional results including APL throughout training, full ICTD sweeps, and visualizations of the latent spaces for the baselines in Appendix A.3.2.

Original: Behavior concept discovery.
Revised: Behavior concept discovery.

Original: Next, we test the representation power of the latent spaces learned by MOHBA, and illustrate a means of discovering ‘behavior concepts’ in the latent space.
Revised: Next, we test the representation power of the latent spaces learned by MOHBA, and illustrate a means of discovering ‘behavior concepts’ in the latent space.

Original: We adopt the completeness-aware concept explanations framework of Yeh et al.
Revised: We adopt the completeness-aware concept explanations framework of Yeh et al.

Original: [13], with slight changes to make it amenable to this setting (see Appendix A.2.5).
Revised: [13], with slight changes to make it amenable to this setting (see Appendix A.2.5).

Original: At a high-level, given a set of inputs, a set of concept vectors C in the same space as inputs, and prediction targets (e.g., domain characteristics of interest)
Revised: At a high-level, given a set of inputs, a set of concept vectors C in the same space as inputs, and prediction targets (e.g., domain characteristics of interest)

Original: , Yeh et al.
Revised: , Yeh et al.

Original: [13] deﬁne a framework to compute class-conditioned Shapley values, called
Revised: [13] deﬁne a framework to compute class-conditioned Shapley values, called

Original: ConceptSHAP.
Revised: ConceptSHAP.

Original: To do so, every input is projected onto each of the concept vectors, yielding a vector of ‘concept scores’, which is then passed to a simple prediction head g
Revised: To do so, every input is projected onto each of the concept vectors, yielding a vector of ‘concept scores’, which is then passed to a simple prediction head g

Original: (e.g., small MLP or linear model) to predict the targets and compute the ConceptSHAP.
Revised: (e.g., small MLP or linear model) to predict the targets and compute the ConceptSHAP.

Original: ConceptSHAP provides numerical scores interpreted as the importance of each concept in C for predicting a given target class, which is useful for identifying key concepts (and nearby inputs) associated with certain domain characteristics.
Revised: ConceptSHAP provides numerical scores interpreted as the importance of each concept in C for predicting a given target class, which is useful for identifying key concepts (and nearby inputs) associated with certain domain characteristics.

Original: The concept explanation framework of Yeh et al.
Revised: The concept explanation framework of Yeh et al.

Original: [13] is then used to ﬁrst predict the correct classes (using only z ω , rather than the trajectory τ , as input), identify core concepts
Revised: [13] is then used to ﬁrst predict the correct classes (using only z ω , rather than the trajectory τ , as input), identify core concepts

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we use this technique to identify concepts in z ω space associated with varying characteristics in the hill-climbing domain.
Revised: we use this technique to identify concepts in z ω space associated with varying characteristics in the hill-climbing domain.

Original: The concept set C considered is generated using K-means in the z ω space (see Appendix A.2.5 for details).
Revised: The concept set C considered is generated using K-means in the z ω space (see Appendix A.2.5 for details).

Original: The ﬁrst row of Table 3 shows classes corresponding to increasing levels of agent dispersion in the domain (with dispersion deﬁned as the sum of L 2 -distances of agents from their centroid at the ﬁnal timestep).
Revised: The ﬁrst row of Table 3 shows classes corresponding to increasing levels of agent dispersion in the domain (with dispersion deﬁned as the sum of L 2 -distances of agents from their centroid at the ﬁnal timestep).

Original: For each trajectory τ , we compute the associated dispersion, creating 5 classes of equal-sized bins (labels 0 through 4 mapping to bins of increasing dispersion).
Revised: For each trajectory τ , we compute the associated dispersion, creating 5 classes of equal-sized bins (labels 0 through 4 mapping to bins of increasing dispersion).

Original: We create an 80-20 train-validation split, then train a 2-layer (8 hidden units each) MLP g via a softmax-cross entropy loss to predict the classes using only z ω as input (rather than the actual trajectory τ ).
Revised: We create an 80-20 train-validation split, then train a 2-layer (8 hidden units each) MLP g via a softmax-cross entropy loss to predict the classes using only z ω as input (rather than the actual trajectory τ ).

Original: We attain a validation accuracy of 54.27%, signifying the predictive capabilities of z ω .
Revised: We attain a validation accuracy of 54.27%, signifying the predictive capabilities of z ω .

Original: Next, we compute the class-conditioned ConceptSHAP, thus identifying the top-scoring concept vector for each class.
Revised: Next, we compute the class-conditioned ConceptSHAP, thus identifying the top-scoring concept vector for each class.

Original: In Table 3, we visualize the 20 trajectories with the closest z ω to the top-scoring concept for each class (with the mean agent dispersion across these trajectories listed under each image).
Revised: In Table 3, we visualize the 20 trajectories with the closest z ω to the top-scoring concept for each class (with the mean agent dispersion across these trajectories listed under each image).

Original: Intuitively, the identiﬁed concepts involve agents becoming increasingly dispersed, with agents
Revised: Intuitively, the identiﬁed concepts involve agents becoming increasingly dispersed, with agents

Original: ﬁrst nearly stationary at the center (class 0), then all converging to the same hill (class 1), spreading across two hills (class 2), and ﬁnally covering all hills (classes 3 and 4).
Revised: ﬁrst nearly stationary at the center (class 0), then all converging to the same hill (class 1), spreading across two hills (class 2), and ﬁnally covering all hills (classes 3 and 4).

Original: The second row of Table 3 repeats this experiment, now using classes associated with agents’ sumof-returns.
Revised: The second row of Table 3 repeats this experiment, now using classes associated with agents’ sumof-returns.

Original: We attain a validation set accuracy of 60.98%, and observe behaviors associated with generally increasing reward by identifying top clusters in each class (with some overlaps, e.g., classes 1 and 2 with high standard deviation in returns).
Revised: We attain a validation set accuracy of 60.98%, and observe behaviors associated with generally increasing reward by identifying top clusters in each class (with some overlaps, e.g., classes 1 and 2 with high standard deviation in returns).

Original: Overall, these experiments help quantify the representational capacity of z ω and identify clusters associated with distinct behavioral characteristics.
Revised: Overall, these experiments help quantify the representational capacity of z ω and identify clusters associated with distinct behavioral characteristics.

Original: [Figure]
Revised: [Figure]

Original: Scalability to high-dimensional domains.
Revised: Scalability to high-dimensional domains.

Original: We next test MOHBA’s scalability to high-dimensional domains using the 2-Agent HalfCheetah multiagent MuJoCo domain [14], where agents 0 and 1, respectively, control the back and front limbs of a cheetah to coordinate movement.
Revised: We next test MOHBA’s scalability to high-dimensional domains using the 2-Agent HalfCheetah multiagent MuJoCo domain [14], where agents 0 and 1, respectively, control the back and front limbs of a cheetah to coordinate movement.

Original: Each agent’s state consists of a 6D vector summarizing velocity and position information for the 3 joints it controls, with its 3D action space corresponding to motor torques applied to these joints.
Revised: Each agent’s state consists of a 6D vector summarizing velocity and position information for the 3 joints it controls, with its 3D action space corresponding to motor torques applied to these joints.

Original: The agents coordinate to maximize the forward-speed of the cheetah over episodes of 200 timesteps each.
Revised: The agents coordinate to maximize the forward-speed of the cheetah over episodes of 200 timesteps each.

Original: We generate data using 30 independent trials of MARL training, collecting 1 e 5 total trajectories to train MOHBA.
Revised: We generate data using 30 independent trials of MARL training, collecting 1 e 5 total trajectories to train MOHBA.

Original: Figure 5a visualizes the behavior space learned by MOHBA, wherein we observe several clusters.
Revised: Figure 5a visualizes the behavior space learned by MOHBA, wherein we observe several clusters.

Original: We highlight three example trajectories stemming from distinct clusters in the joint behavior space z ω , showing the corresponding HalfCheetah behaviors in Fig.
Revised: We highlight three example trajectories stemming from distinct clusters in the joint behavior space z ω , showing the corresponding HalfCheetah behaviors in Fig.

Original: 5b.
Revised: 5b.

Original: The left panel of each example provides a view of the cheetah’s overall movement, where we observe key behavior differences: in
Revised: The left panel of each example provides a view of the cheetah’s overall movement, where we observe key behavior differences: in

Original: Example A, the cheetah runs forward using subtle vibration of its limbs; in Example B, it bounces forward with its torso arched up due to its front limb (agent 1) being more extended than its back limb (agent 0); in Example C, it moves closer to the ground with its torso arched down.
Revised: Example A, the cheetah runs forward using subtle vibration of its limbs; in Example B, it bounces forward with its torso arched up due to its front limb (agent 1) being more extended than its back limb (agent 0); in Example C, it moves closer to the ground with its torso arched down.

Original: In the latent space (Fig. 5a), we observe that for Examples B and C, the z 0 α (back limb) latents are close to one another, while the z 1 α (front limb) latents are far apart.
Revised: In the latent space (Fig. 5a), we observe that for Examples B and C, the z 0 α (back limb) latents are close to one another, while the z 1 α (front limb) latents are far apart.

Original: To investigate these local agent behavioral differences, the right panel of Fig.
Revised: To investigate these local agent behavioral differences, the right panel of Fig.

Original: 5b provides an overlaid view of the same frames, with the torso now aligned (making it easier to discern the behaviors of agent 0 vs. agent 1).
Revised: 5b provides an overlaid view of the same frames, with the torso now aligned (making it easier to discern the behaviors of agent 0 vs. agent 1).

Original: Here we observe that the back limb (agent 0) behaves similarly across both examples, while the front limb (agent 1) stays much closer to the head for Example C, in contrast to Example B, which coincides with the ﬁndings in the z α space.
Revised: Here we observe that the back limb (agent 0) behaves similarly across both examples, while the front limb (agent 1) stays much closer to the head for Example C, in contrast to Example B, which coincides with the ﬁndings in the z α space.

Original: Appendix A.3.1 provides additional results for a 4-agent MuJoCo AntWalker environment.
Revised: Appendix A.3.1 provides additional results for a 4-agent MuJoCo AntWalker environment.

Original: Application to externally-trained policies: hide-and-seek game.
Revised: Application to externally-trained policies: hide-and-seek game.

Original: We next apply MOHBA to policies trained by external teams.
Revised: We next apply MOHBA to policies trained by external teams.

Original: We consider the OpenAI hide-and-seek environment [10], wherehiders and 2 seekers compete in a rich environment with various interactive objects (boxes and ramps).
Revised: We consider the OpenAI hide-and-seek environment [10], wherehiders and 2 seekers compete in a rich environment with various interactive objects (boxes and ramps).

Original: OpenAI has open-sourced policies [81] annotated by humans as exhibiting distinctive behaviors at key stages of training.
Revised: OpenAI has open-sourced policies [81] annotated by humans as exhibiting distinctive behaviors at key stages of training.

Original: We consider four policy checkpoints that correspond to the following human annotations: A) ‘running and chasing’, B) ‘fort building’, C) ‘ramp use’, and D) ‘ramp defense’.
Revised: We consider four policy checkpoints that correspond to the following human annotations: A) ‘running and chasing’, B) ‘fort building’, C) ‘ramp use’, and D) ‘ramp defense’.

Original: The state-space used for each of the agents is 100-dimensional, consisting of the agent’s own state (position, rotation, and velocity), states of the other 3 agents, and the states of 3 boxes and 1 ramp in the environment (position, velocity, and box size); each agent’s action space consists of a
Revised: The state-space used for each of the agents is 100-dimensional, consisting of the agent’s own state (position, rotation, and velocity), states of the other 3 agents, and the states of 3 boxes and 1 ramp in the environment (position, velocity, and box size); each agent’s action space consists of a

Original: 3-dimensional force vector, and a ‘glue’ and ‘lock’ action for interacting with objects.
Revised: 3-dimensional force vector, and a ‘glue’ and ‘lock’ action for interacting with objects.

Original: We collecttrajectories per policy checkpoint, each being 200 timesteps long.
Revised: We collecttrajectories per policy checkpoint, each being 200 timesteps long.

Original: These trajectories exhibit a wide distribution of behaviors, as agent and object initializations are random in each episode.
Revised: These trajectories exhibit a wide distribution of behaviors, as agent and object initializations are random in each episode.

Original: We then mix the trajectories collected from all policy checkpoints, then train MOHBA on this shufﬂed dataset.
Revised: We then mix the trajectories collected from all policy checkpoints, then train MOHBA on this shufﬂed dataset.

Original: Figure 6 visualizes the behavior spaces discovered by MOHBA in the hide-and-seek domain.
Revised: Figure 6 visualizes the behavior spaces discovered by MOHBA in the hide-and-seek domain.

Original: Agents 0 and 1 in this ﬁgure correspond to ‘hiders’, whereas agents 2 and 3 are ‘seekers’.
Revised: Agents 0 and 1 in this ﬁgure correspond to ‘hiders’, whereas agents 2 and 3 are ‘seekers’.

Original: We label each of the trajectories in this ﬁgure with the human-expert annotations provided by OpenAI.
Revised: We label each of the trajectories in this ﬁgure with the human-expert annotations provided by OpenAI.

Original: In Fig.
Revised: In Fig.

Original: 6, we observe the presence of behavior clusters that correspond well to the human-expert labels, both at the joint and local agent levels.
Revised: 6, we observe the presence of behavior clusters that correspond well to the human-expert labels, both at the joint and local agent levels.

Original: Interestingly, for the seekers (agents 2 and 3), policy A (‘running and chasing’) is highly distinctive and well-separated from the other behaviors.
Revised: Interestingly, for the seekers (agents 2 and 3), policy A (‘running and chasing’) is highly distinctive and well-separated from the other behaviors.

Original: Moreover, despite the order of emergent behaviors in the original hide-and-seek MARL training being A → B → C → D, policies B (‘fort building’) and D (‘ramp defense’) appear to be behaviorally slightly closer to one another than C (‘ramp use’) and D, both in the joint space and also the local spaces of the seekers (agent 2 and 3).
Revised: Moreover, despite the order of emergent behaviors in the original hide-and-seek MARL training being A → B → C → D, policies B (‘fort building’) and D (‘ramp defense’) appear to be behaviorally slightly closer to one another than C (‘ramp use’) and D, both in the joint space and also the local spaces of the seekers (agent 2 and 3).

Original: This could perhaps be due to both the ‘fort building’ and ‘ramp defense’ policies being associated with situations where the seekers cannot easily ﬁnd the hiders, due to the hiders using obstacles to block entrances (B) and moving ramps to prevent their effective use (D).
Revised: This could perhaps be due to both the ‘fort building’ and ‘ramp defense’ policies being associated with situations where the seekers cannot easily ﬁnd the hiders, due to the hiders using obstacles to block entrances (B) and moving ramps to prevent their effective use (D).

Original: Overall, these experiments help to validate MOHBA’s learned latent spaces using policy labels manually annotated by human experts, and highlight its applicability to behaviorally-rich domains.
Revised: Overall, these experiments help to validate MOHBA’s learned latent spaces using policy labels manually annotated by human experts, and highlight its applicability to behaviorally-rich domains.

Original:  Discussion
Revised:  Discussion

Original: Our proposed method, MOHBA, leverages trajectory data to better understand multiagent behaviors during and after training.
Revised: Our proposed method, MOHBA, leverages trajectory data to better understand multiagent behaviors during and after training.

Original: MOHBA assumes no knowledge of agents’ underlying training algorithms, does not require access to their hidden states or internal models, and applies even to reward-free settings.
Revised: MOHBA assumes no knowledge of agents’ underlying training algorithms, does not require access to their hidden states or internal models, and applies even to reward-free settings.

Original: Our experiments showcased a variety of applications of MOHBA, including the analysis of joint and local agent behaviors, monitoring of behavior emergence throughout training, discovery of behavioral concepts associated with certain domain criteria, and disentanglement of third-partylabeled behaviors from open-source policies such as those for OpenAI hide-and-seek [10].
Revised: Our experiments showcased a variety of applications of MOHBA, including the analysis of joint and local agent behaviors, monitoring of behavior emergence throughout training, discovery of behavioral concepts associated with certain domain criteria, and disentanglement of third-partylabeled behaviors from open-source policies such as those for OpenAI hide-and-seek [10].

Original: While we believe our approach is an important step in terms of increasing the understanding of multiagent systems, there are several limitations and potential societal impacts of note.
Revised: While we believe our approach is an important step in terms of increasing the understanding of multiagent systems, there are several limitations and potential societal impacts of note.

Original: One limitation is related to the collection of a dataset of agent behaviors.
Revised: One limitation is related to the collection of a dataset of agent behaviors.

Original: Understanding emergent behaviors throughout MARL training using MOHBA requires the storage of large-scale trajectory datasets, which could potentially take a lot of storage and compute power to generate.
Revised: Understanding emergent behaviors throughout MARL training using MOHBA requires the storage of large-scale trajectory datasets, which could potentially take a lot of storage and compute power to generate.

Original: It would be interesting to consider follow-ups to our model that learn behavioral clusters in a streaming fashion, thus building knowledge of behaviors over time and permitting older data to be discarded throughout agent training.
Revised: It would be interesting to consider follow-ups to our model that learn behavioral clusters in a streaming fashion, thus building knowledge of behaviors over time and permitting older data to be discarded throughout agent training.

Original: Moreover, in our datasets, we collected trajectories at uniform intervals throughout original MARL training.
Revised: Moreover, in our datasets, we collected trajectories at uniform intervals throughout original MARL training.

Original: However, it might be interesting to consider a non-uniform collection scheme, e.g., collecting trajectories only when they are detected to diverge from behavioral clusters of a pre-trained MOHBA model.
Revised: However, it might be interesting to consider a non-uniform collection scheme, e.g., collecting trajectories only when they are detected to diverge from behavioral clusters of a pre-trained MOHBA model.

Original: Finally, it might be interesting to enable the agents to ﬁrst learn joint behaviors / ‘skills’ using the proposed method, and leverage them to transfer to related downstream tasks (potentially more sample-efﬁciently than training from scratch).
Revised: Finally, it might be interesting to enable the agents to ﬁrst learn joint behaviors / ‘skills’ using the proposed method, and leverage them to transfer to related downstream tasks (potentially more sample-efﬁciently than training from scratch).

Original: The study of behavioral interactions in multiagent systems can potentially be used for both positive and negative societal applications.
Revised: The study of behavioral interactions in multiagent systems can potentially be used for both positive and negative societal applications.

Original: For example, such an approach could be used to prevent certain undesirable behaviors by agents that interact with humans (e.g, self-driving cars), but potentially also used by adversaries to predict and exploit other behaviors (e.g., exploiting certain human preferences for harm), or even inadvertently cause harm due to misinterpretation of certain behavioral modes.
Revised: For example, such an approach could be used to prevent certain undesirable behaviors by agents that interact with humans (e.g, self-driving cars), but potentially also used by adversaries to predict and exploit other behaviors (e.g., exploiting certain human preferences for harm), or even inadvertently cause harm due to misinterpretation of certain behavioral modes.

Original: As such, further research and evaluation will be required prior to deployment of this and related behavioral analysis approaches to human-facing domains.
Revised: As such, further research and evaluation will be required prior to deployment of this and related behavioral analysis approaches to human-facing domains.

Original: Nonetheless, as agent capabilities continue to grow, our view is that behavioral analysis of multiagent systems will become increasingly important and should complement traditional reward-based performance monitoring.
Revised: Nonetheless, as agent capabilities continue to grow, our view is that behavioral analysis of multiagent systems will become increasingly important and should complement traditional reward-based performance monitoring.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank Asma Ghandeharioun, Meredith Morris, and Kathy Meier-Hellstern for their helpful feedback and support during the paper writing phase.
Revised: We thank Asma Ghandeharioun, Meredith Morris, and Kathy Meier-Hellstern for their helpful feedback and support during the paper writing phase.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We include these in the Discussions section.
Revised: [Yes] We include these in the Discussions section.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We include these in the discussions section.
Revised: [Yes] We include these in the discussions section.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] These are provided in the main paper and the proofs included in the appendix.
Revised: [Yes] These are provided in the main paper and the proofs included in the appendix.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Proofs are provided in the appendix.
Revised: [Yes] Proofs are provided in the appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide details for experiment reproducibility in Appendix A.2.
Revised: [Yes] We provide details for experiment reproducibility in Appendix A.2.

Original: We also include model highlevel code in Appendix A.7.
Revised: We also include model highlevel code in Appendix A.7.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We provide all training details in Appendix A.2.
Revised: [Yes] We provide all training details in Appendix A.2.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We provide all computational details in Appendix A.2.
Revised: [Yes] We provide all computational details in Appendix A.2.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: Our work does not use existing assets.
Revised: Our work does not use existing assets.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not crowdsourced or human subject data.
Revised: Our work does not crowdsourced or human subject data.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: Our work does not crowdsourced or human subject data.
Revised: Our work does not crowdsourced or human subject data.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] Our work does not crowdsourced or human subject data.
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: A Appendix for ‘Beyond Rewards: a Hierarchical Perspective on Ofﬂine Multiagent Behavioral Analysis’ A.1 Derivations

Original: [-A.1.
-]
Revised: [-A.1.
-]

Original: On the Conditional Independence of π i from z ω given z iα

Original: In (1), we assumed that each agent’s latent-conditioned policy is conditionally-independent of the high-level latent behavior z ω given its low-level latent z iα .

Original: This section provides justiﬁcation of this assumption.

Original: One of the prototypical paradigms in multiagent (MARL) training is that of ‘centralized training, decentralized execution’.

Original: In this regime, each agent’s decision-making policy is conditioned only on its available local information / local observations during execution; thus, the behaviors exhibited by each agent are ultimately informed by their local information, rather than global (joint) information.

Original: In our setting, z iα serves to provide this local behavior context for each agent.

Original: Thus, the assumption of the policy being conditionally-independent of z ω given z iα corresponds well to the assumption of agents only using local information (rather than joint information) in MARL to inform their policy/decision-making.

Original: Having said this, we also note that there is a strong relationship between local and joint behavioral distributions in our setting, which helps provide a coordination signal to the policy implicitly through z iα .

Original: Speciﬁcally, our derivation results in a local behavior prior term, p ( z iα | z ω ) ; using this term, the joint behavior latent z ω is able to inﬂuence the learned space over local behavior latents z iα .

Original: Thus, the information-ﬂow in our framework can be more intuitively described as follows:

Original: The joint behavior latent observes a trajectory and summarizes the joint behavior exhibited in it (e.g., team-wide cooperation, competition, etc.) in z ω . 2.

Original: Subsequently, this joint latent z ω affects the local behavior spaces z iα , which are then sampled and used to inform each agent how to behave locally in order to achieve the joint behavior observed.

Original: Overall, the above relationship between the two latent spaces implies makes the conditionalindependence of the policy from z ω a reasonable simplifying assumption.

Original: A.1.2 Derivation of the Variational Lower Bound

Original: This section details the derivation of the variational lower bound described in Section 3 of the main text, which is used for training MOHBA.

Original: Similar to traditional variational autoencoder approaches [73, 74], we approximate the maximization of the latent-conditioned trajectory probability (4) in the main text using the evidence lower bound

Original: [Equation]

Original: where q φ ( ·| τ ) and p θ ( · ) are, respectively, learned posterior and prior distributions.

Original: Using (4) from the main text to expand this expression yields

Original: [Equation]

Original: Note that the terms p ( s 0 ) and p ( s t +1 | s t , a t ) stem from the underlying MA-MDP environment

Original: [Equation]

Original: and thus cannot be optimized via parameters φ and θ .

Original: Dropping these extraneous terms yields,

Original: [Equation]

Original: [Equation]

Original: The ﬁrst term simply relates to trajectory reconstruction, inducing our agent-wise policies π iθ to behave similarly to the observed trajectories in the dataset.

Original: The second term is a regularization term involving our two latent parameters, which we further simplify using assumptions used in previous multi-level VAE models [67].

Original: First, we assume that the posterior distribution is factorizable when conditioned on a particular trajectory, as follows,

Original: [Equation]

Original: and similarly for the prior,

Original: [Equation]

Original: We next simplify the KL-divergence component of (10) as follows by combining it with (12) and (14):

Original: [Equation]

Original: Combining this result with (10) and additionally modulating the KL-terms in the lower bound as in β -VAEs [72], our overall objective simpliﬁes to

Original: [Equation]

Original: A.2 Experiment Details and Hyperparameters

Original: This section provides an overview of the details for our experiments.

Original: A.2.1 Data Generation Details

Original: The multiagent trajectory data analyzed in this paper is generated using the Acme RL library [75], using the TD3 algorithm [76] in a decentralized fashion, with dataset management handled using

Original: RLDS [77].

Original: Each agent uses a 2-layer MLP (256 hidden units for each layer) for its TD3 network (using the vanilla TD3 network speciﬁcations in Acme), with the hyperparameters used for data generation summarized for each domain in Table 4.

Original: Note that the large number of training trials/seeds used in this data generation pipeline serve to produce a wide variety of agent behaviors in the dataset subsequently analyzed by MOHBA

Original: [Table]

Original: [Table]

Original: A.2.2 MOHBA Hyperparameters

Original: We conduct a wide hyperparameter sweep for training MOHBA itself, summarized in Table 5.

Original: Our joint and local encoder models each consist of a bidirectional LSTM, followed by a 2-layer MLP head for mapping to latent distribution parameters; a GMM head is used for the joint encoder, and a

Original: Gaussian for the local encoder.

Original: Our local prior model consists of a 2-layer MLP with Gaussian head, with our joint prior model simply being the learned parameters of a GMM.

Original: All models use ReLU for intermediate layer activations.

Original: Note that we found that cyclically-annealing [82] the β term in our variational lower bound from 0 to the values speciﬁed in Table 5 to help avoid KL-vanishing.

Original: Sweeps are conducted over the MOHBA latent space dimensionality ( 4 and 8 -dimensional), MLP hidden sizes ( 64 and 128 units), LSTM hidden sizes ( 64 and 128 units), Adam optimizer [83] learning rates ( 0 . 001 and 0 . 0001 ), with 3 seeds per parameter set (standard deviations are reported over all seeds).

Original: All MOHBA and baseline training is conducted using the Adam optimizer [83], with Adam parameters β 1 = 0 .

Original: 9 , β 2 = 0 .

Original: 999 , gradient clipping using a max global norm threshold of 10 . 0 , and learning rates swept over as indicated in the tables.

Original: A.2.3 Baseline Model Hyperparameters

Original: We similarly conduct a wide sweep over hyperparameters for the considered baselines, summarized in Table 6.

Original: A.2.4 Computational Details

Original: For MARL trajectory data generation, we used an internal CPU cluster for both the 3-agent hillclimbing and 2-agent coordination domains, using TPUs for only the multiagent MuJoCo data generation.

Original: For training MOHBA itself, we used TPUs for the 3-agent hill-climbing and 2-agent coordination domains, and a Tesla V100 GPU cluster for the MuJoCo environment.

Original: For training the

Original: LSTM and VAE baselines, we used Tesla P100 GPU clusters.

Original: [Table]

Original: [Table]

Original: A.2.5 Concept Discovery Framework

Original: This section describes details of how we apply the completeness-aware concept-based explanation framework of Yeh et al.

Original: [13] to discover interesting concepts in our setting.

Original: Given a characteristic of interest (e.g., the level of dispersion of agents), we deﬁne a training set consisting of joint latents z ω and class labels y

Original: (e.g., classes corresponding to different intervals of team returns).

Original: Yeh et al.

Original: [13] seek to identify a set of concept vectors that are sufﬁcient for predicting the labels given the inputs.

Original: Speciﬁcally, they let C =

Original: { c j } mj

Original: =1 denote the set of concepts, which are unique vectors where c j ∈ R D ω ∀ j .

Original: Normalizing z ω , we can then use the inner product (cid:104) z ω , c j (cid:105) as a similarity measure between z ω and concept c j .

Original: The concept product is deﬁned ν c ( z ω ) =

Original: TH ( (cid:104) z ω , c j (cid:105) , κ ) ∈ R m , where TH ( · , κ ) clips values less than κ to 0 .

Original: Normalizing the concept product yields the concept score ˆ ν c ( z ω )

Original: =

Original: ν c ( z ω ) /

Original: || ν c

Original: ( z ω )

Original: || 2 ∈ R m , where elements provide a measure of similarity of the input z ω to each of the m concept vectors.

Original: Using these deﬁnitions, we can gauge the representational power of z ω by learning a mapping g : ˆ ν c ( z ω )

Original: →

Original: y .

Original: In practice, g is a simple model (e.g., shallow network or linear projection) so as to gauge the expressivity of the latent space.

Original: Given a mapping g , we use the classiﬁcation accuracy as a ‘completeness score’ [13] for the set of concepts C , deﬁned η = sup g P z ω ,y ∼ V (cid:2)

Original: y = arg max y (cid:48) g ( ν c ( z ω )) (cid:3) , where V is a validation set.

Original: Importantly, this approach permits us to compute class-conditioned Shapley values, called ConceptSHAP in the framework of [13].

Original: Speciﬁcally, for a given class k , the class-conditioned ConceptSHAP value for each concept c j is deﬁned,

Original: [Equation]

Original: where η k is the completeness score for class k (computed simply as the classiﬁcation accuracy of the model for the subset of validation points with ground truth label k ).

Original: ConceptSHAP provides a measure of importance of each concept c j for predicting the outcomes associated with a given class, which we can then use to identify the trajectories.

Original: The experiments in our main paper use the above setup to both quantitatively evaluate z ω in terms of representation power, and also reveal interesting concepts associated with relevant characteristics in the domain.

Original: For each of the prediction problems considered in the main paper (agent dispersion and agent returns), we create an 80-20 training-validation data split (over 5 classes).

Original: For concept generation we use

Original: K-means with 16 and 24 clusters, a 2-layer MLP (with 8 hidden units per layer) for the prediction head g , training with a batch size of 64 , and a concept threshold κ of 0.0 and 0.3 for the dispersion and agent return experiments, respectively, and train the prediction head for 1 e 4 steps.

Original: As in Yeh et al.

Original: [13], we use KernelSHAP [84] to approximate ConceptSHAP efﬁciently.

Original: A.3 Additional Results A.3.1 Additional Large-scale Experiments

Original: MultiAgent MuJoCo

Original: AntWalker Domain Results.

Original: We also consider the MultiAgent MuJoCo

Original: AntWalker domain, wherein 4 agents each control one of 4 ant legs to coordinate movement towards the + x direction (Fig. 7a).

Original: To collect data for this domain, we conduct a wider MARL parameter sweep using both the TD3 and SAC algorithms, with varying training batch sizes and learning rates to gather a widely-varying dataset of behaviors.

Original: We subsequently train MOHBA on this data, visualizing the learned behavior clusters in Fig.

Original: 7b.

Original: We observe several behavior clusters of interest; notably, a large joint cluster exists in the top-right region of the joint and local returns, which, upon inspection of underlying trajectory videos, corresponds to cases where agents attain very low return.

Original: Similarly, there exists a smaller cluster in the lower-left region of the joint returns that also attains very low performance.

Original: The remaining clusters correspond to the agents displaying various ant-poses, and moving only incrementally.

Original: One exceptional cluster also exists in the left region of the joint behavior space, which attains medium-level return (points that are primarily red in color).

Original: On closer inspection, the AntWalker behavior in this cluster corresponds to one of the agents learning a reasonably good walking gait, while the remaining three agents remain stationary.

Original: A.3.2 Additional Baseline Results

Original: Figures 8 and 9 provide additional results for the baseline comparisons conducted in the main paper.

Original: Speciﬁcally, (a) in each ﬁgure provides an expanded comparison of ICTD (with a sweep over the # of clusters in K-means).

Original: In (b) of each ﬁgure, we visualize the convergence of the APL throughout training for MOHBA and the baselines.

Original: Subﬁgures (c) and (d), respectively, show a PCA and UMAP projection of the latent space discovered by the LSTM baseline (recall from the main paper that ‘joint embeddings’ used for the LSTM are simply the ﬁnal hidden state).

Original: Likewise, subﬁgures (e) and (f) show the same projections for the VAE baseline.

Original: As especially evident in the coordination domain (Fig. 9), the joint LSTM baseline embeddings (whether using PCA or UMAP projections, Figs.

Original: 9c and 9d, respectively) do not reveal to the fairly interpretable 3 clusters of coordinated behaviors discussed in the main paper.

Original: By contrast, the VAE baseline (in both the PCA and UMAP case, Figs. 9e and 9f, respectively) does produce these clusters; despite this, note that the key limitation of this baseline is its non-hierarchical nature, which makes identiﬁcation of the local-to-joint behavioral correspondences noted in the main paper signiﬁcantly more difﬁcult.

Original: A.4 Hyperparameter Ablations and Effects on Latent Spaces

Original: This section conducts an analysis of the effect of key hyperparameters on the latent spaces learned by

Original: MOHBA.

Original: We focus our analysis on the KL-loss β term in the training objective, alongside the latent z ω and z α dimensionality.

Original: We both compare the policy reconstruction loss (5) across ablations to attain a quantitative comparison of the quality of the latent encodings, and additionally compare the structure of the learned latent spaces themselves as a function of these parameters for a qualitative understanding.

Original: First, in terms of the raw policy reconstruction loss, we ﬁnd in Figs.

Original: 10a, 11a and 12a that the latent dimensionality has a higher impact on policy reconstruction performance than the weighing term β ; intuitively, increasing latent dimensionality leads to better reconstruction (lower loss) due to the latent

Original: [Figure]

Original: A g e n t

Original: R e t u r n

Original: [Figure]

Original: S u m o f a g e n t - w i s e r e t u r n s (

Original: [Figure]

Original: space being able to encode more behavioral information about the agent trajectories.

Original: By contrast, the effects of the KL weighing term are more negligible in terms of reconstruction loss.

Original: Second, we inspect the effects of these hyperparameter sweeps on the learned latent distributions themselves.

Original: Speciﬁcally, the respective panels (b)-(e) of each of Fig.

Original: 10, Fig. 11, and Fig.visualize the change in latent space structure as a function of the latent dimensionality and β .

Original: At a high-level, prominent behavioral clusters mentioned in the main text are re-discovered throughout these parameter sweeps (e.g., the three local clusters for the hill climbing environment, the joint and local clusters for the coordination environment, and the various walking gait clusters for the

Original: HalfCheetah environment).

Original: For the more complex latent spaces (e.g., HalfCheetah), increasing the dimensionality of z tends to increase the number of joint clusters

Original: identiﬁed

Original: (e.g., compare Fig. 12c versus Fig. 12e); this is intuitive as a larger latent space results in a richer encoding, capable of distinguishing more nuanced behaviors.

Original: Increasing the KL β term from 1 e − 4 to 1 e − 2 tends to

Original: APL

Original: ICTD

Original: K=

Original: K=

Original: K=

Original: LSTM 4 .

Original: 8 ± 0 .0 .

Original: 60 ± 0 .0 .

Original: 51 ± 0 .0 .

Original: 47 ± 0 .

Original: VAE 6 .

Original: 9 ± 0 .0 .

Original: 44 ± 0 .0 .

Original: 30 ± 0 .0 .

Original: 18 ± 0 .

Original: MOHBA 1 .

Original: 3 ± 0 .0 .

Original: 46 ± 0 .0 .

Original: 31 ± 0 .0 .

Original: 17 ± 0 .

Original: [Figure]

Original: LSTM

Original: [Figure]

Original: result in clusters that overlap more / are ‘softer’ and slightly less distinguishable (e.g., comparing z ω in Fig. 12c versus Fig. 12d); this also aligns well with intuition, as increasing the β term prioritizes the KL divergence between the posterior and prior, which deprioritizes disentangling the behaviors for the policy reconstruction term (5).

Original: Overall, these results provide us intuition in terms of the role of these hyperparameters in the behavior clusters learned.

Original: At a high level, it appears that the sensitivity of the results to these hyperparameters is fairly low (in the range of values explored in these experiments), thus allowing the high level behaviors discovered to remain reasonably distinctive across the various sweeps.

Original: APL

Original: ICTD

Original: K=

Original: K=

Original: K=

Original: LSTM 2 .

Original: 9 ± 0 .0 .

Original: 41 ± 0 .0 .

Original: 40 ± 0 .0 .

Original: 34 ± 0 .

Original: VAE 5 . 0 ± 0 .0 .

Original: 18 ± 0 .0 .

Original: 17 ± 0 .0 .

Original: 16 ± 0 .

Original: MOHBA 2 .

Original: 4 ± 0 .0 .

Original: 18 ± 0 .0 .

Original: 17 ± 0 .0 .

Original: 16 ± 0 .

Original: [Figure]
Revised: [N/A]

Original: LSTM Baseline

Original: VAE Baseline

Original: [Figure]

Original: A.5 Additional Dataset Details / Reward Distributions

Original: This section provides additional details and statistics regarding the trajectory datasets used for evaluation.

Original: Speciﬁcally, one of the core properties of datasets typically used for ofﬂine RL is the inclusion of a large number of sub-optimal and random trajectories [85].

Original: As noted earlier, the trajectory data for the agents in our experiments is collected from numerous underlying MARL training runs for each domain.

Original: To help ensure the collection and analysis of a diverse set of behaviors including sub-optimal ones, we not only collect trajectories at the end of MARL training, but rather from policy checkpoints throughout all of training.

Original: All such trajectories, including those stemming from randomly-initialized agent policies at the beginning of training, are shufﬂed together and used to simultaneously train MOHBA.

Original: [Table]

Original: Here we more closely inspect our datasets to better understand the distribution of agent behaviors.

Original: At a high level, we ﬁrst note that our datasets consist of a signiﬁcant proportion of sub-optimal trajectories.

Original: Speciﬁcally, in Table 7, we note the percent of trajectories in each of the considered datasets that attain less than 50% of maximum observed return.

Original: Note that in the high-dimensional

Original: MultiAgent MuJoCo environments especially, a large proportion of low-reward trajectories exist in the dataset, with the HalfCheetah dataset being a particularly notable one where 92.69% of datasets attain less than 50% of max observed return.

Original: We additionally provide a more detailed overview of trajectory returns in each of Figs. 13a, 14a, 15a and 16a.

Original: These ﬁgures visualize the distribution of individual agent returns in each of the considered environments (respectively, hill-climbing, coordination game, 2-agent HalfCheetah, and 4-agent

Original: AntWalker).

Original: Notably, many of the trajectories in the HalfCheetah domain are skewed towards medium and low-return behaviors, with very few high-return trajectories present (Fig. 13a).

Original: Interestingly, in the AntWalker dataset we observe a bimodal return distribution (seemingly consisting of very low return and very high return trajectories, as seen in Fig. 14a).

Original: Closer inspection of this domain reveals that a large number of training runs result in highly sub-optimal behaviors with very low returns; in

Original: Fig.

Original: 14b, we visualize the distribution of returns with these low-return trajectories excluded, observing a similar distribution to the HalfCheetah dataset (consisting primarily of medium-return trajectories).

Original: Similarly, a large proportion of trajectories with extremely low returns exist in the hill-climbing and coordination game domains (Figs. 15a and 16a); we veriﬁed that these low-return trajectories consist primarily of random agent behaviors collected early in training.

Original: Overall, this analysis is helpful in determining that the analyzed datasets exhibit a wide range of trajectories (in terms of agent returns), covering random behaviors, sub-optimal behaviors, and high-return behaviors, similar to typical ofﬂine RL datasets [85].

Original: As part of this investigation, we also analyze the discovered behavior spaces (both at joint and local agent levels) with respect to the return distributions.

Original: Speciﬁcally, Figs.

Original: 13b, 14c, 15b and 16b visualize the behavior spaces learned by MOHBA, which are then labeled by the ground truth return corresponding to each trajectory.

Original: Interestingly, despite MOHBA not using any reward or return information in learning agent behavior spaces, clear clusters corresponding to low-return (random/early-training trajectories), medium-return trajectories, and high-return trajectories are automatically discovered in all of the considered domains.

Original: For example, in the hill-climbing environment, many of the trajectories corresponding to low returns are clustered in the center of the joint latent space in Fig.

Original: 15b; in the

Original: HalfCheetah environment, a prominent cluster of high-return trajectories is visible in the top-left and bottom-left local latent spaces of agents 0 and 1, respectively (Fig. 14c); in the AntWalker environment, the lowest-return trajectories are clustered in distinctive regions both at the joint and local level, with a medium-return trajectory set also visible in the latent spaces (Fig. 14c).

Original: Overall, these results indicate the richness of the learned latent spaces in terms of not only raw trajectories but also their capacity to cluster trajectories that exhibit similar performance, even without observing the reward function.

Original: A.6 Visualizing z ω and z α

Original: This section summarizes the procedure used to generate the behavior space ﬁgures associated with z ω and z α .

Original: The visualization procedure we use is as follows:

Original: We train MOHBA using the ofﬂine behavior datasets of interest.

Original: Following training of MOHBA, we pass each trajectory τ through both the joint encoder q φ ( z ω | τ ) and the agent-wise local encoders q φ ( z iα | τ ) , sampling a joint latent z ω and a set of

Original: local latents

Original: z iα

Original: for all N

Original: agents accordingly

Original: Thus, for K

Original: such trajectories

Original: we obtain a set of latent parameters { ( z ω , z 1 α , · · · , z Nα ) k } Kk =1 . 3.

Original: As the latent parameters are high -dimensional in nature we
Revised: Our work does not crowdsourced or human subject data.

Original: visualize

Original: their

Original: 2D

Original: projection (e.g., using Principal Component Analysis), thus yielding the behavior space visualizations such as those in Fig. 3a.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: n

Original: [Figure]

Original: A.7 Model high-level code

Original: Listing 1 provides the high-level code of the MOHBA model.

Original: Note that the components of our model are standard network modules (e

Original: [Figure]

Original: g., MLPs, bidirectional LSTMs, etc.), which are indicated below and can be implemented using any desired neural network library.

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]