Concept-based Understanding of Emergent Multi-Agent Behavior
Abstract
This work studies concept-based interpretability in the context of multi-agent learning.
Unlike supervised learning, where there have been efforts to understand a model’s decisions, multi-agent interpretability remains under-investigated.
This is in part due to the increased complexity of the multi-agent setting—interpreting the decisions of multiple agents over time is combinatorially more complex than understanding individual, static decisions—but is also a reﬂection of the limited availability of tools for understanding multi-agent behavior.
Interactions between agents, and coordination generally, remain difﬁcult to gauge in MARL.
In this work, we propose Concept Bottleneck Policies (CBPs) as a method for learning intrinsically interpretable, concept-based policies with MARL.
We demonstrate that, by conditioning each agent’s action on a set of human-understandable concepts, our method enables post-hoc behavioral analysis via concept intervention that is infeasible with standard policy architectures.
Experiments show that concept interventions over CBPs reliably detect when agents have learned to coordinate with each other in environments that do not demand coordination, and detect those environments in which coordination is required.
Moreover, we ﬁnd evidence that
CBPs can detect coordination failures (such as lazy agents) and expose the lowlevel inter-agent information that underpins emergent coordination.
Finally, we demonstrate that our approach matches the performance of standard, non-conceptbased policies; thereby achieving interpretability without sacriﬁcing performance.
Introduction
Multi-agent learning techniques continue to play a crucial role in the development of scalable and generally-capable AI systems.
In addition to well-known successes in board and card games (e g.,
Go [1], Chess and Shogi [2], Poker [3], Stratego [4], Hanabi [5]) and massively-multiplayer online games (e.g., StarCraft [6], Dota2 [7]), multi-agent learning has enabled agents to develop a range of coordination capabilities, such as navigating game-theoretic social dilemmas [8], balancing low-level control with high-level strategy in football [9], allocating roles and conventions [10]; and even developing socioeconomic behaviors such as bartering [11] and tax policy design [12].
For all its success, multi-agent learning still lacks in a critical area: interpretability.
Understanding emergent multi-agent behavior is challenging—for one, it is difﬁcult to decipher the nature of a learned coordination strategy (or whether agents have learned to coordinate at all)—and these issues are only compounded by the inherent opacity of neural networks, which state of the art multiagent methods rely upon heavily.
For this reason, recent work has shifted focus from traditional measures of performance (reward, evaluations against human experts, etc) to better understanding emergent behaviors [13].
Related methods perform behavioral analysis in a post-hoc manner, either by visualizing trajectories [14], quantifying basic statistics related to agent behavior [9], or measuring the predictability of game-related concepts from a network’s activations [15]. In the supervised learning
[Figure]
literature, an alternative to post-hoc analysis is intrinsic interpretability , in which a model’s decisions incorporate human-understandable concepts directly, rather than extracting them from models after training.
Intrinsically interpretable models can be preferable when concept-based explanations cannot be learned automatically, or if extracted concepts are unreliable [16].
Of particular relevance is the work of Koh et al.
[17], which showed that it is possible to train an end-to-end neural network that is “bottlenecked" by human-understandable concepts, enabling intrinsic, concept-based interpretability while maintaining high performance in image classiﬁcation tasks.
In this work, we bridge the multi-agent learning and concept-based interpretability domains by introducing Concept Bottleneck Policies (CBPs)—a class of intrinsically-interpretable, conceptbased models for MARL.
CBPs force agents to make decisions by ﬁrst predicting an intermediate set of human-understandable concepts, then using those concepts to select actions.
CBPs increase interpretability while also targeting important facets of the multi-agent learning problem.
For example, if a subset of the concepts available to each agent correspond to other agents in the environment, learning through a concept bottleneck incentivizes agents to model each other, which has been shown in prior work to beneﬁt performance [18, 19].
Moreover, by intervening on concepts related to other agents, we can obtain in-depth analysis of emergent multi-agent behaviors, and speciﬁcally learned coordination, that is much more difﬁcult to attain from reward or qualitative analysis alone.
In our experiments, we show that concept intervention reveals key information underlying learned multi-agent coordination.
Speciﬁcally, concept interventions can distinguish agents who have learned to coordinate from those that act independently, expose the distinct inter-agent factors that drive coordination between agents (when it emerges), and even diagnose common multi-agent failure modes such as lazy agents.
Finally, beyond behavioral analysis, we demonstrate experimentally that Concept
Bottleneck Policies perform comparably to non-concept based policies on difﬁcult coordination tasks; meaning that CBPs achieve concept-based interpretability without sacriﬁng performance.
In sum, our contributions are as follows: (i)
We introduce Concept Bottleneck Policies as an interpretable, concept-based learning architecture for MARL; (ii) We highlight the role CBPs can play in better understanding multi-agent behavior (and speciﬁcally coordination); (iii) We show that CBPs can learn effective policies that match the performance of non-concept-based network architectures.
Related Work
Our work lies at the intersection of interpretability and MARL.
In interpretability, computer vision works have used saliency maps to help provide local explanations (at the pixel level) for model predictions [20–23].
Interpretability using grounded concepts has been previously explored to provide more meaningful, human-understandable explanations of model decisions [24–30].
As described earlier, concept bottleneck models [17] constrain the networks through such grounded concepts to enable direct interpretability of the model itself.
RL interpretability techniques have included those that either directly increase agents’ model transparency [31, 32], where we take transparency loosely to mean the ability of a human observer to understand the underlying decision-making process of the agent; or analyze agent behaviors from a post-hoc perspective [13, 15, 33–38].
Approaches that directly increase transparency include those that use model compression in combination with decision trees to yield more interpretable agent policies [31], or rely upon causal decision trees with limited depth to explore causality of agent decisions [32], However, these approaches have not yet been extended to MARL settings involving complex observations (e.g., images), where function approximators are typically needed to solve underlying tasks.
Post-hoc RL interpretability methods have used computer vision-based saliency techniques to better attribute decisions to image observation regions [33], highlighting states that lead to major differences in agent behaviors [34], and summarize key agent behaviors using measures such as action uncertainty [35].
Increasingly, techniques making use of natural language have been used to enable human understanding of agents through instructions [39] or policy explanations [40].
Interpretability of MARL agents has received limited attention in prior works, with primary investigations gauging agents’ internal representations by making predictions about future outcomes [36], visualizing latent clusterings of agents’ neural activation vectors [37, 41], visualizing emergent role assignments [42], and most recently using ofﬂine behavioral analysis to learn behavioral spaces over agents [13].
Recent works have also conducted analysis of decision-making in two-player games such as Chess [15], and Hex [38].
In contrast to our work, these approaches focus on post-hoc interpretability, whereas ours makes agent decisions directly transparent, thus enabling causal analysis of their decisions (as illustrated in our experiments).
Background
Partially-Observable Markov Games
A partially-observable Markov game [43] of N agents is deﬁned by the tuple M = ( S , A , T , R , ⌦ , O ) where S , ⌦ , and A deﬁne the game’s global state- space, joint observation-space, and joint action-space, respectively.
In each state, each agent i selects an action a
i 2 A i , yielding a joint action
a = ( a 1 , ..., a N ) for all agents.
Following action selection, the environment transitions to a new state according to the transition function T : S ⇥ A !
S , produces new observations for each agent according to the observation function
O : S ⇥ A !
⌦ , and emits a reward to each agent deﬁned by the reward function R : S ⇥ A !
R N .
The action selection of each agent is dictated by a policy ⇡ i : ⌦ i !
A i and the collection of all individual policies ⇡ = ( ⇡ 1 , ..., ⇡ N ) is referred to as the joint policy.
Concept Bottleneck Models Concept bottleneck models refer to a class of intrinsicallyinterpretable neural network architectures for supervised learning [17].
The network makes predictions by ﬁrst estimating a set of human-understandable concepts, then producing an output based on those concept estimates–i.e. the network is “bottlenecked" by concepts.
Formally, given a dataset { ( x ( j ) , y ( j ) , c ( j ) ) } nj =1 consisting of inputs x 2 R d , outputs y 2 R , and human-understandable concepts c 2 R k (where k is the number of unique concepts), a concept bottleneck model learns two mappings – f : R d !
R k from input-space to concept-space, and g : R k !
R from concept-space to output-space.
The model can then make predictions ˆ y =
f ( g ( x )) as a composition of those mappings.
Figure 1a highlights the increased interpretability achieved by concept bottlenecks in the context of the arthritis classiﬁcation task posed by Koh et al.
Unlike standard networks that learn an uninterpretable mapping from MRI images to arthritis severity labels, the concept bottleneck’s output is conditioned on an intermediate set of human-understandable concepts.
Since these concepts correspond to real-world features—e.g., the presence of bone spursin the arthritis task—they can be used by human observers to understand model failures or misclassiﬁcations.
Moreover, because the model’s predictions are conditioning on these concepts, we can manually change the model’s output (without retraining or ﬁne-tuning) by intervening on its concept estimates.
Concept Bottleneck Policies for MARL
In this section, we introduce Concept Bottleneck Policies as an intrinsically interpretable, conceptbased policy learning method for MARL.
First, we bridge concept-based interpretability and MARL by extending the Markov game formalism to include the components necessary for concept-based learning.
Then, we introduce the concept bottleneck architecture and show how it can be learned within any existing policy learning scheme.
Finally, we demonstrate our method’s potential as a tool for interpreting emergent multi-agent behavior.
In particular, we introduce a behavioral analysis technique using concept intervention and show that, using this technique, we can both detect successful coordination and expose coordination failures (e.g., lazy agents).
4.1 Concept-based Markov Games
To support concept learning, we extend the deﬁnition of Markov games from Section 3 in two important ways.
First, we assume that, in addition to its state space S , the environment maintains an interpretable concept-state space C , where each concept-state c 2 C is a vector of human-understandable concepts that are extracted from the environment and describe key features of the environment, such as the position of agents or objects (we note that this assumption holds in popular RL settings like
Atari [44] and the multi-agent games we study here).
Second, though agents never observe all of c directly (only the concepts related to themselves), we allow them to generate estimates ˆ c of the environment’s underlying concept state.
Thus, at each time-step t , the environment is described by both its state s t and concept-state c t , and each agent i produces both an action a i,t and a set of concept estimates c i,t .
4.2 Concept Bottleneck Architecture
In principle, there are a number of ways in which an agent’s concept estimates ˆ c can be modeled within the RL framework.
Inspired by concept bottleneck models, we are most interested in examining policy architectures in which an agent’s action is conditioned entirely on it’s own concept estimates.
To this end, we factorize an agent i ’s policy ⇡
i ( a i | o i ) into two sub-policies: ⇡ conc i : O i !
C i mapping observations o
i to concept estimates ˆ c i ; and ⇡ act i : C i !
A i mapping concept estimates ˆ c i to actions a i .
The composition of ⇡ conc i and ⇡ act i yields a standard policy mapping observations to actions:
[Equation]
More concretely, given an observation o i,t for some agent i and time-step t , the concept bottleneck ﬁrst produces concept estimates ˆ c i,t ⇠ ⇡ conc i , then uses those estimates alone to select an action a i,t ⇠ ⇡ act i .
Importantly, structuring the policy network such that actions are conditioned on concept estimates creates an intrinsically interpretable policy—the policy is forced to provide a human-understandable rendering of the factors driving its own decision making.
Figure 1b shows the proposed architecture.
Importantly, ˆ c can be thought of as an agent’s beliefs about the underlying game state , which has implications for behavioral analysis.
For example, if two agents i and j collide while moving in the environment, we can examine both ˆ c
i,t and ˆ c j,t to identify which of the two agents incorrectly modeled the location its teammate.
4.3 Concept Bottleneck Learning
Under the concept bottleneck framing, learning a strong but interpretable policy requires both (i) learning how to predict concepts accurately, and (ii) learning how to select actions from those concepts effectively.
To achieve the former, we introduce the following concept loss:
[Equation]
where each component L C j measures the concept prediction error between the j ’th predicted concept and its concept label.
Note that the deﬁnition of each L C j is dictated by the values that both c j and ˆ c j take on—mean-squared error if c j is a scalar, log loss if c j is binary, cross entropy if c j is categorical, etc.
In practice, each pair of concepts c (from the environment’s concept state) and concept estimates ˆ c are stored in an agent’s replay buffer during training alongside standard ( s, a, r, s 0 ) tuples.
To learn a policy from concept estimates, we can attach Equation (1) as an auxiliary loss to the reward-based loss deﬁned by any base
RL algorithm (e.g., PPO [45], TD3 [46], etc).
In general, if
L RL is a generic reward-based loss deﬁned by some RL algorithm, we can construct the following joint concept bottleneck policy (CBP) objective:
[Equation]
where the concept loss coefﬁcient  weights the relative importance of concept prediction.
Later, we examine  ’s impact on training and, in particular, the learned behavior of a multi-agent team.
4.4 Behavioral Analysis via Concept Intervention
In addition to its intrinsic interpretability, a key feature of our method is its support of test-time intervention analysis.
Once a CBP is trained, we can freeze the policy network, roll out a trajectory, and at each time-step replace an individual concept estimate (e.g., the j -th concept estimate ˆ c j ) with a replacement value ¯ c j .
In the simplest case, we can mask out the estimate completely by setting ¯ c j = 0 .
We can then observe the effect, if any,
that ¯ c j has on the agent’s behavior.
Such test-time intervention is a powerful tool for analyzing multi-agent behavior.
In multi-agent settings, it is often difﬁcult to determine from reward alone if agents have learned to coordinate or if they are simply acting independently [37].
Using concept intervention, we propose a direct test for coordination.
For an agent i to coordinate with another agent j , i must condition its policy on some information about j (either j ’s position, orientation, etc).
It follows that if the agents are coordinating and we mask out i ’s concept estimates pertaining to j , we should observe a decrease in team performance.
Conversely, if we do not see performance degrade, then i and j must not be explicitly coordinating (i.e. directly using signals from each other).
By masking out the concepts pertaining to an agent’s teammates, therefore, we can identify whether or not agents are coordinating.
Concept bottlenecks also provide a means for detecting common MARL failures, such as lazy agents.
Here we deﬁne a lazy agent as one that does not contribute to increasing team reward through its own actions.
For an agent i to contribute, it must therefore be conditioning its policy on information about its own interactions with the environment.
We propose to deﬁne a lazy agent i as one that has learned a sub-optimal policy that does not encode such information, leading to unproductive behavior.
We can therefore test for the degree of laziness of an agent by masking out its concept estimates about itself and examining the extent to which team performance degrades as a result.
If team performance remains the same with agent i thus incapacitated, we conclude that i is a lazy agent.
Experiments
In this section, we evaluate the following hypotheses pertaining to Concept Bottleneck Policies:
H1:
Identifying Emergent Coordination Can post-hoc concept intervention identify the level of coordination required by an environment?
To what extent can it identify emergent coordination from policies that act independently?
Moreover, if agents are coordinating, can intervention expose the speciﬁc inter-agent features that underlie that coordination?
H2: Identifying Lazy Agents Can post-hoc concept intervention identify common failure modes such as lazy agents and, in general, measure an agent’s contribution to the larger multi-agent system?
H3:
Bottleneck Performance
Can CBPs match the performance of traditional, non-concept-based neural network policies; thereby achieving intrinsic interpretability without sacriﬁcing performance?
Environment We use Melting Pot Collaborative Cooking as an experimental domain [10].
In
Collaborative Cooking, agents must work together to cook and deliver soups in a kitchen-like environment.
Solving the cooking task requires sophisticated coordination, involving both task partitioning—splitting a recipe into parts—and role assignment—distributing sub-tasks among agents.
For these reasons, Collaborative Cooking is investigated in a number of prior works [47–49] and is emerging as a strong benchmark for multi-agent learning.
We consider four variants of the game for
N =2 agents.
Each environment supports the following agent- and object-oriented concepts: (i) agent position; (ii) agent orientation; (iii) if an agent has a tomato, dish or soup; (iv) cooking pot position; (v) soup cooking progress; (vi) cooking pot tomato count; (vii) tomato position; (viii) dish position.
Further details are provided in Appendix A.
Training and architecture For each of our experiments, we use PPO [45] as a backbone algorithm, which has been shown to be a state-of-the-art multi-agent algorithm [50, 51].
We augment the PPO objective with the concept loss deﬁned in Equation (1) (we refer to this combination as
ConceptPPO moving forward).
Each agent’s policy network consists of CNN and MLP encoders (for image and position/orientation inputs, respectively), followed by a two-layer MLP and a linear mapping that compresses the encoded inputs into concept predictions.
Concept estimates are fed through a two-layer MLP, which produces the ﬁnal action.
ReLU activation is used throughout (except in the bottleneck layer itself).
As a baseline, we use vanilla PPO (no concept loss) with the same architecture.
We train 10 individual policies across each of the following values of  :
5.1 H1: Identifying Emergent Coordination
We evaluate the efﬁcacy of Concept Bottleneck Policies as a mechanism for understanding emergent multi-agent behavior using the post-hoc intervention technique introduced in Section 4.4.
Environmental Coordination Demands For each Concept PPO and PPO policy trained in our cooking environments, we mask out, for each agent, all of the concepts related to that agent’s teammate.
We measure the average cumulative reward attained over 100 trajectories each.
The results of this intervention test are shown in Fig.
The level of coordination required by the environment is apparent from the severity of performance degradation that results from intervening on each agent’s estimates of its teammate.
Most notable is the contrast between the basic environment (blue) and the impassable environment (purple).
The impassable environment requires strict coordination—agents can only access a subset of ingredients and must pass items to each other across the center divider.
In this case, intervention performance drops to near-zero across all policies.
In the basic environment, on the other hand, there are no obstacles and agents have access to their own supply of ingredients; and so both policies in which agents coordinate and policies in which agents act independently are successful.
Consequently, the impact of intervention is much less severe, as the performance of policies that coordinate (and fail under intervention) is averaged in with independent policies that are unaffected by intervention.
Altogether, these results indicate that our method accurately distinguishes environments that require coordination from those that do not.
Identifying Coordination vs. Independent Behaviors Our next intervention test aims to disentangle, within a single environment, policies that learn to coordinate from those that opt for independent action.
We hone in on the basic environment—because it most clearly supports independent policies as a successful strategy—and plot the performance of each policy individually (rather than averaging across all policies).
As shown in Fig.
3, we again ﬁnd a stark contrast between coordination and non-coordination, but this time at the level of individual policies.
In the trajectory snapshot in Fig.
3a, we see an emergent strategy in which two agents coordinate through role assignment.
In particular, the orange agent maneuvers in the bottom-right corner of the environment picking up tomatoes and bringing them to the cooking pot; while the blue agent stays in the top-right corner running dishes to and from the cooking pot to deliver soup.
As expected, intervention over this strategy leads to a catastrophic drop in performance.
Each agent’s coordination hinges on an accurate modeling of its teammate.
In the trajectory snapshot in Fig.
3b, we ﬁnd a much
[Figure]
different strategy in which agents complete the cooking task independently on opposite sides of the environment.
Here, intervention does not hurt performance (or hurts performance only marginally), as neither agent needs to closely monitor its teammate to complete the task.
Finally, because intervention does not impact performance when agents are not coordinating, this result provides further evidence that coordination detection through intervention is accurate and reliable.
Factors of Coordination Next, we examine the policies that have learned to coordinate and attempt to pinpoint the speciﬁc inter-agent features that drive their coordination.
To this end, rather than intervening over all concepts pertaining to an agent’s teammate at once, we iterate over each of them individually.
As we hypothesize, if we know that the agents are coordinating, then intervening over concepts related to the other agent should result in a sharp drop in performance.
The results of this intervention test are shown in Fig.
3c.
Interestingly, performance only drops when intervening over the orientation concept, indicating that agents are primarily using the orientation of the other agent as a signal to drive coordination.
This is curious, as we would expect coordination to involve multiple individual sources of inter-agent information; or combinations thereof.
We emphasize that identifying the speciﬁc factors that underlie an agent’s coordination strategy would be much more difﬁcult without the concept intervention enabled by our concept bottleneck policies.
For completeness, we conduct two additional experiments to support this analysis.
First, we rule out the possibility that intervening with a ﬁxed mask to zero-out orientation creates an OOD (or otherwise adversarial) input that the agents’ policies cannot handle.
We do this by manufacturing intervention masks that are both in- and out-of-distribution, using an empirical sample of the orientations experienced by each agent at test-time; and show that our results are consistent across both cases.
Second, we train a new set of ConceptPPO policies without orientation as a concept and re-run this iterative intervention over concepts pertaining to each agent’s teammate.
These results show that, without orientation, agent coordination latches on to another single concept estimate as its driving signal.
Detailed discussion of these supporting results is provided in Appendix B.
5.2 H2: Identifying Lazy Agents
In addition to coordination successes, the concept intervention technique enabled by our method allows us to test for coordination failures .
Here we test for the presence of lazy agents by masking out each agent’s concept predictions about itself—including the agents own position, orientation, etc.
According to our hypothesis, if an agent is acting productively in the environment, removing this information will greatly hinder that agent’s performance; and team performance as a whole.
If performance does not decrease, the agent likely isn’t contributing to the task.
Figure 4 shows the results of this lazy agent test for four policies, each differing in the strength of their contribution to the team.
As predicted by our hypothesis, the productivity of an agent can be exposed through the impact of the intervention test.
When agents are both contributing to the task
[Figure]
Crucially, the magnitude of performance degradation is a direct function of the productivity of each agent.
Thus, these results not only demonstrate that our method can reliably diagnose lazy agents, but also suggest that it can quantify the degree of laziness as a function of performance degradation under intervention.
We stress that diagnosing lazy agents and the degree to which an agent contributes to a multi-agent strategy would be more difﬁcult without our concept bottleneck architecture.
5.3 H3: Bottleneck Performance
To evaluate the general performance of our method, we compare the performance of Concept PPO to the non-concept PPO baseline.
We measure performance as the average cumulative reward obtained over 100 test-time trajectories (and ﬁve random seeds each).
Figure 5 provides an overview of these results.
We ﬁnd evidence that ConceptPPO can match the performance of PPO across each of our environments for small values of the concept cost coefﬁcient (  < = 0 . 5 ).
This is an important result from the perspective of interpretability.
It demonstrates that, if  is tuned appropriately, it is possible to train intrinsically-interpretable policy networks—where, notably, decisions are expressed in human-understandable concepts—without sacriﬁcing in task performance.
Figure 5 also shows, though, that over-valuing the concept prediction loss causes performance to degrade.
In particular, performance falls for  > 0 .
75 and, in all but the basic environment, collapses to zero for larger values (   5 . 0) .
In these cases, agent policies begin to overvalue concept prediction accuracy relative to reward, which changes the nature of their emergent coordination strategy.
As  increases, agents are more incentivized to place themselves in areas of the state-space where they
[Figure]
can accurately predict concepts, including those related to the other agent.
We hypothesize that this causes agents to develop behaviors that are less efﬁcient from a reward perspective, but more effective from a predictability perspective, such as clustering themselves together spatially.
State Visitation Analysis To illustrate the behavioral changes induced by  , we measure the distribution of states visited by each agent across a subset of the  values used during training.
The results for each agent are plotted as a heatmap in Fig.
As expected, we ﬁnd that agents visit smaller and smaller regions of the state space as  increases until, ﬁnally, behavior collapses.
In the most extreme cases (e.g.,  = 10 . [-
-]0 in the impassable environment) agents opt not to move at all—after all, the easiest way to predict concepts accurately is to not change anything in the environment.
Conclusion
We proposed Concept Bottleneck Policies as an intrinsically interpretable, concept-based policy learning method for MARL.
We demonstrated that our method is effective for understanding emergent multi-agent behavior.
In particular, Concept Bottleneck Policies support test-time concept intervention, which can be used to identify when a multi-agent team has learned to coordinate, what inter-agent features drive that coordination, and to what extent coordination is required in a particular environment.
Moreover, concept intervention can help diagnose coordination failures like lazy agents.
Experimental results also show that our method achieves comparable levels of performance to traditional non-concept-based policy learning methods.
Acknowledgments and Disclosure of Funding
We would like to thank Yannick Assogba, Lucas Dixon, Asma Ghandeharioun, Adam Pearce, Ryan
Mullins, Joel Z. Leibo, Edgar Duéñez Guzmán, Peter Hase, Srivatsan Krishnan and Blair Bilodeau for their helpful feedback on this work