Paper: Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation (ICLR-2024)

Reviewer comment (Experiments):
The approach seems somewhat limited in scope, requiring a considerable amount of work from a designer and a description of the world model. While the authors claim the proposed approach can "generate natural language explanations for an agent's behavior based only on observations of states and actions," in practice there are significant additional requirements. The agent's policy must be summarized into a decision tree, which can require a lengthy training phase and access to the environment. The LLM is also provided with a comprehensive description of the world model, which appears necessary to generate reasonable explanations. This raises questions about scenarios in which the pipeline could be applied, since it seems to require access to low-level environment knowledge (in the form of a model) such that other explanation forms might be more appropriate.

Author response:
**1) The approach seems somewhat limited in scope, requiring a considerable amount of work from a designer and a description of the world model**

We argue that our approach actually requires considerably less effort than previous methods, which typically involve building a dataset of annotated natural language explanations for agent state-action pairs and fine-tuning a language model. Policy distillation is a fairly straightforward process (we use DAGGER in this work, see our response to #2 below) that only has to occur once. Similarly, a domain expert has to define a natural language description of the task MDP and a few in-context learning examples, but these are also only one-time efforts and are trivial in comparison to constructing an entire dataset for fine-tuning (as most prior methods require). We will clarify the writing to more clearly indicate that the description of the MDP is a requirement.

Reviewer comment (Experiments):
The system requires behaviors that can be summarized into a decision tree, which is a fairly stringent constraint. The ability of the system to be summarized in terms of a behavior tree might be more challenging than generating the explanations themselves. There is no way to know in advance if the behavior tree is a good proxy for the agent's behavior, and there are no experiments provided to judge how accurate these behavior trees are.

Author response:
**2) Being able to summarize behaviors into a decision tree**

We use dataset aggregation (DAGGER)[1] to sample state-action pairs and train the decision tree (using CART). DAGGER proceeds by collecting a set of visited states at each iteration under the current distilled policy and the corresponding actions from the agent’s original policy, which is aggregated to the set of all previous state-action pairs and used to train the decision tree.

The state of our environment is an NxMx5 tensor where N is the number of rows (5 in our case) and M is the number of columns (4). The last dimension represents a binary feature vector for each room and consists of: the presence of the medic, the presence of the engineer, the presence of rubble, the presence of a victim (may be hidden if there is rubble), and whether the room has been explored.

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.

We quantified the distilled policy accuracy by rolling out 5000 episodes with random initializations, and computing the action accuracy using the original policy’s action as the target. The accuracy values are given below where we can observe a range from ~83% to 100%. We will add these results to the appendix of our paper.

|             | explore | exploit | fixed   |
|-------------|---------|---------|---------|
| medic       | 87.72%  | 87.05%  | 100.00% |
| engineer    | 82.94%  | 96.42%  | 100.00% |

We would like to point out a few things with regard to the distilled policy.
1. Our environment was intentionally designed so as to exhibit ambiguous states, where different behaviors will produce the same action at time t, but different actions at time t+1. This ambiguity is the cause for some of the loss in performance (especially the explore policies).
2. We only produce explanations when the action of the distilled DT matches that of the original policy. So in the worst case here, we still explain ~83% of the agent’s actions, which we think is reasonable.

Reviewer comment (Theory):
Multiple places in the paper refer to "optimal behavior" without a clearly defined reward function. While it is presumably to reveal and assist all victims as quickly as possible, and while participants likely view exploratory behavior as suboptimal for this reason, clearer specification and discussion of this optimality criterion would be helpful.

Author response:
**3) On the subject of ‘optimality’**

We don’t assume optimal behavior, and will change the subtitle of Figure 5. The major confusion is likely raised by Figure 3, where the example includes the LLM response “The medic did not consider this information when making its decision, which would imply a non-optimal decision-making process on the part of the medic.” Note that this text is entirely generated by the LLM. Interestingly, GPT-4 assumes the exploitative policy as optimal and the explorative policy as suboptimal. Relevant discussions on this have been covered in section 4.1.

Reviewer comment (Evaluation):
The user study design appears potentially flawed. Instead of testing whether non-expert human end users understand why an agent behaved in a certain way, it asks participants to choose which explanation they prefer between two examples. This could measure preference for prose quality rather than correctness or informativeness, as ChatGPT is known to produce convincing but potentially inaccurate statements.

Author response:
**4) Regarding concerns about the user studies**

The fact that humans didn’t identify explanations unfaithful to the agent doesn’t necessarily lead to the conclusion that our explanations didn’t convey sufficient information. Instead, it points out that users may overly pay attention to prose, so it’s particularly important to ensure the content provided to the users is faithful. To demonstrate this, we describe our user study in more depth. For each of the 20 cases, the user is given a figure of the current state, a figure of the future state (in which the agent accomplishes its next goal), and two explanations of random order. Then they are asked the question “which explanation best helps you understand the agent's behavior”. We provide the future state such that participants can better evaluate explanation helpfulness with respect to the agent’s actual behavior.

In the study introduction, we explicitly informed participants that explanations may not be correct, so they were aware of this. However, the fact that participant preference was not affected by hallucination indicates either that, a) participants were unable to identify hallucinations even given the current and future states, or b) participants did identify it but considered it inconsequential. Our study design does not allow us to disentangle these effects, but it does not emphasize the necessity of using explanations which minimizes inaccuracies and hallucinations, which we show that our approach excels at.

Due to our study design, we could not ask participants to predict the agent’s next action (because we provided them with the future state). In this case, we opted for giving more information to participants such that they can better judge explanation helpfulness by considering the agent’s future behavior, rather than withholding it and analyzing explanation comprehension by prompting participants to predict actions themselves.

Reviewer comment (Evaluation):
The conclusion drawn from user studies that hallucinations do not impact user preference may be misleading. This outcome could indicate that users are not truly understanding the agent's behavior, instead basing their preference on prose quality rather than substance.

Author response:
**4) Regarding concerns about the user studies**

The fact that humans didn’t identify explanations unfaithful to the agent doesn’t necessarily lead to the conclusion that our explanations didn’t convey sufficient information. Instead, it points out that users may overly pay attention to prose, so it’s particularly important to ensure the content provided to the users is faithful. To demonstrate this, we describe our user study in more depth. For each of the 20 cases, the user is given a figure of the current state, a figure of the future state (in which the agent accomplishes its next goal), and two explanations of random order. Then they are asked the question “which explanation best helps you understand the agent's behavior”. We provide the future state such that participants can better evaluate explanation helpfulness with respect to the agent’s actual behavior.

In the study introduction, we explicitly informed participants that explanations may not be correct, so they were aware of this. However, the fact that participant preference was not affected by hallucination indicates either that, a) participants were unable to identify hallucinations even given the current and future states, or b) participants did identify it but considered it inconsequential. Our study design does not allow us to disentangle these effects, but it does not emphasize the necessity of using explanations which minimizes inaccuracies and hallucinations, which we show that our approach excels at.

Due to our study design, we could not ask participants to predict the agent’s next action (because we provided them with the future state). In this case, we opted for giving more information to participants such that they can better judge explanation helpfulness by considering the agent’s future behavior, rather than withholding it and analyzing explanation comprehension by prompting participants to predict actions themselves.

Reviewer comment (Experiments):
The user studies might be more effective if they evaluated participants' comprehension of the agent's behavior. For example, users could be asked to predict the agent's next action after reading an explanation, and their performance compared to users who are given the high-level policy description (e.g., "explore" vs "exploit" policies).

Author response:
**4) Regarding concerns about the user studies**

The fact that humans didn’t identify explanations unfaithful to the agent doesn’t necessarily lead to the conclusion that our explanations didn’t convey sufficient information. Instead, it points out that users may overly pay attention to prose, so it’s particularly important to ensure the content provided to the users is faithful. To demonstrate this, we describe our user study in more depth. For each of the 20 cases, the user is given a figure of the current state, a figure of the future state (in which the agent accomplishes its next goal), and two explanations of random order. Then they are asked the question “which explanation best helps you understand the agent's behavior”. We provide the future state such that participants can better evaluate explanation helpfulness with respect to the agent’s actual behavior.

In the study introduction, we explicitly informed participants that explanations may not be correct, so they were aware of this. However, the fact that participant preference was not affected by hallucination indicates either that, a) participants were unable to identify hallucinations even given the current and future states, or b) participants did identify it but considered it inconsequential. Our study design does not allow us to disentangle these effects, but it does not emphasize the necessity of using explanations which minimizes inaccuracies and hallucinations, which we show that our approach excels at.

Due to our study design, we could not ask participants to predict the agent’s next action (because we provided them with the future state). In this case, we opted for giving more information to participants such that they can better judge explanation helpfulness by considering the agent’s future behavior, rather than withholding it and analyzing explanation comprehension by prompting participants to predict actions themselves.

Reviewer comment (Evaluation):
There is concern about the high rate of hallucinations (~30%) for the proposed BR (Path) approach.

Author response:
Regarding the high hallucination rate, the maximum rate for BR (Path) is 20% for the explorative short-term policy, with the average rate falling at 10.7% across all policies. This is significantly lower than 30%, and we note that a considerable amount of hallucinations are from reasoning instead of from facts. Examples of them are “If the engineer is in room (0, 0) and moves north, they will be in room (0, 0)” while the engineer cannot move north in the first place, “there is no rubble blocking the way” while rubble is not blocking the way in our settings. Wrong presentation of facts is extremely rare in our proposed method.

Reviewer comment (Reproducibility):
The paper does not appear to include an example of the "Template" response, which would be important for understanding the user studies. An example should be provided.

Author response:
**5) Minor comments**

As stated in the paper, Template refers to a textual representation of the decision path. Examples have been provided both in Figure 2 and in our appendix. Thanks for pointing out these issues, we’ll update the manuscript accordingly.

Reviewer comment (Theory):
The proposed method's claim of being model-agnostic is questionable because it seems to require that the representation of the environment itself is interpretable. Without interpretable state representations, there is no obvious way for the decision tree policy and corresponding decision path to be translated into an interpretable natural language template for the LLM to consume. This requirement is met in the gridworld environment used in the paper, but not in many other models and agent policies, which may use pixel-level or low-level state inputs. Therefore, the promised benefit that only simulation access to a policy is needed is not fully delivered. The method can only be applied if the underlying state representation is sufficiently interpretable, and this assumption should be explicitly stated in the paper's framing. Under such framing, comparisons should be made against XAI approaches that also leverage interpretable state representations, such as plan recognition and goal recognition algorithms.

Author response:
**1) How model-agnostic is this approach given that it seems to rely on interpretable state representations?**

You are correct that the distillation process used here requires inputs which can be directly consumed by a decision tree (and also interpreted by an LLM, but the DT is the bigger constraint in our experience). However, we also feel that our approach is applicable to environments which contain different state spaces, e.g. images or uninterpretable latent representations. We argue that such representations can be handled by first extracting a set of semantic features, e.g. visual concepts [1], which are then used to distill a structured behavior representation. Ultimately this is an issue that applies to all interpretable/explainable methods – how do we construct human-intelligible representations from inherently uninterpretable ones – and so we don’t feel that this is unique to our method, particularly as we are not directly addressing this problem.

[1] Koh, Pang Wei, et al. "Concept bottleneck models." ICML 2020.

Reviewer comment (Experiments):
The ability of the method to generalize to other environments and policies is uncertain. The current experiments appear highly tailored to generating explanations for purely explorative or purely exploitative policies in a search and rescue setting. While the natural language explanations presented seem intuitively plausible, a much broader variety of possible policies exists, and it is unclear whether humans or LLMs could generate compelling explanations for those. The example decision paths shown in the paper could be consistent with many different goals, strategies, and intentions, raising concerns that the LLMs' ability to generate the "right" explanations may be an artifact of the few-shot examples and test set design. It is unclear whether the approach would work if the policies were, for example, exploratory in one part of the state space and exploitative in another, or if they mixed exploration and exploitation through strategies like Thompson sampling.

Author response:
**2) Can this method generalize to a wider range of policies, or to a richer range of environments?**

These are reasonable points, which we think can be reduced to: how descriptive can a decision tree/path be of an agent’s strategy, intent, and goal? We agree that it may be possible for a given decision path to be interpreted multiple ways, and that the generated explanation is influenced by the in-context learning examples which are given in the prompt. However, we feel that is also a strength of this approach: if an agent is capable of producing similar-looking decision paths that may have different underlying intents, then this can be represented through in-context learning examples which themselves express contradictory explanations or uncertainty. We noted this behavior ourselves in the paper, e.g. when providing the model with no behavior representation (no DT path), the LLM often exhibited uncertainty and refrained from generating specific explanations for agent behavior (even when using the “standard” set of in-context examples in this case). There is even a potential for prompting the LLM for a set of explanations and then measuring agreement above them to quantify certainty.

Despite this, we emphasize that our approach has several benefits over previous methods:
1. We leverage the encoded knowledge within the pre-trained LLM to avoid having to curate annotated datasets of states and explanations, which is impractical when attempting to explain arbitrary agent behavior.
2. The LLM itself is capable of performing reasoning over the given inputs (guided by the in-context examples), which often leads to more useful explanations as a human observer no longer has to perform this reasoning themselves (e.g. from textual translations of decision paths).

Lastly, we want to note that decision trees are rather expressive in terms of behavior. For example, with your example of an agent that is exploratory in the top-half of the grid and exploitative in the bottom-half, we would expect a decision tree to represent this with different decision path structures for rooms in the top half vs bottom half.

Reviewer comment (Evaluation):
There are concerns about what constitutes a "good" explanation of an agent's behavior and how this is evaluated. In explaining human behavior, people often use mental states like goals, beliefs, and intentions as part of theory-of-mind reasoning. The authors seem to adopt a similar view by annotating LLM-generated explanations for accuracy in terms of goals, intents, and strategies. However, this may not be appropriate for artificial agents, especially since black-box policies can behave very differently from humans and may lack such mental states entirely. For example, a decision-tree policy is purely reactive and not goal-directed, making it a kind of metaphysical error to attribute goals to it unless it imitates a goal-directed policy.

Author response:
**3) How were the examples hand-annotated and how did we avoid over-fitting to to our choice of policies?**

We first note that the agent policies themselves were exhibiting goal-directed behavior in our experiments by design (we explicitly crafted policies to exhibit exploitation and exploration type behaviors). Any reasonably accurate distilled policy will then mimic such goal-directed behavior as well. This actually ties into our broader point: we don’t necessarily agree that goal-directed metrics are inappropriate for explaining agent behavior in general. In some sense, any explanation which attempts to reason about high-level behavior is at risk of “anthropomorphic bias” unless a policy explicitly includes hierarchical behavior planning or computational theory of mind models. And yet, we can clearly observe goal-directed behavior in standard deep neural network policies that do not include such mechanisms [2] – to such a degree that it is useful for humans to reason about agent behavior in terms of them. So while there may be cases in which inferred goals do not explicitly represent the underlying policy, we feel that the idea and metrics in general still have merit, particularly from a comprehensibility standpoint for humans.

[2] Vinyals, Oriol, et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learning." Nature 2019.

We include additional details regarding the hand-annotation below.

We designed *policy-agnostic quantitative metrics* to make sure our evaluation approach can be applied to other tasks and domains. These are Strategy, Category, Goal, Action, and Intent accuracy. These metrics are intended to work with any environment and agent that exhibits high-level strategies and low-level goals so as to be useful for future work. The downside is that such metrics must be manually annotated from natural language explanations, requiring considerable human effort. However, we find these provide far more detailed insights than automated metrics which may be erroneous (e.g. asking an LLM to evaluate itself) or only operate at the dataset-level (e.g. BLEU, BERTscore). We define our metrics in the manuscript in Sec. 4 but restate them here with additional details.

Suppose that the engineer is taking an exploitative strategy, and the decision path largely consists of decision rules operating over rubble locations (similar to the same path in Fig. 2). 

• Strategy: Whether the overall strategy is identified. In this case, whether the explanation contains “The agent seems to prioritize the removal of rubble” for an exploitative engineer, as opposed to “The agent seems to be prioritizing exploration” for an explorative engineer.

• Category: Whether the agent’s goal category was identified in the explanation. For example, if the explanation is “The engineer is prioritizing the removal of rubble and moves south to room (0,1) as that brings it closer to room (0,4) which contains rubble”, then the explanation correctly identifies that the agent is seeking out rubble (which happens to be in room (0,4) in this case, although the specific location is not part of the category).

• Goal: Whether the agent’s specific goal was correctly identified in the explanation. In the explanation above, if the agent’s actual goal was to remove the rubble in room (0,4) (which we can observe by looking at future actions), then the goal is correct. However, if the agent instead moved south to room (0,1) in order to then move east and remove rubble in room (1,1), the goal would be incorrect even though the category would still be correct.

• Action: Whether the agent’s next action was successfully predicted.

• Intent: Whether the agent’s intent for taking the next action was successfully identified. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (0, 4) which contains rubble” for an exploitative policy, Intent is correct. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (1, 4) which is unexplored” for an exploitative policy (which should take rubble in room (0, 4) as priority), Intent is incorrect.

Reviewer comment (Evaluation):
To avoid anthropomorphic bias in explanation generation, a different standard for a "good" explanation could be that it supports accurate prediction as well as accurate intervention and counterfactual reasoning. While attributing mental states to humans is useful for planning interventions, for artificial agents explanatory quality could also be measured by whether the explanations support useful interventions. This could involve evaluating intervention utility in addition to future action prediction (which the authors already do). Including such an evaluation could improve the paper. Without it, there is a risk that the evaluation methodology is overfit to the specific policies considered, which allow for relatively simple anthropomorphic explanations based on exploratory or exploitative goals.

Author response:
**4) Concerns over user study evaluation and interventions/counterfactuals**

These are good points, and we would like to provide additional details for our user study here. When evaluating explanation helpfulness (Sec. 5.1) we showed two state images to the participant: the current state in which the agent takes the current action that we wish to explain, and the future state that occurs after the agent achieves its next goal (see discussion on goal-directed behavior above). This enables each participant to judge the explanations not only on their comprehensibility, but also on whether it is specifically helpful in understanding the agent’s actual behavior.

As you pointed out, we did indeed evaluate action prediction accuracy. However, we argue that we also evaluated intervention/counterfactual accuracy via our second user study in Sec. 5.2. In this study, participants were presented with an agent state/action/explanation and offered the chance to interact with the LLM to issue clarification, counterfactual, or other queries. As we observed, participants found this interaction helpful when the agent’s actions did not align with the participant’s expectations. We will add examples of these interactions to the appendix and we list an example below.

*LLM Response 1 (Explanation)*

`The engineer moves east to room (1, 0) because this room contains rubble. Based on the features, the engineer appears to prioritize removing rubble from rooms. Since the engineer is in room (0, 0), moving east to room (1, 0) aligns with this goal, demonstrating the engineer's focus on clearing rubble as part of the search and rescue operation. The other mentioned rooms do not contain rubble, making room (1, 0) the logical choice for the engineer's next action.`

*Counterfactual Query*

`What would the engineer do if room (0,0) also contains rubble? Answer briefly.`

*LLM Response 2*

`If room (0,0) also contains rubble, the engineer would likely stay in room (0,0) and take action to remove the rubble there, given their priority of clearing rubble in the search and rescue operation.`

Reviewer comment (Evaluation):
For the experiment setup section, can you explain more on the metrics, for example, what's the difference between Category and Goal?

Author response:
**1) Additional information on metrics**

We designed *policy-agnostic quantitative metrics* to make sure our evaluation approach can be applied to other tasks and domains. These are Strategy, Category, Goal, Action, and Intent accuracy. These metrics are intended to work with any environment and agent that exhibits high-level strategies and low-level goals so as to be useful for future work. The downside is that such metrics must be manually annotated from natural language explanations, requiring considerable human effort. However, we find these provide far more detailed insights than automated metrics which may be erroneous (e.g. asking an LLM to evaluate itself) or only operate at the dataset-level (e.g. BLEU, BERTscore). We define our metrics in the manuscript in Sec. 4 but restate them here with additional details.

Suppose that the engineer is taking an exploitative strategy, and the decision path largely consists of decision rules operating over rubble locations (similar to the same path in Fig. 2). 

• Strategy: Whether the overall strategy is identified. In this case, whether the explanation contains “The agent seems to prioritize the removal of rubble” for an exploitative engineer, as opposed to “The agent seems to be prioritizing exploration” for an explorative engineer.

• Category: Whether the agent’s goal category was identified in the explanation. For example, if the explanation is “The engineer is prioritizing the removal of rubble and moves south to room (0,1) as that brings it closer to room (0,4) which contains rubble”, then the explanation correctly identifies that the agent is seeking out rubble (which happens to be in room (0,4) in this case, although the specific location is not part of the category).

• Goal: Whether the agent’s specific goal was correctly identified in the explanation. In the explanation above, if the agent’s actual goal was to remove the rubble in room (0,4) (which we can observe by looking at future actions), then the goal is correct. However, if the agent instead moved south to room (0,1) in order to then move east and remove rubble in room (1,1), the goal would be incorrect even though the category would still be correct.

• Action: Whether the agent’s next action was successfully predicted.

• Intent: Whether the agent’s intent for taking the next action was successfully identified. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (0, 4) which contains rubble.” for an exploitative policy, Intent is correct. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (1, 4) which is unexplored.” for an exploitative policy (which should take rubble in room (0, 4) as priority), Intent is incorrect.

Reviewer comment (Reproducibility):
Can you provide more details on the decision tree based distillation, such as what features used?

Author response:
**2) More details on tree-based distillation**

We use dataset aggregation (DAGGER)[1] to sample state-action pairs and train the decision tree (using CART). DAGGER proceeds by collecting a set of visited states at each iteration under the current distilled policy and the corresponding actions from the agent’s original policy, which is aggregated to the set of all previous state-action pairs and used to train the decision tree.

The state of our environment is an NxMx5 tensor where N is the number of rows (5 in our case) and M is the number of columns (4). The last dimension represents a binary feature vector for each room and consists of: the presence of the medic, the presence of the engineer, the presence of rubble, the presence of a victim (may be hidden if there is rubble), and whether the room has been explored.

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.

Reviewer comment (Experiments):
As proposed in the conclusion, this method is limited by using only non-dense inputs, however, for real world application, many of them will include dense inputs like continuous features, this makes the proposed method limited.

Author response:
**4) Generalization to environments with continuous/dense inputs**

Our approach readily supports continuous observations and action spaces as-is; decision trees can split over continuous features and also be used for regression tasks (continuous actions), and most tree-construction algorithms natively support this, e.g. CART, ID3. High-dimensional inputs such as images can also be supported by first extracting a set of semantic features from the images, e.g. visual concepts [1].

As for complex environments which induce complex behaviors, this gets into the issue of how complex of a policy a decision tree can encode. While we do not directly investigate this question in our work, we can conjecture here about what we expect. Previous works [2] have shown that DTs can achieve surprisingly good performance purely through distillation, however, it is likely that this approach may not work for all complex policies. When that is the case, we expect that our method would also work with other model representations for the behavior. For example, instead of a decision tree maybe we utilize a gradient boosted tree or a differentiable decision tree (that is “crispified” [3]). While we have not investigated these specific approaches yet, LLMs have shown at least some ability to interpret even complex neural network representations [4], so we expect it would perform at least as well on decision tree variants.

[1] Koh, Pang Wei, et al. "Concept bottleneck models." ICML 2020.
[2] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." NeurIPS 2018.
[3] Paleja, Rohan, et al. "Learning Interpretable, High-Performing Policies for Autonomous Driving." RSS 2022.
[4] Bills, Steven, et al. "Language models can explain neurons in language models." 2023.

Reviewer comment (Experiments):
There are few details given of the agent, the decision tree, environment and the language model used. Authors have used one type of agent, in one environment and used one language model, which raises further questions of the methods generalizability and scalability to complex agents and larger environments.

Author response:
**1) Few details are given of the agent, the decision tree, environment, and the language model**

We have provided details regarding our experiments in Appendix A, but we restate them here with some additional detail for clarity:

**Environment**: The environment is a partially observable multi-agent urban search and rescue task, consisting of a grid of inter-connected rooms that may contain rubble and/or victims. There are two agents with heterogeneous skills: a medic which can save victims and an engineer which can remove rubble. The goal of the task is to save the victims, however, they may be trapped by rubble if both a victim and rubble are present in the same room. In this case, the agents must work together: the engineer must remove the rubble so the medic can save the victim.

**Agent**: We leverage two types of agent behaviors, an exploration type and an exploitation type. The exploration type will first explore all rooms to identify exposed victims and rubble (which may possibly hide trapped victims). The exploitation type acts as a greedy agent, which begins by exploring but acts to either save victims or remove rubble (depending on the agent type) as soon as one is found. All of our results produce explanations for both the medic and engineer agents.

**Language model**: We use GPT4 via OpenAI’s developer API. We include a concise description of the environment (similar to above), a description of the behavior representation (if there is one), and three in-context learning examples.

**Decision tree**: We use dataset aggregation (DAGGER)[1] to sample state-action pairs and train the decision tree (using CART). DAGGER proceeds by collecting a set of visited states at each iteration under the current distilled policy and the corresponding actions from the agent’s original policy, which is aggregated to the set of all previous state-action pairs and used to train the decision tree.

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.

Reviewer comment (Evaluation):
The human-subject experiments are only used to evaluate the preferences of the participants. Explanations can be highly preferred but be misleading (i.e. unfaithful to the underlying agent).

Author response:
**3) Preference is heavily involved in the user study regarding the evaluation of behavior explanations**

The quality of the generated explanations are evaluated from a holistic perspective, with the quantitative metrics evaluating faithfulness and the human study evaluating preference. We believe that a good explanation needs to be both faithful and comprehensible from a human's perspective. It is true that users didn't prefer the non-hallucinated explanations over hallucinated ones (Sec. 5.1), but whether this is because they considered the hallucination inconsequential or because they didn’t detect it in the first place we cannot say. However, this supports our argument that we need to generate explanations which minimize hallucinations in the first place, and our quantitative results are intended to show that this is the case. If we use explanations with hallucinations and users blindly trust them, this can mislead the user and potentially lead to dangerous situations.

Reviewer comment (Theory):
Can this framework handle domains with continuous actions? If this is the case, what are the changes that the framework needs?

Author response:
**2) Concerns about generalization to other environments, language models, and continuous action spaces**

Our method was evaluated on a single environment due to the time-consuming nature of hand-annotating responses for quantitative evaluation and running user studies. However, we argue that the depth of our analysis on this environment offers intriguing insights, and this doesn’t limit the generalization of our approach to other environments.

**Environment and policy-agnostic metrics**: We designed *policy-agnostic quantitative metrics* to make sure our evaluation approach can be applied to other tasks and domains. These are Strategy, Category, Goal, Action, and Intent accuracy. These metrics are intended to work with any environment and agent that exhibits high-level strategies and low-level goals so as to be useful for future work. The downside is that such metrics must be manually annotated from natural language explanations, requiring considerable human effort. However, we find these provide far more detailed insights than automated metrics which may be erroneous (e.g. asking an LLM to evaluate itself) or only operate at the dataset-level (e.g. BLEU, BERTscore). We define our metrics in the manuscript in Sec. 4 but restate them here with additional details.

Suppose that the engineer is taking an exploitative strategy, and the decision path largely consists of decision rules operating over rubble locations (similar to the path in Fig. 2). 

• Strategy: Whether the overall strategy is identified. In this case, whether the explanation contains “The agent seems to prioritize the removal of rubble” for an exploitative engineer, as opposed to “The agent seems to be prioritizing exploration” for an explorative engineer.

• Category: Whether the agent’s goal category was identified in the explanation. For example, if the explanation is “The engineer is prioritizing the removal of rubble and moves south to room (0,1) as that brings it closer to room (0,4) which contains rubble”, then the explanation correctly identifies that the agent is seeking out rubble (which happens to be in room (0,4) in this case, although the specific location is not part of the category).

• Goal: Whether the agent’s specific goal was correctly identified in the explanation. In the explanation above, if the agent’s actual goal was to remove the rubble in room (0,4) (which we can observe by looking at future actions), then the goal is correct. However, if the agent instead moved south to room (0,1) in order to then move east and remove rubble in room (1,1), the goal would be incorrect even though the category would still be correct.

• Action: Whether the agent’s next action was successfully predicted.

• Intent: Whether the agent’s intent for taking the next action was successfully identified. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (0, 4) which contains rubble.” for an exploitative policy, Intent is correct. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (1, 4) which is unexplored.” for an exploitative policy (which should take rubble in room (0, 4) as priority), Intent is incorrect.

**Environments with high-dimensional inputs, continuous action spaces, and higher complexity**: Our approach readily supports continuous action spaces as-is; decision trees can be used for regression tasks (continuous actions) and most tree-construction algorithms natively support this, e.g. CART, ID3. High-dimensional inputs such as images can also be supported by first extracting a set of semantic features from the images, e.g. visual concepts [2].

As for complex environments which induce complex behaviors, this gets into the issue of how complex of a policy a decision tree can encode. While we do not directly investigate this question in our work, we can conjecture here about what we expect. Previous works [3] have shown that DTs can achieve surprisingly good performance purely through distillation, however, it is likely that this approach may not work for all complex policies. When that is the case, we expect that our method would also work with other model representations for the behavior. For example, instead of a decision tree maybe we utilize a gradient boosted tree or a differentiable decision tree (that is “crispified” [4]). While we have not investigated these specific approaches yet, LLMs have shown at least some ability to interpret even complex neural network representations [5], so we expect it would perform at least as well on decision tree variants.

[2] Koh, Pang Wei, et al. "Concept bottleneck models." ICML 2020.

[3] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." NeurIPS 2018.

[4] Paleja, Rohan, et al. "Learning Interpretable, High-Performing Policies for Autonomous Driving." RSS 2022.

[5] Bills, Steven, et al. "Language models can explain neurons in language models." 2023.