Original: T RAJECTORY - BASED E XPLAINABILITY F RAMEWORK FOR O FFLINE RL
Revised: Trajectory-based Explainability Framework for Offline RL

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: Abstract

Original: Explanation is a key component for the adoption of reinforcement learning (RL) in many real-world decision-making problems.
Revised: Explanation is a key component for the adoption of reinforcement learning (RL) in many real-world decision-making problems.

Original: In the literature, the explanation is often provided by saliency attribution to the features of the RL agent’s state.
Revised: In the literature, the explanation is often provided by saliency attribution to the features of the RL agent’s state.

Original: In this work, we propose a complementary approach to these explanations, particularly for offline RL, where we attribute the policy decisions of a trained RL agent to the trajectories encountered by it during training.
Revised: In this work, we propose a complementary approach to these explanations, particularly for offline RL, where we attribute the policy decisions of a trained RL agent to the trajectories encountered by it during training.

Original: To do so, we encode trajectories in offline training data individually as well as collectively (encoding a set of trajectories).
Revised: To do so, we encode trajectories in offline training data individually as well as collectively (encoding a set of trajectories).

Original: We then attribute policy decisions to a set of trajectories in this encoded space by estimating the sensitivity of the decision with respect to that set.
Revised: We then attribute policy decisions to a set of trajectories in this encoded space by estimating the sensitivity of the decision with respect to that set.

Original: Further, we demonstrate the effectiveness of the proposed approach in terms of quality of attributions as well as practical scalability in diverse environments that involve both discrete and continuous state and action spaces such as grid-worlds, video games (Atari) and continuous control (MuJoCo).
Revised: Further, we demonstrate the effectiveness of the proposed approach in terms of quality of attributions as well as practical scalability in diverse environments that involve both discrete and continuous state and action spaces such as grid-worlds, video games (Atari) and continuous control (MuJoCo).

Original:  I NTRODUCTION
Revised:  Introduction

Original: Reinforcement learning (Sutton & Barto, 2018) has enjoyed great popularity and has achieved huge success, especially in the online settings, post advent ofthe deep reinforcement learning (Mnih et al., 2013; Schulman et al., 2017; Silver et al., 2017; Haarnoja et al., 2018).
Revised: Reinforcement learning (Sutton & Barto, 2018) has enjoyed great popularity and has achieved huge success, especially in the online settings, post advent of deep reinforcement learning (Mnih et al., 2013; Schulman et al., 2017; Silver et al., 2017; Haarnoja et al., 2018).

Original: Deep RL algorithms are now able to handle high-dimensional observations such as visual inputs with ease.
Revised: Deep RL algorithms are now able to handle high-dimensional observations such as visual inputs with ease.

Original: However, using these algorithms in the real world requires- i) efficient learning from minimal exploration to avoid catastrophic decisions due to insufficient knowledge of the environment, and ii) being explainable.
Revised: However, using these algorithms in the real world requires i) efficient learning from minimal exploration to avoid catastrophic decisions due to insufficient knowledge of the environment, and ii) being explainable.

Original: The first aspect is  being studied under offline RL (Levine et al., 2020) where the agent is trained on collected experience rather than exploring directly in the environment. There is a huge body of work on offline RL (Levine et al., 2020; Kumar et al., 2020;
Revised: The first aspect is extensively being studied under offline RL (Levine et al., 2020) where the agent is trained on collected experience rather than exploring directly in the environment (Kumar et al., 2020;

Original: Yu et al., 2020; Kostrikov et al., 2021).
Revised: Yu et al., 2020; Kostrikov et al., 2021).

Original: However, more work is needed to address the explainability aspect of RL decision-making.
Revised: However, more work is needed to address the explainability aspect of RL decision-making.

Original: Previously, researchers have attempted explaining decisions of RL agent by highlighting important features of the agent’s state (input observation) (Puri et al., 2019;
Revised: Previously, researchers have attempted explaining decisions of RL agent by highlighting important features of the agent’s state (input observation) (Puri et al., 2019;

Original: Iyer et al., 2018; Greydanus et al., 2018).
Revised: Iyer et al., 2018; Greydanus et al., 2018).

Original: While these approaches are useful, we take a complementary route.
Revised: While these approaches are useful, we take a complementary route.

Original: Instead of identifying salient state-features, we wish to identify the past experiences (trajectories) that led the RL agent to learn certain behaviours.
Revised: Instead of identifying salient state-features, we wish to identify the past experiences (trajectories) that led the RL agent to learn certain behaviours.

Original: We call this approach as trajectory-aware RL explainability.
Revised: We call this approach as trajectory-aware RL explainability.

Original: Such explainability confers faith in the decisions suggested by the RL agent in critical scenarios (surgical (Loftus et al., 2020), nuclear (Boehnlein et al., 2022), etc.) by looking at the trajectories responsible for the decision.
Revised: Such explainability confers faith in the decisions suggested by the RL agent in critical scenarios (surgical (Loftus et al., 2020), nuclear (Boehnlein et al., 2022), etc.) by looking at the trajectories responsible for the decision.

Original: While this sort of training data attribution has been shown to be highly effective in supervised learning (Nguyen et al., 2021), to the best of our knowledge, this is the first work to study data attribution-based explainability in RL.
Revised: While this sort of training data attribution has been shown to be highly effective in supervised learning (Nguyen et al., 2021), to the best of our knowledge, this is the first work to study data attribution-based explainability in RL.

Original: In the present work, we restrict ourselves to the offline RL setting.
Revised: In the present work, we restrict ourselves to the offline RL setting.

Original: Contributions of this work are enumerated below:
Revised: Contributions of this work are enumerated below:

Original: A novel explainability framework for reinforcement learning that aims to find experiences(trajectories) that lead an RL agent  learn certain behaviour.
Revised: A novel explainability framework for reinforcement learning that aims to find experiences (trajectories) that lead an RL agent to learn certain behaviour.

Original: A solution for trajectory attribution in offline RL setting based on state-of-the-art sequence modeling techniques.
Revised: A solution for trajectory attribution in offline RL setting based on state-of-the-art sequence modeling techniques.

Original: In our solution, we present a methodology that generates a single embedding for a trajectory of states, actions, and rewards.
Revised: In our solution, we present a methodology that generates a single embedding for a trajectory of states, actions, and rewards.

Original: We also extend this method to generate a single encoding of data containing a set of trajectories.
Revised: We also extend this method to generate a single encoding of data containing a set of trajectories.

Original: Analysis of trajectory explanations produced by our technique along with analysis of the trajectory embeddings generated, where we demonstrate how different embedding clusters represent different semantically meaningful behaviours A.6.
Revised: Analysis of trajectory explanations produced by our technique along with analysis of the trajectory embeddings generated, where we demonstrate how different embedding clusters represent different semantically meaningful behaviours (see Appendix A.6).

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  Background and Related Work

Original: Explainability in RL.
Revised: Explainability in RL.

Original: Explainable AI (XAI) refers to the field of machine learning (ML) that focuses on developing tools for explaining the decisions of ML models.
Revised: Explainable AI (XAI) refers to the field of machine learning (ML) that focuses on developing tools for explaining the decisions of ML models.

Original: Explainable RL (XRL) (Puiutta & Veith, 2020) is a sub-field of XAI that specializes in interpreting behaviours of RL agents.
Revised: Explainable RL (XRL) (Puiutta & Veith, 2020) is a sub-field of XAI that specializes in interpreting behaviours of RL agents.

Original: Prior works include approaches that distill the RL policy into simpler models such as decision tree (Coppens et al., 2019) or to human understandable high-level decision language (Verma et al., 2018).
Revised: Prior works include approaches that distill the RL policy into simpler models such as decision tree (Coppens et al., 2019) or to human understandable high-level decision language (Verma et al., 2018).

Original: However, such policy simplification fails to approximate the behavior of complex RL models.
Revised: However, such policy simplification fails to approximate the behavior of complex RL models.

Original: In addition, causality-based approaches (Pawlowski et al., 2020; Madumal et al., 2020) aim to explain an action’s agent by identifying the cause behind it using counterfactual samples.
Revised: In addition, causality-based approaches (Pawlowski et al., 2020; Madumal et al., 2020) aim to explain an action’s agent by identifying the cause behind it using counterfactual samples.

Original: To the best of our knowledge, for the first time, we explore the direction of explaining an agent’s behaviour by attributing its actions to past encountered trajectories rather than highlighting state features.
Revised: To the best of our knowledge, for the first time, we explore the direction of explaining an agent’s behaviour by attributing its actions to past encountered trajectories rather than highlighting state features.

Original: Sequence Modeling in Offline RL.
Revised: Sequence Modeling in Offline RL.

Original: Recently, the RL problem of maximizing long-term return has been cast as taking the best possible action given the sequence of past interactions in terms of states, actions, rewards (Chen et al., 2021; Janner et al., 2021;
Revised: Recently, the RL problem of maximizing long-term return has been cast as taking the best possible action given the sequence of past interactions in terms of states, actions, rewards (Chen et al., 2021; Janner et al., 2021;

Original: Reed et al., 2022; Park et al., 2018).
Revised: Reed et al., 2022; Park et al., 2018).

Original: Such sequence modelling approaches to RL, especially the ones based on transformer architecture (Vaswani et al., 2017), have produced state-of-the-art results in various offline RL benchmarks, and offer rich latent representations to work with.
Revised: Such sequence modelling approaches to RL, especially the ones based on transformer architecture (Vaswani et al., 2017), have produced state-of-the-art results in various offline RL benchmarks, and offer rich latent representations to work with.

Original: However, little to no work has been done in the direction of understanding these sequence representations and their applications.
Revised: However, little to no work has been done in the direction of understanding these sequence representations and their applications.

Original: In this work, we base our solution on these sequence modelling approaches to leverage their high efficiency in capturing the policy and environment dynamics of the offline RL systems.T RAJECTORY A TTRIBUTION [Figure] 𝜏 ! :(
Revised: In this work, we base our solution on these sequence modelling approaches to leverage their high efficiency in capturing the policy and environment dynamics of the offline RL systems.

Original: 𝑠 " ,𝑎 " ,𝑟 " ,𝑠 # ,𝑎 # ,𝑟 # ,…,𝑠 $ ,𝑎 $ ,𝑟 $ )  …

Original: Step 2:  Trajectory Clustering  …  …  …  …  …  …  …  …  …  …  … … C C C n …
Revised:  Trajectory Attribution

Original: Step 3: Data Embedding  …  …  …  …  … …

Original: [Figure]

Original: {𝜏 ! }𝜋 % 𝜋 # … 𝑡 # 𝑡 & 𝑡 ’ 𝑡 ( 𝑡 !

Original: [Figure]

Original: Preliminaries.
Revised: Preliminaries.

Original: We denote the offline RL dataset using D that comprises a set of n τ trajectories.
Revised: We denote the offline RL dataset using D that comprises a set of n τ trajectories.

Original: Each trajectory τ j ∈ D comprises of a sequence of observation ( o k ), action ( a k ) and per-step reward ( r k ) tuples with k ranging from 1 to the length of the trajectory τ j .
Revised: Each trajectory τ j ∈ D comprises of a sequence of observation ( o k ), action ( a k ) and per-step reward ( r k ) tuples with k ranging from 1 to the length of the trajectory τ j .

Original: We begin by training an offline RL agent on this data using any standard offline RL algorithm.
Revised: We begin by training an offline RL agent on this data using any standard offline RL algorithm.

Original: Algorithm.
Revised: Algorithm.

Original: Having obtained the learned policy using an offline RL algorithm, our objective now is to attribute this policy, i.e., the action chosen by this policy, at a given state to a set of trajectories.
Revised: Having obtained the learned policy using an offline RL algorithm, our objective now is to attribute this policy, i.e., the action chosen by this policy, at a given state to a set of trajectories.

Original: We intend to achieve this in the following way.
Revised: We intend to achieve this in the following way.

Original: We want to find the smallest set of trajectories, the absence of which from the training data leads to different behaviour at the state under consideration.
Revised: We want to find the smallest set of trajectories, the absence of which from the training data leads to different behaviour at the state under consideration.

Original: That is, we posit that this set of trajectories contains specific behaviours and respective feedbacks from the environment that train the RL agent to make decisions in a certain manner.
Revised: That is, we posit that this set of trajectories contains specific behaviours and respective feedbacks from the environment that train the RL agent to make decisions in a certain manner.

Original: This identified set of trajectories would then be provided as attribution for the original decision.
Revised: This identified set of trajectories would then be provided as attribution for the original decision.

Original: While this basic idea is straightforward and intuitive, it is not computationally feasible beyond a few small RL problems with discrete state and action spaces.
Revised: While this basic idea is straightforward and intuitive, it is not computationally feasible beyond a few small RL problems with discrete state and action spaces.

Original: The key requirement to scale this approach to large, continuous state and action space problems, is to group the trajectories into clusters which can then be used to analyse their role in the decision-making of the RL agent.
Revised: The key requirement to scale this approach to large, continuous state and action space problems, is to group the trajectories into clusters which can then be used to analyse their role in the decision-making of the RL agent.

Original: In this work, we propose to cluster the trajectories using trajectory embeddings produced with the help of state-of- the-art sequence modelling approaches.
Revised: In this work, we propose to cluster the trajectories using trajectory embeddings produced with the help of state-of-the-art Step [Figure] : Trajectory Encoding 𝜏 ! :(

Original: Fig.
Revised: 𝑠 " ,𝑎 " ,𝑟 " ,𝑠 # ,𝑎 # ,𝑟 # ,…,𝑠 $ ,𝑎 $ ,𝑟 $ ) 0 … Step 2: Trajectory Clustering 0 … 0 … 0 … 0 … 0 … 0 … 0 … 0 … 0 … 0 … 0 … … C C

Revised: C n …

Revised: Step 3: Data Embedding 0 … 0 … 0 … 0 … 0 … …

Revised: [Figure]

Revised: 𝜏 !

Revised: [Figure]

Revised: 𝑡 # 𝑡 & 𝑡 ’ 𝑡 (

Revised: [Figure]

Revised: 𝜋 )*+, (𝑠,𝑎) 𝜋 #

Revised: [Figure]

Revised: sequence modelling approaches.

Revised: Fig.

Original: 1 gives an overview of our proposed approach, the steps of which are described below: (i) Trajectory Encoding.
Revised: 1 gives an overview of our proposed approach, the steps of which are described below: (i) Trajectory Encoding.

Original: First, we tokenize the trajectories in the offline data according to the specifications of the sequence encoder used (e.g. decision transformer/trajectory transformer).
Revised: First, we tokenize the trajectories in the offline data according to the specifications of the sequence encoder used (e.g. decision transformer/trajectory transformer).

Original: The observation, action and reward tokens of a trajectory are then fed to the sequence encoder to produce corresponding latent representations, which we refer to as output tokens.
Revised: The observation, action and reward tokens of a trajectory are then fed to the sequence encoder to produce corresponding latent representations, which we refer to as output tokens.

Original: We define the trajectory embedding as an average of these output tokens.
Revised: We define the trajectory embedding as an average of these output tokens.

Original: This technique is inspired by average-pooling techniques (Choi et al., 2021; Briggs, 2021) in NLP used to create sentence embedding from embeddings of words present in it.
Revised: This technique is inspired by average-pooling techniques (Choi et al., 2021; Briggs, 2021) in NLP used to create sentence embedding from embeddings of words present in it.

Original: (Refer to Alg. 1.)
Revised: (Refer to Alg. 1.)

Original: (ii) Trajectory Clustering.
Revised: (ii) Trajectory Clustering.

Original: Having obtained trajectory embeddings, we cluster them using X-Means clustering algorithm (Pelleg et al., 2000) with implementation provided by Novikov (2019).
Revised: Having obtained trajectory embeddings, we cluster them using X-Means clustering algorithm (Pelleg et al., 2000) with implementation provided by Novikov (2019).

Original: While in principle, any suitable clustering algorithm can be used here, we chose X-Means as it is a simple extension to the K-means clustering algorithm (Lloyd, 1982); it determines the number of clusters n c automatically.
Revised: While in principle, any suitable clustering algorithm can be used here, we chose X-Means as it is a simple extension to the K-means clustering algorithm (Lloyd, 1982); it determines the number of clusters n c automatically.

Original: This enables us to identify all possible patterns in the trajectories without forcing n c as a hyperparameter (Refer to Alg. 2).
Revised: This enables us to identify all possible patterns in the trajectories without forcing n c as a hyperparameter (Refer to Alg. 2).

Original: (iii) Data Embedding.
Revised: (iii) Data Embedding.

Original: We need a way to identify the least change in the original data that leads to the change in behaviour of the RL agent.
Revised: We need a way to identify the least change in the original data that leads to the change in behaviour of the RL agent.

Original: To achieve this, we propose a representation for data comprising collection of trajectories.
Revised: To achieve this, we propose a representation for data comprising collection of trajectories.

Original: The representation has to be agnostic to the order in which trajectories are present in the collection.
Revised: The representation has to be agnostic to the order in which trajectories are present in the collection.

Original: So, we follow the set-encoding procedure prescribed in (Zaheer et al., 2017) where we first sum the embeddings of the trajectories in the collection, normalize this sum by division with a constant and further apply a non-linearity, in our case, simply, softmax over the feature dimension to generate a single data embedding (Refer to Alg. 3).
Revised: So, we follow the set-encoding procedure prescribed in (Zaheer et al., 2017) where we first sum the embeddings of the trajectories in the collection, normalize this sum by division with a constant and further apply a non-linearity, in our case, simply, softmax over the feature dimension to generate a single data embedding (Refer to Alg. 3).

Original: We use this technique to generate data embeddings for n c + 1 sets of trajectories.The first
Revised: We use this technique to generate data embeddings for n c + 1 sets of trajectories.

Original:  set represents the entire training data whose embedding is denoted by ¯ d orig .
Revised: The first set represents the entire training data whose embedding is denoted by ¯ d orig .

Original: The remaining n c sets are constructed as follows.
Revised: The remaining n c sets are constructed as follows.

Original: For each trajectory cluster c j , we construct a set with the entire training data but the trajectories contained in c j .We call
Revised: For each trajectory cluster c j , we construct a set with the entire training data but the trajectories contained in c j .

Original: this

Original:  set the complementary data set corresponding to cluster c j and the corresponding data embedding 
Revised: We call this set the complementary data set corresponding to cluster c j and the corresponding data embedding as

Original: as

Original: the complementary data embedding ¯ d j  (iv) Training Explanation Policies.
Revised: the complementary data embedding ¯ d j . (iv) Training Explanation Policies.

Original: In this step, for each cluster c j , using its complementary data set, we train an offline RL agent.
Revised: In this step, for each cluster c j , using its complementary data set, we train an offline RL agent.

Original: We ensure that all the training conditions (algorithm, weight initialization, optimizers, hyperparameters, etc.) are identical to the training of original RL policy, except for the modification in the training data.
Revised: We ensure that all the training conditions (algorithm, weight initialization, optimizers, hyperparameters, etc.) are identical to the training of original RL policy, except for the modification in the training data.

Original: We call this newly learned policy as the explanation policy corresponding to cluster c j .
Revised: We call this newly learned policy as the explanation policy corresponding to cluster c j .

Original: We thus get n c explanation policies at the end of this step.
Revised: We thus get n c explanation policies at the end of this step.

Original: In addition, we compute data embeddings for complementary data sets (Refer to Alg. 4).
Revised: In addition, we compute data embeddings for complementary data sets (Refer to Alg. 4).

Original: (v) Cluster Attribution.
Revised: (v) Cluster Attribution.

Original: In this final step, given a state, we note the actions suggested by all the explanation policies at this state.
Revised: In this final step, given a state, we note the actions suggested by all the explanation policies at this state.

Original: We then compute the distances of these actions (where we assume a metric over the action space) from the action suggested by the original RL agent at the state.
Revised: We then compute the distances of these actions (where we [Figure] O b s e r v a t i o n C o n t e x t

Revised: T r a j e c t

Revised: o r y

Revised: ( i )

Revised: T r a j e c t

Revised: o r y ( ii )

Revised: T r a j e c t

Revised: o r y ( iii ) assume a metric over the action space) from the action suggested by the original RL agent at the state.

Original: The explanation policies corresponding to the maximum of these distances form the candidate attribution set.
Revised: The explanation policies corresponding to the maximum of these distances form the candidate attribution set.

Original: For each policy in this candidate attribution set, we compute the distance between its respective complementary data embedding and the data embedding of the entire training data using the Wasserstein metric for capturing distances between softmax simplices (Vallender, 1974).
Revised: For each policy in this candidate attribution set, we compute the distance between its respective complementary data embedding and the data embedding of the entire training data using the Wasserstein metric for capturing distances between softmax simplices (Vallender, 1974).

Original: We then select the policy that has the smallest data distance and attribute the decision of the RL agent to the cluster corresponding to this policy(Refer to Alg. 5).
Revised: We then select the policy that has the smallest data distance and attribute the decision of the RL agent to the cluster corresponding to this policy(Refer to Alg. 5).

Original: Our approach comprised of all the five steps is summarized in Algorithm 6.
Revised: Our approach comprised of all the five steps is summarized in Algorithm 6.

Original:  R ESULTS
Revised:  Results

Original: Next, we present experimental results on the Atari Seaquest environment (Bellemare et al., 2013) to show the effectiveness of our approach in generating trajectory explanations.
Revised: Next, we present experimental results on the Atari Seaquest environment (Bellemare et al., 2013) to show the effectiveness of our approach in generating trajectory explanations.

Original: The details of the experimental setup are given in Sec.
Revised: The details of the experimental setup are given in Sec.

Original: A.2.
Revised: A.2.

Original: 2 shows Seaquest agent (submarine) suggesting action ‘left’ for given observation in the context of past few frames. The corresponding attributed trajectories provide the insights into how the submarine is aligning itself to target enemies coming from the left. Table 1 presents quantitative analysis of the proposed trajectory attribution using evaluation metrics described in A.2. Initial state value estimate ( E ( V ( s 0 )) ) for original policy π orig compares with
Revised: Fig.

Original: [Figure] FIRE FIRE [Figure]
Revised: 2 shows trajectory attribution results for RL agent playing Seaquest Atari game.

Original: LEFT

Original: O b s e r v a t i o n C o n t e x t T r a j e c t o r y
Revised: Moreover, table 1 presents quantitative analysis of the proposed trajectory attribution using evaluation metrics described in Sec.

Original: ( i )
Revised: A.2.

Original: T r a j e c t o r y ( ii ) T r a j e c t o r y ( iii )  estimates for explanation policies trained on different complementary datasets.
Revised: Initial state value estimate ( E ( V ( s 0 )) ) for original policy π orig compares with estimates for explanation policies trained on different complementary datasets.

Original: This indicates that π orig , having access to all behaviours, is able to outperform other policies that are trained on data lacking information about important behaviours (e.g. fighting in the top-right corner).
Revised: This indicates that π orig , having access to all behaviours, is able to outperform other policies that are trained on data lacking information about important behaviours (e.g. fighting in the top-right corner).

Original: Further, local mean absolute action-value difference ( E ( | ∆ Q π orig | ) =
Revised: Further, local mean absolute action-value difference ( E ( | ∆ Q π orig | ) =

Original: E ( | Q π orig ( π orig ( s )) − Q π orig ( π j ( s )) | ) ) and action difference ( E ( 1 ( π orig ( s ) ̸
Revised: E ( | Q π orig ( π orig ( s )) − Q π orig ( π j ( s )) | ) ) and action difference ( E ( 1 ( π orig ( s ) ̸

Original: = π j ( s )) ) turn out to be highly correlated, i.e., the explanation policies that suggest the most contrasting actions are usually perceived by the original policy as lowreturn actions.
Revised: = π j ( s ))) ) turn out to be highly correlated, i.e., the explanation policies that suggest the most contrasting actions are usually perceived by the original policy as low-return actions.

Original: This evidence supports the proposed trajectory algorithm as we want to identify the behaviours which when removed make agent choose actions that are not considered suitable originally.
Revised: This evidence supports the proposed trajectory algorithm as we want to identify the behaviours which when removed make agent choose actions that are not considered suitable originally.

Original: In addition, in columns 4 and 5, we provide normalized Wasserstein distances between the data embeddings ( W dist ( ¯ d, ¯ d j ) ) and the cluster attribution distribution ( P ( c final = c j ) ), respectively.
Revised: In addition, in columns 4 and 5, we provide normalized Wasserstein distances between the data embeddings ( W dist ( ¯ d, ¯ d j ) ) and the cluster attribution distribution ( P ( c final = c j ) ), respectively.

Original: The latter depicts how RL decisions are dependent on various behaviour clusters.[Table]
Revised: The latter depicts how RL decisions are dependent on various behaviour clusters.

Original: The results on Grid-world and MuJoCo HalfCheetah
Revised: The results on Grid-world and MuJoCo HalfCheetah

Original: (Todorov et al., 2012) are given in Sec.
Revised: (Todorov et al., 2012) are given in Sec.

Original: A.5.1; additional qualitative results are given in Sec.
Revised: A.5.1; additional qualitative results are given in Sec.

Original: A.5.2; clustering analysis is provided in Sec.
Revised: A.5.2; clustering analysis is provided in Sec.

Original: A.6.
Revised: A.6.

Revised: [Table]

Original:  D ISCUSSION
Revised:  Discussion

Original: In this work, we proposed a novel explanation technique that attributes decisions suggested by an
Revised: In this work [Table] we proposed a novel explanation technique that attributes decisions suggested by an

Original: RL agent to trajectories encountered by the agent in the past.
Revised: RL agent to trajectories encountered by the agent in the past.

Original: We provided an algorithm that enables us to perform trajectory attribution in offline RL.
Revised: We provided an algorithm that enables us to perform trajectory attribution in offline RL.

Original: The key idea behind our approach was to study sensitivity of original RL agent’s policy to the trajectory clusters obtained using sequence modelling techniques.
Revised: The key idea behind our approach was to study sensitivity of original RL agent’s policy to the trajectory clusters obtained using sequence modelling techniques.

Original: We demonstrated the utility of our method using experiments in grid-world, Seaquest and HalfCheetah environments.
Revised: We demonstrated the utility of our method using experiments in grid-world, Seaquest and HalfCheetah environments [Table]

Original: A A PPENDIX A. A LGORITHMS
Revised: A Appendix A.1 Algorithms

Revised: Algorithm 1:

Original: /* Encoding given set of trajectories individually */
Revised: /* Encoding given set of trajectories individually */

Original: Input : Offline Data { τ i } , Sequence Encoder E
Revised: Input : Offline Data { τ i }, Sequence Encoder E

Original: Initialize: Initialize array T to collect the trajectory embeddings 1 for τ j in { τ i } do / * Using E , get output tokens for all the o , a & r in τ j */ ( e o 1 ,j , e a 1 ,j , e r 1 ,j , ..., e o T ,j , e a T ,j , e r T ,j ) ← E ( o 1 ,j , a 1 ,j , r 1 ,j , ..., o T ,j , a T ,j , r T ,j ) // where 3 T = #input tokens /
Revised: Initialize : Initialize array T to collect the trajectory embeddings 1 for τ j in { τ i } do /* Using E , get output tokens for all the o , a & r in τ j */ ( e o 1 ,j , e a 1 ,j , e r 1 ,j , ..., e o T ,j , e a T ,j , e r T ,j ) ← E ( o 1 ,j , a 1 ,j , r 1 ,j , ..., o T ,j , a T ,j , r T ,j ) // where 3 T = #input tokens /

Original: * Take mean of outputs to generate τ j ’s embedding t j
Revised: * Take mean of outputs to generate τ j ’s embedding t j

Original: */ t j ← ( e o 1 ,j + e a 1 ,j + e r 1 ,j + e o 2 ,j + e a 2 ,j + e r 2 ,j + ...
Revised: */ t j ← ( e o 1 ,j + e a 1 ,j + e r 1 ,j + e o 2 ,j + e a 2 ,j + e r 2 ,j + ...

Original: + e o T ,j + e a T ,j + e r T ,j ) / (3 T )
Revised: + e o T ,j + e a T ,j + e r T ,j ) / (3 T )

Original: Append t j to T
Revised: Append t j to T

Original: Output  Return 
Revised: Output : Return the trajectory embeddings T = { t i } Algorithm

Original: /* Clustering the trajectories using their embeddings */
Revised: /* Clustering the trajectories using their embeddings */

Original: Input : Trajectory embeddings T =
Revised: Input : Trajectory embeddings T =

Original: { t i } , clusteringAlgo 1 C ← clusteringAlgo(T) //
Revised: { t i } , clusteringAlgo 1 C ← clusteringAlgo(T) //

Original: Cluster using provided clustering algorithm
Revised: Cluster using provided clustering algorithm

Original: Output : Return trajectory clusters C
Revised: Output : Return trajectory clusters C

Original: = { c i } n c i =
Revised: = { c i } n c i =

Original: Algorithm 
Revised: Algorithm 3:

Original: /* Generating data embedding for a given set of trajectories */
Revised: /* Generating data embedding for a given set of trajectories */

Original: Input : Trajectory embeddings T =
Revised: Input : Trajectory embeddings T =

Original: { t i } , Normalizing factor M , Softmax temperature T soft 1 ¯ s ← (cid:80)
Revised: { t i } , Normalizing factor M , Softmax temperature T soft 1 ¯ s ← (cid:80)

Original: i t i M //
Revised: i t i M //

Original: Sum the trajectory embeddings and normalize them 2 ¯ d ← { d j | d j = exp ( s j /T soft ) (cid:80)
Revised: Sum the trajectory embeddings and normalize them 2 ¯ d ← { d j | d j = exp ( s j /T soft ) (cid:80)

Original: k exp ( s k /T soft ) } // Take softmax along feature dimension
Revised: k exp ( s k /T soft ) } // Take softmax along feature dimension

Original: Output : Return the data embedding ¯ d
Revised: Output : Return the data embedding ¯ d

Revised: Algorithm 4:

Original: /* Train explanation policies and compute corresponding data embeddings 
Revised: /* Train explanation policies and compute corresponding data embeddings */

Original: Input : Offline data { τ i } , Traj.
Revised: Input : Offline data { τ i } , Traj.

Original: Embeddings T , Traj.
Revised: Embeddings T , Traj.

Original: Clusters C , offlineRLAlgo 1 for c j in C do { τ i } j ← { τ i } − c j // Compute complementary dataset corresp.
Revised: Clusters C , offlineRLAlgo 1 for c j in C do { τ i } j ← { τ i } − c j // Compute complementary dataset corresp.

Original: to c j
Revised: to c j

Original: T j ← gatherTrajectoryEmbeddings ( T, { τ i } j ) // Gather corresp.
Revised: T j ← gatherTrajectoryEmbeddings ( T, { τ i } j ) // Gather corresp.

Original: τ embeds
Revised: τ embeds

Original: Explanation policy, π j ← offlineRLAlgo ( { τ i } j )
Revised: Explanation policy, π j ← offlineRLAlgo ( { τ i } j )

Original: Complementary data embedding, ¯ d j ← generateDataEmbedding ( T j , M, T soft )
Revised: Complementary data embedding, ¯ d j ← generateDataEmbedding ( T j , M, T soft )

Original: Output : Explanation policies { π j } , Complementary data embeddings { ¯ d j }
Revised: Output : Explanation policies { π j } , Complementary data embeddings { ¯ d j }

Revised: Algorithm

Original: /* Generating cluster attributions for a orig
Revised: /* Generating cluster attributions for a orig

Original: =
Revised: =

Original: π orig ( s ) */
Revised: π orig ( s ) */

Original: Input : State s , Original Policy π orig , Explanation Policies { π j } , Original Data Embedding ¯ d orig , Complementary Data Embeddings { ¯ d j } 1 Original action, a orig ← π orig ( s) 2 Actions suggested by explanation policies, a j ← π j ( s ) 3 d a orig ,a j ← calcActionDistance( a orig , a j ) // Compute action distance 4 K ← argmax ( d a orig ,a j ) // Get candidate clusters using argmax 5 w k ← W dist ( ¯ d orig , ¯ d k ) // Compute Wasserstein distance b/w complementary data embeddings of candidate clusters & orig data embedding 6 c final ← argmin ( w k ) //
Revised: Input : State s , Original Policy π orig , Explanation Policies { π j } , Original Data Embedding ¯ d orig , Complementary Data Embeddings { ¯ d j } 1 Original action, a orig ← π orig ( s

Original: Choose cluster with min data embedding dist.

Original: Output : c final
Revised: Output : c final

Original: Input : Offline Data { τ i } , States needing explanation S exp , Sequence Encoder E , offlineRLAlgo, clusteringAlgo, Normalizing constant M , Softmax Temperature T soft /
Revised: Input : Offline Data { τ i }, States needing explanation S exp , Sequence Encoder E , offlineRLAlgo, clusteringAlgo, Normalizing constant M , Softmax Temperature T soft /

Original: * Train original offline RL policy */ 1 π orig
Revised: * Train original offline RL policy */ 1 π orig

Original: ← offlineRLAlgo ( { τ i } ) /
Revised: ← offlineRLAlgo ( { τ i } ) /

Original: * Encode individual trajectories */ 2 T = encodeTrajectories( { τ i } , E ) // Algo./* Cluster the trajectories */ 3 C ← clusterTrajectories(T, clusteringAlgo) // Algo./* Compute data embedding for the entire dataset */ 4 ¯ d orig = generateDataEmbedding(T, M, T soft ) // Algo./* Generate explanation policies and their corresponding complementary data embeddings */ 5 { π j } , { ¯ d j } ← trainExpPolicies( { τ i } , T, C , offlineRLAlgo) // Algo./* Attributing policy decisions for given set of states */  fors ∈ S do
Revised: * Encode individual trajectories */ 2 T = encodeTrajectories({ τ i }, E ) // Algo./* Cluster the trajectories */ 3 C ← clusterTrajectories(T, clusteringAlgo) // Algo./* Compute data embedding for the entire dataset */ 4 ¯ d orig = generateDataEmbedding(T, M, T soft ) // Algo./* Generate explanation policies and their corresponding complementary data embeddings */ 5 { π j } , { ¯ d j } ← trainExpPolicies( { τ i } , T, C , offlineRLAlgo) // Algo./* Attributing policy decisions for given set of states */ 6 for

Original: ( s
Revised: S do generateClusterAttribution( s, π , { π j } , ¯ d orig , ¯ d j ) //

Original: // Algo.
Revised: Algo.

Original: A. E XPERIMENTAL S ETUP
Revised: A.2 Experimental Setup

Original: We first describe the environments, models, and metrics designed to study the reliability of our trajectory explanations.
Revised: We first describe the environments, models, and metrics designed to study the reliability of our trajectory explanations.

Original: RL Environments.
Revised: RL Environments.

Original: We perform experiments on three environments: i) Grid-world (Fig. 3) which has discrete state and action spaces, ii) Seaquest from Atari suite which has environments with continuous visual observations and discrete action spaces (Bellemare et al., 2013), and iii)
Revised: We perform experiments on three environments: i) Grid-world (Fig. 3) which has discrete state and action spaces, ii) Seaquest from Atari suite which has environments with continuous visual observations and discrete action spaces (Bellemare et al., 2013), and iii)

Original: HalfCheetah from MuJoCo environments which are control environments with continuous state and action spaces (Todorov et al., 2012).
Revised: HalfCheetah from MuJoCo environments which are control environments with continuous state and action spaces (Todorov et al., 2012).

Original: Offline Data and Sequence Encoders.
Revised: Offline Data and Sequence Encoders.

Original: For grid-world, we collect offline data of 60 trajectories from policy rollout of other RL agents and train an LSTM-based trajectory encoder following the procedure described in trajectory transformer, replacing the transformer with LSTM.
Revised: For grid-world, we collect offline data of 60 trajectories from policy rollout of other RL agents and train an LSTM-based trajectory encoder following the procedure described in trajectory transformer, replacing the transformer with LSTM.

Original: For Seaquest, we collect offline data of 717 trajectories from the D4RL-Atari repository1 and use a pre-trained decision transformer2 as trajectory encoder.
Revised: For Seaquest, we collect offline data of 717 trajectories from the D4RL-Atari repository and use a pre-trained decision transformer as trajectory encoder.

Original: Similarly, for HalfCheetah, we collect offline data of 1000 trajectories from the D4RL repository3 (Fu et al., 2020) and use a pre-trained trajectory transformer4 as a trajectory encoder.
Revised: Similarly, for HalfCheetah, we collect offline data oftrajectories from the D4RL repository (Fu et al., 2020) and use a pre-trained trajectory transformer as a trajectory encoder.

Original: To cluster high-level skills in long trajectory sequences, we divide the Seaquest trajectories into 30-length sub-trajectories and the HalfCheetah trajectories into 25length sub-trajectories.
Revised: To cluster high-level skills in long trajectory sequences, we divide the Seaquest trajectories into 30-length sub-trajectories and the HalfCheetah trajectories into 25-length sub-trajectories.

Original: These choices were made based on the transformers’ input block sizes and the quality of clustering.
Revised: These choices were made based on the transformers’ input block sizes and the quality of clustering.

Original: Offline RL Training and Data Embedding.
Revised: Offline RL Training and Data Embedding.

Original: We train the offline RL agents for each environment using the data collected as follows - for grid-world, we use model-based offline RL, and for Seaquest and HalfCheetah, we employ DiscreteSAC (Christodoulou, 2019) and SAC (Haarnoja et al., 2018), respectively, using d3rlpy implementations (Takuma Seno, 2021).
Revised: We train the offline RL agents for each environment using the data collected as follows - for grid-world, we use model-based offline RL, and for Seaquest and HalfCheetah, we employ DiscreteSAC (Christodoulou, 2019) and SAC (Haarnoja et al., 2018), respectively, using d3rlpy implementations (Takuma Seno, 2021).

Original: We compute data embedding of entire training data for each of the environments.
Revised: We compute data embedding of entire training data for each of the environments.

Original: See Sec.
Revised: See Sec.

Original: A.4 for additional training details.
Revised: A.4 for additional training details.

Original: Encoding of Trajectories and Clustering.
Revised: Encoding of Trajectories and Clustering.

Original: We encode the trajectory data using sequence encoders and cluster the output trajectory embeddings using the X-means algorithm.
Revised: We encode the trajectory data using sequence encoders and cluster the output trajectory embeddings using the X-means algorithm.

Original: More specifically, we obtain 10 trajectory clusters for grid-world, 8 for Seaquest, and 10 for HalfCheetah.
Revised: More specifically, we obtain 10 trajectory clusters for grid-world, 8 for Seaquest, and 10 for HalfCheetah.

Original: Complementary Data Sets.
Revised: Complementary Data Sets.

Original: We obtain complementary data sets using the aforementioned cluster information and provide 10 complementary data sets for grid-world, 8 for Seaquest, and 10 for
Revised: We obtain complementary data sets using the aforementioned cluster information and provide 10 complementary data sets for grid-world, 8 for Seaquest, and 10 for

Original: HalfCheetah.
Revised: HalfCheetah.

Original: Next, we compute data embeddings corresponding to these newly formed data sets.
Revised: Next, we compute data embeddings corresponding to these newly formed data sets.

Revised: https://github.com/takuseno/d4rl-atarihttps://huggingface.co/edbeeching/decision_transformer_atari https://github.com/Farama-Foundation/D4RLhttps:

Original: Explanation Policies.
Revised: Explanation Policies.

Original: Subsequently, we train explanation policies on the complementary data sets for each environment.
Revised: Subsequently, we train explanation policies on the complementary data sets for each environment.

Original: The training produces 10 additional policies for grid-world, 8 policies for
Revised: The training produces 10 additional policies for grid-world, 8 policies for

Original: Seaquest, and 10 policies for HalfCheetah.
Revised: Seaquest, and 10 policies for HalfCheetah.

Original: In summary, we train the original policy on the entire data, obtain data embedding for the entire data, cluster the trajectories and obtain their explanation policies and complementary data embeddings.
Revised: In summary, we train the original policy on the entire data, obtain data embedding for the entire data, cluster the trajectories and obtain their explanation policies and complementary data embeddings.

Original: Trajectory Attribution.
Revised: Trajectory Attribution.

Original: Finally, we attribute a decision made by the original policy for a given state to a trajectory cluster.
Revised: Finally, we attribute a decision made by the original policy for a given state to a trajectory cluster.

Original: We choose top-3 trajectories from these attributed clusters by matching the context for the state-action under consideration with trajectories in the cluster in our experiments.
Revised: We choose top-3 trajectories from these attributed clusters by matching the context for the state-action under consideration with trajectories in the cluster in our experiments.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We compare policies trained on different data using three metrics (deterministic nature of policies is assumed throughout the discussion)
Revised: We compare policies trained on different data using three metrics (deterministic nature of policies is assumed throughout the discussion)

Original: – 1) Initial State Value Estimate denoted by E ( V ( s 0 )) which is a measure of expected long-term returns to evaluate offline RL training as described in Paine et al.
Revised: – 1) Initial State Value Estimate denoted by E ( V ( s 0 )) which is a measure of expected long-term returns to evaluate offline RL training as described in Paine et al.

Original: (2020), 2) Local Mean Absolute Action-Value Difference: defined as E ( | ∆ Q
Revised: (2020), 2) Local Mean Absolute Action-Value Difference: defined as E ( | ∆ Q

Original: π orig | )
Revised: π orig | )

Original: =
Revised: =

Original: E ( | Q π orig ( π orig ( s )) − Q π orig ( π j ( s )) | ) that measures how original policy perceives suggestions given by explanation policies, and 3) Action Contrast Measure: a measure of difference in actions suggested by explanation policies and the original action.
Revised: E ( | Q π orig ( π orig ( s )) − Q π orig ( π j ( s )) | ) that measures how original policy perceives suggestions given by explanation policies, and 3) Action Contrast Measure: a measure of difference in actions suggested by explanation policies and the original action.

Original: Here, we use E ( 1 ( π orig ( s ) ̸
Revised: Here, we use E ( 1 ( π orig ( s ) ̸

Original: = π j ( s )) for discrete action space and E (( π orig ( s ) − π j ( s )) 2 ) for continuous action space.
Revised: = π j ( s )) for discrete action space and E (( π orig ( s ) − π j ( s )) 2 ) for continuous action space.

Original: Further, we compute distances between embeddings of original and complementary data sets using Wasserstein metric: W dist ( ¯ d orig , ¯ d j ) , later normalized to [0, 1].
Revised: Further, we compute distances between embeddings of original and complementary data sets using Wasserstein metric: W dist ( ¯ d orig , ¯ d j ) , later normalized to [0, 1].

Original: Finally, the cluster attribution frequency is measured using metric P ( c final = c j ) .
Revised: Finally, the cluster attribution frequency is measured using metric P ( c final = c j ) .

Original: A. G RID - WORLD E NVIRONMENT D ETAILS
Revised: A.3 Grid-world Environment Details

Original: [Figure]
Revised: [Figure]

Original: Lava Wall Action: UP A. A DDTIONAL T RAINING D ETAILS
Revised: A.4 Addtional Training Details

Original: Seaquest Atari Environment – We employed Discrete SAC to train the original policy along with explanation policies, where the training was performed until saturation in the performance.
Revised: Seaquest Atari Environment – We employed Discrete SAC to train the original policy along with explanation policies, where the training was performed until saturation in the performance.

Original: We used the critic learning rate of 3 × 10 − 4 and the actor learning rate of 3 × 10 − 4 with a batch size of 256.
Revised: We used the critic learning rate of 3 × 10 − 4 and the actor learning rate of 3 × 10 − 4 with a batch size of 256.

Original: The trainings were performed parallelly on a single Nvidia-A100 GPU hardware.
Revised: The trainings were performed parallelly on a single Nvidia-A100 GPU hardware.

Original: HalfCheetah MuJoCo Environment – We used SAC to train the original policy as well as explanation policies where we trained the agents until training performance saturated.
Revised: HalfCheetah MuJoCo Environment – We used SAC to train the original policy as well as explanation policies where we trained the agents until training performance saturated.

Original: We again used the critic learning rate of 3 × 10 − 4 and the actor learning rate of 3 × 10 − 4 with a batch size of 512.
Revised: We again used the critic learning rate of 3 × 10 − 4 and the actor learning rate of 3 × 10 − 4 with a batch size of 512.

Original: The policy trainings were performed parallelly on a single Nvidia-A GPU.
Revised: The policy trainings were performed parallelly on a single Nvidia-A GPU.

Original: A. A DDITIONAL T RAJECTORY A TTRIBUTION R ESULTS A.5. T RAJECTORY A TTRIBUTION R ESULTS FOR G RID - WORLD AND H ALF C HEETAH
Revised: A.5 Additional Trajectory Attribution Results A.5.1 Trajectory Attribution Results for Grid-world and HalfCheetah Qualitative

Original: Qualitative Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 4 depicts a grid-world state - (1, 1), corresponding decision by the trained offline RL agent - ‘right’, and attribution trajectories explaining the decision.
Revised: 4 depicts a grid-world state - (1, 1), corresponding decision by the trained offline RL agent - ‘right’ [Figure] and attribution trajectories explaining the decision.

Original: As we can observe, the decision is influenced not only by trajectory (traj.-i) that goes through (1, 1) but also by other distant trajectories(trajs.-ii, iii).
Revised: As we can observe, the decision is influenced not only by trajectory (traj.-i) that goes through (1, 1) but also by other distant trajectories(trajs.-ii, iii).

Original: These examples demonstrate that distant experiences (e.g. traj.-iii) could significantly influence the RL agent’s decisions, deeming trajectory attribution an essential component of future XRL techniques.
Revised: These examples demonstrate that distant experiences (e.g. traj.-iii) could significantly influence the RL agent’s decisions, deeming trajectory attribution an essential component of future XRL techniques.

Original: Further, Fig. 5 shows HalfCheetah observation, the agent suggested in terms of hinge torques, and corresponding attributed trajectories showing runs that influence suggested set of torques.
Revised: Further, Fig. 5 shows HalfCheetah observation, the agent suggested in terms of hinge torques [Figure] and corresponding attributed trajectories showing runs that influence suggested set of torques.

Original: Additional results are given in Sec.
Revised: Additional results are given in Sec.

Original: A.5.
Revised: A.

Original: [Figure] 
Revised: [Figure] [Figure] j e c t o r y ( i )

Revised: T r a j e c t

Revised: o r y ( ii )

Original: [Figure]
Revised: [Figure]

Revised: j e

Revised: [Figure]

Original: Quantitative Results.
Revised: Quantitative Results.

Original: Tables 2 and 3 present quantitative analysis of the proposed trajectory attribution for Grid-world and HalfCheetah respectively.
Revised: Tables 2 and 3 present quantitative analysis of the proposed trajectory attribution for Grid-world and HalfCheetah respectively.

Original: The initial state value estimate for the original policy π orig matches or exceeds estimates for explanation policies trained on different complementary data sets in both environment settings.
Revised: The initial state value estimate for the original policy π orig matches or exceeds estimates for explanation policies trained on different complementary data sets in both environment settings.

Original: This indicates that the original policy, having access to all behaviours, is able to outperform other policies that are trained on data lacking information about important behaviours (e [Figure] g. grid-world: reaching a distant goal, HalfCheetah: stabilizing the frame while taking strides).
Revised: This indicates that the original policy, having access to all behaviours, is able to outperform other policies that are trained on data lacking information about important behaviours (e.g. grid-world: reaching a distant goal, HalfCheetah: stabilizing the frame while taking strides).

Original: Furthermore, local mean absolute action-value difference and action differences turn out to be highly correlated (Tab. 1 and 3), i.e., the explanation policies that suggest the most contrasting actions are usually perceived by the original policy as low-return actions.
Revised: Furthermore, local mean absolute action-value difference and action differences turn out to be highly correlated (Tab. 1 and 3), i.e., the explanation policies that suggest the most contrasting actions are usually perceived by the original policy as low-return actions.

Original: This evidence supports the proposed trajectory algorithm as we want to identify the behaviours which when removed make agent choose actions that are not considered suitable originally.
Revised: This evidence supports the proposed trajectory algorithm as we want to identify the behaviours which when removed make agent choose actions that are not considered suitable originally.

Original: In addition, we provide the distances between the data embeddings in the penultimate column.
Revised: In addition, we provide the distances between the data embeddings in the penultimate column.

Original: The cluster attribution distribution is represented in the last column which depicts how RL decisions are dependent on various behaviour clusters.
Revised: The cluster attribution distribution is represented in the last column which depicts how RL decisions are dependent on various behaviour clusters.

Original: Interestingly, in the case of grid-world, we found that only a few clusters containing information about reaching goals and avoiding lava had the most significant effect on the original RL policy.
Revised: Interestingly, in the case of grid-world, we found that only a few clusters containing information about reaching goals and avoiding lava had the most significant effect on the original RL policy.

Original: [Table]
Revised: [Table]

Original: V ( s 0 )) , better is the trained policy.

Original: High E ( | ∆ Q π orig | )) along with high E (( π orig ( s ) − π j ( s )) 2 ) is desirable.

Original: The policies with lower W dist ( ¯ d, ¯ d j ) and high action contrast are given priority while attribution.

Original: The cluster attribution distribution is given in the final column.

Original: [Table]
Revised: [Table]

Original: A.5. A DDITIONAL Q UALITATIVE R ESULTS
Revised: A.5.2 Additional Qualitative Results

Original: [Figure]
Revised: [Figure]

Original: A. C LUSTERING A NALYSIS
Revised: A.6 Clustering Analysis

Original: [Figure]
Revised: [Figure]

Revised: c)

Revised: We find these clusters represent semantically meaningful high-level behaviours

Original: We observe that the trajectory embeddings obtained from the sequence encoders when clustered together 7 demonstrate characteristic high-level behaviours.
Revised: We observe that the trajectory embeddings obtained from the sequence encoders when clustered together 7 demonstrate characteristic high-level behaviours.

Original: For instance, in the case of grid-world (Refer 8), the clusters comprise semantically similar trajectories where the agent demonstrates behaviours such as ‘ falling into the lava ’, ‘ achieving the goal in the first quadrant ’, ‘ mid-grid journey to the goal ’, etc.
Revised: For instance, in the case of grid-world (Refer 8), the clusters comprise semantically similar trajectories where the agent demonstrates behaviours such as ‘ falling into the lava ’, ‘ achieving the goal in the first quadrant ’, ‘ mid-grid journey to the goal ’, etc.

Original: For Seaquest 9, we obtain trajectory clusters that represent high-level behaviours such as ‘ filling in oxygen ’, ‘ fighting along the surface ’, ‘ submarine bursting due to collision with enemy ’, etc. and for HalfCheetah in Fig.
Revised: For Seaquest 9, we obtain trajectory clusters that represent high-level behaviours such as ‘ filling in oxygen ’, ‘ fighting along the surface ’, ‘ submarine bursting due to collision with enemy ’, etc. and for HalfCheetah in Fig.

Original: 10, we obtain trajectory clusters that represent high-level actions such as ‘ taking long forward strides ’, ‘ jumping on hind leg ’, ‘ running with head down ’, etc.
Revised: 10, we obtain trajectory clusters that represent high-level actions such as ‘ taking long forward strides ’, ‘ jumping on hind leg ’, ‘ running with head down ’, etc.

Original: Although these results look quite promising, in this work we mainly focus on trajectory attribution that leverages these findings.
Revised: Although these results look quite promising, in this work we mainly focus on trajectory attribution that leverages these findings.

Original: In future, we wish to analyse the trajectory embeddings and the behaviour patterns in greater detail.
Revised: In future, we wish to analyse the trajectory embeddings and the behaviour patterns in greater detail.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]