Original: Eﬃcient Bayesian Inverse Reinforcement Learning via Conditional Kernel Density Estimation
Revised: Eﬃcient Bayesian Inverse Reinforcement Learning via Conditional Kernel Density Estimation

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: Inverse reinforcement learning (IRL) methods attempt to recover the reward function of an agent by observing its behavior.
Revised: Inverse reinforcement learning (IRL) methods attempt to recover the reward function of an agent by observing its behavior.

Original: Given the large amount of uncertainty in the underlying reward function, it is often useful to model this function probabilistically, rather than estimate a single reward function.
Revised: Given the large amount of uncertainty in the underlying reward function, it is often useful to model this function probabilistically, rather than estimate a single reward function.

Original: However, existing Bayesian approaches to IRL use a
Revised: However, existing Bayesian approaches to IRL use a

Original: Q-value function to approximate the likelihood, leading to a computationally intractable and inﬂexible framework.
Revised: Q-value function to approximate the likelihood, leading to a computationally intractable and inﬂexible framework.

Original: Here, we introduce kernel density Bayesian IRL (KD-BIRL), a method that uses kernel density estimation to approximate the likelihood.
Revised: Here, we introduce kernel density Bayesian IRL (KD-BIRL), a method that uses kernel density estimation to approximate the likelihood, or the probability of the observed states and actions given a reward function.

Original: This lends itself to an eﬃcient posterior inference for the reward function given a sequence of agent observations.
Revised: This approximation allows for eﬃcient posterior inference of the reward function given a sequence of agent observations.

Original: Using both linear and nonlinear reward functions in a Gridworld environment, we demonstrate that the KD-BIRL posterior centers around the true reward function, and that our method is more eﬃcient than existing approaches.
Revised: Empirically, using both linear and nonlinear reward functions in a Gridworld environment, we demonstrate that the KD-BIRL posterior centers around the true reward function.

Original: 1. Introduction
Revised: 1. Introduction

Original: Reinforcement learning (RL) methods ﬁnd policies and sequences of actions that maximize an agent’s long-term expected reward in a Markov decision process (MDP).
Revised: Reinforcement learning (RL) methods ﬁnd policies and sequences of actions that maximize an agent’s long-term expected reward in a Markov decision process (MDP).

Original: However, in many oﬀ-policy, observational data settings, we observe a sequence of states and actions for an agent who is carrying out a policy based on a reward function that is unknown to the observer.
Revised: However, in many oﬀ-policy, observational data settings, we observe a sequence of states and actions for an agent who is carrying out a policy based on a reward function that is unknown to the observer.

Original: In these cases, it is of interest to infer the reward function that the agent is using in order to understand the factors driving certain behavioral patterns.
Revised: In these cases, it is of interest to infer the reward function that the agent is using in order to understand the factors driving certain behavioral patterns.

Original: For example, in a hospital setting, we may observe the treatment schedule  a patientreceived along with measurements of the patient’s health state,which can be represented as  physiological covariates and clinical interventions.
Revised: For example, in a hospital setting, we may observe the treatment schedule for a patient along with measurements of the patient’s health state, as represented by physiological covariates and clinical interventions.

Original: In this case, we may be interested in understanding the doctor’s reward function—which is typically complex and mostly unobserved—and how this function drives certain treatment decisions based on a patient’s state.
Revised: In this case, we may be interested in understanding the doctor’s reward function—which is typically complex and mostly unobserved—and how this function drives certain treatment decisions based on a patient’s state.

Original: For this purpose, inverse reinforcement learning (IRL) methods aim to recover the reward function (i.e., objectives or priorities) of an agent given observations of the agent’s behavior.
Revised: For this purpose, inverse reinforcement learning (IRL) methods aim to recover the reward function (i.e., objectives or priorities) of an agent given observations of the agent’s behavior.

Original: The earliest IRL algorithms focused on estimating a single reward function (a point estimate) that best explained the observed behavior (Abbeel and Ng, 2004; Ng and Russell, 2000).
Revised: Early IRL algorithms focused on estimating a single reward function (a point estimate) that best explained the observed behavior (Abbeel and Ng, 2004; Ng and Russell, 2000).

Original: These IRL frameworks led to applications in path planning (Mombaur et al., 2010), urban navigation (urb), and robotics settings such as quadruped locomotion (Ratliﬀ et al., 2006a; Kolter et al., 2008).
Revised: These IRL frameworks led to applications in path planning (Mombaur et al., 2010), urban navigation (Ziebart et al., 2008), and robotics settings, such as quadruped locomotion (Ratliﬀ et al., 2006a; Kolter et al., 2008).

Original: IRL is also broadly useful to generate better RL policies; if we can  infer the reward function from a set of desired behavior demonstrations, we can then ﬁt new policies using this reward function, which may lead to better outcomes.
Revised: Additionally, a challenge in RL is designing reward functions for achieving desired behavior patterns; IRL can be used in this setting to infer the reward function from a set of desired behavior demonstrations, which can then be used to ﬁt improved RL policies.

Original: Despite the success ofthese early IRL approaches, there are major limitations to estimating a single reward function, and it is often impossible to recover the exact reward function of an agent for a number of reasons.
Revised: Despite the success of early IRL approaches, there are major limitations to point estimation in this setting.

Original: First, the IRL problem is likely non-identiﬁable (urb; Ziebart et al.,  2009; Ratliﬀ et al., 2006b; Abbeel and Ng, 2004).
Revised: First, the IRL problem is often non-identiﬁable (Ziebart et al., 2008, 2009; Ratliﬀ et al., 2006b; Abbeel and Ng, 2004).

Original: That is, there may be many reward functions that explain a set of behaviors in an agent equally well.
Revised: That is, there are multiple (and possibly inﬁnite) reward functions that explain a set of behaviors in an agent equally well.

Original: Second, if the training data are insuﬃcient to identify the reward function, point estimatesgenerally fail to capture the uncertainty  in these estimates.
Revised: Second, for ﬁnite training datasets, point estimates fail to capture the uncertainty and noise in the data generating process.

Original: Thus, it is advantageous to take a Bayesian approach and expressour uncertainty through a posterior distribution over the estimated reward function (Ramachandran and Amir, 2007; Balakrishnan et al., 2020; Chan and van der Schaar, 2021; Michini and How, 2012a,b; Choi and Kim, 2012).
Revised: Thus, it is advantageous to take a Bayesian approach and express uncertainty through a posterior distribution over the estimated reward function (Ramachandran and Amir, 2007; Balakrishnan et al., 2020; Chan and van der Schaar, 2021; Michini and How, 2012a,b; Choi and Kim, 2012).

Original: A Bayesian approach communicates a degree of conﬁdence that relies on the dataset distribution, and the posterior will ideally place mass on all regions of the reward function space that could explain the observed behavior.
Revised: A Bayesian approach communicates a degree of conﬁdence that relies on the dataset distribution, placing mass on all regions of the reward function space that could explain the observed behavior.

Original: Despite advances inthis area, existing Bayesian IRL methods tend to be computationally demanding.
Revised: Despite advances in Bayesian IRL, existing methods are computationally demanding.

Original: This is because, in sampling-based inference for the posterior,  each reward function must be evaluated using forward RL by ﬁtting a policy using that sampled function.
Revised: This is because, in sampling-based inference for the posterior, these methods require value iteration for each instance of a reward function.

Original: A single instance of forward RL is already computationally demanding, so requiring it for each sampled reward function compounds the problem.
Revised: A single instance of value iteration is in itself computationally demanding, so repeating it for each sampled reward function compounds the problem.

Original: To address this issue, we introduce kernel density Bayesian inverse reinforcement learning (KD-BIRL), an eﬃcient IRL framework that eliminates the need to ﬁt a new policy for each sampled reward function.
Revised: To address this issue, we introduce kernel density Bayesian inverse reinforcement learning (KD-BIRL), an eﬃcient IRL framework that eliminates the need to ﬁt a new policy for each sampled reward function.

Original: Our contributions to the IRL literature are as follows: (1) KD-BIRL calculates the likelihood using a conditional kernel density estimation that does not require forward RL to be run at inference time; (2) We show that KD-BIRL isfar more computationally feasible than related approaches; (3) We demonstrate that KD-BIRL returns accurate posterior estimates with both linear and nonlinear reward functions.
Revised: Our contributions to the IRL literature are as follows: (1) We introduce KD-BIRL, an IRL method that calculates the likelihood using a conditional kernel density estimation that does not require value iteration at inference time; (2) We show that KD-BIRL is more computationally feasible than related approaches; (3) We demonstrate that KD-BIRL returns accurate posterior estimates with both linear and nonlinear reward functions.

Original: 2. Related Work
Revised: 2. Related Work

Original: In the ﬁrst paper to propose Bayesian IRL (Ramachandran and Amir, 2007), the authors use a Q -value function to calculate the likelihood of seeing a given state and action pair ( s, a ) given a reward function R .
Revised: The ﬁrst work exploring Bayesian IRL (Ramachandran and Amir, 2007) used a Q -value function to calculate the likelihood of seeing a given state and action pair ( s, a ) given a reward function R .

Original: This method isgenerally unsuitable for complex state spaces because it requires us to repeatedly solve an environment’s
Revised: This method is unsuitable for complex state spaces because it requires us to repeatedly solve an environment’s

Original: MDP using forward RL. In particular, the algorithm uses Markov chain Monte Carlo (MCMC) sampling to compute the Gibbs posterior; every iteration requires a likelihood calculation, and each of these calculations require that forward RL is run to reﬁt the Q -value function to the current reward function.
Revised: MDP using value iteration to learn a Q -value function. The proposed inference algorithm uses Markov chain Monte Carlo (MCMC) sampling to compute a Gibbs posterior. Every sampling iteration requires running value iteration to estimate the Q -value function for the sampled reward function.

Original: This  is computationally prohibitive, especially with inﬁnite or real-valued state spaces.
Revised: This step is computationally prohibitive, especially with inﬁnite or real-valued state spaces.

Original: To address this issue, it is necessary to either formulate a diﬀerent likelihood estimator or minimize the number of calls to forward RL.
Revised: To address this issue, it is necessary to either formulate a diﬀerent likelihood estimator or minimize the number of times value iteration is performed.

Original: Recent work proposes anew likelihood calculation that relies on human-recorded pairwise preferences over the demonstrations (Brown and Niekum, 2019).
Revised: Recently, a likelihood approximation was proposed that relies on human-recorded pairwise preferences over the demonstrations (Brown and Niekum, 2019).

Original: While this greatly reduces the computational complexity ofthe earlier methods,it relies on having these  preferences available.
Revised: While this greatly reduces the computational complexity of earlier methods, these human-recorded preferences are not available in most settings.

Original: The preferences may not be known, especially when the state spaces are large and the number available demonstrations are small.

Original: Another approach samples more appropriate reward functions using variational inference (Chan and van der Schaar, 2021); this approach requires far fewer calls to forward RL, making it more appropriate for high-dimensional environments.
Revised: Another line of work has proposed approximating the exact reward function posterior using variational inference (Chan and van der Schaar, 2021); this approach requires far fewer instances of value iteration, making it more appropriate for high-dimensional environments.

Original: However, it still relies on a Q -value function estimate to compute the  observed data likelihood, and this is computationally expensive.
Revised: However, it still relies on an expensive Q -value function estimate to compute the likelihood of the observed data.

Original: The use of a Q -value function implies that forward RL must be run every time the likelihood of a reward function is evaluated, which is time consuming for even moderately complex environments.

Original: 3. Kernel density Bayesian inverse reinforcement learning
Revised: 3. Background: Inverse reinforcement learning

Original: Inthis section we present the kernel density Bayesian inverse reinforcement learning (KDBIRL) algorithm. We ﬁrst discuss the setup of  the IRL problem, and then describe the likelihood calculation and how to estimate the reward function posterior distribution.
Revised: In the IRL setting, the data we wish to model are a set of expert demonstrations, { ( s i , a i ) } ni =1 , where the demonstration at time i is a 2-tuple ( s i , a i ) representing the agent’s state and chosen action.

Original: The input to  an IRL algorithm is a set of expert demonstrations of an optimal policy for a ﬁxed but unspeciﬁed reward function. Given these demonstrations, the goal is to recover the real-valued reward function R : S → R , where  S is the state space. Each sample in a demonstration at time i is a 2-tuple ( s i , a i ) representing the agent’s state and chosen action.
Revised: These demonstrations are assumed to arise from an agent acting according to an optimal policy, π (cid:63) : S → A , for a ﬁxed but unknown reward function R : S → R , where A is the action space and S is the state space. Given these demonstrations, the IRL objective is to recover R .

Original: Note that rewards are not available explicitly in these demonstrations. Given demonstrations of an optimal policy with respect to an unknown reward function π (cid:63) : S → A , where A is the action space, IRL seeks to characterize that reward function R such that π (cid:63) is optimal. We can derive π (cid:63) as follows.
Revised: Speciﬁcally, IRL seeks the set of reward functions such that π (cid:63) is optimal. We now formalize this problem.

Original: Suppose that S ⊆ R p , and let s ∈ S denote a state vector.
Revised: Suppose that S ⊆ R p , and let s ∈ S denote a state vector.

Original: Let R denote the space of reward functions.
Revised: Let R denote the space of reward functions.

Original: The value function for a given policy π is V π = E
Revised: The value function for a given policy π is V π = E

Original: [ (cid:80) ∞ t =

Revised: [ (cid:80) ∞ t =

Original: γ t R ( s t ) | b 0 , π ] where b 0 is the probability of starting in state s t .
Revised: γ t R ( s t ) | b 0 , π ] where b 0 is the probability of starting in state s t , and γ ∈ (0 , 1) is a discount factor.

Original: Without loss of generality, assume that the optimal policy is given by π (cid:63) ( s )
Revised: Without loss of generality, assume that the optimal policy is given by π (cid:63) ( s )

Original: = a (cid:63) , where a (cid:63) is the optimal action to be taken at state s .
Revised: = a (cid:63) , where a (cid:63) is the optimal action to be taken at state s .

Original: In order to identify a reward function R for which π is optimal, it must be true that 
Revised: In order to identify a reward function R for which π is optimal, it must be true that E s (cid:48) ∼ P sa(cid:63) [ V

Original: E s (cid:48) ∼ P sa(cid:63)
Revised: [Equation] ( s [Equation]

Original: [ V π ( s (cid:48) )] ≥ E s (cid:48) ∼ P sa
Revised: )] ≥ E [Equation] ∼ [Equation]

Original:  [ V π ( s (cid:48) )], for all states s ∈ S and all actions a ∈ A \ a (cid:63) , where P sa  is the state transition probability.
Revised: sa [ V [Equation] ( s (cid:48) )], for all states s ∈ S and all actions a ∈ A \ a (cid:63) , where P sa = p ( s (cid:48) | s, a ) is the state transition probability.

Original: In other words, the long-term expected value of the optimal policy under the reward must be equal to or greater than the long-term expected value of selecting an action that deviates from the optimal policy.
Revised: In other words, the long-term expected value of the optimal policy under the reward must be equal to or greater than the long-term expected value of selecting an action that deviates from the optimal policy.

Original: Generalizing beyond linear functions, our goalhere is to infer a posterior distribution over an expert reward function R from n demonstrations of the expert policy { ( s ei , a ei ) } ni =1 . 
Revised: Taking a Bayesian approach to IRL, our goal is to infer the posterior distribution over the reward function R given n demonstrations of the expert policy { ( s ei , a ei ) } ni =1 . We call this dataset the set of expert demonstrations .

Original: The posterior density is proportional to the product of the prior distribution  p ( R ) on the rewards and the likelihood of the  demonstrations given the reward:
Revised: The posterior density is proportional to the product of the prior distribution on the rewards, p ( R ), and the likelihood of the expert demonstrations given the reward:

Original: [Equation]
Revised: [Equation]

Original: In the original Bayesian IRL framework (Ramachandran and Amir, 2007), the form of the “likelihood” is
Revised: In the original Bayesian IRL framework (Ramachandran and Amir, 2007), the form of the “likelihood” is

Original: [Equation]
Revised: [Equation]

Original: where Q (cid:63) ( s i , a i , R ) is the optimal Q -value function computed with reward R and α >is a temperature parameter.
Revised: where Q (cid:63) ( s i , a i , R ) is the optimal Q -value function computed with reward R and α > 0 is an inverse temperature parameter.

Original: In fact, Equation (2) is a loss-based function instead of a likelihood, and the resulting posterior is the Gibbs posterior instead of a true posterior.
Revised: Equation (2) is a loss-based function instead of a likelihood, and the resulting posterior is the Gibbs posterior instead of a true posterior (Shawe-Taylor and Williamson, 1997).

Original: Additionally, the likelihood above depends on the Q -value function, which is expensive to recompute for each estimate of the reward function.
Revised: Additionally, the likelihood above depends on the Q -value function, which is expensive to recompute for each sampled reward function.

Original: We propose  asimple nonparametricmethod, a conditional kernel density estimator (CKDE) based on Gaussian kernels.
Revised: We propose kernel density Bayesian inverse reinforcement learning (KD-BIRL), which uses a nonparametric conditional kernel density estimator (CKDE) with Gaussian kernels to estimate the likelihood in the IRL posterior (Equation 1).

Original: CKDE has been well-studied, with known asymptotic behaviors (van der Vaart, 2000).
Revised: CKDEs have been well-studied, with known asymptotic behaviors (van der Vaart, 2000).

Revised: To ﬁt the KDE, we propose sampling another set of demonstrations to augment the observed expert demonstrations.

Original: This calculation requires m demonstration samples from agents that follow diﬀerent reward functions.
Revised: Speciﬁcally, we sample m demonstrations from agents whose behaviors are optimal for reward functions distinct from the expert reward function.

Original: The training dataset  consists of 3-tuples 
Revised: This dataset, which we call the training dataset , consists of 3-tuples { ( s j , a j , R j ) } mj

Original: { ( s j , a j , R j ) } mj =1 , where R j is the reward functions generating ( s j , a j ).
Revised: =1 , where R j is the reward functions generating ( s j , a j ).

Original: Given these samples, the conditional density  given the reward  R is
Revised: Using the training dataset, the conditional density for a state-action pair ( s, a ) given a reward function R is

Original: [Equation]
Revised: [Equation]

Original: where d s : ( S×A ) × ( S×A ) → R is a distance metric to compare ( s, a ) tuples, d r : R×R → R is a distance metric to compare reward functions, and h, h (cid:48) > 0 are smoothing hyperparameters.
Revised: where d s : ( S × A ) × ( S × A ) → R is a distance metric to compare ( s, a ) tuples, d r : R × R → R is a distance metric to compare reward functions, and h, h (cid:48) > 0 are smoothing hyperparameters.

Original: We can now estimate  p ( s ei , a ei | R ) by (cid:98) p n ( s ei , a ei | R ), where ( s ei , a ei ) are samples in the  expert trajectories:
Revised: We can now estimate the likelihood p ( s ei , a ei | R ) by (cid:98) p n ( s ei , a ei | R ), where ( s ei , a ei ) is a single sample from the dataset of expert demonstrations:

Original: [Equation]
Revised: [Equation]

Original: The estimated posterior is thencalculated as
Revised: The estimated posterior is then

Original: [Equation]

Original: [Equation]

Original: h

Original: [Equation]
Revised: [Equation]

Original: j

Original: [Equation]

Original: ) (cid:80) [Equation] Note that once the conditional density (cid:98) p m is obtained, it is not necessary to run forward RL to evaluate the posterior at any reward  R .
Revised: Importantly, note that once the conditional density (cid:98) p m is obtained, it is not necessary to perform value iteration to evaluate the posterior for a given reward function R .

Original: This eliminates the vast majority of the computational complexity of existing BIRL algorithms and allows us to easily apply this algorithm to complex environments with large state spaces.
Revised: This drastically reduces the computational complexity compared to existing BIRL algorithms and opens the door for performing Bayesian IRL in complex environments with large state spaces.

Original: Additionally, as m → ∞ , (cid:98) p m converges to the true conditional density and the posterior converges to the true posterior (van der Vaart, 2000).
Revised: Additionally, as m → ∞ , (cid:98) p m converges to the true likelihood and the posterior converges to the true posterior (van der Vaart, 2000).

Original: 4. Experiments
Revised: 5. Experiments

Original: In this section, we perform a series of experiments to test the accuracy and robustness of this approach and compare it to AVRIL (Chan and van der Schaar, 2021), a competing method.
Revised: In this section, we perform a series of experiments to test the accuracy and robustness of KD-BIRL and compare it to AVRIL (Chan and van der Schaar, 2021), a competing method.

Original: We perform these experiments in a modiﬁed Gridworld environment, with  uniform prior on the reward function.
Revised: We perform these experiments in a modiﬁed Gridworld environment with a linear reward function, and we specify a uniform prior on the reward function parameters.

Original: Gridworld has a two-dimensional discrete state space that lies on an (cid:96) × (cid:96) grid.
Revised: Gridworld has a two-dimensional discrete state space made up of tiles on an (cid:96) × (cid:96) grid.

Original: Speciﬁcally, the MDP is: S =
Revised: The MDP is: S =

Original: [ (cid:96) ]
Revised: [ (cid:96) ]

Original: × [ (cid:96) ], A = { [0 , 0] , [0 , 1] , [1 , 0] , [ − 1 , 0] , [0 , − 1] } , R : S → R , where we choose a grid size of (cid:96) = 4, and the actions correspond to not moving ([0 , 0]), going up ([0 , 1]),  right ([1 , 0]),  left ([ − 1 , 0]), and  down ([0 , − 1]). 
Revised: × [ (cid:96) ], A = { [0 , 0] , [0 , 1] , [1 , 0] , [ − 1 , 0] , [0 , − 1] } , R : S → R , where we choose a grid size of (cid:96) = 4, and the actions correspond to not moving ([0 , 0]), moving up ([0 , 1]), moving right ([1 , 0]), moving left ([ − 1 , 0]), and moving down ([0 , − 1]). A grid contains a target state (or [Figure]

Original: Each grid contains a target state, and the reward  depends on the coordinates of the state, with the maximum reward received at the target state.
Revised: goal state). The reward observed in a state depends on the coordinates of the state, with the maximum reward received at the target state.

Original: Our method requires that we deﬁne two distance metrics: one associated with the distance between reward functions ( d r ), and one associated with the distance between state-action pairs ( d s ).
Revised: KD-BIRL requires that we deﬁne two distance metrics: one associated with the distance between reward functions ( d r ), and one associated with the distance between state-action pairs ( d s ).

Original: In this setting, for d r , we use the cosine distance for linear rewards and the Euclidean distance for nonlinear rewards, and for d s , we use the Euclidean distance.
Revised: In this setting, for d r , we use the cosine distance for linear rewards and the Euclidean distance for nonlinear rewards, and for d s , we use the Euclidean distance.

Original: We choose n = 30, m = 1000, h = 0 .
Revised: We choose n = 30, m = 1000, h = 0 .

Original: 03, h (cid:48)
Revised: 03, h (cid:48)

Original: = 0 .
Revised: = 0 .

Original: 03, and k is between 4 and 8.
Revised: 03, and (cid:96) is between 4 and 8.

Original: We ﬁrst validate that our algorithm is able to recover the posterior distribution of a simple linear reward function R ( s )
Revised: We ﬁrst validate that our algorithm is able to recover the posterior distribution of a simple linear reward function R ( s )

Original: =
Revised: =

Original: β (cid:62) s , wheres ∈ S and β ∈ R 2 .
Revised: β (cid:62) s , where β ∈ R 2 is a coeﬃcient vector.

Original: We show results for  twosets of linear reward function parameters: β = [ − 1 , 1] and β = [1 , 1] (Figure 1).
Revised: We ﬁt KDBIRL for expert demonstrations generated from two linear reward functions: β = [ − 1 , 1] and β = [1 , 1] (Figure 1).

Original: In these results, we also visualize the demonstration density, a measure of the relative state[Figure] [Figure] occupancy of the expert demonstrations.
Revised: In these results, we also visualize the demonstration density, a measure of the relative state occupancy of the expert demonstrations.

Original: We also visualize the posterior distribution  by evaluating it on a grid of reward parameters to understand how uncertain our estimates are.
Revised: We also visualize the posterior distribution over reward functions by evaluating the posterior (Equation 3) on a grid of reward parameters to understand how uncertain our estimates are, and whether their mass tends to be situated around the functions used to generate the data.

Original: We ﬁnd that KD-BIRL is conﬁdent about reward parameters that most eﬀectively characterize the expert demonstrations and its posterior centers around the true reward function (Figure 2).
Revised: We ﬁnd that KD-BIRL is conﬁdent about reward parameters that most eﬀectively characterize the expert demonstrations, and its posterior centers around the true reward function (Figure 2).

Original: Finally, we consider non-linear reward functions.
Revised: Finally, we consider non-linear reward functions.

Original: Here, we use a 
Revised: Here, we use a function where the reward is 0 in all states except the target state, which receives a reward of 1 (Figure 3). In this experiment, we specify the set of possible reward functions to be fully nonparametic, [Figure]

Original: [Figure]
Revised: [Figure]

Original: function where the reward is 0 in all states except the target state, which receives a reward of 1 (Figure 3).
Revised: and allow each of the 16 possible states to have its own estimated scalar reward.

Original: We also compare our estimate to that of AVRIL (Chan and van der Schaar, 2021). 
Revised: We also compare our estimate to that of AVRIL (Chan and van der Schaar, 2021). We ﬁnd that

Revised: KD-BIRL’s posterior has a sharp peak at the target state, and much lower mass elsewhere.

Original: Our experiments suggest that KD-BIRL can recover posterior estimates over linear and non-linear reward functions in a Gridworld environment.
Revised: Collectively, our experiments suggest that KD-BIRL can recover posterior estimates over linear and non-linear reward functions in a Gridworld environment.

Original: We ﬁnd that while the AVRIL algorithm is more computationally eﬃcient than earlier approaches, it can still return inaccurate posterior estimates (Figure 3).
Revised: We ﬁnd that while the AVRIL algorithm is more computationally eﬃcient than earlier approaches, it can still return inaccurate posterior estimates (Figure 3).

Original: 5. Discussion and future directions
Revised: 6. Discussion and future directions

Original: In this work, we introduce  a novel approach toestimate conditional densities for the purpose of inverse reinforcement learning.
Revised: In this work, we introduce KD-BIRL, a novel approach to inverse reinforcement learning using conditional kernel density estimators.

Original: We demonstrate that this approach to Bayesian IRL removes much of the computational complexity associated with similar algorithms and does not need to  solve an MDP using forward RL to calculate the likelihood of a given set of reward parameters.
Revised: We demonstrate that this approach to Bayesian IRL removes much of the computational complexity associated with similar algorithms and does not need to repeatedly solve an MDP using value iteration to calculate the likelihood of a given set of reward parameters.

Original: Several future directions remain.
Revised: Several future directions remain.

Original: This work is currently adapted best for on-policy (i.e., simulation) environments, and we believe that additional work must be done to apply it on oﬀ-policy environments such as  clinical  settings.
Revised: This work is currently well-suited for on-policy (i.e., simulation) environments, and we believe that additional work must be done to apply it on oﬀ-policy environments such as medical or clinical decision making settings.

Original: The particular choice of distance metrics used in the conditional density estimate is also likely to depend on the environment and reward function type; we pick metrics that are most suitable to the Gridworld environment in this paper, and we believe that additional experimentation must be done to adapt this to diﬀerent environments.
Revised: The particular choice of distance metrics used in the conditional density estimate is also likely to depend on the environment and reward function type; we pick metrics that are most suitable to the Gridworld environment in this paper, and we believe that additional experimentation must be done to adapt this to diﬀerent environments.

Original: The same is true for the hyperparameters h and h (cid:48) ; we currently use rule-of-thumb hyperparameters (Silverman, 1986), but additional work is needed to determine if this is reasonable.
Revised: The same is true for the hyperparameters h and h (cid:48) ; we currently use rule-of-thumb hyperparameters (Silverman, 1986), but additional work could further optimize these values.

Original: Furthermore, our current calculation of the conditional density is most suitable for environments with a low-dimensional state space.
Revised: Furthermore, our current calculation of the conditional density is most suitable for environments with a low-dimensional state space.

Original: To extend this to a higher dimensional state space, like in clinical environments, we anticipate using a Gaussian process could be beneﬁcial. 
Revised: To extend this to a higher dimensional state space, like in clinical environments, we anticipate using a Gaussian process could be beneﬁcial. 7. Acknowledgements

Original: Finally, additional work must be done to develop methods to evaluate this algorithm in oﬀ-policy environments, where the true reward function is unknown.
Revised: This work was funded by the Helmsley Trust grant AWD1006624, NIH NCI 5U2CCA233195, NIH NHLBI R01 HL133218, and NSF CAREER AWD1005627. BEE is on the SAB of Creyon Bio, Arrepath, and Freenome.

Revised: D. Cai was supported in part by a Google Ph.D.

Revised: Fellowship in Machine Learning.