Original: B AYESIAN Q- LEARNING W ITH I MPERFECT E XPERT D EMONSTRATIONS
Revised: B AYESIAN Q- LEARNING W ITH I MPERFECT E XPERT D EMONSTRATIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Guided exploration with expert demonstrations improves data efficiency for reinforcement learning, but current algorithms often overuse expert information.
Revised: Guided exploration with expert demonstrations improves data efficiency for reinforcement learning, but current algorithms often overuse expert information.

Original: We propose a novel algorithm to speed up Q-learning with the help of a limited amount of imperfect expert demonstrations.
Revised: We propose a novel algorithm to speed up Q-learning with the help of a limited amount of imperfect expert demonstrations.

Original: The algorithm is based on a Bayesian framework to model suboptimal expert actions and derives Q-values’ update rules by maximizing the posterior probability.
Revised: The algorithm is based on a Bayesian framework to model suboptimal expert actions and derives Q-values’ update rules by maximizing the posterior probability.

Original: It weighs expert information by the uncertainty of learnt Q-values and avoids excessive reliance on expert data, gradually reducing the usage of uninformative expert data.
Revised: It weighs expert information by the uncertainty of learnt Q-values and avoids excessive reliance on expert data, gradually reducing the usage of uninformative expert data.

Original: Experimentally, we evaluate our approach on a sparse-reward chain environment and six more complicated Atari games with delayed rewards.
Revised: Experimentally, we evaluate our approach on a sparse-reward chain environment and six more complicated Atari games with delayed rewards.

Original: With the proposed methods, we can achieve better results than Deep Q-learning from Demonstrations (Hester et al., 2017) in most environments.
Revised: With the proposed methods, we can achieve better results than Deep Q-learning from Demonstrations (Hester et al., 2017) in most environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) trains an agent to maximize expected cumulative rewards through online interactions with an environment (Sutton & Barto, 2018).
Revised: Reinforcement learning (RL) trains an agent to maximize expected cumulative rewards through online interactions with an environment (Sutton & Barto, 2018).

Original: To speed up learning, online RL can be combined with offline expert demonstrations (Hester et al., 2018), which guides the agent toward high-rewarding behaviours and thus improves data efficiency.
Revised: To speed up learning, online RL can be combined with offline expert demonstrations (Hester et al., 2018), which guides the agent toward high-rewarding behaviours and thus improves data efficiency.

Original: Most existing expert demonstration methods (Brys et al., 2015; Abbeel & Ng, 2004) either clone behaviours from the expert or bonus expert actions.
Revised: Most existing expert demonstration methods (Brys et al., 2015; Abbeel & Ng, 2004) either clone behaviours from the expert or bonus expert actions.

Original: However, agents are provided with imperfect expert data and gain distracting guidance.
Revised: However, agents are provided with imperfect expert data and gain distracting guidance.

Original: Some works start handling the imperfect expert data but still lack a solid theoretical foundation (Nair et al., 2018; Zhang et al., 2022).
Revised: Some works start handling the imperfect expert data but still lack a solid theoretical foundation (Nair et al., 2018; Zhang et al., 2022).

Original: Our paper adopts Bayesian frameworks (Ghavamzadeh et al., 2015; Dearden et al., 1998), which provide an analytical method to incorporate extra sub-optimal expert information for learning.
Revised: Our paper adopts Bayesian frameworks (Ghavamzadeh et al., 2015; Dearden et al., 1998) which provide an analytical method to incorporate extra sub-optimal expert information for learning.

Original: In the probabilistic model for Bayesian inference, the sub-optimal expert decision is assumed to rely on the
Revised: In the probabilistic model for Bayesian inference, the sub-optimal expert decision is assumed to rely on the

Original: Boltzmann distribution dependent on the optimal expected returns from state-action pairs, called the optimal Q-values.
Revised: Boltzmann distribution dependent on the optimal expected returns from state-action pairs, called the optimal Q-values.

Original: Intuitively, our paper considers that the expert would prefer actions with higher optimal Q-values.
Revised: Intuitively, our paper considers that the expert would prefer actions with higher optimal Q-values.

Original: Based on this relationship, the agent can infer values of the optimal Q-values from the given expert data during online learning and correct estimated Q-values to better correspond to the expert behaviours.
Revised: Based on this relationship, the agent can infer values of the optimal Q-values from the given expert data during online learning and correct estimated Q-values to better correspond to the expert behaviours.

Original: This inference is equivalent to maximizing the posterior distribution of
Revised: This inference is equivalent to maximizing the posterior distribution of

Original: Q-values conditioned on expert data, but computing the maximum of a posterior probability is a difficult task (Hoffman et al., 2013; Diaconis & Ylvisaker, 1979; West et al., 1985).
Revised: Q-values conditioned on expert data, but computing the maximum of a posterior probability is a difficult task (Hoffman et al., 2013; Diaconis & Ylvisaker, 1979; West et al., 1985).

Original: Therefore, our paper proposes to utilize the generalized extended Kalman filter (GEKF) (Fahrmeir, 1992) to derive a posterior maximum, requiring fewer restrictions on Q-values functions than other Bayesian RL frameworks (Dearden et al., 1998; Engel et al., 2003; Osband et al., 2019).
Revised: Therefore, our paper proposes to utilize the generalized extended Kalman filter (GEKF) (Fahrmeir, 1992) to derive a posterior maximum, requiring fewer restrictions on Q-values functions than other Bayesian RL frameworks (Dearden et al., 1998; Engel et al., 2003; Osband et al., 2019).

Original: Under the assumptions from the GEKF framework, we derive an iterative forward way to compute
Revised: Under the assumptions from the GEKF framework, we derive an iterative forward way to compute

Original: Q-values, which consists of a Q-value-update step as in the Q-learning algorithm and an expert correction step.
Revised: Q-values, which consists of a Q-value-update step as in the Q-learning algorithm and an expert correction step.

Original: In the correction step, our update rule weighs expert information according to an agent’s uncertainty of its self-learning result, measured by the estimated posterior variance of learned Q-values.
Revised: In the correction step, our update rule weighs expert information according to an agent’s uncertainty of its self-learning result, measured by the estimated posterior variance of learned Q-values.

Original: Thus, the larger Q-values’ variance is, the more our expert correction encourages expert behaviours by increasing expert actions’ Q-values and decreasing non-experts’ Q-values.
Revised: Thus, the larger Q-values’ variance is, the more our expert correction encourages expert behaviours by increasing expert actions’ Q-values and decreasing non-experts’ Q-values.

Original: This mechanism reduces the influence of uninformative expert data and avoids excessive guided exploration.
Revised: This mechanism reduces the influence of uninformative expert data and avoids excessive guided exploration.

Original: Furthermore, we propose our computationally efficient deep algorithm, Bayesian Q-learning from
Revised: Furthermore, we propose our computationally efficient deep algorithm, Bayesian Q-learning from

Original: Demonstrations (BQfD), built on top of the Deep Q-learning Network (DQN) (Mnih et al., 2013).
Revised: Demonstrations (BQfD), built on top of the Deep Q-learning Network (DQN) (Mnih et al., 2013).

Original: The algorithm embeds reliable expert knowledge into Q-values and leads to a more efficient exploration, as shown on a sparse-reward chain environment DeepSea and six randomly chosen Atari games.
Revised: The algorithm embeds reliable expert knowledge into Q-values and leads to a more efficient exploration, as shown on a sparse-reward chain environment DeepSea and six randomly chosen Atari games.

Original: In most environments, our algorithm learns faster than Deep Q-learning from Demonstrations (DQfD)
Revised: In most environments, our algorithm learns faster than Deep Q-learning from Demonstrations (DQfD)

Original: (Hester et al., 2018) and Prioritized Double Duelling (PDD) DQN (Wang et al., 2016).
Revised: (Hester et al., 2018) and Prioritized Double Duelling (PDD) DQN (Wang et al., 2016).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Reinforcement learning from demonstrations (RLfD) has drawn attention in recent years.
Revised: Reinforcement learning from demonstrations (RLfD) has drawn attention in recent years.

Original: This method only requires a small number of offline expert demonstrations and can noticeably improve performance.
Revised: This method only requires a small number of offline expert demonstrations and can noticeably improve performance.

Original: Deep Q-learning from demonstrations (DQfD) (Hester et al., 2018) includes an additional margin classification loss to ensure that Q -values of expert actions are higher than other actions.
Revised: Deep Q-learning from demonstrations (DQfD) (Hester et al., 2018) includes an additional margin classification loss to ensure that Q -values of expert actions are higher than other actions.

Original: Reinforcement learning from demonstrations through shaping (Brys et al., 2015) assigns high potential to a state-action pair ( s, a ) when the action a is used by the expert in the neighbourhood of the state s .
Revised: Reinforcement learning from demonstrations through shaping (Brys et al., 2015) assigns high potential to a state-action pair ( s, a ) when the action a is used by the expert in the neighbourhood of the state s .

Original: Wu et al.
Revised: Wu et al.

Original: (Wu et al., 2020) further leverages reward potential computed by generative models.
Revised: (Wu et al., 2020) further leverages reward potential computed by generative models.

Original: However, this requires an added complexity of training generative models.
Revised: However, this requires an added complexity of training generative models.

Original: Also, expert data is used implicitly to learn rewards inversely at the beginning (Abbeel & Ng, 2004; Brown et al., 2019).
Revised: Also, expert data is used implicitly to learn rewards inversely at the beginning (Abbeel & Ng, 2004; Brown et al., 2019).

Original: However, these approaches do not consider the case where the expert is imperfect and can often have difficulty exceeding expert performance.
Revised: However, these approaches do not consider the case where the expert is imperfect and can often have difficulty exceeding expert performance.

Original: In contrast, Nair et al.
Revised: In contrast, Nair et al.

Original: (Nair et al.
Revised: (Nair et al.

Original: 2018) handle suboptimal demonstrations by only cloning expert actions when their current estimated Q-values are higher than others.
Revised: 2018) handle suboptimal demonstrations by only cloning expert actions when their current estimated Q-values are higher than others.

Original: Zhang et al.
Revised: Zhang et al.

Original: (Zhang et al., 2022) utilizes expert information when the cumulative Q-values on expert state-action pairs are larger than cumulative value functions.
Revised: (Zhang et al., 2022) utilizes expert information when the cumulative Q-values on expert state-action pairs are larger than cumulative value functions.

Original: However, the learned Q-value is often unreliable and frequently changes, making it a poor judgment of expert data quality.
Revised: However, the learned Q-value is often unreliable and frequently changes, making it a poor judgment of expert data quality.

Original: At the same time, our measurement based on the posterior variance of estimated Q-values is more reasonable.
Revised: At the same time, our measurement based on the posterior variance of estimated Q-values is more reasonable.

Original: Jing et al.
Revised: Jing et al.

Original: (Jing et al., 2020) models the suboptimal expert policy as a local optimum to maximize the expected returns and then constrains the agent to learn within a region around the expert policy, measured by KL divergence between occupancy measures.
Revised: (Jing et al., 2020) models the suboptimal expert policy as a local optimum to maximize the expected returns and then constrains the agent to learn within a region around the expert policy, measured by KL divergence between occupancy measures.

Original: However, the local optimum condition is hard to satisfy, and the limitation of learning around the expert policy cannot deal with mistaken expert actions.
Revised: However, the local optimum condition is hard to satisfy, and the limitation of learning around the expert policy cannot deal with mistaken expert actions.

Original: In contrast, our assumption on Boltzmann distributed expert policy is much weaker.
Revised: In contrast, our assumption on Boltzmann distributed expert policy is much weaker.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: A Markov decision process (Sutton & Barto, 2018) is a tuple M = ⟨S , A , P, γ, R, ρ ⟩ where S is the state space, A is the action space, P is the time-homogeneous transition probability matrix with
Revised: A Markov decision process (Sutton & Barto, 2018) is a tuple M = ⟨S , A , P, γ, R, ρ ⟩ where S is the state space, A is the action space, P is the time-homogeneous transition probability matrix with

Original: P ( s ′ | s, a ) as elements, γ ∈
Revised: P ( s ′ | s, a ) as elements, γ ∈

Original: (0 , 1) is the discount factor, R is the reward function, with R ( s, a ) denoting a random vector describing rewards received after state-action pair ( s, a ) and ρ is the distribution of the initial state.
Revised: (0 , 1) is the discount factor, R is the reward function, with R ( s, a ) denoting a random vector describing rewards received after state-action pair ( s, a ) and ρ is the distribution of the initial state.

Original: At each time step h , the agent samples an action A h from a policy µ , and then transits to the next state S h +1 and gains a reward R ( S h , A h ) .
Revised: At each time step h , the agent samples an action A h from a policy µ , and then transits to the next state S h +1 and gains a reward R ( S h , A h ) .

Original: Our paper focuses on finite horizon cases with horizon H , where an agent stops at time step H .
Revised: Our paper focuses on finite horizon cases with horizon H , where an agent stops at time step H .

Original: Also, our paper works on finite action and state spaces.
Revised: Also, our paper works on finite action and state spaces.

Original: The expected discounted cumulative return starting from each state-action pair ( s, a ) at time step h and following the policy µ is called the Q-value, denoted by q µh ( s, a ) , and is defined as follows:
Revised: The expected discounted cumulative return starting from each state-action pair ( s, a ) at time step h and following the policy µ is called the Q-value, denoted by q µh ( s, a ) , and is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where τ denotes trajectories with S h
Revised: where τ denotes trajectories with S h

Original: = s ,
Revised: = s ,

Original: A h
Revised: A h

Original: = a , S t ∼ P ( ·| S t − 1 , A t − 1 ) ,
Revised: = a , S t ∼ P ( ·| S t − 1 , A t − 1 ) ,

Original: A t ∼ µ ( ·| S h ) and rewards R ( S t , A t ) .
Revised: A t ∼ µ ( ·| S h ) and rewards R ( S t , A t ) .

Original: The unique optimal Q-value function at time h is defined as:
Revised: The unique optimal Q-value function at time h is defined as:

Original: [Equation]
Revised: [Equation]

Original: The optimal Q-values also satisfy the optimal Bellman equation for all states and actions:[Equation] q
Revised: The optimal Q-values also satisfy the optimal Bellman equation for all states and actions:

Original: [Equation]
Revised: [Equation]

Original: The optimal Bellman operator T is defined by the equation.
Revised: The optimal Bellman operator T is defined by the equation.

Original: Moreover, the optimal Q-values at each time h for all state-action pairs can be combined into one vector, q ∗ h ∈ R |S|×|A| , each element representing the optimal Q-value for a state-action pair.
Revised: Moreover, the optimal Q-values at each time h for all state-action pairs can be combined into one vector, q ∗ h ∈ R |S|×|A| , each element representing the optimal Q-value for a state-action pair.

Original: Q- LEARNING
Revised: Q- LEARNING

Original: Optimal Q-values can be computed by Q-learning (Watkins & Dayan, 1992; Jin et al., 2018), which estimates Q-values directly and is based on the following update rule:
Revised: Optimal Q-values can be computed by Q-learning (Watkins & Dayan, 1992; Jin et al., 2018), which estimates Q-values directly and is based on the following update rule:

Original: [Equation]
Revised: [Equation]

Original: where Q is an estimation of Q-values, S ′ ∼ P ( ·| s, a ) is a sampled next state, and α ∈ (0 , 1) is the learning rate.
Revised: where Q is an estimation of Q-values, S ′ ∼ P ( ·| s, a ) is a sampled next state, and α ∈ (0 , 1) is the learning rate.

Original: A choice of adaptive learning rate is to assign each state-action pair a learning rate and decay it with respect to the number of visitations to the state-action pair.
Revised: A choice of adaptive learning rate is to assign each state-action pair a learning rate and decay it with respect to the number of visitations to the state-action pair.

Original: We use n l ( s, a ) to represent the number of visitation times of state-action pair ( s, a ) until and including episode l.
Revised: We use n l ( s, a ) to represent the number of visitation times of state-action pair ( s, a ) until and including episode l

Original:  B AYESIAN M ODEL - FREE F RAMEWORK
Revised: . B AYESIAN M ODEL - FREE F RAMEWORK

Original: Bayesian reinforcement learning assumes that there is an initial guess or a prior probability over the model parameters, denoted by P ( M ) .
Revised: Bayesian reinforcement learning assumes that there is an initial guess or a prior probability over the model parameters, denoted by P ( M ) .

Original: Then in the model-based case, the agent gradually learns a posterior probability for the model parameters conditioned on observed trajectories (Duff, 2003).
Revised: Then in the model-based case, the agent gradually learns a posterior probability for the model parameters conditioned on observed trajectories (Duff, 2003).

Original: Next, an agent can learn a policy based on the most likely MDP or by sampling MDPs from the posterior.
Revised: Next, an agent can learn a policy based on the most likely MDP or by sampling MDPs from the posterior.

Original: In the model-free case, we treat Q-values as random variables and implicitly embed parametric uncertainties from different MDPs.
Revised: In the model-free case, we treat Q-values as random variables and implicitly embed parametric uncertainties from different MDPs.

Original: A prior distribution of the optimal Q-values at time h is defined as:
Revised: A prior distribution of the optimal Q-values at time h is defined as:

Original: [Equation]
Revised: [Equation]

Original: Then algorithms compute the posterior probability of Q-values (Dearden et al., 1998;
Revised: Then algorithms compute the posterior probability of Q-values (Dearden et al., 1998;

Original: Osband et al., 2018).
Revised: Osband et al., 2018).

Original:  M ODEL
Revised:  M ODEL

Original: In this section, we present the probabilistic model used for Bayesian inference and then derive the update rule of Q-values by maximizing posterior probability.
Revised: In this section, we present the probabilistic model used for Bayesian inference and then derive the update rule of Q-values by maximizing posterior probability.

Original: 4. P ROBABILISTIC M ODEL OF S UBOPTIMAL E XPERT A CTIONS
Revised: 4. P ROBABILISTIC M ODEL OF S UBOPTIMAL E XPERT A CTIONS

Original: We start by modelling the relationship between expert actions and the optimal Q-values.
Revised: We start by modelling the relationship between expert actions and the optimal Q-values.

Original: The expert chooses the optimal actions most frequently but may contain mistakes and select low probability actions.
Revised: The expert chooses the optimal actions most frequently but may contain mistakes and select low probability actions.

Original: The Boltzmann distribution can describe this behaviour with the optimal Q-values as parameters.
Revised: The Boltzmann distribution can describe this behaviour with the optimal Q-values as parameters.

Original: Under this distribution, expert actions are sampled proportional to the exponential of optimal Q-values up to a constant multiplier.
Revised: Under this distribution, expert actions are sampled proportional to the exponential of optimal Q-values up to a constant multiplier.

Original: As known, this assumed expert policy maximizes the expected returns regularized by the entropy (Haarnoja et al., 2018), describing almost optimal but sometimes mistaken expert behaviours.
Revised: As known, this assumed expert policy maximizes the expected returns regularized by the entropy (Haarnoja et al., 2018), describing almost optimal but sometimes mistaken expert behaviours.

Original: The above assumption is formally presented as follows and shown on the left of figure 1, where the expert action at time h and state s is denoted by A exp,h (s).
Revised: The above assumption is formally presented as follows and shown on the left of figure 1, where the expert action at time h and state s is denoted by A exp,h (s).

Original: Assumption 1.
Revised: Assumption 1.

Original: Assume that expert demonstrations are drawn from a policy π expert dependent on the optimal Q-values.
Revised: Assume that expert demonstrations are drawn from a policy π expert dependent on the optimal Q-values.

Original: Also, this policy follows the Boltzmann distribution:
Revised: Also, this policy follows the Boltzmann distribution:

Original: [Equation]
Revised: [Equation]

Original: where η is any positive constant.
Revised: where η is any positive constant.

Original: Moreover, we capture the relationship between the optimal Q-values at different time steps in our probabilistic model through the Bellman equation 1.
Revised: Moreover, we capture the relationship between the optimal Q-values at different time steps in our probabilistic model through the Bellman equation 1.

Original: Considering the Bellman equation cannot be computed without the knowledge of environment dynamics, our model relies on samples of the reward and the next state, which are widely accepted in the field (Watkins & Dayan, 1992).
Revised: Considering the Bellman equation cannot be computed without the knowledge of environment dynamics, our model relies on samples of the reward and the next state, which are widely accepted in the field (Watkins & Dayan, 1992).

Original: We re-write the Bellman equation in a stochastic form as
Revised: We re-write the Bellman equation in a stochastic form as

Original: [Equation]
Revised: [Equation]

Original: where R ( s, a ) and the next state S ′ are sampled according to the rules of the underlying MDP.
Revised: where R ( s, a ) and the next state S ′ are sampled according to the rules of the underlying MDP.

Original: The random noise ν is defined as
Revised: The random noise ν is defined as

Original: [Equation]
Revised: [Equation]

Original: which captures the randomness from the dynamics.
Revised: which captures the randomness from the dynamics.

Original: Then, the whole model of relationships among rewards, optimal Q-values, and expert actions along a trajectory is presented on the right of figure 1.
Revised: Then, the whole model of relationships among rewards, optimal Q-values, and expert actions along a trajectory is presented on the right of figure 1.

Original: 4. GEKF F RAMEWORK
Revised: 4. GEKF F RAMEWORK

Original: Next, our desired estimated Q-values should not only follow the stochastic Bellman equation, but they should also most likely give the Boltzmann distribution that the expert is following.
Revised: Next, our desired estimated Q-values should not only follow the stochastic Bellman equation, but they should also most likely give the Boltzmann distribution that the expert is following.

Original: This objective is equivalent to maximizing the posterior probabilities of optimal Q-values equaling our estimations conditioned on expert information while constraining to the stochastic Bellman equation.
Revised: This objective is equivalent to maximizing the posterior probabilities of optimal Q-values equaling our estimations conditioned on expert information while constraining to the stochastic Bellman equation.

Original: In order to compute the posterior probability, we need help from a framework under which the posterior probability density function or the posterior mode is of analytical form.
Revised: In order to compute the posterior probability, we need help from a framework under which the posterior probability density function or the posterior mode is of analytical form.

Original: Thus, our paper leverages the generalized extended Kalman filter (GEKF) (Fahrmeir, 1992), which can analyze the time series of Q-values with extra expert information and provides an estimation of the posterior maximum.
Revised: Thus, our paper leverages the generalized extended Kalman filter (GEKF) (Fahrmeir, 1992), which can analyze the time series of Q-values with extra expert information and provides an estimation of the posterior maximum.

Original: This framework requires extra expert information following an exponential family distribution, which is already assumed.
Revised: This framework requires extra expert information following an exponential family distribution, which is already assumed.

Original: Furthermore, the framework requires the random noise ν in the stochastic Bellman equation 3 at each
Revised: Furthermore, the framework requires the random noise ν in the stochastic Bellman equation 3 at each

Original: Q-value update step to be Gaussian, which is also assumed in Osband et al.
Revised: Q-value update step to be Gaussian, which is also assumed in Osband et al.

Original: (Osband et al., 2019).
Revised: (Osband et al., 2019).

Original: Meanwhile, this assumption does not hurt in the latter stage of training since the influence of the random noise gradually approaches zero as the number of samples increases, and the learning rate decays.
Revised: Meanwhile, this assumption does not hurt in the latter stage of training since the influence of the random noise gradually approaches zero as the number of samples increases, and the learning rate decays.

Original: Notice that the random noise should have a zero expectation and bounded variance when rewards and time horizons are bounded.
Revised: Notice that the random noise should have a zero expectation and bounded variance when rewards and time horizons are bounded.

Original: Thus, our paper considers ν as a Gaussian random variable with zero expectation and a fixed variance λ , as shown in the following assumption.
Revised: Thus, our paper considers ν as a Gaussian random variable with zero expectation and a fixed variance λ , as shown in the following assumption.

Original: Assumption 2.
Revised: Assumption 2.

Original: For all state-action pair ( s, a ) ,the noise ν is modeled independently by a Gaussian random variable ν ∼ N (0 , λ ) .
Revised: For all state-action pair ( s, a ) ,the noise ν is modeled independently by a Gaussian random variable ν ∼ N (0 , λ ) .

Original: Also, the GEKF framework approximates the posterior distribution by Gaussian and linearizes the maximization operator in Q-value updates as in Osband et al.
Revised: Also, the GEKF framework approximates the posterior distribution by Gaussian and linearizes the maximization operator in Q-value updates as in Osband et al.

Original: (Osband et al., 2019).
Revised: (Osband et al., 2019).

Original: 4. D ERIVATION OF Q- VALUES U PDATE R ULES
Revised: 4. D ERIVATION OF Q- VALUES U PDATE R ULES

Original: We derive our Q-value estimations Q under the above assumptions given by the GEKF framework (Fahrmeir, 1992).
Revised: We derive our Q-value estimations Q under the above assumptions given by the GEKF framework (Fahrmeir, 1992).

Original: Firstly, this estimation should satisfy the stochastic Bellman equation in equation 3.
Revised: Firstly, this estimation should satisfy the stochastic Bellman equation in equation 3.

Original: Secondly, it should maximize the posterior probability of the optimal Q-values conditioned on expert demonstrations.
Revised: Secondly, it should maximize the posterior probability of the optimal Q-values conditioned on expert demonstrations.

Original: This task is stated as follows:
Revised: This task is stated as follows:

Original: [Equation]
Revised: [Equation]

Original: Recall that q ∗ h ∈ R |S||A| represents the Q-value vector at step h with each element as the value of one state-action pair, and similar for R h ; moreover, A exp,h ∈ R |S| represents the expert actions chosen at all states at step h .
Revised: Recall that q ∗ h ∈ R |S||A| represents the Q-value vector at step h with each element as the value of one state-action pair, and similar for R h ; moreover, A exp,h ∈ R |S| represents the expert actions chosen at all states at step h .

Original: Then, we can eliminate terms independent of Q-values in the posterior and obtain:
Revised: Then, we can eliminate terms independent of Q-values in the posterior and obtain:

Original: [Equation]
Revised: [Equation]

Original: The three distributions are modelled through our previous assumptions: the expert action follows a
Revised: The three distributions are modelled through our previous assumptions: the expert action follows a

Original: Boltzmann distribution, Q-values at step h follow a Gaussian distribution with the expectation given by the Bellman equation, and Q-values at step H are all set to be zero.
Revised: Boltzmann distribution, Q-values at step h follow a Gaussian distribution with the expectation given by the Bellman equation, and Q-values at step H are all set to be zero.

Original: Next, the task in equation 4 forms a constrained optimization problem and can further be transformed into a two-point boundary value problem (Fahrmeir, 1992; Sage & Melsa, 1971).
Revised: Next, the task in equation 4 forms a constrained optimization problem and can further be transformed into a two-point boundary value problem (Fahrmeir, 1992; Sage & Melsa, 1971).

Original: This boundary value problem is then solved by the invariant embedding method, which provides approximate solutions for non-linear partial differential equations.
Revised: This boundary value problem is then solved by the invariant embedding method, which provides approximate solutions for non-linear partial differential equations.

Original: The derivation is shown in the appendix following the steps in Sage (1997)
Revised: The derivation is shown in the appendix following the steps in Sage (1997)

Original: (Sage & Melsa, 1971).
Revised: (Sage & Melsa, 1971).

Original: Finally, in a backward view, our estimations can be computed step-by-step as presented in algorithm 1.
Revised: Finally, in a backward view, our estimations can be computed step-by-step as presented in algorithm 1.

Original: In the prediction step, the algorithm updates Q-values based on sampled rewards and sampled next states by the Bellman equation as the Q-learning algorithm (Watkins & Dayan, 1992).
Revised: In the prediction step, the algorithm updates Q-values based on sampled rewards and sampled next states by the Bellman equation as the Q-learning algorithm (Watkins & Dayan, 1992).

Original: This update of estimated Q-values can be written as
Revised: This update of estimated Q-values can be written as

Original: [Equation]
Revised: [Equation]

Original: using a transformation matrix T ∈ R |S||A|×|S||A| .
Revised: using a transformation matrix T ∈ R |S||A|×|S||A| .

Original: Its element at each state-action pair and its next-state-maximal-action pair equals to one, that is T [( s, a ) , ( S ′ , a ′ )]
Revised: Its element at each state-action pair and its next-state-maximal-action pair equals to one, that is T [( s, a ) , ( S ′ , a ′ )]

Original: = 1 where S ′ ∼ P ( ·| s, a ) and a ′ =
Revised: = 1 where S ′ ∼ P ( ·| s, a ) and a ′ =

Original: argmax b Q
Revised: argmax b Q

Original: h
Revised: h

Original: +

Revised: +

Original: ( S ′ , b ) .
Revised: ( S ′ , b ) .

Original: And all other elements are zero.
Revised: And all other elements are zero.

Original: Then, the algorithm computes the estimated posterior variance W .
Revised: Then, the algorithm computes the estimated posterior variance W .

Original: Our derivation gives that this variance not only depends on the transformation matrix T , but also the Hessian of log expert policy
Revised: Our derivation gives that this variance not only depends on the transformation matrix T , but also the Hessian of log expert policy

Original: U ∈ R |S||A|×|S||A| .
Revised: U ∈ R |S||A|×|S||A| .

Original: So the update first computes N h =
Revised: So the update first computes N h =

Original: T Th W h +1 T
Revised: T Th W h +1 T

Original: h + λI , and then calculates the estimated variance as W h
Revised: h + λI , and then calculates the estimated variance as W h

Original: =
Revised: =

Original: ( N − 1 h + U h ) − 1 .
Revised: ( N − 1 h + U h ) − 1 .

Original: After computing the variance, expert information corrects Q-values as
Revised: After computing the variance, expert information corrects Q-values as

Original: [Equation]
Revised: [Equation]

Original: Recall that η is the parameter of the expert policy.
Revised: Recall that η is the parameter of the expert policy.

Original: So a bonus weighted by the variance is given to expert actions and encourages the agent to explore them when it is not confident of the current learning result.
Revised: So a bonus weighted by the variance is given to expert actions and encourages the agent to explore them when it is not confident of the current learning result.

Original: When the variance of a Q-value estimate is low, the agent stops guided exploration by the expert and follows its own learnt policy.
Revised: When the variance of a Q-value estimate is low, the agent stops guided exploration by the expert and follows its own learnt policy.

Original: Thus, the agent can deal with inadequate expert data by gradually ignoring it.
Revised: Thus, the agent can deal with inadequate expert data by gradually ignoring it.

Original: Meanwhile, this expert modification does not break the stochastic Bellman equation in equation 3 since it is listed as the constraint in the posterior maximization task in equation 4.
Revised: Meanwhile, this expert modification does not break the stochastic Bellman equation in equation 3 since it is listed as the constraint in the posterior maximization task in equation 4.

Original: A LGORITHM

Original: Our paper proposes an algorithm Bayesian Q-learning from Demonstrations (BQfD) to realize the

Original: GEKF update rules of Q-values.

Original: This algorithm samples a trajectory at the beginning of each turn and then updates Q-values by the sampled-based Bellman rule and the expert correction backward.

Original: Initialize: Q-values at the last step Q H = 0 , Weight matrices of Q-value functions W H = 0 |S||A|×|S||A| , and Identity matrix I .
Revised: Initialize: Q-values at the last step Q H = 0 , Weight matrices of Q-value functions W H = 0 |S||A|×|S||A| , and Identity matrix I .

Original: for time step h
Revised: for time step h

Original: = H − 1 , ..., 0 do
Revised: = H − 1 , ..., 0 do

Original: Prediction step:
Revised: Prediction step:

Original: Update Q-value of each state-action pair based on sampled reward and next state though Bellman equation
Revised: Update Q-value of each state-action pair based on sampled reward and next state though Bellman equation

Original: [Equation]
Revised: [Equation]

Original: Correction step:
Revised: Correction step:

Original: Compute the estimated posterior variance
Revised: Compute the estimated posterior variance

Original: [Equation]
Revised: [Equation]

Original: Embed in the expert correction
Revised: Embed in the expert correction

Original: [Equation]
Revised: [Equation]

Original: end for
Revised: end for

Revised:  A LGORITHM

Revised: Our paper proposes an algorithm Bayesian Q-learning from Demonstrations (BQfD) to realize the

Revised: GEKF update rules of Q-values.

Revised: This algorithm samples a trajectory at the beginning of each turn and then updates Q-values by the sampled-based Bellman rule and the expert correction backward.

Original: Thus, our algorithm only updates the Q-value of the state-action pair ( S h , A h ) at each time step h and introduces a learning rate α for Bellman equation updates to reduce variances.
Revised: Thus, our algorithm only updates the Q-value of the state-action pair ( S h , A h ) at each time step h and introduces a learning rate α for Bellman equation updates to reduce variances.

Original: This learning rate α n ( s,a ) ( s, a ) is considered to be state-action pair dependent and decays as the number of that pair’s visitations n ( s, a ) increases.
Revised: This learning rate α n ( s,a ) ( s, a ) is considered to be state-action pair dependent and decays as the number of that pair’s visitations n ( s, a ) increases.

Original: However, the posterior variance calculation in the correction step is computationally inefficient due to the Hessian and inverse operations.
Revised: However, the posterior variance calculation in the correction step is computationally inefficient due to the Hessian and inverse operations.

Original: To avoid these computations, we would like to approximate the posterior variance at a state-action pair ( s, a ) .
Revised: To avoid these computations, we would like to approximate the posterior variance at a state-action pair ( s, a ) .

Original: Given a first-order decaying learning rate α n ( s,a ) ( s, a ) = 1 β + n ( s,a ) , short written as α and ignoring the variance change by expert information, the variance is roughly a function of the visitation times, denoted by w n ( s,a ) =
Revised: Given a first-order decaying learning rate α n ( s,a ) ( s, a ) = 1 β + n ( s,a ) , short written as α and ignoring the variance change by expert information, the variance is roughly a function of the visitation times, denoted by w n ( s,a ) =

Original: O ( 1 ( n ( s,a ) λ ) .
Revised: O ( 1 ( n ( s,a ) λ ) .

Original: Furthermore, our paper utilizes deep neural networks and proposes a deep BQfD based on Prioritized
Revised: Furthermore, our paper utilizes deep neural networks and proposes a deep BQfD based on Prioritized

Original: Dueling Double (PDD) Deep Q-learning Network (DQN) (Mnih et al., 

Revised: Dueling Double (PDD) Deep Q-learning Network (DQN) (Mnih et al.,

Original: ; Wang et al., 2016;
Revised: ; Wang et al., 2016;

Original: Hessel et al., 2018).
Revised: Hessel et al., 2018).

Original: Notice that our current update rule requires expert information for every state, while a limited set of expert demonstrations is usually provided.
Revised: Notice that our current update rule requires expert information for every state, while a limited set of expert demonstrations is usually provided.

Original: Meanwhile, the indicator value in the expert correction may have conflict values if an expert provides multiple trajectories but chooses different actions at a state.
Revised: Meanwhile, the indicator value in the expert correction may have conflict values if an expert provides multiple trajectories but chooses different actions at a state.

Original: Instead, our algorithm suggests storing expert data in the replay buffer and only performing expert correction when sampling expert data from the buffer.
Revised: Instead, our algorithm suggests storing expert data in the replay buffer and only performing expert correction when sampling expert data from the buffer.

Original: Also, our algorithm weighs expert data to reduce off-policy biases.
Revised: Also, our algorithm weighs expert data to reduce off-policy biases.

Original: The importance sampling ratio is hard to calculate, so our algorithm uses a computationally-efficient weight
Revised: The importance sampling ratio is hard to calculate, so our algorithm uses a computationally-efficient weight

Original: [Equation]
Revised: [Equation]

Original: where the power ζ is a rescaling factor to avoid miniature weights.
Revised: where the power ζ is a rescaling factor to avoid miniature weights.

Original: This ratio assigns higher values to more frequent state-action pairs under the current ϵ -greedy policy with respect to Q-estimates.
Revised: This ratio assigns higher values to more frequent state-action pairs under the current ϵ -greedy policy with respect to Q-estimates.

Original: The detailed algorithm is presented in algorithm 3 in the appendix.
Revised: The detailed algorithm is presented in algorithm 3 in the appendix.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: To evaluate the discernment of expert data, we test our algorithm using optimal and misleading expert data in a sparse-reward chain environment called DeepSea (Osband et al., 2019).
Revised: To evaluate the discernment of expert data, we test our algorithm using optimal and misleading expert data in a sparse-reward chain environment called DeepSea (Osband et al., 2019).

Original: Then, we further test the data efficiency of our algorithm with the help of a limited amount of suboptimal human data on six complicated Atari games (Brockman et al., 2016).
Revised: Then, we further test the data efficiency of our algorithm with the help of a limited amount of suboptimal human data on six complicated Atari games (Brockman et al., 2016).

Original: We compare against other two discrete-action algorithms:
Revised: We compare against other two discrete-action algorithms:

Original: Prioritized Dueling Double (PDD)
Revised: Prioritized Dueling Double (PDD)

Original: Deep Q-learning Network (DQN) (Mnih et al., 2013;
Revised: Deep Q-learning Network (DQN) (Mnih et al., 2013;

Original: Wang et al., 2016; Hessel et al., 2018) with EZ-greedy (Dabney et al., 2020) exploration and Deep
Revised: Wang et al., 2016; Hessel et al., 2018) with EZ-greedy (Dabney et al., 2020) exploration and Deep

Original: Q-learning from demonstrations (DQfD).
Revised: Q-learning from demonstrations (DQfD).

Original: The DeepSea environment requires less than one hour of training on an Nvidia 2070 GPU.
Revised: The DeepSea environment requires less than one hour of training on an Nvidia 2070 GPU.

Original: Each Atari game requires 21 days of training on Compute Canada server with either one Nvidia V100 or one Nvidia P100. 
Revised: Each Atari game requires 21 days of training on Compute Canada server with either one Nvidia V100 or one Nvidia P100. 6. C HAIN E NVIRONMENT

Revised: DeepSea is a deterministic chain environment of length N with two actions: left and right.

Revised: A figure to describe the environment is shown in the appendix.

Revised: The agent is initialized on the left in each episode and terminates after N steps.

Revised: The agent receives rewards of 0 for left actions and − 0 .

Original: [Figure]
Revised: 01 N for the right actions except for a fixed but unknown reward at the most right state. The unknown but fixed reward consists of either a treasure with a bonus of +1 or a bomb with a cost of − 1 . Regardless of the environment, we always provide agents with an expert trajectory towards the most right. Thus, in the treasure case, we test our algorithms’ efficiency; in the bomb case, we evaluate our algorithms’ robustness in the presence of highly misleading expert information.

Revised: To provide a fair comparison, we include an expert trajectory to the replay buffer of DQN and provide no pretraining for any algorithms.

Original: The results for the treasure case are shown on the left of figure 2.
Revised: The results for the treasure case are shown on the left of figure 2.

Original: BQfD and DQfD can gain the optimal reward after one training episode much faster than DQN since the expert data directly leads to the reward, and these two algorithms extract expert information efficiently.
Revised: BQfD and DQfD can gain the optimal reward after one training episode much faster than DQN since the expert data directly leads to the reward, and these two algorithms extract expert information efficiently.

Original: But DQfD assumes the expert data to be optimal and causes catastrophic performance when expert demonstrations are misguiding, as seen in the bomb case, shown on the right of figure 2.
Revised: But DQfD assumes the expert data to be optimal and causes catastrophic performance when expert demonstrations are misguiding, as seen in the bomb case, shown on the right of figure 2.

Original: On the contrary, our learningfrom-demonstrations approach successfully learns the optimal policy, though the convergence speed is hindered due to confusing information.
Revised: On the contrary, our learningfrom-demonstrations approach successfully learns the optimal policy, though the convergence speed is hindered due to confusing information.

Original: This result proves that BQfD efficiently extracts expert information and gradually stops using expert information.
Revised: This result proves that BQfD efficiently extracts expert information and gradually stops using expert information.

Original: 6. A RCADE L EARNING E NVIRONMENT E XPERIMENT S ETTINGS
Revised: 6. A RCADE L EARNING E NVIRONMENT E XPERIMENT S ETTINGS

Original: We further evaluated our approach on Arcade Learning Environments (Brockman et al., 2016; Mnih et al., 2013): Breakout, ChopperCommand, Freeway, MsPacman, VideoPinball, and Seaquest.
Revised: We further evaluated our approach on Arcade Learning Environments (Brockman et al., 2016; Mnih et al., 2013): Breakout, ChopperCommand, Freeway, MsPacman, VideoPinball, and Seaquest.

Original: The architecture for all three algorithms is the same as PDD DQN (Mnih et al., 2013; Wang et al., 2016;
Revised: The architecture for all three algorithms is the same as PDD DQN (Mnih et al., 2013; Wang et al., 2016;

Original: Hester et al., 2018; Hessel et al., 2018) with prioritized replay buffer (Schaul et al., 2015)
Revised: Hester et al., 2018; Hessel et al., 2018) with prioritized replay buffer (Schaul et al., 2015).

Original: Baselines’ hyperparameters follow the choices in DQfD and the  replay buffer.
Revised: Baselines’ hyperparameters follow the choices in DQfD and the prioritized replay buffer.

Original: Meanwhile, we tune hyperparameters for BQfD on  environment  the  results are in appendix are provided with
Revised: Meanwhile, we tune hyperparameters for BQfD on the environment Seaquest, and the detailed results are shown

Original: -generated trajectory pre trained

Original: [Table]
Revised: [Table]

Revised: in the appendix.

Revised: All approaches are provided with one human-generated expert trajectory and are pre-trained with this expert data.

Revised: Given the randomness during training, all results shown are averaged over three random seeds.

Original: R ESULTS
Revised: R ESULTS

Original: The overall results are shown in table 1.
Revised: The overall results are shown in table 1.

Original: Here, we present final-learned evaluation rewards: the average means and standard deviations of evaluation rewards over the last 2M steps.
Revised: Here, we present final-learned evaluation rewards: the average means and standard deviations of evaluation rewards over the last 2M steps.

Original: Our algorithm outperforms the baselines in Breakout, ChopperCommand, Seaquest
Revised: Our algorithm outperforms the baselines in Breakout, ChopperCommand, Seaquest

Original: [Table]
Revised: [Table]

Original: and VideoPinball.
Revised: and VideoPinball.

Original: In MsPacman, it still performs better than DQN but slightly worse than DQfD.
Revised: In MsPacman, it still performs better than DQN but slightly worse than DQfD.

Original: In general, both three algorithms perform much worse than humans on MsPacman.
Revised: In general, both three algorithms perform much worse than humans on MsPacman.

Original: DQN dominates the game in Freeway [Table] but our algorithm recovers from weak expert data compared to DQfD.
Revised: DQN dominates the game in Freeway, but our algorithm recovers from weak expert data compared to DQfD.

Original: The learning curve of Breakout and Freeway are presented in figure 3.
Revised: The learning curve of Breakout and Freeway are presented in figure 3.

Original: Initially, the three algorithms’ performances do not show significant differences.
Revised: Initially, the three algorithms’ performances do not show significant differences.

Original: But the average evaluation performance slowly tilts in favour of BQfD; it learns fastest, and the order of DQN and DQfD varies.
Revised: But the average evaluation performance slowly tilts in favour of BQfD; it learns fastest, and the order of DQN and DQfD varies.

Original: Surprisingly, BQfD’s advantage arises in the latter stages of training instead of improving learning speed in the early stages.
Revised: Surprisingly, BQfD’s advantage arises in the latter stages of training instead of improving learning speed in the early stages.

Original: One reason might be that expert data leads to adequate explorations.
Revised: One reason might be that expert data leads to adequate explorations.

Original: Also, our approach encourages the agent to deviate from expert states and learn from trials and errors.
Revised: Also, our approach encourages the agent to deviate from expert states and learn from trials and errors.

Original:  F UTURE W ORK
Revised:  F UTURE W ORK

Original: Learning with demonstrations is an exciting direction.
Revised: Learning with demonstrations is an exciting direction.

Original: Though our approach is limited to discrete action problems, it can be extended to continuous actions and policy gradient algorithms left as future work.
Revised: Though our approach is limited to discrete action problems, it can be extended to continuous actions and policy gradient algorithms left as future work.

Original: Furthermore, a more accurate approximation of posterior variance should be considered for future work.
Revised: Furthermore, a more accurate approximation of posterior variance should be considered for future work.

Original: Moreover, when experts are not limited to being optimal but instead generate informative trajectories, the quality of expert demonstrations becomes more challenging to judge.
Revised: Moreover, when experts are not limited to being optimal but instead generate informative trajectories, the quality of expert demonstrations becomes more challenging to judge.

Original: The task of selecting the correct expert becomes an important question.
Revised: The task of selecting the correct expert becomes an important question.