Original: H ELLINGER D ISTANCE C ONSTRAINED R EGRESSION
Revised: H ELLINGER D ISTANCE C ONSTRAINED R EGRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper introduces the off-policy reinforcement learning method that usesthe
Revised: This paper introduces an off-policy reinforcement learning method that uses

Original: Hellinger distance between sampling policy  and current policy  as a constraint.
Revised: Hellinger distance between sampling policy (from what samples were collected) and current policy (policy being optimized) as a constraint.

Original: Hellinger distance squared multiplied by two is greater than or equal to total variation distance squared and less than or equal to Kullback-Leibler divergence, therefore  lower bound for expected discounted return for the new policy is improved comparing to  KL.
Revised: Hellinger distance squared multiplied by two is greater than or equal to total variation distance squared and less than or equal to Kullback-Leibler divergence, therefore a lower bound for expected discounted return for the new policy is improved compared to the lower bound for training with KL.

Original: Also, Hellinger distance is less than or equal to 1, so there is a policy-independent lower bound for expected discounted return.
Revised: Also, Hellinger distance is less than or equal to 1, so there is a policy-independent lower bound for expected discounted return.

Original: HDCR is capable of training with Experience Replay, a common setting for distributed RL when collecting trajectories using different policies and learning from this data centralized.
Revised: HDCR is capable of training with Experience Replay, a common setting for distributed RL when collecting trajectories using different policies and learning from this data centralized.

Original: HDCR shows results comparable to or better than Advantageweighted Behavior Model and Advantage-Weighted Regression on MuJoCo tasks using  ofﬂine datasets collected by random agents and datasets  obtained during the ﬁrst iterations of online training of HDCR agent.
Revised: HDCR shows results comparable to or better than Advantage-weighted Behavior Model and Advantage-Weighted Regression on MuJoCo tasks using tiny ofﬂine datasets collected by random agents. On bigger datasets (100k timesteps) obtained by pretrained behavioral policy, HDCR outperforms ABM and AWR methods on 3 out of 4 tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Policy gradient algorithms are methods of model-free reinforcement learning that optimize policy through differentiating expected discounted return.
Revised: Policy gradient algorithms are methods of model-free reinforcement learning that optimize policy through differentiating expected discounted return.

Original: Despite the simplicity, to converge, these methods should stay on-policy because of the ﬁrst-order approximation of state visitation frequencies.
Revised: Despite the simplicity, to converge, these methods should stay on-policy because of the ﬁrst-order approximation of state visitation frequencies.

Original: This issue makes agents learn through trial-and-error, using data only once.
Revised: This issue makes agents learn through trial-and-error, using data only once.

Original: To make policy gradient updates more off-policy, we can add a constraint on the update to decrease the step size if the current policy is too far from the sampling policy.
Revised: To make policy gradient updates more off-policy, we can add a constraint on the update to decrease the step size if the current policy is too far from the sampling policy.

Original: One of the ﬁrst methods was to add total variation distance squared as a constraint for mixture policies.
Revised: One of the ﬁrst methods was to add total variation distance squared as a constraint for mixture policies.

Original: Later it was proven that there is a lower bound for new policy’s expected discounted return (Kakade & Langford, 2002).
Revised: Later it was proven that there is a lower bound for new policy’s expected discounted return (Kakade & Langford, 2002).

Original: Recently it was proven that this lower bound exists for all types of updates (Schulman et al., 2015).
Revised: Recently it was proven that this lower bound exists for all types of updates (Schulman et al., 2015).

Original: Next, total variation distance squared was replaced by Kullback-Leibler divergence that is greater than or equal to the previous one (Pinker’s inequality (Levin & Peres, 2017)), so that the lower bound was decreased (Schulman et al., 2015).
Revised: Next, total variation distance squared was replaced by Kullback-Leibler divergence that is greater than or equal to the previous one (Pinker’s inequality (Levin & Peres, 2017)), so that the lower bound was decreased (Schulman et al., 2015).

Original: With the use of Lagrangian, have been derived offpolicy method called Advantage-Weighted Regression (Peng et al., 2019), which also used KL as a constraint.
Revised: Using Lagrangian, have been derived off-policy method called Advantage-Weighted Regression (Peng et al., 2019), which also used KL as a constraint.

Original: This article proposes a new method whose lower bound of expected discounted return is greater than or equal to the bound with KL.
Revised: This article proposes a new method whose lower bound of expected discounted return is greater than or equal to the bound with KL.

Original: We achieve this by replacing total variation distance by Hellinger distance, which decreases lower bound.
Revised: We achieve this by replacing total variation distance by Hellinger distance, which decreases lower bound.

Original: Therefore strictness stays the same.
Revised: Therefore strictness stays the same.

Original: Then we derive an offpolicy method called Hellinger Distance Constrained Regression using the new constraint.
Revised: Then we derive an offpolicy method called Hellinger Distance Constrained Regression using the new constraint.

Original: It can be used on discrete and continuous action spaces since derivation uses Lebesgue integrals rather than a summation or Riemann integrals.
Revised: It can be used on discrete and continuous action spaces since derivation uses Lebesgue integrals rather than a summation or Riemann integrals.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: To better present the problem, we start from basic deﬁnitions, go through the history of improvements, and then describe the disadvantages of using KL divergence as a constraint.
Revised: To better present the problem, we start from basic deﬁnitions, go through the history of improvements, and then describe the disadvantages of using KL divergence as a constraint.

Original: We consider an inﬁnite-horizon discounted Markov decision process (MDP), deﬁned by the tuple ( S , A , P, r, ρ 0 , γ ), where S is a set of states (ﬁnite or inﬁnite), A is a set of actions (ﬁnite or inﬁnite), P : S ×
Revised: We consider an inﬁnite-horizon discounted Markov decision process (MDP), deﬁned by the tuple ( S , A , P, r, ρ 0 , γ ), where S is a set of states (ﬁnite or inﬁnite), A is a set of actions (ﬁnite or inﬁnite), P : S ×

Original: A × S → R is the transition probability distribution, r : S → R is the reward function, ρ 0 : S → R is the distribution of the initial state s 0 , and γ ∈ (0 , 1) is the discount factor.
Revised: A × S → R is the transition probability distribution, r : S → R is the reward function, ρ 0 : S → R is the distribution of the initial state s 0 , and γ ∈ (0 , 1) is the discount factor.

Original: Let π denote a stochastic policy π : S ×
Revised: Let π denote a stochastic policy π : S ×

Original: A → [0 , 1] , and then its expected discounted return is:
Revised: A → [0 , 1] , and then its expected discounted return is:

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: This paper uses state-action value function Q π , state value function V π , and advantage function A π with the following deﬁnitions:
Revised: This paper uses state-action value function Q π , state value function V π , and advantage function A π with the following deﬁnitions:

Original: [Equation]
Revised: [Equation]

Original: Let ρ π ( s ) be unnormalized visitation frequencies of state s where actions are chosen according to π :
Revised: Let ρ π ( s ) be unnormalized visitation frequencies of state s where actions are chosen according to π :

Original: [Equation]
Revised: [Equation]

Original: Following identity expresses the expected return of another policy ˜ π
Revised: Following identity expresses the expected return of another policy ˜ π

Original: in terms of the advantage over π , accumulated over states (see Schulman et al.
Revised: in terms of the advantage over π , accumulated over states (see Schulman et al.

Original: (2015) for proof):
Revised: (2015) for proof):

Original: [Equation]
Revised: [Equation]

Original: In approximately optimal learning, we replace state visitation frequency ρ ˜ π by ρ π , since this drastically decrease optimization complexity:
Revised: In approximately optimal learning, we replace state visitation frequency ρ ˜ π by ρ π , since this drastically decrease optimization complexity:

Original: [Equation]
Revised: [Equation]

Original: Let π old denote current policy, then the lower bound for the expected discounted return for the new policy π new will be (see Schulman et al.
Revised: Let π old denote current policy, then the lower bound for the expected discounted return for the new policy π new will be (see Schulman et al.

Original: (2015) for proof):
Revised: (2015) for proof):

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Theoretical Trust-Region Policy Optimization algorithm relays on Pinsker’s inequality (see (Tsybakov, 2009) for proof):
Revised: Theoretical Trust-Region Policy Optimization algorithm relays on Pinsker’s inequality (see (Tsybakov, 2009) for proof):

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: new (
Revised: new (

Original: [Equation]
Revised: [Equation]

Original: | s
Revised: | s

Original: [Equation]
Revised: [Equation]

Original: where D KL ( π old ( ·
Revised: where D KL ( π old ( ·

Original: [Equation]
Revised: [Equation]

Original: new (
Revised: new (

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: old
Revised: old

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: old
Revised: old

Original: [Equation]
Revised: [Equation]

Original: new
Revised: new

Original: [Equation]
Revised: [Equation]

Original: To retain strictness and decrease calculation complexity, total variation distance squared was replaced with Kullback-Leibler divergence D KL ( π old ( ·| s ) ||
Revised: To retain strictness and decrease calculation complexity, total variation distance squared was replaced with Kullback-Leibler divergence D KL ( π old ( ·| s ) ||

Original: π new ( ·| s )) :
Revised: π new ( ·| s )) :

Original: [Equation]
Revised: [Equation]

Original: However, this replacement greatly decreases the lower bound for the expected discounted return for the new policy.
Revised: However, this replacement greatly decreases the lower bound for the expected discounted return for the new policy.

Original: Moreover, Kullback-Leibler divergence has no upper bound.
Revised: Moreover, Kullback-Leibler divergence has no upper bound.

Original: Therefore we have no policy-independent lower bound for this type of update.
Revised: Therefore we have no policy-independent lower bound for this type of update.

Original:  H ELLINGER D ISTANCE IN P OLICY O PTIMIZATION
Revised:  H ELLINGER D ISTANCE IN P OLICY O PTIMIZATION

Original: We can improve lower bound (comparing to KL) by replacing D TV ( π old ( · | s )
Revised: We can improve lower bound (compared to KL) by replacing D TV ( π old ( · | s )

Original: ||
Revised: ||

Original: π new ( · | s )) with
Revised: π new ( · | s )) with

Original: Hellinger distance H ( π old ( ·| s ) ||
Revised: Hellinger distance H ( π old ( ·| s ) ||

Original: π new ( · | s )) :
Revised: π new ( · | s )) :

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 (see Appendix A or (Tsybakov, 2009, section 2.4) for proof) proves correctness and improvement (comparing to KL) to the lower bound.
Revised: Theorem 1 (see Appendix A or (Tsybakov, 2009, section 2.4) for proof) proves correctness and improvement (compared to KL) to the lower bound.

Original: Let p ( v ) and q ( v ) be two probability density functions then:
Revised: Let p ( v ) and q ( v ) be two probability density functions then:

Original: [Equation]
Revised: [Equation]

Original: Replacing p ( v ) and q ( v ) with π old ( · | s ) and π new ( · | s ) respectively, new lower bound follows:
Revised: Replacing p ( v ) and q ( v ) with π old ( · | s ) and π new ( · | s ) respectively, new lower bound follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) =
Revised: where (cid:15) =

Original: max s,a | A π ( s, a ) | ,
Revised: max s,a | A π ( s, a ) | ,

Original: [Equation]
Revised: [Equation]

Original: It is worth to note that H ( π old ( · | s ) ||
Revised: It is worth to note that H ( π old ( · | s ) ||

Original: π new ( · | s )) ≤ 1 .
Revised: π new ( · | s )) ≤ 1 .

Original:  H ELLINGER D ISTANCE C ONSTRAINED R EGRESSION (HDCR)
Revised:  H ELLINGER D ISTANCE C ONSTRAINED R EGRESSION (HDCR)

Original: We could use presented lower bound as in TRPO, but instead, we derive an ofﬂine regression algorithm by introducing the following optimization problem where µ is the sampling policy:
Revised: We could use presented lower bound as in TRPO, but instead, we derive an ofﬂine regression algorithm by introducing the following optimization problem where µ is the sampling policy:

Original: [Equation]
Revised: [Equation]

Original: Constructing Lagrangian, differentiating it with respect to π 
Revised: Constructing Lagrangian, differentiating it with respect to π , and solving for π gives us the following optimal policy (see Appendix B for derivation):

Original: [Equation]
Revised: [Equation]

Original: and solving for π gives us the following optimal policy (see Appendix B for derivation):
Revised: is a Lagrangian multiplier.

Original: [Equation]
Revised: [Equation]

Original: Constructing a regression problem of KL divergence between optimal policy π ∗ and current policy π and simplifying gives us the following supervised regression problem (see Appendix B for derivation): 
Revised: Constructing a regression problem of KL divergence between optimal policy π ∗ and current policy π and simplifying gives us the following supervised regression problem (see Appendix B for derivation): [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using notation of ABM paper (Siegel et al., 2020) ”advantage-weighting” function is f
Revised: Using notation of ABM paper (Siegel et al., 2020) ”advantage-weighting” function is f

Original: ( A ( s, a ))
Revised: ( A ( s, a ))

Original: = 1 ( β − A ( s,a )) 2 .
Revised: = 1 ( β − A ( s,a )) 2 .

Original: If we use HDCR with Experience Replay, in equation 14, we replace µ ( ·| s ) in expectation E a ∼ µ ( · | s ) and advantage function A µ ( s, a ) .
Revised: If we use HDCR with Experience Replay, in equation 14, we replace µ ( ·| s ) in expectation E a ∼ µ ( · | s ) and advantage function A µ ( s, a ) .

Original: Let Π =
Revised: Let Π =

Original: { π i , π i +1 , ..., π i + N } be a set of sampling policies from which actions were sampled, w ( π i ) probability of selecting policy
Revised: { π i , π i +1 , ..., π i + N } be a set of sampling policies from which actions were sampled, w ( π i ) probability of selecting policy

Original: π i , then:
Revised: π i , then:

Original: [Equation]
Revised: [Equation]

Original: The proof will repeat the proof for AWR with Experience Replay (Peng et al., 2019).
Revised: The proof will repeat the proof for AWR with Experience Replay (Peng et al., 2019).

Original: Practically we simply sample uniformly from the replay buffer and using a value function estimator.
Revised: Practically we simply sample uniformly from the replay buffer and using a value function estimator.

Original: This type of sampling provides us an approximation of expectation and state value function.
Revised: This type of sampling provides us an approximation of expectation and state value function.

Original: Let D denote a set of stored trajectories (replay buffer), A ( s, a ; φ ) is an advantage function parameterized by vector φ .
Revised: Let D denote a set of stored trajectories (replay buffer), A ( s, a ; φ ) is an advantage function parameterized by vector φ .

Original: The most popular method to obtain A ( s, a ; φ ) for ofﬂine learning is to use state-action function estimator Q ( s, a ; φ ) parameterized by φ :
Revised: The most popular method to obtain A ( s, a ; φ ) for ofﬂine learning is to use state-action function estimator Q ( s, a ; φ ) parameterized by φ :

Original: [Equation]
Revised: [Equation]

Original: Despite Q ( s, a ; φ ) being closer to the expectation of discounted return following policy π (because of ”taking” the ﬁrst action according to π rather then µ ), we found Monte-Carlo return more efﬁcient on tiny ofﬂine datasets.
Revised: Despite Q ( s, a ; φ ) being closer to the expectation of discounted return following policy π (because of ”taking” the ﬁrst action according to π rather then µ ), we found Monte-Carlo return more efﬁcient on tiny ofﬂine datasets.

Original: Greater performance using MC return can be explained by a lack of experience ”produced” by certain actions.
Revised: Greater performance using MC return can be explained by a lack of experience ”produced” by certain actions.

Original: Monte-Carlo estimation of A ( s, a ; φ ) can be described as follows where R D s t ,a t = (cid:80) Tl =

Revised: Monte-Carlo estimation of A ( s, a ; φ ) can be described as follows where R D s t ,a t = (cid:80) Tl =

Original: γ l r t + l and V ( s ; φ ) is a state value function estimator parameterized by vector φ :
Revised: γ l r t + l and V ( s ; φ ) is a state value function estimator parameterized by vector φ :

Original: [Equation]
Revised: [Equation]

Original: Also, we can use Generalized Advantage Estimation (Schulman et al., 2016), where γ ∈ [0 , 1] and δ φ k t is a one-step temporal difference for state s t calculated using old vector φ k :
Revised: Also, we can use Generalized Advantage Estimation (Schulman et al., 2016), where γ ∈ [0 , 1] and δ φ k t is a one-step temporal difference for state s t calculated using old vector φ k :

Original: [Equation]
Revised: [Equation]

Original: Finally, we propose the following reinforcement learning algorithm:
Revised: Finally, we propose the following reinforcement learning algorithm:

Original: Algorithm 1: Hellinger Distance Constrained Regression θ 1 ← random initial weights
Revised: Algorithm 1: Hellinger Distance Constrained Regression θ 1 ← random initial weights

Original: D ← ∅ for iteration k = 1 , ..., k max do add trajectories { τ i } sampled via π θ k to D φ k +1 ← arg min φ E s,
Revised: D ← ∅ for iteration k = 1 , ..., k max do add trajectories { τ i } sampled via π θ k to D φ k +1 ← arg min φ E s,

Original: a ∼D A 2 ( s, a ; φ ) θ k +1 ← arg max θ E s,a ∼D (cid:104) log π θ ( a | s ) 1 ( β − A ( s,a ; φ k ))
Revised: a ∼D A 2 ( s, a ; φ ) θ k +1 ← arg max θ E s,a ∼D (cid:104) log π θ ( a | s ) 1 ( β − A ( s,a ; φ k ))

Original: (cid:105) end
Revised: (cid:105) end

Original: [Figure][Figure] AWR HDCR
Revised: [Figure]

Original: Ant

Original: [Figure][Table] [Table] [Table]
Revised: [Figure]

Original: HalfCheetah

Original: [Table]
Revised: [Table]

Original: Hopper-

Original: [Table] [Table] [Table] Walker2d
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments, we evaluate the algorithm on MuJoCo (Todorov et al., 2012) tasks.
Revised: In our experiments, we evaluate the algorithm on MuJoCo (Todorov et al., 2012) tasks.

Revised: 5. T INY DATASETS

Original: We use setting inspired by Behavioral Modelling Priors for Ofﬂine Reinforcement Learning paper (Siegel et al., 2020). We have the following differences: on a tiny dataset, we use random actions sampled uniformly (random agent).
Revised: For evaluating on extremely small datasets we use setting inspired by Behavioral Modelling Priors for Ofﬂine Reinforcement Learning paper (Siegel et al., 2020) but instead of using actions from a behaviorial policy we use random actions while generating buffer.

Original: First, we collect 2048 timestamps or more, until episode termination (whichever occurred later), from each of 5 seeds using random actions.
Revised: First, we collect 2048 timestamps or more, until episode termination (whichever occurred later), from each of 5 seeds using random actions.

Original: Then we load collected trajectories to a replay buffer for the agent training.
Revised: Then we load collected trajectories to a replay buffer for the agent training.

Original: Separate networks with the same architecture (except the last layer) represent the policy and value function and consist of 2 hidden layers of 256 ELU units.
Revised: Separate networks with the same architecture (except the last layer) represent the policy and value function and consist of 2 hidden layers of 256 ELU units.

Original: Each train iteration uses only old data obtained by random agents.
Revised: Each train iteration uses only old data obtained by random agents.

Original: Each iteration, the value function is updating withgradient steps and policy with 50 steps using a uniformly sampled batches of 512 samples using all data from the replay buffer.
Revised: Each iteration, the value function is updating withgradient steps and policy with 50 steps using a uniformly sampled batches of 512 samples using all data from the replay buffer.

Original: Learning rates for Adam optimizer are 2 × 10 − 3 and 2 × 10 − 4 for critic and actor, respectively.
Revised: Learning rates for Adam optimizer are 2 × 10 − 3 and 2 × 10 − 4 for critic and actor, respectively.

Original: We compare 3 different ”advantage-weight” functions f ( A ( s, a )) :
Revised: We compare 3 different ”advantage-weight” functions f ( A ( s, a )) :

Original: • Hellinger Distance Constrained Regression, where f ( A ( s, a )) =
Revised: • Hellinger Distance Constrained Regression, where f ( A ( s, a )) =

Original: 1 ( β − A ( s,a )) 2 ; • Advantage-weighted Behavior Model, where f ( A ( s, a ))
Revised: 1 ( β − A ( s,a )) 2 ; • Advantage-weighted Behavior Model, where f ( A ( s, a ))

Original: =
Revised: =

Original: I A ( s,a ) > 0 , I x> 

Revised: I A ( s,a ) > 0 , I x>

Original: = 

Revised: =

Original: if x >otherwise I x> 0 = 0 ; • Advantage-Weighted Regression, where f ( A ( s, a )) =
Revised: if x >otherwise I x> 0 = 0 ; • Advantage-Weighted Regression, where f ( A ( s, a )) =

Original: exp( 1 β A ( s, a )) .
Revised: exp( 1 β A ( s, a )) .

Original: AWR method uses β = 1 . 0 as it is in implementation released by authors.
Revised: AWR method uses β = 1 . 0 as it is in implementation released by authors.

Original: HDCR uses β = 1 . 0 .
Revised: HDCR uses β = 1 . 0 .

Original: For
Revised: For

Original: TD( λ ) we use λ = 0 .
Revised: TD( λ ) we use λ = 0 .

Original: On simple tasks as Hopper-v2, all methods are able to learn (Figure 1), and HDCR shows slightly better results (Table 1).
Revised: On simple tasks as Hopper-v2, all methods are able to learn (Figure 1), and HDCR shows slightly better results (Table 1).

Original: While on difﬁcult tasks as Ant-v2, all algorithms do not improve their results through iterations.
Revised: While on difﬁcult tasks as Ant-v2, all algorithms do not improve their results through iterations.

Original: Moreover, evaluation returns decrease.
Revised: Moreover, evaluation returns decrease.

Revised: 5. L ARGE DATASETS

Original: Next, we perform training with the same setting on 16k
Revised: Next, we perform tests using buffers with the size of 100k timesteps.

Original: ﬁrst timestamps obtained from a single onpolicy training by HDCR.
Revised: Buffer is ﬁlled by running a pretrained behavioral policy.

Original: Figure 2 shows that agents can learn on 2 out of 4 environments.
Revised: This setting replicates the setting from Off-Policy Deep Reinforce- [Figure] [Figure]

Original: Onout of 4 environments, HDCR shows (Table 2) better results than ABM and AWR.
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: ment Learning without Exploration (Fujimoto et al., 2019) paper.

Original: D ISCUSSION
Revised: Therefore we also provide results of BCQ method achieved by authors’ implementation trained from the same datasets. For calculating advantage we use equation 16 where we approximate integral by taking mean of 10 Q-values obtained by 10 actions sampled from the policy. While BCQ outperforms all the presented methods, it uses a gradient of Q-value function in actor training, which provides better generalization. This provides better results on evaluation but affects stability and performance.

Revised: Against other methods that update the policy function directly, HDCR shows better results on 3 environments out of 4 (Figure 2 and Table 2).

Revised:  D ISCUSSION

Original: We theoretically proved that Hellinger distance improves the lower bound of expected discounted return compared to Kullback-Leibler divergence and proposed a simple off-policy reinforcement learning method that uses Hellinger distance as a constraint.
Revised: We theoretically proved that Hellinger distance improves the lower bound of expected discounted return compared to Kullback-Leibler divergence and proposed a simple off-policy reinforcement learning method that uses Hellinger distance as a constraint.

Original: The expected discounted return for a new policy now has a policy-independent lower bound.
Revised: The expected discounted return for a new policy now has a policy-independent lower bound.

Original: This bound guarantees that return will not decrease in ”one” shot.
Revised: This bound guarantees that return will not decrease in ”one” shot.

Original: Experiments show that HDCR outperforms both ABM and AWR on tiny datasets obtained by random agents.
Revised: Experiments show that HDCR outperforms both ABM and AWR on tiny datasets obtained by random agents.

Original: This performance proves the efﬁciency of using Hellinger distance by allowing bigger step sizes, retaining lower bound.
Revised: This performance proves the efﬁciency of using Hellinger distance by allowing bigger step sizes, retaining lower bound.

Original: On bigger datasets, HDCR shows comparable or better results compared to AWR and ABM training from ofﬂine data collected by HDCR on ﬁrst on-policy train iterations.
Revised: On bigger datasets, HDCR shows comparable or better results than AWR and ABM.

Original: A T HEOREM 1 PROOF
Revised: A T HEOREM 1 PROOF

Original: Let (Ω , A ) be a measurable space, P and Q be two probability measures on that space, v be a σ -ﬁnite measure on (Ω , A ) such that P (cid:28) v ( P ( A ) = 0 for any A ∈
Revised: Let (Ω , A ) be a measurable space, P and Q be two probability measures on that space, v be a σ -ﬁnite measure on (Ω , A ) such that P (cid:28) v ( P ( A ) = 0 for any A ∈

Original: A such that µ ( A ) = 0 ) and
Revised: A such that µ ( A ) = 0 ) and

Original: Q (cid:28) v .
Revised: Q (cid:28) v .

Original: And let us denote Radon–Nikodym derivatives (in our derivations we can assume them as probability density functions) as follows:
Revised: And let us denote Radon–Nikodym derivatives (in our derivations we can assume them as probability density functions) as follows:

Original: [Equation]
Revised: [Equation]

Original: So, p and q satisfy following conditions:
Revised: So, p and q satisfy following conditions:

Original: [Equation]
Revised: [Equation]

Original: Then we deﬁne distances between probability measures:
Revised: Then we deﬁne distances between probability measures:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Following Lemmas will be used in Theorem 1 proof:
Revised: Following Lemmas will be used in Theorem 1 proof:

Original: Lemma 1.
Revised: Lemma 1.

Original: Given two probability distributions p and q total variance divergence can be calculated as follows:
Revised: Given two probability distributions p and q total variance divergence can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let B = { v : P ( v ) > Q ( v ) } ,
Revised: Let B = { v : P ( v ) > Q ( v ) } ,

Original: B c
Revised: B c

Original: = { v : P ( v ) ≤ Q ( v ) } and A ∈ A :
Revised: = { v : P ( v ) ≤ Q ( v ) } and A ∈ A :

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.
Revised: Lemma 3.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Rewriting left part in 4 integrals over following sets { max( p ( v ) , q ( v ))
Revised: Rewriting left part in 4 integrals over following sets { max( p ( v ) , q ( v ))

Original: [Equation]
Revised: [Equation]

Original: { max( p ( v ) , q ( v )) =
Revised: { max( p ( v ) , q ( v )) =

Original: q ( v ) } , { min( p ( v ) , q ( v ))
Revised: q ( v ) } , { min( p ( v ) , q ( v ))

Original: =
Revised: =

Original: q ( v ) } and { min( p ( v ) , q ( v ))
Revised: q ( v ) } and { min( p ( v ) , q ( v ))

Original: =
Revised: =

Original: q ( v ) } and stacking integrals back gives us P (Ω) + Q (Ω) = 2 .
Revised: q ( v ) } and stacking integrals back gives us P (Ω) + Q (Ω) = 2 .

Original: Theorem 1.
Revised: Theorem 1.

Original: For any two probability density functions p and q
Revised: For any two probability density functions p and q

Original: [Equation]
Revised: [Equation]

Original: the following double inequality is true:
Revised: the following double inequality is true:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First inequality can be proved as follows:
Revised: First inequality can be proved as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof of the second inequality:
Revised: Proof of the second inequality:

Original: [Equation]
Revised: [Equation]

Original: B HDCR DERIVATION
Revised: B HDCR DERIVATION

Original: Given optimization problem:
Revised: Given optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Constructing Lagrangian where α : S → R is a function for obtaining Lagrange multiplier for every state, β is also a Lagrange multiplier:
Revised: Constructing Lagrangian where α : S → R is a function for obtaining Lagrange multiplier for every state, β is also a Lagrange multiplier:

Original: [Equation]
Revised: [Equation]

Original: Differentiating with respect to π ( a | s ) gives us following result:
Revised: Differentiating with respect to π ( a | s ) gives us following result:

Original: [Equation]
Revised: [Equation]

Original: Solving for π ( a | s ) :
Revised: Solving for π ( a | s ) :

Original: [Equation]
Revised: [Equation]

Original: Substituting π ( a | s ) = µ ( ·| s ) and taking expectation over actions taken according to µ gives us expression for α s :
Revised: Substituting π ( a | s ) = µ ( ·| s ) and taking expectation over actions taken according to µ gives us expression for α s :

Original: [Equation]
Revised: [Equation]

Original: Then optimal policy π ∗ ( a | s ) can be written as follows:
Revised: Then optimal policy π ∗ ( a | s ) can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: To obtain regression problem we construct Kullback–Leibler divergence between optimal policy π ∗ and current policy π :
Revised: To obtain regression problem we construct Kullback–Leibler divergence between optimal policy π ∗ and current policy π :

Original: [Equation]
Revised: [Equation]

Original: Regression problem follows:
Revised: Regression problem follows:

Original: [Equation]
Revised: [Equation]