Original: M AKING S ENSE OF R EINFORCEMENT L EARNING AND P ROBABILISTIC I NFERENCE
Revised: M AKING S ENSE OF R EINFORCEMENT L EARNING AND P ROBABILISTIC I NFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning (RL) combines a control problem with statistical estimation: the system dynamics are not known to the agent, but can be learned through experience.
Revised: Reinforcement learning (RL) combines a control problem with statistical estimation: the system dynamics are not known to the agent, but can be learned through experience.

Original: A recent line of research casts ‘RL as inference’ and suggests a particular framework to generalize the RL problem as probabilistic inference.
Revised: A recent line of research casts ‘RL as inference’ and suggests a particular framework to generalize the RL problem as probabilistic inference.

Original: Our paper surfaces  key shortcomings in that approach, and clariﬁes the sense in which
Revised: Our paper surfaces a key shortcoming in that approach, and clariﬁes the sense in which

Original: RL can be coherently cast as an inference problem.
Revised: RL can be coherently cast as an inference problem.

Original: In particular, an RL agent must consider the effects of its actions upon future rewards and observations: the exploration-exploitation tradeoff.
Revised: In particular, an RL agent must consider the effects of its actions upon future rewards and observations: the exploration-exploitation tradeoff.

Original: In all but the most simple settings, the resulting inference is computationally intractable so that practical RL algorithms must resort to approximation.
Revised: In all but the most simple settings, the resulting inference is computationally intractable so that practical RL algorithms must resort to approximation.

Original: We show that the popular ‘RL as inference’ approximation can perform poorly in even the simplest settings.
Revised: We demonstrate that the popular ‘RL as inference’ approximation can perform poorly in even very basic problems.

Original: Despite this, we demonstrate that with a small modiﬁcation theRL as inference framework  can provably perform well, and we connect the resulting algorithm with Thompson sampling and the recently proposed K-learning algorithm.
Revised: However, we show that with a small modiﬁcation the framework does yield algorithms that can provably perform well, and we show that the resulting algorithm is equivalent to the recently proposed K-learning, which we further connect with Thompson sampling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Probabilistic inference is a procedure of making sense of uncertain data using Bayes’ rule.
Revised: Probabilistic inference is a procedure of making sense of uncertain data using Bayes’ rule.

Original: The optimal control problem is to take actions in a known system in order to maximize the cumulative rewards through time.
Revised: The optimal control problem is to take actions in a known system in order to maximize the cumulative rewards through time.

Original: Probabilistic graphical models (PGMs) offer a coherent and ﬂexible language to specify causal relationships, for which a rich literature of learning and inference techniques have developed (Koller & Friedman, 2009).
Revised: Probabilistic graphical models (PGMs) offer a coherent and ﬂexible language to specify causal relationships, for which a rich literature of learning and inference techniques have developed (Koller & Friedman, 2009).

Original: Although control dynamics might also be encoded as a PGM, the relationship between action planning and probabilistic inference is not immediately clear.
Revised: Although control dynamics might also be encoded as a PGM, the relationship between action planning and probabilistic inference is not immediately clear.

Original: For inference, it is typically enough to specify the system and pose the question, and the objectives for learning emerge automatically.
Revised: For inference, it is typically enough to specify the system and pose the question, and the objectives for learning emerge automatically.

Original: In control, the system and objectives aretrivially known, but the question of how to approach a solution may remain extremely complex (Bertsekas, 2005).
Revised: In control, the system and objectives are known, but the question of how to approach a solution may remain extremely complex (Bertsekas, 2005).

Original: Perhaps surprisingly, there is a deep sense in which inference and control can represent a dual view of the same problem.
Revised: Perhaps surprisingly, there is a deep sense in which inference and control can represent a dual view of the same problem.

Original: This relationship is most clearly stated in the case of linear quadratic systems, where the Ricatti equations relate the optimal control policy in terms of the system dynamics (Welch et al., 1995).
Revised: This relationship is most clearly stated in the case of linear quadratic systems, where the Ricatti equations relate the optimal control policy in terms of the system dynamics (Welch et al., 1995).

Original: In fact, this connection extends to a wide range of systems, where control tasks can be related to a dual inference problem through rewards as exponentiated probabilities in a distinct, but coupled, PGM (Todorov, 2007; 2008).
Revised: In fact, this connection extends to a wide range of systems, where control tasks can be related to a dual inference problem through rewards as exponentiated probabilities in a distinct, but coupled, PGM (Todorov, 2007; 2008).

Original: A great beneﬁt of this connection is that it can allow the tools of inference to make progress in control problems, and vice-versa.
Revised: A great beneﬁt of this connection is that it can allow the tools of inference to make progress in control problems, and vice-versa.

Original: In both cases the connections provide new insights, inspire new algorithms and enrich our understanding (Toussaint & Storkey, 2006; Ziebart et al., 2008; Kappen et al., 2012).
Revised: In both cases the connections provide new insights, inspire new algorithms and enrich our understanding (Toussaint & Storkey, 2006; Ziebart et al., 2008; Kappen et al., 2012).

Original: Reinforcement learning (RL) is the problem of learning to control an unknown system (Sutton &
Revised: Reinforcement learning (RL) is the problem of learning to control an unknown system (Sutton &

Original: Barto, 2018).
Revised: Barto, 2018).

Original: Like the control setting, an RL agent should take actions to maximize its cumulative rewards through time.
Revised: Like the control setting, an RL agent should take actions to maximize its cumulative rewards through time.

Original: Like the inference problem, the agent is initially uncertain of the system dynamics, but can learn through the states and actions it observes.
Revised: Like the inference problem, the agent is initially uncertain of the system dynamics, but can learn through the transitions it observes.

Original: This leads to a fundamental tradeoff: the agent may be able to improve its understanding through exploring poorly-understood states and actions, but it may be able to attain higher immediate reward through exploiting its existing knowledge (Kearns & Singh, 2002).
Revised: This leads to a fundamental tradeoff: the agent may be able to improve its understanding through exploring poorly-understood states and actions, but it may be able to attain higher immediate reward through exploiting its existing knowledge (Kearns & Singh, 2002).

Original: In many ways, RL combines control and inference into a general framework for decision making under uncertainty.
Revised: In many ways, RL combines control and inference into a general framework for decision making under uncertainty.

Original: Although there has been ongoing research in this area for many decades, there has been a recent explosion of interest as RL techniques have made high-proﬁle breakthroughs in grand challenges of artiﬁcial intelligence research (Mnih et al., 2013; Silver et al., 2016).
Revised: Although there has been ongoing research in this area for many decades, there has been a recent explosion of interest as RL techniques have made high-proﬁle breakthroughs in grand challenges of artiﬁcial intelligence research (Mnih et al., 2013; Silver et al., 2016).

Original: A popular line of research has sought to cast ‘RL as inference’, mirroring the dual relationship for control in known systems.
Revised: A popular line of research has sought to cast ‘RL as inference’, mirroring the dual relationship for control in known systems.

Original: This approach is most clearly stated in the tutorial and review of Levine (2018), and provides a key reference for research in this ﬁeld.
Revised: This approach is most clearly stated in the tutorial and review of Levine (2018), and provides a key reference for research in this ﬁeld.

Original: It suggests that a generalization of the
Revised: It suggests that a generalization of the

Original: RL problem can be cast as probabilistic inference through inference over exponentiated rewards, in a continuation of previous work in optimal control (Todorov, 2009).
Revised: RL problem can be cast as probabilistic inference through inference over exponentiated rewards, in a continuation of previous work in optimal control (Todorov, 2009).

Original: This perspective promises several beneﬁts: a probabilistic perspective on rewards, the ability to apply powerful inference algorithms to solve RL problems and a natural exploration strategy.
Revised: This perspective promises several beneﬁts: a probabilistic perspective on rewards, the ability to apply powerful inference algorithms to solve RL problems and a natural exploration strategy.

Original: In this paper we will outline  important ways in which this perspective is incomplete.
Revised: In this paper we will outline an important way in which this perspective is incomplete.

Original: These shortcomings ultimately result in algorithms that can perform poorly in even very simple decision problems.
Revised: This shortcoming ultimately results in algorithms that can perform poorly in even very simple decision problems.

Revised: Importantly, these are not simply technical issues that show up in some edge cases, but fundamental failures of this approach that arise in even the most simple decision problems.

Original: In this paper we revisit an alternative framing of ‘RL as inference’.
Revised: In this paper we revisit an alternative framing of ‘RL as inference’.

Original: In fact, we show that the original RL problem was already an inference problem all along.
Revised: In fact, we show that the original RL problem was already an inference problem all along.

Original: Importantly, this inference problem includes inference over the agent’s future actions and observations.
Revised: Importantly, this inference problem includes inference over the agent’s future actions and observations.

Original: Of course, this perspective is not new, and has long been known as simply the Bayes-optimal solution, see, e.g. , Ghavamzadeh et al.
Revised: Of course, this perspective is not new, and has long been known as simply the Bayes-optimal solution, see, e.g. , Ghavamzadeh et al.

Original: The problem is that, due to the exponential lookahead, this inference problem is fundamentally intractable for all but the simplest problems Gittins (1979).
Revised: The problem is that, due to the exponential lookahead, this inference problem is fundamentally intractable for all but the simplest problems (Gittins, 1979).

Original: For this reason, RL research focuses on computationally efﬁcient approaches that maintain a level of statistical efﬁciency (Osband et al., 2017).
Revised: For this reason, RL research focuses on computationally efﬁcient approaches that maintain a level of statistical efﬁciency (Osband et al., 2017).

Original: We provide a review of the RL problem in Section 2, together with a simple and coherent framing of ‘RL as inference’.
Revised: We provide a review of the RL problem in Section 2, together with a simple and coherent framing of RL as probabilistic inference.

Original: In Section 3 we present three approximations to the intractable Bayes-optimal policy.
Revised: In Section 3 we present three approximations to the intractable Bayes-optimal policy.

Original: We begin with the celebrated Thompson sampling algorithm, then we review the popular ‘RL as inference’ framing, as presented by Levine (2018), and highlight some clear and simple shortcomings in this approach.
Revised: We begin with the celebrated Thompson sampling algorithm, then we review the popular ‘RL as inference’ framing, as presented by Levine (2018), and highlight a clear and simple shortcoming in this approach.

Original: Finally, we review K-learning (O’Donoghue, 2018), which we reinterpret as a modiﬁcation to the RL as inference framework that provides a principled approach to the statistical inference problem, as well as a presenting a relationship with Thompson sampling.In
Revised: Finally, we review K-learning (O’Donoghue, 2018), which we re-interpret as a modiﬁcation to the RL as inference framework that provides a principled approach to the statistical inference problem, as well as a presenting a relationship with Thompson sampling.

Original:  Section 4 we present computational studies that support our claims.
Revised: In Section 4 we present computational studies that support our claims.

Original:  R EINFORCEMENT LEARNING
Revised:  R EINFORCEMENT LEARNING

Original: We consider the problem of an agent taking actions in an unknown environment in order to maximize cumulative rewards through time.
Revised: We consider the problem of an agent taking actions in an unknown environment in order to maximize cumulative rewards through time.

Original: For simplicity, this paper will model the environment as a ﬁnite horizon, discrete Markov Decision Process (MDP) M =
Revised: For simplicity, this paper will model the environment as a ﬁnite horizon, discrete Markov Decision Process (MDP) M =

Original: ( S , A , R , P , H, ρ ) .
Revised: ( S , A , R , P , H, ρ ) .

Original: 2 Here S =
Revised: 2 Here S =

Original: { 1 , .., S } is the state space,
Revised: { 1 , .., S } is the state space,

Original: A = { 1 , .., A } is the action space and each episode is of ﬁxed length H ∈ N .
Revised: A = { 1 , .., A } is the action space and each episode is of ﬁxed length H ∈ N .

Original: Each episode (cid:96) ∈ N begins with state s 0 ∼ ρ then for timesteps h = 0 , .., H − 1 the agent selects action a h , observes transition s h +1 with probability P
Revised: Each episode (cid:96) ∈ N begins with state s 0 ∼ ρ then for timesteps h = 0 , .., H − 1 the agent selects action a h , observes transition s h +1 with probability P

Original: ( s h +1 , s h , a h ) ∈ [0 , 1] and receives reward r h +1 ∼ 
Revised: ( s h +1 , s h , a h ) ∈ [0 , 1] and receives reward r h +1 ∼ R

Original: R ( s h , a h ) , where we denote by µ ( s h , a h ) =
Revised: ( s h , a h ) , where we denote by µ ( s h , a h ) =

Original: E r h +1 the mean reward.
Revised: E r h +1 the mean reward.

Original: We deﬁne a policy π to be a mapping from S to  distributions over A and write Π for the space of all policies.
Revised: We deﬁne a policy π to be a mapping from S to probability distributions over A and write Π for the space of all policies.

Original: For any timestep t = ( (cid:96), h ) , we deﬁne F t = ( s 00 , a 00 , r 01 , .., s h − 1 , a h − 1 , r h ) to be the sequence of observations made before time t .
Revised: For any timestep t = ( (cid:96), h ) , we deﬁne F t = ( s 00 , a 00 , r 01 , .., s (cid:96)h − 1 , a (cid:96)h − 1 , r (cid:96)h ) to be the sequence of observations made before time t .

Original: An RL algorithm maps histories to policies π t = alg
Revised: An RL algorithm maps histories to policies π t = alg

Original: ( S , A , F t ) .
Revised: ( S , A , F t ) .

Original: Our goal in the design of RL algorithms is to obtain good performance (cumulative rewards) for an unknown M ∈ M , where M is some family of possible environments.
Revised: Our goal in the design of RL algorithms is to obtain good performance (cumulative rewards) for an unknown M ∈ M , where M is some family of possible environments.

Original: Note that this is a different problem from typical ‘optimal control’, that seeks to optimize performance for one particular known
Revised: Note that this is a different problem from typical ‘optimal control’, that seeks to optimize performance for one particular known

Original: MDP M ; although you might still fruitfully apply an RL algorithm to solve problems of that type.
Revised: MDP M ; although you might still fruitfully apply an RL algorithm to solve problems of that type.

Original: For any environment M and any policy π we can deﬁne the action-value function
Revised: For any environment M and any policy π we can deﬁne the action-value function,

Original: [Equation]
Revised: [Equation]

Original: Where the expectation in (1) is taken with respect to the action selection a j for j > h from the policy π and evolution of the ﬁxed MDP M .
Revised: Where the expectation in (1) is taken with respect to the action selection a j for j > h from the policy π and evolution of the ﬁxed MDP M .

Original: We deﬁne the value function V M,πh
Revised: We deﬁne the value function V M,πh

Original: [Equation]
Revised: [Equation]

Original: and write V M,(cid:63)h
Revised: and write Q M,(cid:63)h

Original: [Equation]
Revised: [Equation]

Original:  for the optimal value over policies.
Revised: s [Equation] a ) for the optimal Q-values over policies, and the optimal value function is given by V M [Equation]

Original:  In order to compare algorithm performance across different environments, it is natural to normalize in terms of the regret , or shortfall in cumulative rewards relative to the optimal value,
Revised: h [Equation] In order to compare algorithm performance across different environments, it is natural to normalize in terms of the regret , or shortfall in cumulative rewards relative to the optimal value,

Original: [Equation]
Revised: [Equation]

Original: This quantity depends on the unknown MDP M ,  but the expectations are taken with respect to the dynamics of M and the learning algorithm alg .
Revised: This quantity depends on the unknown MDP M , which is ﬁxed from the start and kept the same throughout, but the expectations are taken with respect to the dynamics of M and the learning algorithm alg .

Original: For any particular MDP M , the optimal regret of zero can be attained by the non-learning algorithm alg M that returns the optimal policy for M .
Revised: For any particular MDP M , the optimal regret of zero can be attained by the nonlearning algorithm alg M that returns the optimal policy for M .

Original: In order to assess the quality of a reinforcement learning algorithm, which is designed to work across some family of M ∈ M , we need some method to condense performance over a set to a single number.
Revised: In order to assess the quality of a reinforcement learning algorithm, which is designed to work across some family of M ∈ M , we need some method to condense performance over a set to a single number.

Original: There are two main approaches to this:
Revised: There are two main approaches to this:

Original: [Equation]
Revised: [Equation]

Original: where φ is a prior over the famliy M .
Revised: where φ is a prior over the family M .

Original: These differing objectives are often framed as Bayesian (average-case) (3) and frequentist (worst-case) (4) RL.
Revised: These differing objectives are often framed as Bayesian (average-case) (3) and frequentist (worst-case) (4) RL 3 .

Original: Although these two settings are typically studied in isolation, it should be clear that they are intimately related through the choice of M and φ .
Revised: Although these two settings are typically studied in isolation, it should be clear that they are intimately related through the choice of M and φ .

Original: Our next section will investigate what it would mean to ‘solve’ the RL problem.
Revised: Our next section will investigate what it would mean to ‘solve’ the RL problem.

Original: Importantly, we show that both frequentist and Bayesian perspectives already amount to a problem in probabilistic inference, without the need for additional re-interpretation.
Revised: Importantly, we show that both frequentist and Bayesian perspectives already amount to a problem in probabilistic inference, without the need for additional re-interpretation.

Original: 2. S OLVING THE RL PROBLEM THROUGH PROBABILISTIC INFERENCE
Revised: 2. S OLVING THE RL PROBLEM THROUGH PROBABILISTIC INFERENCE

Original: If you want to ‘solve’ the RL problem, then formally the objective is clear: ﬁnd the RL algorithm that minimizes your chosen objective, (3) or (4).
Revised: If you want to ‘solve’ the RL problem, then formally the objective is clear: ﬁnd the RL algorithm that minimizes your chosen objective, (3) or (4).

Original: To anchor our discussion, we introduce a simple decision problem designed to highlight some key aspects of reinforcement learning.
Revised: To anchor our discussion, we introduce a simple decision problem designed to highlight some key aspects of reinforcement learning.

Original: We will revisit this problem setting as we discuss approximations to the optimal policy.
Revised: We will revisit this problem setting as we discuss approximations to the optimal policy.

Original: Problem 1 (One unknown action) .
Revised: Problem 1 (One unknown action) .

Original: Fix N ∈ N ≥ 3 , (cid:15) > 

Revised: Fix N ∈ N ≥ 3 , (cid:15) >

Original: and deﬁne M N,(cid:15)
Revised: and deﬁne M N,(cid:15)

Original: =
Revised: =

Original: { M + N,(cid:15) , M − N,(cid:15) } .
Revised: { M + N,(cid:15) , M − N,(cid:15) } .

Original: Both
Revised: Both

Original: M + and M − share S = { 1 } , H = 1 and A = { 1 , .., N } ; they only differ through their rewards: 
Revised: M + and M − share S = { 1 } , H = 1 and A = { 1 , .., N } ; they only differ through their rewards: R + (1) = 1 , R + (2) = +2 ,

Original: [Equation]
Revised: R + ( a ) = 1 − (cid:15) for a = 3 , .., N, R − (1) = 1 , R − (2) = − 2 , R − ( a ) = 1 − (cid:15) for a = 3 , .., N.

Original: Where R ( a ) = x ∈ R is a shorthand for deterministic reward of x when choosing action a .
Revised: Where R ( a ) = x ∈ R is a shorthand for deterministic reward of x when choosing action a .

Original: Problem 1 is extremely simple, it involves no generalization and no long-term consequences: it is an independent bandit problem with only one unknown action.
Revised: Problem 1 is extremely simple, it involves no generalization and no long-term consequences: it is an independent bandit problem with only one unknown action.

Original: For known M + , M − the optimal policy is trivial: choose a t = 2 in M + and a t = 1 in M − for all t .
Revised: For known M + , M − the optimal policy is trivial: choose a t = 2 in M + and a t = 1 in M − for all t .

Original: An RL agent faced with unknown
Revised: An RL agent faced with unknown

Original: M ∈ M should attempt to optimize the RL objectives (3) or (4).
Revised: M ∈ M should attempt to optimize the RL objectives (3) or (4).

Original: Unusually, and only because
Revised: Unusually, and only because

Original: Problem 1 is so simple, we can actually compute the optimal solutions to both in terms of L (the total number of episodes) and φ = ( p + , p − ) where p + = P ( M = M + ) , the probability of being in
Revised: Problem 1 is so simple, we can actually compute the optimal solutions to both in terms of L (the total number of episodes) and φ = ( p + , p − ) where p + = P ( M = M + ) , the probability of being in

Original: M + .
Revised: M + .

Original: For L > 3 an optimal minimax RL algorithm is to ﬁrst choose a 0 = 2 and observe r 1 .
Revised: For L > 3 an optimal minimax RL algorithm is to ﬁrst choose a 0 = 2 and observe r 1 .

Original: If r 1 = 2 then you know you are in M + so pick a t = 2 for all t
Revised: If r 1 =then you know you are in M + so pick a t = 2 for all t

Original: = 1 , 2 .. , for Regret( L ) = [-
-]
Revised: = 1 , 2 .. , for Regret( L ) = [-
-]0 .

Original: for all L = 1 , 2 , .. .

Original: If r 1 = − 2 then you know you are in M − so pick a t = 1 for all t
Revised: If r 1 = − 2 then you know you are in M − so pick a t = 1 for all t

Original: = 1 , 2 .. , for Regret( L ) = 3for all L = 1 , 2 , .. .
Revised: = 1 , 2 .. , for Regret( L ) = 3 .

Original: The minimax regret of this algorithm is 3 , which cannot be bested by any algorithm.
Revised: The minimax regret of this algorithm is 3 , which cannot be bested by any algorithm.

Original: Actually, the same RL algorithm is also Bayes -optimal for any φ
Revised: Actually, the same RL algorithm is also Bayes -optimal for any φ

Original: =
Revised: =

Original: ( p + , p − ) provided p + L > 3 .
Revised: ( p + , p − ) provided p + L > 3 .

Original: This relationship is not a coincidence.
Revised: This relationship is not a coincidence.

Original: All admissible solutions to the minimax problem (4) arein given by solutions to the average-case (3) for some ‘worst-case’ prior ˜ φ (Wald, 1950).
Revised: All admissible solutions to the minimax problem (4) are given by solutions to the average-case (3) for some ‘worst-case’ prior ˜ φ (Wald, 1950).

Original: As such, for ease of exposition, our discussion will focus on the Bayesian (or average-case) setting.
Revised: As such, for ease of exposition, our discussion will focus on the Bayesian (or average-case) setting.

Original: However, readers should understand that the same arguments apply to the minimax objective.
Revised: However, readers should understand that the same arguments apply to the minimax objective.

Original: In Problem 1, the key probabilistic inference the agent must consider is the effects of it own actions upon the future rewards, or whether it has chosen action 2 .
Revised: In Problem 1, the key probabilistic inference the agent must consider is the effects of it own actions upon the future rewards, i.e. , whether it has chosen action 2 .

Original: More generally, where actions are independent and episode length H
Revised: Slightly more generally, where actions are independent and episode length H

Original: = 1 , the optimal RL algorithm can be computed via Gittins indices, but these  are very much the exception (Gittins, 1979).
Revised: = 1 , the optimal RL algorithm can be computed via Gittins indices, but these problems are very much the exception (Gittins, 1979).

Original: In problems with generalization or long-term consequences, computing the Bayes-optimal solution is computationally intractable.
Revised: In problems with generalization or long-term consequences, computing the Bayes-optimal solution is computationally intractable.

Original: One example of an algorithm that converges to Bayes-optimal solution in the limit of inﬁnite computation is given by Bayes-adaptive Monte-Carlo Planning (Guez et al., 2012).
Revised: One example of an algorithm that converges to Bayes-optimal solution in the limit of inﬁnite computation is given by Bayes-adaptive Monte-Carlo Planning (Guez et al., 2012).

Original: The problem is that, even for very simple problems, the lookahead tree of interactions between actions, observations and algorithmic updates grows exponentially in the search depth (Strehl et al., 2006).
Revised: The problem is that, even for very simple problems, the lookahead tree of interactions between actions, observations and algorithmic updates grows exponentially in the search depth (Strehl et al., 2006).

Original: Worse still, direct computational approximations to the Bayes-optimal solution can fail exponentially badly should they fall short of the required computation (Munos, 2014).
Revised: Worse still, direct computational approximations to the Bayes-optimal solution can fail exponentially badly should they fall short of the required computation (Munos, 2014).

Original: As a result, research in reinforcement learning amounts to trying to ﬁnd computationally tractable approximations to (3), that maintain some degree of statistical efﬁciency.
Revised: As a result, research in reinforcement learning amounts to trying to ﬁnd computationally tractable approximations to the Bayes-optimal policy that maintain some degree of statistical efﬁciency.

Original:  A PPROXIMATIONS FOR COMPUTATIONAL AND STATISTICAL EFFICIENCY
Revised:  A PPROXIMATIONS FOR COMPUTATIONAL AND STATISTICAL EFFICIENCY

Original: The exponential explosion of future actions and observations means the solving for Bayes-optimal solution is computationally intractable.
Revised: The exponential explosion of future actions and observations means the solving for Bayes-optimal solution is computationally intractable.

Original: To counter this, most computationally efﬁcient approaches to RL simplify the problem at time t to only consider inference over the data F t that has been gathered prior to time t .
Revised: To counter this, most computationally efﬁcient approaches to RL simplify the problem at time t to only consider inference over the data F t that has been gathered prior to time t .

Original: The most common family of these algorithms are ‘certainty equivalent’ (under an identity utility): they take a point estimate for their best guess of the environment ˆ M , and try to optimize their control given these estimates V ˆ M, ∗ .
Revised: The most common family of these algorithms are ‘certainty equivalent’ (under an identity utility): they take a point estimate for their best guess of the environment ˆ M , and try to optimize their control given these estimates V ˆ M,(cid:63) .

Original: Typically, these algorithms are used in conjunction with some dithering scheme for random action selection ( e.g. , epsilon-greedy), to mitigate premature and suboptimal convergence (Watkins, 1989).
Revised: Typically, these algorithms are used in conjunction with some dithering scheme for random action selection ( e.g. , epsilon-greedy), to mitigate premature and suboptimal convergence (Watkins, 1989).

Original: However, since these algorithms do not prioritize their exploration, they may take exponentially long to ﬁnd the optimal policy (Osband et al., 2014).
Revised: However, since these algorithms do not prioritize their exploration, they may take exponentially long to ﬁnd the optimal policy (Osband et al., 2014).

Original: In order for an RL algorithm to be statistically efﬁcient, it must consider the value of information.
Revised: In order for an RL algorithm to be statistically efﬁcient, it must consider the value of information.

Original: To do this, an agent must ﬁrst maintain some notion of epistemic uncertainty, so that it can direct its exploration towards states and actions that it does not understand well (O’Donoghue et al., 2018).
Revised: To do this, an agent must ﬁrst maintain some notion of epistemic uncertainty, so that it can direct its exploration towards states and actions that it does not understand well (O’Donoghue et al., 2018).

Original: Here again, probabilistic inference ﬁnds a natural home in RL: we should build up posterior estimates for the unknown problem parameters, and use this distribution to drive efﬁcient exploration.
Revised: Here again, probabilistic inference ﬁnds a natural home in RL: we should build up posterior estimates for the unknown problem parameters, and use this distribution to drive efﬁcient exploration.

Original: 3. T HOMPSON SAMPLING
Revised: 3. T HOMPSON SAMPLING

Original: One of the oldest heuristics for balancing exploration with exploitation is given by Thompson sampling, or probability matching (Thompson, 1933).
Revised: One of the oldest heuristics for balancing exploration with exploitation is given by Thompson sampling, or probability matching (Thompson, 1933).

Original: Each episode, Thompson sampling (TS) randomly selects a policy according to the probability it is the optimal policy, conditioned upon the data seen prior to that episode.
Revised: Each episode, Thompson sampling (TS) randomly selects a policy according to the probability it is the optimal policy, conditioned upon the data seen prior to that episode.

Original: Thompson sampling is a simple and effective method that successfully balances exploration with exploitation (Russo et al., 2018).
Revised: Thompson sampling is a simple and effective method that successfully balances exploration with exploitation (Russo et al., 2018).

Original: Implementing Thompson sampling amounts to an inference problem at each episode.
Revised: Implementing Thompson sampling amounts to an inference problem at each episode.

Original: For each s, a, h deﬁne the binary random variable O h ( s, a ) where O h ( s, a )
Revised: For each s, a, h deﬁne the binary random variable O h ( s, a ) where O h ( s, a )

Original: = 1 denotes the event that action a is optimal for state s in timestep h .
Revised: = 1 denotes the event that action a is optimal for state s in timestep h .

Original: The TS policy for episode L is thus given by the inference problem,
Revised: The TS policy for episode (cid:96) is thus given by the inference problem,

Original: [Equation]
Revised: [Equation]

Original: where P ( O | F L ) is the joint probability over all the binary optimality variables.
Revised: where P ( O | F (cid:96) ) is the joint probability over all the binary optimality variables (hereafter we shall suppress the dependence on F (cid:96) ).

Original: To understand how Thompson sampling guides exploration let us consider its performance in Problem 1 when implemented with a uniform prior φ
Revised: To understand how Thompson sampling guides exploration let us consider its performance in Problem 1 when implemented with a uniform prior φ

Original: = ( 12 , 12 ) .
Revised: = ( 12 , 12 ) .

Original: In the ﬁrst timestep the agent samples M 0 ∼ φ .
Revised: In the ﬁrst timestep the agent samples M 0 ∼ φ .

Original: If it samples M + it will choose action a 0 = 2 and learn the true system dynamics, choosing the optimal arm thereafter.
Revised: If it samples M + it will choose action a 0 = 2 and learn the true system dynamics, choosing the optimal arm thereafter.

Original: If it samples M − it will choose action a 0 = 1 and repeat the identical decision in the next timestep.
Revised: If it samples M − it will choose action a 0 = 1 and repeat the identical decision in the next timestep.

Original: Note that this procedure achieves BayesRegret 2.5 according to φ , but also minimax regret 3 , which matches the optimal minimax performance despite its uniform prior.
Revised: Note that this procedure achieves BayesRegret 2.5 according to φ , but also minimax regret 3 , which matches the optimal minimax performance despite its uniform prior.

Original: Recent interest in TS was kindled by strong empirical performance in bandit tasks (Chapelle & Li, 2011).
Revised: Recent interest in TS was kindled by strong empirical performance in bandit tasks (Chapelle & Li, 2011).

Original: Following work has shown that this algorithm satisﬁes strong Bayesian regret bounds close to the  lower bounds for MDPs (Osband & Van Roy, 2017; 2016).
Revised: Following work has shown that this algorithm satisﬁes strong Bayesian regret bounds close to the known lower bounds for MDPs, under certain assumptions (Osband & Van Roy, 2017; 2016).

Original: However, although much simpler than the Bayes-optimal solution, the inference problem in (5) can still be prohibitively expensive.
Revised: However, although much simpler than the Bayes-optimal solution, the inference problem in (5) can still be prohibitively expensive.

Original: Table 1 describes on approach to  sampling from (5) implicitly by maintaining an explicit model over3 For the purposes of this paper, we will focus on optimistic approaches to exploration, although more sophisticated information-seeking approaches merit investigation in future work (Russo & Van Roy, 2014). For the problem deﬁnition in Section 2 there is always a deterministic optimal policy for M . MDP parameters.
Revised: Table 1 describes one approach to performing the sampling required in (5) implicitly, by maintaining an explicit model over MDP parameters.

Original: This algorithm can be computationally intractable as the MDP becomes large and so attempts to scale Thompson sampling to complex systems have focused on approximate posterior samples via randomized value functions, but it is not yet clear under which settings these approximations should be expected to perform well
Revised: This algorithm can be computationally intractable as the MDP becomes large and so attempts to scale Thompson sampling to complex systems have focused on approximate posterior samples via randomized value functions, but it is not yet clear under which settings these approximations should be expected to perform well

Original: (Osband et al., 2017).
Revised: (Osband et al., 2017).

Original: As we look for practical, scalable approaches to posterior inference one promising (and popular) approach is known commonly as ‘RL as inference’.
Revised: As we look for practical, scalable approaches to posterior inference one promising (and popular) approach is known commonly as ‘RL as inference’.

Original: Before episode L
Revised: Before episode (cid:96) Sample

Original: [Equation]
Revised: [Equation]

Original: Bellman equation
Revised: Bellman equation

Revised: [Equation]

Revised: Policy

Original: [Equation]
Revised: [Equation]

Original: 3. T HE ‘RL AS INFERENCE ’ FRAMEWORK AND ITS LIMITATIONS
Revised: 3. T HE ‘RL AS INFERENCE ’ FRAMEWORK AND ITS LIMITATIONS

Original: The computational challenges of Thompson sampling suggest an approximate algorithm that replaces (5) with a parametric distribution suitable for expedient computation.
Revised: The computational challenges of Thompson sampling suggest an approximate algorithm that replaces (5) with a parametric distribution suitable for expedient computation.

Original: It is possible to view the algorithms of the ‘RL as inference’ approach in this light (Rawlik et al., 2013; Todorov, 2009;
Revised: It is possible to view the algorithms of the ‘RL as inference’ approach in this light (Rawlik et al., 2013; Todorov, 2009;

Original: Toussaint, 2009; Deisenroth et al., 2013); see Levine (2018) for a recent survey.
Revised: Toussaint, 2009; Deisenroth et al., 2013); see Levine (2018) for a recent survey.

Original: These algorithms choose to model the probability of optimality according to,
Revised: These algorithms choose to model the probability of optimality according to,

Original: [Equation]
Revised: [Equation]

Original: for some β > 0 , where τ h ( s, a ) is a trajectory (a sequence of state-action pairs) starting from ( s, a ) at timestep h , and where E L denotes the expectation under the posterior at episode L .
Revised: for some β > 0 , where τ h ( s, a ) is a trajectory (a sequence of state-action pairs) starting from ( s, a ) at timestep h , and where E (cid:96) denotes the expectation under the posterior at episode (cid:96) .

Original: With this potential in place one can perform Bayesian inference over these unobserved ‘optimality’ variables, obtaining posteriors over the policy or other variables of interest.
Revised: With this potential in place one can perform Bayesian inference over the unobserved ‘optimality’ variables, obtaining posteriors over the policy or other variables of interest.

Original: (This optimality variable is slightly different to the presentation in  Levine (2018) but ultimately it produces the same family of algorithms; we provide such a derivation in the appendix for completeness).
Revised: This presentation of the RL as inference framework is slightly closer to the one in Deisenroth et al. (2013, § 2.4.2.2) than to Levine (2018), but ultimately it produces the same family of algorithms. We provide such a derivation in the appendix for completeness.

Original: Applying inference procedures to (6) leads naturally to RL algorithms with some ‘soft’ Bellman updates, and added entropy regularization.
Revised: Applying inference procedures to (6) leads naturally to RL algorithms with some ‘soft’ Bellman updates, and added entropy regularization.

Original: We describe the general structure of these algorithms in
Revised: We describe the general structure of these algorithms in

Original: Table 2.
Revised: Table 2.

Original: These algorithmic connections can help reveal connections to policy gradient, actor-critic, and maximum entropy RL methods (Mnih et al., 2016; O’Donoghue et al., 2017; Haarnoja et al., 2017; 2018;
Revised: These algorithmic connections can help reveal connections to policy gradient, actor-critic, and maximum entropy RL methods (Mnih et al., 2016; O’Donoghue et al., 2017; Haarnoja et al., 2017; 2018;

Original: Eysenbach et al., 2018).
Revised: Eysenbach et al., 2018).

Original: The problem is that this resultant ‘posterior’ from (6) does not generally bear any close relationship to the agent’s epistemic probability that ( s, a, h ) is optimal.
Revised: The problem is that this resultant ‘posterior’ derived using (6) does not generally bear any close relationship to the agent’s epistemic probability that ( s, a, h ) is optimal.

Original: Bellman equation

Original: [Table]
Revised: [Table]

Original: To understand how ‘RL as inference’ guides decision making, let us consider its performance in
Revised: To understand how ‘RL as inference’ guides decision making, let us consider its performance in

Original: Problem 1.
Revised: Problem 1.

Original: Practical implementations of ‘RL as inference’ estimate E L µ through observations.
Revised: Practical implementations of ‘RL as inference’ estimate E (cid:96) µ through observations.

Original: For
Revised: For

Original: N large, and without prior guidance, the agent in then extremely unlikely to select action a t =and so resolve its epistemic uncertainty.
Revised: N large, and without prior guidance, the agent is then extremely unlikely to select action a t =and so resolve its epistemic uncertainty.

Original: Even for an informed prior φ = ( 12 , 12 ) action selection according to the exploration strategy of Boltzmann dithering isexponentially unlikely to sample action 2 for which E L µ (2) = 

Revised: Even for an informed prior φ = ( 12 , 12 ) action selection according to the exploration strategy of Boltzmann dithering is unlikely to sample action 2 for which E (cid:96) µ (2) =

Original: (Levine, 2018; Cesa-Bianchi et al., 2017).
Revised: (Levine, 2018; Cesa-Bianchi et al., 2017).

Original: This is because the N −actions with E L µ ≥ 1 − (cid:15) are much more likely to be sampled.
Revised: This is because the N − 1 ‘distractor’ actions with E (cid:96) µ ≥ 1 − (cid:15) are much more probable under the Boltzmann policy.

Original: This problem is the same problem that afﬂicts most dithering approaches to exploration.
Revised: This problem is the same problem that afﬂicts most dithering approaches to exploration.

Original: ‘RL as inference’ as a framework does not incorporate an agents epistemic uncertainty, and so can lead to poor policies for even simple problems.
Revised: ‘RL as inference’ as a framework does not incorporate an agents epistemic uncertainty, and so can lead to poor policies for even simple problems.

Original: While (6) allows the construction of a dual ‘posterior distribution’, this distribution does not generally bear any relation to the typical posterior an agent might compute conditioned upon the data it has gathered according to (5).
Revised: While (6) allows the construction of a dual ‘posterior distribution’, this distribution does not generally bear any relation to the typical posterior an agent should compute conditioned upon the data it has gathered, e.g. , equation (5).

Original: Despite these shortcomings RL as inference has inspired many interesting and novel techniques, as well as delivered algorithms with good performance on problems where exploration is not the bottleneck (Gregor et al., 2016).
Revised: Despite this shortcoming RL as inference has inspired many interesting and novel techniques, as well as delivered algorithms with good performance on problems where exploration is not the bottleneck (Eysenbach et al., 2018).

Original: However, due to the use of language about ‘optimality’ and ‘posterior inference’ etc. , it may come as a surprise to some that this framework does not truly tackle the Bayesian RL problem.
Revised: However, due to the use of language about ‘optimality’ and ‘posterior inference’ etc. , it may come as a surprise to some that this framework does not truly tackle the Bayesian RL problem.

Original: Indeed, algorithms using ‘RL as inference’ can perform very poorly on problems where accurate uncertainty quantiﬁcation is crucial to performance.
Revised: Indeed, algorithms using ‘RL as inference’ can perform very poorly on problems where accurate uncertainty quantiﬁcation is crucial to performance.

Original: We hope that this paper sheds some light on the topic.
Revised: We hope that this paper sheds some light on the topic.

Original: 3. M AKING SENSE OF ‘RL AS I NFERENCE ’ VIA K- LEARNING
Revised: 3. M AKING SENSE OF ‘RL AS I NFERENCE ’ VIA K- LEARNING

Original: In this section we suggest a subtle alteration to the ‘RL as inference’ framework that develops a coherent notion of optimality.
Revised: In this section we suggest a subtle alteration to the ‘RL as inference’ framework that develops a coherent notion of optimality.

Original: The K-learning algorithm was originally introduced through a risk-seeking exponential utility (O’Donoghue, 2018).
Revised: The K-learning algorithm was originally introduced through a riskseeking exponential utility (O’Donoghue, 2018).

Original: In this paper we re-derive this algorithm as a principled approximate inference procedure with clear connections to Thompson sampling, and  highlight its similarities to the ‘RL as inference’ framework.
Revised: In this paper we re-derive this algorithm as a principled approximate inference procedure with clear connections to Thompson sampling, and we highlight its similarities to the ‘RL as inference’ framework.

Original: We believe that this may offer a road towards combining the respective strengths of Thompson sampling and  ‘RL as inference’ frameworks.
Revised: We believe that this may offer a road towards combining the respective strengths of Thompson sampling and the ‘RL as inference’ frameworks.

Original: First, consider the following approximate conditional optimality probability atstate-action ( s, a ) :
Revised: First, consider the following approximate conditional optimality probability at ( s, a, h ) :

Original: [Equation]
Revised: [Equation]

Original: (7) for some β > 0 , and note that this is conditioned on the random variable Q M,(cid:63)h ( s, a ) .
Revised: for some β > 0 , and note that this is conditioned on the random variable Q M,(cid:63)h ( s, a ) .

Original: We can marginalize over possible Q-values yielding
Revised: We can marginalize over possible Q-values yielding

Original: [Equation]
Revised: [Equation]

Original: where G Qh ( s, a, · ) is the cumulant generating function of the random variable Q M,(cid:63)h ( s, a ) (Kendall, 1946).
Revised: where G Qh ( s, a, · ) denotes the cumulant generating function of the random variable Q M,(cid:63)h ( s, a ) (Kendall, 1946).

Original: Equations (6) and (7) are closely linked, but there is a crucial difference.
Revised: Clearly K-learning and the ‘RL as inference’ framework are similar, since equations (6) and (7) are closedly linked, but there is a crucial difference.

Original: The K-learning expectation (7) is with respect to the posterior over Q M,(cid:63)h ( s, a ) to give a parametric approximation to the probability of optimality. This includes the epistemic uncertainty explicitly and so results in more uncertain actions being taken more frequently. Before episode L
Revised: Notice that the integral performed in (8) is with respect to the posterior over Q M,(cid:63)h ( s, a ) , which includes the epistemic uncertainty explicitly.

Original: [Table]
Revised: [Table]

Original:  Bellman equation
Revised: Given the approximation to the posterior probability of optimality in (8) we could sample actions from it as our policy, as done by Thompson sampling (5). However, that requires computation of the cumulant generating function G Qh ( s, a, β ) , which is non-trivial. It was shown in (O’Donoghue, 2018) that an upper bound to the cumulant generating function could be computed by solving a particular ‘soft’ Bellman equation. The resulting K-values, denoted K h ( s, a ) at ( s, a, h ) , are also optimistic for the expected optimal Q-values.

Revised: Speciﬁcally, for a sequence { β (cid:96) } the following holds

Original: [Equation]
Revised: [Table]

Original: Policy

Original: [Equation]
Revised: [Equation]

Original: Following the policy in (8) requires computation of the cumulant generating function, which is nontrivial. O’Donoghue (2018) showed that the K-values that are the solution to a particular Bellman equation produced a guaranteed upper bound on the cumulant generating function at β , which by
Revised: [Table]

Original: Jensen’s inequality is optimistic for the expected Q-value,
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Following a Boltzmann policy over these K-values satisﬁes a Bayesian regret bound which matches the current best bound for Thompson samplingin tabular MDPs up to logarithmic factors.
Revised: Following a Boltzmann policy over these K-values satisﬁes a Bayesian regret bound which matches the current best bound for Thompson sampling up to logarithmic factors under the same set of assumptions.

Original: We summarize the K-learning algorithm in table (3), where ρ > 0 is a constant.
Revised: We summarize the K-learning algorithm in table (3), where β > 0 is a constant and and G µ ( s, a, β ) denotes the cumulant generating function of µ under the posterior.

Original: Comparing tables 2 and 3 it is clear that soft Q-learning and K-learning share some similarities: They both use a ‘soft’ value function and  Boltzmann policies.
Revised: Comparing tables 2 and 3 it is clear that soft Q-learning and K-learning share some similarities: They both solve a ‘soft’ value function and use Boltzmann policies.

Original: However, the differences are important.
Revised: However, the differences are important.

Original: Firstly, K-learning has an explicit schedule for the inverse temperature parameter β L = ρ √ L , and secondly it includes a ‘bonus’ based on visit counts in the  reward signal.
Revised: Firstly, K-learning has an explicit schedule for the inverse temperature parameter β (cid:96) = β √ (cid:96) , and secondly it replaces the expected reward (with respect to the posterior) with a quantity that is optimistic for the expected reward under the posterior.

Original: These two relatively small changes make K-learning a principled exploration and inference strategy.
Revised: These two relatively small changes make K-learning a principled exploration and inference strategy.

Original: To understand how K-learning drives exploration, consider its performance on Problem 1.
Revised: To understand how K-learning drives exploration, consider its performance on Problem 1.

Original: Since this is a bandit problem we can compute the cumulant generating functions for each arm and then use the policy given by (8).
Revised: Since this is a bandit problem we can compute the cumulant generating functions for each arm and then use the policy given by (8).

Original: For any non-trivial prior and choice of β > 0 the cumulant generating function is optimistic for arm 2 which results in the policy selecting arm 2 more frequently, thereby resolving its epistemic uncertainty.
Revised: For any non-trivial prior and choice of β > 0 the cumulant generating function is optimistic for arm 2 which results in the policy selecting arm 2 more frequently, thereby resolving its epistemic uncertainty.

Original: As β → ∞ K-learning will converge on pulling arm 2 with probability one.
Revised: As β → ∞ K-learning converges on pulling arm 2 with probability one.

Original: In contrast to Soft Q-learning, actions a = 3 , .., N are exponentially unlikely to be selected as the exploration parameter β grows.
Revised: In contrast to Soft Q-learning, actions a = 3 , .., N are exponentially unlikely to be selected as the exploration parameter β grows.

Original: 3.3. T HE K- LEARNING POLICY APPROXIMATES THE T HOMPSON SAMPLINGPOLICY
Revised: 3.3. C ONNECTIONS BETWEEN K- LEARNING AND T HOMPSON SAMPLING

Original: Since K-learning can be viewed as approximating the posterior probability of optimality of each action one may ask how similar are the two distributions.
Revised: Since K-learning can be viewed as approximating the posterior probability of optimality of each action it is natural to ask how close an approximation it is.

Original: A natural way to measure this similarity is the KL divergence between the distributions,
Revised: A natural way to measure this similarity is the Kullback–Leibler (KL) divergence between the distributions,

Original: [Equation]
Revised: [Equation]

Revised: where we are using the notation

Revised: O h ( s )

Original: Note that this is different to the usual notion of distance that is taken in variational Bayesian methods, which would typically reverse the order of the arguments in the KL divergence (Blundell et al., 2015).
Revised: = O h ( s, · ) and π K h ( s ) = π K h ( s, · ) . This is different to the usual notion of distance that is taken in variational Bayesian methods, which would typically reverse the order of the arguments in the KL divergence (Blundell et al., 2015).

Original: However, in RL that ‘direction’ is not appropriate: a distribution minimizing D KL ( q || P ( O h ( s ))) may put zero probability on regions of support of P ( O h ( s )) .
Revised: However, in RL that ‘direction’ is not appropriate: a distribution minimizing D KL ( π h ( s ) || P ( O h ( s ))) may put zero probability on regions of support of P ( O h ( s )) .

Original: This means an action with non-zero probability of being optimal might never be taken.
Revised: This means an action with non-zero probability of being optimal might never be taken.

Original: On the other hand a policy minimizing D KL ( P ( O h
Revised: On the other hand a policy minimizing D KL ( P ( O h

Original: ( s ))
Revised: ( s ))

Original: || q ) must assign a non-zero probability to every action that has a non-zero probability of being optimal, or incur an inﬁnite KL divergence penalty. 
Revised: || π h ( s )) must assign a non-zero probability to every action that has a non-zero probability of being optimal, or incur an inﬁnite KL divergence penalty. With this characterization in mind, and noting that the

Revised: Thompson sampling policy satisﬁes E (cid:96)

Original: With this characterization in mind, our next result links the policies of K-learning to Thompson sampling.
Revised: π TS h ( s ) = P ( O h ( s )) , our next result links the policies of K-learning to Thompson sampling.

Original: Theorem 1.
Revised: Theorem 1.

Original: The K-learning value function V KL and policy π KL deﬁned in table 3 satisfy the following bound at every state s ∈ S and h = 0 , . . .
Revised: The K-learning value function V K and policy π K deﬁned in table 3 satisfy the following bound at every state s ∈ S and h = 0 , . . .

Original: H :
Revised: H :

Original: [Equation]
Revised: [Equation]

Original: We defer the proof to Appendix 5.2.
Revised: We defer the proof to Appendix 5.2.

Original: This theorem tells us that the distance between the true probability of optimality, i.e. , the Thompson sampling policy, and the K-learning policy is bounded for any choice of β < ∞ .
Revised: This theorem tells us that the distance between the true probability of optimality and the K-learning policy is bounded for any choice of β < ∞ .

Original: In other words, if there is an action that might be optimal then K-learning will eventually take that action.
Revised: In other words, if there is an action that might be optimal then K-learning will eventually take that action.

Original: 3. W HY IS ‘RL AS I NFERENCE ’ SO POPULAR ?
Revised: 3. W HY IS ‘RL AS I NFERENCE ’ SO POPULAR ?

Original: The sections above outline some surprising ways that the ‘RL as inference’ framework can drive suboptimal behaviour in even simple domains.
Revised: The sections above outline some surprising ways that the ‘RL as inference’ framework can drive suboptimal behaviour in even simple domains.

Original: The question remains, why do so many popular and effective algorithms lie within this class?
Revised: The question remains, why do so many popular and effective algorithms lie within this class?

Original: The ﬁrst, and most important point, is that these algorithms canalso perform extremely well in domains where efﬁcient exploration is not a bottleneck.
Revised: The ﬁrst, and most important point, is that these algorithms can perform extremely well in domains where efﬁcient exploration is not a bottleneck.

Original: Further they are often easy to implement and amenable to function approximation (Peters et al., 2010; Kober & Peters, 2009; Abdolmaleki et al., 2018).
Revised: Furthermore, they are often easy to implement and amenable to function approximation (Peters et al., 2010; Kober & Peters, 2009; Abdolmaleki et al., 2018).

Original: Our discussion of K-learning in Section 3.3 shows that a relatively simple ﬁx to this problem formulation can result in a framing of RL as inference that maintains a coherent notion of optimality.
Revised: Our discussion of K-learning in Section 3.3 shows that a relatively simple ﬁx to this problem formulation can result in a framing of RL as inference that maintains a coherent notion of optimality.

Original: Computational results show that, in tabular domains, Klearning can be competitive with, or even outperform Thompson sampling strategies, but extending these results to large-scale domains with generalization is an open question (O’Donoghue, 2018;
Revised: Computational results show that, in tabular domains, Klearning can be competitive with, or even outperform Thompson sampling strategies, but extending these results to large-scale domains with generalization is an open question (O’Donoghue, 2018;

Original: Osband et al., 2017).
Revised: Osband et al., 2017).

Original: The other observation is that the ‘RL as inference’ can provide useful insights to the structure of particular algorithms for RL.
Revised: The other observation is that the ‘RL as inference’ can provide useful insights to the structure of particular algorithms for RL.

Original: It is valid to note that, under certain conditions, following policy gradient is equivalent to a dual inference problem where the ‘probabilities’ play the role of dummy variables, but are not supposed to represent the probability of optimality in the RL problem.
Revised: It is valid to note that, under certain conditions, following policy gradient is equivalent to a dual inference problem where the ‘probabilities’ play the role of dummy variables, but are not supposed to represent the probability of optimality in the RL problem.

Original: In this light, Levine (2018) presents the inference framework as a way to generalize a wide range of state of the art RL algorithms.
Revised: In this light, Levine (2018) presents the inference framework as a way to generalize a wide range of state of the art RL algorithms.

Original: However, when taking this view, you should remember that this inference duality is limited to certain RL algorithms, and without some modiﬁcations (e.g. Section 3.3) this perspective is in danger of overlooking important aspects of the RL problem.
Revised: However, when taking this view, you should remember that this inference duality is limited to certain RL algorithms, and without some modiﬁcations (e.g. Section 3.3) this perspective is in danger of overlooking important aspects of the RL problem.

Original:  C OMPUTATIONAL EXPERIMENTS 4. O NE UNKNOWN ACTION (P ROBLEM 1)
Revised:  C OMPUTATIONAL EXPERIMENTS 4. O NE UNKNOWN ACTION (P ROBLEM 1)

Original: Consider the environment of Problem 1 with uniform prior φ = ( 12 , 12 ) .
Revised: Consider the environment of Problem 1 with uniform prior φ = ( 12 , 12 ) .

Original: We ﬁx (cid:15) = 1 e − 3 and consider how the Bayesian regret varies with N > 3 .
Revised: We ﬁx (cid:15) = 1 e − 3 and consider how the Bayesian regret varies with N > 3 .

Original: Figure 1a compares how the regret scales for Bayes-optimal ( 1 . 5 ), Thompson sampling ( 2 . 5 ), K-learning ( ≤ 2 . 2 ) and Soft Q-learning (which grows linearly in N for the optimal β → 0 , but would typically grow exponentially for β > 0 ).
Revised: Figure 1 compares how the regret scales for Bayes-optimal ( 1 . 5 ), Thompson sampling ( 2 . 5 ), K-learning ( ≤ 2 . 2 ) and Soft Q-learning (which grows linearly in N for the optimal β → 0 , but would typically grow exponentially for β > 0 ).

Original: This highlights that, even in a simple problem, there can be great value in considering the value of information.
Revised: This highlights that, even in a simple problem, there can be great value in considering the value of information.

Original:  4. ‘D EEP S EA ’ PROBLEM WITH TABULAR REPRESENTATION
Revised: Soft-Q does not scale gracefully with N . 4. ‘D EEP S EA ’ EXPLORATION

Original: Our next set of experiments considers thethe ‘DeepSea’ MDPsas introduced by Osband et al.
Revised: Our next set of experiments considers the ‘DeepSea’ MDPs introduced by Osband et al.

Original: At a high level this problem represents a ‘needle in a haystack’,and is designed to require efﬁcient exploration, the complexity of which grows with the problem size N ∈ N .
Revised: At a high level this problem represents a ‘needle in a haystack’, designed to require efﬁcient exploration, the complexity of which grows with the problem size N ∈ N .

Original: DeepSea  is a scalable variant of the ‘chain MDPs’ long considered in exploration research (Jaksch et al., 2010). 
Revised: DeepSea (Figure 2) is a scalable variant of the ‘chain MDPs’ popular in exploration research (Jaksch et al., 2010). right one and falls one row.

Revised: There is a small negative reward for right, and zero reward for left.

Revised: There is only one rewarding state, at the bottom right cell.

Original:  Algorithms that do not perform deep exploration will take an exponential number of episodes to learn the optimal policy, but those that prioritize informative states and action can learn in polynomial time of N .
Revised: The only way the agent can receive positive reward is to choose to go right in each timestep. Algorithms that do not perform deep exploration will take an exponential number of episodes to learn the optimal policy, but those that prioritize informative states and actions can learn much faster.

Original: Figure 1b presents results for Thompson sampling (PSRL), K-learning and Soft-Q learning with identical priors and posterior updates (Gaussian rewards, Dirichlet transitions).
Revised: Figure 3a shows the ‘time to learn’ for tabular implementations of K-learning (Section 3.3), Soft-Q (Section 3.2) and Thompson sampling (Section 3.1). We implement each of the algorithms with a N (0 , 1) prior for rewards and Dirichlet(1 /N ) prior for transitions.

Revised: Since these problems are small and tabular, we can use conjugate prior updates and exact MDP planning via value iteration.

Original: As expected, Thompson sampling and K-learning scale gracefully to large domains, Soft-Q  does not.
Revised: As expected, Thompson sampling and K-learning scale gracefully to large domains but Soft-Q learning does not.

Original: 4. B EHAVIOUR SUITE FOR R EINFORCEMENT L EARNING
Revised: 4. B EHAVIOUR S UITE FOR R EINFORCEMENT L EARNING

Original: So far our results have purely focused on the tabular setting, but the main focus of ‘RL as inference’ is for scalable algorithms that work with generalization.
Revised: So far our experiments have been conﬁned to the tabular setting, but the main focus of ‘RL as inference’ is for scalable algorithms that work with generalization.

Revised: In this section we show that the same insights we built in the tabular setting extend to the setting of deep RL.

Original: To show that the problems we have identiﬁed are not ﬁxed by implementing Soft-Q learning with neural networks we  evaluate this algorithm on bsuite : a suite of benchmark tasks designed to highlight key issues in RL (Osband et al., 2019). 
Revised: To do this we implement variants of Deep Q Networks with a single layer, 50-unit MLP (Mnih et al., 2013). To adapt K-learning and Thompson sampling to this deep RL setting we use an ensemble of size 20 with randomized prior functions to approximate the posterior distribution over neural network Q-values (Osband et al., 2018) (full experimental details are included in Appendix 5.4). We then evaluate all of the algorithms on bsuite : a suite of benchmark tasks designed to highlight key issues in RL (Osband et al., 2019). In particular, bsuite includes an evaluation on the DeepSea problems but with a one-hot pixel representation of the agent position.

Original: As expected, the core issues of uncertainty and exploration remain poor for  Soft-Q even when paired with deep neural architectures.
Revised: In Figure 3b we see that the results for these deep RL implementations closely match the observed scaling for the tabular setting. In particular, the algorithms motivated by Thompson sampling and K-learning both scale gracefully to large problem sizes, where Soft-Q learning is unable to drive deep exploration. Our bsuite evaluation includes many more experiments that can be ﬁt into this paper, but we provide a link to the complete results at bit.ly/rl-inference-bsuite . In general, the results for Thompson sampling and Klearning are similar, with Soft Q performing signiﬁcantly worse on ‘exploration’ tasks.

Original: We push  these results to Appendix 6.
Revised: We push a summary of these results to Appendix 6.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper aims to make sense of reinforcement learning and probabilistic inference.
Revised: This paper aims to make sense of reinforcement learning and probabilistic inference.

Original: We review the reinforcement learning problem and show that this problem of optimal learning already combined the problems of control and inference.
Revised: We review the reinforcement learning problem and show that this problem of optimal learning already combined the problems of control and inference.

Original: As we highlight this connection, we also clarify some potentially confusing details in the popular ‘RL as inference’ framework.
Revised: As we highlight this connection, we also clarify some potentially confusing details in the popular ‘RL as inference’ framework.

Original: We show that, since this problem formulation ignores the role of epistemic uncertainty, that algorithms derived from that framework can perform poorly on even simple tasks.
Revised: We show that, since this problem formulation ignores the role of epistemic uncertainty, that algorithms derived from that framework can perform poorly on even simple tasks.

Original: Importantly, we also offer a way forward, to reconcile the views of RL and inference in a way that maintains the best pieces of both.
Revised: Importantly, we also offer a way forward, to reconcile the views of RL and inference in a way that maintains the best pieces of both.

Original: In particular, we show that a simple variant to the RL as inference framework (K-learning) can incorporate uncertainty estimates to drive efﬁcient exploration.
Revised: In partic- ular, we show that a simple variant to the RL as inference framework (K-learning) can incorporate uncertainty estimates to drive efﬁcient exploration.

Original: We support our claims with a series of simple didactic experiments.
Revised: We support our claims with a series of simple didactic experiments.

Original: We leave the crucial questions of how to scale these insights up to large complex domains for future work.
Revised: We leave the crucial questions of how to scale these insights up to large complex domains for future work.

Original: A PPENDIX 5. S OFT Q- LEARNING DERIVATION
Revised: A PPENDIX 5. S OFT Q- LEARNING DERIVATION

Original: We present a derivation of soft Q-learning from the RL as inference parametric approximation to the probability of optimality.
Revised: We present a derivation of soft Q-learning from the RL as inference parametric approximation to the probability of optimality.

Original: Although our presentation is slightly different to that of Levine (2018) we show here that the resulting algorithms are essentially identical.
Revised: Although our presentation is slightly different to that of Levine (2018) we show here that the resulting algorithms are essentially identical.

Original: Recall from equation (6) that the parametric approximation  is given by[Equation] where we use the notation τ ( s [Equation]
Revised: Recall from equation (6) that the parametric approximation to optimality we consider is given by

Original: a

Original: [Equation]
Revised: [Equation]

Original:  h  ) to denote a trajectory starting fromstate-action ( s [Equation] a ) at timestep h .
Revised: where τ h ( s, a ) is a trajectory starting from ( s, a ) at time h and β > 0 is a hyper-parameter.

Original: Now we must marginalize out  τ  using the (unknown) system dynamics.
Revised: Now we must marginalize out the possible trajectories τ h using the (unknown) system dynamics.

Original: Since this is a certaintyequivalent algorithm we shall use the expected value of the transition probabilities, under the posterior at timestep L . In which case we obtain P
Revised: Since this is a certainty-equivalent algorithm we shall use the expected value of the transition probabilities, under the posterior at episode (cid:96) , which means we can write

Original: [Equation]
Revised: [Equation]

Original: now we make the additional assumption that  p ( y | s ) is uniform across all actions so we can ignore it (this assumption is also required in  Levine (2018)
Revised: and we make the additional assumption that the ‘prior’ p ( a | s ) is uniform across all actions a for each s (this assumption is standard in this framework, see Levine (2018)). In this case we obtain ˜ P ( O h ( s, a )) = (cid:88)

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  with this we can rewrite
Revised: Now with this we can rewrite

Original: [Equation]
Revised: [Equation]

Original: where Z is the normalization constant [Equation]
Revised: where Z is the normalization constant, and using Jensen’s

Original: from Jensen’s then we get
Revised: we have the following bound

Original: [Equation]
Revised: [Equation]

Original: now if we introduce the soft Q-values that satisfy the  Bellman equation
Revised: now if we introduce the soft Q-values that satisfy the soft Bellman equation

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: and we have the soft Q-learning algorithm (the approximation comes from the fact we used Jensen’s inequality 
Revised: and we have the soft Q-learning algorithm (the approximation comes from the fact we used Jensen’s inequality to provide a bound)

Original: [Equation]
Revised: [Equation]

Original: 5. P ROOF OF THEOREM
Revised: 5. P ROOF OF THEOREM

Original: Theorem.
Revised: Theorem.

Original: The K-learning value function V KL and policy π KL deﬁned in table 3 satisfy the following bound at every state s ∈ S and h = 0 , . . .
Revised: The K-learning value function V K and policy π K deﬁned in table 3 satisfy the following bound at every state s ∈ S and h = 0 , . . .

Original: H :
Revised: H :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix some particular state s ∈ S [Equation] and let the joint posterior over value and policy be denoted by
Revised: Fix some particular state s ∈ S , and let the joint posterior over value and optimality be denoted by

Original: [Equation]
Revised: [Equation]

Original: where we use f to denote the conditional distribution over Q-values conditioned on optimality
Revised: where we use f to denote the conditional distribution over Q-values conditioned on optimality.

Revised: Recall that from equation (7) we have approximated the conditional posterior probability of optimality as

Original: [Equation]
Revised: [Equation]

Original: and where π tss h ,a h = P ( O h ( s, a )) is the Thompson sampling policy.
Revised: for some β >

Original: Recall that from equation (7) we have approximated the posterior probability of optimality as
Revised: [Equation] which when yields

Original: [Equation]
Revised: [Equation]

Original: Form Bayes rule this implies the following approximation to the conditional distribution
Revised: From Bayes’ rule this implies the following approximation to the conditional distribution

Original: [Equation]
Revised: [Equation]

Original: This is known as the exponential tilt of the posterior distribution P ( Q M,(cid:63)h ( s, a )) and has a myriad of applications in statistics (Asmussen & Glynn, 2007).
Revised: This is known as the exponential tilt of the posterior distribution P ( Q M,(cid:63)h ( s, a )) and has a myriad of applications in statistics (Asmussen & Glynn, 2007).

Original: This yields the following approximation to the joint distribution ˜ f

Original: ( Q M,(cid:63)h ( s, a ) |O h

Original: ( s, a )) ˜ P ( O h ( s, a )) . However, the K-learning policy does not follow (7) since computing the cumulant generating function is non-trivial.
Revised: From this we could derive an approximation to the joint posterior (11), however, the K-learning policy does not follow (8) since computing the cumulant generating function is non-trivial.

Original: Instead we compute the K-values, which are the solution to a Bellman equation that provide a guaranteed upper bound on the cumulant generating function, and the K-learning policy is thus
Revised: Instead we compute the K-values, which are the solution to a Bellman equation that provide a guaranteed upper bound on the cumulant generating function, and the K-learning policy is thus

Original: [Equation]
Revised: [Equation]

Original: and recall we have 
Revised: where we have (O’Donoghue, 2018)

Original: [Equation]
Revised: [Equation]

Original: With that in mind we take our approximate joint posterior  to be
Revised: With that in mind we take our approximation to the joint posterior (11) to be

Original: [Equation]
Revised: [Equation]

Original: Now consider the KL-divergence between the true joint posterior and our approximate one, a quick calculation yields
Revised: Now consider the KL-divergence between the true joint posterior and our approximate one, a quick calculation yields

Original: [Equation]
Revised: [Equation]

Original: where we used the subscript s to denote quantities restricted to state s .
Revised: ) for timestep h and state s .

Original: Considering the terms in (12) separately we have
Revised: Considering the terms on the right hand side of (14) separately we have

Original: [Equation]
Revised: [Equation]

Original: where H denotes the entropy, and (cid:88)
Revised: where H denotes the entropy

Original: [Equation]
Revised: [Equation]

Original: Now we sum these two terms
Revised: and using (12) (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  using (11) and the following identities 
Revised: Now we sum these two terms, using (13) and the following identities (cid:88)

Original: [Equation]
Revised: [Equation]

Original: and 
Revised: and (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  P [Equation] O h 
Revised: ≤ H ( P ( O h ( s ))) , since log( P ( O h ( s, a ) | Q M,(cid:63)h ( s, a ))) ≤

Original: [Equation]
Revised: [Equation]

Original: , we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: The theorem follows from this and the fact that the K-learning value function is deﬁned as
Revised: The theorem follows from this and the fact that the K-learning value function is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: as well as the fact thatD KL ( P ( O h ( s )) || π h ( s ) KL ) ≤ D KL ( P h ( s ) ||
Revised: as well as the fact that

Original: ˜ P h ( s )) from equation (12).
Revised: [Equation] from equation (14) [Equation]

Original: 5. P ROBLEM 1 K- LEARNING DETAILS
Revised: 5. P ROBLEM 1 K- LEARNING DETAILS

Original: For a bandit problem the K-learning policy is given by
Revised: For a bandit problem the K-learning policy is given by

Original: [Equation]
Revised: [Equation]

Original: which requires the cumulant generating function of the posterior over each arm.
Revised: which requires the cumulant generating function of the posterior over each arm.

Original: For arm 1 and the distractor arms there is no uncertainty, in which case the cumulant generating function is given by
Revised: For arm 1 and the distractor arms there is no uncertainty, in which case the cumulant generating function is given by

Original: [Equation]
Revised: [Equation]

Original: In the case of arm 2 the cumulant generating function is
Revised: In the case of arm 2 the cumulant generating function is

Original: [Equation]
Revised: [Equation]

Original: In (O’Donoghue, 2018) it was shown that the optimal choice of β is given by
Revised: In (O’Donoghue, 2018) it was shown that the optimal choice of β is given by

Original: [Equation]
Revised: [Equation]

Original: which requires solving a convex optimization problem in variable β − 1 .
Revised: which requires solving a convex optimization problem in variable β − 1 .

Original: In the case of problem 1 the optimal choice of β ≈ 10 . 23 , which yields π kl 2 ≈ 0 .
Revised: In the case of problem 1 the optimal choice of β ≈ 10 . 23 , which yields π kl 2 ≈ 0 .

Original: Then, once arm 2 has been pulled once and the true reward of arm 2 has been revealed, its cumulant generating function has the same form as the others, and then the optimal choice of β is simply
Revised: Then, once arm 2 has been pulled once and the true reward of arm 2 has been revealed, its cumulant generating function has the same form as the others, and then the optimal choice of β is simply

Original: [Equation]
Revised: [Equation]

Original: at which point K-learning is greedy with respect to the optimal arm.
Revised: at which point K-learning is greedy with respect to the optimal arm.

Revised: 5. I MPLEMENTATION DETAILS FOR BSUITE EVALUATION

Revised: All three algorithms use the same neural network architecture consisting of an MLP (multilayer perceptron) with a single hidden layer with 50 hidden units.

Original: bsuite report: Varying β in Soft Q learning 
Revised: All three algorithms used a replay buffer of the most recent 10 4 transitions to allow re-use of data. For all three the Adam optimizer (Kingma & Ba, 2014) was used with learning rate 10 − 3 and batch-size 128 , and learning is performed at every time-step. For both K-learning and soft Q-learning the temperature was set at β − 1 = 0 . For

Revised: Bootstrap DQN we chose an ensemble of size 20 , and used the randomized prior functions (Osband et al., 2018) with scale 3 . .

Revised: For K-learning, in order to estimate the cumulant generating function of the reward, we used an ensemble of neural networks predicting the reward for each state and action and used these to calculate the empirical cumulant generating function over them.

Revised: Each of these was a single hidden layer MLP with 10 hidden units.

Revised: Finally, we noted that actually training a small ensemble of K-networks performed better than a single network, we used an ensemble of sizefor this purpose as well as using randomized priors to encourage diversity between the elements of the ensemble with scale 1 . 0 .

Revised: The K-learning policy was the Boltzmann policy over all the ensemble

Revised: K-values at each state.

Original: The Behaviour Suite for Reinforcement Learning , or bsuite for short, is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent.
Revised: The Behaviour Suite for Reinforcement Learning , or bsuite for short, is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent.

Original: The aim of the bsuite project is to collect clear, informative and scalable problems that capture key issues in the design of efﬁcient and general learn- ing algorithms and study agent behaviour through their performance on these shared benchmarks.
Revised: The aim of the bsuite project is to collect clear, informative and scalable problems that capture key issues in the design of efﬁcient and general learn- ing algorithms and study agent behaviour through their performance on these shared benchmarks.

Original: This report provides a snapshot of agent performance on bsuite2019 , obtained by running the experiments from github.com/deepmind/bsuite Osband et al.
Revised: This report provides a snapshot of agent performance on bsuite2019 , obtained by running the experiments from github.com/deepmind/bsuite Osband et al.

Original: 6. A GENT DEFINITION
Revised: 6. A GENT DEFINITION

Original: All agents correspond to different instantiations of the DQN agent with Soft Q learning (Mnih et al., 2015; O’Donoghue et al., 2017).
Revised: All agents were run with the same network architecture (a single layer MLP with 50 hidden units a ReLU activation) adapting DQN (Mnih et al., 2013).

Original: We use a 2 layer MLP with 50 hidden units, Adam optimizer and learning rate 1e-3.
Revised: Full hyperparameters in Appendix 5.4. • boot dqn : bootstrapped DQN with prior networks (Osband et al., 2016; 2018). • k learn : K-learning via ensemble with prior networks (O’Donoghue, 2018; Osband et al., 2018).

Original: The only parameter we vary is the temperature β as deﬁned in Table 2.
Revised: • soft q : soft Q-learning with temperature β − 1 = 0 . 01 (O’Donoghue et al., 2017).

Original: 6. S UMMARY SCORES
Revised: 6. S UMMARY SCORES

Original: Each bsuite experiment outputs a summary score in [0,1].
Revised: Each bsuite experiment outputs a summary score in [0,1].

Original: We aggregate these scores by according to key experiment type, according to the standard analysis notebook.
Revised: We aggregate these scores by according to key experiment type, according to the standard analysis notebook.

Original: A detailed analysis of each of these experiments may be found in a notebook hosted on Colaboratory: (withheld for anonynomity).
Revised: A detailed analysis of each of these experiments may be found in a notebook hosted on Colaboratory: bit.ly/rl-inference-bsuite .

Original: 6. R ESULTS COMMENTARY
Revised: 6. R ESULTS COMMENTARY

Original: Although weswept the temperature parameter β over a large range of values, we do not see a huge difference in the performance of the agent.
Revised: Overall, we see that the algorithms K-learning and Bootstrapped DQN perform extremely similarly across bsuite evaluations.

Original: For each setting of β the agent performs reasonably on thebasic tasks (except for very small β = 1 where action selection is too random). Importantly, there is no setting of β that appreciably improves the agent’s performance on the hard exploration tasks.
Revised: However, there is a clear signal that soft Q-learning performs markedly worse on the tasks requiring efﬁcient exploration. This observation is consistent with the hypothesis that algorithms motivated by ‘RL as Inference’ fail to account for the value of exploratory actions.

Original: See the colab for a full description of results.
Revised: Beyond this major difference in exploration score, we see that Bootstrapped DQN outperforms the other algorithms on problems varying ‘Scale’. This too is not surprising, since both soft Q and Klearning rely on a temperature tuning that will be problem-scale dependent. Finally, we note that soft Q also performs worse on some ‘basic’ tasks, notably ‘bandit’ and ‘mnist’. We believe that the relatively high temperature (tuned for best performance on Deep Sea) leads to poor performance on these tasks with larger action spaces, due to too many random actions.