Original: Directed Exploration via Uncertainty-Aware Critics
Revised: Directed Exploration via Uncertainty-Aware Critics

Original: Abstract
Revised: Abstract

Original: The exploration-exploitation dilemma is still an open problem in Reinforcement Learning (RL), especially when coped with deep architectures and in the context of continuous action spaces.
Revised: The exploration-exploitation dilemma is still an open problem in Reinforcement Learning (RL), especially when coped with deep architectures and in the context of continuous action spaces.

Original: Uncertainty quantification has been extensively used as a means to achieve efficient directed exploration.
Revised: Uncertainty quantification has been extensively used as a means to achieve efficient directed exploration.

Original: However, state-of-the-art methods for continuous actions still suffer from high sample complexity requirements.
Revised: However, state-of-the-art methods for continuous actions still suffer from high sample complexity requirements.

Original: Indeed, they either completely lack strategies for propagating the epistemic uncertainty throughout the updates, or they mix it with aleatory uncertainty while learning the full return distribution (e.g., distributional RL).
Revised: Indeed, they either completely lack strategies for propagating the epistemic uncertainty throughout the updates, or they mix it with aleatory uncertainty while learning the full return distribution (e.g., distributional RL).

Original: In this paper, we propose Wasserstein
Revised: In this paper, we propose Wasserstein

Original: Actor-Critic (WAC), an actor-critic architecture inspired by the recent Wasserstein Q-Learning (WQL) (Metelli et al., 2019), that employs approximate Q-posteriors to represent the epistemic uncertainty and Wasserstein barycenters for uncertainty propagation across the state-action space.
Revised: Actor-Critic (WAC), an actor-critic architecture inspired by the recent Wasserstein Q-Learning (WQL) (Metelli et al., 2019), that employs approximate Q-posteriors to represent the epistemic uncertainty and Wasserstein barycenters for uncertainty propagation across the state-action space.

Original: WAC enforces exploration in a principled way by guiding the policy learning process with the optimization of an upper bound of the Q-value estimates.
Revised: WAC enforces exploration in a principled way by guiding the policy learning process with the optimization of an upper bound of the Q-value estimates.

Original: Furthermore, we study some peculiar issues that arise when using function approximation, coupled with the uncertainty estimation, and propose a regularized loss for the uncertainty estimation.
Revised: Furthermore, we study some peculiar issues that arise when using function approximation, coupled with the uncertainty estimation, and propose a regularized loss for the uncertainty estimation.

Original: Finally, we evaluate our algorithm on a suite of continuous-actions domains, where exploration is crucial, in comparison with stateof-the-art baselines.
Revised: Finally, we evaluate our algorithm on a suite of continuous-actions domains, where exploration is crucial, in comparison with stateof-the-art baselines.

Original: Our experiments show a clear benefit of using uncertainty-aware critics for continuous-actions control.
Revised: Our experiments show a clear benefit of using uncertainty-aware critics for continuous-actions control.

Original: 1. Introduction
Revised: 1. Introduction

Original: Reinforcement Learning (RL Sutton & Barto, 2018) is one of the most widely used frameworks for solving sequential decision-making problems, especially in model-free settings, where a model of the environment dynamics is not available.
Revised: Reinforcement Learning (RL Sutton & Barto, 2018) is one of the most widely used frameworks for solving sequential decision-making problems, especially in model-free settings, where a model of the environment dynamics is not available.

Original: When an agent acts in an uncertain environment, it faces the choice between exploring with the hope of discovering more profitable behaviors or exploiting the current information about the actions’ values.
Revised: When an agent acts in an uncertain environment, it faces the choice between exploring with the hope of discovering more profitable behaviors or exploiting the current information about the actions’ values.

Original: This exploration-exploitation dilemma is particularly challenging in continuous-state spaces, where function approximation is required to generalize across states, and, differently from the tabular case, an accurate estimate of the uncertainty on the value estimates is not available point-wise.
Revised: This exploration-exploitation dilemma is particularly challenging in continuous-state spaces, where function approximation is required to generalize across states, and, differently from the tabular case, an accurate estimate of the uncertainty on the value estimates is not available point-wise.

Original: Continuousaction tasks pose additional challenges since most exploration methods require the maximization of some objective (e.g., upper bound of the Q-value) over the action space.
Revised: Continuousaction tasks pose additional challenges since most exploration methods require the maximization of some objective (e.g., upper bound of the Q-value) over the action space.

Original: While in the discrete case, this maximization can be performed by enumeration, in the continuous case it requires solving a complex optimization problem, increasing the computational demands.
Revised: While in the discrete case, this maximization can be performed by enumeration, in the continuous case it requires solving a complex optimization problem, increasing the computational demands.

Original: Actor-Critic (AC) methods (Haarnoja et al., 2018; Ciosek et al., 2019; Schulman et al., 2015) represent the current state of the art for continuous control.
Revised: Actor-Critic (AC) methods (Haarnoja et al., 2018; Ciosek et al., 2019; Schulman et al., 2015) represent the current state of the art for continuous control.

Original: Despite their widespread adoption, these methods still suffer from high sample complexity.
Revised: Despite their widespread adoption, these methods still suffer from high sample complexity.

Original: Efficient exploration strategies have been extensively studied in the literature as a means of reducing sample complexity mainly in tabular domains (Auer et al., 2008; Ian et al., 2013; Metelli et al., 2019; O’Donoghue et al., 2018).
Revised: Efficient exploration strategies have been extensively studied in the literature as a means of reducing sample complexity mainly in tabular domains (Auer et al., 2008; Ian et al., 2013; Metelli et al., 2019; O’Donoghue et al., 2018).

Original: Classical exploration strategies, like ϵ -greedy or Boltzmann (Sutton & Barto, 2018), inject noise around the current greedy policy to enforce exploration.
Revised: Classical exploration strategies, like ϵ -greedy or Boltzmann (Sutton & Barto, 2018), inject noise around the current greedy policy to enforce exploration.

Original: Although in simple settings this is enough to guarantee convergence (Szepesvári, 1997), this exploration strategy is not efficient in the general case.
Revised: Although in simple settings this is enough to guarantee convergence (Szepesvári, 1997), this exploration strategy is not efficient in the general case.

Original: Another line of approaches considers the maximum entropy setting to improve exploration, and avoids the deterministic collapse of the policies, e.g., Soft Actor Critic (SAC,
Revised: Another line of approaches considers the maximum entropy setting to improve exploration, and avoids the deterministic collapse of the policies, e.g., Soft Actor Critic (SAC,

Original: Haarnoja et al., 2018).
Revised: Haarnoja et al., 2018).

Original: In this setting, stochastic policies are preferred by optimizing the expected return regularized with an entropy term.
Revised: In this setting, stochastic policies are preferred by optimizing the expected return regularized with an entropy term.

Original: This too represents a form of undirected exploration since the policies are forced to be stochastic, thanks to the entropy bonus, but the induced noise does not consciously shift its focus towards promising regions of the state space.
Revised: This too represents a form of undirected exploration since the policies are forced to be stochastic, thanks to the entropy bonus, but the induced noise does not consciously shift its focus towards promising regions of the state space.

Original: A common trend in the RL literature consists in endowing existing methods with some form of uncertainty quantification and using it to perform directed exploration while focusing on the most promising regions.
Revised: A common trend in the RL literature consists in endowing existing methods with some form of uncertainty quantification and using it to perform directed exploration while focusing on the most promising regions.

Original: For instance, optimistic approaches have been applied to both Q-learning (Jin et al., 2018) and SAC.
Revised: For instance, optimistic approaches have been applied to both Q-learning (Jin et al., 2018) and SAC.

Original: In particular, a recent extension of SAC, Optimistic Actor-Critic(OAC,
Revised: In particular, a recent extension of SAC, Optimistic Actor-Critic(OAC,

Original: Ciosek et al., 2019), proved to improve sample efficiency over the standard SAC.
Revised: Ciosek et al., 2019), proved to improve sample efficiency over the standard SAC.

Original: Indeed, uncertainty quantification is a fundamental step to define efficient exploration strategies.
Revised: Indeed, uncertainty quantification is a fundamental step to define efficient exploration strategies.

Original: The most used exploration strategies, coming from the Multi-Armed Bandit (MAB, Lattimore, 2020 - 2020) literature, use uncertainty estimates to explore either based on optimism (Auer et al., 2002) or posterior sampling (PS, Thompson, 1933).
Revised: The most used exploration strategies, coming from the Multi-Armed Bandit (MAB, Lattimore, 2020 - 2020) literature, use uncertainty estimates to explore either based on optimism (Auer et al., 2002) or posterior sampling (PS, Thompson, 1933).

Original: These methods have been extended for the RL settings, starting from tabular domains (Auer et al., 2008; Ian et al., 2013;
Revised: These methods have been extended for the RL settings, starting from tabular domains (Auer et al., 2008; Ian et al., 2013;

Original: Metelli et al., 2019), with theoretical guarantees on the sample complexity and/or regret.
Revised: Metelli et al., 2019), with theoretical guarantees on the sample complexity and/or regret.

Original: Uncertainty quantification methods have been proposed for the Deep Reinforcement
Revised: Uncertainty quantification methods have been proposed for the Deep Reinforcement

Original: Learning (DRL) settings too, but the guarantees no longer hold up.
Revised: Learning (DRL) settings too, but the guarantees no longer hold up.

Original: This is done by means of posterior distributions to represent uncertainty.
Revised: This is done by means of posterior distributions to represent uncertainty.

Original: Posterior distributions are either modeled explicitly (Metelli et al., 2019; Lee et al., 2021) or implicitly by using an ensemble of value estimates (Wang et al., 2021).
Revised: Posterior distributions are either modeled explicitly (Metelli et al., 2019; Lee et al., 2021) or implicitly by using an ensemble of value estimates (Wang et al., 2021).

Original: Ensemble methods allow quantifying the uncertainty but do not propagate it across the state action-space when performing the critic updates.
Revised: Ensemble methods allow quantifying the uncertainty but do not propagate it across the state action-space when performing the critic updates.

Original: Uncertainty propagation is a fundamental tool of any principled uncertainty estimation approach since most AC methods rely on bootstrapping when updating the critics.
Revised: Uncertainty propagation is a fundamental tool of any principled uncertainty estimation approach since most AC methods rely on bootstrapping when updating the critics.

Original: This results in Q-value estimates that also incorporate uncertainty about the bootstrapped values.
Revised: This results in Q-value estimates that also incorporate uncertainty about the bootstrapped values.

Original: Distributional RL (O’Donoghue et al., 2018) allows for uncertainty propagation but considers only aleatoric uncertainty, being aimed at estimating the full return distribution.
Revised: Distributional RL (O’Donoghue et al., 2018) allows for uncertainty propagation but considers only aleatoric uncertainty, being aimed at estimating the full return distribution.

Original: This is not straightforward in practice, and, furthermore, it is not strictly necessary in the classical RL setting where the goal is to maximize the expected return.
Revised: This is not straightforward in practice, and, furthermore, it is not strictly necessary in the classical RL setting where the goal is to maximize the expected return.

Original: To the best of our knowledge, the only method capable of propagating the epistemic uncertainty, without the need to learn the return distribution is Wasserstein Q-learning (WQL Metelli et al., 2019), which has only been proposed for the discrete action case.
Revised: To the best of our knowledge, the only method capable of propagating the epistemic uncertainty, without the need to learn the return distribution is Wasserstein Q-learning (WQL Metelli et al., 2019), which has only been proposed for the discrete action case.

Original: In this paper, we address the problem of uncertainty estimation and propagation in the context of continuousaction RL.
Revised: In this paper, we address the problem of uncertainty estimation and propagation in the context of continuousaction RL.

Original: Starting from the methodology introduced in
Revised: Starting from the methodology introduced in

Original: WQL (Metelli et al., 2019), we devise a novel actor-critic algorithm, Wasserstein Actor-Critic (WAC), which employs
Revised: WQL (Metelli et al., 2019), we devise a novel actor-critic algorithm, Wasserstein Actor-Critic (WAC), which employs

Original: Q-posteriors both to quantify uncertainty on the critic estimates to drive exploration, as well as a tool to propagate it across the state-action space, by means of Wasserstein barycenters (Section 3).
Revised: Q-posteriors both to quantify uncertainty on the critic estimates to drive exploration, as well as a tool to propagate it across the state-action space, by means of Wasserstein barycenters (Section 3).

Original: The Q-posteriors quantify the epistemic uncertainty of the Q-values and incorporate both the uncertainty due to the empirical estimate of the stochastic transition and immediate reward sample, as well as the Qvalue uncertainty of the next states imported during the bootstrapping of the Temporal Difference (TD, Sutton & Barto, 2018) updates.
Revised: The Q-posteriors quantify the epistemic uncertainty of the Q-values and incorporate both the uncertainty due to the empirical estimate of the stochastic transition and immediate reward sample, as well as the Qvalue uncertainty of the next states imported during the bootstrapping of the Temporal Difference (TD, Sutton & Barto, 2018) updates.

Original: Furthermore, we consider some practical problems that arise while quantifying uncertainty by means of Q-posteriors coped with function approximators, especially neural networks.
Revised: Furthermore, we consider some practical problems that arise while quantifying uncertainty by means of Q-posteriors coped with function approximators, especially neural networks.

Original: To this end, we propose a new regularization approach for the uncertainty networks to avoid the collapse of the uncertainty estimates due to uncontrolled generalization (Section 4).
Revised: To this end, we propose a new regularization approach for the uncertainty networks to avoid the collapse of the uncertainty estimates due to uncontrolled generalization (Section 4).

Original: WAC uses the Q-posteriors to explore efficiently, by optimizing an upper bound of the
Revised: WAC uses the Q-posteriors to explore efficiently, by optimizing an upper bound of the

Original: Q-values.
Revised: Q-values.

Original: Unlike OAC (Ciosek et al., 2019), which employs bootstrapped uncertainty estimates from an ensemble of critics (two) to define the upper bound, we employ the Qposteriors, which will eventually shrink to point estimates.
Revised: Unlike OAC (Ciosek et al., 2019), which employs bootstrapped uncertainty estimates from an ensemble of critics (two) to define the upper bound, we employ the Qposteriors, which will eventually shrink to point estimates.

Original: Furthermore, WAC recovers SAC for a specific hyperparameter configuration and, more importantly, is able to explore more efficiently with negligible additional computational costs.
Revised: Furthermore, WAC recovers SAC for a specific hyperparameter configuration and, more importantly, is able to explore more efficiently with negligible additional computational costs.

Original: After revewing the literature (Section 5), we present a thorough experimental evaluation over some simple 1D navigation domains, as well as some Mujoco (Todorov et al., 2012) tasks designed for exploration to assess the effect of uncertainty estimation and propagation on exploration and sample complexity (Section 6).
Revised: After revewing the literature (Section 5), we present a thorough experimental evaluation over some simple 1D navigation domains, as well as some Mujoco (Todorov et al., 2012) tasks designed for exploration to assess the effect of uncertainty estimation and propagation on exploration and sample complexity (Section 6).

Original: 2. Preliminaries
Revised: 2. Preliminaries

Original: Markov Decision Processes We consider infinite-horizon discounted Markov Decision Processes (MDP, Puterman, 2014).
Revised: Markov Decision Processes We consider infinite-horizon discounted Markov Decision Processes (MDP, Puterman, 2014).

Original: An MDP is a 5-tuple M =( S , A , P , R ,γ ) , defined by the state space S , the action space A , a transition kernel P : S ×A→ ∆( S ) , a reward R : S ×A→
Revised: An MDP is a 5-tuple M =( S , A , P , R ,γ ) , defined by the state space S , the action space A , a transition kernel P : S ×A→ ∆( S ) , a reward R : S ×A→

Original: ∆( R ) and the discount factor γ ∈ [0 , 1) .
Revised: ∆( R ) and the discount factor γ ∈ [0 , 1) .

Original: 1 Let r : S ×A→ R be the expectation of the reward R that we assume bounded in [ r min ,r max ] .
Revised: 1 Let r : S ×A→ R be the expectation of the reward R that we assume bounded in [ r min ,r max ] .

Original: The behavior of an agent is described by a policy π : S → ∆( A ) .
Revised: The behavior of an agent is described by a policy π : S → ∆( A ) .

Original: When the agent observes state s ∈S , it chooses an action according to policy a ∼ π ( s ) , executes it and observes a reward, r ∼R ( s,a ) and the next state sampled from the transition kernel s ′ ∼P
Revised: When the agent observes state s ∈S , it chooses an action according to policy a ∼ π ( s ) , executes it and observes a reward, r ∼R ( s,a ) and the next state sampled from the transition kernel s ′ ∼P

Original: ( s,a ) .
Revised: ( s,a ) .

Original: The performance of a policy π is measured by its state-value function V π ( s )=
Revised: The performance of a policy π is measured by its state-value function V π ( s )=

Original: E [ (cid:80) ∞ t =0 γ t r t | s 0 =
Revised: E [ (cid:80) ∞ t =0 γ t r t | s 0 =

Original: s ] , where the expectation is taken w.r.t.
Revised: s ] , where the expectation is taken w.r.t.

Original: to the stochasticity of the reward, the transition model, and the policy π .
Revised: to the stochasticity of the reward, the transition model, and the policy π .

Original: Similarly, the action-value function is defined as Q π ( s,a )=
Revised: Similarly, the action-value function is defined as Q π ( s,a )=

Original: E [ (cid:80) ∞ t =

Revised: E [ (cid:80) ∞ t =

Original: γ t r t | s 

Revised: γ t r t | s

Original: =
Revised: =

Original: s,a 0 = a ] , where we fix the first action and follow policy π for the next steps.
Revised: s,a 0 = a ] , where we fix the first action and follow policy π for the next steps.

Original: The value functions satisfy the Bellman equations,
Revised: The value functions satisfy the Bellman equations,

Original: V π ( s )=
Revised: V π ( s )=

Original: E [ r ( s,a )+ γV π
Revised: E [ r ( s,a )+ γV π

Original: ( s ′ )]
Revised: ( s ′ )]

Original: for every s ∈S , Q π ( s,a )=
Revised: for every s ∈S , Q π ( s,a )=

Original: r ( s,a )+ γ E
Revised: r ( s,a )+ γ E

Original: [ Q π ( s ′ ,a ′ )] for every ( s,a ) ∈S ×A .
Revised: [ Q π ( s ′ ,a ′ )] for every ( s,a ) ∈S ×A .

Original: The optimal action-value function Q ∗ is the maximum, over all policies, of the Q function, for all state action pairs, Q ∗ ( s,a )=
Revised: The optimal action-value function Q ∗ is the maximum, over all poli- cies, of the Q function, for all state action pairs, Q ∗ ( s,a )=

Original: sup π { Q π ( s,a ) } .
Revised: sup π { Q π ( s,a ) } .

Original: Q ∗ satisfies the Bellman optimality equation Q ∗ ( s,a )=
Revised: Q ∗ satisfies the Bellman optimality equation Q ∗ ( s,a )=

Original: r ( s,a )+ γ E [sup a ′ ∈A { Q ∗ ( s ′ ,a ′ ) } ] .
Revised: r ( s,a )+ γ E [sup a ′ ∈A { Q ∗ ( s ′ ,a ′ ) } ] .

Original: The
Revised: The

Original: Bellman equations form the basis of TD-learning, which updates the estimation of the V or Q functions in the current state using estimates of the next-states V or Q function.
Revised: Bellman equations form the basis of TD-learning, which updates the estimation of the V or Q functions in the current state using estimates of the next-states V or Q function.

Original: The goal of learning algorithms in this setting is to find the optimal policy π ∗ , which is defined as the policy that acts greedily w.r.t.
Revised: The goal of learning algorithms in this setting is to find the optimal policy π ∗ , which is defined as the policy that acts greedily w.r.t.

Original: [Equation]
Revised: Q ∗ , π ∗ ( ·| s ) ∈ ∆(argmax a ∈A { Q ∗ ( s,a ) } ) , ∀ s ∈S .

Original: Actor-Critic Methods AC methods maintain a parameterized value-function Q ω ( critic ) to estimate the value of the current (or a given target) policy, and a parameterized policy π θ ( actor ), trained through gradient descent.
Revised: Actor-Critic Methods AC methods maintain a parameterized value-function Q ω ( critic ) to estimate the value of the current (or a given target) policy, and a parameterized policy π θ ( actor ), trained through gradient descent.

Original: In particular, SAC (Haarnoja et al., 2018), employs an entropy-regularized architecture.
Revised: In particular, SAC (Haarnoja et al., 2018), employs an entropy-regularized architecture.

Original: It maintains two parameterized action-value functions { Q ω 1 ,Q ω 2 } to estimate the entropy-regularized value function of policy π θ .
Revised: It maintains two parameterized action-value functions { Q ω 1 ,Q ω 2 } to estimate the entropy-regularized value function of policy π θ .

Original: They are trained on the same samples and differ only on the initialization of ω 1 and ω 2 .
Revised: They are trained on the same samples and differ only on the initialization of ω 1 and ω 2 .

Original: The actor optimizes a “lower bound” of the action-value function,
Revised: The actor optimizes a “lower bound” of the action-value function,

Original: Q LB ( s,a )=
Revised: Q LB ( s,a )=

Original: min { Q ω 1 ( s,a ) ,Q ω 2 ( s,a ) } .
Revised: min { Q ω 1 ( s,a ) ,Q ω 2 ( s,a ) } .

Original: To update the critic, given a sample ( s,a,r,s ′ ) , SAC uses the SARSA (Sutton & Barto, 2018) update rule, Q { ω 1 , ω 2 } ( s,a ) ← r + γQ LB ( s ′ ,a ′ ) , where a ′ ∼ π θ ( s ′ ) .
Revised: To update the critic, given a sample ( s,a,r,s ′ ) , SAC uses the SARSA (Sutton & Barto, 2018) update rule, Q { ω 1 , ω 2 } ( s,a ) ← r + γQ LB ( s ′ ,a ′ ) , where a ′ ∼ π θ ( s ′ ) .

Original: Specifically, SAC maintains experience collected with previous policies π θ in a replay buffer D .
Revised: Specifically, SAC maintains experience collected with previous policies π θ in a replay buffer D .

Original: The critic is trained to minimize the (entropy regularized)
Revised: The critic is trained to minimize the (entropy regularized)

Original: Bellman error over this replay buffer, as follows:
Revised: Bellman error over this replay buffer, as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) Q ( s,a )=
Revised: where (cid:101) Q ( s,a )=

Original: Q LB ( s,a ) − α log π θ ( s ′ ,a ′ ) , Q LB ( s,a )=
Revised: Q LB ( s,a ) − α log π θ ( s ′ ,a ′ ) , Q LB ( s,a )=

Original: min { Q ω 1 ( s,a ) ,Q ω 2 ( s,a ) } is the lower bound of the Qvalues given from two target networks which are updated slowly to improve stability (Mnih et al., 2015), a ′ ∼ π θ ( s ′ ) , and α> 0 specifies the level of entropy regularization.
Revised: min { Q ω 1 ( s,a ) ,Q ω 2 ( s,a ) } is the lower bound of the Qvalues given from two target networks which are updated slowly to improve stability (Mnih et al., 2015), a ′ ∼ π θ ( s ′ ) , and α> 0 specifies the level of entropy regularization.

Original: The actor network is trained to optimize an entropy-regularized objective.
Revised: The actor network is trained to optimize an entropy-regularized objective.

Original: Since the target Q -function is a parameterized function approximator, the policy can directly follow the gradient of the critic:
Revised: Since the target Q -function is a parameterized function approximator, the policy can directly follow the gradient of the critic:

Original: [Equation]
Revised: [Equation]

Original: Wasserstein TD-Learning Bayesian RL approaches (Dearden et al., 1998; Metelli et al., 2019) maintain, for each stateaction pair ( s,a ) ∈S ×A , a probability distribution Q ( s,a ) , called a Q-posterior , used to represent the epistemic uncertainty over the value estimates.
Revised: Wasserstein TD-Learning Bayesian RL approaches (Dearden et al., 1998; Metelli et al., 2019) maintain, for each stateaction pair ( s,a ) ∈S ×A , a probability distribution Q ( s,a ) , called a Q-posterior , used to represent the epistemic uncertainty over the value estimates.

Original: In practice, Q is an approximate distribution in a class Q (e.g., Gaussians).
Revised: In practice, Q is an approximate distribution in a class Q (e.g., Gaussians).

Original: V-Posteriors can be defined, as in the classical way, as an average of the Qposteriors, by employing the notion of barycenters defined in terms of a given divergence.
Revised: V-Posteriors can be defined, as in the classical way, as an average of the Qposteriors, by employing the notion of barycenters defined in terms of a given divergence.

Original: As in (Metelli et al., 2019), we choose Wasserstein (Villani, 2008) divergence since the variance of our Q-posteriors vanishes as the number of samples grows to infinity, and the Wasserstein divergence allows computing the distance between distributions with disjoint support.
Revised: As in (Metelli et al., 2019), we choose Wasserstein ( , 2008) divergence since the variance of our Q-posteriors vanishes as the number of samples grows to infinity, and the Wasserstein divergence allows computing the distance between distributions with disjoint support.

Original: Given two probability distributions, µ and ν , the L p -Wasserstein distance between µ and ν is defined as: 
Revised: Given two probability distributions, µ and ν , the L p -Wasserstein distance between µ and ν is defined as: W p ( µ,ν )=(inf ρ ∈ Γ( µ,ν ) E X,Y ∼ ρ

Original: [Equation] /p , where Γ( µ,ν ) is the set of all joint measures with marginals µ and ν , and d is a metric.
Revised: [ d ( X,Y ) p ]) 1 /p , where Γ( µ,ν ) is the set of all joint measures with marginals µ and ν , and d is a metric.

Original: Given a class of probability distribuAlgorithm Input: critic parameters ω 1 , ω 2 , policy parameters θ , θ T Initialize Q { 1 , 2 } ( s,a ) with the prior Q 0 Initialize replay buffer D←∅ for epoch
Revised: Given a class of probability distributions N , a set of probability distributions { µ i } ni =1 , µ i ∈ N and a set of weights { ξ i } ni =1 , (cid:80) ni

Original: =1, 2 ,... do for t =1 , 2 ,... do Take action a t ∼ π θ ( ·| s t ) Observe s t +1 and r t +1 D←D (cid:83) ( s t ,a t ,r t +1 ,s t +1 ) end for σ { 1 , 2 } old ← σ ω { 1 , 2 } for iteration =1 , 2 ,... do Update critic weights ω { 1 , 2 } using Equation (8) Update actor weights θ ( θ T ) using Equation (6) (Equation (7)) end for end for = tions N , a set of probability distributions { µ i } ni =1 , µ i ∈ N and a set of weights { ξ i } ni =1 , (cid:80) n i =1 ξ i =1 and ξ i ≥ 0 , the
Revised: =1 ξ i =1 and ξ i ≥ 0 , the

Original: L 2 -Wasserstein barycenter is defined as (Agueh & Carlier,2011)
Revised: L 2 -Wasserstein barycenter is defined as (Agueh & Carlier,

Original: [Equation]
Revised: [Equation]

Original: Using the concept of Wasserstein barycenter, we can also propagate the uncertainty of the value function estimates across the state-action space.
Revised: Using the concept of Wasserstein barycenter, we can also propagate the uncertainty of the value function estimates across the state-action space.

Original: Indeed, having observed a transition ( s,a,r,s ′ ) , the Wasserstein Temporal Difference (WTD, Metelli et al., 2019) update rule is defined via the computation of the barycenter of the current Q-posterior and the TD- target posterior , defined as T t = r + γ Q t ( s ′ ,a ′ ) :
Revised: Indeed, having observed a transition ( s,a,r,s ′ ) , the Wasserstein Temporal Difference (WTD, Metelli et al., 2019) update rule is defined via the computation of the barycenter of the current Q-posterior and the TD- target posterior , defined as T t = r + γ Q t ( s ′ ,a ′ ) :

Original: [Equation]
Revised: [Equation]

Original: where α t is the learning rate, and a ′ is the action taken in the next step.
Revised: where α t is the learning rate, and a ′ is the action taken in the next step.

Original: Depending on the policy chosen to take action a ′ the update can be on-policy or off-policy.
Revised: Depending on the policy chosen to take action a ′ the update can be on-policy or off-policy.

Original: The presence of γ in the definition of T t shrinks the posteriors, vanishing the uncertainty when the number of samples grows to infinity.
Revised: The presence of γ in the definition of T t shrinks the posteriors, vanishing the uncertainty when the number of samples grows to infinity.

Original: When the Q-posteriors become point estimates, the update rule reduces to the classic TD update rule.
Revised: When the Q-posteriors become point estimates, the update rule reduces to the classic TD update rule.

Original: For some choices of distribution classes, Equation (3) can be computed in closed form.
Revised: For some choices of distribution classes, Equation (3) can be computed in closed form.

Original: We will focus on Gaussian posteriors; this is not a limiting choice, as the sample mean is approximately Gaussian with enough samples, even if the return distribution is not Gaussian in the general case.
Revised: We will focus on Gaussian posteriors; this is not a limiting choice, as the sample mean is approximately Gaussian with enough samples, even if the return distribution is not Gaussian in the general case.

Original: 3. Wasserstein Actor-Critic
Revised: 3. Wasserstein Actor-Critic

Original: In this section, we introduce Wasserstein Actor-Critic (WAC), which extends WQL (Metelli et al., 2019) to handle environments with continuous-action spaces.
Revised: In this section, we introduce Wasserstein Actor-Critic (WAC), which extends WQL (Metelli et al., 2019) to handle environments with continuous-action spaces.

Original: We present the algorithm, define the update rules, and a regularization for the uncertainty estimates.
Revised: We present the algorithm, define the update rules, and a regularization for the uncertainty estimates.

Original: Distributional Critic Similar to Bayesian approaches,
Revised: Distributional Critic Similar to Bayesian approaches,

Original: WAC uses distributional critics to represent the epistemic uncertainty on the Q-value estimates.
Revised: WAC uses distributional critics to represent the epistemic uncertainty on the Q-value estimates.

Original: For each state-action pair ( s,a ) ∈S ×A , we maintain an approximate distribution
Revised: For each state-action pair ( s,a ) ∈S ×A , we maintain an approximate distribution

Original: Q ( s,a ) ∼Q ( s,a ) , representing a Q-posterior over the possible values of Q ( s,a ) to model the uncertainty estimate on the value function.
Revised: Q ( s,a ) ∼Q ( s,a ) , representing a Q-posterior over the possible values of Q ( s,a ) to model the uncertainty estimate on the value function.

Original: While these distributions will generally depend on the aleatoric uncertainty of the environment (state  transition and reward), our updates will vanish the variance as we collect samples.
Revised: While these distributions will generally depend on the aleatoric uncertainty of the environment (state Input: parameters ω 1 , ω 2 , policy parameters θ , θ T Initialize do Observe transition and reward), our updates will vanish the variance as we collect samples.

Original: This represents our main difference w.r.t.
Revised: This represents our main difference w.r.t.

Original: Distributional RL, as we do not require learning the whole return distribution, while still propagating uncertainty across the state-action space.
Revised: Distributional RL, as we do not require learning the whole return distribution, while still propagating uncertainty across the state-action space.

Original: More specifically, given a replay buffer of past behavior D , our critic minimizes the
Revised: More specifically, given a replay buffer of past behavior D , our critic minimizes the

Original: L 2 -Wasserstein distance between the Q-posterior Q ω and the target posterior r + γ Q ω , defined through the target parameters ω and target policy π θ T :
Revised: L 2 -Wasserstein distance between the Q-posterior Q ω and the target posterior r + γ Q ω , defined through the target parameters ω and target policy π θ T :

Original: [Equation]
Revised: [Equation]

Original: Like in the original WQL paper, different flavors of the algorithm can be proposed, based on the combination of: (i) distribution classes Q , (ii) behavioral π θ , and (iii) target π θ T policies.
Revised: Like in the original WQL paper, different flavors of the algorithm can be proposed, based on the combination of: (i) distribution classes Q , (ii) behavioral π θ , and (iii) target π θ T policies.

Original: In the discrete-action case, the exploration policy can follow an optimistic approach (maximizing an upper bound on the Q-values) or a posterior sampling approach (sampling actions with the probability of being optimal).
Revised: In the discrete-action case, the exploration policy can follow an optimistic approach (maximizing an upper bound on the Q-values) or a posterior sampling approach (sampling actions with the probability of being optimal).

Original: However, when moving to continuous actions, even sampling from the distribution of the maximum requires computing a product integral (D’Eramo et al., 2017).
Revised: However, when moving to continuous actions, even sampling from the distribution of the maximum requires computing a product integral (D’Eramo et al., 2017).

Original: In addition, we could only sample from the distribution approximately (e.g., through a Monte Carlo random walk over the action space).
Revised: In addition, we could only sample from the distribution approximately (e.g., through a Monte Carlo random walk over the action space).

Original: For this reason, we focus on the optimistic exploration, that just requires optimizing upper bounds.
Revised: For this reason, we focus on the optimistic exploration, that just requires optimizing upper bounds.

Original: Finally, although other distribution classes, like particle-models, could be employed, we limit our discussion to Gaussian posteriors, as their parametrization allows for direct control over the distribution variance.
Revised: Finally, although other distribution classes, like particle-models, could be employed, we limit our discussion to Gaussian posteriors, as their parametrization allows for direct control over the distribution variance.

Original: Similar to WQL, we maintain a parameterized distributional critic using a function approximator (e.g., neural network) that outputs the parameters of the distribution.
Revised: Similar to WQL, we maintain a parameterized distributional critic using a function approximator (e.g., neural network) that outputs the parameters of the distribution.

Original: For the Gaussian case, Q ( s,a ) ∼N ( µ ω ( s,a ) ,σ ω ( s,a )) , the Wasserstein distance has a closed form, and the critic objective becomes:
Revised: For the Gaussian case, Q ( s,a ) ∼N ( µ ω ( s,a ) ,σ ω ( s,a )) , the Wasserstein distance has a closed form, and the critic objective becomes:

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) µ ω ( s,a )= µ ω ( s,a ) − α log π θ T ( s,a ) .
Revised: where (cid:101) µ ω ( s,a )= µ [Equation] ( s [Equation]

Revised: [Equation]

Original: In practice, µ ω and σ ω can use either a shared network architecture or two different networks.
Revised: log π [Equation] T ( s [Equation]

Revised: a ) .

Revised: In practice, µ ω and σ ω can use either a shared network architecture or two different networks.

Original: Similar to WQL, we initialize the posterior networks using the bias of the last layer of the network.
Revised: Similar to WQL, we initialize the posterior networks using the bias of the last layer of the network.

Original: If the reward function is limited in the interval [ r min ,r max ] , the Q values will be in the range [ q min ,q max ] with q min = r min / (1 − γ ) and q max = r max / (1 − γ ) .
Revised: If the reward function is limited in the interval [ r min ,r max ] , the Q values will be in the range [ q min ,q max ] with q min = r min / (1 − γ ) and q max = r max / (1 − γ ) .

Original: We therefore initialize the uncertainty networks to σ 0 =( q max − q min ) / √ 12 , i.e. the Gaussian minimizing the KL divergence with the uniform distribution in [ q min ,q max ] .
Revised: We therefore initialize the uncertainty networks to σ 0 =( q max − q min ) / √ 12 , i.e. the Gaussian minimizing the KL divergence with the uniform distribution in [ q min ,q max ] .

Original: Actor The actor in WAC is updated by optimizing an upper bound U δ ω of the estimated Q-value, which we can efficiently compute using Gaussian posterior: U δ ω ( s,a )= µ ω ( s,a )+ σ ω ( s,a )Φ − 1 ( δ ) , where Φ − 1 is the quantile function of the standard normal and δ ∈ (0 , 1) .
Revised: Actor The actor in WAC is updated by optimizing an upper bound U δ ω of the estimated Q-value, which we can efficiently compute using Gaussian posterior: U δ ω ( s,a )= µ ω ( s,a )+ σ ω ( s,a )Φ − 1 ( δ ) , where Φ − 1 is the quantile function of the standard normal and δ ∈ (0 , 1) .

Original: When actions are finite, no actor is needed, as we can compute the maximum by enumeration.
Revised: When actions are finite, no actor is needed, as we can compute the maximum by enumeration.

Original: However, in our case, we need an actor that follows U δ ω ( s,a ) , which is differentiable in ω , leading to the minimization of the objective:
Revised: However, in our case, we need an actor that follows U δ ω ( s,a ) , which is differentiable in ω , leading to the minimization of the objective:

Original: [Equation]
Revised: [Equation]

Original: where θ are the parameters of the behavioral policy.
Revised: where θ are the parameters of the behavioral policy.

Original: In practice, as discussed in Section 2, we employ a double critic strategy to improve stability, like SAC, i.e., we maintain two distributional critics and use the minimum of the two upper bounds, as target:
Revised: In practice, as discussed in Section 2, we employ a double critic strategy to improve stability, like SAC, i.e., we maintain two distributional critics and use the minimum of the two upper bounds, as target:

Original: U δ ω ( s,a )=
Revised: U δ ω ( s,a )=

Original: min { U δ

Original: [Equation]

Original:  1 ( s, [Equation] ) ,U δ ω 2 ( s,a [Equation]
Revised: min { U δ ω 1 ( s,a ) ,U δ ω 2 ( s,a ) } .

Original: Target Policy We propose two alternatives for the target policy π θ T , corresponding to different estimators for the target posterior T t .
Revised: Target Policy We propose two alternatives for the target policy π θ T , corresponding to different estimators for the target posterior T t .

Original: First, we can use the same policy we use for exploration, i.e., θ = θ T , like SAC.
Revised: First, we can use the same policy we use for exploration, i.e., θ = θ T , like SAC.

Original: This has the advantage of not requiring a second parameterized policy.
Revised: This has the advantage of not requiring a second parameterized policy.

Original: We call this version Optimistic Estimator-WAC (OE-WAC), which represents an on-policy algorithm.
Revised: We call this version Optimistic Estimator-WAC (OE-WAC), which represents an on-policy algorithm.

Original: Alternatively, we can use a greedy policy that optimizes the expected value of the Q-posteriors (the mean critic µ ω ( s,a ) in the Gaussian case).
Revised: Alternatively, we can use a greedy policy that optimizes the expected value of the Q-posteriors (the mean critic µ ω ( s,a ) in the Gaussian case).

Original: In this case, the target policy minimizes:
Revised: In this case, the target policy minimizes:

Original: [Equation]
Revised: [Equation]

Original: We call this version Mean Estimator-WAC (ME-WAC).
Revised: We call this version Mean Estimator-WAC (ME-WAC).

Original: The best version to use between the two is task-dependent.
Revised: The best version to use between the two is task-dependent.

Original: Generally, OE-WAC is more suitable for environments that require large exploration, whereas ME-WAC is more suitable for simpler environments where OE-WAC might overexplore and might suffer from some instability.
Revised: Generally, OE-WAC is more suitable for environments that require large exploration, whereas ME-WAC is more suitable for simpler environments where OE-WAC might over- explore and might suffer from some instability.

Original: 4. Regularized Uncertainty Estimation
Revised: 4. Regularized Uncertainty Estimation

Original: Our Q-posteriors are initialized to high uncertainty at the beginning of the learning process.
Revised: Our Q-posteriors are initialized to high uncertainty at the beginning of the learning process.

Original: Since they represent epistemic uncertainty, their variance will shrink as we observe more samples.
Revised: Since they represent epistemic uncertainty, their variance will shrink as we observe more samples.

Original: This is apparent in Equation (5), where the targets σ ω are multiplied with γ .
Revised: This is apparent in Equation (5), where the targets σ ω are multiplied with γ .

Original: In tabular settings, the updates are localized, i.e., they affect a single state-action pair, without interfering with the others.
Revised: In tabular settings, the updates are localized, i.e., they affect a single state-action pair, without interfering with the others.

Original: However, when function approximators are involved, generalizing uncertainty in an uncontrolled way might cause non-visited areas of the stateaction space to take low uncertainty values, which might be undesired.
Revised: However, when function approximators are involved, generalizing uncertainty in an uncontrolled way might cause non-visited areas of the stateaction space to take low uncertainty values, which might be undesired.

Original: 3 Consider the example in Figure 1.
Revised: 3 Consider the example in Figure 1.

Original: The graph shows the uncertainty estimate as a function of the action, in a fixed state s .
Revised: The graph shows the uncertainty estimate as a function of the action, in a fixed state s .

Original: Starting from an initial high constant estimate of σ 0 , at the beginning of the learning process, we will observe samples like the red crosses in the figure, i.e., with lower uncertainty since it gets shrunk with γ .
Revised: Starting from an initial high constant estimate of σ 0 , at the beginning of the learning process, we will observe samples like the red crosses in the figure, i.e., with lower uncertainty since it gets shrunk with γ .

Original: Among all the possible fitting lines, we would prefer an estimate like σ 31 , which keeps high uncertainty in unseen regions, and would like to avoid failures like σ 11 .
Revised: Among all the possible fitting lines, we would prefer an estimate like σ 31 , which keeps high uncertainty in unseen regions, and would like to avoid failures like σ 11 .

Original: This requires controlling the “smoothness” properties of the approximator.
Revised: This requires controlling the “smoothness” properties of the approximator.

Original: To avoid the additional computational burden, we propose a simple scheme based on synthetic samples.
Revised: To avoid the additional computational burden, we propose a simple scheme based on synthetic samples.

Original: Specifically, we periodically save the weights of the uncertainty network σ old and use it as the target for state-action pairs drawn uniformly from the state-action space.
Revised: Specifically, we periodically save the weights of the uncertainty network σ old and use it as the target for state-action pairs drawn uniformly from the state-action space.

Original: More formally, our distributional critic minimizes:
Revised: More formally, our distributional critic minimizes:

Original: [Equation]
Revised: [Equation]

Original: where J C ( ω { 1 , 2 } ) is defined in Equation (5) and λ ≥ 0 defines the relative weight of the regularization.
Revised: where J C ( ω { 1 , 2 } ) is defined in Equation (5) and λ ≥ 0 defines the relative weight of the regularization.

Original: Furthermore, in practice we add a second parameter, ρ ∈ [0 , 1] which represents the fraction of fake samples (w.r.t.
Revised: Furthermore, in practice we add a second parameter, ρ ∈ [0 , 1] which represents the fraction of fake samples (w.r.t.

Original: the samples used for J C ( ω { 1 , 2 } ) ) drawn for regularization.
Revised: the samples used for J C ( ω { 1 , 2 } ) ) drawn for regularization.

Original: Specifically, if we estimate J C ( ω { 1 , 2 } using N samples from replay buffer
Revised: Specifically, if we estimate J C ( ω { 1 , 2 } using N samples from replay buffer

Original: D , we will estimate the expectation in Equation (8) with
Revised: D , we will estimate the expectation in Equation (8) with

Original: M = ρN samples from U ( S ×A ) .
Revised: M = ρN samples from U ( S ×A ) .

Original: In Section 6, we will see that for simple environments, small values of ρ are enough, while for more complex tasks the regularizer will require larger ρ values.
Revised: In Section 6, we will see that for simple environments, small values of ρ are enough, while for more complex tasks the regularizer will require larger ρ values.

Original: Algorithm 1 reports the pseudocode of
Revised: Algorithm 1 reports the pseudocode of

Original: WAC, embedding the regularized uncertainty estimation.
Revised: WAC, embedding the regularized uncertainty estimation.

Original: To investigate the effectiveness of the regularized uncertainty loss, on an illustrative example, we trained two different agents, in a one-dimensional Linear Quadratic Regulator (LQG, Dorato et al., 2000).
Revised: To investigate the effectiveness of the regularized uncertainty loss, on an illustrative example, we trained two different agents, in a one-dimensional Linear Quadratic Regulator (LQG, Dorato et al., 2000).

Original: This task has a onedimensional state and action spaces, which allow us to visualize the uncertainty estimates.
Revised: This task has a onedimensional state and action spaces, which allow us to visualize the uncertainty estimates.

Original: As function approximator, we used a two-layer MLP (Bishop, 2006) of 128 neurons per layer.
Revised: As function approximator, we used a two-layer MLP (Bishop, 2006) of 128 neurons per layer.

Original: Figure 2 shows the comparison of the uncertainty estimation with and without the regularized uncertainty loss.
Revised: Figure 2 shows the comparison of the uncertainty estimation with and without the regularized uncertainty loss.

Original: The first row depicts the case without regularization (Equation 5) and the second row with regularization (Equation 8).
Revised: The first row depicts the case without regularization (Equation 5) and the second row with regularization (Equation 8).

Original: On the left, we show the empirical state-action visitation distribution.
Revised: On the left, we show the empirical state-action visitation distribution.

Original: The agent starts in one of the borders of the state space and has to reach the center in a few steps while calibrating the actions (there is a punishment for high actions).
Revised: The agent starts in one of the borders of the state space and has to reach the center in a few steps while calibrating the actions (there is a punishment for high actions).

Original: This is apparent in both left plots, with the highest densities in the borders and the center.
Revised: This is apparent in both left plots, with the highest densities in the borders and the center.

Original: We consider desirable to obtain uncertainty estimates that mirror these state-action densities, as the epistemic uncertainty is inversely proportional to the state-action visitation.
Revised: We consider desirable to obtain uncertainty estimates that mirror these state-action densities, as the epistemic uncertainty is inversely proportional to the state-action visitation.

Original: While in both cases, the state-action densities are similar, the uncertainty estimates are completely different.
Revised: While in both cases, the state-action densities are similar, the uncertainty estimates are completely different.

Original: In the top-right figure, we see the output of the uncertainty critic, without regularization, completely fails to represent the uncertainty.
Revised: In the top-right figure, we see the output of the uncertainty critic, without regularization, completely fails to represent the uncertainty.

Original: On the bottomright corner, we can see that the regularized uncertainty critic, almost perfectly matches the state-action densities, with low uncertainty in the center and a gradual increase as we move away and then again a sharp decrease close to the borders.
Revised: On the bottomright corner, we can see that the regularized uncertainty critic, almost perfectly matches the state-action densities, with low uncertainty in the center and a gradual increase as we move away and then again a sharp decrease close to the borders.

Original: In Section 6 and Appendix B we show a more thorough investigation of the effect of the regularized uncertainty loss.
Revised: In Section 6 and Appendix B we show a more thorough investigation of the effect of the regularized uncertainty loss.

Original: 5. Related Works
Revised: 5. Related Works

Original: There is a large body of literature studying efficient exploration techniques in RL.
Revised: There is a large body of literature studying efficient exploration techniques in RL.

Original: In the tabular settings, provably efficient methods have been devised, both in the modelbased (Auer et al., 2008; Jaksch et al., 2010; Ian et al., 2013) and model-free (Jin et al., 2018; Strehl et al., 2006;
Revised: In the tabular settings, provably efficient methods have been devised, both in the modelbased (Auer et al., 2008; Jaksch et al., 2010; Ian et al., 2013) and model-free (Jin et al., 2018; Strehl et al., 2006;

Original: Metelli et al., 2019) settings.
Revised: Metelli et al., 2019) settings.

Original: These methods cannot be easily extended to the Deep RL setting, or when extensions are proposed, they lose their theoretical guarantees.
Revised: These methods cannot be easily extended to the Deep RL setting, or when extensions are proposed, they lose their theoretical guarantees.

Original: In this section, we focus on tractable exploration methods proposed for Deep RL for continuous action spaces.
Revised: In this section, we focus on tractable exploration methods proposed [Figure] [Figure] [Figure]

Revised: for Deep RL for continuous action spaces.

Original: Two main exploration frameworks exist: uncertainty-based methods and intrinsic motivation methods.
Revised: Two main exploration frameworks exist: uncertainty-based methods and intrinsic motivation methods.

Original: Uncertainty-Based Methods Classical value-based methods (including ACs) maintain a point estimate of the value functions for each state (or state-action pairs).
Revised: Uncertainty-Based Methods Classical value-based methods (including ACs) maintain a point estimate of the value functions for each state (or state-action pairs).

Original: Exploration policies, like ϵ -greedy or Boltzmann (Sutton & Barto, 2018), add noise around the greedy action derived from these point estimates.
Revised: Exploration policies, like ϵ -greedy or Boltzmann (Sutton & Barto, 2018), add noise around the greedy action derived from these point estimates.

Original: These methods are not efficient, mainly because the exploration is not directed towards unvisited regions of[Figure] [Figure] the state space.
Revised: These methods are not efficient, mainly because the exploration is not directed towards unvisited regions of the state space.

Original: The entropy regularization of SAC is a form of undirected exploration too, as the policies are trained to sacrifice some returns to preserve stochastic behavior.
Revised: The entropy regularization of SAC is a form of undirected exploration too, as the policies are trained to sacrifice some returns to preserve stochastic behavior.

Original: In recent years, several methods that move away from point estimates have been proposed.
Revised: In recent years, several methods that move away from point estimates have been proposed.

Original: Ensemble methods (Chen et al., 2021; Wang et al., 2021) implicitly model the epistemic uncertainty of the Q-value estimates by maintaining multiple Q-function approximators.
Revised: Ensemble methods (Chen et al., 2021; Wang et al., 2021) implicitly model the epistemic uncertainty of the Q-value estimates by maintaining multiple Q-function approximators.

Original: OAC (Ciosek et al., 2019) explicitly models the uncertainty on the value estimates by computing the variance of two critics, and it uses it to compute an exploration policy that optimizes an upper bound of the Q -values.
Revised: OAC (Ciosek et al., 2019) explicitly models the uncertainty on the value estimates by computing the variance of two critics, and it uses it to compute an exploration policy that optimizes an upper bound of the Q -values.

Original: Unfortunately, this uncertainty estimate is just heuristic and only stems from the disagreement between the two Q -networks with different initialization.
Revised: Unfortunately, this uncertainty estimate is just heuristic and only stems from the disagreement between the two Q -networks with different initialization.

Original: Indeed, the networks are also trained with the same samples, and same target Q -values, so any disagreement is purely due to the random initialization only.
Revised: Indeed, the networks are also trained with the same samples, and same target Q -values, so any disagreement is purely due to the random initialization only.

Original: Recently,
Revised: Recently,

Original: SUNRISE (Lee et al., 2021) proposes a framework to unify ensemble methods for epistemic uncertainty estimation and shows considerable performance improvements in discrete and continuous action spaces.
Revised: SUNRISE (Lee et al., 2021) proposes a framework to unify ensemble methods for epistemic uncertainty estimation and shows considerable performance improvements in discrete and continuous action spaces.

Original: Distributional RL (Bellemare et al., 2017), on the other hand, models the aleatoric uncertainty, as its goal is to estimate the whole return distribution.
Revised: Distributional RL (Bellemare et al., 2017), on the other hand, models the aleatoric uncertainty, as its goal is to estimate the whole return distribution.

Original: First proposed for problems with a discrete action space (Bellemare et al., 2017; Dabney et al., 2018; Mavrin et al., 2019), it has been successfully extended also to the
Revised: First proposed for problems with a discrete action space (Bellemare et al., 2017; Dabney et al., 2018; Mavrin et al., 2019), it has been successfully extended also to the

Original: AC setting in TOP (Moskovitz et al., 2021).
Revised: AC setting in TOP (Moskovitz et al., 2021).

Original: TOP models both aleatoric and epistemic uncertainty and adapts the level of optimism/pessimism by means of a Multi-Armed Bandit (MAB, Lattimore, 2020 - 2020) approach.
Revised: TOP models both aleatoric and epistemic uncertainty and adapts the level of optimism/pessimism by means of a Multi-Armed Bandit (MAB, Lattimore, 2020 - 2020) approach.

Original: While TOP deals with uncertainty propagation, it mixes the epistemic and aleatoric uncertainty while estimating the return distribution.
Revised: While TOP deals with uncertainty propagation, it mixes the epistemic and aleatoric uncertainty while estimating the return distribution.

Original: To the best of our knowledge, WAC is the first method able to propagate epistemic uncertainty in continuous action spaces, without the need for estimating the full return distributions.
Revised: To the best of our knowledge, WAC is the first method able to propagate epistemic uncertainty in continuous action spaces, without the need for estimating the full return distributions.

Original: Intrinsic Motivation Tractable model-free methods based on intrinsic motivation have been proposed in recent years.
Revised: Intrinsic Motivation Tractable model-free methods based on intrinsic motivation have been proposed in recent years.

Original: Methods based on pseudo-counts (Bellemare et al., 2016;
Revised: Methods based on pseudo-counts (Bellemare et al., 2016;

Original: Ostrovski et al., 2017) assign exploration bonuses according to the novelty of the state-action pairs visited.
Revised: Ostrovski et al., 2017) assign exploration bonuses according to the novelty of the state-action pairs visited.

Original: While they have been applied with good results to deep architectures, they generally rely on (often pre-trained) density-models, which are not straightforward to maintain.
Revised: While they have been applied with good results to deep architectures, they generally rely on (often pre-trained) density-models, which are not straightforward to maintain.

Original: Other methods apply exploration bonuses based on the state-action visitation density of the policy.
Revised: Other methods apply exploration bonuses based on the state-action visitation density of the policy.

Original: MADE
Revised: MADE

Original: (Zhang et al., 2021) adds an exploration bonus, based on the deviation of the state visitation density of the new policy from the last observed policies.
Revised: (Zhang et al., 2021) adds an exploration bonus, based on the deviation of the state visitation density of the new policy from the last observed policies.

Original: While it has been applied to continuous state-action spaces, it comes with a considerable computational cost to estimate the state densities and also requires pre-training of density models.
Revised: While it has been applied to continuous state-action spaces, it comes with a considerable computational cost to estimate the state densities and also requires pre-training of density models.

Original: State entropy maximiza- tion (Mutti et al., 2021; Seo et al., 2021; Yarats et al., 2021) has also been applied as an incentive to explore the whole state-action space, including hard to reach regions.
Revised: State entropy maximization (Mutti et al., 2021; Seo et al., 2021; Yarats et al., 2021) has also been applied as an incentive to explore the whole state-action space, including hard to reach regions.

Original: These methods generally scale better to continuous domains, as they do not explicitly need to estimate the state occupancy but only the entropy of this distribution.
Revised: These methods generally scale better to continuous domains, as they do not explicitly need to estimate the state occupancy but only the entropy of this distribution.

Original: Numerous methods have also been proposed, with bonuses based on the information-gain (Houthooft et al., 2016; Achiam & Sastry, 2017; Pathak et al., 2019), but come with considerable computational costs to estimate these bonuses.
Revised: Numerous methods have also been proposed, with bonuses based on the information-gain (Houthooft et al., 2016; Achiam & Sastry, 2017; Pathak et al., 2019), but come with considerable computational costs to estimate these bonuses.

Original: 6. Experiments
Revised: 6. Experiments

Original: In this section, we present the empirical evaluation of WAC in various continuous control domains.
Revised: In this section, we present the empirical evaluation of WAC in various continuous control domains.

Original: We start from simple domains like LQG and a continuous version of Riverswim (Strehl & Littman, 2008), where we can better visualize the effects of the Q-posteriors in the learning and exploration process.
Revised: We start from simple domains like LQG and a continuous version of Riverswim (Strehl & Littman, 2008), where we can better visualize the effects of the Q-posteriors in the learning and exploration process.

Original: Then we evaluate WAC on a set of Mujoco (Todorov et al., 2012) tasks designed for exploration.
Revised: Then we evaluate WAC on a set of Mujoco (Todorov et al., 2012) tasks designed for exploration.

Original: 1D Navigation
Revised: 1D Navigation

Original: The goal of this set of experiments is to measure the effect of uncertainty estimation on exploration.
Revised: The goal of this set of experiments is to measure the effect of uncertainty estimation on exploration.

Original: We keep track of the cumulative coverage of the state-action space, i.e., the portion of the total volume visited with relative frequency larger than ϵ> 0 .
Revised: We keep track of the cumulative coverage of the state-action space, i.e., the portion of the total volume visited with relative frequency larger than ϵ> 0 .

Original: For this reason, we perform an empirical evaluation of WAC in two simple 1D navigation tasks.
Revised: For this reason, we perform an empirical evaluation of WAC in two simple 1D navigation tasks.

Original: We consider a one-dimensional LQG, an environment with no particular exploration challenges, and a more challenging continuous-action version of the Riverswim chain, where long sequences of rewardless actions are needed to reach high reward states.
Revised: We consider a one-dimensional LQG, an environment with no particular exploration challenges, and a more challenging continuous-action version of the Riverswim chain, where long sequences of rewardless actions are needed to reach high reward states.

Original: A full description of the environments is reported in Appendix A.
Revised: A full description of the environments is reported in Appendix A.

Original: Figure 3 shows the results of these experiments.
Revised: Figure 3 shows the results of these experiments.

Original: For each environment, we train WAC, varying the parameters λ and ρ of the regularized uncertainty loss in Equation (8).
Revised: For each environment, we train WAC, varying the parameters λ and ρ of the regularized uncertainty loss in Equation (8).

Original: For each value, we report the coverage averaged over all training epochs.
Revised: For each value, we report the coverage averaged over all training epochs.

Original: Firstly, we observe that both parameters directly control the amount of exploration.
Revised: Firstly, we observe that both parameters directly control the amount of exploration.

Original: Indeed, the coverage is monotonically increasing with both λ and ρ .
Revised: Indeed, the coverage is monotonically increasing with both λ and ρ .

Original: As expected, low values of ρ cause higher variance, as fewer samples are employed to estimate the uncertainty regularization.
Revised: As expected, low values of ρ cause higher variance, as fewer samples are employed to estimate the uncertainty regularization.

Original: This can be seen in all the curves of the leftmost plot, as well as in the third plot, where the black curve corresponding to ρ =0 .
Revised: This can be seen in all the curves of the leftmost plot, as well as in the third plot, where the black curve corresponding to ρ =0 .

Original: 25 suffers from a high variance.
Revised: 25 suffers from a high variance.

Original: In Appendix B we perform a similar study for OAC, varying the parameters β and δ which control exploration, and we observe that the coverage is not so easily controllable with these parameters.
Revised: In Appendix B we perform a similar study for OAC, varying the parameters β and δ which control exploration, and we observe that the coverage is not so easily controllable with these parameters.

Original: We attribute this to the heuristic nature of the uncertainty estimation of OAC, based on the critics’ disagreement only.
Revised: We attribute this to the heuristic nature of the uncertainty estimation of OAC, based on the critics’ disagreement only.

Original:  To assess whether a principled uncertainty estimation and propagation translate into lower sample complexity, we perform an empirical evaluation in a set of Mujoco (Todorov et al., 2012) tasks, where the amount of exploration needed to solve the task can be controlled.
Revised: 2D Navigation To assess whether a principled uncertainty estimation and propagation translate into lower sample complexity, we perform an empirical evaluation in a set of Mujoco (Todorov et al., 2012) tasks, where the amount of exploration needed to solve the task can be controlled.

Original: We start from the 2D navigation task used in (Moro et al., 2022), where the agent has to reach a goal state in a 2D world, by avoiding obstacles.
Revised: We start from the 2D navigation task used in (Moro et al., 2022), where the agent has to reach a goal state in a 2D world, by avoiding obstacles.

Original: The reward is the negative Euclidean distance from the goal state.
Revised: The reward is the negative Euclidean distance from the goal state.

Original: While this is a dense reward , the obstacle presence generates local optima which the agent needs to overcome by exploring efficiently.
Revised: While this is a dense reward , the obstacle presence generates local optima which the agent needs to overcome by exploring efficiently.

Original: We progressively make the task more challenging by adding additional walls and obstacles.
Revised: We progressively make the task more challenging by adding additional walls and obstacles.

Original: A visual representation of the tasks is shown in Figure 4a.
Revised: A visual representation of the tasks is shown in Figure 4a.

Original: We leave the full description of the environments in Appendix A .
Revised: We leave the full description of the environments in Appendix A .

Original: We name the tasks as “Point x ” with x ∈{ 1 , 2 , 3 , 4 } , where a higher x means a more difficult exploration challenge.
Revised: We name the tasks as “Point x ” with x ∈{ 1 , 2 , 3 , 4 } , where a higher x means a more difficult exploration challenge.

Original: We compare the performance of WAC, in both versions defined in Section 3, with SAC and OAC.
Revised: We compare the performance of WAC, in both versions defined in Section 3, with SAC and OAC.

Original: In each task, we track the cumulative return, as well as the number of episodes completed in a fixed number of steps (higher is better).
Revised: In each task, we track the cumulative return, as well as the number of episodes completed in a fixed number of steps (higher is better).

Original: We use the implementation of
Revised: We use the implementation of

Original: SAC and OAC used in (Ciosek et al., 2019), and extend the repository with our implementation, to guarantee comparable results.
Revised: SAC and OAC used in (Ciosek et al., 2019), and extend the repository with our implementation, to guarantee comparable results.

Original: The same network architectures are used for all algorithms.
Revised: The same network architectures are used for all algorithms.

Original: For the common hyperparameters, we only tune SAC and use the same values for WAC and OAC, by additionally tuning the algorithm specific parameters ( δ and β for OAC and λ and ρ for WAC).
Revised: For the common hyperparameters, we only tune SAC and use the same values for WAC and OAC, by additionally tuning the algorithm specific parameters ( δ and β for OAC and λ and ρ for WAC).

Original: Details on the hyperparameter tuning are in Appendix A .
Revised: Details on the hyperparameter tuning are in Appendix A .

Original: In Figure 4b, we present the average return as a function of the training epochs, whereas in Figure 4c we present the number of episodes completed in 3000 steps of interaction.
Revised: In Figure 4b, we present the average return as a function of the training epochs, whereas in Figure 4c we present the number of episodes completed in 3000 steps of interaction.

Original: Starting from left to right, we increase the difficulty of the task.
Revised: Starting from left to right, we increase the difficulty of the task.

Original: We can see that for the easiest task, all algorithms are able to find the optimal policy of quickly avoiding an obstacle in the middle to reach the goal state, even though
Revised: We can see that for the easiest task, all algorithms are able to find the optimal policy of quickly avoiding an obstacle in the middle to reach the goal state, even though

Original: SAC learns slower compared to the others.
Revised: SAC learns slower compared to the others.

Original: Being a simple exploration task, ME-WAC performs better and is more stable than OE-WAC.
Revised: Being a simple exploration task, ME-WAC performs better and is more stable than OE-WAC.

Original: OAC is also able to quickly solve the task.
Revised: OAC is also able to quickly solve the task.

Original: While the difference in return is negligible, the number of completed episodes shows an advantage for MEWAC, which completes more episodes faster.
Revised: While the difference in return is negligible, the number of completed episodes shows an advantage for MEWAC, which completes more episodes faster.

Original: Finally, we underline that even though the task does not require particular exploration, WAC does not over-explore, but rather solves with a speed comparable with the other baselines.
Revised: Finally, we underline that even though the task does not require particular exploration, WAC does not over-explore, but rather solves with a speed comparable with the other baselines.

Original: The clear advantages of WAC in terms of exploration can  be seen starting from the second task, where the exploration requirements are increased.
Revised: The clear advantages of WAC in terms of exploration can [Figure] be seen starting from the second task, where the exploration requirements are increased.

Original: Both versions of WAC learn faster and with less variance compared with both SAC and
Revised: Both versions of WAC learn faster and with less variance compared with both SAC and

Original: OAC.
Revised: OAC.

Original: The difference is even more apparent in the number of episodes completed, where SAC and WAC have disjoint confidence intervals.
Revised: The difference is even more apparent in the number of episodes completed, where SAC and WAC have disjoint confidence intervals.

Original: In the third task, SAC completely fails in learning to reach the goal, while OAC succeeds in some of the seeds only, showing a high variance.
Revised: In the third task, SAC completely fails in learning to reach the goal, while OAC succeeds in some of the seeds only, showing a high variance.

Original: WAC, on the other hand, outperforms them both in terms of return and[Figure] [Figure] completed episodes.
Revised: WAC, on the other hand, outperforms them both in terms of return and completed episodes.

Original: ME-WAC performs better, even though the task requires a good amount of exploration.
Revised: ME-WAC performs better, even though the task requires a good amount of exploration.

Original: Compared to ME-WAC, OE-WAC over-explores and it shows a slower learning curve.
Revised: Compared to ME-WAC, OE-WAC over-explores and it shows a slower learning curve.

Original: The last task is solved by the WAC agents only.
Revised: The last task is solved by the WAC agents only.

Original: SAC and OAC never reach the goal state.
Revised: SAC and OAC never reach the goal state.

Original: WAC outperforms them, in both versions, with statistical significance.
Revised: WAC outperforms them, in both versions, with statistical significance.

Original: We also see the need for larger exploration, apparent from the difference in performance between OE-WAC and MEWAC.
Revised: We also see the need for larger exploration, apparent from the difference in performance between OE-WAC and MEWAC.

Original: Finally, Figure 4d presents the number of episodes completed in a sparse reward version of the same tasks.
Revised: Finally, Figure 4d presents the number of episodes completed in a sparse reward version of the same tasks.

Original: In this scenario, we do not show the return as it is proportional to the number of completed episodes.
Revised: In this scenario, we do not show the return as it is proportional to the number of completed episodes.

Original: We only trained OEWAC agents in these tasks, as they present a substantial exploration challenge.
Revised: We only trained OEWAC agents in these tasks, as they present a substantial exploration challenge.

Original: The advantage of WAC is extremely evident in these tasks.
Revised: The advantage of WAC is extremely evident in these tasks.

Original: SAC and OAC are only able to solve the simplest task.
Revised: SAC and OAC are only able to solve the simplest task.

Original: In a sparse reward setting, SAC and OAC will only explore randomly so they fully rely on the chance of reaching the goal state with random actions.
Revised: In a sparse reward setting, SAC and OAC will only explore randomly so they fully rely on the chance of reaching the goal state with random actions.

Original: OAC explores more compared to SAC, but since the exploration does not depend on the state-action visitations, but only on the disagreement between the critics, sparse reward tasks are a great challenge.
Revised: OAC explores more compared to SAC, but since the exploration does not depend on the state-action visitations, but only on the disagreement between the critics, sparse reward tasks are a great challenge.

Original: WAC, instead, will still explore, even when facing sparse rewards since the uncertainty will gradually decline in visited regions, so the upper bounds will favor reaching unvisited ones.
Revised: WAC, instead, will still explore, even when facing sparse rewards since the uncertainty will gradually decline in visited regions, so the upper bounds will favor reaching unvisited ones.

Original: Indeed, OE-WAC outperforms both baselines in all the tasks with sparse rewards.
Revised: Indeed, OE-WAC outperforms both baselines in all the tasks with sparse rewards.

Original: 7. Conclusions
Revised: 7. Conclusions

Original: In this paper, we presented a novel AC algorithm to perform directed exploration.
Revised: In this paper, we presented a novel AC algorithm to perform directed exploration.

Original: We presented WAC, which extends the recently proposed WQL to the continuous-actions case.
Revised: We presented WAC, which extends the recently proposed WQL to the continuous-actions case.

Original: Furthermore, we addressed a problem of uncertainty estimation that arises when using function approximation, related to the generalization of the uncertainty estimates.
Revised: Furthermore, we addressed a problem of uncertainty estimation that arises when using function approximation, related to the generalization of the uncertainty estimates.

Original: We proposed a simple, yet effective, regularization method, based on synthetic samples that allowed us to better generalize the uncertainty across the state-action space.
Revised: We proposed a simple, yet effective, regularization method, based on synthetic samples that allowed us to better generalize the uncertainty across the state-action space.

Original: Finally, we performed a thorough empirical evaluation to investigate the advantages of performing a principled uncertainty estimation and propagation in continuous-action domains.
Revised: Finally, we performed a thorough empirical evaluation to investigate the advantages of performing a principled uncertainty estimation and propagation in continuous-action domains.

Original: We observed that the uncertainty estimates of WAC can effectively steer exploration towards promising regions of the state-action space, even under sparse rewards, especially when comparing it with heuristic uncertainty estimation based on ensemble methods.
Revised: We observed that the uncertainty estimates of WAC can effectively steer exploration towards promising regions of the state-action space, even under sparse rewards, especially when comparing it with heuristic uncertainty estimation based on ensemble methods.

Original: Future work includes extending the current method to posterior sampling exploration strategies.
Revised: Future work includes extending the current method to posterior sampling exploration strategies.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. Experimental Details A.1. Environments Description
Revised: A. Experimental Details A.1. Environments Description

Original: RiverSwim We extend the classical Riverswim domain (Strehl & Littman, 2008) to a continuous setting.
Revised: RiverSwim We extend the classical Riverswim domain (Strehl & Littman, 2008) to a continuous setting.

Original: In this environment, the agent has to navigate a 1 dimensional state space, ranging from 0 to max _ state , by applying a 1 dimensional action, representing the intended movement a ∈ [ − 1 1] .
Revised: In this environment, the agent has to navigate a 1 dimensional state space, ranging from 0 to max _ state , by applying a 1 dimensional action, representing the intended movement a ∈ [ − 1 1] .

Original: The initial state is a uniformly distributed in [0 0 . 5] .
Revised: The initial state is a uniformly distributed in [0 0 . 5] .

Original: When an action is chosen the agent moves left or right on the state space.
Revised: When an action is chosen the agent moves left or right on the state space.

Original: The distance of the movement is equal to the absolute value of the action (if the result is outside the state space a clip operations brings it back inside).
Revised: The distance of the movement is equal to the absolute value of the action (if the result is outside the state space a clip operations brings it back inside).

Original: The direction d ∈{− 1 , 0 , 1 } of the movement is stochastic, according to the following probabilities:
Revised: The direction d ∈{− 1 , 0 , 1 } of the movement is stochastic, according to the following probabilities:

Original: [Equation]
Revised: [Equation]

Original: Given the current state s t , the action a t and the direction d t sampled according the previous probabilities, the next state is s t +1 =
Revised: Given the current state s t , the action a t and the direction d t sampled according the previous probabilities, the next state is s t +1 =

Original: clip ( s t + d t | a t | ) , where clip clips the state in the range [0 max _ state ]
Revised: clip ( s t + d t | a t | ) , where clip clips the state in the range [0 max _ state ]

Original: The reward depends on the starting state s and the action sign: 
Revised: The reward depends on the starting state s and the action sign: r =  

Original: [Equation]
Revised: [Equation]

Original: 5 · 10 −if s ≤ if s ≥ ( max _ state − 1) and a> otherwise
Revised: [Equation] if s ≤ if s ≥ ( max _ state − 1) and a> otherwise

Original: The optimal policy is to always perform a =1 which gives the agent the best chance of moving toward high reward states.
Revised: The optimal policy is to always perform a =1 which gives the agent the best chance of moving toward high reward states.

Original: In our experiments:
Revised: In our experiments:

Original: • max _ state =
Revised: • max _ state =

Original: LQG We test our agents also on an instance of a Linear Quadratic Gaussian control.
Revised: LQG We test our agents also on an instance of a Linear Quadratic Gaussian control.

Original: Given a state x , an action a , and v ∼N (0 , 0 . 5) the next state and the cost c (= − r ) are defined as:
Revised: Given a state x , an action a , and v ∼N (0 , 0 . 5) the next state and the cost c (= − r ) are defined as:

Original: [Equation]
Revised: [Equation]

Original: In our experiments, we use A =1 ,B =1 ,Q =0 .
Revised: In our experiments, we use A =1 ,B =1 ,Q =0 .

Original: 9 ,R =0 .
Revised: 9 ,R =0 .

Original: The agents starts from the borders of the state space and, with this configuration, the goal is to reach the origin of the state space while balancing the actions.
Revised: The agents starts from the borders of the state space and, with this configuration, the goal is to reach the origin of the state space while balancing the actions.

Original: Point
Revised: Point

Original: This environment models a sphere moving inside a two-dimensional maze.
Revised: This environment models a sphere moving inside a two-dimensional maze.

Original: The goal of the agent is to get close enough to a goal state on the right side of the maze, while avoiding obstacles.
Revised: The goal of the agent is to get close enough to a goal state on the right side of the maze, while avoiding obstacles.

Original: Once the agent gets close enough to the goal (Euclidean distance < 2 ) the episode ends.
Revised: Once the agent gets close enough to the goal (Euclidean distance < 2 ) the episode ends.

Original: The state space includes the agent position in the 2 -dimensional space, as well as the velocities.
Revised: The state space includes the agent position in the 2 -dimensional space, as well as the velocities.

Original: The action space is also 2 -dimensional, controlling the actuators in both directions.
Revised: The action space is also 2 -dimensional, controlling the actuators in both directions.

Original: We use 2 reward functions for this task.
Revised: We use 2 reward functions for this task.

Original: In the dense reward version of the environment, the reward is the negative Euclidean distance of the sphere from the center of the goal.
Revised: In the dense reward version of the environment, the reward is the negative Euclidean distance of the sphere from the center of the goal.

Original: This represents a dense reward signal, which makes optimization easier but also introduces local maximums due to the presence of the obstacles.
Revised: This represents a dense reward signal, which makes optimization easier but also introduces local maximums due to the presence of the obstacles.

Original: In the sparse version the reward is always − 1 , so the optimal policy is to reach the goal as quickly as possible so that the episodes ends.
Revised: In the sparse version the reward is always − 1 , so the optimal policy is to reach the goal as quickly as possible so that the episodes ends.

Original: We devise 4 environment configuarations, with different levels of difficulty.
Revised: We devise 4 environment configuarations, with different levels of difficulty.

Original: The first environment (Figure 5a) has a U-shaped wall in the middle.
Revised: The first environment (Figure 5a) has a U-shaped wall in the middle.

Original: The agent has to overcome it by either running into it and then moving back to outflank it, so it can escape the local maximum, or preferably, it has to go around it without
Revised: The agent has to overcome it by either running into it and then moving back to outflank it, so it can escape the local maximum, or preferably, it has to go around it without

Original: [Figure]
Revised: [Figure]

Original: touching it.
Revised: touching it.

Original: The U-shaped of the wall makes the task more difficult since the agent is unlikely to be able to escape the local maximum once it has hit the wall by simply playing random actions.
Revised: The U-shaped of the wall makes the task more difficult since the agent is unlikely to be able to escape the local maximum once it has hit the wall by simply playing random actions.

Original: The second environment (Figure 5b) has 2 simpler obstacles to overcome compared to first environment, yet the agent now has to learn how to overcome two obstacles which overall makes the second task more difficult than the first one.
Revised: The second environment (Figure 5b) has 2 simpler obstacles to overcome compared to first environment, yet the agent now has to learn how to overcome two obstacles which overall makes the second task more difficult than the first one.

Original: To overcome the obstacle of the third environment (Figure 5c) the agent has to perform a rather complete exploration of the space since the gateway to the second room is quite narrow and it is easy to be stuck in a local maximum at the border due to the reward function based on the Euclidean Distance.
Revised: To overcome the obstacle of the third environment (Figure 5c) the agent has to perform a rather complete exploration of the space since the gateway to the second room is quite narrow and it is easy to be stuck in a local maximum at the border due to the reward function based on the Euclidean Distance.

Original: The forth and last environment shown in Figure 5d, puts together the challenges of the previous two, it has 2 obstacles to overcome and it also requires a rather complete exploration of the space to advance to the goal.
Revised: The forth and last environment shown in Figure 5d, puts together the challenges of the previous two, it has 2 obstacles to overcome and it also requires a rather complete exploration of the space to advance to the goal.

Original: A.2. Tuning procedures
Revised: A.2. Tuning procedures

Original: In this appendix, we present the hyperparameter tuning employed, as well as the final values used in our experiments.
Revised: In this appendix, we present the hyperparameter tuning employed, as well as the final values used in our experiments.

Original: For all the approximators employed, including critics and actors, we use 2 layer MLPs.
Revised: For all the approximators employed, including critics and actors, we use 2 layer MLPs.

Original: For what concerns LQG all algorithms could solve it easily independently from the hyper-parameters chosen.
Revised: For what concerns LQG all algorithms could solve it easily independently from the hyper-parameters chosen.

Original: The same can be said for RiverSwim, except for SAC which could not solve for any set of hyper-parameters contained in the grid search we performed.
Revised: The same can be said for RiverSwim, except for SAC which could not solve for any set of hyper-parameters contained in the grid search we performed.

Original: In point environment we performed a grid search on all environments with dense rewards starting with SAC (3 runs with[Table] different seeds for each node of the grid).
Revised: In point environment we performed a grid search on all environments with dense rewards starting with SAC (3 runs with different seeds for each node of the grid).

Original: We choose the set of hyper-parameters that could solve the most runs for the two most difficult environments in which at least one run could learn a policy that reached the goal.
Revised: We choose the set of hyper-parameters that could solve the most runs for the two most difficult environments in which at least one run could learn a policy that reached the goal.

Original: The best recovered values are reported in Table 1.
Revised: The best recovered values are reported in Table 1.

Original: [Table]
Revised: [Table]

Original: Afterwards, we performed a grid search on OAC and WAC, where we fixed all the hyper-parameters they share with SAC to the best values we found on SAC hyper-parameter tuning and we tuned only on their additional hyper-parameters.
Revised: Afterwards, we performed a grid search on OAC and WAC, where we fixed all the hyper-parameters they share with SAC to the best values we found on SAC hyper-parameter tuning and we tuned only on their additional hyper-parameters.

Original: Once again, we choose the hyper-parameters sets that allow each algorithm to perform best the 2 most difficult environment it could solve.
Revised: Once again, we choose the hyper-parameters sets that allow each algorithm to perform best the 2 most difficult environment it could solve.

Original: The final values are reported in Table 2 and Table 3.
Revised: The final values are reported in Table 2 and Table 3.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B. Additional Experiments B.1. Tuning on δ
Revised: B. Additional Experiments B.1. Tuning on δ

Original: In Section 6 we have shown how by tuning λ and ρ we can control the amount of exploration.
Revised: In Section 6 we have shown how by tuning λ and ρ we can control the amount of exploration.

Original: We now show some experiment that illustrate how the hyper parameter δ can also be use to control exploration, since it defines what percentile of the estimated Gaussian distribution we use as upper bound.
Revised: We now show some experiment that illustrate how the hyper parameter δ can also be use to control exploration, since it defines what percentile of the estimated Gaussian distribution we use as upper bound.

Original: The results are reported in Figure 6.
Revised: The results are reported in Figure 6.

Original: We observe that also δ is directly related with the coverage.
Revised: We observe that also δ is directly related with the coverage.

Original: Indeed by increasing δ , we employ larger upper bounds for the value estimates, and this directly translates to larger coverage of the state-action space.
Revised: Indeed by increasing δ , we employ larger upper bounds for the value estimates, and this directly translates to larger coverage of the state-action space.

Original: B.2. Coverage in OAC
Revised: B.2. Coverage in OAC

Original: We performed a similar study to the one presented in Section 6 on WAC for OAC, to investigate whether we could control how much the algorithm explores based on the values of the hyper-parameters.
Revised: We performed a similar study to the one presented in Section 6 on WAC for OAC, to investigate whether we could control how much the algorithm explores based on the values of the hyper-parameters.

Original: However, what we found is that is hard to predict OAC’s exploration based on its hyper parameters δ and β UB .
Revised: However, what we found is that is hard to predict OAC’s exploration based on its hyper parameters δ and β UB .

Original: In OAC, δ controls how much the exploration policy differs from the target policy.
Revised: In OAC, δ controls how much the exploration policy differs from the target policy.

Original: From Figure 7, we can see that δ can even negatively affect exploration, if we allow the exploration policy to differ too much from the target policy.
Revised: From Figure 7, we can see that δ can even negatively affect exploration, if we allow the exploration policy to differ too much from the target policy.

Original: The dependence on β , which controls the definition of the upper bound (similar to our δ in OAC), suggests that the uncertainty estimate of OAC is not directly related to exploration either.
Revised: The dependence on β , which controls the definition of the upper bound (similar to our δ in OAC), suggests that the uncertainty estimate of OAC is not directly related to exploration either.

Original: We attribute both these results to the heuristic estimation of uncertainty that OAC employees, based only on the disagreement between the two critics.
Revised: We attribute both these results to the heuristic estimation of uncertainty that OAC employees, based only on the disagreement between the two critics.

Original: We argue that this uncertainty estimation is not enough to direct exploration meaningfully.
Revised: We argue that this uncertainty estimation is not enough to direct exploration meaningfully.

Original: B.3. Exploration Heatmaps in Point
Revised: B.3. Exploration Heatmaps in Point

Original: In this section, we show some additional heatmaps which represents the visited states (we have ignored velocities so we could visualize the location of the agent) over 300 epochs of all algorithms we have tested throughout the paper.
Revised: In this section, we show some additional heatmaps which represents the visited states (we have ignored velocities so we could visualize the location of the agent) over 300 epochs of all algorithms we have tested throughout the paper.

Original: Figureshows the heatmaps from runs on Point 3 environment, with dense rewards.
Revised: Figureshows the heatmaps from runs on Point 3 environment, with dense rewards.

Original: In Figure 8a we see that SAC cannot get past the first wall and does not explore the space around the local maximum enough to reach other maxima.
Revised: In Figure 8a we see that SAC cannot get past the first wall and does not explore the space around the local maximum enough to reach other maxima.

Original: In Figure 8b
Revised: In Figure 8b

Original: we
Revised: we

Original: C o v e r a g e [Figure]
Revised: C o v e r a g e (a) Coverage in LQG.

Original: [Figure]
Revised: (b) Coverage in Riverswim.

Original: UB 0.
Revised: UB 0.

Original: see that OAC finds a better maximum but still a local one.
Revised: see that OAC finds a better maximum but still a local one.

Original: WAC does find the same local maximum as OAC, in fact we can see in Figure 8c it visits it many times, yet once the uncertainty estimate is low enough it is able to keep exploring and ultimately reach the goal.
Revised: WAC does find the same local maximum as OAC, in fact we can see in Figure 8c it visits it many times, yet once the uncertainty estimate is low enough it is able to keep exploring and ultimately reach the goal.

Original: We have also reported in Figure 8d the same heatmap created by the target policy which follows the critic of the mean instead of the upper bound.
Revised: We have also reported in Figure 8d the same heatmap created by the target policy which follows the critic of the mean instead of the upper bound.

Original: Figure 9 shows the cumulative visited states in the Point 2 environment with sparse reward.
Revised: Figure 9 shows the cumulative visited states in the Point 2 environment with sparse reward.

Original: We can observe that SAC, having no uncertainty estimate and no informative rewards, mostly explores around the starting states with very simple policies that follow straight lines.
Revised: We can observe that SAC, having no uncertainty estimate and no informative rewards, mostly explores around the starting states with very simple policies that follow straight lines.

Original: OAC is able to reach areas of the maze which are further away from the starting point but it still can’t reach the goal.
Revised: OAC is able to reach areas of the maze which are further away from the starting point but it still can’t reach the goal.

Original: Finally, WAC manages to reach the goal of the maze and explores almost every area of the it.
Revised: Finally, WAC manages to reach the goal of the maze and explores almost every area of the it.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]