Original: G REEDY A CTOR -C RITIC : A N EW C ONDITIONAL C ROSS -E NTROPY M ETHOD FOR P OLICY I MPROVEMENT
Revised: G REEDY A CTOR -C RITIC : A N EW C ONDITIONAL C ROSS -E NTROPY M ETHOD FOR P OLICY I MPROVEMENT

Original: A BSTRACT
Revised: A BSTRACT

Original: Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor).
Revised: Many policy gradient methods are variants of Actor-Critic (AC), where a value function (critic) is learned to facilitate updating the parameterized policy (actor).

Original: The update to the actor involves a log-likelihood update weighted by the actionvalues, with the addition of entropy regularization for soft variants.
Revised: The update to the actor involves a log-likelihood update weighted by the actionvalues, with the addition of entropy regularization for soft variants.

Original: In this work, we explore an alternative update for the actor, based on an extension of the cross entropy method (CEM) to condition on inputs (states).
Revised: In this work, we explore an alternative update for the actor, based on an extension of the cross entropy method (CEM) to condition on inputs (states).

Original: The idea is to start with a broader policy and slowly concentrate around maximally valued actions, using a maximum likelihood update towards actions in the top percentile per state.
Revised: The idea is to start with a broader policy and slowly concentrate around maximally valued actions, using a maximum likelihood update towards actions in the top percentile per state.

Original: The speed of this concentration is controlled by a proposal policy, that concentrates at a slower rate than the actor.
Revised: The speed of this concentration is controlled by a proposal policy, that concentrates at a slower rate than the actor.

Original: We first provide a policy improvement result in an idealized setting, and then prove that our conditional CEM (CCEM) strategy tracks a CEM update per state, even with changing action-values.
Revised: We first provide a policy improvement result in an idealized setting and then prove that our conditional CEM (CCEM) strategy tracks a CEM update per state, even with changing action-values.

Original: We empirically show that our GreedyAC algorithm, that uses CCEM for the actor update, performs better than Soft Actor-Critic and is much less sensitive to entropy-regularization
Revised: We empirically show that our GreedyAC algorithm, that uses CCEM for the actor update, performs better than Soft Actor-Critic and is much less sensitive to entropy-regularization

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many policy optimization strategies update the policy towards the Boltzmann policy.
Revised: Many policy optimization strategies update the policy towards the Boltzmann policy.

Original: This strategy became popularized by Soft Q-Learning (Haarnoja et al., 2017) and Soft Actor-Critic (SAC) (Haarnoja et al., 2018a), but has a long history in reinforcement learning (Kober & Peters, 2008; Neumann, 2011).
Revised: This strategy became popularized by Soft Q-Learning (Haarnoja et al., 2017) and Soft Actor-Critic (SAC) (Haarnoja et al., 2018a), but has a long history in reinforcement learning (Kober & Peters, 2008; Neumann, 2011).

Original: In fact, recent work (Vieillard et al., 2020a; Chan et al., 2021) has highlighted that an even broader variety of policy optimization methods can be seen as optimizing either a forward or reverse
Revised: In fact, recent work (Vieillard et al., 2020a; Chan et al., 2021) has highlighted that an even broader variety of policy optimization methods can be seen as optimizing either a forward or reverse

Original: KL divergence to the Boltzmann policy, as in SAC.
Revised: KL divergence to the Boltzmann policy, as in SAC.

Original: In fact, even the original Actor-Critic (AC) update (Sutton, 1984) can be seen as optimizing a reverse KL divergence, with zero-entropy.
Revised: In fact, even the original Actor-Critic (AC) update (Sutton, 1984) can be seen as optimizing a reverse KL divergence, with zero-entropy.

Original: The use of the Boltzmann policy underlies many methods for good reason: it guarantees policy improvement (Haarnoja et al., 2018a).
Revised: The use of the Boltzmann policy underlies many methods for good reason: it guarantees policy improvement (Haarnoja et al., 2018a).

Original: More specifically, this is the case when learning entropyregularized action-values Q πτ for a policy π with regularization parameter τ > 0 .
Revised: More specifically, this is the case when learning entropyregularized action-values Q πτ for a policy π with regularization parameter τ > 0 .

Original: The Boltzmann policy for a state is proportional to exp( Q πτ ( s, a ) τ − 1 ) .
Revised: The Boltzmann policy for a state is proportional to exp( Q πτ ( s, a ) τ − 1 ) .

Original: The level of emphasis on high-valued actions is controlled by τ : the higher the magnitude of the entropy level (larger τ ), the less the probabilities in the Boltzmann policy are peaked around maximally valued actions.
Revised: The level of emphasis on high-valued actions is controlled by τ : the higher the magnitude of the entropy level (larger τ ), the less the probabilities in the Boltzmann policy are peaked around maximally valued actions.

Original: This choice, however, has several limitations.
Revised: This choice, however, has several limitations.

Original: The policy improvement guarantee is for the entropyregularized MDP, rather than the original MDP.
Revised: The policy improvement guarantee is for the entropyregularized MDP, rather than the original MDP.

Original: Entropy regularization is used to encourage exploration (Ziebart et al., 2008; Mei et al., 2019) and improve the optimization surface (Ahmed et al., 2019;
Revised: Entropy regularization is used to encourage exploration (Ziebart et al., 2008; Mei et al., 2019) and improve the optimization surface (Ahmed et al., 2019;

Original: Shani et al., 2020), resulting in a trade-off between improving the learning process and converging to the optimal policy.
Revised: Shani et al., 2020), resulting in a trade-off between improving the learning process and converging to the optimal policy.

Original: Additionally, SAC and other methods are well-known to be sensitive to the entropy regularization parameter (Pourchot & Sigaud, 2019).
Revised: Additionally, SAC and other methods are well-known to be sensitive to the entropy regularization parameter (Pourchot & Sigaud, 2019).

Original: Prior work has explored optimizing entropy during learning (Haarnoja et al., 2018b), however, this optimization introduces yet another hyperparameter to tune, and this approach may be less performant than a simple grid search (see
Revised: Prior work has explored optimizing entropy during learning (Haarnoja et al., 2018b), however, this optimization introduces yet another hyperparameter to tune, and this approach may be less performant than a simple grid search (see

Original: Appendix D).
Revised: Appendix D).

Original: It is reasonable to investigate alternative policy improvement approaches that could potentially improve our actor-critic algorithms.
Revised: It is reasonable to investigate alternative policy improvement approaches that could potentially improve our actor-critic algorithms.

Original: In this work we propose a new greedification strategy towards this goal.
Revised: In this work we propose a new greedification strategy towards this goal.

Original: The basic idea is to iteratively take the top percentile of actions, ranked according to the learned action-values.
Revised: The basic idea is to iteratively take the top percentile of actions, ranked according to the learned action-values.

Original: The procedure slowly concentrates on the maximal action(s), across states, for the given action-values.
Revised: The procedure slowly concentrates on the maximal action(s), across states, for the given action-values.

Original: The update itself is simple: N ∈ N actions are sampled according to a proposal policy , the actions are sorted based on the magnitude of the action-values, and the policy is updated to increase the probability of the ⌈ ρN ⌉ maximally valued actions for ρ ∈ (0 , 1) .
Revised: The update itself is simple: N ∈ N actions are sampled according to a proposal policy , the actions are sorted based on the magnitude of the action-values, and the policy is updated to increase the probability of the ⌈ ρN ⌉ maximally valued actions for ρ ∈ (0 , 1) .

Original: We call this algorithm for the actor Conditional CEM (CCEM), because it is an extension of the well-known Cross-Entropy Method (CEM) (Rubinstein, 1999) to condition on inputs 1 .
Revised: We call this algorithm for the actor Conditional CEM (CCEM), because it is an extension of the well-known Cross-Entropy Method (CEM) (Rubinstein, 1999) to condition on inputs 1 .

Original: We leverage theory for CEM to validate that our algorithm concentrates on maximally valued actions across states over time.
Revised: We leverage theory for CEM to validate that our algorithm concentrates on maximally valued actions across states over time.

Original: We introduce GreedyAC, a new AC algorithm that uses CCEM for the actor.
Revised: We introduce GreedyAC, a new AC algorithm that uses CCEM for the actor.

Original: GreedyAC has several advantages over using Boltzmann greedification.
Revised: GreedyAC has several advantages over using Boltzmann greedification.

Original: First, we show that our new greedification operator ensures a policy improvement for the original MDP, rather than a different entropy-regularized MDP.
Revised: First, we show that our new greedification operator ensures a policy improvement for the original MDP, rather than a different entropy-regularized MDP.

Original: Second, we can still leverage entropy to prevent policy collapse, but only incorporate it into the proposal policy.
Revised: Second, we can still leverage entropy to prevent policy collapse, but only incorporate it into the proposal policy.

Original: This ensures the agent considers potentially optimal actions for longer, but does not skew the actor.
Revised: This ensures the agent considers potentially optimal actions for longer, but does not skew the actor.

Original: In fact, it is possible to decouple the role of entropy for exploration and policy collapse within GreedyAC: the actor could have a small amount of entropy to encourage exploration, and the proposal policy a higher level of entropy to avoid policy collapse.
Revised: In fact, it is possible to decouple the role of entropy for exploration and policy collapse within GreedyAC: the actor could have a small amount of entropy to encourage exploration, and the proposal policy a higher level of entropy to avoid policy collapse.

Original: Potentially because of this decoupling, we find that GreedyAC is much less sensitive to the choice of entropy regularizer, as compared to SAC.
Revised: Potentially because of this decoupling, we find that GreedyAC is much less sensitive to the choice of entropy regularizer, as compared to SAC.

Original: This design of the algorithm may help it avoid getting stuck in a locally optimal action, and empirical evidence for CEM suggests it can be quite effective for this purpose (Rubinstein & Kroese, 2004).
Revised: This design of the algorithm may help it avoid getting stuck in a locally optimal action, and empirical evidence for CEM suggests it can be quite effective for this purpose (Rubinstein & Kroese, 2004).

Original: In addition to our theoretical support for CCEM, we provide an empirical investigation comparing GreedyAC, SAC, and a vanilla AC, highlighting that
Revised: In addition to our theoretical support for CCEM, we provide an empirical investigation comparing GreedyAC, SAC, and a vanilla AC, highlighting that

Original: GreedyAC performs consistently well, even in problems like the Mujoco environment Swimmer and pixel-based control where SAC performs poorly.
Revised: GreedyAC performs consistently well, even in problems like the Mujoco environment Swimmer and pixel-based control where SAC performs poorly.

Original:  B ACKGROUND AND P ROBLEM F ORMULATION
Revised:  B ACKGROUND AND P ROBLEM F ORMULATION

Original: The interaction between the agent and environment is formalized by a Markov decision process ( S , A , P , R , γ ) , where S is the state space, A is the action space, P : S ×
Revised: The interaction between the agent and environment is formalized by a Markov decision process ( S , A , P , R , γ ) , where S is the state space, A is the action space, P : S ×

Original: A × S → [0 , ∞ ) is the one-step state transition dynamics, R : S
Revised: A × S → [0 , ∞ ) is the one-step state transition dynamics, R : S

Original: ×
Revised: ×

Original: A × S → R is the reward function, and γ ∈ [0 , 1] is the discount rate.
Revised: A × S → R is the reward function, and γ ∈ [0 , 1] is the discount rate.

Original: We assume an episodic problem setting, where the start state S 0 ∼ d 0 for start state distribution d 0 : S → [0 , ∞ ) and the length of the episode T is random, depending on when the agent reaches termination.
Revised: We assume an episodic problem setting, where the start state S 0 ∼ d 0 for start state distribution d 0 : S → [0 , ∞ ) and the length of the episode T is random, depending on when the agent reaches termination.

Original: At each discrete timestep t = 1 , 2 , . . .
Revised: At each discrete timestep t = 1 , 2 , . . .

Original: , T , the agent finds itself in some state
Revised: , T , the agent finds itself in some state

Original: S t and selects an action A t drawn from its stochastic policy π : S ×
Revised: S t and selects an action A t drawn from its stochastic policy π : S ×

Original: A → [0 , ∞ ) .
Revised: A → [0 , ∞ ) .

Original: The agent then transitions to state S t +

Revised: The agent then transitions to state S t +

Original: according to P and observes a scalar reward R t +1 .
Revised: according to P and observes a scalar reward R t +1 .

Original: =
Revised: =

Original: R ( S t , A t , S t +1 ) .
Revised: R ( S t , A t , S t +1 ) .

Original: For a parameterized policy π w with parameters w , the agent attempts to maximize the objective
Revised: For a parameterized policy π w with parameters w , the agent attempts to maximize the objective

Original: J ( w ) =
Revised: J ( w ) =

Original: E π w [ (cid:80) Tt =

Revised: E π w [ (cid:80) Tt =

Original: γ t R t +1 ] , where the expectation is according to start state distribution d 0 , transition dynamics P , and policy π w .
Revised: γ t R t +1 ] , where the expectation is according to start state distribution d 0 , transition dynamics P , and policy π w .

Original: Policy gradient methods, like REINFORCE (Williams, 1992), attempt to obtain (unbiased) estimates of the gradient of this objective to directly update the policy.
Revised: Policy gradient methods, like REINFORCE (Williams, 1992), attempt to obtain (unbiased) estimates of the gradient of this objective to directly update the policy.

Original: The difficulty is that the policy gradient is expensive to sample, because it requires sampling return trajectories from states sampled from the visitation distribution under π w , as per the policy gradient theorem (Sutton et al., 1999).
Revised: The difficulty is that the policy gradient is expensive to sample, because it requires sampling return trajectories from states sampled from the visitation distribution under π w , as per the policy gradient theorem (Sutton et al., 1999).

Original: Theory papers analyze such an idealized algorithm (Kakade & Langford, 2002; Agarwal et al., 2021), but in practice this strategy is rarely used.
Revised: Theory papers analyze such an idealized algorithm (Kakade & Langford, 2002; Agarwal et al., 2021), but in practice this strategy is rarely used.

Original: Instead, it is much more common to (a) ignore bias in the state distribution (Thomas, 2014; Imani et al., 2018; Nota & Thomas, 2020) and (b) use biased estimates of the return, in the form of a value function critic.
Revised: Instead, it is much more common to (a) ignore bias in the state distribution (Thomas, 2014; Imani et al., 2018; Nota & Thomas, 2020) and (b) use biased estimates of the return, in the form of a value function critic.

Original: The action-value function Q π ( s, a ) .
Revised: The action-value function Q π ( s, a ) .

Original: =
Revised: =

Original: E π
Revised: E π

Original: [ (cid:80) T − t k =1 γ t R t + k | S t = s, A t = a ] is the expected return from a given state and action, when following policy π .
Revised: [ (cid:80) T − t k =1 γ t R t + k | S t = s, A t = a ] is the expected return from a given state and action, when following policy π .

Original: Many PG methods—specifically variants of Actor-Critic—estimate these action-values with parameterized Q θ ( s, a ) , to use the update Q θ ( s, a ) ∇ ln π w ( a | s ) or one with a baseline [ Q θ ( s, a ) − V ( s )] ∇ ln π w ( a | s ) where the value function V ( s ) is also typically learned.
Revised: Many PG methods—specifically variants of Actor-Critic—estimate these action-values with parameterized Q θ ( s, a ) , to use the update Q θ ( s, a ) ∇ ln π w ( a | s ) or one with a baseline [ Q θ ( s, a ) − V ( s )] ∇ ln π w ( a | s ) where the value function V ( s ) is also typically learned.

Original: The state s is sampled from a replay buffer, and a ∼ π w ( ·| s ) , for the update.
Revised: The state s is sampled from a replay buffer, and a ∼ π w ( ·| s ) , for the update.

Original: There has been a flurry of work, and success, pursuing this path, including methods such as OffPAC (Degris et al., 2012b), SAC (Haarnoja et al., 2018a), SQL (Haarnoja et al., 2017), TRPO (Schulman et al., 2015) and many other variants of related ideas (Peters et al., 2010; Silver et al., 2014; Schulman et al., 2016; Lillicrap et al., 2016; Wang et al., 2017; Gu et al., 2017; Schulman et al., 2017;
Revised: There has been a flurry of work, and success, pursuing this path, including methods such as OffPAC (Degris et al., 2012b), SAC (Haarnoja et al., 2018a), SQL (Haarnoja et al., 2017), TRPO (Schulman et al., 2015) and many other variants of related ideas (Peters et al., 2010; Silver et al., 2014; Schulman et al., 2016; Lillicrap et al., 2016; Wang et al., 2017; Gu et al., 2017; Schulman et al., 2017;

Original: Abdolmaleki et al., 2018; Mei et al., 2019; Vieillard et al., 2020b).
Revised: Abdolmaleki et al., 2018; Mei et al., 2019; Vieillard et al., 2020b).

Original: Following close behind are unification results that make sense of this flurry of work (Tomar et al., 2020; Vieillard et al., 2020a;
Revised: Following close behind are unification results that make sense of this flurry of work (Tomar et al., 2020; Vieillard et al., 2020a;

Original: Chan et al., 2021; Lazi´c et al., 2021).
Revised: Chan et al., 2021; Lazi´c et al., 2021).

Original: They highlight that many methods include a mirror descent component—to minimize KL to the most recent policy—and an entropy-regularization component (Vieillard et al., 2020a).
Revised: They highlight that many methods include a mirror descent component—to minimize KL to the most recent policy—and an entropy-regularization component (Vieillard et al., 2020a).

Original: In particular, these methods are better thought of as (approximate) policy iteration approaches that update towards the Boltzmann policy, in some cases using a mirror descent update.
Revised: In particular, these methods are better thought of as (approximate) policy iteration approaches that update towards the Boltzmann policy, in some cases using a mirror descent update.

Original: The Boltzmann policy B τ Q ( s, a ) for a given Q is
Revised: The Boltzmann policy B τ Q ( s, a ) for a given Q is

Original: [Equation]
Revised: [Equation]

Original: for entropy parameter τ .
Revised: for entropy parameter τ .

Original: As τ → 0 , this policy puts all weight on greedy actions.
Revised: As τ → 0 , this policy puts all weight on greedy actions.

Original: As τ → ∞ , all actions are weighted uniformly.
Revised: As τ → ∞ , all actions are weighted uniformly.

Original: This policy could be directly used as the new greedy policy.
Revised: This policy could be directly used as the new greedy policy.

Original: However, because it is expensive to sample from B τ Q ( s, a ) , typically a parameterized policy π w is learned to approximate B τ Q ( s, a ) , by minimizing a KL divergence.
Revised: However, because it is expensive to sample from B τ Q ( s, a ) , typically a parameterized policy π w is learned to approximate B τ Q ( s, a ) , by minimizing a KL divergence.

Original: As the entropy goes to zero, we get an unregularized update that corresponds to the vanilla AC update (Chan et al., 2021).
Revised: As the entropy goes to zero, we get an unregularized update that corresponds to the vanilla AC update (Chan et al., 2021).

Original:  C ONDITIONAL CEM
Revised:  C ONDITIONAL CEM

Original: Though using the Boltzmann policy has been successful, it does have some limitations.
Revised: Though using the Boltzmann policy has been successful, it does have some limitations.

Original: The primary limitation is that it is sensitive to the choice of entropy (Pourchot & Sigaud, 2019; Chan et al., 2021).
Revised: The primary limitation is that it is sensitive to the choice of entropy (Pourchot & Sigaud, 2019; Chan et al., 2021).

Original: A natural question is what other strategies we can use for this greedification step in these approximate policy iteration algorithms, and how they compare to this common approach.
Revised: A natural question is what other strategies we can use for this greedification step in these approximate policy iteration algorithms, and how they compare to this common approach.

Original: We propose and motivate a new approach in this section, and then focus the paper on providing insight into its benefits and limitations, in contrast to using the Boltzmann policy.
Revised: We propose and motivate a new approach in this section, and then focus the paper on providing insight into its benefits and limitations, in contrast to using the Boltzmann policy.

Original: Let us motivate our approach, by describing the well-known global optimization algorithm called the
Revised: Let us motivate our approach, by describing the well-known global optimization algorithm called the

Original: Cross Entropy Method (CEM) (Rubinstein, 1999).
Revised: Cross Entropy Method (CEM) (Rubinstein, 1999).

Original: Global optimization strategies are designed to find the global optimum of a general function f ( β ) for some parameters β .
Revised: Global optimization strategies are designed to find the global optimum of a general function f ( β ) for some parameters β .

Original: For example, for parameters β of a neural network, f may be the loss function on a sample of data.
Revised: For example, for parameters β of a neural network, f may be the loss function on a sample of data.

Original: An advantage of these methods is that they do not rely on gradient-based strategies, which are prone to getting stuck in local optima.
Revised: An advantage of these methods is that they do not rely on gradient-based strategies, which are prone to getting stuck in local optima.

Original: Instead, they use randomized search strategies, that have optimality guarantees in some settings (Hu et al., 2012) and have been shown to be effective in practice (Peters & Schaal, 2007;
Revised: Instead, they use randomized search strategies, that have optimality guarantees in some settings (Hu et al., 2012) and have been shown to be effective in practice (Peters & Schaal, 2007;

Original: Hansen et al., 2003; Szita & L ¨ orincz, 2006; Salimans et al., 2017).
Revised: Hansen et al., 2003; Szita & L ¨ orincz, 2006; Salimans et al., 2017).

Original: CEM maintains a distribution p ( β ) over parameters β , iteratively narrowing the range of plausible solutions.
Revised: CEM maintains a distribution p ( β ) over parameters β , iteratively narrowing the range of plausible solutions.

Original: The algorithm maintain a current threshold f t , that slowly increases over time as it narrows on the maximal β .
Revised: The algorithm maintain a current threshold f t , that slowly increases over time as it narrows on the maximal β .

Original: On iteration t , N parameter vectors β 1 , . . .
Revised: On iteration t , N parameter vectors β 1 , . . .

Original: , β N are sample from p t ; the algorithm only keeps β ∗ 1 , . .
Revised: , β N are sample from p t ; the algorithm only keeps β ∗ 1 , . .

Original: , β ∗ h where f ( β ∗ i ) ≥ f t and discards the rest.
Revised: , β ∗ h where f ( β ∗ i ) ≥ f t and discards the rest.

Original: The KL divergence is reduced between p t and this empirical distribution ˆ I = { β ∗ 1 , . .
Revised: The KL divergence is reduced between p t and this empirical distribution ˆ I = { β ∗ 1 , . .

Original: , β ∗ h } , for h ≤ N .
Revised: , β ∗ h } , for h ≤ N .

Original: This step corresponds to increasing the likelihood of the β in the set
Revised: This step corresponds to increasing the likelihood of the β in the set

Original: ˆ I .
Revised: ˆ I .

Original: Iteratively, the distribution over parameters p t narrows around β with higher values under f .
Revised: Iteratively, the distribution over parameters p t narrows around β with higher values under f .

Original: To make it more likely to find the global optimum, the initial distribution p 0 is a wide distribution, such as a Gaussian distribution with mean zero µ 0 = 0 and a diagonal covariance Σ 0 of large magnitude.
Revised: To make it more likely to find the global optimum, the initial distribution p 0 is a wide distribution, such as a Gaussian distribution with mean zero µ 0 = 0 and a diagonal covariance Σ 0 of large magnitude.

Original: Distribution( N, ρ )
Revised: Distribution( N, ρ )

Original: Evaluate and sort in descending order:
Revised: Evaluate and sort in descending order:

Original: [Equation]
Revised: Q θ ( S t , a i 1 ) ≥ . . .

Original:  ) return ˆ I ( S t )
Revised: ≥ Q θ ( S t , a i N ) return ˆ I ( S t )

Original: = { a i 1 , . . .
Revised: = { a i 1 , . . .

Original: , a i h } (where h = ⌈ ρN ⌉ )
Revised: , a i h } (where h = ⌈ ρN ⌉ )

Original: CEM attempts to find the single-best set of optimal parameters for a single optimization problem.
Revised: CEM attempts to find the single-best set of optimal parameters for a single optimization problem.

Original: The straightforward use in reinforcement learning is to learn the single-best set of policy parameters w (Szita & L¨orincz, 2006; Mannor et al., 2003).
Revised: The straightforward use in reinforcement learning is to learn the single-best set of policy parameters w (Szita & L¨orincz, 2006; Mannor et al., 2003).

Original: Our goal, however, is to (repeatedly) find maximally valued actions a ∗ conditioned on each state for Q ( s, · ) .
Revised: Our goal, however, is to (repeatedly) find maximally valued actions a ∗ conditioned on each state for Q ( s, · ) .

Original: The global optimization strategy could be run on each step to find the exact best action for each current state, as in QT-Opt
Revised: The global optimization strategy could be run on each step to find the exact best action for each current state, as in QT-Opt

Original: (Kalashnikov et al., 2018)
Revised: (Kalashnikov et al., 2018)

Original: and follows-ups (Simmons-Edler et al., 2019; Shao et al., 2022), but this is expensive and throws away prior information about the function surface obtained when previous optimizations were executed.
Revised: and follows-ups (Simmons-Edler et al., 2019; Shao et al., 2022), but this is expensive and throws away prior information about the function surface obtained when previous optimizations were executed.

Original: We extend CEM to be (a) conditioned on state and (b) learned iteratively over time.
Revised: We extend CEM to be (a) conditioned on state and (b) learned iteratively over time.

Original: The key modification when extending CEM to Conditional CEM (CCEM), to handle these two key differences, is to introduce another proposal policy that concentrates more slowly.
Revised: The key modification when extending CEM to Conditional CEM (CCEM), to handle these two key differences, is to introduce another proposal policy that concentrates more slowly.

Original: This proposal policy is entropyregularized to ensure that we keep a broader set of potential actions when sampling, in case changing action-values are very different since the previous update to that state.
Revised: This proposal policy is entropyregularized to ensure that we keep a broader set of potential actions when sampling, in case changing action-values are very different since the previous update to that state.

Original: The main policy (the actor) does not use entropy regularization, allowing it to more quickly start acting according to currently greedy actions, without collapsing.
Revised: The main policy (the actor) does not use entropy regularization, allowing it to more quickly start acting according to currently greedy actions, without collapsing.

Original: We visualize this in Figure 1.
Revised: We visualize this in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: We use uniform policies, for interpretability.
Revised: We use uniform policies, for interpretability.

Original: In the rightmost figure, we show an actual progression of CCEM with Gaussian policies, when executed on the action-values depicted in the leftmost figure.
Revised: In the rightmost figure, we show an actual progression of CCEM with Gaussian policies, when executed on the action-values depicted in the leftmost figure.

Original: The Actor policy (in black) concentrates more quickly than the Proposal policy (inred).
Revised: The Actor policy (in black) concentrates more quickly than the Proposal policy (in

Original: The CCEM algorithm is presented in Algorithm 2.
Revised: The CCEM algorithm is presented in Algorithm 2.

Original: On each step, the proposal policy, ˜ π w ′ t ( ·| S t ) , is sampled to provide a set of actions a 1 , . .
Revised: On each step, the proposal policy, ˜ π w ′ t ( ·| S t ) , is sampled to provide a set of actions a 1 , . .

Original: . , a N from which we construct the empirical distribution ˆ
Revised: . , a N from which we construct the empirical distribution ˆ

Original: I ( S t ) =
Revised: I ( S t ) =

Original: { a ∗ 1 , . . .
Revised: { a ∗ 1 , . . .

Original: , a ∗ h } of maximally valued actions.
Revised: , a ∗ h } of maximally valued actions.

Original: The actor parameters w t are updated using a gradient ascent step on the log-likelihood of the actions ˆ I ( S t ) .
Revised: The actor parameters w t are updated using a gradient ascent step on the log-likelihood of the actions ˆ I ( S t ) .

Original: The proposal parameters w ′ t are updated using a similar update, but with an entropy regularizer.
Revised: The proposal parameters w ′ t are updated using a similar update, but with an entropy regularizer.

Original: To obtain ˆ I ( S t ) , we select a ∗ i ⊂ { a 1 , . .
Revised: To obtain ˆ I ( S t ) , we select a ∗ i ⊂ { a 1 , . .

Original: . , a N } where Q ( S t , a ∗ i ) are in the top (1 − ρ ) quantile values.
Revised: . , a N } where Q ( S t , a ∗ i ) are in the top (1 − ρ ) quantile values.

Original: For example, for ρ = 0 . 2 , approximately the top 20% of actions are chosen, with h = ⌈ ρN ⌉ .
Revised: For example, for ρ = 0 . 2 , approximately the top 20% of actions are chosen, with h = ⌈ ρN ⌉ .

Original: Implicitly, f t is Q θ ( S t , a ∗ h ) for a ∗ h the action with the lowest value in this top percentile.
Revised: Implicitly, f t is Q θ ( S t , a ∗ h ) for a ∗ h the action with the lowest value in this top percentile.

Original: This procedure is summarized in Algorithm 1.
Revised: This procedure is summarized in Algorithm 1.

Original: Greedy Actor-Critic, in Algorithm 3, puts this all together.
Revised: Greedy Actor-Critic, in Algorithm 3, puts this all together.

Original: We use experience replay, and the CCEM algorithm on a mini-batch.
Revised: We use experience replay, and the CCEM algorithm on a mini-batch.

Original: The updates involve obtaining the sets
Revised: The updates involve obtaining the sets

Original: ˆ I ( S )
Revised: ˆ I ( S )

Original: for every S in the mini-batch
Revised: for every S in the mini-batch

Original: B and updating with the gradient 1 | B | (cid:80) S ∈ B (cid:80) a ∈ ˆ
Revised: B and updating with the gradient 1 | B | (cid:80) S ∈ B (cid:80) a ∈ ˆ

Original: I ( S ) ∇ w ln π w ( a | S ) .
Revised: I ( S ) ∇ w ln π w ( a | S ) .

Original: The Sarsa update to the critic involves (1) sampling an on-policy action from the actor A ′ ∼ π w ( ·| S ′ ) for each tuple in the minibatch and (2) using the update 1 | B | (cid:80) ( S,A,S ′ ,R,A ′ ) ∈ B ( R + γQ θ ( S ′ , A ′ )
Revised: The Sarsa update to the critic involves (1) sampling an on-policy action from the actor A ′ ∼ π w ( ·| S ′ ) for each tuple in the minibatch and (2) using the update 1 | B | (cid:80) ( S,A,S ′ ,R,A ′ ) ∈ B ( R + γQ θ ( S ′ , A ′ )

Original: − Q θ ( S, A )) ∇ θ Q θ ( S, A ) .
Revised: − Q θ ( S, A )) ∇ θ Q θ ( S, A ) .

Original: Other critic updates are possible; we discuss alternatives and connections to related algorithms in
Revised: Other critic updates are possible; we discuss alternatives and connections to related algorithms in

Original: Appendix A.
Revised: Appendix A.

Original: Algorithm
Revised: Algorithm

Original: Input: S t and Q θ , N ∈ N , ρ ∈ (0 , 1) if actions discrete and | A | ≤ 1 /ρ then 
Revised: Input: S t and Q θ , N ∈ N , ρ ∈ (0 , 1) if actions discrete and | A | ≤ 1 /ρ then ˆ [Equation] max

Original: [Equation]
Revised: [Equation]

Original: else
Revised: else

Original: Sample N actions a i ∼ ˜ π w ′ ( ·| S t )
Revised: Sample N actions a i ∼ ˜ π w ′ ( ·| S t )

Original: Obtain ˆ I ( S t ) using Algorithmend if w 
Revised: Obtain ˆ I ( S t ) using Algorithmend if w ←

Original: [Equation]
Revised: [Equation]

Original:  ′ ( ·| S t ))]
Revised: w [Equation] + τ ∇ w ′ H (˜ π w ′ ( ·| S t ))]

Original: Initialize parameters θ, w , w ′ , replay buffer B
Revised: Initialize parameters θ, w , w ′ , replay buffer B

Original: Obtain initial state S while agent interacting with the environment do
Revised: Obtain initial state S while agent interacting with the environment do

Original: Take action A ∼ π w ( ·| S ) , observe R , S ′
Revised: Take action A ∼ π w ( ·| S ) , observe R , S ′

Original: Add ( S, A, S ′ , R ) to the buffer B
Revised: Add ( S, A, S ′ , R ) to the buffer B

Original: Grab a random mini-batch B from buffer B
Revised: Grab a random mini-batch B from buffer B

Original: Update θ using Sarsa for policy π w on B
Revised: Update θ using Sarsa for policy π w on B

Original: Update w , w ′ using Algorithm 2 on B .
Revised: Update w , w ′ using Algorithm 2 on B .

Original: end while
Revised: end while

Original: CCEM for Discrete Actions.
Revised: CCEM for Discrete Actions.

Original: Although we can use the same algorithm for discrete actions, we can make it simpler when we have a small number of discrete actions.
Revised: Although we can use the same algorithm for discrete actions, we can make it simpler when we have a small number of discrete actions.

Original: Our algorithm is designed around the fact that it is difficult to solve for the maximal action for Q θ ( S t , a ) for continuous actions; we slowly identify this maximal action across states.
Revised: Our algorithm is designed around the fact that it is difficult to solve for the maximal action for Q θ ( S t , a ) for continuous actions; we slowly identify this maximal action across states.

Original: For a small set of discrete actions, it is easy to get this maximizing action.
Revised: For a small set of discrete actions, it is easy to get this maximizing action.

Original: If | A | < 1 /ρ , then the top percentile consists of the one top action (or the top actions if there are ties); we can directly set ˆ
Revised: If | A | < 1 /ρ , then the top percentile consists of the one top action (or the top actions if there are ties); we can directly set ˆ

Original: I
Revised: I

Original: S t ) =
Revised: S t ) =

Original: arg max a ∈
Revised: arg max a ∈

Original: A Q θ ( S t , a ) and do not need to maintain a proposal policy.
Revised: A Q θ ( S t , a ) and do not need to maintain a proposal policy.

Original: For this reason, we focus our theory on the continuous-action setting, which is the main motivation for using CEM for the actor update.
Revised: For this reason, we focus our theory on the continuous-action setting, which is the main motivation for using CEM for the actor update.

Original:  T HEORETICAL G UARANTEES
Revised:  T HEORETICAL G UARANTEES

Original: In this section, we motivate that the target policy underlying CCEM guarantees policy improvement, and characterize the ODE underlying CCEM.
Revised: In this section, we motivate that the target policy underlying CCEM guarantees policy improvement, and characterize the ODE underlying CCEM.

Original: We show it tracks a CEM update in expectation across states and slowly concentrates around maximally valued actions even while the action-values are changing.
Revised: We show it tracks a CEM update in expectation across states and slowly concentrates around maximally valued actions even while the action-values are changing.

Original: 4. P OLICY I MPROVEMENT UNDER AN I DEALIZED S ETTING
Revised: 4. P OLICY I MPROVEMENT UNDER AN I DEALIZED S ETTING

Original: We first consider the setting where we have access to Q π , as is typically done for characterizing the policy improvement properties of an operator (Haarnoja et al., 2018a; Ghosh et al., 2020; Chan et al., 2021) as well as for the original policy improvement theorem (Sutton & Barto, 2018).
Revised: We first consider the setting where we have access to Q π , as is typically done for characterizing the policy improvement properties of an operator (Haarnoja et al., 2018a; Ghosh et al., 2020; Chan et al., 2021) as well as for the original policy improvement theorem (Sutton & Barto, 2018).

Original: Our update moves our policy towards a percentile-greedy policy that redistributes probability solely to the (1 − ρ ) -quantile according to magnitudes under Q ( s, a ) .
Revised: Our update moves our policy towards a percentile-greedy policy that redistributes probability solely to the (1 − ρ ) -quantile according to magnitudes under Q ( s, a ) .

Original: More formally, let f ρQ ( π ; s ) be the threshold such that (cid:82) { a ∈ A | Q ( s,a ) ≥ f ρQ ( π ; s ) } π ( a | s )
Revised: More formally, let f ρQ ( π ; s ) be the threshold such that (cid:82) { a ∈ A | Q ( s,a ) ≥ f ρQ ( π ; s ) } π ( a | s )

Original: da = ρ , namely that gives the set of actions in the top 1 − ρ quantile, according to magnitudes under Q ( s, · ) .
Revised: da = ρ , namely that gives the set of actions in the top 1 − ρ quantile, according to magnitudes under Q ( s, · ) .

Original: Then we can define the percentile-greedy policy as
Revised: Then we can define the percentile-greedy policy as

Original: [Equation]

Original: else

Original: [Equation]
Revised: [Equation]

Original: 2) where diving by ρ renormalizes the distribution.
Revised: where diving by ρ renormalizes the distribution.

Original: Computing this policy would be onerous; instead, we only sample the KL divergence to this policy, using a sample percentile.
Revised: Computing this policy would be onerous; instead, we only sample the KL divergence to this policy, using a sample percentile.

Original: Nonetheless, this percentile-greedy policy represents the target policy that the actor updates towards (in the limit of samples N for the percentile).
Revised: Nonetheless, this percentile-greedy policy represents the target policy that the actor updates towards (in the limit of samples N for the percentile).

Original: Intuitively, this target policy should give policy improvement, as it redistributes weight for low valued actions proportionally to high-valued actions.
Revised: Intuitively, this target policy should give policy improvement, as it redistributes weight for low valued actions proportionally to high-valued actions.

Original: We formalize this in the following theorem.
Revised: We formalize this in the following theorem.

Original: We write π ρ ( a | s ) instead of π ρ ( a | s, Q π , π ) , when it is clear from context.
Revised: We write π ρ ( a | s ) instead of π ρ ( a | s, Q π , π ) , when it is clear from context.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For a given policy π , action-value Q π and ρ > 0 , the percentile-greedy policy π ρ in π and Q π is guaranteed to be at least as good as π in all states:
Revised: For a given policy π , action-value Q π and ρ > 0 , the percentile-greedy policy π ρ in π and Q π is guaranteed to be at least as good as π in all states:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is a straightforward modification of the policy improvement theorem.
Revised: The proof is a straightforward modification of the policy improvement theorem.

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: by the definition of percentiles, for any state s .
Revised: by the definition of percentiles, for any state s .

Original: Rewriting (cid:82) A π ( a | s ) Q
Revised: Rewriting (cid:82) A π ( a | s ) Q

Original: π
Revised: π

Original: ( s, a ) da =
Revised: ( s, a ) da =

Original: E π
Revised: E π

Original: [ Q π ( s, A )] ,
Revised: [ Q π ( s, A )] ,

Original: V π ( s ) =
Revised: V π ( s ) =

Original: E π
Revised: E π

Original: [ Q π ( s, A )] ≤ E π ρ [ Q π ( s, A )] =
Revised: [ Q π ( s, A )] ≤ E π ρ [ Q π ( s, A )] =

Original: E π ρ [ R t +1 + γ E π
Revised: E π ρ [ R t +1 + γ E π

Original: [ Q π ( S t +1 , A t +1 ) | S t
Revised: [ Q π ( S t +1 , A t +1 ) | S t

Original: = s ] ≤ E π ρ [ R t +1 + γ E π ρ [ Q π ( S t+1 , A t +1 )] | S
Revised: = s ] ≤ E π ρ [ R t +1 + γ E π ρ [ Q π ( S t

Original: [Equation]
Revised: [Equation]

Original: This result is a sanity check to ensure the target policy is sensible in our update.
Revised: This result is a sanity check to ensure the target policy is sensible in our update.

Original: Note that the
Revised: Note that the

Original: Boltzmann policy only guarantees improvement under the entropy-regularized action-values.
Revised: Boltzmann policy only guarantees improvement under the entropy-regularized action-values.

Original: 4. CCEM T RACKS THE G REEDY A CTION
Revised: 4. CCEM T RACKS THE G REEDY A CTION

Original: Beyond the idealized setting, we would like to understand the properties of the stochastic algorithm.
Revised: Beyond the idealized setting, we would like to understand the properties of the stochastic algorithm.

Original: CCEM is not a gradient descent approach, so we need to reason about its dynamics—namely the underlying ODE.
Revised: CCEM is not a gradient descent approach, so we need to reason about its dynamics—namely the underlying ODE.

Original: We expect CCEM to behave like CEM per state, but with some qualifiers.
Revised: We expect CCEM to behave like CEM per state, but with some qualifiers.

Original: First,
Revised: First,

Original: CCEM uses a parameterized policy conditioned on state, meaning that there is aliasing between the action distributions per state.
Revised: CCEM uses a parameterized policy conditioned on state, meaning that there is aliasing between the action distributions per state.

Original: CEM, on the other hand, does not account for such aliasing.
Revised: CEM, on the other hand, does not account for such aliasing.

Original: We identify conditions on the parameterized policy and use an ODE that takes expectations over states.
Revised: We identify conditions on the parameterized policy and use an ODE that takes expectations over states.

Original: Second, the function we are attempting to maximize is also changing with time, because the actionvalues are updating.
Revised: Second, the function we are attempting to maximize is also changing with time, because the actionvalues are updating.

Original: We address this issue using a two-timescale stochastic approximation approach, where the action-values Q θ change more slowly than the policy, allowing the policy to track the maximally valued actions.
Revised: We address this issue using a two-timescale stochastic approximation approach, where the action-values Q θ change more slowly than the policy, allowing the policy to track the maximally valued actions.

Original: The policy itself has two timescales, to account for its own parameters changing at different timescales.
Revised: The policy itself has two timescales, to account for its own parameters changing at different timescales.

Original: Actions for the maximum likelihood step are selected according to older (slower) parameters w ′ , so that it is as if the primary (faster) parameters w are updated using samples from a fixed distribution.
Revised: Actions for the maximum likelihood step are selected according to older (slower) parameters w ′ , so that it is as if the primary (faster) parameters w are updated using samples from a fixed distribution.

Original: These two policies correspond to our proposal policy (slow) and actor (fast).
Revised: These two policies correspond to our proposal policy (slow) and actor (fast).

Original: We show that the ODE for the CCEM parameters w t is based on the gradient
Revised: We show that the ODE for the CCEM parameters w t is based on the gradient

Original: [Equation]
Revised: [Equation]

Original: where θ and w ′ are changing at slower timescales, and so effectively fixed from the perspective of the faster changing w t .
Revised: where θ and w ′ are changing at slower timescales, and so effectively fixed from the perspective of the faster changing w t .

Original: The term per-state is exactly the update underlying CEM, and so we can think of this ODE as one for an expected CEM Optimizer, across states for parameterized policies.
Revised: The term per-state is exactly the update underlying CEM, and so we can think of this ODE as one for an expected CEM Optimizer, across states for parameterized policies.

Original: We say that CCEM tracks this expected CEM Optimizer, because θ and w ′ are changing with time.
Revised: We say that CCEM tracks this expected CEM Optimizer, because θ and w ′ are changing with time.

Original: We provide an informal theorem statement here for Theorem B.1, with a proof-sketch.
Revised: We provide an informal theorem statement here for Theorem B.1, with a proof-sketch.

Original: The main result, including all conditions, is given in Appendix B.
Revised: The main result, including all conditions, is given in Appendix B.

Original: We discuss some of the (limitations of the) conditions  the proof sketch.
Revised: We discuss some of the (limitations of the) conditions after the proof sketch.

Original: Informal Result: Let θ t be the action-value parameters with stepsize α q,t , and w t be the policy parameters with stepsize α a,t , with w ′ t a more slowly changing set of policy parameters set to w ′ t
Revised: Informal Result: Let θ t be the action-value parameters with stepsize α q,t , and w t be the policy parameters with stepsize α a,t , with w ′ t a more slowly changing set of policy parameters set to w ′ t

Original: = (1 − α ′ a,t ) w ′ t + α ′ a,t w t for stepsize α ′ a,t ∈ (0 , 1] .
Revised: = (1 − α ′ a,t ) w ′ t + α ′ a,t w t for stepsize α ′ a,t ∈ (0 , 1] .

Original: Assume: (1) States S t are sampled from a fixed marginal distribution.
Revised: Assume: (1) States S t are sampled from a fixed marginal distribution.

Original: (2) ∇ w ln π w ( ·| s ) is locally Lipschitz w.r.t.
Revised: (2) ∇ w ln π w ( ·| s ) is locally Lipschitz w.r.t.

Original: w , ∀ s ∈ S .
Revised: w , ∀ s ∈ S .

Original: (3) Parameters w t and θ t remain bounded almost surely.
Revised: (3) Parameters w t and θ t remain bounded almost surely.

Original: (4) Stepsizes are chosen for three different timescales: w t evolves faster than w ′ t and w ′ t evolves faster than θ t .
Revised: (4) Stepsizes are chosen for three different timescales: w t evolves faster than w ′ t and w ′ t evolves faster than θ t .

Original: Under these four conditions, the CCEM Actor tracks the expected CEM Optimizer.
Revised: Under these four conditions, the CCEM Actor tracks the expected CEM Optimizer.

Original: Proof Sketch:
Revised: Proof Sketch:

Original: The stochastic update to the Actor is not a direct gradient-descent update.
Revised: The stochastic update to the Actor is not a direct gradient-descent update.

Original: Each update to the Actor is a CEM update, which requires a different analysis to ensure that the stochastic noise remains bounded and is asymptotically negligible.
Revised: Each update to the Actor is a CEM update, which requires a different analysis to ensure that the stochastic noise remains bounded and is asymptotically negligible.

Original: Further, the classical results of CEM also do not immediately apply, because such updates assume distribution parameters can be directly computed.
Revised: Further, the classical results of CEM also do not immediately apply, because such updates assume distribution parameters can be directly computed.

Original: Here, distribution parameters are conditioned on state, as outputs from a parametrized function.
Revised: Here, distribution parameters are conditioned on state, as outputs from a parametrized function.

Original: We identify conditions on the parametrized policy to ensure well-behaved CEM updates.
Revised: We identify conditions on the parametrized policy to ensure well-behaved CEM updates.

Original: The multi-timescale analysis allows us to focus on the updates of the Actor w t , assuming the actionvalue parameter θ and action-sampling parameter w ′ are quasi-static.
Revised: The multi-timescale analysis allows us to focus on the updates of the Actor w t , assuming the actionvalue parameter θ and action-sampling parameter w ′ are quasi-static.

Original: These parameters are allowed to change with time—as they will in practice—but are moving at a sufficiently slower timescale relative to w t and hence the analysis can be undertaken as if they are static.
Revised: These parameters are allowed to change with time—as they will in practice—but are moving at a sufficiently slower timescale relative to w t and hence the analysis can be undertaken as if they are static.

Original: The first step in the proof is to formulate the update to the weights as a projected stochastic recursion— simply meaning a stochastic update where after each update the weights are projected to a compact, convex set to keep them bounded.
Revised: The first step in the proof is to formulate the update to the weights as a projected stochastic recursion— simply meaning a stochastic update where after each update the weights are projected to a compact, convex set to keep them bounded.

Original: The stochastic recursion is reformulated into a summation involving the mean vector field g θ ( w t ) (which depends on the action-value parameters θ ), martingale noise, and a loss term ℓ θt that is due to having approximate quantiles.
Revised: The stochastic recursion is reformulated into a summation involving the mean vector field g θ ( w t ) (which depends on the action-value parameters θ ), martingale noise, and a loss term ℓ θt that is due to having approximate quantiles.

Original: The key steps are then to show almost surely that the mean vector field g θ is locally Lipschitz, the martingale noise is quadratically bounded and that the loss term ℓ θ t decays to zero asymptotically.
Revised: The key steps are then to show almost surely that the mean vector field g θ is locally Lipschitz, the martingale noise is quadratically bounded and that the loss term ℓ θ t decays to zero asymptotically.

Original: For the first and second, we identify conditions on the policy parameterization that guarantee these.
Revised: For the first and second, we identify conditions on the policy parameterization that guarantee these.

Original: For the final case, we adapt the proof for sampled quantiles approaching true quantiles for CEM, with modifications to account for expectations over the conditioning variable, the state.
Revised: For the final case, we adapt the proof for sampled quantiles approaching true quantiles for CEM, with modifications to account for expectations over the conditioning variable, the state.

Original: ■
Revised: ■

Original: This result has several limitations.
Revised: This result has several limitations.

Original: First, it does not perfectly characterize the CCEM algorithm that we actually use.
Revised: First, it does not perfectly characterize the CCEM algorithm that we actually use.

Original: We do not use the update w ′ t
Revised: We do not use the update w ′ t

Original: =
Revised: =

Original: (1 − α ′ a,t ) w ′ t + α ′ a,t w t , and instead use entropy regularization to make w ′ t concentrate more slowly than w t .
Revised: (1 − α ′ a,t ) w ′ t + α ′ a,t w t , and instead use entropy regularization to make w ′ t concentrate more slowly than w t .

Original: The principle is similar; empirically we found entropy regularization to be an effective strategy to achieve this condition.
Revised: The principle is similar; empirically we found entropy regularization to be an effective strategy to achieve this condition.

Original: Second, the theory assumes the state distribution is fixed, and not influenced by π w .
Revised: Second, the theory assumes the state distribution is fixed, and not influenced by π w .

Original: It is standard to analyze the properties of (off-policy) algorithms for fixed datasets as a first step, as was done for
Revised: It is standard to analyze the properties of (off-policy) algorithms for fixed datasets as a first step, as was done for

Original: Q-learning (Jaakkola et al., 1994).
Revised: Q-learning (Jaakkola et al., 1994).

Original: It allows us to separate concerns, and just ask: does our method concentrate on maximal actions across states?
Revised: It allows us to separate concerns, and just ask: does our method concentrate on maximal actions across states?

Original: An important next step is to characterize the full
Revised: An important next step is to characterize the full

Original: Greedy Actor-Critic algorithm, beyond just understanding the properties of the CCEM component.
Revised: Greedy Actor-Critic algorithm, beyond just understanding the properties of the CCEM component.

Original:  E MPIRICAL R ESULTS
Revised:  E MPIRICAL R ESULTS

Original: We are primarily interested in investigating sensitivity to hyperparameters.
Revised: We are primarily interested in investigating sensitivity to hyperparameters.

Original: This sensitivity reflects how difficult it can be to get AC methods working on a new task—relevant for both applied settings and research.
Revised: This sensitivity reflects how difficult it can be to get AC methods working on a new task—relevant for both applied settings and research.

Original: AC methods have been notoriously difficult to tune due to the interacting time scales of the actor and critic (Degris et al., 2012a), further compounded by the sensitivity in the entropy scale.
Revised: AC methods have been notoriously difficult to tune due to the interacting time scales of the actor and critic (Degris et al., 2012a), further compounded by the sensitivity in the entropy scale.

Original: The use of modern optimizers may reduce some of the sensitivity in stepsize selection; these experiments help understand if that is the case.
Revised: The use of modern optimizers may reduce some of the sensitivity in stepsize selection; these experiments help understand if that is the case.

Original: Further, a very well-tuned algorithm may not be representative of performance across problems.
Revised: Further, a very well-tuned algorithm may not be representative of performance across problems.

Original: We particularly examine the impacts of selecting a single set of hyperparameters across environments, in contrast to tuning per environment.
Revised: We particularly examine the impacts of selecting a single set of hyperparameters across environments, in contrast to tuning per environment.

Original: We chose to conduct experiments in small, challenging domains appropriately sized for extensive experiment repetition.
Revised: We chose to conduct experiments in small, challenging domains appropriately sized for extensive experiment repetition.

Original: Ensuring significance in results and carefully exploring hyperparameter sensitivity required many experiments.
Revised: Ensuring significance in results and carefully exploring hyperparameter sensitivity required many experiments.

Original: Our final plots required ∼ 30,000 runs across all environments, algorithms, and hyperparameters.
Revised: Our final plots required ∼ 30,000 runs across all environments, algorithms, and hyperparameters.

Original: Further, contrary to popular belief, classic control domains are a challenge for Deep RL agents (Ghiassian et al., 2020), and performance differences in these environments have been shown to extend to larger environments (Obando-Ceron & Castro, 2021).
Revised: Further, contrary to popular belief, classic control domains are a challenge for Deep RL agents (Ghiassian et al., 2020), and performance differences in these environments have been shown to extend to larger environments (Obando-Ceron & Castro, 2021).

Original: 5. A LGORITHMS
Revised: 5. A LGORITHMS

Original: We focus on comparing GreedyAC to Soft Actor-Critic (SAC) both since this allows us to compare to a method that uses the Boltzmann target policy on action-values and because SAC continues to have the most widely reported success 2 .
Revised: We focus on comparing GreedyAC to Soft Actor-Critic (SAC) both since this allows us to compare to a method that uses the Boltzmann target policy on action-values and because SAC continues to have the most widely reported success 2 .

Original: We additionally include VanillaAC as a baseline, a basic AC variant which does not include any of the tricks SAC utilizes to improve performance, such as action reparameterization to estimate the policy gradient or double Q functions to mitigate maximization bias.
Revised: We additionally include VanillaAC as a baseline, a basic AC variant which does not include any of the tricks SAC utilizes to improve performance, such as action reparameterization to estimate the policy gradient or double Q functions to mitigate maximization bias.

Original: For discrete actions, policies are parameterized using Softmax distributions.
Revised: For discrete actions, policies are parameterized using Softmax distributions.

Original: For continuous actions, policies are parameterized using Gaussian distributions, except SAC which uses a squashed
Revised: For continuous actions, policies are parameterized using Gaussian distributions, except SAC which uses a squashed

Original: Gaussian policy as per the original work.
Revised: Gaussian policy as per the original work.

Original: We tested SAC with a Gaussian policy, and it performed worse.
Revised: We tested SAC with a Gaussian policy, and it performed worse.

Original: All algorithms use neural networks.
Revised: All algorithms use neural networks.

Original: Feedforward networks consist of two hidden layers ofunits (classic control environments) or 256 units (Swimmer-v3 environment).
Revised: Feedforward networks consist of two hidden layers ofunits (classic control environments) or 256 units (Swimmer-v3 environment).

Original: Convolutional layers consists of one convolutional layer with 3 kernels of size 16 followed by a fully connected layer of size 128.
Revised: Convolutional layers consists of one convolutional layer with 3 kernels of size 16 followed by a fully connected layer of size 128.

Original: All algorithms use the Adam optimizer (Kingma & Ba, 2014), experience replay, and target networks for the value functions.
Revised: All algorithms use the Adam optimizer (Kingma & Ba, 2014), experience replay, and target networks for the value functions.

Original: See Appendix C.1 for a full discussion of hyperparameters.
Revised: See Appendix C.1 for a full discussion of hyperparameters.

Original: 5. E NVIRONMENTS
Revised: 5. E NVIRONMENTS

Original: We use the classic versions of Mountain Car (Sutton & Barto, 2018),
Revised: We use the classic versions of Mountain Car (Sutton & Barto, 2018),

Original: Pendulum (Degris et al., 2012a), and Acrobot (Sutton & Barto, 2018).
Revised: Pendulum (Degris et al., 2012a), and Acrobot (Sutton & Barto, 2018).

Original: Each environment is run with both continuous and discrete action spaces; states are continuous.
Revised: Each environment is run with both continuous and discrete action spaces; states are continuous.

Original: Discrete actions consist of the two extreme continuous actions and 0.
Revised: Discrete actions consist of the two extreme continuous actions and 0.

Original: All environments use a discount factor of γ = 0 . 99 , and episodes are cut off at 1,000 timesteps, teleporting the agent back to the start state (but not causing termination).
Revised: All environments use a discount factor of γ = 0 . 99 , and episodes are cut off at 1,000 timesteps, teleporting the agent back to the start state (but not causing termination).

Original: To demonstrate the potential of GreedyAC at scale, we also include experiments on Freeway and Breakout from MinAtar (Young & Tian, 2019) as well as on Swimmer-v3 from OpenAI Gym (Brockman et al., 2016).
Revised: To demonstrate the potential of GreedyAC at scale, we also include experiments on Freeway and Breakout from MinAtar (Young & Tian, 2019) as well as on Swimmer-v3 from OpenAI Gym (Brockman et al., 2016).

Original: On MinAtar, episodes are cutoff at 2,500 timesteps.
Revised: On MinAtar, episodes are cutoff at 2,500 timesteps.

Original: In Mountain Car, the goal is to drive an underpowered car up a hill.
Revised: In Mountain Car, the goal is to drive an underpowered car up a hill.

Original: State consists of the position in [ − 1 . 2 , 0 . 6] and velocity in [ − 0 .
Revised: State consists of the position in [ − 1 . 2 , 0 . 6] and velocity in [ − 0 .

Original: The agent starts in a random position in [ − 0 .
Revised: The agent starts in a random position in [ − 0 .

Original: 6 , − 0 .
Revised: 6 , − 0 .

Original: 4] and velocity 0.
Revised: 4] and velocity 0.

Original: The action is the force to apply to the car, in [ − 1 , 1] .
Revised: The action is the force to apply to the car, in [ − 1 , 1] .

Original: The reward is -1 per step.
Revised: The reward is -1 per step.

Original: In Pendulum, the goal is to hold a pendulum with a fixed base in a vertical position.
Revised: In Pendulum, the goal is to hold a pendulum with a fixed base in a vertical position.

Original: State consists of the angle (normalized in [ − π, π ) ) and angular momentum (in [ − 1 , 1] ).
Revised: State consists of the angle (normalized in [ − π, π ) ) and angular momentum (in [ − 1 , 1] ).

Original: The agent starts with the pendulum facing downwards and 0 velocity.
Revised: The agent starts with the pendulum facing downwards and 0 velocity.

Original: The action is the torque applied to the fixed base, in [ − 2 , 2] .
Revised: The action is the torque applied to the fixed base, in [ − 2 , 2] .

Original: The reward is the cosine of the angle of the pendulum from the positive y-axis.
Revised: The reward is the cosine of the angle of the pendulum from the positive y-axis.

Original: In Acrobot, the agent controls a doubly-linked pendulum with a fixed base.
Revised: In Acrobot, the agent controls a doubly-linked pendulum with a fixed base.

Original: The goal is to swing the second link one link’s length above the fixed base.
Revised: The goal is to swing the second link one link’s length above the fixed base.

Original: State consists of the angle of each link (in [ − π, π ) ) and the angular velocity of each link (in [ − 4 π, 4 π ] and [ − 9 π, 9 π ] respectively).
Revised: State consists of the angle of each link (in [ − π, π ) ) and the angular velocity of each link (in [ − 4 π, 4 π ] and [ − 9 π, 9 π ] respectively).

Original: The agent starts with random angles and angular velocities in [ − 0 . 1 , 0 . 1] .
Revised: The agent starts with random angles and angular velocities in [ − 0 . 1 , 0 . 1] .

Original: The action is the torque applied to the joint between the two links, in [ − 1 , 1] .
Revised: The action is the torque applied to the joint between the two links, in [ − 1 , 1] .

Original: The reward is -1 per step.
Revised: The reward is -1 per step.

Original: 5. E XPERIMENTAL D ETAILS
Revised: 5. E XPERIMENTAL D ETAILS

Original: We sweep hyperparameters for 40 runs, tuning over the first 10 runs and reporting results using the final 30 runs for the best hyperparameters.
Revised: We sweep hyperparameters for 40 runs, tuning over the first 10 runs and reporting results using the final 30 runs for the best hyperparameters.

Original: We sweep critic step size α = 10 x for x ∈ {− 5 , − 4 , . .
Revised: We sweep critic step size α = 10 x for x ∈ {− 5 , − 4 , . .

Original: , − 1 } .
Revised: , − 1 } .

Original: We set the actor step size to be κ × α and sweep κ ∈ (cid:8) 10 − 3 , 10 − 2 , 10 − 1 , 1 , 2 , 10 (cid:9) .
Revised: We set the actor step size to be κ × α and sweep κ ∈ (cid:8) 10 − 3 , 10 − 2 , 10 − 1 , 1 , 2 , 10 (cid:9) .

Original: We sweep entropy scales τ = 10 y for y ∈ {− 3 , − 2 , − 1 , 0 , 1 } .
Revised: We sweep entropy scales τ = 10 y for y ∈ {− 3 , − 2 , − 1 , 0 , 1 } .

Original: For the classic control experiments, we used fixed batch sizes of 32 samples and a replay buffer capacity of 100,000 samples.
Revised: For the classic control experiments, we used fixed batch sizes of 32 samples and a replay buffer capacity of 100,000 samples.

Original: For the MinAtar experiments, we used fixed batch sizes of 32 samples and a buffer capacity of 1 million.
Revised: For the MinAtar experiments, we used fixed batch sizes of 32 samples and a buffer capacity of 1 million.

Original: For the
Revised: For the

Original: Swimmer experiments, we used fixed batch sizes of 100 samples and a buffer capacity of 1 million.
Revised: Swimmer experiments, we used fixed batch sizes of 100 samples and a buffer capacity of 1 million.

Original: For CCEM, we fixed ρ = 0 .
Revised: For CCEM, we fixed ρ = 0 .

Original: 1 and sample N = 30 actions.
Revised: 1 and sample N = 30 actions.

Original: To select hyperparameters across environment, we must normalize performance to provide an aggregate score.
Revised: To select hyperparameters across environment, we must normalize performance to provide an aggregate score.

Original: We use near-optimal performance as the normalizer for each environment, with a score of 1 meaning equal to this performance.
Revised: We use near-optimal performance as the normalizer for each environment, with a score of 1 meaning equal to this performance.

Original: We only use this normalization to average scores across environments.
Revised: We only use this normalization to average scores across environments.

Original: We report learning curves using the original unnormalized returns.
Revised: We report learning curves using the original unnormalized returns.

Original: For more details, see Appendix C.2.
Revised: For more details, see Appendix C.2.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: [Figure]
Revised: [Figure]

Original: Per-environment Tuning: We first examine how well the algorithms can perform when they are tuned perenvironment.
Revised: Per-environment Tuning: We first examine how well the algorithms can perform when they are tuned perenvironment.

Original: In Figure 2, we see that
Revised: In Figure 2, we see that

Original: SAC performs well in Pendulum-CA (continuous actions) and in PendulumDA (discrete actions) but poorly in the other settings.
Revised: SAC performs well in Pendulum-CA (continuous actions) and in PendulumDA (discrete actions) but poorly in the other settings.

Original: SAC learns slower than
Revised: SAC learns slower than

Original: GreedyAC and VanillaAC on Acrobot.
Revised: GreedyAC and VanillaAC on Acrobot.

Original: GreedyAC performs worse than SAC in Pendulum-CA, but still performs acceptably, nearly reaching the same final performance.
Revised: GreedyAC performs worse than SAC in Pendulum-CA, but still performs acceptably, nearly reaching the same final performance.

Original: SAC performs poorly on both versions of Mountain
Revised: SAC performs poorly on both versions of Mountain

Original: Car.
Revised: Car.

Original: That AC methods struggle with Acrobot is common wisdom, but here we see that both GreedyAC and VanillaAC do well on this problem.
Revised: That AC methods struggle with Acrobot is common wisdom [Figure] but here we see that both GreedyAC and VanillaAC do well on this problem.

Original: GreedyAC is the clear winner in Mountain Car
Revised: GreedyAC is the clear winner in Mountain Car

Original: [Figure]
Revised: [Figure]

Original: Across-environment Tuning: We next examine the performance of the algorithms when they are forced to select one hyperparameter setting across continuous- or discrete-action environments separately, shown in Figure 3.
Revised: Across-environment Tuning: We next examine the performance of the algorithms when they are forced to select one hyperparameter setting across continuous- or discrete-action environments separately, shown in Figure 3.

Original: We expect algorithms that are less sensitive to their parameters to suffer less degradation.
Revised: We expect algorithms that are less sensitive to their parameters to suffer less degradation.

Original: Under this regime, GreedyAC has a clear advantage over SAC.
Revised: Under this regime, GreedyAC has a clear advantage over SAC.

Original: GreedyAC maintains acceptable performance across all environments, sometimes learning more slowly than under per-environment tuning, but having reasonable behavior.
Revised: GreedyAC maintains acceptable performance across all environments, sometimes learning more slowly than under per-environment tuning, but having reasonable behavior.

Original: SAC performs poorly on two-thirds of the environments.
Revised: SAC performs poorly on two-thirds of the environments.

Original: GreedyAC is less sensitive than VanillaAC under across-environment tuning and performs at least as good as VanillaAC.
Revised: GreedyAC is less sensitive than VanillaAC under across-environment tuning and performs at least as good as VanillaAC.

Original: Hyperparameter Sensitivity: We examine the sensitivity of GreedyAC and SAC to their entropy scales, focusing on the continuous action environments.
Revised: Hyperparameter Sensitivity: We examine the sensitivity of GreedyAC and SAC to their entropy scales, focusing on the continuous action environments.

Original: We plot sensitivity curves, with one plotted for each entropy scale, with the stepsize on the x-axis and average return across all steps and allruns on the y-axis.
Revised: We plot sensitivity curves, with one plotted for each entropy scale, with the stepsize on the x-axis and average return across all steps and allruns on the y-axis.

Original: Because there are two stepsizes, we have two sets of plots.
Revised: Because there are two stepsizes, we have two sets of plots.

Original: When examining the sensitivity to the critic stepsize, we select the best actor stepsize.
Revised: When examining the sensitivity to the critic stepsize, we select the best actor stepsize.

Original: We do the same for the actor stepsize plots.
Revised: We do the same for the actor stepsize plots.

Original: We provide the plots with individual lines in Appendix C.3 and here focus on a more summarized view.
Revised: We provide the plots with individual lines in Appendix C.3 and here focus on a more summarized view.

Original: [Figure]
Revised: [Figure]

Original: Figure 4 depicts the range of performance obtained across entropy scales.
Revised: Figure 4 depicts the range of performance obtained across entropy scales.

Original: The plot is generated by filling in the region between the curves for each entropy scale.
Revised: The plot is generated by filling in the region between the curves for each entropy scale.

Original: If this sensitivity region is broad, then the algorithm performed very differently across different entropy scales and so is sensitive to the entropy.
Revised: If this sensitivity region is broad, then the algorithm performed very differently across different entropy scales and so is sensitive to the entropy.

Original: SAC has much wider sensitivity regions than GreedyAC.
Revised: SAC has much wider sensitivity regions than GreedyAC.

Original: Those of GreedyAC are generally narrow, indicating that the stepsize rather than entropy was the dominant factor.
Revised: Those of GreedyAC are generally narrow, indicating that the stepsize rather than entropy was the dominant factor.

Original: Further, the bands of performance are generally at the top of the plot.
Revised: Further, the bands of performance are generally at the top of the plot.

Original: When SAC exhibits narrower regions than GreedyAC, those regions are lower on the plot, indicating overall poor performance.
Revised: When SAC exhibits narrower regions than GreedyAC, those regions are lower on the plot, indicating overall poor performance.

Original:  S CALING G REEDY 
Revised:  S CALING G REEDY -AC

Original: [Figure]
Revised: [Figure]

Original: We ran GreedyAC on two pixel-based control problems, Breakout and Freeway, from the MinAtar suite (Young & Tian, 2019).
Revised: We ran GreedyAC on two pixel-based control problems, Breakout and Freeway, from the MinAtar suite (Young & Tian, 2019).

Original: Recent work has shown that MinAtar results are indicative of those in much larger scale problems (Obando-Ceron & Castro, 2021).
Revised: Recent work has shown that MinAtar results are indicative of those in much larger scale problems (Obando-Ceron & Castro, 2021).

Original: We set the actor step-size scale to 1.
Revised: We set the actor step-size scale to 1.

Original: and a critic step-size of 10 − 3 for both GreedyAC and SAC—the defaults of SAC.
Revised: and a critic step-size of 10 − 3 for both GreedyAC and SAC—the defaults of SAC.

Original: We set the entropy scale of SAC to 10 − 3 based on a grid search.
Revised: We set the entropy scale of SAC to 10 − 3 based on a grid search.

Original: Figure 5a above clearly indicates GreedyAC can learn a good policy from high-dimensional inputs; comparable performance to DQN Rainbow.
Revised: Figure 5a above clearly indicates GreedyAC can learn a good policy from high-dimensional inputs; comparable performance to DQN Rainbow.

Original: Finally, we ran GreedyAC on Swimmer-v3 from OpenAI Gym (Brockman et al., 2016).
Revised: Finally, we ran GreedyAC on Swimmer-v3 from OpenAI Gym (Brockman et al., 2016).

Original: We tuned over one run and then ran the tuned hyperparameters for an additional 9 runs to generate Figure 5b.
Revised: We tuned over one run and then ran the tuned hyperparameters for an additional 9 runs to generate Figure 5b.

Original: We report online and offline performance.
Revised: We report online and offline performance.

Original: Offline evaluation is performed every 10,000 steps, for 10 episodes, where only the mean action is selected and learning is disabled.
Revised: Offline evaluation is performed every 10,000 steps, for 10 episodes, where only the mean action is selected and learning is disabled.

Original: We report SAC’s final performance on Swimmer from the SpinningUp benchmark 3 .
Revised: We report SAC’s final performance on Swimmer from the SpinningUp benchmark 3 .

Original: GreedyAC is clearly not state-of-the-art here—most methods are not—however, GreedyAC steadily improves throughout the experiment.
Revised: GreedyAC is clearly not state-of-the-art here—most methods are not—however, GreedyAC steadily improves throughout the experiment.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced a new Actor-Critic (AC)
Revised: In this work, we introduced a new Actor-Critic (AC)

Original: algorithm called GreedyAC, that uses a new update to the Actor based on an extension of the cross-entropy method (CEM).
Revised: algorithm called GreedyAC, that uses a new update to the Actor based on an extension of the cross-entropy method (CEM).

Original: The idea is to (a) define a percentile-greedy target policy and (b) update the actor towards this target policy, by reducing a KL divergence to it.
Revised: The idea is to (a) define a percentile-greedy target policy and (b) update the actor towards this target policy, by reducing a KL divergence to it.

Original: This percentile-greedy policy guarantees policy improvement, and we prove that
Revised: This percentile-greedy policy guarantees policy improvement, and we prove that

Original: our
Revised: our

Original: Conditional CEM algorithm tracks the actions of maximal value under changing action-values.
Revised: Conditional CEM algorithm tracks the actions of maximal value under changing action-values.

Original: We conclude with an in-depth empirical study, showing that GreedyAC has significantly lower sensitivity to its hyperparameters than SAC does.
Revised: We conclude with an in-depth empirical study, showing that GreedyAC has significantly lower sensitivity to its hyperparameters than SAC does.

Original: A R ELATED P OLICY O PTIMIZATION A LGORITHMSAs
Revised: A R ELATED P OLICY O PTIMIZATION A LGORITHMS

Original:  mentioned in the main text, there are many policy optimization algorithms that can be seen as approximate policy iteration (API) rather than performing gradient descent on a policy objective.
Revised: As mentioned in the main text, there are many policy optimization algorithms that can be seen as approximate policy iteration (API) rather than performing gradient descent on a policy objective.

Original: An overview and survey are given by Vieillard et al.
Revised: An overview and survey are given by Vieillard et al.

Original: (2020a)
Revised: (2020a)

Original: and Chan et al.
Revised: and Chan et al.

Original: There, many methods are shown to either minimize a forward or reverse KL-divergence to the Boltzmann policy.
Revised: There, many methods are shown to either minimize a forward or reverse KL-divergence to the Boltzmann policy.

Original: Our approach similarly updates the actor using a KL-divergence to a target policy, but here that target policy is the percentile-greedy policy.
Revised: Our approach similarly updates the actor using a KL-divergence to a target policy, but here that target policy is the percentile-greedy policy.

Original: By doing a maximum likelihood update with actions sampled under the percentile-greedy policy, we are reducing the forward KL-divergence to the percentile-greedy policy in Equation 2.
Revised: By doing a maximum likelihood update with actions sampled under the percentile-greedy policy, we are reducing the forward KL-divergence to the percentile-greedy policy in Equation 2.

Original: Our CCEM update for the actor is new, but there are several approaches that resemble the idea, particularly those that try to match an expert.
Revised: Our CCEM update for the actor is new, but there are several approaches that resemble the idea, particularly those that try to match an expert.

Original: This includes dual policy iteration methods (DPI) (Parisotto et al., 2016; Sun et al., 2018; Steckelmacher et al., 2019) and RL as classification methods (Lagoudakis & Parr, 2003;
Revised: This includes dual policy iteration methods (DPI) (Parisotto et al., 2016; Sun et al., 2018; Steckelmacher et al., 2019) and RL as classification methods (Lagoudakis & Parr, 2003;

Original: Lazaric et al., 2010; Farahmand et al., 2015).
Revised: Lazaric et al., 2010; Farahmand et al., 2015).

Original: DPI has two policies, one which is guiding the other.
Revised: DPI has two policies, one which is guiding the other.

Original: For example, one policy might be an expensive tree search and another a learned neural network, trained to mimic the first (expert or guide) policy.
Revised: For example, one policy might be an expensive tree search and another a learned neural network, trained to mimic the first (expert or guide) policy.

Original: CCEM, on the other hand, uses two policies differently.
Revised: CCEM, on the other hand, uses two policies differently.

Original: Our actor does not imitate our proposal policy.
Revised: Our actor does not imitate our proposal policy.

Original: Rather, the proposal policy is used to improve the search over the nonconcave surface of Q. It samples actions more broadly, to make it more likely to find a maximizing action.
Revised: Rather, the proposal policy is used to improve the search over the nonconcave surface of Q. It samples actions more broadly, to make it more likely to find a maximizing action.

Original: Further, the actor increases the likelihood of only the top actions and does not imitate the proposal policy.
Revised: Further, the actor increases the likelihood of only the top actions and does not imitate the proposal policy.

Original: In contrast, Bootstrap DPI (Steckelmacher et al., 2019, Equation 5) uses an update based on Actor-Mimic (Parisotto et al., 2016), where the policy increases likelihood of actions for the softmax policies it is trying to mimic.
Revised: In contrast, Bootstrap DPI (Steckelmacher et al., 2019, Equation 5) uses an update based on Actor-Mimic (Parisotto et al., 2016), where the policy increases likelihood of actions for the softmax policies it is trying to mimic.

Original: The resemblance arises from the fact that (Steckelmacher et al., 2019, Equation 5) can be seen as a sum over forward KL divergences to softmax policies (for discrete actions), just like we have a forward
Revised: The resemblance arises from the fact that (Steckelmacher et al., 2019, Equation 5) can be seen as a sum over forward KL divergences to softmax policies (for discrete actions), just like we have a forward

Original: KL divergence but to the percentile-greedy policy (for discrete or continuous actions).
Revised: KL divergence but to the percentile-greedy policy (for discrete or continuous actions).

Original: The other class of algorithms, RL as classification, also look similar due to using a forward KL divergence.
Revised: The other class of algorithms, RL as classification, also look similar due to using a forward KL divergence.

Original: They reduce the problem to identifying “positive” actions in a state (producing maximal returns) and “negative” actions in a state (producing non-maximal returns).
Revised: They reduce the problem to identifying “positive” actions in a state (producing maximal returns) and “negative” actions in a state (producing non-maximal returns).

Original: If a cross-entropy loss is used, then this corresponds to maximizing the likelihood of the positive actions and minimizing the likelihood of the negative ones.
Revised: If a cross-entropy loss is used, then this corresponds to maximizing the likelihood of the positive actions and minimizing the likelihood of the negative ones.

Original: More generally, other classification algorithms can be used that do not involve maximizing likelihood (like SVMs).
Revised: More generally, other classification algorithms can be used that do not involve maximizing likelihood (like SVMs).

Original: The RL as classifications algorithms primarily focus on how to obtain these positive and negative actions, and otherwise look quite different from Greedy
Revised: The RL as classifications algorithms primarily focus on how to obtain these positive and negative actions, and otherwise look quite different from Greedy

Original: AC, in addition to being restricted to a discrete set of actions.
Revised: AC, in addition to being restricted to a discrete set of actions.

Original: Finally, we can also consider the connection to Conservative Policy Iteration (CPI)
Revised: Finally, we can also consider the connection to Conservative Policy Iteration (CPI)

Original: (Kakade &
Revised: (Kakade &

Original: Langford, 2002) and a generalization called Deep CPI (Vieillard et al., 2020b).
Revised: Langford, 2002) and a generalization called Deep CPI (Vieillard et al., 2020b).

Original: CPI updates the policy to be an interpolation between the greedy policy G ( Q ) and the current policy π , to get the new policy π ′ =
Revised: CPI updates the policy to be an interpolation between the greedy policy G ( Q ) and the current policy π , to get the new policy π ′ =

Original: (1 − α ) π
Revised: (1 − α ) π

Original: + αG ( Q ) for α ∈ [0 , 1] .
Revised: + αG ( Q ) for α ∈ [0 , 1] .

Original: Deep CPI extends this idea to parameterized policies, instead minimizing a forward KL to this interpolation policy.
Revised: Deep CPI extends this idea to parameterized policies, instead minimizing a forward KL to this interpolation policy.

Original: Greedy AC could be seen as another way to obtain a conservative update, because it does not move the actor all the way to the greedy policy.
Revised: Greedy AC could be seen as another way to obtain a conservative update, because it does not move the actor all the way to the greedy policy.

Original: Instead, it moves towards the percentile-greedy policy (in Equation 2), which shifts probability to the upper percentile of actions.
Revised: Instead, it moves towards the percentile-greedy policy (in Equation 2), which shifts probability to the upper percentile of actions.

Original: Similarly to the interpolation policy, this percentile-greedy policy depends on the previous policy and on increasing probability for maximally valued actions.
Revised: Similarly to the interpolation policy, this percentile-greedy policy depends on the previous policy and on increasing probability for maximally valued actions.

Original: As yet,
Revised: As yet,

Original: Deep CPI has not been shown to enjoy the same theoretical guarantees as CPI: minimizing a forward
Revised: Deep CPI has not been shown to enjoy the same theoretical guarantees as CPI: minimizing a forward

Original: KL to the interpolation policy does not provide the same guarantees.
Revised: KL to the interpolation policy does not provide the same guarantees.

Original: It remains an open question how to implement this conservative update in deep RL, and it would be interesting to understand if the CCEM update could provide an alternative route to obtaining such guarantees.
Revised: It remains an open question how to implement this conservative update in deep RL, and it would be interesting to understand if the CCEM update could provide an alternative route to obtaining such guarantees.

Original: B C ONVERGENCE A NALYSIS OF THE A CTOR
Revised: B C ONVERGENCE A NALYSIS OF THE A CTOR

Original: We provided an informal proof statement and proof sketch in Section 4.2, to provide intuition for the result.
Revised: We provided an informal proof statement and proof sketch in Section 4.2, to provide intuition for the result.

Original: Here, we provide the formal proof in the following subsections.
Revised: Here, we provide the formal proof in the following subsections.

Original: We first provide some definitions, particularly for the quantile function which is central to the analysis.
Revised: We first provide some definitions, particularly for the quantile function which is central to the analysis.

Original: We then lay out the assumptions, and discuss some policy parameterizations to satisfy those assumptions.
Revised: We then lay out the assumptions, and discuss some policy parameterizations to satisfy those assumptions.

Original: We finally state the theorem, with proof, and provide one lemma needed to prove the theorem in the final subsection.
Revised: We finally state the theorem, with proof, and provide one lemma needed to prove the theorem in the final subsection.

Original: B. N OTATION AND D EFINITIONS
Revised: B. N OTATION AND D EFINITIONS

Original: Notation:
Revised: Notation:

Original: For a set A , let ˚ A represent the interior of A , while ∂A is the boundary of A .
Revised: For a set A , let ˚ A represent the interior of A , while ∂A is the boundary of A .

Original: The abbreviation a.s. stands for almost surely and i.o. stands for infinitely often .
Revised: The abbreviation a.s. stands for almost surely and i.o. stands for infinitely often .

Original: Let N represent the set { 0 , 1 , 2 , . . . } .
Revised: Let N represent the set { 0 , 1 , 2 , . . . } .

Original: For a set A , we let I A to be the indicator function/characteristic function of A and is defined as I A ( x )
Revised: For a set A , we let I A to be the indicator function/characteristic function of A and is defined as I A ( x )

Original: =
Revised: =

Original: if x ∈ A and 0 otherwise.
Revised: if x ∈ A and 0 otherwise.

Original: Let E g [ · ] , V g [ · ] and P g ( · ) denote the expectation, variance and probability measure w.r.t.
Revised: Let E g [ · ] , V g [ · ] and P g ( · ) denote the expectation, variance and probability measure w.r.t.

Original: g .
Revised: g .

Original: For a σ -field F , let E [ ·| F ] represent the conditional expectation w.r.t.
Revised: For a σ -field F , let E [ ·| F ] represent the conditional expectation w.r.t.

Original: F .
Revised: F .

Original: A function f : X → Y is called Lipschitz continuous if ∃ L ∈ (0 , ∞ ) s.t.
Revised: A function f : X → Y is called Lipschitz continuous if ∃ L ∈ (0 , ∞ ) s.t.

Original: ∥ f ( x 1 ) − f ( x 2 ) ∥ ≤ L ∥ x 1 − x 2 ∥ , ∀ x 1 , x 2 ∈ X .
Revised: ∥ f ( x 1 ) − f ( x 2 ) ∥ ≤ L ∥ x 1 − x 2 ∥ , ∀ x 1 , x 2 ∈ X .

Original: A function f is called locally Lipschitz continuous if for every x ∈ X , there exists a neighbourhood U of X such that f | U is Lipschitz continuous.
Revised: A function f is called locally Lipschitz continuous if for every x ∈ X , there exists a neighbourhood U of X such that f | U is Lipschitz continuous.

Original: Let
Revised: Let

Original: C ( X, Y ) represent the space of continuous functions from X to Y .
Revised: C ( X, Y ) represent the space of continuous functions from X to Y .

Original: Also, let B r ( x ) represent an open ball of radius r with centered at x .
Revised: Also, let B r ( x ) represent an open ball of radius r with centered at x .

Original: For a positive integer M , let [ M ] .
Revised: For a positive integer M , let [ M ] .

Original: = { 1 , 2 . . .
Revised: = { 1 , 2 . . .

Original: M } .
Revised: M } .

Original: Definition 1.
Revised: Definition 1.

Original: A function Γ : U ⊆
Revised: A function Γ : U ⊆

Original: IR d 1 → V ⊆
Revised: IR d 1 → V ⊆

Original: IR
Revised: IR

Original: d 2 is Frechet differentiable at x ∈ U if there exists a bounded linear operator (cid:98) Γ x : IR d 1 → IR d 2 such that the limit
Revised: d 2 is Frechet differentiable at x ∈ U if there exists a bounded linear operator (cid:98) Γ x : IR d 1 → IR d 2 such that the limit

Original: [Equation]
Revised: [Equation]

Original: exists and is equal to (cid:98) Γ x ( y ) .
Revised: exists and is equal to (cid:98) Γ x ( y ) .

Original: We say Γ is Frechet differentiable if Frechet derivative of Γ exists at every point in its domain.
Revised: We say Γ is Frechet differentiable if Frechet derivative of Γ exists at every point in its domain.

Original: Definition 2.
Revised: Definition 2.

Original: Given a bounded real-valued continuous function H : IR d → IR with H ( a ) ∈ [ H l , H u ] and a scalar ρ ∈ [0 , 1] , we define the (1 − ρ ) -quantile of H ( A ) w.r.t.
Revised: Given a bounded real-valued continuous function H : IR d → IR with H ( a ) ∈ [ H l , H u ] and a scalar ρ ∈ [0 , 1] , we define the (1 − ρ ) -quantile of H ( A ) w.r.t.

Original: the PDF g (denoted as f ρ ( H, g ) ) as follows:
Revised: the PDF g (denoted as f ρ ( H, g ) ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where P g is the probability measure induced by the PDF g , i.e. , for a Borel set A , P g ( A ) .
Revised: where P g is the probability measure induced by the PDF g , i.e. , for a Borel set A , P g ( A ) .

Original: = (cid:82)
Revised: = (cid:82)

Original: A g ( a ) da .
Revised: A g ( a ) da .

Original: This quantile operator will be used to succinctly write the quantile for Q θ ( S, · ) , with actions selected according to π w , i.e.,
Revised: This quantile operator will be used to succinctly write the quantile for Q θ ( S, · ) , with actions selected according to π w , i.e.,

Original: [Equation]
Revised: [Equation]

Original: B. A SSUMPTIONS
Revised: B. A SSUMPTIONS

Original: Assumption 1.
Revised: Assumption 1.

Original: Given a realization of the transition dynamics of the MDP in the form of a sequence of transition tuples O .
Revised: Given a realization of the transition dynamics of the MDP in the form of a sequence of transition tuples O .

Original: = { ( S t , A t , R t , S ′ t ) } t ∈ N , where the state S t ∈ S is drawn using a latent sampling distribution ν , while A t ∈ A is the action chosen at state S t , the transitioned state
Revised: = { ( S t , A t , R t , S ′ t ) } t ∈ N , where the state S t ∈ S is drawn using a latent sampling distribution ν , while A t ∈ A is the action chosen at state S t , the transitioned state

Original: S ∋ S ′ t ∼ P ( S t , A t , · ) and the reward IR ∋ R t .
Revised: S ∋ S ′ t ∼ P ( S t , A t , · ) and the reward IR ∋ R t .

Original: =
Revised: =

Original: R ( S t , A t , S ′ t ) .
Revised: R ( S t , A t , S ′ t ) .

Original: We further assume that the reward is uniformly bounded, i.e. , | R ( · , · , · ) | < R max < ∞ .
Revised: We further assume that the reward is uniformly bounded, i.e. , | R ( · , · , · ) | < R max < ∞ .

Original: We analyze the long run behaviour of the conditional cross-entropy recursion (actor) which is defined as follows:
Revised: We analyze the long run behaviour of the conditional cross-entropy recursion (actor) which is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: whereΞ t . =
Revised: where

Original: { A t, 1 , A

Original: [Equation]
Revised: [Equation]

Original: Here, Γ W {·} is the projection operator onto the compact (closed and bounded) and convex set
Revised: Here, Γ W {·} is the projection operator onto the compact (closed and bounded) and convex set

Original: W ⊂ IR m with a smooth boundary ∂W .
Revised: W ⊂ IR m with a smooth boundary ∂W .

Original: Therefore, Γ W maps vectors in IR m to the nearest vectors in W w.r.t.
Revised: Therefore, Γ W maps vectors in IR m to the nearest vectors in W w.r.t.

Original: the Euclidean distance (or equivalent metric).
Revised: the Euclidean distance (or equivalent metric).

Original: Convexity and compactness ensure that the projection is unique and belongs to W .
Revised: Convexity and compactness ensure that the projection is unique and belongs to W .

Original: Assumption 2.
Revised: Assumption 2.

Original: The pre-determined, deterministic, step-size sequences { α a,t } t ∈ N , { α ′ a,t } t ∈ N and { α q,t } t ∈ N are positive scalars which satisfy the following:
Revised: The pre-determined, deterministic, step-size sequences { α a,t } t ∈ N , { α ′ a,t } t ∈ N and { α q,t } t ∈ N are positive scalars which satisfy the following:

Original: [Equation]
Revised: [Equation]

Original: The first conditions in Assumption 2 are the classical Robbins-Monro conditions (Robbins & Monro, 1985) required for stochastic approximation algorithms.
Revised: The first conditions in Assumption 2 are the classical Robbins-Monro conditions (Robbins & Monro, 1985) required for stochastic approximation algorithms.

Original: The last two conditions enable the different stochastic recursions to have separate timescales.
Revised: The last two conditions enable the different stochastic recursions to have separate timescales.

Original: Indeed, it ensures the w t recursion is faster compared to the recursions of θ t and w ′ t .
Revised: Indeed, it ensures the w t recursion is faster compared to the recursions of θ t and w ′ t .

Original: This timescale divide is needed to obtain the desired asymptotic behaviour, as we describe in the next section.
Revised: This timescale divide is needed to obtain the desired asymptotic behaviour, as we describe in the next section.

Original: Assumption 3.
Revised: Assumption 3.

Original: The pre-determined, deterministic, sample length schedule { N t ∈ N } t ∈ N is positive and strictly monotonically increases to ∞ and inf t ∈ N N t +1 N t > 1 .
Revised: The pre-determined, deterministic, sample length schedule { N t ∈ N } t ∈ N is positive and strictly monotonically increases to ∞ and inf t ∈ N N t +1 N t > 1 .

Original: Assumption 3 states that the number of samples increases to infinity and is primarily required to ensure that the estimation error arising due to the estimation of sample quantiles eventually decays to 0 .
Revised: Assumption 3 states that the number of samples increases to infinity and is primarily required to ensure that the estimation error arising due to the estimation of sample quantiles eventually decays to 0 .

Original: Practically, one can indeed consider a fixed, finite, positive integer for N t which is large enough to accommodate the acceptable error.
Revised: Practically, one can indeed consider a fixed, finite, positive integer for N t which is large enough to accommodate the acceptable error.

Original: Assumption 4.
Revised: Assumption 4.

Original: The sequence { θ t } t ∈ N satisfies θ t ∈ Θ , where Θ ⊂ IR n is a convex, compact set.
Revised: The sequence { θ t } t ∈ N satisfies θ t ∈ Θ , where Θ ⊂ IR n is a convex, compact set.

Original: Also, for θ ∈ Θ , let Q θ ( s
Revised: Also, for θ ∈ Θ , let Q θ ( s

Original: , a ) ∈
Revised: , a ) ∈

Original: [ Q θl , Q θu ] , ∀ s ∈ S , a ∈ A .
Revised: [ Q θl , Q θu ] , ∀ s ∈ S , a ∈ A .

Original: Assumption 4 assumes stability of the Expert, and minimally only requires that the values remain in a bounded range.
Revised: Assumption 4 assumes stability of the Expert, and minimally only requires that the values remain in a bounded range.

Original: We make no additional assumptions on the convergence properties of the Expert, as we simply need stability to prove the Actor tracks the update.
Revised: We make no additional assumptions on the convergence properties of the Expert, as we simply need stability to prove the Actor tracks the update.

Original: Assumption 5.
Revised: Assumption 5.

Original: For θ ∈ Θ and s ∈ S , let P
Revised: For θ ∈ Θ and s ∈ S , let P

Original: A ∼ π w ′ ( ·| s ) ( Q θ ( s, A ) ≥ ℓ ) > 0 , ∀ ℓ ∈ [ Q θl , Q θu ] and ∀ w ′ ∈ W .
Revised: A ∼ π w ′ ( ·| s ) ( Q θ ( s, A ) ≥ ℓ ) > 0 , ∀ ℓ ∈ [ Q θl , Q θu ] and ∀ w ′ ∈ W .

Original: Assumption 5 implies that there always exists a strictly positive probability mass beyond every threshold ℓ ∈ [ Q θl , Q θu ] .
Revised: Assumption 5 implies that there always exists a strictly positive probability mass beyond every threshold ℓ ∈ [ Q θl , Q θu ] .

Original: This assumption is easily satisfied when Q θ ( s, a ) is continuous in a and π w ( ·| s ) is a continuous probability density function.
Revised: This assumption is easily satisfied when Q θ ( s, a ) is continuous in a and π w ( ·| s ) is a continuous probability density function.

Original: Assumption 6.
Revised: Assumption 6.

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: (cid:21)

Original: [Equation]

Original: [Equation]

Original: Assumption 7.
Revised: Assumption 7.

Original: For s ∈ S , ∇ w ln π w ( ·| s ) is locally Lipschitz continuous w.r.t.
Revised: For s ∈ S , ∇ w ln π w ( ·| s ) is locally Lipschitz continuous w.r.t.

Original: w .
Revised: w .

Original: Assumptions 6 and 7 are technical requirements that can be more easily characterized when we consider π w to belong to the natural exponential family (NEF) of distributions.
Revised: Assumptions 6 and 7 are technical requirements that can be more easily characterized when we consider π w to belong to the natural exponential family (NEF) of distributions.

Original: Definition 3.
Revised: Definition 3.

Original: Natural exponential family of distributions (NEF)(Morris, 1982):
Revised: Natural exponential family of distributions (NEF)(Morris, 1982):

Original: These probability distributions over IR m are represented by
Revised: These probability distributions over IR m are represented by

Original: [Equation]
Revised: [Equation]

Original: where η is the natural parameter, h : IR m −→ IR , while T : IR m −→ IR d (called the sufficient statistic) and K ( η ) .
Revised: where η is the natural parameter, h : IR m −→ IR , while T : IR m −→ IR d (called the sufficient statistic) and K ( η ) .

Original: = ln (cid:82) h ( x ) e η ⊤ T ( x ) d x (called the cumulant function of the family).
Revised: = ln (cid:82) h ( x ) e η ⊤ T ( x ) d x (called the cumulant function of the family).

Original: The space Λ is defined as Λ .
Revised: The space Λ is defined as Λ .

Original: = { η ∈ IR d | | K ( η ) | < ∞} .
Revised: = { η ∈ IR d | | K ( η ) | < ∞} .

Original: Also, the above representation is assumed minimal.
Revised: Also, the above representation is assumed minimal.

Original: A few popular distributions which belong to the NEF family include Binomial, Poisson, Bernoulli,
Revised: A few popular distributions which belong to the NEF family include Binomial, Poisson, Bernoulli,

Original: Gaussian, Geometric and Exponential distributions.
Revised: Gaussian, Geometric and Exponential distributions.

Original: We parametrize the policy π w ( ·| S ) using a neural network, which implies that when we consider
Revised: We parametrize the policy π w ( ·| S ) using a neural network, which implies that when we consider

Original: NEF for the stochastic policy, the natural parameter η of the NEF is being parametrized by w .
Revised: NEF for the stochastic policy, the natural parameter η of the NEF is being parametrized by w .

Original: To be more specific, we have { ψ w : S → Λ | w ∈ IR m } to be the function space induced by the neural network of the actor, i.e. , for a given state s ∈ S , ψ w ( s ) represents the natural parameter of the NEF policy π w ( ·| s ) .
Revised: To be more specific, we have { ψ w : S → Λ | w ∈ IR m } to be the function space induced by the neural network of the actor, i.e. , for a given state s ∈ S , ψ w ( s ) represents the natural parameter of the NEF policy π w ( ·| s ) .

Original: Further,
Revised: Further,

Original: [Equation]
Revised: [Equation]

Original: Therefore Assumption 7 can be directly satisfied by assuming that ψ w is twice continuously differentiable w.r.t.
Revised: Therefore Assumption 7 can be directly satisfied by assuming that ψ w is twice continuously differentiable w.r.t.

Original: w .
Revised: w .

Original: Assumption 8.
Revised: Assumption 8.

Original: For every θ ∈ Θ , s ∈ S and w ∈ W , f ρθ ( w ; s ) (from Eq.
Revised: For every θ ∈ Θ , s ∈ S and w ∈ W , f ρθ ( w ; s ) (from Eq.

Original: equation 5) exists and is unique.
Revised: equation 5) exists and is unique.

Original: The above assumption ensures that the true (1 − ρ ) -quantile is unique and the assumption is usually satisfied for most distributions and a well-behaved Q θ .
Revised: The above assumption ensures that the true (1 − ρ ) -quantile is unique and the assumption is usually satisfied for most distributions and a well-behaved Q θ .

Original: B. M AIN T HEOREM
Revised: B. M AIN T HEOREM

Original: To analyze the algorithm, we employ here the ODE-based analysis as proposed in (Borkar, 2008;
Revised: To analyze the algorithm, we employ here the ODE-based analysis as proposed in (Borkar, 2008;

Original: Kushner & Clark, 2012).
Revised: Kushner & Clark, 2012).

Original: The actor recursions (Eqs.
Revised: The actor recursions (Eqs.

Original: (6-7)) represent a classical two timescale stochastic approximation recursion, where there exists a bilateral coupling between the individual stochastic recursions (6) and (7).
Revised: (6-7)) represent a classical two timescale stochastic approximation recursion, where there exists a bilateral coupling between the individual stochastic recursions (6) and (7).

Original: Since the step-size schedules { α a,t } t ∈ N and { α ′ a,t } t ∈ N satisfy α ′ a,t α a,t → 0 , we have α ′ a,t → 0 relatively faster than α a,t → 0 .
Revised: Since the step-size schedules { α a,t } t ∈ N and { α ′ a,t } t ∈ N satisfy α ′ a,t α a,t → 0 , we have α ′ a,t → 0 relatively faster than α a,t → 0 .

Original: This disparity induces a pseudoheterogeneous rate of convergence (or timescales) between the individual stochastic recursions which further amounts to the asymptotic emergence of a stable coherent behaviour which is quasiasynchronous.
Revised: This disparity induces a pseudoheterogeneous rate of convergence (or timescales) between the individual stochastic recursions which further amounts to the asymptotic emergence of a stable coherent behaviour which is quasiasynchronous.

Original: This pseudo-behaviour can be interpreted using multiple viewpoints.
Revised: This pseudo-behaviour can be interpreted using multiple viewpoints.

Original: When viewed from the faster timescale recursion— controlled by α a,t —the slower timescale recursion—controlled by α ′ a,t —appears quasi-static, i.e., almost a constant.
Revised: When viewed from the faster timescale recursion— controlled by α a,t —the slower timescale recursion—controlled by α ′ a,t —appears quasi-static, i.e., almost a constant.

Original: Likewise, when observed from the slower timescale, the faster timescale recursion seems equilibrated.
Revised: Likewise, when observed from the slower timescale, the faster timescale recursion seems equilibrated.

Original: The existence of this stable long run behaviour under certain standard assumptions of stochastic approximation algorithms is rigorously established in (Borkar, 1997) and also in Chapter 6 of (Borkar, 2008).
Revised: The existence of this stable long run behaviour under certain standard assumptions of stochastic approximation algorithms is rigorously established in (Borkar, 1997) and also in Chapter 6 of (Borkar, 2008).

Original: For our stochastic approximation setting (Eqs.
Revised: For our stochastic approximation setting (Eqs.

Original: (6-7)), we can directly apply this appealing characterization of the long run behaviour of the two timescale stochastic approximation algorithms—after ensuring the compliance of our setting to the pre-requisites demanded by the characterization—by considering the slow timescale stochastic recursion (7) to be quasi-stationary ( i.e. , w ′ t ≡ w ′ , a.s. , ∀ t ∈ N ), while analyzing the limiting behaviour of the faster timescale recursion (6).
Revised: (6-7)), we can directly apply this appealing characterization of the long run behaviour of the two timescale stochastic approximation algorithms—after ensuring the compliance of our setting to the pre-requisites demanded by the characterization—by considering the slow timescale stochastic recursion (7) to be quasi-stationary ( i.e. , w ′ t ≡ w ′ , a.s. , ∀ t ∈ N ), while analyzing the limiting behaviour of the faster timescale recursion (6).

Original: Similarly, we let θ t to be quasi-stationary too ( i.e. , θ t ≡ θ , a.s. , ∀ t ∈ N ).
Revised: Similarly, we let θ t to be quasi-stationary too ( i.e. , θ t ≡ θ , a.s. , ∀ t ∈ N ).

Original: The asymptotic behaviour of the slower timescale recursion is further analyzed by considering the faster timescale temporal variable w t with the limit point so obtained during quasi-stationary analysis.
Revised: The asymptotic behaviour of the slower timescale recursion is further analyzed by considering the faster timescale temporal variable w t with the limit point so obtained during quasi-stationary analysis.

Original: Define the filtration { F t } t ∈ N , a family of increasing natural σ -fields, where
Revised: Define the filtration { F t } t ∈ N , a family of increasing natural σ -fields, where

Original: [Equation]
Revised: [Equation]

Original: Theorem B.1.
Revised: Theorem B.1.

Original: Let w ′ t ≡ w ′ , θ t ≡ θ, ∀ t ∈ N a.s.
Revised: Let w ′ t ≡ w ′ , θ t ≡ θ, ∀ t ∈ N a.s.

Original: Let Assumptions 1-8 hold.
Revised: Let Assumptions 1-8 hold.

Original: Then the stochastic sequence { w t } t ∈ N generated by the stochastic recursion (6) asymptotically tracks the ODE:
Revised: Then the stochastic sequence { w t } t ∈ N generated by the stochastic recursion (6) asymptotically tracks the ODE:

Original: [Equation]
Revised: [Equation]

Original: In other words, lim t →∞ w t ∈ K a.s. , where K is set of stable equilibria of the ODE (10) contained inside W .
Revised: In other words, lim t →∞ w t ∈ K a.s. , where K is set of stable equilibria of the ODE (10) contained inside W .

Original: Proof.
Revised: Proof.

Original: Firstly, we rewrite the stochastic recursion (6) under the hypothesis that θ t and w ′ t are quasi-stationary, i.e. , θ t ≡ a.s. θ and w ′ t ≡ a.s. w ′ as follows:
Revised: Firstly, we rewrite the stochastic recursion (6) under the hypothesis that θ t and w ′ t are quasi-stationary, i.e. , θ t ≡ a.s. θ and w ′ t ≡ a.s. w ′ as follows:

Original: [Equation]
Revised: [Equation]

Original: where f ρθ ( w ′ ; S ) .
Revised: where f ρθ ( w ′ ; S ) .

Original: =
Revised: =

Original: f ρ ( Q θ ( S, · ) , π w ′ ( ·| S )) and ∇ w t .
Revised: f ρ ( Q θ ( S, · ) , π w ′ ( ·| S )) and ∇ w t .

Original: = ∇ w = w t , i.e. , the gradient w.r
Revised: = ∇ w = w t , i.e. , the gradient w.r

Original: [Equation]
Revised: [Equation]

Original: t. w at w t .
Revised: t. w at w t .

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: Then we can rewrite equation11 =
Revised: Then we can rewrite equation

Original: [Equation]
Revised: [Equation]

Original: A few observations are in order:
Revised: A few observations are in order:

Original: B1.
Revised: B1.

Revised: M t +

Original: { M t +1 } t ∈ N is a martingale difference noise sequence w.r.t.
Revised: t ∈ N is a martingale difference noise sequence w.r.t.

Original: the filtration { F t } t ∈ N , i.e. , M t + is F t +1 -measurable and integrable, ∀ t ∈ N and E [ M t +1 | F t ] = 

Revised: the filtration { F t } t ∈ N , i.e. , M t + is F t + -measurable and integrable, ∀ t ∈ N and E [ M t +1 | F t ] =

Original: a.s. , ∀ t ∈ N . B2.
Revised: a.s. , ∀ t ∈ N . B2.

Original: g θ is locally Lipschitz continuous.
Revised: g θ is locally Lipschitz continuous.

Original: This follows from Assumption 7.
Revised: This follows from Assumption 7.

Original: B3.
Revised: B3.

Original: ℓ θt → 0 a.s. as t → ∞ .
Revised: ℓ θt → 0 a.s. as t → ∞ .

Original: (By Lemma 2 below).
Revised: (By Lemma 2 below).

Original: B4.
Revised: B4.

Original: The iterates { w t } t ∈ N is bounded almost surely, i e. , [Equation]
Revised: The iterates { w t } t ∈ N is bounded almost surely, i.e. , sup t ∈ N ∥ w t ∥ < ∞ a.s.

Original: This is ensured by the explicit application of the projection operator Γ W {·} over the iterates { w t } t ∈ N at every iteration onto the bounded set W . B5.
Revised: This is ensured by the explicit application of the projection operator Γ W {·} over the iterates { w t } t ∈ N at every iteration onto the bounded set W . B5.

Original: ∃ L ∈ (0 , ∞ ) s.t.
Revised: ∃ L ∈ (0 , ∞ ) s.t.

Original: E (cid:2)
Revised: E (cid:2)

Original: ∥ M t +1 ∥ 2 | F t (cid:3) ≤ L (cid:0) 1 + ∥ w t ∥ 2 (cid:1) a.s.
Revised: ∥ M t +1 ∥ 2 | F t (cid:3) ≤ L (cid:0) 1 + ∥ w t ∥ 2 (cid:1) a.s.

Original: This follows from Assumption 6 (ii)
Revised: This follows from Assumption 6 (ii).

Original: Now, we rewrite the stochastic recursion (15) as follows:
Revised: Now, we rewrite the stochastic recursion (15) as follows:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) Γ W is the Frechet derivative (Definition 3).
Revised: where (cid:98) Γ W is the Frechet derivative (Definition 3).

Original: The above stochastic recursion is also a stochastic approximation recursion with the vector field (cid:98) Γ W w t ( g θ ( w t )) , the noise term (cid:98) Γ W w t ( M t +1 ) , the bias term (cid:98) Γ W w t (cid:0) ℓ θt (cid:1) with an additional error term o ( α a,t ) which is asymptotically inconsequential.
Revised: The above stochastic recursion is also a stochastic approximation recursion with the vector field (cid:98) Γ W w t ( g θ ( w t )) , the noise term (cid:98) Γ W w t ( M t +1 ) , the bias term (cid:98) Γ W w t (cid:0) ℓ θt (cid:1) with an additional error term o ( α a,t ) which is asymptotically inconsequential.

Original: Also, note that Γ W is single-valued map since the set W is assumed convex and also the limit exists since the boundary ∂W is considered smooth.
Revised: Also, note that Γ W is single-valued map since the set W is assumed convex and also the limit exists since the boundary ∂W is considered smooth.

Original: Further, for w ∈ ˚ W , we have
Revised: Further, for w ∈ ˚ W , we have

Original: [Equation]
Revised: [Equation]

Original: i.e. , (cid:98) Γ W w ( · ) is an identity map for w ∈ ˚ W .
Revised: i.e. , (cid:98) Γ W w ( · ) is an identity map for w ∈ ˚ W .

Original: Now by appealing to Theorem 2, Chapter 2 of (Borkar, 2008) along with the observations B1-B5, we conclude that the stochastic recursion (6) asymptotically tracks the following ODE almost surely:
Revised: Now by appealing to Theorem 2, Chapter 2 of (Borkar, 2008) along with the observations B1-B5, we conclude that the stochastic recursion (6) asymptotically tracks the following ODE almost surely:

Original: [Equation]
Revised: [Equation]

Original: The interchange of expectation and the gradient in the last equality follows from dominated convergence theorem and Assumption 7 (Rubinstein & Shapiro, 1993).
Revised: The interchange of expectation and the gradient in the last equality follows from dominated convergence theorem and Assumption 7 (Rubinstein & Shapiro, 1993).

Original: The above ODE is a gradient flow with dynamics restricted inside W .
Revised: The above ODE is a gradient flow with dynamics restricted inside W .

Original: This further implies that the stochastic recursion (6) converges to a (possibly sample path dependent) asymptotically stable equilibrium point of the above ODE inside
Revised: This further implies that the stochastic recursion (6) converges to a (possibly sample path dependent) asymptotically stable equilibrium point of the above ODE inside

Original: W .
Revised: W .

Original: B. P ROOF OF L EMMA 2 TO SATISFY C ONDITION
Revised: B. P ROOF OF L EMMA 2 TO SATISFY C ONDITION

Original: In this section, we show that ℓ θt → 0 a.s. as t → ∞ , in Lemma 2.
Revised: In this section, we show that ℓ θt → 0 a.s. as t → ∞ , in Lemma 2.

Original: To do so, we first need to prove several supporting lemmas.
Revised: To do so, we first need to prove several supporting lemmas.

Original: Lemma 1 shows that, for a given Actor and Expert, the sample quantile converges to the true quantile.
Revised: Lemma 1 shows that, for a given Actor and Expert, the sample quantile converges to the true quantile.

Original: Using this lemma, we can then prove Lemma 2.
Revised: Using this lemma, we can then prove Lemma 2.

Original: In the following subsection, we provide three supporting lemmas about convexity and Lipschitz properties of the sample quantiles, required for the proof Lemma 1.
Revised: In the following subsection, we provide three supporting lemmas about convexity and Lipschitz properties of the sample quantiles, required for the proof Lemma 1.

Original: For this section, we require the following characterization of f ρ ( Q θ ( s, · ) , w ′ ) .
Revised: For this section, we require the following characterization of f ρ ( Q θ ( s, · ) , w ′ ) .

Original: Please refer Lemmaof (Homem-de Mello, 2007) for more details.
Revised: Please refer Lemmaof (Homem-de Mello, 2007) for more details.

Original: [Equation]
Revised: [Equation]

Original: where Ψ( y, ℓ ) .
Revised: where Ψ( y, ℓ ) .

Original: = ( y − ℓ )(1 − ρ )
Revised: = ( y − ℓ )(1 − ρ )

Original: I { y ≥ ℓ } + ( ℓ − y ) ρI { ℓ ≥ y } .
Revised: I { y ≥ ℓ } + ( ℓ − y ) ρI { ℓ ≥ y } .

Original: Similarly, the sample estimate of the true (1 − ρ ) -quantile, i.e. , (cid:98) f
Revised: Similarly, the sample estimate of the true (1 − ρ ) -quantile, i.e. , (cid:98) f

Original: ρ .
Revised: ρ .

Original: =
Revised: =

Original: Q ( ⌈ (1 − ρ ) N ⌉ ) θ,s , (where Q ( i ) θ,s is the i -th order statistic of the random sample { Q θ ( s, A ) } A ∈ Ξ with Ξ .
Revised: Q ( ⌈ (1 − ρ ) N ⌉ ) θ,s , (where Q ( i ) θ,s is the i -th order statistic of the random sample { Q θ ( s, A ) } A ∈ Ξ with Ξ .

Original: = { A i } Ni =1 iid ∼ π w ′ ( ·| s ) ) can be characterized as the unique solution of the stochastic counterpart of the above optimization problem, i.e. ,
Revised: = { A i } Ni =1 iid ∼ π w ′ ( ·| s ) ) can be characterized as the unique solution of the stochastic counterpart of the above optimization problem, i.e. ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: Assume θ t ≡ θ , w ′ t ≡ w ′ , ∀ t ∈ N .
Revised: Assume θ t ≡ θ , w ′ t ≡ w ′ , ∀ t ∈ N .

Original: Also, let Assumptions 3-5 hold.
Revised: Also, let Assumptions 3-5 hold.

Original: Then, for a given state s ∈ S ,
Revised: Then, for a given state s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) f ρt .
Revised: where (cid:98) f ρt .

Original: =
Revised: =

Original: Q ( ⌈ ( [Equation] ρ ) N t ⌉ ) θ,s , (where Q ( i ) θ,s is the i -th order statistic of the random sample { Q θ ( s, A ) } A ∈ Ξ t with Ξ t .
Revised: Q ( ⌈ (1 − ρ ) N t ⌉ ) θ,s , (where Q ( i ) θ,s is the i -th order statistic of the random sample { Q θ ( s, A ) } A ∈ Ξ t with Ξ t .

Original: = { A i } N t i =1 iid ∼ π w ′ ( ·| s ) ).
Revised: = { A i } N t i =1 iid ∼ π w ′ ( ·| s ) ).

Original: Proof.
Revised: Proof.

Original: The proof is similar to arguments in Lemma 7 of (Hu et al., 2007).
Revised: The proof is similar to arguments in Lemma 7 of (Hu et al., 2007).

Original: Since state s and expert parameter θ are considered fixed, we assume the following notation in the proof.
Revised: Since state s and expert parameter θ are considered fixed, we assume the following notation in the proof.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) f
Revised: where (cid:98) f

Original: ρt and f ρ ( Q θ ( s, · ) , w ′ ) are defined in Equations equation 19 and equation 20.
Revised: ρt and f ρ ( Q θ ( s, · ) , w ′ ) are defined in Equations equation 19 and equation 20.

Original: Consider the open cover { B r ( ℓ ) , ℓ ∈ [ Q θl , Q θu ] } of [ Q θl , Q θu ] .
Revised: Consider the open cover { B r ( ℓ ) , ℓ ∈ [ Q θl , Q θu ] } of [ Q θl , Q θu ] .

Original: Since [ Q θl , Q θu ] is compact, there exists a finite sub-cover, i.e. , ∃{ ℓ 1 , ℓ 2 , . .
Revised: Since [ Q θl , Q θu ] is compact, there exists a finite sub-cover, i.e. , ∃{ ℓ 1 , ℓ 2 , . .

Original: ℓ M } s.t.
Revised: ℓ M } s.t.

Original: ∪ M i =1 B r ( ℓ i ) =

Original: [ Q θ l , Q θu ] .
Revised: [Equation]

Original: Let ϑ [Equation] ℓ ) .
Revised: r

Original: [Equation]
Revised: [Equation]

Original: E [Equation] ∼ [Equation]
Revised: [ Q

Original: |

Original: [Equation]
Revised: [Equation]

Original: [Ψ [Equation] Q [Equation]
Revised: l , Q θu ] .

Original: A [Equation] [Equation] and (cid:98) ϑ
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Now, by triangle inequality, we have for ℓ ∈ [ Q θl , Q θu ] ,[Equation]
Revised: Now, by triangle inequality, we have for ℓ ∈ [ Q θl , Q θu ] ,

Original: [Equation]
Revised: [Equation]

Original: where L ρ and (cid:98) L ρ are the Lipschitz constants of ϑ ( · ) and (cid:98)
Revised: where L ρ and (cid:98) L ρ are the Lipschitz constants of ϑ ( · ) and (cid:98)

Original: ϑ t ( · ) respectively.
Revised: ϑ t ( · ) respectively.

Original: For δ > 0 , take r = δ ( L ρ + (cid:98) L ρ ) / 2 .
Revised: For δ > 0 , take r = δ ( L ρ + (cid:98) L ρ ) / 2 .

Original: Also, by Kolmogorov’s strong law of large numbers (Theorem 2.3.10 of (Sen & Singer, 2017)), we have (cid:98) ϑ
Revised: Also, by Kolmogorov’s strong law of large numbers (Theorem 2.3.10 of (Sen & Singer, 2017)), we have (cid:98) ϑ

Original: t ( ℓ ) → ϑ ( ℓ ) a.s.
Revised: t ( ℓ ) → ϑ ( ℓ ) a.s.

Original: This implies that there exists T ∈ N s.t.
Revised: This implies that there exists T ∈ N s.t.

Original: | ϑ ( ℓ j ) − (cid:98) ϑ
Revised: | ϑ ( ℓ j ) − (cid:98) ϑ

Original: t ( ℓ j ) | < δ/ 2 , ∀ t ≥ T , ∀ j ∈ [ M ] .
Revised: t ( ℓ j ) | < δ/ 2 , ∀ t ≥ T , ∀ j ∈ [ M ] .

Original: Then from Eq.
Revised: Then from Eq.

Original: (22), we have
Revised: (22), we have

Original: [Equation]
Revised: [Equation]

Revised: θu ] .

Original: This implies (cid:98) ϑ
Revised: This implies (cid:98) ϑ

Original: t converges uniformly to ϑ .
Revised: t converges uniformly to ϑ .

Original: By Lemmas 3 and 4, (cid:98) ϑ t and ϑ are strictly convex and
Revised: By Lemmas 3 and 4, (cid:98) ϑ t and ϑ are strictly convex and

Original: Lipschitz continuous, and so because (cid:98) ϑ t converges uniformly to ϑ , this means that the sequence of minimizers of (cid:98)
Revised: Lipschitz continuous, and so because (cid:98) ϑ t converges uniformly to ϑ , this means that the sequence of minimizers of (cid:98)

Original: ϑ
Revised: ϑ

Original: t converge to the minimizer of ϑ (see Lemma 5, Appendix B.6 for an explicit justification).
Revised: t converge to the minimizer of ϑ (see Lemma 5, Appendix B.6 for an explicit justification).

Original: These minimizers correspond to (cid:98) f ρt and f ρ ( Q θ ( s, · ) , w ′ ) respectively, and so lim N t →∞ (cid:98) f
Revised: These minimizers correspond to (cid:98) f ρt and f ρ ( Q θ ( s, · ) , w ′ ) respectively, and so lim N t →∞ (cid:98) f

Original: ρt =
Revised: ρt =

Original: f ρ ( Q θ ( s, · ) , w ′ )
Revised: f ρ ( Q θ ( s, · ) , w ′ )

Original: a.s.
Revised: a.s.

Original: Now, for δ > 0 and r .
Revised: Now, for δ > 0 and r .

Original: =
Revised: =

Original: δ ( L ρ + (cid:98) L ρ ) / 2 , we obtain the following from Eq.
Revised: δ ( L ρ + (cid:98) L ρ ) / 2 , we obtain the following from Eq.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where P π w ′ .
Revised: where P π w ′ .

Original: =
Revised: =

Original: P
Revised: P

Original: A ∼ π w ′ ( ·| s ) .
Revised: A ∼ π w ′ ( ·| s ) .

Original: And the last inequality follows from Hoeffding’s inequality (Hoeffding, 1963) along with the fact that E π w ′ (cid:104)(cid:98) ϑ t ( ℓ j ) (cid:105)
Revised: And the last inequality follows from Hoeffding’s inequality (Hoeffding, 1963) along with the fact that E π w ′ (cid:104)(cid:98) ϑ t ( ℓ j ) (cid:105)

Original: =
Revised: =

Original: ϑ ( ℓ j ) and sup
Revised: ϑ ( ℓ j ) and sup

Original: [Equation]
Revised: [Equation]

Original: Now, the sub-differential of ϑ ( ℓ ) is given by
Revised: Now, the sub-differential of ϑ ( ℓ ) is given by

Original: [Equation]
Revised: [Equation]

Original: By the definition of sub-gradient we obtain
Revised: By the definition of sub-gradient we obtain

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where 
Revised: where C .

Revised: =

Original: [Equation]
Revised: [Equation]

Original: Further,
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: From Eqs.
Revised: From Eqs.

Original: (23) and (26) [Equation] we obtain for ϵ >
Revised: (23) and (26), we obtain for ϵ >

Original: [Equation] 1 − 2 α t ϵ16( Q θu − Q θl )(cid:19) . For α ∈ (0 , 1 / 2) and inf t ∈ N N t +1 N t ≥ τ > 1 (by Assumption 3), then ∞ (cid:88) t =
Revised: P

Original: [Equation]
Revised: [Equation]

Original: exp
Revised: exp

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: For α ∈ (0 , 1 / 2) and inf t ∈ N N t +1 N t ≥ τ > 1 (by Assumption 3), then ∞ (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Borel-Cantelli’s Lemma (Durrett, 1991), we have
Revised: Therefore, by Borel-Cantelli’s Lemma (Durrett, 1991), we have

Original: [Equation]
Revised: [Equation]

Original: Thus we have N αt (cid:16) (cid:98)
Revised: Thus we have N αt (cid:16) (cid:98)

Original: f
Revised: f

Original: ρt | s,θ − f ρ | s,θ (cid:17) → 0 a.s. as N t → ∞ .
Revised: ρt | s,θ − f ρ | s,θ (cid:17) → 0 a.s. as N t → ∞ .

Original: Lemma 2.
Revised: Lemma 2.

Original: Almost surely, ℓ θt → 0 as N t → ∞ .
Revised: Almost surely, ℓ θt → 0 as N t → ∞ .

Original: Proof of Lemma 2: Consider
Revised: Proof of Lemma 2: Consider

Original: E

Original: [Equation]
Revised: [Equation]

Original: Notice that, because of the conditions on π w ′ ( ·| s )
Revised: Notice that, because of the conditions on π w ′ ( ·| s )

Original: [Equation]
Revised: [Equation]

Original: we know that the sample average converges with an exponential rate in the number of samples, for arbitrary w ′ ∈ W .
Revised: we know that the sample average converges with an exponential rate in the number of samples, for arbitrary w ′ ∈ W .

Original: Namely, for ϵ > 0 and N ∈ N [Equation] we have[Equation]
Revised: Namely, for ϵ > 0 and N ∈ N , we have

Original: S

Original: [Equation]
Revised: [Equation]

Original: where C 1 , c 2 , c 3 , c 4 > 0 .
Revised: where C 1 , c 2 , c 3 , c 4 > 0 .

Original: Therefore, for α ′ >
Revised: Therefore [Equation] for α ′ >

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where f ρθ,s .
Revised: where f ρθ,s .

Original: =
Revised: =

Original: f ρ ( Q θ ( s, · ) , π w ′ ( ·| s )) and inf t ∈ N N t +1 N t ≥ τ > 1 (by Assumption 3).
Revised: f ρ ( Q θ ( s, · ) , π w ′ ( ·| s )) and inf t ∈ N N t +1 N t ≥ τ > 1 (by Assumption 3).

Original: For c 3 − c 4 α ′ > 0 ⇒ α ′ <
Revised: For c 3 − c 4 α ′ > 0 ⇒ α ′ <

Original: c
Revised: c

Original: 3 /c 4 , we have
Revised: 3 /c 4 , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Borel-Cantelli’s Lemma (Durrett, 1991), we have
Revised: Therefore, by Borel-Cantelli’s Lemma (Durrett, 1991), we have

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: The above result implies that the sample average converges at a rate O ( N α ′ t ) , where 0 < α ′ <
Revised: The above result implies that the sample average converges at a rate O ( N α ′ t ) , where 0 < α ′ <

Original: c 3 /cindependent of w , w ′ ∈ W .
Revised: c 3 /cindependent of w , w ′ ∈ W .

Original: By Lemma 1, we have the sample quantiles (cid:98) f ρt also converging to the true quantile at a rate O ( N αt ) independent of w , w ′ ∈ W .
Revised: By Lemma 1, we have the sample quantiles (cid:98) f ρt also converging to the true quantile at a rate O ( N αt ) independent of w , w ′ ∈ W .

Original: Now the claim follows directly from
Revised: Now the claim follows directly from

Original: Assumption 6 (ii) and bounded convergence theorem.
Revised: Assumption 6 (ii) and bounded convergence theorem.

Original: ■
Revised: ■

Original: B. S UPPORTING L EMMAS FOR L EMMA
Revised: B. S UPPORTING L EMMAS FOR L EMMA

Original: Lemma 3.
Revised: Lemma 3.

Original: Let Assumption 5 hold.
Revised: Let Assumption 5 hold.

Original: For θ ∈ Θ , w ′ ∈ W , s ∈ S and ℓ ∈ [ Q θl , Q θu ] , we have
Revised: For θ ∈ Θ , w ′ ∈ W , s ∈ S and ℓ ∈ [ Q θl , Q θu ] , we have

Original: E A ∼ π w ′ ( ·| s )
Revised: E A ∼ π w ′ ( ·| s )

Original: [Ψ( Q θ ( s, A ) , ℓ )] is Lipschitz continuous.
Revised: [Ψ( Q θ ( s, A ) , ℓ )] is Lipschitz continuous.

Original: 1 N (cid:80)
Revised: 1 N (cid:80)

Original: A ∈ Ξ | Ξ | = N Ψ( Q θ ( s, A ) , ℓ ) (with Ξ iid ∼ π w ′ ( ·| s ) ) is Lipschitz continuous with Lipschitz constantindependent of the
Revised: A ∈ Ξ | Ξ | = N Ψ( Q θ ( s, A ) , ℓ ) (with Ξ iid ∼ π w ′ ( ·| s ) ) is Lipschitz continuous with Lipschitz constant

Original:  sample length N .
Revised: independent of the sample length N .

Original: Proof.
Revised: Proof.

Original: Let ℓ 1 , ℓ 2 ∈
Revised: Let ℓ 1 , ℓ 2 ∈

Original: [ Q θl , Q θu ] , ℓ 2 ≥ ℓ 1 .
Revised: [ Q θl , Q θu ] , ℓ 2 ≥ ℓ 1 .

Original: By Assumption 5 we have P A ∼ π w ′ ( ·| s ) ( Q θ ( s, A ) ≥ ℓ 1 ) >and P A ∼ π w ′ ( ·| s ) ( Q θ ( s, A ) ≥ ℓ 2 ) > 0 .
Revised: By Assumption 5 we have P A ∼ π w ′ ( ·| s ) ( Q θ ( s, A ) ≥ ℓ 1 ) >and P A ∼ π w ′ ( ·| s ) ( Q θ ( s, A ) ≥ ℓ 2 ) > 0 .

Original: Now,
Revised: Now,

Original: [Equation]
Revised: [Equation]

Original: Similarly [Equation] we can prove the later claim also.
Revised: Similarly, we can prove the later claim also.

Original: This completes the proof of Lemma3.
Revised: This completes the proof of Lemma

Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Let Assumption 5 hold.
Revised: Let Assumption 5 hold.

Original: Then, for θ ∈ Θ , w ′ ∈ W , s ∈ S and ℓ ∈ [ Q θl , Q θu ] , we have E A ∼ π w ′ ( ·| s )
Revised: Then, for θ ∈ Θ , w ′ ∈ W , s ∈ S and ℓ ∈ [ Q θl , Q θu ] , we have E A ∼ π w ′ ( ·| s )

Original: [Ψ( Q θ ( s, A ) , ℓ)] and 1 N (cid:80) A ∈ Ξ | Ξ | = N Ψ ( Q θ ( s, A ) , ℓ ) (with Ξ iid ∼ π w ′ ( ·| s ) ) are strictly convex.
Revised: [Ψ( Q θ ( s, A ) , ℓ

Original: Proof.
Revised: Proof.

Original: For λ ∈ [0 , 1] and ℓ 1 , ℓ 2 ∈
Revised: For λ ∈ [0 , 1] and ℓ 1 , ℓ 2 ∈

Original: [ Q l , Q u ] with ℓ 1 ≤ ℓ 2 , we have
Revised: [ Q l , Q u ] with ℓ 1 ≤ ℓ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: We consider how one of these components simplifies
Revised: We consider how one of these components simplifies

Original: [Equation]
Revised: [Equation]

Original: Similarly, we get
Revised: Similarly, we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, for Equation equation 28, we get equation
Revised: Therefore [Equation] for Equation equation [Equation]

Revised: we get

Original: [Equation]
Revised: [Equation]

Original: We can prove the second claim similarly.
Revised: We can prove the second claim similarly.

Original: This completes the proof of Lemma4.
Revised: This completes the proof of Lemma

Revised: [Equation]

Original: B. L EMMA
Revised: B. L EMMA

Original: Lemma 5.
Revised: Lemma 5.

Original: Let { f n ∈ C
Revised: Let { f n ∈ C

Original: (IR , IR) } n ∈ N be a sequence of strictly convex, continuous functions converging uniformly to a strict convex function f .
Revised: (IR , IR) } n ∈ N be a sequence of strictly convex, continuous functions converging uniformly to a strict convex function f .

Original: Let x ∗ n = arg min x f n ( x ) and x ∗ =
Revised: Let x ∗ n = arg min x f n ( x ) and x ∗ =

Original: arg min x ∈ IR f
Revised: arg min x ∈ IR f

Original: ( x ) .
Revised: ( x ) .

Original: Then lim n →∞ x ∗ n = x ∗ .
Revised: Then lim [Equation] n = x ∗ .

Original: Proof.
Revised: Proof.

Original: Let c = lim inf n x ∗ n .
Revised: Let c = lim inf n x ∗ n .

Original: We employ proof by contradiction here.
Revised: We employ proof by contradiction here.

Original: For that, we assume x ∗ > c .
Revised: For that, we assume x ∗ > c .

Original: Now, note that f ( x ∗ ) <
Revised: Now, note that f ( x ∗ ) <

Original: f ( c ) and f ( x ∗ ) <
Revised: f ( c ) and f ( x ∗ ) <

Original: f
Revised: f

Original: (( x ∗ + c ) / 2) (by the definition of x ∗ ).
Revised: (( x ∗ + c ) / 2) (by the definition of x ∗ ).

Original: Also, by the strict convexity of f , we have f (( x ∗ + c ) / 2) <
Revised: Also, by the strict convexity of f , we have f (( x ∗ + c ) / 2) <

Original: ( f ( x ∗ )
Revised: ( f ( x ∗ )

Original: + f ( c )) /
Revised: + f ( c )) /

Original: 2 < f
Revised: 2 < f

Original: ( c ) .
Revised: ( c ) .

Original: Therefore, we have f 
Revised: Therefore, we have f (

Original: [Equation]
Revised: [Equation]

Original:  Let r 1 ∈ IR be such that f ( c ) > r 1 > f
Revised: > [Equation] Let r 1 ∈ IR be such that f ( c ) > r 1 > f

Original: (( x ∗ + c ) / 2) .
Revised: (( x ∗ + c ) / 2) .

Original: Now, since ∥ f n − f ∗ ∥ ∞ → 0 as n → ∞ , there exists an positive integer N s.t.
Revised: Now, since ∥ f n − f ∗ ∥ ∞ → 0 as n → ∞ , there exists an positive integer N s.t.

Original: | f n ( c ) − f ( c ) | < f ( c ) − r 1 , ∀ n ≥ N and ϵ > 0 .
Revised: | f n ( c ) − f ( c ) | < f ( c ) − r 1 , ∀ n ≥ N and ϵ > 0 .

Original: Therefore, f n ( c ) − f ( c ) > r 1 − f ( c ) ⇒ f n ( c ) > r 1 .
Revised: Therefore, f n ( c ) − f ( c ) > r 1 − f ( c ) ⇒ f n ( c ) > r 1 .

Original: Similarily, we can show that f n (( x ∗ + c ) / 2) > r 1 .
Revised: Similarily, we can show that f n (( x ∗ + c ) / 2) > r 1 .

Original: Therefore, we have f n ( c ) >
Revised: Therefore, we have f n ( c ) >

Original: f n (( x ∗ + c ) / 2) .
Revised: f n (( x ∗ + c ) / 2) .

Original: Similarily, we can show that f n (( x ∗ + c ) / 2) >
Revised: Similarily, we can show that f n (( x ∗ + c ) / 2) >

Original: f n ( x ∗ ) .
Revised: f n ( x ∗ ) .

Original: Finally, we obtain
Revised: Finally, we obtain

Original: [Equation]
Revised: [Equation]

Original:  Now, by the extreme value theorem of the continuous functions, we obtain that for n ≥ N , f n achieves minimum (say at x p in the closed interval [ c, ( x ∗ + c ) / 2] .
Revised: [Equation] Now, by the extreme value theorem of the continuous functions, we obtain that for n ≥ N , f n achieves minimum (say at x p in the closed interval [ c, ( x ∗ + c ) / 2] .

Original: Note that f n ( x p ) ≮ f n (( x ∗ + c ) / 2) (if so then f n ( x p ) will be a local minimum of f n since f n ( x ∗ ) <
Revised: Note that f n ( x p ) ≮ f n (( x ∗ + c ) / 2) (if so then f n ( x p ) will be a local minimum of f n since f n ( x ∗ ) <

Original: f n (( x ∗ + c ) / 2) ).
Revised: f n (( x ∗ + c ) / 2) ).

Original: Also, f n ( x p ) ̸ = f n (( x ∗ + c ) / 2) .
Revised: Also, f n ( x p ) ̸ = f n (( x ∗ + c ) / 2) .

Original: Therefore, f n achieves it minimum in the closed interval [ c, ( x ∗ + c ) / 2] at the point ( x ∗ + c ) / 2 .
Revised: Therefore, f n achieves it minimum in the closed interval [ c, ( x ∗ + c ) / 2] at the point ( x ∗ + c ) / 2 .

Original: This further implies that x ∗ n > ( x ∗ + c ) / 2 .
Revised: This further implies that x ∗ n > ( x ∗ + c ) / 2 .

Original: Therefore, lim inf n x ∗ n ≥ ( x ∗ + c ) / 2 ⇒ c ≥ ( x ∗ + c ) / 2 ⇒ c ≥ x ∗ .
Revised: Therefore, lim inf n x ∗ n ≥ ( x ∗ + c ) / 2 ⇒ c ≥ ( x ∗ + c ) / 2 ⇒ c ≥ x ∗ .

Original: This is a contradiction and implies [Equation]
Revised: This is a contradiction and implies lim inf n x ∗ n ≥ x ∗ .

Original: Now consider g n ( x ) =
Revised: Now consider g n ( x ) =

Original: f n ( − x ) .
Revised: f n ( − x ) .

Original: Note that g n is also continuous and strictly convex.
Revised: Note that g n is also continuous and strictly convex.

Original: Indeed, for λ ∈ [0 , 1] , we have g n ( λx 1 +(1 − λ ) x 2 ) =
Revised: Indeed, for λ ∈ [0 , 1] , we have g n ( λx 1 +(1 − λ ) x 2 ) =

Original: f n ( − λx 1 − (1 − λ ) x 2 ) <
Revised: f n ( − λx 1 − (1 − λ ) x 2 ) <

Original: λf ( − x 1 )+(1 − λ ) f ( − x 2 ) =
Revised: λf ( − x 1 )+(1 − λ ) f ( − x 2 ) =

Original: λg ( x 1 ) + (1 − λ ) g ( x 2 ) .
Revised: λg ( x 1 ) + (1 − λ ) g ( x 2 ) .

Original: Applying the result from Eq.
Revised: Applying the result from Eq.

Original: (31) to the sequence { g n } n ∈ N , we obtain that lim inf n ( − x ∗ n ) ≥ − x ∗ .
Revised: (31) to the sequence { g n } n ∈ N , we obtain that lim inf n ( − x ∗ n ) ≥ − x ∗ .

Original: This implies lim sup n x ∗ n ≤ x ∗ .
Revised: This implies lim sup n x ∗ n ≤ x ∗ .

Original: Therefore, lim inf n [Equation]
Revised: Therefore, lim inf n x ∗ n ≥ x ∗ ≥ lim sup n x ∗ n ≥ lim sup n x ∗ n .

Original: Hence, lim inf n x ∗ n = lim sup n x ∗ n = x ∗
Revised: Hence, lim inf n x ∗ n = lim sup n x ∗ n = x ∗

Original: C E XPERIMENTAL D ETAILS C. H YPERPARAMETER D ETAILS
Revised: C E XPERIMENTAL D ETAILS C. H YPERPARAMETER D ETAILS

Original: In this section, we outline the tuned hyperparameters for each algorithm on each environment in our experiments.
Revised: In this section, we outline the tuned hyperparameters for each algorithm on each environment in our experiments.

Original: For each algorithm, hyperparameters were tuned over an initial 10 runs with different random seeds.
Revised: For each algorithm, hyperparameters were tuned over an initial 10 runs with different random seeds.

Original: Each algorithm saw the same 10 initial random seeds.
Revised: Each algorithm saw the same 10 initial random seeds.

Original: For a list of all hyperparameters swept, see Section 5.3.
Revised: For a list of all hyperparameters swept, see Section 5.3.

Original: In Table 1, we list the tuned hyperparameters for each algorithm when tuning across continuous-action environments.
Revised: In Table 1, we list the tuned hyperparameters for each algorithm when tuning across continuous-action environments.

Original: In Table 2, we list the tuned hyperparameters for each algorithm when tuning across discrete-action environments.
Revised: In Table 2, we list the tuned hyperparameters for each algorithm when tuning across discrete-action environments.

Original: In Tables 3, 4, and 5, we list the tuned hyperparameters when tuning per-environment for GreedyAC, VanillaAC, and SAC respectively.
Revised: In Tables 3, 4, and 5, we list the tuned hyperparameters when tuning per-environment for GreedyAC, VanillaAC, and SAC respectively.

Original: Finally [Table] Table 6 outlines the hyperparamters used in the experiments on Swimmer.
Revised: Finally, Table 6 outlines the hyperparamters used in the experiments on Swimmer.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: C. N ORMALIZATION A PPROACH

Original: For each environment, we find the best return achieved by any agent, across all runs, as a simple approximation to a near-optimal return.
Revised: For each environment, we find the best return achieved by any agent, across all runs, as a simple approximation to a near-optimal return.

Original: Table 7 lists these returns for each environment.
Revised: Table 7 lists these returns for each environment.

Original: Then, to obtain a normalized score, we use 1 − BestValue − AlgValue
Revised: Then, to obtain a normalized score, we use 1 − BestValue − AlgValue

Original: | BestValue | , where the numerator is guaranteed to be nonnegative.
Revised: | BestValue | , where the numerator is guaranteed to be nonnegative.

Original: If AlgValue =
Revised: If AlgValue =

Original: BestValue we get the highest value of 1.
Revised: BestValue we get the highest value of 1.

Original: If AlgValue is half of BestValue, we get 0 .
Revised: If AlgValue is half of BestValue, we get 0 .

Original: 5 BestValue | BestValue | = 0 .
Revised: 5 BestValue | BestValue | = 0 .

Original: If AlgValue is significantly worse than BestValue, the score is much lower.
Revised: If AlgValue is significantly worse than BestValue, the score is much lower.

Original: The AlgValue that we normalize is the point depicted on the sensitivity plot.
Revised: The AlgValue that we normalize is the point depicted on the sensitivity plot.

Original: It corresponds to the
Revised: It corresponds to the

Original: Average Return across timesteps and across runs for the algorithm, with that hyperparameter setting in that environment.
Revised: Average Return across timesteps and across runs for the algorithm, with that hyperparameter setting in that environment.

Original: For the experiments in Figure 3, where we tune across the complete set of discrete- or continuousaction environments, we first compute the normalized scores just described.
Revised: For the experiments in Figure 3, where we tune across the complete set of discrete- or continuousaction environments, we first compute the normalized scores just described.

Original: Then, we compute the average normalized scores for each algorithm and hyperparameter setting across discrete-action and continuous-action environments separately.
Revised: Then, we compute the average normalized scores for each algorithm and hyperparameter setting across discrete-action and continuous-action environments separately.

Original: We then choose the hyperparameter setting for each algorithm for the discrete- and continuous-action environments based on the hyperparameter setting which resulted in the highest normalized scores.
Revised: We then choose the hyperparameter setting for each algorithm for the discrete- and continuous-action environments based on the hyperparameter setting which resulted in the highest normalized scores.

Original: The learning curves for these hyperparameters, for each algorithm, are shown in Figure 3.
Revised: The learning curves for these hyperparameters, for each algorithm, are shown in Figure 3.

Original: [Table]
Revised: [Table]

Original: C. S ENSITIVITY P LOTS
Revised: C. S ENSITIVITY P LOTS

Original: We plot parameter sensitivity curves, which include a line for each entropy scale, with the stepsize on the x-axis.
Revised: We plot parameter sensitivity curves, which include a line for each entropy scale, with the stepsize on the x-axis.

Original: Because there are two stepsizes, we have two sets of plots – one for the critic stepsize and one for the actor stepsize.
Revised: Because there are two stepsizes, we have two sets of plots – one for the critic stepsize and one for the actor stepsize.

Original: When examining the sensitivity to the critic stepsize, we select the corresponding best actor stepsize.
Revised: When examining the sensitivity to the critic stepsize, we select the corresponding best actor stepsize.

Original: This means that for each point ( critic stepsize , entropy scale ) =
Revised: This means that for each point ( critic stepsize , entropy scale ) =

Original: ( α, τ ) on the sensitivity plot for the critic stepsize, we find the best actor stepsize and report the performance for that triplet averaged over all 40 runs.
Revised: ( α, τ ) on the sensitivity plot for the critic stepsize, we find the best actor stepsize and report the performance for that triplet averaged over all 40 runs.

Original: We do the same procedure when plotting the actor stepsize on the x-axis [Figure] but maximizing over critic stepsize.
Revised: We do the same procedure when plotting the actor stepsize on the x-axis, but maximizing over critic stepsize

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D A BLATION S TUDY ON SAC
Revised: D A BLATION S TUDY ON SAC

Original: [Figure]
Revised: [Figure]

Original: Modern variants of SAC utilize a trick to automatically adapt the entropy scale hyperparameter during training (Haarnoja et al., 2018b).
Revised: Modern variants of SAC utilize a trick to automatically adapt the entropy scale hyperparameter during training (Haarnoja et al., 2018b).

Original: In order to gauge which variant of SAC to use in this work, we performed an ablation study where we studied SAC with and without automatic entropy tuning.
Revised: In order to gauge which variant of SAC to use in this work, we performed an ablation study where we studied SAC with and without automatic entropy tuning.

Original: We ran SAC with automatic entropy tuning for 10 runs.
Revised: We ran SAC with automatic entropy tuning for 10 runs.

Original: Hyperparameters were swept in the same sets as listed in Section 5.3.
Revised: Hyperparameters were swept in the same sets as listed in Section 5.3.

Original: Additionally, we swept entropy scale step-sizes β
Revised: Additionally, we swept entropy scale step-sizes β

Original: = 10 z for in z ∈ {− 4 , − 3 , − 2 } for automatic entropy tuning.
Revised: = 10 z for in z ∈ {− 4 , − 3 , − 2 } for automatic entropy tuning.

Original: Figure 8 shows the learning curves of SAC with automatic entropy tuning, over 10 runs, and SAC without automatic entropy tuning over the 40 runs conducted for the experiments in the main text.
Revised: Figure 8 shows the learning curves of SAC with automatic entropy tuning, over 10 runs, and SAC without automatic entropy tuning over the 40 runs conducted for the experiments in the main text.

Original: As can be seen in the figure, performing a grid search over the entropy scale hyperparameter never degrades performance compared to using automatic entropy tuning, and in some cases results in better performance than when using automatic entropy tuning.
Revised: As can be seen in the figure, performing a grid search over the entropy scale hyperparameter never degrades performance compared to using automatic entropy tuning, and in some cases results in better performance than when using automatic entropy tuning.

Original: Because of this, we decided to use manual entropy tuning through a grid search in our experiments, which also allows us to characterize the sensitivity of SAC’s performance with respect to the entropy scale hyperparameter.
Revised: Because of this, we decided to use manual entropy tuning through a grid search in our experiments, which also allows us to characterize the sensitivity of SAC’s performance with respect to the entropy scale hyperparameter.