Original: C ONTINUOUS C ONTROL WITH A CTION Q UANTIZATION FROM D EMONSTRATIONS
Revised: C ONTINUOUS C ONTROL WITH A CTION Q UANTIZATION FROM D EMONSTRATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In Reinforcement
Revised: In Reinforcement

Original: Learning (RL), discrete actions, as opposed to continuous actions, result in less complex exploration problems and the immediate computation of the maximum of the action-value function which is central to dynamic programmingbased methods.
Revised: Learning (RL), discrete actions, as opposed to continuous actions, result in less complex exploration problems and the immediate computation of the maximum of the action-value function which is central to dynamic programmingbased methods.

Original: In this paper, we propose a novel method: Action Quantization from Demonstrations (AQuaDem) to learn a discretization of continuous action spaces by leveraging the priors of demonstrations.
Revised: In this paper, we propose a novel method: Action Quantization from Demonstrations (AQuaDem) to learn a discretization of continuous action spaces by leveraging the priors of demonstrations.

Original: This dramatically reduces the exploration problem, since the actions faced by the agent not only are in a ﬁnite number but also are plausible in light of the demonstrator’s behavior.
Revised: This dramatically reduces the exploration problem, since the actions faced by the agent not only are in a ﬁnite number but also are plausible in light of the demonstrator’s behavior.

Original: By discretizing the action space we can apply any discrete action deep RL algorithm to the continuous control problem.
Revised: By discretizing the action space we can apply any discrete action deep RL algorithm to the continuous control problem.

Original: We evaluate the proposed method on three different setups: RL with demonstrations, RL with play data –demonstrations of a human playing in an environment but not solving any speciﬁc task– and Imitation
Revised: We evaluate the proposed method on three different setups: RL with demonstrations, RL with play data –demonstrations of a human playing in an environment but not solving any speciﬁc task– and Imitation

Original: Learning.
Revised: Learning.

Original: For all three setups, we only consider human data, which is more challenging than synthetic data.
Revised: For all three setups, we only consider human data, which is more challenging than synthetic data.

Original: We found that AQuaDem consistently outperforms state-of-the-art continuous control methods, both in terms of performance and sample efﬁciency on a variety of hard manipulation tasks
Revised: We found that AQuaDem consistently outperforms state-of-the-art continuous control methods, both in terms of performance and sample efﬁciency on a variety of hard manipulation tasks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With several successes on highly challenging tasks including strategy games such as Go (Silver et al., 2016), StarCraft (Vinyals et al., 2019) or Dota 2 (Berner et al., 2019) as well as robotic manipulation (Andrychowicz et al., 2020), Reinforcement Learning (RL) holds a tremendous potential for solving sequential decision making problems.
Revised: With several successes on highly challenging tasks including strategy games such as Go (Silver et al., 2016), StarCraft (Vinyals et al., 2019) or Dota 2 (Berner et al., 2019) as well as robotic manipulation (Andrychowicz et al., 2020), Reinforcement Learning (RL) holds a tremendous potential for solving sequential decision making problems.

Original: RL relies on Markov Decision Processes (MDP) (Puterman, 2014) as its cornerstone, a general framework under which vastly different problems can be casted.
Revised: RL relies on Markov Decision Processes (MDP) (Puterman, 2014) as its cornerstone, a general framework under which vastly different problems can be casted.

Original: There is a clear separation in the class of MDPs between the ﬁnite discrete action setup, where an agent faces a ﬁnite number of possible actions, and the continuous action setup, where an agent faces an inﬁnite number of actions.
Revised: There is a clear separation in the class of MDPs between the ﬁnite discrete action setup, where an agent faces a ﬁnite number of possible actions, and the continuous action setup, where an agent faces an inﬁnite number of actions.

Original: When the number of actions is small, the former is arguably simpler, since exploration is more manageable with a ﬁnite, reasonable number of actions, and computing the maximum of the action-value function is straightforward (and implicitly deﬁnes a greedily-improved policy).
Revised: When the number of actions is small, the former is arguably simpler, since exploration is more manageable with a ﬁnite, reasonable number of actions, and computing the maximum of the action-value function is straightforward (and implicitly deﬁnes a greedily-improved policy).

Original: In the continuous action setup, the parametrized policy either directly optimizes the expected value function that is estimated through Monte Carlo rollouts (Williams, 1992), which makes it demanding in interactions with the environment, or tracks the maximum of a parametrized state-action value function (Konda and Tsitsiklis, 2000) hence introducing additional sources of approximations.
Revised: In the continuous action setup, the parametrized policy either directly optimizes the expected value function that is estimated through Monte Carlo rollouts (Williams, 1992), which makes it demanding in interactions with the environment, or tracks the maximum of a parametrized state-action value function (Konda and Tsitsiklis, 2000) hence introducing additional sources of approximations.

Original: Therefore, a workaround consists in turning a continuous control problem into a discrete one.
Revised: Therefore, a workaround consists in turning a continuous control problem into a discrete one.

Original: The simplest approach is to naively ( e.g. uniformly) discretize the action space, an idea which dates back to the “bang-bang” controller (Bushaw, 1952).
Revised: The simplest approach is to naively ( e.g. uniformly) discretize the action space, an idea which dates back to the “bang-bang” controller (Bushaw, 1952).

Original: However, such a discretization scheme suffers from the curse of dimensionality.
Revised: However, such a discretization scheme suffers from the curse of dimensionality.

Original: Various methods have addressed this limitation by making the assumption of independence (Andrychowicz et al., 2020)
Revised: Various methods have addressed this limitation by making the assumption of independence (Andrychowicz et al., 2020)

Original: (strong thus limiting) or of causal dependence (Metz et al., 2017; Vinyals et al., 2019) between the action dimensions, yet, they are typically complex and task-speciﬁc (e.g. autoregressive policies and/or complex architectures like pointer networks).
Revised: (strong thus limiting) or of causal dependence (Metz et al., 2017; Vinyals et al., 2019) between the action dimensions, yet, they are typically complex and task-speciﬁc (e.g. autoregressive policies and/or complex architectures like pointer networks).

Original: In this work, we introduce a novel approach leveraging the prior of human demonstrations for reducing a continuous action spaces to a discrete set of meaningful actions.
Revised: In this work, we introduce a novel approach leveraging the prior of human demonstrations for reducing a continuous action spaces to a discrete set of meaningful actions.

Original: The proposed method does not suffer from the curse of dimensionality and does not require any task speciﬁc assumption.
Revised: The proposed method does not suffer from the curse of dimensionality and does not require any task speciﬁc assumption.

Original: Demonstrations typically consist of transitions experienced by a human in the targeted environment, performing the task at hand or not.
Revised: Demonstrations typically consist of transitions experienced by a human in the targeted environment, performing the task at hand or not.

Original: They are of particular interest in cases where the reward function is hard to deﬁne (Russell, 1998; Ng et al., 1999), to facilitate exploration (Salimans and Chen, 2018;
Revised: They are of particular interest in cases where the reward function is hard to deﬁne (Russell, 1998; Ng et al., 1999), to facilitate exploration (Salimans and Chen, 2018;

Original: Nair et al., 2018) or to build behavioral priors (Singh et al., 2020).
Revised: Nair et al., 2018) or to build behavioral priors (Singh et al., 2020).

Original: We thus propose Action Quantization from Demonstrations, or AQuaDem, a novel paradigm where we learn a state dependent discretization of a continuous action space using demonstrations, enabling the use of discrete-action deep RL methods by virtue of this learned discretization.
Revised: We thus propose Action Quantization from Demonstrations, or AQuaDem, a novel paradigm where we learn a state dependent discretization of a continuous action space using demonstrations, enabling the use of discrete-action deep RL methods by virtue of this learned discretization.

Original: We formalize this paradigm, provide a neural implementation and analyze it through visualizations in simple grid worlds.
Revised: We formalize this paradigm, provide a neural implementation and analyze it through visualizations in simple grid worlds.

Original: We empirically evaluate this discretization strategy on three downstream task setups: Reinforcement
Revised: We empirically evaluate this discretization strategy on three downstream task setups: Reinforcement

Original: Learning with demonstrations, Reinforcement Learning with play data, and Imitation Learning.
Revised: Learning with demonstrations, Reinforcement Learning with play data, and Imitation Learning.

Original: We test the resulting methods on robotics tasks and show that they outperform state-of-the-art continuous control methods both in terms of sample-efﬁciency and performance on every setup.
Revised: We test the resulting methods on robotics tasks and show that they outperform state-of-the-art continuous control methods both in terms of sample-efﬁciency and performance on every setup.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Markov Decision Process.
Revised: Markov Decision Process.

Original: We model the sequential decision making problem as a Markov Decision Process (MDP) (Puterman, 2014; Sutton and Barto, 2018).
Revised: We model the sequential decision making problem as a Markov Decision Process (MDP) (Puterman, 2014; Sutton and Barto, 2018).

Original: An MDP is a tuple ( S , A , P , r, γ, ρ 0 ) , where S is the state space, A is the action space, P is the transition kernel, r is the deterministic reward function, γ the discount factor and ρ 

Revised: An MDP is a tuple ( S , A , P , r, γ, ρ 0 ) , where S is the state space, A is the action space, P is the transition kernel, r is the deterministic reward function, γ the discount factor and ρ

Original: the initial state distribution.
Revised: the initial state distribution.

Original: Throughout the paper, we distinguish discrete action spaces, which simply amount to a set { 1 , . .
Revised: Throughout the paper, we distinguish discrete action spaces, which simply amount to a set { 1 , . .

Original: , K } , from continuous action spaces which consist in an interval of R d where d is the dimensionality of the action space.
Revised: , K } , from continuous action spaces which consist in an interval of R d where d is the dimensionality of the action space.

Original: A stationary policy π is a mapping from states to distributions over actions.
Revised: A stationary policy π is a mapping from states to distributions over actions.

Original: The value function V π of a policy π is deﬁned as the expected discounted cumulative reward from starting in a particular state and acting according to π : V π ( s ) =
Revised: The value function V π of a policy π is deﬁned as the expected discounted cumulative reward from starting in a particular state and acting according to π : V π ( s ) =

Original: E (cid:2) (cid:80)
Revised: E (cid:2) (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ( s t ) , s t +1 ∼ P ( s t , a t )) (cid:3) .
Revised: ( s t ) , s t +1 ∼ P ( s t , a t )) (cid:3) .

Original: An optimal policy π ∗ maximizes the value function V π ∗ for all states.
Revised: An optimal policy π ∗ maximizes the value function V π ∗ for all states.

Original: The action-value function Q π is deﬁned as the expected discounted cumulative reward from starting in a particular state, taking an action and then acting according to π : Q π ( s, a ) =
Revised: The action-value function Q π is deﬁned as the expected discounted cumulative reward from starting in a particular state, taking an action and then acting according to π : Q π ( s, a ) =

Original: r ( s, a ) + γ E (cid:2)
Revised: r ( s, a ) + γ E (cid:2)

Original: V π ( s (cid:48) ) | s (cid:48) ∼ P ( s, a ) (cid:3) .
Revised: V π ( s (cid:48) ) | s (cid:48) ∼ P ( s, a ) (cid:3) .

Original: Value-based RL.
Revised: Value-based RL.

Original: The Bellman (1957) operator T connects an action-value function Q for the state-action pair ( s, a ) to the action-value function in the subsequent state s (cid:48) :
Revised: The Bellman (1957) operator T connects an action-value function Q for the state-action pair ( s, a ) to the action-value function in the subsequent state s (cid:48) :

Original: T π ( Q )( s, a ) :
Revised: T π ( Q )( s, a ) :

Original: = r ( s, a ) + γ E (cid:2) Q ( s (cid:48) , a (cid:48) ) | a (cid:48) ∼ π ( s ) , s (cid:48) ∼ P ( s, a ) (cid:3) .
Revised: = r ( s, a ) + γ E (cid:2) Q ( s (cid:48) , a (cid:48) ) | a (cid:48) ∼ π ( s ) , s (cid:48) ∼ P ( s, a ) (cid:3) .

Original: Value Iteration (VI) (Bertsekas, 2000) is the basis for methods using the Bellman equation to derive algorithms estimating the optimal policy π ∗ .
Revised: Value Iteration (VI) (Bertsekas, 2000) is the basis for methods using the Bellman equation to derive algorithms estimating the optimal policy π ∗ .

Original: The prototypical example is the Q -learning algorithm
Revised: The prototypical example is the Q -learning algorithm

Original: (Watkins and Dayan, 1992), which is the basis of e.g. DQN (Mnih et al., 2015), and consists in the repeated application of a stochastic approximation of the Bellman operator Q ( s, a ) :
Revised: (Watkins and Dayan, 1992), which is the basis of e.g. DQN (Mnih et al., 2015), and consists in the repeated application of a stochastic approximation of the Bellman operator Q ( s, a ) :

Original: = r ( s, a ) +
Revised: = r ( s, a ) +

Original: γ max a (cid:48) Q ( s (cid:48) , a (cid:48) ) , where ( s, a, s (cid:48) ) is a transition sampled from the MDP.
Revised: γ max a (cid:48) Q ( s (cid:48) , a (cid:48) ) , where ( s, a, s (cid:48) ) is a transition sampled from the MDP.

Original: The Q -learning algorithm exempliﬁes two desirable traits of VI-inspired methods in discrete action spaces that are 1) bootstrapping: the current Q -value estimate at the next state s (cid:48) is used to compute a ﬁner estimate of the Q -value at state s , and 2) the exact derivation of the maximum
Revised: The Q -learning algorithm exempliﬁes two desirable traits of VI-inspired methods in discrete action spaces that are 1) bootstrapping: the current Q -value estimate at the next state s (cid:48) is used to compute a ﬁner estimate of the Q -value at state s , and 2) the exact derivation of the maximum

Original: Q -value at a given state.
Revised: Q -value at a given state.

Original: For continuous action spaces, state-of-the-art methods (Haarnoja et al., 2018; Fujimoto et al., 2018) are also fundamentally close to a VI scheme, as they rely on Bellman consistency, with the difference being that the argument maximizing the Q -value, in other words the parametrized policy, is approximate.
Revised: For continuous action spaces, state-of-the-art methods (Haarnoja et al., 2018; Fujimoto et al., 2018) are also fundamentally close to a VI scheme, as they rely on Bellman consistency, with the difference being that the argument maximizing the Q -value, in other words the parametrized policy, is approximate.

Original: Demonstration data.
Revised: Demonstration data.

Original: Additional data consisting of transitions from an agent may be available.
Revised: Additional data consisting of transitions from an agent may be available.

Original: These demonstrations may contain the reward information or not.
Revised: These demonstrations may contain the reward information or not.

Original: In the context of Imitation Learning (Pomerleau, 1991; Ng et al., 1999; 2000; Ziebart et al., 2008), the assumption is that the agent generating the demonstration data is near-optimal and that demonstration rewards are not provided.
Revised: In the context of Imitation Learning (Pomerleau, 1991; Ng et al., 1999; 2000; Ziebart et al., 2008), the assumption is that the agent generating the demonstration data is near-optimal and that demonstration rewards are not provided.

Original: The objective is then to match the distribution of the agent with the one of the expert.
Revised: The objective is then to match the distribution of the agent with the one of the expert.

Original: In the context of Reinforcement Learning with demonstrations (RLfD) (Hester et al., 2018; Vecerik et al., 2017), demonstration rewards are provided.
Revised: In the context of Reinforcement Learning with demonstrations (RLfD) (Hester et al., 2018; Vecerik et al., 2017), demonstration rewards are provided.

Original: They are typically used in the form of auxiliary objectives together with a standard learning agent whose goal is to maximize the environment reward.
Revised: They are typically used in the form of auxiliary objectives together with a standard learning agent whose goal is to maximize the environment reward.

Original: In the context of Reinforcement Learning with play (Lynch et al., 2020), demonstration rewards are not provided as play data is typically not task-speciﬁc.
Revised: In the context of Reinforcement Learning with play (Lynch et al., 2020), demonstration rewards are not provided as play data is typically not task-speciﬁc.

Original: Demonstration data can come from various sources, although a common assumption is that it is generated by a single, unimodal Markovian policy.
Revised: Demonstration data can come from various sources, although a common assumption is that it is generated by a single, unimodal Markovian policy.

Original: However, most of available data comes from agents that do not fulﬁll this condition.
Revised: However, most of available data comes from agents that do not fulﬁll this condition.

Original: In particular, for human data, and even more so when coming from several individuals, the behavior generating the episodes may not be unimodal nor Markovian.
Revised: In particular, for human data, and even more so when coming from several individuals, the behavior generating the episodes may not be unimodal nor Markovian.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we introduce the AQuaDem framework and a practical neural network implementation together with an accompanying objective function.
Revised: we introduce the AQuaDem framework and a practical neural network implementation together with an accompanying objective function.

Original: We provide a series of visualizations to study the candidate actions learned with AQuaDem in gridworld experiments
Revised: We provide a series of visualizations to study the candidate actions learned with AQuaDem in gridworld experiments

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: +k4AD146HON6SZ5g8IZlzyxaFWsKvEjsjBRRhppnfjndiCQBDRXhIGXHtmLlpiAUI5yOC04iaQxkCH3a0TSEgEo3nT4yxsda6eJeJHSFCk/V3xMpBFKOAl93BqAGct6biP95nUT1Lt2UhXGiaEhmi3oJxyrCk1RwlwlKFB9pAkQwfSsmAxBAlM6uoEOw519eJM3Tin1esepnxep1FkceHaAjVEI2ukBVdItqqIEIekTP6BW9GU/Gi/FufMxac0Y2s4/+wPj8Ae6fmAI=</latexit> ⇡ ( s ) =
Revised: +k4AD146HON6SZ5g8IZlzyxaFWsKvEjsjBRRhppnfjndiCQBDRXhIGXHtmLlpiAUI5yOC04iaQxkCH3a0TSEgEo3nT4yxsda6eJeJHSFCk/V3xMpBFKOAl93BqAGct6biP95nUT1Lt2UhXGiaEhmi3oJxyrCk1RwlwlKFB9pAkQwfSsmAxBAlM6uoEOw519eJM3Tin1esepnxep1FkceHaAjVEI2ukBVdItqqIEIekTP6BW9GU/Gi/FufMxac0Y2s4/+wPj8Ae6fmAI=</latexit> ⇡ ( s ) =

Original: argmax k Q ( s,a k )
Revised: argmax k Q ( s,a k )

Original: Visualization of the AQuaDem framework (ofﬂine) with a downstream algorithm (online).
Revised: Visualization of the AQuaDem framework (ofﬂine) with a downstream algorithm (online).

Original: 3. AQ UA D EM : A CTION Q UANTIZATION FROM D EMONSTRATIONS
Revised: 3. AQ UA D EM : A CTION Q UANTIZATION FROM D EMONSTRATIONS

Original: Our objective is to reduce a continuous control problem to a discrete action one, on which we can apply discrete-action RL methods.
Revised: Our objective is to reduce a continuous control problem to a discrete action one, on which we can apply discrete-action RL methods.

Original: Using demonstrations, we wish to assign to each state s ∈ S a set of K candidate actions from A .
Revised: Using demonstrations, we wish to assign to each state s ∈ S a set of K candidate actions from A .

Original: The resulting action space is therefore a discrete ﬁnite set of K state-conditioned vectors.
Revised: The resulting action space is therefore a discrete ﬁnite set of K state-conditioned vectors.

Original: In a given state s ∈ S , picking action k ∈ { 1 , . .
Revised: In a given state s ∈ S , picking action k ∈ { 1 , . .

Original: , K } stands for picking the k th candidate action for that particular state.
Revised: , K } stands for picking the k th candidate action for that particular state.

Original: The AQuaDem framework refers to the discretization of the action space, and the resulting discrete action algorithms used with AQuaDem on continuous control tasks are detailed in Section 4.
Revised: The AQuaDem framework refers to the discretization of the action space, and the resulting discrete action algorithms used with AQuaDem on continuous control tasks are detailed in Section 4.

Original: We propose to learn the discrete action space through a modiﬁed version of the Behavioral Cloning (BC)
Revised: We propose to learn the discrete action space through a modiﬁed version of the Behavioral Cloning (BC)

Original: (Pomerleau, 1991) reconstruction loss that captures the multimodality of demonstrations.
Revised: (Pomerleau, 1991) reconstruction loss that captures the multimodality of demonstrations.

Original: Indeed the typical BC implementation consists in building a deterministic mapping between states and actions Φ : S (cid:55)→ A .
Revised: Indeed the typical BC implementation consists in building a deterministic mapping between states and actions Φ : S (cid:55)→ A .

Original: But in practice, and in particular when the demonstrator is human, the demonstrator can take multiple actions in a given state (we say that its behavior is multimodal ) which are all good candidates for AQuaDem.
Revised: But in practice, and in particular when the demonstrator is human, the demonstrator can take multiple actions in a given state (we say that its behavior is multimodal ) which are all good candidates for AQuaDem.

Original: We thus learn a mapping Ψ : S (cid:55)→
Revised: We thus learn a mapping Ψ : S (cid:55)→

Original: A K from states to a set of K candidate actions and optimize a reconstruction loss based on a soft minimum between the candidate actions and the demonstrated action.
Revised: A K from states to a set of K candidate actions and optimize a reconstruction loss based on a soft minimum between the candidate actions and the demonstrated action.

Original: Suppose we have a dataset of expert demonstrations D
Revised: Suppose we have a dataset of expert demonstrations D

Original: = { ( s i , a i ) } 1: n .
Revised: = { ( s i , a i ) } 1: n .

Original: In the continuous action setting, the vanilla BC approach consists in ﬁnding a parametrized function f Φ that minimizes the reconstruction error between predicted actions and actions in the dataset D .
Revised: In the continuous action setting, the vanilla BC approach consists in ﬁnding a parametrized function f Φ that minimizes the reconstruction error between predicted actions and actions in the dataset D .

Original: To ease notations, we will conﬂate the function f
Revised: To ease notations, we will conﬂate the function f

Original: Φ with its parameters Φ and simply note it Φ : S (cid:55)→ A .
Revised: Φ with its parameters Φ and simply note it Φ : S (cid:55)→ A .

Original: The objective is thus to minimize: min Φ E s,a ∼D (cid:107) Φ
Revised: The objective is thus to minimize: min Φ E s,a ∼D (cid:107) Φ

Original: [Equation]
Revised: [Equation]

Original: s ) − a
Revised: s ) − a

Original: [Equation]
Revised: [Equation]

Original: Instead, we propose to learn a set of K actions
Revised: Instead, we propose to learn a set of K actions

Original: Ψ k
Revised: Ψ k

Original: ( s ) for each state by minimizing the following loss:
Revised: ( s ) for each state by minimizing the following loss:

Original: [Equation]
Revised: [Equation]

Original: where the temperature T is a hyperparameter.
Revised: where the temperature T is a hyperparameter.

Original: Equation (1) corresponds to minimizing a softminimum between the candidates actions Ψ 1 ( s ) , . .
Revised: Equation (1) corresponds to minimizing a softminimum between the candidates actions Ψ 1 ( s ) , . .

Original: , Ψ K ( s ) and the demonstrated action a .
Revised: , Ψ K ( s ) and the demonstrated action a .

Original: Note that with K = 1 , this is exactly the BC loss.
Revised: Note that with K = 1 , this is exactly the BC loss.

Original: The larger the temperature T is, the more the loss imposes all candidate actions to be close to the demonstrated action a thus reducing to the BC loss.
Revised: The larger the temperature T is, the more the loss imposes all candidate actions to be close to the demonstrated action a thus reducing to the BC loss.

Original: The lower the temperature T is, the more the loss only imposes a single candidate action to be close to the demonstrated action a .
Revised: The lower the temperature T is, the more the loss only imposes a single candidate action to be close to the demonstrated action a .

Original: We provide empirical evidence of this phenomenon in Section 3.2.
Revised: We provide empirical evidence of this phenomenon in Section 3.2.

Original: Equation (1) is also interpretable in the context of Gaussian mixture models (see Appendix A).
Revised: Equation (1) is also interpretable in the context of Gaussian mixture models (see Appendix A).

Original: The Ψ function enables us to deﬁne a new MDP where the continuous action space is replaced by a discrete action space of size K corresponding to the K action candidates returned by Ψ at each state.
Revised: The Ψ function enables us to deﬁne a new MDP where the continuous action space is replaced by a discrete action space of size K corresponding to the K action candidates returned by Ψ at each state.

Original: 3. V ISUALIZATION
Revised: 3. V ISUALIZATION

Original: In this section, we analyze the actions learned through the AQuaDem framework, in a toy grid world environment.
Revised: In this section, we analyze the actions learned through the AQuaDem framework, in a toy grid world environment.

Original: We introduce a continuous action grid world with demonstrations in Figure 1.
Revised: We introduce a continuous action grid world with demonstrations in Figure 1.

Original: We deﬁne a neural network Ψ and optimize its parameters by minimizing the objective function deﬁned in Equation (1) (implementation details can be found in Appendix D.1).
Revised: We deﬁne a neural network Ψ and optimize its parameters by minimizing the objective function deﬁned in Equation (1) (implementation details can be found in Appendix D.1).

Original: We display the resulting candidate actions across the state space in Figure 2.
Revised: We display the resulting candidate actions across the state space in Figure 2.

Original: As each color of the arrows depicts a single head of the Ψ network, we observe that the action candidates are smooth : action candidates slowly vary as the state vary, which prevents to have inconsistent action candidates in nearby states.
Revised: As each color of the arrows depicts a single head of the Ψ network, we observe that the action candidates are smooth : action candidates slowly vary as the state vary, which prevents to have inconsistent action candidates in nearby states.

Original: Note that BC actions tend to be diagonal even in the bottom left part of the action space, where the demonstrator only takes horizontal or vertical actions.
Revised: Note that BC actions tend to be diagonal even in the bottom left part of the action space, where the demonstrator only takes horizontal or vertical actions.

Original: On the contrary, the action candidates learned by AQuaDem include the actions taken by the demonstrator conditioned on the states.
Revised: On the contrary, the action candidates learned by AQuaDem include the actions taken by the demonstrator conditioned on the states.

Original: Remark that in the case of K = 2 , the action right is learned independently of the state position (middle plot in
Revised: Remark that in the case of K = 2 , the action right is learned independently of the state position (middle plot in

Original: Figure 2) although it is only executed in a subspace of the action space.
Revised: Figure 2) although it is only executed in a subspace of the action space.

Original: In the case of K = 3 , actions are completely state-independent.
Revised: In the case of K = 3 , actions are completely state-independent.

Original: In non-trivial tasks, the state dependence induced by the AQuaDem framework is essential, as we show in the ablation study in Appendix C and in the analysis of the actions learned in a more realistic setup in Appendix B.
Revised: In non-trivial tasks, the state dependence induced by the AQuaDem framework is essential, as we show in the ablation study in Appendix C and in the analysis of the actions learned in a more realistic setup in Appendix B.

Original: Inﬂuence of the temperature.
Revised: Inﬂuence of the temperature.

Original: The temperature controls the degree of smoothness of the softminimum deﬁned in Equation (1).
Revised: The temperature controls the degree of smoothness of the softminimum deﬁned in Equation (1).

Original: We show that with larger temperatures, the soft-minimum converges to the average which is well represented in Figure 3 rightmost plot where the proﬁle of
Revised: We show that with larger temperatures, the soft-minimum converges to the average which is well represented in Figure 3 rightmost plot where the proﬁle of

Original: AQuaDem’s action candidates conﬂate with actions learned by BC.
Revised: AQuaDem’s action candidates conﬂate with actions learned by BC.

Original: With lower temperatures, the actions taken by the demonstrator are recovered, but if the temperature is too low ( T = 0 . 001 ), some actions that are not taken by the demonstrator might appear as candidates (blue arrows in the leftmost ﬁgure).
Revised: With lower temperatures, the actions taken by the demonstrator are recovered, but if the temperature is too low ( T = 0 . 001 ), some actions that are not taken by the demonstrator might appear as candidates (blue arrows in the leftmost ﬁgure).

Original: This occurs because the soft minimum converges to a hard minimum with lower temperatures meaning that as long as one candidate is close enough to the demonstrated action, the other candidates can be arbitrarily far off.
Revised: This occurs because the soft minimum converges to a hard minimum with lower temperatures meaning that as long as one candidate is close enough to the demonstrated action, the other candidates can be arbitrarily far off.

Original: In this work, we treat the temperature as a hyperparameter, although a natural direction for future work is to aggregate actions learned for different temperatures.
Revised: In this work, we treat the temperature as a hyperparameter, although a natural direction for future work is to aggregate actions learned for different temperatures.

Original: 3. D ISCUSSION
Revised: 3. D ISCUSSION

Original: On losing the optimal policy.
Revised: On losing the optimal policy.

Original: In any form of discretization scheme, the resulting class of policies might not include the optimal policy of the original MDP.
Revised: In any form of discretization scheme, the resulting class of policies might not include the optimal policy of the original MDP.

Original: In the case of AQuaDem, this mainly depends on the quality of the demonstrations.
Revised: In the case of AQuaDem, this mainly depends on the quality of the demonstrations.

Original: For standard continuous control methods, the parametrization of the policy also constrains the space of possible policies, potentially not including the optimal one.
Revised: For standard continuous control methods, the parametrization of the policy also constrains the space of possible policies, potentially not including the optimal one.

Original: This is a lesser problem since policies tend to be represented with functions with universal approximation capabilities.
Revised: This is a lesser problem since policies tend to be represented with functions with universal approximation capabilities.

Original: Nevertheless, for most continuous control methods, the policy improvement step is approximate, while in the case of AQuaDem it is exact, since it amounts to selecting the argmax of the Q -values.
Revised: Nevertheless, for most continuous control methods, the policy improvement step is approximate, while in the case of AQuaDem it is exact, since it amounts to selecting the argmax of the Q -values.

Original: On the multimodality of demonstrations.
Revised: On the multimodality of demonstrations.

Original: The multimodality of demonstrations enables us to deﬁne multiple plausible actions for the agent to take in a given state, guided by the priors of the demonstrations.
Revised: The multimodality of demonstrations enables us to deﬁne multiple plausible actions for the agent to take in a given state, guided by the priors of the demonstrations.

Original: We argue that the assumption of multimodality of the demonstrator should actually be systematic (Mandlekar et al., 2021).
Revised: We argue that the assumption of multimodality of the demonstrator should actually be systematic (Mandlekar et al., 2021).

Original: Indeed, the demonstrator can be e.g. non-Markovian, optimizing for something different than a reward function like curiosity (Barto et al., 2013), or they can be in the process of learning how to interact with the environment.
Revised: Indeed, the demonstrator can be e.g. non-Markovian, optimizing for something different than a reward function like curiosity (Barto et al., 2013), or they can be in the process of learning how to interact with the environment.

Original: When demonstrations are gathered from multiple demonstrators, this naturally leads to multiple modalities in the demonstrations.
Revised: When demonstrations are gathered from multiple demonstrators, this naturally leads to multiple modalities in the demonstrations.

Original: And even in the case where the demonstrator is optimal, multiple actions might be equally good ( e.g. in navigation tasks).
Revised: And even in the case where the demonstrator is optimal, multiple actions might be equally good ( e.g. in navigation tasks).

Original: Finally, the demonstrator can interact with an environment without any task speciﬁc intent, which we refer to as play (Lynch et al., 2020) and also induces a multimodal behavior.
Revised: Finally, the demonstrator can interact with an environment without any task speciﬁc intent, which we refer to as play (Lynch et al., 2020) and also induces a multimodal behavior.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the AQuaDem framework on three different downstream tasks setups: RL with demonstrations, RL with play data and Imitation Learning.
Revised: In this section, we evaluate the AQuaDem framework on three different downstream tasks setups: RL with demonstrations, RL with play data and Imitation Learning.

Original: For all experiments, we detail the networks architectures, hyperparameters search, and training procedures in the Appendix.
Revised: For all experiments, we detail the networks architectures, hyperparameters search, and training procedures in the Appendix.

Original: 4. R EINFORCEMENT L EARNING WITH DEMONSTRATIONS
Revised: 4. R EINFORCEMENT L EARNING WITH DEMONSTRATIONS

Original: Setup.
Revised: Setup.

Original: In the Reinforcement Learning with demonstrations setup (RLfD), the environment of interest comes with a reward function and demonstrations (which include the reward), and the goal is to learn a policy that maximizes the expected return.
Revised: In the Reinforcement Learning with demonstrations setup (RLfD), the environment of interest comes with a reward function and demonstrations (which include the reward), and the goal is to learn a policy that maximizes the expected return.

Original: This setup is particularly interesting for sparse reward tasks, where the reward function is easy to deﬁne (say reaching a goal state) and where RL methods typically fail because the exploration problem is too hard.
Revised: This setup is particularly interesting for sparse reward tasks, where the reward function is easy to deﬁne (say reaching a goal state) and where RL methods typically fail because the exploration problem is too hard.

Original: We consider the Adroit tasks (Rajeswaran et al., 2017) represented in Figure 9, for which human demonstrations are available ( episodes acquired using a virtual reality system).
Revised: We consider the Adroit tasks (Rajeswaran et al., 2017) represented in Figure 9, for which human demonstrations are available ( episodes acquired using a virtual reality system).

Original: These environments come with a dense reward function that we replace with the following sparse reward : 1 if the goal is achieved, 0 otherwise.
Revised: These environments come with a dense reward function that we replace with the following sparse reward : 1 if the goal is achieved, 0 otherwise.

Original: Algorithm & baselines.
Revised: Algorithm & baselines.

Original: The algorithm we propose is a two-fold training procedure: 1) we learn a discretization of the action space in a fully ofﬂine fashion using the AQuaDem framework from human demonstrations; 2) we train a discrete action deep RL algorithm on top of this this discretization.
Revised: The algorithm we propose is a two-fold training procedure: 1) we learn a discretization of the action space in a fully ofﬂine fashion using the AQuaDem framework from human demonstrations; 2) we train a discrete action deep RL algorithm on top of this this discretization.

Original: We refer to this algorithm as AQuaDQN.
Revised: We refer to this algorithm as AQuaDQN.

Original: The RL algorithm considered is Munchausen DQN (Vieillard et al., 2020) as it is the state of the art on the Atari benchmark (Bellemare et al., 2013)
Revised: The RL algorithm considered is Munchausen DQN (Vieillard et al., 2020) as it is the state of the art on the Atari benchmark (Bellemare et al., 2013)

Original: (although we use the non-distributional version of it which simply amounts to DQN (Mnih et al., 2015) with a regularization term).
Revised: (although we use the non-distributional version of it which simply amounts to DQN (Mnih et al., 2015) with a regularization term).

Original: To make as much use of the demonstrations as possible, we maintain two replay buffers: one containing interactions with the environment, the other containing the demonstrations that we sample using a ﬁxed ratio similarly to DQfD (Hester et al., 2018), although we do not use the additional recipes of DQfD (multiple n -step evaluation of the bootstrapped estimate of Q ,
Revised: To make as much use of the demonstrations as possible, we maintain two replay buffers: one containing interactions with the environment, the other containing the demonstrations that we sample using a ﬁxed ratio similarly to DQfD (Hester et al., 2018), although we do not use the additional recipes of DQfD (multiple n -step evaluation of the bootstrapped estimate of Q ,

Original: BC regularization term) for the sake of simplicity.
Revised: BC regularization term) for the sake of simplicity.

Original: When sampling demonstrations, the actions are discretized by taking the closest AQuaDem action candidate (using the Euclidean norm).
Revised: When sampling demonstrations, the actions are discretized by taking the closest AQuaDem action candidate (using the Euclidean norm).

Original: We consider
Revised: We consider

Original: SAC and SAC from demonstrations (SACfD) –a modiﬁed version of SAC where demonstrations are added to the replay buffer (Vecerik et al., 2017)– as baselines against the proposed method.
Revised: SAC and SAC from demonstrations (SACfD) –a modiﬁed version of SAC where demonstrations are added to the replay buffer (Vecerik et al., 2017)– as baselines against the proposed method.

Original: We do not include naive discretization baselines here, as the dimension of the action space is at least 24, which would lead to a 2 24 (cid:39) 16 M actions with a binary discretization scheme.
Revised: We do not include naive discretization baselines here, as the dimension of the action space is at least 24, which would lead to a 2 24 (cid:39) 16 M actions with a binary discretization scheme.

Original: Evaluation & results.
Revised: Evaluation & results.

Original: We train the different methods on 1 M environment interactions on 10 seeds for the chosen hyperparameters (a single set of hyperameters for all tasks) and evaluate the agents every 50 k environment interactions (without exploration noise) onepisodes.
Revised: We train the different methods on 1 M environment interactions on 10 seeds for the chosen hyperparameters (a single set of hyperameters for all tasks) and evaluate the agents every 50 k environment interactions (without exploration noise) onepisodes.

Original: An episode is considered a success if the goal is achieved during the episode.
Revised: An episode is considered a success if the goal is achieved during the episode.

Original: The AQuaDem discretization is trained ofﬂine using 50 k gradient steps on batches of size 256 .
Revised: The AQuaDem discretization is trained ofﬂine using 50 k gradient steps on batches of size 256 .

Original: The number of actions considered were 10 , 15 , 20 and we found 10 to be performing the best.
Revised: The number of actions considered were 10 , 15 , 20 and we found 10 to be performing the best.

Original: Figure 4 shows the AQuaDem loss through the training procedure of the discretization step, and the Figure 5 shows the returns of the trained agents as well as their success rate.
Revised: Figure 4 shows the AQuaDem loss through the training procedure of the discretization step, and the Figure 5 shows the returns of the trained agents as well as their success rate.

Original: On Door, Pen, and
Revised: On Door, Pen, and

Original: Hammer, the AQuaDQN agent reaches high success rate, largely outperforming SACfD in terms of success and sample efﬁciency.
Revised: Hammer, the AQuaDQN agent reaches high success rate, largely outperforming SACfD in terms of success and sample efﬁciency.

Original: On Relocate, all methods reach poor results (although AQuaDQN slightly outperforms the baselines).
Revised: On Relocate, all methods reach poor results (although AQuaDQN slightly outperforms the baselines).

Original: The task requires a larger degree of generalisation than the other three since the goal state and the initial ball position are changing at each episode.
Revised: The task requires a larger degree of generalisation than the other three since the goal state and the initial ball position are changing at each episode.

Original: We show in Figure 6 that when tuned uniquely on the Relocate environment and with more environment interactions, AQuaDQN manages to reach a 50% success rate where other methods still fail.
Revised: We show in Figure 6 that when tuned uniquely on the Relocate environment and with more environment interactions, AQuaDQN manages to reach a 50% success rate where other methods still fail.

Original: Notice that on the Door environment, the SAC and
Revised: Notice that on the Door environment, the SAC and

Original: SACfD agents outperform the AQuaDQN agent in terms of ﬁnal return (but not in term of success rate).
Revised: SACfD agents outperform the AQuaDQN agent in terms of ﬁnal return (but not in term of success rate).

Original: The behavior of these agents are however different from the demonstrator since they consist in slapping the handle and abruptly pulling it back.
Revised: The behavior of these agents are however different from the demonstrator since they consist in slapping the handle and abruptly pulling it back.

Original: We provide videos of all resulting agents (one episode for each seed which is not cherry picked) to demonstrate that AQuaDQN consistently learns a behavior that is qualitatively closer to the demonstrator.
Revised: We provide videos of all resulting agents (one episode for each seed which is not cherry picked) to demonstrate that AQuaDQN consistently learns a behavior that is qualitatively closer to the demonstrator.

Original: baselines when all are tuned on the Relocate environment.
Revised: baselines when all are tuned on the Relocate environment.

Original: We represent the median performance in terms of success rate as well as the interquartile range over 10 seeds.
Revised: We represent the median performance in terms of success rate as well as the interquartile range over 10 seeds.

Original: 4. I MITATION L EARNING
Revised: 4. I MITATION L EARNING

Original: Setup.
Revised: Setup.

Original: In Imitation Learning, the task is not speciﬁed by the reward function but by the demonstrations themselves.
Revised: In Imitation Learning, the task is not speciﬁed by the reward function but by the demonstrations themselves.

Original: The goal is to mimic the demonstrated behavior.
Revised: The goal is to mimic the demonstrated behavior.

Original: There is no reward function and the notion of success is ill-deﬁned (Hussenot et al., 2021).
Revised: There is no reward function and the notion of success is ill-deﬁned (Hussenot et al., 2021).

Original: A number of existing works (Ho and
Revised: A number of existing works (Ho and

Original: Ermon, 2016; Ghasemipour et al., 2019; Dadashi et al., 2021) cast the problem into matching the state distributions of the agent and of the expert.
Revised: Ermon, 2016; Ghasemipour et al., 2019; Dadashi et al., 2021) cast the problem into matching the state distributions of the agent and of the expert.

Original: Imitation Learning is of particular interest when designing a satisfying reward function –one that would lead the desired behavior to be the only optimal policy– is harder than directly demonstrating this behavior.
Revised: Imitation Learning is of particular interest when designing a satisfying reward function –one that would lead the desired behavior to be the only optimal policy– is harder than directly demonstrating this behavior.

Original: In this setup, there is no reward provided, not in the environment interactions nor in the demonstrations.
Revised: In this setup, there is no reward provided, not in the environment interactions nor in the demonstrations.

Original: We again consider the Adroit environments and the human demonstrations which consist of 25 episodes acquired via a virtual reality system.
Revised: We again consider the Adroit environments and the human demonstrations which consist of 25 episodes acquired via a virtual reality system.

Original: Algorithm & baselines.
Revised: Algorithm & baselines.

Original: Again, the algorithm we propose has two stages.
Revised: Again, the algorithm we propose has two stages.

Original: 1) We learn –fully ofﬂine– a discretization of the action space using AQuaDem.
Revised: 1) We learn –fully ofﬂine– a discretization of the action space using AQuaDem.

Original: We train a discrete action version of the GAIL algorithm (Ho and Ermon, 2016) in the discretized environment.
Revised: We train a discrete action version of the GAIL algorithm (Ho and Ermon, 2016) in the discretized environment.

Original: More precisely, we interleave the training of a discriminator between demonstrations and agent experiences, and the training of a Munchausen DQN agent that maximizes the confusion of this discriminator.
Revised: More precisely, we interleave the training of a discriminator between demonstrations and agent experiences, and the training of a Munchausen DQN agent that maximizes the confusion of this discriminator.

Original: The
Revised: The

Original: Munchausen DQN takes one of the candidates actions given by AQuaDem.
Revised: Munchausen DQN takes one of the candidates actions given by AQuaDem.

Original: We call this algorithm
Revised: We call this algorithm

Original: AQuaGAIL.
Revised: AQuaGAIL.

Original: As a baseline, we consider the GAIL algorithm with a SAC (Haarnoja et al., 2018) agent directly maximizing the confusion of the discriminator.
Revised: As a baseline, we consider the GAIL algorithm with a SAC (Haarnoja et al., 2018) agent directly maximizing the confusion of the discriminator.

Original: This results in a very similar algorithm as the one proposed by Kostrikov et al.
Revised: This results in a very similar algorithm as the one proposed by Kostrikov et al.

Original: We also include the results of BC (Pomerleau, 1991).
Revised: We also include the results of BC (Pomerleau, 1991).

Original: Evaluation & results.
Revised: Evaluation & results.

Original: We train AQuaGAIL and GAIL for 1M environment interactions onseeds for the selected hyperparameters (a single set for all tasks).
Revised: We train AQuaGAIL and GAIL for 1M environment interactions onseeds for the selected hyperparameters (a single set for all tasks).

Original: BC is trained for 60k gradient steps with batch size 256 .
Revised: BC is trained for 60k gradient steps with batch size 256 .

Original: We evaluate the agents every 50 k environment steps during training (without exploration noise) on 30 episodes.
Revised: We evaluate the agents every 50 k environment steps during training (without exploration noise) on 30 episodes.

Original: The AQuaDem discretization is trained ofﬂine using 50 k gradient steps on batches of size 256 .
Revised: The AQuaDem discretization is trained ofﬂine using 50 k gradient steps on batches of size 256 .

Original: The results are provided in Figure 7.
Revised: The results are provided in Figure 7.

Original: Evaluating imitation learning algorithms has to be done carefully as the goal to “mimic a behavior” is ill-deﬁned.
Revised: Evaluating imitation learning algorithms has to be done carefully as the goal to “mimic a behavior” is ill-deﬁned.

Original: Here, we provide the results according to two metrics.
Revised: Here, we provide the results according to two metrics.

Original: On top, the success rate is deﬁned in Section 4.1.
Revised: On top, the success rate is deﬁned in Section 4.1.

Original: Notice that the human demonstrations do not have a success score of 1 on every task.
Revised: Notice that the human demonstrations do not have a success score of 1 on every task.

Original: We see that, except for
Revised: We see that, except for

Original: Relocate, which is a hard task to solve with only 25 human demonstrations due to the necessity to generalize to new positions of the ball and the target, AQuaGAIL solves the tasks as successfully as the humans, outperforming GAIL and BC.
Revised: Relocate, which is a hard task to solve with only 25 human demonstrations due to the necessity to generalize to new positions of the ball and the target, AQuaGAIL solves the tasks as successfully as the humans, outperforming GAIL and BC.

Original: Notice that our results corroborate previous work (Orsini et al., 2021) that showed poor performance of GAIL on human demonstrations after 1M steps.
Revised: Notice that our results corroborate previous work (Orsini et al., 2021) that showed poor performance of GAIL on human demonstrations after 1M steps.

Original: The second metric we provide, on the bottom, is the Wasserstein distance between the state distribution of the demonstrations and the one of the agent.
Revised: The second metric we provide, on the bottom, is the Wasserstein distance between the state distribution of the demonstrations and the one of the agent.

Original: We compute it using the POT library (Flamary et al., 2021) and use the Sinkhorn distance, a regularized version of the Wasserstein distance, as it is faster to compute.
Revised: We compute it using the POT library (Flamary et al., 2021) and use the Sinkhorn distance, a regularized version of the Wasserstein distance, as it is faster to compute.

Original: The “human” Wasserstein distance score is computed by randomly taking 5 episodes out of the 25 human demonstrations and compute the Wasserstein distance to the remaining 20.
Revised: The “human” Wasserstein distance score is computed by randomly taking 5 episodes out of the 25 human demonstrations and compute the Wasserstein distance to the remaining 20.

Original: We repeat this procedure 100 times and plot the median (and the interquartile range) of the obtained values.
Revised: We repeat this procedure 100 times and plot the median (and the interquartile range) of the obtained values.

Original: Remark that AQuaGAIL is able to get much closer behavior to the human than BC and GAIL on all four environments in terms of Wasserstein distance.
Revised: Remark that AQuaGAIL is able to get much closer behavior to the human than BC and GAIL on all four environments in terms of Wasserstein distance.

Original: This supports that AQuaDem leads to policies much closer to the demonstrator.
Revised: This supports that AQuaDem leads to policies much closer to the demonstrator.

Original: We provide videos of the trained agents as an additional qualitative empirical evidence to support this claim.
Revised: We provide videos of the trained agents as an additional qualitative empirical evidence to support this claim.

Original: 4. R EINFORCEMENT L EARNING WITH PLAY DATA
Revised: 4. R EINFORCEMENT L EARNING WITH PLAY DATA

Original: Setup.
Revised: Setup.

Original: The Reinforcement Learning with play data is an under-explored yet natural setup (Gupta et al., 2019).
Revised: The Reinforcement Learning with play data is an under-explored yet natural setup (Gupta et al., 2019).

Original: In this setup, the environment of interest has multiple tasks, a shared observation and action space for each task, and a reward function speciﬁc to each of the tasks.
Revised: In this setup, the environment of interest has multiple tasks, a shared observation and action space for each task, and a reward function speciﬁc to each of the tasks.

Original: We also assume that we have access to play data , introduced by Lynch et al.
Revised: We also assume that we have access to play data , introduced by Lynch et al.

Original: (2020), which consists in episodes from a human demonstrator interacting with an environment with the sole intention to play with it.
Revised: (2020), which consists in episodes from a human demonstrator interacting with an environment with the sole intention to play with it.

Original: The goal is to learn an optimal policy for each of the tasks.
Revised: The goal is to learn an optimal policy for each of the tasks.

Original: We consider the Robodesk tasks (Kannan et al., 2021) shown in Figure 9, for which we acquired play data.
Revised: We consider the Robodesk tasks (Kannan et al., 2021) shown in Figure 9, for which we acquired play data.

Original: We expand on the environment as well as the data collection procedure in the Appendix D.2.
Revised: We expand on the environment as well as the data collection procedure in the Appendix D.2.

Original: Algorithm & baselines.
Revised: Algorithm & baselines.

Original: Similarly to the RLfD setting, we propose a two-fold training procedure: 1) we learn a discretization of the action space in a fully ofﬂine fashion using the AQuaDem framework on the play data, 2) we train a discrete action deep RL algorithm using this discretization on each tasks.
Revised: Similarly to the RLfD setting, we propose a two-fold training procedure: 1) we learn a discretization of the action space in a fully ofﬂine fashion using the AQuaDem framework on the play data, 2) we train a discrete action deep RL algorithm using this discretization on each tasks.

Original: We refer to this algorithm as AQuaPlay.
Revised: We refer to this algorithm as AQuaPlay.

Original: Unlike the RLfD setting, the demonstrations do not include any task speciﬁc reward nor goal labels meaning that we cannot incorporate the demonstration episodes in the replay buffer nor use some form of goal-conditioned BC.
Revised: Unlike the RLfD setting, the demonstrations do not include any task speciﬁc reward nor goal labels meaning that we cannot incorporate the demonstration episodes in the replay buffer nor use some form of goal-conditioned BC.

Original: We use SAC as a baseline, which is trained to optimize task speciﬁc rewards.
Revised: We use SAC as a baseline, which is trained to optimize task speciﬁc rewards.

Original: Since the action space dimensionality is fairly low (5-dimensional), we can include naive discretization baselines we call “bang-bang” (Bushaw, 1952) based on different granularities (therefore extending the original meaning of “bang-bang controller”): we refer to them as BB- 2 , BB- 3 and BB- 5 .
Revised: Since the action space dimensionality is fairly low (5-dimensional), we can include naive discretization baselines we call “bang-bang” (Bushaw, 1952) based on different granularities (therefore extending the original meaning of “bang-bang controller”): we refer to them as BB- 2 , BB- 3 and BB- 5 .

Original: Evaluation & results.
Revised: Evaluation & results.

Original: We train the different methods on 1 M environment interactions on 10 seeds for the chosen hyperparameters (a single set of hyperameters for all tasks) and evaluate the agents every 50 k environment interactions (without exploration noise) on 30 episodes.
Revised: We train the different methods on 1 M environment interactions on 10 seeds for the chosen hyperparameters (a single set of hyperameters for all tasks) and evaluate the agents every 50 k environment interactions (without exploration noise) on 30 episodes.

Original: The AQuaDem discretization is trained ofﬂine on play data using 50 k gradient steps on batches of size 256 .
Revised: The AQuaDem discretization is trained ofﬂine on play data using 50 k gradient steps on batches of size 256 .

Original: The number of actions considered were 10 , 20 , 30 , 40 and we found 30 to be performing the best.
Revised: The number of actions considered were 10 , 20 , 30 , 40 and we found 30 to be performing the best.

Original: It is interesting to notice that it is higher than for the previous setups.
Revised: It is interesting to notice that it is higher than for the previous setups.

Original: It aligns with the intuition that with play data, several behaviors needs to be modelled.
Revised: It aligns with the intuition that with play data, several behaviors needs to be modelled.

Original: The results are provided in Figure 8.
Revised: The results are provided in Figure 8.

Original: The
Revised: The

Original: AQuaPlay agent consistently outperforms SAC in this setup.
Revised: AQuaPlay agent consistently outperforms SAC in this setup.

Original: Interestingly, the performance of the BB agent decreases with the discretization granularity, well exemplifying the curse of dimensionality of the method.
Revised: Interestingly, the performance of the BB agent decreases with the discretization granularity, well exemplifying the curse of dimensionality of the method.

Original: In fact, BB with a binary discretization (BB-2) is competitive with AQuaPlay, which validates that discrete action RL algorithms are well performing if the discrete actions are sufﬁcient to solve the task.
Revised: In fact, BB with a binary discretization (BB-2) is competitive with AQuaPlay, which validates that discrete action RL algorithms are well performing if the discrete actions are sufﬁcient to solve the task.

Original: Note however that the Robodesk environment is a relatively low-dimensional action environment, making it possible to have BB as a baseline, which is not the case of e.g. Adroit where the action space is high-dimensional.
Revised: Note however that the Robodesk environment is a relatively low-dimensional action environment, making it possible to have BB as a baseline, which is not the case of e.g. Adroit where the action space is high-dimensional.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Continuous action discretization.
Revised: Continuous action discretization.

Original: The discretization of continuous action spaces has been introduced in control problems by Bushaw (1952) with the “bang-bang” controller (Bellman et al., 1956).
Revised: The discretization of continuous action spaces has been introduced in control problems by Bushaw (1952) with the “bang-bang” controller (Bellman et al., 1956).

Original: This naive discretization is problematic in high-dimensional action spaces, as the number of actions grows exponentially with the action dimensionality.
Revised: This naive discretization is problematic in high-dimensional action spaces, as the number of actions grows exponentially with the action dimensionality.

Original: To mitigate this phenomenon, a possible strategy is to assume that action dimensions are independent (Tavakoli et al., 2018;
Revised: To mitigate this phenomenon, a possible strategy is to assume that action dimensions are independent (Tavakoli et al., 2018;

Original: Andrychowicz et al., 2020; Vieillard et al., 2021; Tang and Agrawal, 2020), or to assume some causal dependence between them and use an autoregressive discretization (Metz et al., 2017; Tessler et al., 2019).
Revised: Andrychowicz et al., 2020; Vieillard et al., 2021; Tang and Agrawal, 2020), or to assume some causal dependence between them and use an autoregressive discretization (Metz et al., 2017; Tessler et al., 2019).

Original: The
Revised: The

Original: AQuaDem framework circumvents the curse of dimensionality as the discretization is based on the demonstrations and hence is dependent on the multimodality of the actions picked by the demonstrator rather than the dimensionality of the action space.
Revised: AQuaDem framework circumvents the curse of dimensionality as the discretization is based on the demonstrations and hence is dependent on the multimodality of the actions picked by the demonstrator rather than the dimensionality of the action space.

Original: Close to our setup is the case where the action space is both discrete and continuous (Neunert et al., 2020) or the action space is discrete and large (Dulac-Arnold et al., 2015).
Revised: Close to our setup is the case where the action space is both discrete and continuous (Neunert et al., 2020) or the action space is discrete and large (Dulac-Arnold et al., 2015).

Original: Those setups are interesting directions for extending AQuaDem.
Revised: Those setups are interesting directions for extending AQuaDem.

Original: Q-learning in continuous action spaces.
Revised: Q-learning in continuous action spaces.

Original: Policy-based methods consist in solving continuous or discrete MDPs based on maximizing the expected return over the parameters of a family of policies.
Revised: Policy-based methods consist in solving continuous or discrete MDPs based on maximizing the expected return over the parameters of a family of policies.

Original: If the return is estimated through Monte Carlo rollouts, this leads to algorithms that are typically sampleinefﬁcient and difﬁcult to train in high-dimensional action spaces (Williams, 1992; Schulman et al., 2015; 2017).
Revised: If the return is estimated through Monte Carlo rollouts, this leads to algorithms that are typically sampleinefﬁcient and difﬁcult to train in high-dimensional action spaces (Williams, 1992; Schulman et al., 2015; 2017).

Original: As a result, a number of policy-based methods inspired from the policy gradient theorem (Sutton et al., 2000), aim at maximizing the return using an approximate version of the Q -value thus making them more sample-efﬁcient.
Revised: As a result, a number of policy-based methods inspired from the policy gradient theorem (Sutton et al., 2000), aim at maximizing the return using an approximate version of the Q -value thus making them more sample-efﬁcient.

Original: One common architecture is to parameterize a Q -value, which is estimated by enforcing Bellman consistency, and deﬁne a policy using an optimization procedure of the parametrized Q -value.
Revised: One common architecture is to parameterize a Q -value, which is estimated by enforcing Bellman consistency, and deﬁne a policy using an optimization procedure of the parametrized Q -value.

Original: Typical strategies to solve the Q -value maximization include enforcing the Q -value to be concave (Gu et al., 2016; Amos et al., 2017) making it easy to optimize through e.g. gradient ascent, to use a black box optimization method (Kalashnikov et al., 2018;
Revised: Typical strategies to solve the Q -value maximization include enforcing the Q -value to be concave (Gu et al., 2016; Amos et al., 2017) making it easy to optimize through e.g. gradient ascent, to use a black box optimization method (Kalashnikov et al., 2018;

Original: Simmons-Edler et al., 2019; Lim et al., 2018), to solve a mixed integer programming problem (Ryu et al., 2020), or to follow a biased estimate of the policy gradient based on the approximate Q -value (Konda and
Revised: Simmons-Edler et al., 2019; Lim et al., 2018), to solve a mixed integer programming problem (Ryu et al., 2020), or to follow a biased estimate of the policy gradient based on the approximate Q -value (Konda and

Original: Tsitsiklis, 2000; Lillicrap et al., 2016;
Revised: Tsitsiklis, 2000; Lillicrap et al., 2016;

Original: Haarnoja et al., 2018; Fujimoto et al., 2018).
Revised: Haarnoja et al., 2018; Fujimoto et al., 2018).

Original: Recently, Asadi et al.
Revised: Recently, Asadi et al.

Original: (2021) proposed to use a network that outputs actions together with their associated Q -values, tuned for each of the tasks at hand, on low-dimensional action spaces.
Revised: (2021) proposed to use a network that outputs actions together with their associated Q -values, tuned for each of the tasks at hand, on low-dimensional action spaces.

Original: Note that maximizing the approximate Q -value is a key problem that does not appear in discrete action space environments, thus justifying the interest of the AQuaDem framework.
Revised: Note that maximizing the approximate Q -value is a key problem that does not appear in discrete action space environments, thus justifying the interest of the AQuaDem framework.

Original: Hierarchical Imitation Learning.
Revised: Hierarchical Imitation Learning.

Original: A number of approaches have explored the learning of primitives or options from demonstrations together with a high-level controller that is either learned from demonstrations (Kroemer et al., 2015; Krishnan et al., 2017; Le et al., 2018;
Revised: A number of approaches have explored the learning of primitives or options from demonstrations together with a high-level controller that is either learned from demonstrations (Kroemer et al., 2015; Krishnan et al., 2017; Le et al., 2018;

Original: Ding et al., 2019; Lynch et al., 2020), or learned from interactions with the environment (Manschitz et al., 2015; Kipf et al., 2019; Shankar et al., 2019), or hand speciﬁed (Pastor et al., 2009; Fox et al., 2019).
Revised: Ding et al., 2019; Lynch et al., 2020), or learned from interactions with the environment (Manschitz et al., 2015; Kipf et al., 2019; Shankar et al., 2019), or hand speciﬁed (Pastor et al., 2009; Fox et al., 2019).

Original: AQuaDem can be loosely interpreted as a two-level procedure as well, where the primitives (action discretization step) are learned fully ofﬂine, however there is no concept of goal nor temporally extended actions.
Revised: AQuaDem can be loosely interpreted as a two-level procedure as well, where the primitives (action discretization step) are learned fully ofﬂine, however there is no concept of goal nor temporally extended actions.

Original: Modeling multimodal demonstrations.
Revised: Modeling multimodal demonstrations.

Original: A number of works have modeled the demonstrator data using multimodal architectures.
Revised: A number of works have modeled the demonstrator data using multimodal architectures.

Original: For example, Chernova and Veloso (2007); Calinon and Billard (2007) introduce Gaussian mixture models in their modeling of the demonstrator data.
Revised: For example, Chernova and Veloso (2007); Calinon and Billard (2007) introduce Gaussian mixture models in their modeling of the demonstrator data.

Original: More recently,
Revised: More recently,

Original: Rahmatizadeh et al.
Revised: Rahmatizadeh et al.

Original: (2018) use Mixture density networks together with a recurrent neural network to model the temporal correlation of actions as well as their multimodality.
Revised: (2018) use Mixture density networks together with a recurrent neural network to model the temporal correlation of actions as well as their multimodality.

Original: (Yu et al., 2018) also uses Mixture density networks to meta-learn a policy from demonstrations for one-shot adaptation.
Revised: (Yu et al., 2018) also uses Mixture density networks to meta-learn a policy from demonstrations for one-shot adaptation.

Original: Another recent line of works has considered the problem of modeling demonstrations using an energy-based model, which is well adapted for multimodalities (Jarrett et al., 2020; Florence et al., 2021).
Revised: Another recent line of works has considered the problem of modeling demonstrations using an energy-based model, which is well adapted for multimodalities (Jarrett et al., 2020; Florence et al., 2021).

Original: Singh et al.
Revised: Singh et al.

Original: (2020) also exploit the demonstrations prior for downstream tasks by learning a prior using a state-conditioned action generative model coupled with a continuous action algorithm.
Revised: (2020) also exploit the demonstrations prior for downstream tasks by learning a prior using a state-conditioned action generative model coupled with a continuous action algorithm.

Original: This is different from AQuaDem that exploits the demonstrations prior to learn a discrete action space in order to use discrete action RL algorithms.
Revised: This is different from AQuaDem that exploits the demonstrations prior to learn a discrete action space in order to use discrete action RL algorithms.

Original:  P ERSPECTIVES AND C ONCLUSION
Revised:  P ERSPECTIVES AND C ONCLUSION

Original: With the AQuaDem paradigm, we provide a simple yet powerful method that enables to use discreteaction deep RL methods on continuous control tasks using demonstrations, thus escaping the complexity or curse of dimensionality of existing discretization methods.
Revised: With the AQuaDem paradigm, we provide a simple yet powerful method that enables to use discreteaction deep RL methods on continuous control tasks using demonstrations, thus escaping the complexity or curse of dimensionality of existing discretization methods.

Original: We showed in three different setups that it provides substantial gains in sample efﬁciency and performance and that it leads to qualitatively better agents, as enlightened by the videos provided in the supplementary material.
Revised: We showed in three different setups that it provides substantial gains in sample efﬁciency and performance and that it leads to qualitatively better agents, as enlightened by the videos provided in the supplementary material.

Original: There are a number of different research avenues opened by AQuaDem.
Revised: There are a number of different research avenues opened by AQuaDem.

Original: Other discrete action speciﬁc methods could be leveraged in a similar way in the context of continuous control: count-based exploration (Tang et al., 2017), planning (Browne et al., 2012) or ofﬂine RL (Lagoudakis and Parr, 2003; Riedmiller, 2005).
Revised: Other discrete action speciﬁc methods could be leveraged in a similar way in the context of continuous control: count-based exploration (Tang et al., 2017), planning (Browne et al., 2012) or ofﬂine RL (Lagoudakis and Parr, 2003; Riedmiller, 2005).

Original: Similarly a number of methods in Imitation Learning (Brantley et al., 2019; Wang et al., 2019) or in ofﬂine RL (Fujimoto and Gu, 2021; Wu et al., 2019) are evaluated on continuous control tasks and are based on Behavioral Cloning regularization which could be reﬁned using the same type of multioutput architecture used in this work.
Revised: Similarly a number of methods in Imitation Learning (Brantley et al., 2019; Wang et al., 2019) or in ofﬂine RL (Fujimoto and Gu, 2021; Wu et al., 2019) are evaluated on continuous control tasks and are based on Behavioral Cloning regularization which could be reﬁned using the same type of multioutput architecture used in this work.

Original: Another possible direction for the
Revised: Another possible direction for the

Original: AQuaDem framework is to be analyzed in the light of risk-MDPs as the constraint of the action space arguably reduces a notion of risk when acting in this environment.
Revised: AQuaDem framework is to be analyzed in the light of risk-MDPs as the constraint of the action space arguably reduces a notion of risk when acting in this environment.

Original: Finally, as the gain of sample efﬁciency is clear in different experimental settings, we believe that the AQuaDem framework could be an interesting avenue for learning controllers on physical systems.
Revised: Finally, as the gain of sample efﬁciency is clear in different experimental settings, we believe that the AQuaDem framework could be an interesting avenue for learning controllers on physical systems.

Original: A C ONNECTION TO G AUSSIAN MIXTURE MODELS
Revised: A C ONNECTION TO G AUSSIAN MIXTURE MODELS

Original: The BC loss can be interpreted as a maximum likelihood objective under the assumption that the demonstrator data comes from a Gaussian distribution.
Revised: The BC loss can be interpreted as a maximum likelihood objective under the assumption that the demonstrator data comes from a Gaussian distribution.

Original: Similarly to Mixture density networks (Bishop, 1994), we propose to replace the Gaussian distribution by a mixture of Gaussian distributions.
Revised: Similarly to Mixture density networks (Bishop, 1994), we propose to replace the Gaussian distribution by a mixture of Gaussian distributions.

Original: Suppose we represent the probability density of an action conditioned on a state by a mixture of K
Revised: Suppose we represent the probability density of an action conditioned on a state by a mixture of K

Original: Gaussian kernels: p ( a | s ) =
Revised: Gaussian kernels: p ( a | s ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk
Revised: Kk

Original: =1 α k
Revised: =1 α k

Original: ( s ) d k ( a | s ) , where α k ( s ) is the mixing coefﬁcient (that can be interpreted as a state conditioned prior probability), and d k ( a | s ) is the conditional density of the target a .
Revised: ( s ) d k ( a | s ) , where α k ( s ) is the mixing coefﬁcient (that can be interpreted as a state conditioned prior probability), and d k ( a | s ) is the conditional density of the target a .

Original: Now assuming that the K kernels are centered on Ψ k ( s ) k
Revised: Now assuming that the K kernels are centered on Ψ k ( s ) k

Original: =1: K and have ﬁxed covariance σ 2 1 where σ is a hyperparameter
Revised: =1: K and have ﬁxed covariance σ 2 1 where σ is a hyperparameter

Original: [Equation]
Revised: [Equation]

Original: we can write the log-likelihood of the demonstrations data D as:
Revised: we can write the log-likelihood of the demonstrations data D as:

Original: [Equation]
Revised: [Equation]

Original: Therefore minimizing the negative log likelihood reduces to minimizing:
Revised: Therefore minimizing the negative log likelihood reduces to minimizing:

Original: [Equation]
Revised: [Equation]

Original: We propose to use a uniform prior α k ( s ) =
Revised: We propose to use a uniform prior α k ( s ) =

Original: [Equation]
Revised: [Equation]

Original: when learning the locations of the centroids, which leads exactly to Equation (1) where the variance σ 2 is the temperature T .
Revised: when learning the locations of the centroids, which leads exactly to Equation (1) where the variance σ 2 is the temperature T .

Original: Note that we initially learned the state conditioned prior α k ( s ) , but we found no empirical evidence that it may be used to improve the performance of the downstream algorithms deﬁned in Section 4.
Revised: Note that we initially learned the state conditioned prior α k ( s ) , but we found no empirical evidence that it may be used to improve the performance of the downstream algorithms deﬁned in Section 4.

Original: B A CTION VISUALIZATION IN A HIGH - DIMENSIONAL ENVIRONMENT
Revised: B A CTION VISUALIZATION IN A HIGH - DIMENSIONAL ENVIRONMENT

Original: For the Door environment (see Figure 9) we represent the actions candidates learned using the
Revised: For the Door environment (see Figure 9) we represent the actions candidates learned using the

Original: AQuaDem framework with videos that can be found in the supplementary material in the folder visualizations (for the best hyperparameters in the RL with demonstrations setting see Appendix D.4.1).
Revised: AQuaDem framework with videos that can be found in the supplementary material in the folder visualizations (for the best hyperparameters in the RL with demonstrations setting see Appendix D.4.1).

Original: As the action space is of high dimensionality, we choose to represent each action dimension on the x-axis, and the value for each dimension on the y-axis.
Revised: As the action space is of high dimensionality, we choose to represent each action dimension on the x-axis, and the value for each dimension on the y-axis.

Original: We connect the dots on the x-axis to facilitate the visualization through time.
Revised: We connect the dots on the x-axis to facilitate the visualization through time.

Original: We replay a trajectory from the human demonstrations and show at each step the 10 actions proposed by the AQuaDem network, and the action actually taken by the human demonstrator.
Revised: We replay a trajectory from the human demonstrations and show at each step the 10 actions proposed by the AQuaDem network, and the action actually taken by the human demonstrator.

Original: Each action candidate has a color consistent across time (meaning that the blue action always correspond to the same head of the Ψ network).
Revised: Each action candidate has a color consistent across time (meaning that the blue action always correspond to the same head of the Ψ network).

Original: Interestingly, the video shows that actions are very state dependent (except some default 0 -action) and evolve smoothly through time.
Revised: Interestingly, the video shows that actions are very state dependent (except some default 0 -action) and evolve smoothly through time.

Original: C A BLATION S TUDY
Revised: C A BLATION S TUDY

Original: In this section, we provide two ablations of the AQuaDQN algorithm.
Revised: In this section, we provide two ablations of the AQuaDQN algorithm.

Original: The ﬁrst ablation is to learn a ﬁxed set of actions independently of the state (which reduces to K -means).
Revised: The ﬁrst ablation is to learn a ﬁxed set of actions independently of the state (which reduces to K -means).

Original: The second ablation consists in using random actions rather than the actions learned by the AQuaDem framework (the actions are given by the AQuaDem network, randomly initialized and not trained).
Revised: The second ablation consists in using random actions rather than the actions learned by the AQuaDem framework (the actions are given by the AQuaDem network, randomly initialized and not trained).

Original: We use the same hyperparameters as the one selected for AQuaDQN.
Revised: We use the same hyperparameters as the one selected for AQuaDQN.

Original: In each case, for a number of actions in { 5, 10, 25 } , the success rate of the agent is 0 for all tasks throughout the training procedure.
Revised: In each case, for a number of actions in { 5, 10, 25 } , the success rate of the agent is 0 for all tasks throughout the training procedure.

Original: D I MPLEMENTATION D. G RID W ORLD V ISUALIZATIONS
Revised: D I MPLEMENTATION D. G RID W ORLD V ISUALIZATIONS

Original: We learn the discretization of the action space using the AQuaDem framework.
Revised: We learn the discretization of the action space using the AQuaDem framework.

Original: The architecture of the network is a common hidden layer of size 256 with relu activation, and a subsequent hidden layer of size 256 with relu activation for each action.
Revised: The architecture of the network is a common hidden layer of size 256 with relu activation, and a subsequent hidden layer of size 256 with relu activation for each action.

Original: We minimize the AQuaDem loss using the Adam optimizer with the learning rate 0 .
Revised: We minimize the AQuaDem loss using the Adam optimizer with the learning rate 0 .

Original: 0003 and the dropout regularization rate 0 .
Revised: 0003 and the dropout regularization rate 0 .

Original: 1 on 20000 gradient steps.
Revised: 1 on 20000 gradient steps.

Original: D. E NVIRONMENTS
Revised: D. E NVIRONMENTS

Original: We considered the Adroit environments and the Robodesk environments, for which we described the observation space and the action space in Table 1.
Revised: We considered the Adroit environments and the Robodesk environments, for which we described the observation space and the action space in Table 1.

Original: [Table]
Revised: [Table]

Original: Adroit The Adroit environments (Rajeswaran et al., 2017) consists in a shadow hand solvingtasks (Figure 9).
Revised: Adroit The Adroit environments (Rajeswaran et al., 2017) consists in a shadow hand solvingtasks (Figure 9).

Original: The environments come with demonstrations which are gathered using virtual reality by a human.
Revised: The environments come with demonstrations which are gathered using virtual reality by a human.

Original: Robodesk The Robodesk environment (Kannan et al., 2021) consists of a simulated Franka Emika
Revised: Robodesk The Robodesk environment (Kannan et al., 2021) consists of a simulated Franka Emika

Original: Panda robot interacting with a table where multiple tasks are possible.
Revised: Panda robot interacting with a table where multiple tasks are possible.

Original: The version of the simulated robot in the Robodesk environment only includes 5 DoFs (vs the 7 DoFs available, 2 were made not controllable).
Revised: The version of the simulated robot in the Robodesk environment only includes 5 DoFs (vs the 7 DoFs available, 2 were made not controllable).

Original: We evaluate AQuaPlay on the 9 base tasks described in Robodesk: open slide , open drawer , push green , stack , upright
Revised: We evaluate AQuaPlay on the 9 base tasks described in Robodesk: open slide , open drawer , push green , stack , upright

Original: block
Revised: block

Original: off table
Revised: off table

Original: , flat block in bin , flat block in shelf , lift upright block , lift ball .
Revised: , flat block in bin , flat block in shelf , lift upright block , lift ball .

Original: We used the RLDS creator github.com/google-research/rlds-creator to generate play data, together with a Nintendo Switch Pro Controller.
Revised: We used the RLDS creator github.com/google-research/rlds-creator to generate play data, together with a Nintendo Switch Pro Controller.

Original: The data is composed by 50 episodes of approximately 3 minutes where the goal of the demonstrator is to interact with the different elements of the environment.
Revised: The data is composed by 50 episodes of approximately 3 minutes where the goal of the demonstrator is to interact with the different elements of the environment.

Original: D. H YPERPARAMETER S ELECTION P ROCEDURE
Revised: D. H YPERPARAMETER S ELECTION P ROCEDURE

Original: In the section we provide the hyperparameter selection procedure for the different setups.
Revised: In the section we provide the hyperparameter selection procedure for the different setups.

Original: For the RLfD setting (Section 4.1) and the IL setting (Section 4.2) the number of hyperparameters is prohibitive to perform grid search.
Revised: For the RLfD setting (Section 4.1) and the IL setting (Section 4.2) the number of hyperparameters is prohibitive to perform grid search.

Original: Therefore, we propose to sample hyperparameters uniformly within the set of all possible hyperparameters.
Revised: Therefore, we propose to sample hyperparameters uniformly within the set of all possible hyperparameters.

Original: For each environment, we sample 1000 conﬁgurations of hyperparameters, and train each algorithm including the baselines.
Revised: For each environment, we sample 1000 conﬁgurations of hyperparameters, and train each algorithm including the baselines.

Original: We compute the average success rate of each individual value on the top 50% of all corresponding conﬁgurations (since poorly performing conﬁgurations are less informative) and select the best performing hyperparameter value independently.
Revised: We compute the average success rate of each individual value on the top 50% of all corresponding conﬁgurations (since poorly performing conﬁgurations are less informative) and select the best performing hyperparameter value independently.

Original: This procedure enables to 1) limit combinatorial explosion with the number of hyperparameters 2) provide a fair evaluation between the baselines and the proposed algorithms as they all rely on the same amount of compute.
Revised: This procedure enables to 1) limit combinatorial explosion with the number of hyperparameters 2) provide a fair evaluation between the baselines and the proposed algorithms as they all rely on the same amount of compute.

Original: In the supplementary material, we provide histograms detailing the inﬂuence of each hyperparameter.
Revised: In the supplementary material, we provide histograms detailing the inﬂuence of each hyperparameter.

Original: For the RLfP setting, we ﬁxed the parameters related to the DQN algorithm with the ones selected in the RLfD setting to limit the hyperparameter search, which enables to perform grid search for 3 seeds, and select the best set of hyperparameters.
Revised: For the RLfP setting, we ﬁxed the parameters related to the DQN algorithm with the ones selected in the RLfD setting to limit the hyperparameter search, which enables to perform grid search for 3 seeds, and select the best set of hyperparameters.

Original: D. R EINFORCEMENT L EARNING WITH DEMONSTRATIONS D.4. AQ UA DQN
Revised: D. R EINFORCEMENT L EARNING WITH DEMONSTRATIONS D.4. AQ UA DQN

Original: We learn the discretization of the action space using the AQuaDem framework.
Revised: We learn the discretization of the action space using the AQuaDem framework.

Original: The architecture of the network is a common hidden layer of size 256 with relu activation, and a subsequent hidden layer of size 256 with relu activation for each action.
Revised: The architecture of the network is a common hidden layer of size 256 with relu activation, and a subsequent hidden layer of size 256 with relu activation for each action.

Original: We minimize the AQuaDem loss using the Adam optimizer and dropout regularization.
Revised: We minimize the AQuaDem loss using the Adam optimizer and dropout regularization.

Original: We train a DQN agent on top of the discretization learned by the AQuaDem framework.
Revised: We train a DQN agent on top of the discretization learned by the AQuaDem framework.

Original: The architecture of the Q-network we use is the default LayerNorm architecture from the Q-network of the ACME library (Hoffman et al., 2020), which consists in a hidden layer of size 512 with layer normalization and tanh activation, followed by two hidden layers of sizes 512 and 256 with elu activation.
Revised: The architecture of the Q-network we use is the default LayerNorm architecture from the Q-network of the ACME library (Hoffman et al., 2020), which consists in a hidden layer of size 512 with layer normalization and tanh activation, followed by two hidden layers of sizes 512 and 256 with elu activation.

Original: We explored multiple Q -value losses for which we used the Adam optimizer: regular
Revised: We explored multiple Q -value losses for which we used the Adam optimizer: regular

Original: DQN (Mnih et al., 2015), double DQN with experience replay (Van Hasselt et al., 2016; Schaul et al., 2016), and Munchausen DQN (Vieillard et al., 2020); the latter led to the best performance.
Revised: DQN (Mnih et al., 2015), double DQN with experience replay (Van Hasselt et al., 2016; Schaul et al., 2016), and Munchausen DQN (Vieillard et al., 2020); the latter led to the best performance.

Original: We maintain a ﬁxed ratio of demonstration episodes and agent episodes in the replay buffer similarly to Hester et al.
Revised: We maintain a ﬁxed ratio of demonstration episodes and agent episodes in the replay buffer similarly to Hester et al.

Original: We also provide as a hyperparameter an optional minimum reward to the transitions of the expert to have a denser reward signal.
Revised: We also provide as a hyperparameter an optional minimum reward to the transitions of the expert to have a denser reward signal.

Original: The hyperparameter sweep for AQuaDQN can be found in Table 2.
Revised: The hyperparameter sweep for AQuaDQN can be found in Table 2.

Original: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps lfd aquadqn.html in the supplementary.
Revised: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps lfd aquadqn.html in the supplementary.

Original: [Table]
Revised: [Table]

Original: When selecting hyperparameters speciﬁcally for Relocate, for Figure 6, the main difference in the chosen values is a dropout rate set to 0.
Revised: When selecting hyperparameters speciﬁcally for Relocate, for Figure 6, the main difference in the chosen values is a dropout rate set to 0.

Original: D.4. SAC AND SAC F D
Revised: D.4. SAC AND SAC F D

Original: We reproduced the authors’ implementations (with an adaptive temperature) and use MLP networks for both the actor and the critic with two hidden layers of size 256 with relu activation.
Revised: We reproduced the authors’ implementations (with an adaptive temperature) and use MLP networks for both the actor and the critic with two hidden layers of size 256 with relu activation.

Original: We use an
Revised: We use an

Original: Adam optimizer to train the SAC losses.
Revised: Adam optimizer to train the SAC losses.

Original: We use a replay buffer of size 1 M, and sample batches of size 256 .
Revised: We use a replay buffer of size 1 M, and sample batches of size 256 .

Original: We introduce a parameter of gradient updates frequency n which indicates a number of n gradient updates on the SAC losses every n environment steps.
Revised: We introduce a parameter of gradient updates frequency n which indicates a number of n gradient updates on the SAC losses every n environment steps.

Original: SACfD is a version of SAC inspired by DDPGfD (Vecerik et al., 2017) where we add expert demonstrations to the replay buffer of the SAC agent with a ratio between the agent episodes and the demonstration episodes which is a hyperparameter.
Revised: SACfD is a version of SAC inspired by DDPGfD (Vecerik et al., 2017) where we add expert demonstrations to the replay buffer of the SAC agent with a ratio between the agent episodes and the demonstration episodes which is a hyperparameter.

Original: We also provide as a hyperparameter an optional minimum reward to the transitions of the expert to have a denser reward signal.
Revised: We also provide as a hyperparameter an optional minimum reward to the transitions of the expert to have a denser reward signal.

Original: We found that the best hyperparameters for SAC are the same for SACfD. The HP sweep for SAC and SACfD can be found in Table 3 and Table 4.
Revised: We found that the best hyperparameters for SAC are the same for SACfD. The HP sweep for SAC and SACfD can be found in Table 3 and Table 4.

Original: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps lfd sac.html and hps lfd sacfd.html in the supplementary.
Revised: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps lfd sac.html and hps lfd sacfd.html in the supplementary.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D. I MITATION L EARNING D.5. AQ UA GAIL
Revised: D. I MITATION L EARNING D.5. AQ UA GAIL

Original: We learn the discretization of the action space using the AQuaDem framework.
Revised: We learn the discretization of the action space using the AQuaDem framework.

Original: The architecture ofthe network is a common hidden layer of size 256 with relu activation
Revised: The architecture ofthe network is a common hidden layer of size 256 with relu activation

Original: [Table]
Revised: [Table]

Original: and a subsequent hidden layer of size 256 with relu activation for each action.
Revised: and a subsequent hidden layer of size 256 with relu activation for each action.

Original: We minimize the AQuaDem loss using the Adam optimizer and dropout regularization.
Revised: We minimize the AQuaDem loss using the Adam optimizer and dropout regularization.

Original: The discriminator is a MLP whose number of layers, number of units per layers are hyperparameters.
Revised: The discriminator is a MLP whose number of layers, number of units per layers are hyperparameters.

Original: We use the Adam optimizer with two possible regularization scheme: dropout and weight decay.
Revised: We use the Adam optimizer with two possible regularization scheme: dropout and weight decay.

Original: The discriminator outputs a value p from which we compute three possible rewards − log( p ) , − 0 .
Revised: The discriminator outputs a value p from which we compute three possible rewards − log( p ) , − 0 .

Original: 5 log( p ) + log(1 − p ) , log(1 − p ) corresponding to the reward balance hyperparameter.
Revised: 5 log( p ) + log(1 − p ) , log(1 − p ) corresponding to the reward balance hyperparameter.

Original: The direct RL algorithm is Munchausen DQN, with the same architecture and hyperparameters described in Section D.4.1.
Revised: The direct RL algorithm is Munchausen DQN, with the same architecture and hyperparameters described in Section D.4.1.

Original: The hyperparameter sweep for AQuaGAIL can be found in Table 5.
Revised: The hyperparameter sweep for AQuaGAIL can be found in Table 5.

Original: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps il aquagail.html in the supplementary.
Revised: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps il aquagail.html in the supplementary.

Original: Hyperparameter
Revised: Hyperparameter

Original: [Table]
Revised: [Table]

Original: discriminator
Revised: discriminator

Original: [Table]
Revised: [Table]

Original: discriminator
Revised: discriminator

Original: [Table]
Revised: [Table]

Original: D.5. GAIL
Revised: D.5. GAIL

Original: We used the same discriminator architecture and hyperparameters as the one described in Section
Revised: We used the same discriminator architecture and hyperparameters as the one described in Section

Original: D.5.1.
Revised: D.5.1.

Original: The direct RL agent is the SAC algorithm whose architecture and hyperparameters are described in Section D.4.2.
Revised: The direct RL agent is the SAC algorithm whose architecture and hyperparameters are described in Section D.4.2.

Original: The hyperparameter sweep for GAIL can be found in Table 6.
Revised: The hyperparameter sweep for GAIL can be found in Table 6.

Original: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps il gail.html in the supplementary.
Revised: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps il gail.html in the supplementary.

Original: [Table]
Revised: [Table]

Original: D.5. B EHAVIORAL C LONING
Revised: D.5. B EHAVIORAL C LONING

Original: The BC network is a MLP whose number of layers, number of units per layers and activation functions are hyperparameters.
Revised: The BC network is a MLP whose number of layers, number of units per layers and activation functions are hyperparameters.

Original: We use the Adam optimizer with two possible regularization scheme: dropout and weight decay.
Revised: We use the Adam optimizer with two possible regularization scheme: dropout and weight decay.

Original: The observation normalization hyperparameter is set to True when each dimension of the observation are centered with the mean and standard deviation of the observations in the demonstration dataset.
Revised: The observation normalization hyperparameter is set to True when each dimension of the observation are centered with the mean and standard deviation of the observations in the demonstration dataset.

Original: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps il bc.html in the supplementary.
Revised: The complete breakdown of the inﬂuence of each hyperparameter is provided in hps il bc.html in the supplementary.

Original: [Table]
Revised: [Table]

Original: D. R EINFORCEMENT L EARNING WITH PLAY DATA D.6. SAC
Revised: D. R EINFORCEMENT L EARNING WITH PLAY DATA D.6. SAC

Original: We used the exact same implementation as the one described in Section D.4.2.
Revised: We used the exact same implementation as the one described in Section D.4.2.

Original: The HP sweep can be found in Table 8.
Revised: The HP sweep can be found in Table 8.

Original: D.6. DQN WITH N AIVE D ISCRETIZATION
Revised: D.6. DQN WITH N AIVE D ISCRETIZATION

Original: We used the exact same implementation as the one described in Section D.4.
Revised: We used the exact same implementation as the one described in Section D.4.

Original: [Table]
Revised: [Table]

Original: and also use the best hyperparameters found in the RLfD setting.
Revised: and also use the best hyperparameters found in the RLfD setting.

Original: As the action space is ( − 1 , 1) 5 , we use three different
Revised: As the action space is ( − 1 , 1) 5 , we use three different

Original: [Table]
Revised: [Table]

Original: discretization meshes: {− 1 , 1 } , {− 1 , 0 , 1 } , {− 1 , − 0 .
Revised: discretization meshes: {− 1 , 1 } , {− 1 , 0 , 1 } , {− 1 , − 0 .

Original: 5 , 1 } which induce a discrete action space of dimension 2 5 , 3 5 , 5 5 respectively.
Revised: 5 , 1 } which induce a discrete action space of dimension 2 5 , 3 5 , 5 5 respectively.

Original: We refer to the resulting algorithm as BB-2, BB-3, and
Revised: We refer to the resulting algorithm as BB-2, BB-3, and

Original: BB-5 (where BB stands for “Bang-bang”).
Revised: BB-5 (where BB stands for “Bang-bang”).

Original: D.6. AQ UA P LAY
Revised: D.6. AQ UA P LAY

Original: We used the exact same implementation as the one described in Section D.4.1, and also use the best hyperparameters found in the RLfD setting for the Munchausen DQN agent.
Revised: We used the exact same implementation as the one described in Section D.4.1, and also use the best hyperparameters found in the RLfD setting for the Munchausen DQN agent.

Original: We performed a sweep on the discretization step that we report in Table 9.
Revised: We performed a sweep on the discretization step that we report in Table 9.

Original: [Table]
Revised: [Table]

Original: E S ANITY C HECK B ASELINES
Revised: E S ANITY C HECK B ASELINES

Original: We provide in Figure 10 the results of our SAC implementation on the 5 classical OpenAI Gym environments, for 20M steps.
Revised: We provide in Figure 10 the results of our SAC implementation on the 5 classical OpenAI Gym environments, for 20M steps.

Original: The hyperparameters are the one of the original papers (original paper, followed by the introduction of the adaptative temperature).
Revised: The hyperparameters are the one of the original papers (original paper, followed by the introduction of the adaptative temperature).

Original: The results correspond to what is provided in the literature.
Revised: The results correspond to what is provided in the literature.

Original: We provide in Figure 11 the results on OpenAI Gym environment sof the conﬁgurations of SAC and
Revised: We provide in Figure 11 the results on OpenAI Gym environment sof the conﬁgurations of SAC and

Original: AQuaDQN that were selected for Adroit (and not Mujoco!).
Revised: AQuaDQN that were selected for Adroit (and not Mujoco!).

Original: The action discretization is learned on the medium-expert dataset from D4RL (no dataset is provided for Humanoid and that is why we do not include it in this ﬁgure).
Revised: The action discretization is learned on the medium-expert dataset from D4RL (no dataset is provided for Humanoid and that is why we do not include it in this ﬁgure).

Original: One can see in Figure 10 that SAC can perform much better on Mujoco
Revised: One can see in Figure 10 that SAC can perform much better on Mujoco

Original: Gym after 1M steps when selecting hyperparameters on the environments themselves.
Revised: Gym after 1M steps when selecting hyperparameters on the environments themselves.

Original: [Figure]
Revised: [Figure]

Revised: r

Revised: [Figure]

Revised: t

Revised: [Figure]

Original: F P ROOF
Revised: F P ROOF

Original: For lighter notations, we write x k = ||
Revised: For lighter notations, we write x k = ||

Original: Ψ k ( s ) − a || 2 and x = ( x 1 , ..., x K ) .
Revised: Ψ k ( s ) − a || 2 and x = ( x 1 , ..., x K ) .

Revised: For a single state-action pair (the empirical expectation being not relevant for studying the effect of the temperature), the

Original: The AQuaDem loss canthus be rewritten:
Revised: AQuaDem loss can be rewritten:

Original: [Equation]
Revised: [Equation]

Original: Let’s deﬁne f T ( x ) = − T log( 1 K (cid:80)
Revised: Let’s deﬁne f T ( x ) = − T log( 1 K (cid:80)

Original: Kk =1 exp( − x k T )) .
Revised: Kk =1 exp( − x k T )) .

Original: The function f T is the same as the loss up to (1) a constant term that does not change the solution of the minimum, (2) a scaling term that can be absorbed in the learning rate.
Revised: The function f T is the same as the loss up to (1) a constant term that does not change the solution of the optimization problem, (2) a scaling term that can be absorbed into the learning rate.

Revised: So, we can study this function for the behavior of the loss with the respect to the temperature.

Original: Now, denoting x m = min k x k .
Revised: Now, denoting x m = min k x k , we’ll ﬁrst study the behavior for low temperature.

Original: [Equation]
Revised: [Equation]

Original: because x k − x m ≥
Revised: Therefore, when the temperature goes to zero, f T behaves as the minimum.

Original: We also have, using Taylorexpansions:
Revised: For large temperatures, we have, using Taylor

Original: [Equation]
Revised: [Equation]

Revised: So, when the temperature goes to inﬁnite, f T behaves as the average.