Original: B EYOND G AMES : B RINGING E XPLORATION TO R OBOTS IN R EAL - WORLD
Revised: B EYOND G AMES : B RINGING E XPLORATION TO R OBOTS IN R EAL -W ORLD

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Exploration has been a long standing problem in both model-based and model-free learning methods for sensorimotor control.
Revised: Exploration has been a long standing problem in both model-based and model-free learning methods for sensorimotor control.

Original: While there has been major advances over the years, most of these successes have been demonstrated in either video games or simulation environments.
Revised: While there has been major advances over the years, most of these successes have been demonstrated in either video games or simulation environments.

Original: This is primarily because the rewards (even the intrinsic ones) are non-differentiable since they are function of the environment (which is a black-box).
Revised: This is primarily because the rewards (even the intrinsic ones) are non-differentiable since they are function of the environment (which is a black-box).

Original: In this paper, we focus on the policy optimization aspect of the intrinsic reward function.
Revised: In this paper, we focus on the policy optimization aspect of the intrinsic reward function.

Original: Speciﬁcally, by using a local approximation, we formulate intrinsic reward as a differentiable function so as to perform policy optimization using likelihood maximization – much like supervised learning instead of reinforcement learning.
Revised: Speciﬁcally, by using a local approximation, we formulate intrinsic reward as a differentiable function so as to perform policy optimization using likelihood maximization – much like supervised learning instead of reinforcement learning.

Original: This leads to a signiﬁcantly sample efﬁcient exploration policy.
Revised: This leads to a signiﬁcantly sample efﬁcient exploration policy.

Original: Our experiments clearly show that our approach outperforms both on-policy and off-policy optimization approaches like REINFORCE and DQN respectively.
Revised: Our experiments clearly show that our approach outperforms both on-policy and off-policy optimization approaches like REINFORCE and DQN respectively.

Original: But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module.
Revised: But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module.

Original: See project videos at https: //doubleblindICLR.github.io/robot-exploration/ .
Revised: See project videos at https: //doubleblindICLR.github.io/robot-exploration/ .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There has been a lot of recent progress in the ﬁeld of Reinforcement Learning (RL).
Revised: There has been a lot of recent progress in the ﬁeld of Reinforcement Learning (RL).

Original: However, most of the successful applications have been conﬁned to the artiﬁcial world of video games (Mnih et al., 2015b) or simulations (Lillicrap et al., 2016).
Revised: However, most of the successful applications have been conﬁned to the artiﬁcial world of video games (Mnih et al., 2015b) or simulations (Lillicrap et al., 2016).

Original: While the ﬁeld of RL was born out of need to make our robots learn how to perform actions, none of the recent advances have translated to success in the ﬁeld of robotics.
Revised: While the ﬁeld of RL was born out of need to make our robots learn how to perform actions, none of the recent advances have translated to success in the ﬁeld of robotics.

Original: Why is that?
Revised: Why is that?

Original: Let us consider the simple task of stacking.
Revised: Let us consider the simple task of stacking.

Original: How does the robot learn to execute successful trajectories for stacking?
Revised: How does the robot learn to execute successful trajectories for stacking?

Original: In model-free Reinforcement Learning(RL) paradigm, the robot will try and try until it is able to stack objects and once it hits a successful instance, it is used as a positive signal (‘reward’) to promote these policy parameters.
Revised: In model-free Reinforcement Learning(RL) paradigm, the robot will try and try until it is able to stack objects and once it hits a successful instance, it is used as a positive signal (‘reward’) to promote these policy parameters.

Original: How does the robot try?
Revised: How does the robot try?

Original: Due to lack of any other signals from the environment, most-often robots use random-exploration policies (or random trajectories).
Revised: Due to lack of any other signals from the environment, most-often robots use random-exploration policies (or random trajectories).

Original: It is clear that if the rewards are sparse, it may take millions of random “tries” before it hits any success.
Revised: It is clear that if the rewards are sparse, it may take millions of random “tries” before it hits any success.

Original: Clearly this approach is only scalable in video-games and not real-world robotics.
Revised: Clearly this approach is only scalable in video-games and not real-world robotics.

Original: Another possibility is to use model-driven approaches.
Revised: Another possibility is to use model-driven approaches.

Original: Here, the robot will learn a model of the world from our millions of interactions and use the model to simulate and search.
Revised: Here, the robot will learn a model of the world from our millions of interactions and use the model to simulate and search.

Original: But what millions of interactions should be performed to build our models?
Revised: But what millions of interactions should be performed to build our models?

Original: Again due to lack of any external information, the most common approach is using random interactions to explore and build the world model (Agrawal et al., 2016; Levine et al., 2016; Pinto et al., 2016).
Revised: Again due to lack of any external information, the most common approach is using random interactions to explore and build the world model (Agrawal et al., 2016; Levine et al., 2016; Pinto et al., 2016).

Original: Building a good model will require enormous number of interactions.
Revised: Building a good model will require enormous number of interactions.

Original: It is clear that one of the biggest stumbling blocks in-front of current robotics is lack of a structured way to explore the world and be efﬁcient in their tries to seek reward or build a model.
Revised: It is clear that one of the biggest stumbling blocks in-front of current robotics is lack of a structured way to explore the world and be efﬁcient in their tries to seek reward or build a model.

Original: Therefore, there has been a lot of signiﬁcant effort to build approaches for exploration and being more sampleefﬁcient than random or on-policy exploration.
Revised: Therefore, there has been a lot of signiﬁcant effort to build approaches for exploration and being more sampleefﬁcient than random or on-policy exploration.

Original: The common theme across these approaches is to introduce “intrinsic” rewards – rewards given by agent to itself based on how environment behaves.
Revised: The common theme across these approaches is to introduce “intrinsic” rewards – rewards given by agent to itself based on how environment behaves.

Original: These rewards are denser compared to external rewards and hence provide early feedback to the exploration policy.
Revised: These rewards are denser compared to external rewards and hence provide early feedback to the exploration policy.

Original: Some examples of intrinsic rewards include “curiosity” (Pathak et al., 2017;
Revised: Some examples of intrinsic rewards include “curiosity” (Pathak et al., 2017;

Original: Oudeyer & Kaplan, 2009; Schmidhuber, 1991) where prediction error is used as reward signal, “diversity rewards” (Eysenbach et al., 2018; Lehman & Stanley, 2011b;a) which discourages the agent from revisiting the same states (or similar states).
Revised: Oudeyer & Kaplan, 2009; Schmidhuber, 1991) where prediction error is used as reward signal, “diversity rewards” (Eysenbach et al., 2018; Lehman & Stanley, 2011b;a) which discourages the agent from revisiting the same states (or similar states).

Original: If the intrinsic rewards are treated as additional
Revised: If the intrinsic rewards are treated as additional

Original: [Figure]
Revised: [Figure]

Original: π f ̂ x t + x t x t x t a t a t a t
Revised: π f ̂ x t + x t x t x t a t a t a t

Original: T E r
Revised: T E r

Original: REINFORCE x t + x t π a t x t a t f ̂ x t + x t + r
Revised: REINFORCE x t + x t π a t x t a t f ̂ x t + x t + r

Original: [Figure]
Revised: [Figure]

Original: rewards in RL and explored in context of original extrinsic rewards; it acts as reward shaping function which improves sample efﬁciency but not by a signiﬁcant amount.
Revised: rewards in RL and explored in context of original extrinsic rewards; it acts as reward shaping function which improves sample efﬁciency but not by a signiﬁcant amount.

Original: Another possibility is to use the inspiration from humans: humans even try to explore the world without the context of task.
Revised: Another possibility is to use the inspiration from humans: humans even try to explore the world without the context of task.

Original: In a similar-manner, we can use these intrinsic rewards to learn taskindependent exploration policy.
Revised: In a similar-manner, we can use these intrinsic rewards to learn taskindependent exploration policy.

Original: This task-independent exploration policy can then be ﬁne-tuned with sparse task-rewards or used to collect data to train model-based RL algorithms.
Revised: This task-independent exploration policy can then be ﬁne-tuned with sparse task-rewards or used to collect data to train model-based RL algorithms.

Original: In fact, in a recent work (Burda et al., 2018), the authors demonstrate how curiosity can be used to train explorationpolicy and then ﬁne-tuned for speciﬁc tasks.
Revised: In fact, in a recent work (Burda et al., 2018), the authors demonstrate how curiosity can be used to train explorationpolicy and then ﬁne-tuned for speciﬁc tasks.

Original: And they demonstrate the power of curiosity ongames.
Revised: And they demonstrate the power of curiosity ongames.

Original: Yes, 54 environments but no real-world physical robots!
Revised: Yes, 54 environments but no real-world physical robots!

Original: Why is that?
Revised: Why is that?

Original: To understand the reason behind sample inefﬁciency of curiosity or intrinsic rewards, notice how the intrinsic rewards are given by agent.
Revised: To understand the reason behind sample inefﬁciency of curiosity or intrinsic rewards, notice how the intrinsic rewards are given by agent.

Original: The agent performs an action and then computes the reward based on its own model and environment behavior.
Revised: The agent performs an action and then computes the reward based on its own model and environment behavior.

Original: For example, in curiosity (Pathak et al., 2017), if the internal model and environment behavior disagree, then the policy is rewarded.
Revised: For example, in curiosity (Pathak et al., 2017), if the internal model and environment behavior disagree, then the policy is rewarded.

Original: From an exploration viewpoint this seems like a good formulation, rewarding actions for which model knows little-to-none.
Revised: From an exploration viewpoint this seems like a good formulation, rewarding actions for which model knows little-to-none.

Original: But the same formulation from an optimization viewpoint, it suffers from all the bad properties of extrinsic rewards.
Revised: But the same formulation from an optimization viewpoint, it suffers from all the bad properties of extrinsic rewards.

Original: The reward is a function of environment behavior with respect to the performed action.
Revised: The reward is a function of environment behavior with respect to the performed action.

Original: Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefﬁcient.
Revised: Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefﬁcient.

Original: Our paper investigates exploration from an optimization viewpoint and asks a simple question: can we formulate curiosity reward as a differentiable function?
Revised: Our paper investigates exploration from an optimization viewpoint and asks a simple question: can we formulate curiosity reward as a differentiable function?

Original: We believe a differentiable reward function would enable to us to be sample efﬁcient and for the ﬁrst time ever, implement exploration on a real-world physical robot.
Revised: We believe a differentiable reward function would enable to us to be sample efﬁcient and for the ﬁrst time ever, implement exploration on a real-world physical robot.

Original: We use a simple yet quite an effective approximation which yields the reward function differentiable.
Revised: We use a simple yet quite an effective approximation which yields the reward function differentiable.

Original: In the end, our optimization is simple min-max over parameters of forward model and policy function results.
Revised: In the end, our optimization is simple min-max over parameters of forward model and policy function results.

Original: Our results indicate that the differentiable reward function formulation is signiﬁcantly better in exploration reaching to interesting interactions in few hundred tries.
Revised: Our results indicate that the differentiable reward function formulation is signiﬁcantly better in exploration reaching to interesting interactions in few hundred tries.

Original: This allows us to implement an exploration policy on a robot and demonstrate pushing and grasping actions using policies trained from scratch.
Revised: This allows us to implement an exploration policy on a robot and demonstrate pushing and grasping actions using policies trained from scratch.

Original:  I NCENTIVIZING E XPLORATION VIA I NTRINSIC R EWARDS
Revised:  I NCENTIVIZING E XPLORATION VIA I NTRINSIC R EWARDS

Original: Consider an agent interacting with the environment E .
Revised: Consider an agent interacting with the environment E .

Original: At time t , it receives the observation x t and then takes an action predicted by its policy, i.e., a t = π ( x t ; θ P ) .
Revised: At time t , it receives the observation x t and then takes an action predicted by its policy, i.e., a t = π ( x t ; θ P ) .

Original: Upon executing the action, it receives, in return, the next observation x t +1 which is ‘generated’ by the environment.
Revised: Upon executing the action, it receives, in return, the next observation x t +1 which is ‘generated’ by the environment.

Original: Our goal is to build an agent that chooses its action in order to maximally explore the state space of the environment in an efﬁcient manner.
Revised: Our goal is to build an agent that chooses its action in order to maximally explore the state space of the environment in an efﬁcient manner.

Original: A popular way to train an agent to perform efﬁcient exploration is to incentivize it by giving feedback on the samples it already executed in the environment via “intrinsic” rewards, i.e., the rewards generated by the agent itself.
Revised: A popular way to train an agent to perform efﬁcient exploration is to incentivize it by giving feedback on the samples it already executed in the environment via “intrinsic” rewards, i.e., the rewards generated by the agent itself.

Original: Strategies for Intrinsic Reward Generation:
Revised: Strategies for Intrinsic Reward Generation:

Original: Given the agent’s transition { x t , a t , x t +1 } , let the intrinsic reward generated by the agent be r i ( x t , a t , x t +1 ) abbvt.
Revised: Given the agent’s transition { x t , a t , x t +1 } , let the intrinsic reward generated by the agent be r i ( x t , a t , x t +1 ) abbvt.

Original: as r it .
Revised: as r it .

Original: A good intrinsic reward formulation would be the one that encourages the agent to perform actions that lead to most informative examples.
Revised: A good intrinsic reward formulation would be the one that encourages the agent to perform actions that lead to most informative examples.

Original: In general, the reward function is considered black-box since it involves environment.
Revised: In general, the reward function is considered black-box since it involves environment.

Original: In this paper, we demonstrate that a simple gradient approximation for maximizing rewards which are intrinsic to the agent leads to a much more sample-efﬁcient policy optimization procedure.
Revised: In this paper, we demonstrate that a simple gradient approximation for maximizing rewards which are intrinsic to the agent leads to a much more sample-efﬁcient policy optimization procedure.

Original: While our formulation could be applied to a more general set of intrinsic rewards, we describe it in the context of prediction-error based curiosity rewards which has recently been shown to be successful across a large variety of simulated environments (Burda et al., 2018).
Revised: While our formulation could be applied to a more general set of intrinsic rewards, we describe it in the context of prediction-error based curiosity rewards which has recently been shown to be successful across a large variety of simulated environments (Burda et al., 2018).

Original: The prediction-error based curiosity reward formulation (Pathak et al., 2017) involves ﬁrst building a prediction model, aka forward model, f θ F of the environment.
Revised: The prediction-error based curiosity reward formulation (Pathak et al., 2017) involves ﬁrst building a prediction model, aka forward model, f θ F of the environment.

Original: Forward model is trained to map the current observation x t and a t to the resulting state x t +1 using maximum likelihood loss and hence it can be learned efﬁciently.
Revised: Forward model is trained to map the current observation x t and a t to the resulting state x t +1 using maximum likelihood loss and hence it can be learned efﬁciently.

Original: The agent’s policy is trained to select the actions a t which result in high loss for the forward model.
Revised: The agent’s policy is trained to select the actions a t which result in high loss for the forward model.

Original: For instance, in deterministic environment, a gaussian density model can be used to deﬁne intrinsic reward r it for the agent:
Revised: For instance, in deterministic environment, a gaussian density model can be used to deﬁne intrinsic reward r it for the agent:

Original: [Equation]
Revised: [Equation]

Original: The agent is trained to maximize r it and the forward model is simultaneously trained in online manner on the data collected by the agent during exploration.
Revised: The agent is trained to maximize r it and the forward model is simultaneously trained in online manner on the data collected by the agent during exploration.

Original: One commonality among different exploration methods (Bellemare et al., 2016; Pathak et al., 2017; Houthooft et al., 2016a), is that the forward model is usually learned in a supervised manner and the agent’s policy is trained using reinforcement learning either in on-policy or off-policy manner.
Revised: One commonality among different exploration methods (Bellemare et al., 2016; Pathak et al., 2017; Houthooft et al., 2016a), is that the forward model is usually learned in a supervised manner and the agent’s policy is trained using reinforcement learning either in on-policy or off-policy manner.

Original: On-Policy Reward Optimization In this case, the policy is directly optimized to maximize the intrinsic reward r it via policy-gradients (Sutton & Barto, 1998).
Revised: On-Policy Reward Optimization In this case, the policy is directly optimized to maximize the intrinsic reward r it via policy-gradients (Sutton & Barto, 1998).

Original: Given the agent’s rollout sequence and the intrinsic reward r it at each timestep t , the policy is trained to maximize the sum of expected reward, i.e., max θ P E π ( x t ; θ P ) (cid:2) (cid:80) t γ t r it (cid:3) discounted by a factor γ .
Revised: Given the agent’s rollout sequence and the intrinsic reward r it at each timestep t , the policy is trained to maximize the sum of expected reward, i.e., max θ P E π ( x t ; θ P ) (cid:2) (cid:80) t γ t r it (cid:3) discounted by a factor γ .

Original: The gradients of this expression are computed using REINFORCE (Williams, 1992).
Revised: The gradients of this expression are computed using REINFORCE (Williams, 1992).

Original: In practice, the existing on-policy algorithms, e.g., A3C (Mnih et al., 2016), PPO (Schulman et al., 2017) etc. are deployed off-the shelf.
Revised: In practice, the existing on-policy algorithms, e.g., A3C (Mnih et al., 2016), PPO (Schulman et al., 2017) etc. are deployed off-the shelf.

Original: Off-Policy Reward Optimization Off-policy algorithms (e.g. DDPG (Lillicrap et al., 2016) or
Revised: Off-Policy Reward Optimization Off-policy algorithms (e.g. DDPG (Lillicrap et al., 2016) or

Original: DQN (Mnih et al., 2015a)) use Q-value function to represent agent’s policy.
Revised: DQN (Mnih et al., 2015a)) use Q-value function to represent agent’s policy.

Original: Q-value, for a given pair of the current state x t and the executed action a t , is deﬁned as the total sum of discounted future rewards.
Revised: Q-value, for a given pair of the current state x t and the executed action a t , is deﬁned as the total sum of discounted future rewards.

Original: Agent’s policy can be written in terms of this Q-value function as a t =
Revised: Agent’s policy can be written in terms of this Q-value function as a t =

Original: π ( x t ; θ P ) =
Revised: π ( x t ; θ P ) =

Original: arg max a t Q ( x t , a t ; θ P ) plus epsilon-noise.
Revised: arg max a t Q ( x t , a t ; θ P ) plus epsilon-noise.

Original: The main beneﬁt in learning Q-values is that they can be trained with samples from buffer even if they are not from the agent’s current policy to minimize the following loss: (cid:107) Q ( x t , a t ; θ P ) − E π ( x t ; θ P ) (cid:2) (cid:80) t γ t
Revised: The main beneﬁt in learning Q-values is that they can be trained with samples from buffer even if they are not from the agent’s current policy to minimize the following loss: (cid:107) Q ( x t , a t ; θ P ) − E π ( x t ; θ P ) (cid:2) (cid:80) t γ t

Original: r
Revised: r

Original: it (cid:3) (cid:107) 2 .
Revised: it (cid:3) (cid:107) 2 .

Original: However, since intrinsic reward distribution r it changes over time one would need to update the intrinsic reward using the most recent f θ F .
Revised: However, since intrinsic reward distribution r it changes over time one would need to update the intrinsic reward using the most recent f θ F .

Original: The key thing to note is that both the approaches to policy optimization treat reward r it as an unknown quantity which can only be estimated via samples.
Revised: The key thing to note is that both the approaches to policy optimization treat reward r it as an unknown quantity which can only be estimated via samples.

Original:  S AMPLE -E FFICIENT D IFFERENTIABLE E XPLORATION
Revised:  S AMPLE -E FFICIENT D IFFERENTIABLE E XPLORATION

Original: As discussed in the previous section, there has been a lot of work in proposing formulations for intrinsic rewards to the agent.
Revised: As discussed in the previous section, there has been a lot of work in proposing formulations for intrinsic rewards to the agent.

Original: However, the optimization procedure for training policies to maximize these intrinsic rewards has more or less remained the same – i.e. – treating the intrinsic reward as a “black-box” even though it is generated by the agent itself.
Revised: However, the optimization procedure for training policies to maximize these intrinsic rewards has more or less remained the same – i.e. – treating the intrinsic reward as a “black-box” even though it is generated by the agent itself.

Original: Let’s consider an example to understand the reason behind the status quo.
Revised: Let’s consider an example to understand the reason behind the status quo.

Original: Consider a robotic-arm agent trying to push multiple objects kept on the table in front of it by looking at the image from an overhead camera.
Revised: Consider a robotic-arm agent trying to push multiple objects kept on the table in front of it by looking at the image from an overhead camera.

Original: Suppose the arm pushes an object such that it collides with another one on the table.
Revised: Suppose the arm pushes an object such that it collides with another one on the table.

Original: The resulting image observation following this action will be the outcome of complex real-world interaction the actual dynamics of which is not known to the agent.
Revised: The resulting image observation following this action will be the outcome of complex real-world interaction the actual dynamics of which is not known to the agent.

Original: More importantly, note that this resulting image observation is a function of the agent’s action (i.e., push in this case).
Revised: More importantly, note that this resulting image observation is a function of the agent’s action (i.e., push in this case).

Original: Since the intrinsic reward r i ( x t , a t , x t +1 ) is function of the next state (which is a function of the agent’s action) .
Revised: Since the intrinsic reward r i ( x t , a t , x t +1 ) is function of the next state (which is a function of the agent’s action) .

Original: This dependency on the unknown environment dynamics absolves the policy optimization to compute any sort of analytical gradient with respect to the intrinsic rewards.
Revised: This dependency on the unknown environment dynamics absolves the policy optimization to compute any sort of analytical gradient with respect to the intrinsic rewards.

Original: To state this intuition mathematically, let T E be the true transition function of the environment
Revised: To state this intuition mathematically, let T E be the true transition function of the environment

Original: such that x t +1 =
Revised: such that x t +1 =

Original: T E ( x t , a t ) .
Revised: T E ( x t , a t ) .

Original: Note that for environments with stochastic dynamics T E will be a distribution, but we describe deterministic case for brevity of the notations.
Revised: Note that for environments with stochastic dynamics T E will be a distribution, but we describe deterministic case for brevity of the notations.

Original: The intrinsic reward function r it , in
Revised: The intrinsic reward function r it , in

Original: Equation (1), can now be rephrased in terms of T E as r i ( x t , a t , T E ( x t , a t )) .
Revised: Equation (1), can now be rephrased in terms of T E as r i ( x t , a t , T E ( x t , a t )) .

Original: It is easy to see that T E is required to compute analytical gradients
Revised: It is easy to see that T E is required to compute analytical gradients

Original: ∂r it /∂a t at any time t .
Revised: ∂r it /∂a t at any time t .

Original: The standard way to optimize policy to maximize sequence of intrinsic rewards is to either use
Revised: The standard way to optimize policy to maximize sequence of intrinsic rewards is to either use

Original: REINFORCE (i.e., on-policy way) or regress to r it to learn value estimates (i.e., off-policy) as discussed in the previous section.
Revised: REINFORCE (i.e., on-policy way) or regress to r it to learn value estimates (i.e., off-policy) as discussed in the previous section.

Original: Both these approaches do not make any use of the structure present in the design of r it .
Revised: Both these approaches do not make any use of the structure present in the design of r it .

Original: While these are unbiased estimators for training policy parameters with respect to r it , they suffer from high variance which is a known issue in reinforcement learning and an active area of research (Schulman et al., 2015).
Revised: While these are unbiased estimators for training policy parameters with respect to r it , they suffer from high variance which is a known issue in reinforcement learning and an active area of research (Schulman et al., 2015).

Original: For instance, REINFORCE roughly amounts to saying that the agent should change the probability of the executed action in proportion to rewards received which ﬂuctuates with the reward trajectories, leading to high variance.
Revised: For instance, REINFORCE roughly amounts to saying that the agent should change the probability of the executed action in proportion to rewards received which ﬂuctuates with the reward trajectories, leading to high variance.

Original: It gives no signal as to what action to take if the current action did not lead to a good reward.
Revised: It gives no signal as to what action to take if the current action did not lead to a good reward.

Original: 3. D IFFERENTIABLE I NTRINSIC R EWARD WITH L OCAL C ONSTANCY A PPROXIMATION
Revised: 3. D IFFERENTIABLE I NTRINSIC R EWARD WITH L OCAL C ONSTANCY A PPROXIMATION

Original: The focus of this paper is on the policy optimization aspect of the intrinsic reward function rather than their formulation.
Revised: The focus of this paper is on the policy optimization aspect of the intrinsic reward function rather than their formulation.

Original: Our goal is to address the question whether we can formulate intrinsic reward as a differentiable function so as to perform policy optimization using likelihood maximization – much like supervised learning instead of reinforcement.
Revised: Our goal is to address the question whether we can formulate intrinsic reward as a differentiable function so as to perform policy optimization using likelihood maximization – much like supervised learning instead of reinforcement.

Original: If possible, this would allow the agent to make use of the structure in r it explicitly.
Revised: If possible, this would allow the agent to make use of the structure in r it explicitly.

Original: For instance, in case of curiosity-driven intrinsic reward, the forward predictor f θ F is trained via maximum likelihood which means it can be learned with much more sample efﬁciency.
Revised: For instance, in case of curiosity-driven intrinsic reward, the forward predictor f θ F is trained via maximum likelihood which means it can be learned with much more sample efﬁciency.

Original: If the policy could also be optimized using direct gradients, the rewarder could very efﬁciently inform the agent to change its action space in the direction where forward prediction loss is high, instead of providing a scalar feedback as in case of reinforcement learning.
Revised: If the policy could also be optimized using direct gradients, the rewarder could very efﬁciently inform the agent to change its action space in the direction where forward prediction loss is high, instead of providing a scalar feedback as in case of reinforcement learning.

Original: Explicit reward (cost) functions are one of the key reasons for success stories in optimal-control based robotics (Deisenroth & Rasmussen, 2011b; Gal et al., 2016), but they don’t scale to high-dimensional state space such as images and rely on having access to a good model of the environment.
Revised: Explicit reward (cost) functions are one of the key reasons for success stories in optimal-control based robotics (Deisenroth & Rasmussen, 2011b; Gal et al., 2016), but they don’t scale to high-dimensional state space such as images and rely on having access to a good model of the environment.

Original: To address our goal, let us revisit the intrinsic reward function in Equation (1).
Revised: To address our goal, let us revisit the intrinsic reward function in Equation (1).

Original: Upon substituting π θ P and T E , we get:
Revised: Upon substituting π θ P and T E , we get:

Original: [Equation]
Revised: [Equation]

Original: Note that the ﬁrst term in the Equation (2) is differentiable since f θ F is a learned function, but second term is not since the environment is not known.
Revised: Note that the ﬁrst term in the Equation (2) is differentiable since f θ F is a learned function, but second term is not since the environment is not known.

Original: Hence, we propose to make a local zeroth-order approximation to the environment so as to make r it differentiable with respect to the action a t .
Revised: Hence, we propose to make a local zeroth-order approximation to the environment so as to make r it differentiable with respect to the action a t .

Original: Local constancy approximation Consider a state x t and the action a t predicted by the policy at that state, a t = π ( x t ; θ P ) .
Revised: Local constancy approximation Consider a state x t and the action a t predicted by the policy at that state, a t = π ( x t ; θ P ) .

Original: The agent reaches x t +1 upon executing the action.
Revised: The agent reaches x t +1 upon executing the action.

Original: We assume that the ﬁnal state x t +1 remains constant in an inﬁnitely small (cid:15)
Revised: We assume that the ﬁnal state x t +1 remains constant in an inﬁnitely small (cid:15)

Original: -neighborhood
Revised: -neighborhood

Original: ball around the action a t executed at the same initial state x t .
Revised: ball around the action a t executed at the same initial state x t .

Original: Mathematically, it can be written as x t +1 =
Revised: Mathematically, it can be written as x t +1 =

Original: T E ( x t , a t )
Revised: T E ( x t , a t )

Original: ≈ T E ( x t , ˜ a t ) for ˜ a t ∈ N (cid:15) ( a t ) where (cid:15) → 0 .
Revised: ≈ T E ( x t , ˜ a t ) for ˜ a t ∈ N (cid:15) ( a t ) where (cid:15) → 0 .

Original: With this approximation in place, we can now compute ∂r it /∂a t at any time t by back propagating gradients through the forward function.
Revised: With this approximation in place, we can now compute ∂r it /∂a t at any time t by back propagating gradients through the forward function.

Original: Intuitively, it basically assumes that minute ﬂuctuations in action won’t change the transition state.
Revised: Intuitively, it basically assumes that minute ﬂuctuations in action won’t change the transition state.

Original: Of course, this does not hold true in certain corner cases, for instance, pushing an object at the very corner edge.
Revised: Of course, this does not hold true in certain corner cases, for instance, pushing an object at the very corner edge.

Original: In those cases, intrinsic reward formulation with local constancy approximation would incentivize all the actions equally in the (cid:15) -neighborhood ball, instead of that particular action.
Revised: In those cases, intrinsic reward formulation with local constancy approximation would incentivize all the actions equally in the (cid:15) -neighborhood ball, instead of that particular action.

Original: This, fortunately, seems like a reasonable settlement in practice as such corner cases require higher sampling to understand where exactly the corner case occurs.
Revised: This, fortunately, seems like a reasonable settlement in practice as such corner cases require higher sampling to understand where exactly the corner case occurs.

Original: 3. D IRECT P OLICY O PTIMIZATION WITH D IFFERENTIABLE I NTRINSIC R EWARDS
Revised: 3. D IRECT P OLICY O PTIMIZATION WITH D IFFERENTIABLE I NTRINSIC R EWARDS

Original: We now leverage the local constancy approximation to formulate a direct gradient-based policy optimization which maximizes intrinsic rewards achieved by the agent without relying on reinforcement learning for policy optimization.
Revised: We now leverage the local constancy approximation to formulate a direct gradient-based policy optimization which maximizes intrinsic rewards achieved by the agent without relying on reinforcement learning for policy optimization.

Original: We ﬁrst discuss the one step case and then provide the general setup.
Revised: We ﬁrst discuss the one step case and then provide the general setup.

Original: Given a transition { x t , a t , x t +1 } , the agent generates an intrinsic reward for itself r it = (cid:107) f
Revised: Given a transition { x t , a t , x t +1 } , the agent generates an intrinsic reward for itself r it = (cid:107) f

Original: ( x t , a t ; θ F ) − x t +1 (cid:107) 2 .
Revised: ( x t , a t ; θ F ) − x t +1 (cid:107) 2 .

Original: Forward model f
Revised: Forward model f

Original: θ F is trained to minimize its loss which amounts to minimizing r it with respect to θ F .
Revised: θ F is trained to minimize its loss which amounts to minimizing r it with respect to θ F .

Original: This is done via direct loss minimization using gradient descent.
Revised: This is done via direct loss minimization using gradient descent.

Original: Upon using local constancy approximation , we can also optimize for policy parameters θ P via differentiable loss function in the same manner using gradient descent.
Revised: Upon using local constancy approximation , we can also optimize for policy parameters θ P via differentiable loss function in the same manner using gradient descent.

Original: However, policy is optimized to maximize the objective.
Revised: However, policy is optimized to maximize the objective.

Original: This results into a min-max optimization over r it .
Revised: This results into a min-max optimization over r it .

Original: The joint objective for a one-step reward horizon is written as follows:
Revised: The joint objective for a one-step reward horizon is written as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that both policy and forward predictor are trained via maximum likelihood in a supervised manner.
Revised: Note that both policy and forward predictor are trained via maximum likelihood in a supervised manner.

Original: Hence, given the local constancy approximation, this should be a much more efﬁcient way to optimize exploration policy unlike reinforcement learning based policy-optimization.
Revised: Hence, given the local constancy approximation, this should be a much more efﬁcient way to optimize exploration policy unlike reinforcement learning based policy-optimization.

Original: We optimize the objective in Equation 3 in an alternating fashion where forward predictor is optimized in the outer-loop, and the policy in inner-loop.
Revised: We optimize the objective in Equation 3 in an alternating fashion where forward predictor is optimized in the outer-loop, and the policy in inner-loop.

Original: Generalization to multi-step reward horizon To optimize policy to maximize a discounted sum of sequence of future intrinsic rewards r it in a differentiable manner, the forward predictor would have to make predictions spanning over multiple time-steps.
Revised: Generalization to multi-step reward horizon To optimize policy to maximize a discounted sum of sequence of future intrinsic rewards r it in a differentiable manner, the forward predictor would have to make predictions spanning over multiple time-steps.

Original: The objective from Equation (3) can be generalized to the multi-step horizon setup by recursively applying the forward predictor as (cid:80) t (cid:107)
Revised: The objective from Equation (3) can be generalized to the multi-step horizon setup by recursively applying the forward predictor as (cid:80) t (cid:107)

Original: ˆ x t +1 − x t +1 (cid:107) 2 where ˆ x t +

Revised: ˆ x t +1 − x t +1 (cid:107) 2 where ˆ x t +

Original: = f
Revised: = f

Original: (ˆ x t , a t ; θ F ) and ˆ x 0 = x 0 .
Revised: (ˆ x t , a t ; θ F ) and ˆ x 0 = x 0 .

Original: Alternatively, one could use LSTM to make forward model itself multi-step.
Revised: Alternatively, one could use LSTM to make forward model itself multi-step.

Original: 3. P RACTICAL CONSIDERATIONS IN IMPLEMENTATION
Revised: 3. P RACTICAL CONSIDERATIONS IN IMPLEMENTATION

Original: Min-max optimization At ﬁrst glance, it might appear that optimizing the objective in Equation (3) could be unstable learning process.
Revised: Min-max optimization At ﬁrst glance, it might appear that optimizing the objective in Equation (3) could be unstable learning process.

Original: However, unlike the other cases of online min-max optimization (e.g. Generative Adversarial Networks (Goodfellow et al., 2014)), this case could be easily made stable.
Revised: However, unlike the other cases of online min-max optimization (e.g. Generative Adversarial Networks (Goodfellow et al., 2014)), this case could be easily made stable.

Original: Here, the goal of our policy optimization is to learn from the forward predictor in the outer-loop and the forward predictor is to improve itself.
Revised: Here, the goal of our policy optimization is to learn from the forward predictor in the outer-loop and the forward predictor is to improve itself.

Original: Hence, we train the forward predictor slightly faster than the policy by keeping higher learning rate to stabilize the learning process.
Revised: Hence, we train the forward predictor slightly faster than the policy by keeping higher learning rate to stabilize the learning process.

Original: Other alternative could be to take few extra gradient steps in the outer-loop minimization than inner-loop maximization.
Revised: Other alternative could be to take few extra gradient steps in the outer-loop minimization than inner-loop maximization.

Original: Back-propagation through forward predictor To directly optimize the policy with respect to the loss function of the forward predictor, we need to backpropagate all the way through action sampling process from the policy.
Revised: Back-propagation through forward predictor To directly optimize the policy with respect to the loss function of the forward predictor, we need to backpropagate all the way through action sampling process from the policy.

Original: In case of continuous action space, one could achieve this via making policy deterministic, i.e. a t =
Revised: In case of continuous action space, one could achieve this via making policy deterministic, i.e. a t =

Original: π θ P with epsilon greedy sampling (Lillicrap et al., 2016).
Revised: π θ P with epsilon greedy sampling (Lillicrap et al., 2016).

Original: Alternatively, in case of discrete action space, we found that straight-through estimator (Bengio et al., 2013) works well in practice.
Revised: Alternatively, in case of discrete action space, we found that straight-through estimator (Bengio et al., 2013) works well in practice.

Original: In this paper, we discretized the action space of our agent.
Revised: In this paper, we discretized the action space of our agent.

Original: Learning forward predictions in the feature space It has been shown that learning forwarddynamics predictor f θ P (Burda et al., 2018; Pathak et al., 2017) in some feature space leads to better generalization instead of making predictions in raw pixel space.
Revised: Learning forward predictions in the feature space It has been shown that learning forwarddynamics predictor f θ P (Burda et al., 2018; Pathak et al., 2017) in some feature space leads to better generalization instead of making predictions in raw pixel space.

Original: Our formulation is trivially extensible to any representation space because all the operations can be performed with φ ( x t ) instead of x t .
Revised: Our formulation is trivially extensible to any representation space because all the operations can be performed with φ ( x t ) instead of x t .

Original:  E XPERIMENTAL S ETUP AND B ASELINES
Revised:  E XPERIMENTAL S ETUP AND B ASELINES

Original: We consider the task of object manipulation in complex scenarios.
Revised: We consider the task of object manipulation in complex scenarios.

Original: Our setup contains a 7-DOF robotic arm which could be tasked to interact with the the objects kept on the table in front of it.
Revised: Our setup contains a 7-DOF robotic arm which could be tasked to interact with the the objects kept on the table in front of it.

Original: The objects are kept randomly in the workspace of the robot on the table.
Revised: The objects are kept randomly in the workspace of the robot on the table.

Original: All of our experiments use raw visual RGBD images as input and predict raw actions as output.
Revised: All of our experiments use raw visual RGBD images as input and predict raw actions as output.

Original: We use position-control for controlling the robotic arm and its action space contains following actions: (a) { X, Y } : the target location in the work-space of the end-effector; (b) Θ : the angle at which gripper of the robot should approach the speciﬁed location; (c) Gripper Status: a boolean value indicating whether to perform a grasping (open the gripper ﬁngers) or pushing gesture (keep ﬁngers close).
Revised: We use position-control for controlling the robotic arm and its action space contains following actions: (a) { X, Y } : the target location in the work-space of the end-effector; (b) Θ : the angle at which gripper of the robot should approach the speciﬁed location; (c) Gripper Status: a boolean value indicating whether to perform a grasping (open the gripper ﬁngers) or pushing gesture (keep ﬁngers close).

Original: Note that in order to perform an accurate grasp or push on objects, the agent has to ﬁgure out accurate location, orientation and the gripper status.
Revised: Note that in order to perform an accurate grasp or push on objects, the agent has to ﬁgure out accurate location, orientation and the gripper status.

Original: We parametrize the action space in the network in a similar manner as (Zeng et al., 2018).
Revised: We parametrize the action space in the network in a similar manner as (Zeng et al., 2018).

Original: The action space is discretized into 224 × 224 { X, Y } locations, 16 orientations for grasping (ﬁngers close) and 16 orientations for pushing.
Revised: The action space is discretized into 224 × 224 { X, Y } locations, 16 orientations for grasping (ﬁngers close) and 16 orientations for pushing.

Original: Input to the policy is a 224 × 224 RGBD image
Revised: Input to the policy is a 224 × 224 RGBD image

Original: and it produces probabilities of the push and grasp action for each input pixel location.
Revised: and it produces probabilities of the push and grasp action for each input pixel location.

Original: Instead of adding therotations in the output, we pass 16 equally spaced rotated inputs to the network and then sample actions based on the output of all the inputs.
Revised: Instead of adding therotations in the output, we pass 16 equally spaced rotated inputs to the network and then sample actions based on the output of all the inputs.

Original: This exploits the convolutional structure of the network.
Revised: This exploits the convolutional structure of the network.

Original: There is no assumption of any sort about either the environment, or the training signal.
Revised: There is no assumption of any sort about either the environment, or the training signal.

Original: Our robotic agents explore the work-space purely out of their own intrinsic reward in a pursuit to develop useful skills.
Revised: Our robotic agents explore the work-space purely out of their own intrinsic reward in a pursuit to develop useful skills.

Original: We have an instantiation of this setup both in the real world and in a simulated environment.
Revised: We have an instantiation of this setup both in the real world and in a simulated environment.

Original: Baselines In this paper, we propose a sample efﬁcient way to optimize policy using intrinsic rewards that does not treat the reward function as a black box.
Revised: Baselines In this paper, we propose a sample efﬁcient way to optimize policy using intrinsic rewards that does not treat the reward function as a black box.

Original: As baselines, we compare the same curiosity reward function but optimized using REINFORCE.
Revised: As baselines, we compare the same curiosity reward function but optimized using REINFORCE.

Original: We also compare to off-policy baseline of DQN.
Revised: We also compare to off-policy baseline of DQN.

Original: In order to make apples-to-apples comparison between the proposed differentiable learning optimization vs. existing black-box optimization, We kept exactly the same setup for all the methods and only changed the optimization procedure.
Revised: In order to make apples-to-apples comparison between the proposed differentiable learning optimization vs. existing black-box optimization, We kept exactly the same setup for all the methods and only changed the optimization procedure.

Original: For obtaining the right hyper-parameters we use
Revised: For obtaining the right hyper-parameters we use

Original: REINFORCE with external touch-rewards (See Appendix A).
Revised: REINFORCE with external touch-rewards (See Appendix A).

Original: As one can see the REINFORCE baseline is quite effective with external rewards.
Revised: As one can see the REINFORCE baseline is quite effective with external rewards.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our goal is to demonstrate a exploration formulation which is sample-efﬁcient enough to be applicable in real-world robotic setups.
Revised: Our goal is to demonstrate a exploration formulation which is sample-efﬁcient enough to be applicable in real-world robotic setups.

Original: The two main components of our proposed methodology are the exploration policy and the forward prediction model.
Revised: The two main components of our proposed methodology are the exploration policy and the forward prediction model.

Original: We evaluate both the components on object manipulation tasks: (a) in a simulated V-REP based environment, and (b) in real-world robotic setup using Sawyer robot arm.
Revised: We evaluate both the components on object manipulation tasks: (a) in a simulated V-REP based environment, and (b) in real-world robotic setup using Sawyer robot arm.

Original: We perform simulation experiments to help us setup the right parameters and do extensive comparisons against REINFORCE and DQN.
Revised: We perform simulation experiments to help us setup the right parameters and do extensive comparisons against REINFORCE and DQN.

Original: We consider two environment setups: sparse and non-sparse.
Revised: We consider two environment setups: sparse and non-sparse.

Original: In the non-sparse setup, the environment contains multiple objects which makes it easier for the policy to stumble upon the object.
Revised: In the non-sparse setup, the environment contains multiple objects which makes it easier for the policy to stumble upon the object.

Original: In the sparse setup, the work-space of the robot only contains one object.
Revised: In the sparse setup, the work-space of the robot only contains one object.

Original: 5. O BJECT M ANIPULATION IN S IMULATION
Revised: 5. O BJECT M ANIPULATION IN S IMULATION

Original: Our main goal is to deploy the exploration policy in real-world, but we ﬁrst begin by studying in depth the performance of our proposed approach in contrast to prior formulations that treat intrinsic reward function as a black-box.
Revised: Our main goal is to deploy the exploration policy in real-world, but we ﬁrst begin by studying in depth the performance of our proposed approach in contrast to prior formulations that treat intrinsic reward function as a black-box.

Original: Simulated Object Interaction Setup We used V-REP simulator to simulate the robot performing grasping and pushing on table top environment.
Revised: Simulated Object Interaction Setup We used V-REP simulator to simulate the robot performing grasping and pushing on table top environment.

Original: This setup is based on (Zeng et al., 2018).
Revised: This setup is based on (Zeng et al., 2018).

Original: It consist of UR5 robot arm with an RG2 gripper.
Revised: It consist of UR5 robot arm with an RG2 gripper.

Original: Dynamics is simulated using Bullet Physics 2.83 physics engine.
Revised: Dynamics is simulated using Bullet Physics 2.83 physics engine.

Original: V-REPs internal inverse kinematics module is used robot motion planning.
Revised: V-REPs internal inverse kinematics module is used robot motion planning.

Original: The objects used in these simulations include 6 3D toy blocks of different shapes, colors of which are randomly chosen during experiments.
Revised: The objects used in these simulations include 6 3D toy blocks of different shapes, colors of which are randomly chosen during experiments.

Original: Perception data is captured using a statically mounted 3D camera in the environment.
Revised: Perception data is captured using a statically mounted 3D camera in the environment.

Original: It provides RGBD images (640x480), without any noise added for depth or color.
Revised: It provides RGBD images (640x480), without any noise added for depth or color.

Original: Evaluation of the exploration policy learned by the agent How do we evaluate if our exploration policy is taking interesting steps?
Revised: Evaluation of the exploration policy learned by the agent How do we evaluate if our exploration policy is taking interesting steps?

Original: In our setup, one attribute that correlates with interesting-ness is performing actions on object.
Revised: In our setup, one attribute that correlates with interesting-ness is performing actions on object.

Original: We use this as a metric to explore how quickly our policy learns to explore interesting part of space.
Revised: We use this as a metric to explore how quickly our policy learns to explore interesting part of space.

Original: Figures 2a and 2b show the performance when the environment consist of single and multiple objects respectively.
Revised: Figures 2a and 2b show the performance when the environment consist of single and multiple objects respectively.

Original: It is clear that both REINFORCE and DQN perform quite poorly and do not show any signiﬁcant improvement even after 3K interactions.
Revised: It is clear that both REINFORCE and DQN perform quite poorly and do not show any signiﬁcant improvement even after 3K interactions.

Original: On the other hand, our approach is able to exploit the structure in the loss to learn signiﬁcantly faster.
Revised: On the other hand, our approach is able to exploit the structure in the loss to learn signiﬁcantly faster.

Original: and achieves 40% interaction rate even after 3K interactions.
Revised: and achieves 40% interaction rate even after 3K interactions.

Original: Table 2 shows the interaction results in multi-step setting after ﬁrst 250 interactions.
Revised: Table 2 shows the interaction results in multi-step setting after ﬁrst 250 interactions.

Original: Again, our multi-step formulation is more effective compared to baseline approaches.
Revised: Again, our multi-step formulation is more effective compared to baseline approaches.

Original: [Figure]
Revised: [Figure]

Original: 5. O BJECT M ANIPULATION USING R OBOTIC A RM IN THE R EAL -W ORLD
Revised: 5. O BJECT M ANIPULATION USING R OBOTIC A RM IN THE R EAL -W ORLD

Original: We now deploy our sample-efﬁcient exploration formulation on real-world robotic setup.
Revised: We now deploy our sample-efﬁcient exploration formulation on real-world robotic setup.

Original: The realworld poses additional challenges unlike simulated environments in terms of behavior, dynamics of varied objects available in the real world.
Revised: The realworld poses additional challenges unlike simulated environments in terms of behavior, dynamics of varied objects available in the real world.

Original: Our robotic setup consisted of a Sawyer-arm with a table placed in front of it.
Revised: Our robotic setup consisted of a Sawyer-arm with a table placed in front of it.

Original: We mounted Kinectv2 at a ﬁxed location from robot to receive RGBD observation of the environment.
Revised: We mounted Kinectv2 at a ﬁxed location from robot to receive RGBD observation of the environment.

Original: We calibrated the system to get extrinsic between camera and robot.
Revised: We calibrated the system to get extrinsic between camera and robot.

Original: In every run robot starts with 3 object placed in front of it (See Appendix for setup).
Revised: In every run robot starts with 3 object placed in front of it (See Appendix for setup).

Original: Objects were manually replaced if robot has completed 100 interactions or if there are no objects in front of it.
Revised: Objects were manually replaced if robot has completed 100 interactions or if there are no objects in front of it.

Original: To see if the robot has interacted with objects, we used to monitor the change in the RGB image.
Revised: To see if the robot has interacted with objects, we used to monitor the change in the RGB image.

Original: This information is only used for our purpose to check the progress of robot, it is not provided to policy.
Revised: This information is only used for our purpose to check the progress of robot, it is not provided to policy.

Original: In order to test the skills learned by the robot during its curious exploration, we tested it on a set of held-out objects.
Revised: In order to test the skills learned by the robot during its curious exploration, we tested it on a set of held-out objects.

Original: Out of total of 30 objects, we created set of 20 objects for training and 10 objects for testing.
Revised: Out of total of 30 objects, we created set of 20 objects for training and 10 objects for testing.

Original: Both, our method and reinforce were trained for 1400 robot interaction with the environment.
Revised: Both, our method and reinforce were trained for 1400 robot interaction with the environment.

Original: Both models were evaluated based on the 80 robot interaction.
Revised: Both models were evaluated based on the 80 robot interaction.

Original: During testing , environment reset was done after every 10 robot steps.
Revised: During testing , environment reset was done after every 10 robot steps.

Original: How good is Exploration Policy?
Revised: How good is Exploration Policy?

Original: The key requirement of a good exploration policy is that it should search the space in an efﬁcient manner, generalize to unseen scenarios and discover complex behaviors which are hard to stumble upon randomly.
Revised: The key requirement of a good exploration policy is that it should search the space in an efﬁcient manner, generalize to unseen scenarios and discover complex behaviors which are hard to stumble upon randomly.

Original: Again we use the same metric (interaction with objects) as before to measure effectiveness of exploration policy.
Revised: Again we use the same metric (interaction with objects) as before to measure effectiveness of exploration policy.

Original: Figure 3(left) shows how effective our differentiable curiosity module is and how it learns to interact with object even with 1400 examples.
Revised: Figure 3(left) shows how effective our differentiable curiosity module is and how it learns to interact with object even with 1400 examples.

Original: This result clearly indicates the importance of using the approximation and hence differentiable reward function.
Revised: This result clearly indicates the importance of using the approximation and hence differentiable reward function.

Original: At the end of 1400 steps, the interaction rate was more than 80%.
Revised: At the end of 1400 steps, the interaction rate was more than 80%.

Original: Our ﬁnal trained exploration policy interacts approximately 91% of times with unseen objects whereas random performs 17%.
Revised: Our ﬁnal trained exploration policy interacts approximately 91% of times with unseen objects whereas random performs 17%.

Original: On the other hand, it seems that REINFORCE just collapses and only 1% of actions involve interaction with object (See Table 3).
Revised: On the other hand, it seems that REINFORCE just collapses and only 1% of actions involve interaction with object (See Table 3).

Original: How good is Forward Prediction Model?
Revised: How good is Forward Prediction Model?

Original: We use the data collected during the exploration to train forward prediction models.
Revised: We use the data collected during the exploration to train forward prediction models.

Original: If the data explored by the agent is interesting, the prediction model should perform well on complex tasks.
Revised: If the data explored by the agent is interesting, the prediction model should perform well on complex tasks.

Original: We evaluate how well is the forward prediction model in terms of planning for goal-driven scenarios.
Revised: We evaluate how well is the forward prediction model in terms of planning for goal-driven scenarios.

Original: We also provide the evaluation in terms of future prediction (Finn et al., 2017) in Appendix.
Revised: We also provide the evaluation in terms of future prediction (Finn et al., 2017) in Appendix.

Original: We compare the planning accuracy of the forward model learned on the data collected by different exploration schemes.
Revised: We compare the planning accuracy of the forward model learned on the data collected by different exploration schemes.

Original: We use the cross-entropy based method to optimize for the action plan given the initial state and the goal state.
Revised: We use the cross-entropy based method to optimize for the action plan given the initial state and the goal state.

Original: Metric is L2 distance from the ground truth action location.
Revised: Metric is L2 distance from the ground truth action location.

Original: As result in the Table 1 indicate the forward model learned using our approach is signiﬁcantly better than the baseline REINFORCE model.
Revised: As result in the Table 1 indicate the forward model learned using our approach is signiﬁcantly better than the baseline REINFORCE model.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The problem of exploration is a well-studied problem in the ﬁeld of reinforcement learning.
Revised: The problem of exploration is a well-studied problem in the ﬁeld of reinforcement learning.

Original: Early approaches focused on studying exploration from theoretical perspective (Strehl & Littman, 2008) and proposed Bayesian formulations (Kolter & Ng, 2009; Deisenroth & Rasmussen, 2011a) which are
Revised: Early approaches focused on studying exploration from theoretical perspective (Strehl & Littman, 2008) and proposed Bayesian formulations (Kolter & Ng, 2009; Deisenroth & Rasmussen, 2011a) which are

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Right
Revised: Right

Original: Table 3 shows the interaction rate of ﬁnal model.
Revised: Table 3 shows the interaction rate of ﬁnal model.

Original: hard to scale in higher dimensions.
Revised: hard to scale in higher dimensions.

Original: In this paper, we focus on the speciﬁc problem of exploration using intrinsic rewards.
Revised: In this paper, we focus on the speciﬁc problem of exploration using intrinsic rewards.

Original: A large family of approaches use “curiosity” as a reward for training the agents.
Revised: A large family of approaches use “curiosity” as a reward for training the agents.

Original: A good summary of early work in curiosity-driven rewards can be found in (Oudeyer et al., 2007;
Revised: A good summary of early work in curiosity-driven rewards can be found in (Oudeyer et al., 2007;

Original: Oudeyer & Kaplan, 2009).
Revised: Oudeyer & Kaplan, 2009).

Original: Most approaches use some form of prediction-error between the learned model and environment behavior (Pathak et al., 2017).
Revised: Most approaches use some form of prediction-error between the learned model and environment behavior (Pathak et al., 2017).

Original: This prediction error can also be formulated as surprise (Schmidhuber, 1991; Achiam & Sastry, 2017; Sun et al., 2011).
Revised: This prediction error can also be formulated as surprise (Schmidhuber, 1991; Achiam & Sastry, 2017; Sun et al., 2011).

Original: Other forms of curiosity can be to explore states and actions where prediction of a forward model is highly-uncertain (Still & Precup, 2012; Houthooft et al., 2016b).
Revised: Other forms of curiosity can be to explore states and actions where prediction of a forward model is highly-uncertain (Still & Precup, 2012; Houthooft et al., 2016b).

Original: Finally, approaches such as (Lopes et al., 2012) try to explore state space which help improve the prediction model most.
Revised: Finally, approaches such as (Lopes et al., 2012) try to explore state space which help improve the prediction model most.

Original: However, most of these efforts have still studied the problem in context of external rewards.
Revised: However, most of these efforts have still studied the problem in context of external rewards.

Original: These intrinsic rewards just guide the search to the space where forward model is uncertain or likely to be wrong.
Revised: These intrinsic rewards just guide the search to the space where forward model is uncertain or likely to be wrong.

Original: Another approach for intrinsic rewards is using explicit visitation counts (Bellemare et al., 2016; Fu et al., 2017).
Revised: Another approach for intrinsic rewards is using explicit visitation counts (Bellemare et al., 2016; Fu et al., 2017).

Original: These exploration strategies guide the exploration policy to “novel” states (Bellemare et al., 2016).
Revised: These exploration strategies guide the exploration policy to “novel” states (Bellemare et al., 2016).

Original: A closely related work uses diversity as a reward for exploration and skill-learning (Eysenbach et al., 2018).
Revised: A closely related work uses diversity as a reward for exploration and skill-learning (Eysenbach et al., 2018).

Original: However, both visitation counts or measuring diversity requires learning a model which keeps the distribution of visited states.
Revised: However, both visitation counts or measuring diversity requires learning a model which keeps the distribution of visited states.

Original: Learning such a model does not seem trivial.
Revised: Learning such a model does not seem trivial.

Original: Another issue is the transferable properties and generalization of such approaches unless the state features are transferable themselves.
Revised: Another issue is the transferable properties and generalization of such approaches unless the state features are transferable themselves.

Original: Finally, apart from intrinsic rewards, other approaches include using an adversarial game (Sukhbaatar et al., 2018) where one agent gives the goal states and hence guiding exploration.
Revised: Finally, apart from intrinsic rewards, other approaches include using an adversarial game (Sukhbaatar et al., 2018) where one agent gives the goal states and hence guiding exploration.

Original: Gregor et al.
Revised: Gregor et al.

Original: (2017) introduce a formulation of empowerment where agent prefers to go to states where it expects it will achieve the most control after learning.
Revised: (2017) introduce a formulation of empowerment where agent prefers to go to states where it expects it will achieve the most control after learning.

Original: Researchers have also tried using perturbation of learned policy for exploration (Fortunato et al., 2017; Plappert et al., 2017) and using value function estimates (Osband et al., 2016).
Revised: Researchers have also tried using perturbation of learned policy for exploration (Fortunato et al., 2017; Plappert et al., 2017) and using value function estimates (Osband et al., 2016).

Original: Again these approaches have mostly been considered in context of external rewards and hence turn out to be sample inefﬁcient.
Revised: Again these approaches have mostly been considered in context of external rewards and hence turn out to be sample inefﬁcient.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Exploration has always been a crucial topic in robotics yet all the recent advances have been shown in either video-games or simulation due to sample inefﬁciency.
Revised: Exploration has always been a crucial topic in robotics yet all the recent advances have been shown in either video-games or simulation due to sample inefﬁciency.

Original: This paper focuses on policy optimization for exploration policy learned via intrinsic rewards.
Revised: This paper focuses on policy optimization for exploration policy learned via intrinsic rewards.

Original: Speciﬁcally, we propose a simple yet effective local approximation which allows us to perform policy optimization using likelihood maximization.
Revised: Speciﬁcally, we propose a simple yet effective local approximation which allows us to perform policy optimization using likelihood maximization.

Original: We demonstrate the effectiveness of our approach on Sawyer robot which learns how to interact with objects (trained from scratch).
Revised: We demonstrate the effectiveness of our approach on Sawyer robot which learns how to interact with objects (trained from scratch).

Original: A PPENDIX A T RAINING USING EXTERNAL REWARD
Revised: A PPENDIX A T RAINING USING EXTERNAL REWARD

Original: [Figure]
Revised: [Figure]

Original: Number of training steps
Revised: Number of training steps

Original: [Figure]
Revised: [Figure]

Original: M e a n
Revised: M e a n

Original: R e w a r d s reinforce_with_ext_rew
Revised: R e w a r d s reinforce_with_ext_rew

Original: In this section, we tried to analyze how policy performs if it is provided with external reward from the environment.
Revised: In this section, we tried to analyze how policy performs if it is provided with external reward from the environment.

Original: The environment gives +1 to the agent if it moves the objects, otherwise 0.
Revised: The environment gives +1 to the agent if it moves the objects, otherwise 0.

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX B P REDICTION L OSS
Revised: A PPENDIX B P REDICTION L OSS

Original: Prediction Loss
Revised: Prediction Loss

Original: We compare the prediction loss of the forward model learned on the data collected by different exploration schemes.
Revised: We compare the prediction loss of the forward model learned on the data collected by different exploration schemes.

Original: We use L2 distance in the ImageNet feature space.
Revised: We use L2 distance in the ImageNet feature space.

Original: As the result in the Table 1 indicate the data collected using our model performs better.
Revised: As the result in the Table 1 indicate the data collected using our model performs better

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: We now demonstrate the comparison of our proposed exploration formulation in Section 3.2 with the recently proposed (concurrent work) large-scale-curiosity (Burda et al., 2018) implementation on benchmark Atari games.

Revised: For baseline, we used the publicly available implementation off-theshelf (Burda et al., 2018).

Revised: The policy training is performed using the curiosity-reward function for the baseline and using our differentiable exploration objective for our method, without relying on any external rewards from the game.

Revised: We measure the external reward from the game as a proxy to evaluate the quality of exploration.

Revised: This external game reward is only shown for evaluation purposes but not used for training.

Revised: Our proposed model in Equation 3 relies on the learned prediction model to train the policy in an efﬁcient manner.

Revised: However, in the absence of any external rewards on these games, the exploration would have to rely on long-term prediction of the learned forward model to make progress that correlated with getting extrinsic reward.

Revised: However, the long-term model learning is still an active area of research (Ebert et al., 2017).

Revised: Since using current approaches, the long-term prediction models are hard to learn, so we use our objective in conjunction with reinforce objective when the long-term prediction quality matters.

Revised: The orange curve in Figure 7 denotes the recent implementation curiosity baseline provided in the paper.

Revised: Note both these environments require long-term modeling of the environment and on both the environments, our objective seems to provide gain in efﬁciency.

Revised: A PPENDIX D A NALYIZING THE D IFFERENTIABLE E XPLORATION O BJECTIVE

Revised: In this section, we discuss a toy experiment to investigate how would the proposed object behave due to the effect of local constancy approximation, explained in Section 3.2.

Revised: Example-1:

Revised: Investigating behavior

Revised: of

Revised: the objective

Revised: Lets consider a simple setting of navigation in the environment where states are represented by the color of rooms that agent visits.

Revised: We compare the behavior of count-based and curiosity-driven exploration with our proposed differentiable exploration objective.

Revised: In both cases, let us assume a tabular representation

Revised: For the count-based exploration reward, count based method will only consider next state being visited irrespective of what path being take to reach over there.

Revised: It will give +1 exploration reward for the ﬁrst 2 transitions mentioned in Table 2.

Revised: However, when agent does transition from green to blue, count based method will give 0 reward, even though this state transition happened for the ﬁrst time.

Revised: In contrast to this, a prediction-based curiosity objective gives +1 reward for the ﬁrst three transitions because they all happened for the ﬁrst time, and 0 reward for the last case.

Revised: [Table]

Revised: Both curiosity and count-based objectives are optimized via policy gradients in general.

Revised: Hence, when reward is 1 it means the action is encouraged and hence the probability of that action is increased as desired.

Revised: However, the more interesting scenario is when the reward is 0 .

Revised: In this case, the policy gradient objective suppresses the current action being taken due to the reinforce objective (Schulman et al., 2015).

Revised: Since the action probability is normalized across action, this amounts to increasing other actions in proportion to their existing log probability.

Revised: This is ﬁne in expectation, but suffers from high variance in small batches.

Revised: Now let us consider our proposed objective from Equation 3.

Revised: In our case, the behavior in ﬁrst three cases is similar to the prediction-based curiosity objective.

Revised: However, in the ﬁnal case, when the same transition is being executed, our model would explicitly try to force the policy to take action that does not lead to the blue as next state.

Revised: Now if the environment contained mostly blue rooms, our objective will explicitly encourage actions that go to a non-blue rooms, in contrast with reinforce which penalizes actions that lead to blue rooms. .

Revised: We now elaborate why this simple difference in behavior has a striking impact in high dimensional scenarios.

Revised: Example-2:

Revised: Investigating gradient direction

Revised: Consider a thought experiment of a white surface with a red magic box kept at a random location.

Revised: The action space of a robotic arm involves going to any location (e.g. a discrete n × n grid) on the surface.

Revised: If the robotic arm end-effector ends up in the area occupied by magic box, something unpredictable happens.

Revised: Now since the area of red magic box is much smaller compared to the white surface, it would take long time to hit the magic area randomly.

Revised: Hence, most of the example for learning dynamics will be from the free surface.

Revised: Suppose a forward dynamics model is learned on a bunch of initial sample interactions (out of which most of them will be on free space) and hence the models prediction error will be very low in white surface and high the red magic area.

Revised: A curiosity-objective learned with reinforce would penalize the actions leading to white space as the reward will be low over there, and the probability of other actions (including going to red area) will increase in proportion of their log-probs.

Revised: In contrast, the direct gradients in our case would directly force the actions to go to non-white locations and hence, quickly reach the locations in red area.

Revised: This examples is representative of scenarios where the number of un-interesting interactions is much more than the interesting interactions which are worth exploring, which is usually the case in the real-world setups.

Revised: D. T HEORETICAL U NDERSTANDING OF O BJECTIVE

Revised: The proposed objective in Equation 3 bears similarities to the experiment-design in optimization literature.

Revised: For a detailed overview, refer to the Chapter-7 in Boyd & Vandenberghe (2004).

Revised: The goal of experiment design is to optimize for the actions such that the error covariance in the model is small, which means going to areas with high prediction error.

Revised: While this is out of the scope of paper, we hope that this connection would help spur discussions for future work to build upon.