Original: Haptics-based Curiosity for Sparse-Reward Tasks
Revised: Haptics-based Curiosity for Sparse-Reward Tasks

Original: Element AI, Montr´eal, 4 CIFAR Fellow
Revised: Fellow

Original: Abstract: Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion.
Revised: Abstract: Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion.

Original: In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks.
Revised: In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks.

Original: Our approach, Haptics-based Curiosity (HaC), learns what visible objects interactions are supposed to “feel” like.
Revised: Our approach, Haptics-based Curiosity (HaC), learns what visible objects interactions are supposed to “feel” like.

Original: We encourage exploration by rewarding interactions where the expectation and the experience do not match.
Revised: We encourage exploration by rewarding interactions where the expectation and the experience do not match.

Original: We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work.
Revised: We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work.

Original: Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difﬁcult tasks through sparse reward and curiosity alone.
Revised: Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difﬁcult tasks through sparse reward and curiosity alone.

Original: We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and ﬁnd that our method performs better and is more sample-efﬁcient.
Revised: We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and ﬁnd that our method performs better and is more sample-efﬁcient.

Original:  Introduction
Revised:  Introduction

Original: Most successes in reinforcement learning (RL) come from games [1, 2] or scenarios where the reward is strongly shaped [3, 4].
Revised: Most successes in reinforcement learning (RL) come from games [1, 2] or scenarios where the reward is strongly shaped [3, 4].

Original: In the former, the environment is often fully observable and the reward is dense and well-deﬁned.
Revised: In the former, the environment is often fully observable and the reward is dense and well-deﬁned.

Original: In the latter, a large amount of work is required to design useful reward functions.
Revised: In the latter, a large amount of work is required to design useful reward functions.

Original: While it may be possible to hand-craft dense reward signals for many real-world tasks, we believe that it is a worthwhile endeavor to investigate learning methods that do not require dense rewards.
Revised: While it may be possible to hand-craft dense reward signals for many real-world tasks, we believe that it is a worthwhile endeavor to investigate learning methods that do not require dense rewards.

Original: Closely related to the sparse rewards problem is the issue of exploration.
Revised: Closely related to the sparse rewards problem is the issue of exploration.

Original: One reason that traditional
Revised: One reason that traditional

Original: RL agents struggle with sparse-reward problems is a lack of exploration.
Revised: RL agents struggle with sparse-reward problems is a lack of exploration.

Original: An agent may not obtain useful rewards without an intuitive exploration strategy when rewards are sparse.
Revised: An agent may not obtain useful rewards without an intuitive exploration strategy when rewards are sparse.

Original: Exploration based on intrinsic curiosity comes naturally to many animals and infants (who start crawling and exploring the environment at around 9 months [5] and oftentimes even before they can crawl by using their hands and mouth to touch and probe objects).
Revised: Exploration based on intrinsic curiosity comes naturally to many animals and infants (who start crawling and exploring the environment at around 9 months [5] and oftentimes even before they can crawl by using their hands and mouth to touch and probe objects).

Original: Touch is a local experience and encodes accurate geometrical information while handling objects.
Revised: Touch is a local experience and encodes accurate geometrical information while handling objects.

Original: Experimental studies in infants has suggested that tactile and visual sensory modalities play a central role in systematic learning of tasks related to object understanding, interaction and manipulation [6, 7].
Revised: Experimental studies in infants has suggested that tactile and visual sensory modalities play a central role in systematic learning of tasks related to object understanding, interaction and manipulation [6, 7].

Original: Ideally, we would like our RL agents to explore the environment in an analogous self-guided fashion to learn the dynamics and object properties, and use this knowledge to solve downstream tasks.
Revised: Ideally, we would like our RL agents to explore the environment in an analogous self-guided fashion to learn the dynamics and object properties, and use this knowledge to solve downstream tasks.

Original: Just as how humans utilize different sensory modalities to explore and understand the world around them, exploration in robots should be more embodied and related to a combination of vision and touch and potentially other sensor modalities.
Revised: Just as how humans utilize different sensory modalities to explore and understand the world around them, exploration in robots should be more embodied and related to a combination of vision and touch and potentially other sensor modalities.

Original: We believe that building autonomous agents that are self-driven and seek to explore via multi-modal interaction are crucial to address key issues in developmental robotics.
Revised: We believe that building autonomous agents that are self-driven and seek to explore via multi-modal interaction are crucial to address key issues in developmental robotics.

Original: Recent works in RL have focused on a curiosity-driven exploration through prediction-based surprise [8, 9, 10].
Revised: Recent works in RL have focused on a curiosity-driven exploration through prediction-based surprise [8, 9, 10].

Original: In this formulation, a forward dynamics model predicts the future, and if its prediction is incorrect when compared to the real future, the agent is surprised and is thus rewarded.
Revised: In this formulation, a forward dynamics model predicts the future, and if its prediction is incorrect when compared to the real future, the agent is surprised and is thus rewarded.

Original: This encourages the agent to look for novel states while improving its visual forward model in return.
Revised: This encourages the agent to look for novel states while improving its visual forward model in return.

Original: However, this formulation can be practically challenging to optimize since there are many states that are visually dissimilar but practically irrelevant (e.g. for a pushing task, moving a robotic end-effector without touching the object creates visual novelty but contributes little to task-related knowledge).
Revised: However, this formulation can be practically challenging to optimize since there are many states that are visually dissimilar but practically irrelevant (e.g. for a pushing task, moving a robotic end-effector without touching the object creates visual novelty but contributes little to task-related knowledge).

Original: One way to constrain this search space over curious behaviors is by involving another modality like haptics.
Revised: One way to constrain this search space over curious behaviors is by involving another modality like haptics.

Original: In this work, we demonstrate that a self-guided cross-modal exploration policy can help solve sparsereward downstream tasks that existing methods without this curiosity struggle to solve.
Revised: In this work, we demonstrate that a self-guided cross-modal exploration policy can help solve sparsereward downstream tasks that existing methods without this curiosity struggle to solve.

Original: Our method uses cross-modal consistency (mismatch between visual and haptic signal) to guide this exploration.
Revised: Our method uses cross-modal consistency (mismatch between visual and haptic signal) to guide this exploration.

Original: To use self-play knowledge in downstream tasks, we relabel past experiences, providing a dense reward signal that allows modern off-policy RL methods to solve the tasks.
Revised: To use self-play knowledge in downstream tasks, we relabel past experiences, providing a dense reward signal that allows modern off-policy RL methods to solve the tasks.

Original: While there are many existing methods that use artiﬁcial curiosity/intrinsic motivation, the majority of these methods either rely on strong domain knowledge (e.g. labels of state dimensions in Forestier et al.
Revised: While there are many existing methods that use artiﬁcial curiosity/intrinsic motivation, the majority of these methods either rely on strong domain knowledge (e.g. labels of state dimensions in Forestier et al.

Original: [4], a goalpicking strategy in Andrychowicz et al.
Revised: [4], a goalpicking strategy in Andrychowicz et al.

Original: [11]) or are prone to get stuck in local optima when a single meaningless stimulus creates enough surprise to capture the attention of the agent (e.g. noisy-TV experiment from [12]).
Revised: [11]) or are prone to get stuck in local optima when a single meaningless stimulus creates enough surprise to capture the attention of the agent (e.g. noisy-TV experiment from [12]).

Original: Other approaches depend on unrealistic assumptions and goal conditioning [11].
Revised: Other approaches depend on unrealistic assumptions and goal conditioning [11].

Original: Our method presents a novel approach in the family of prediction-based models [8, 13, 9] and yields better performance on a wide range of robotic manipulation tasks than purely visionbased and haptics-based approaches [8, 9].
Revised: Our method presents a novel approach in the family of prediction-based models [8, 13, 9] and yields better performance on a wide range of robotic manipulation tasks than purely visionbased and haptics-based approaches [8, 9].

Original: The tasks are chosen with careful consideration—they comprise of preliminary robotic manipulations such as grasping, pushing, and pulling.
Revised: The tasks are chosen with careful consideration—they comprise of preliminary robotic manipulations such as grasping, pushing, and pulling.

Original: In this work, we present the following contributions:
Revised: In this work, we present the following contributions:

Original: • A new curiosity method to help solve sparse-reward tasks that use cross-modal consistency (predicting one modality from another) to guide exploration.
Revised: • A new curiosity method to help solve sparse-reward tasks that use cross-modal consistency (predicting one modality from another) to guide exploration.

Original: We implement it in this work as vision and touch modalities, but the formulation of our method does not require any knowledge about the underlying modalities and can thus be applied to other settings.
Revised: We implement it in this work as vision and touch modalities, but the formulation of our method does not require any knowledge about the underlying modalities and can thus be applied to other settings.

Original: • We create and maintain a manipulation benchmark of simulated tasks, MiniTouch , inspired by Chen et al.
Revised: • We create and maintain a manipulation benchmark of simulated tasks, MiniTouch , inspired by Chen et al.

Original: [14], Andrychowicz et al.
Revised: [14], Andrychowicz et al.

Original: [11], where the robotic arm is equipped with a force/torque sensor.
Revised: [11], where the robotic arm is equipped with a force/torque sensor.

Original: This allows evaluation of models’ performance on different manipulation tasks that can leverage cross-modal learning.
Revised: This allows evaluation of models’ performance on different manipulation tasks that can leverage cross-modal learning.

Original: • We validate the performance of our method on MiniTouch environment comprising of four downstream tasks.
Revised: • We validate the performance of our method on MiniTouch environment comprising of four downstream tasks.

Original: We compare purely vision-based curiosity approaches and standard offpolicy RL algorithms.
Revised: We compare purely vision-based curiosity approaches and standard offpolicy RL algorithms.

Original: Our method improves both performance and sample efﬁciency.
Revised: Our method improves both performance and sample efﬁciency.

Original:  Related Work
Revised:  Related Work

Original: Intrinsic Motivation Intrinsic motivation is an inherent spontaneous tendency to be curious or to seek something novel in order to further enhance one’s skill and knowledge [15, 16, 17].
Revised: Intrinsic Motivation Intrinsic motivation is an inherent spontaneous tendency to be curious or to seek something novel in order to further enhance one’s skill and knowledge [15, 16, 17].

Original: This principle is shown to work well even in the absence of a well-deﬁned goal or objective.
Revised: This principle is shown to work well even in the absence of a well-deﬁned goal or objective.

Original: In reinforcement learning, intrinsic motivation has been a well-researched topic [18, 19, 20, 21, 22, 23].
Revised: In reinforcement learning, intrinsic motivation has been a well-researched topic [18, 19, 20, 21, 22, 23].

Original: An intuitive way to perform intrinsic motivation is through the use of “novelty discovery”.
Revised: An intuitive way to perform intrinsic motivation is through the use of “novelty discovery”.

Original: For example, incentivize the RL agent to visit unusual states or states with substantial information gain [24].
Revised: For example, incentivize the RL agent to visit unusual states or states with substantial information gain [24].

Original: In its simplest form, this can be achieved with up-weighting mechanisms such as state visitation counts [25].
Revised: In its simplest form, this can be achieved with up-weighting mechanisms such as state visitation counts [25].

Original: Count-based methods have also been extended to high-dimensional state spaces [26, 12, 27].
Revised: Count-based methods have also been extended to high-dimensional state spaces [26, 12, 27].

Original: Alternative forms of intrinsic motivation include disagreement [28], empowerment [29, 30].
Revised: Alternative forms of intrinsic motivation include disagreement [28], empowerment [29, 30].

Original: Exploratory intrinsic motivation can also be achieved through “curiosity” [18, 31].
Revised: Exploratory intrinsic motivation can also be achieved through “curiosity” [18, 31].

Original: In this setting, an agent is encouraged to visit states with high predictive errors [10, 8, 9] by training a forward dynamics model that predicts the future state given the current state and action.
Revised: In this setting, an agent is encouraged to visit states with high predictive errors [10, 8, 9] by training a forward dynamics model that predicts the future state given the current state and action.

Original: Instead of making predictions in the raw visual space, Pathak et al.
Revised: Instead of making predictions in the raw visual space, Pathak et al.

Original: [9] mapped images to a feature space where relevant information is represented via an inverse dynamics model.
Revised: [9] mapped images to a feature space where relevant information is represented via an inverse dynamics model.

Original: Burda et al.
Revised: Burda et al.

Original: [8] demonstrate that random features are sufﬁcient for many popular RL game benchmarks.
Revised: [8] demonstrate that random features are sufﬁcient for many popular RL game benchmarks.

Original: This approach may work well with tasks that require navigation to ﬁnd a reward because each unseen position of the agent in the world leads to high intrinsic reward when unseen.
Revised: This approach may work well with tasks that require navigation to ﬁnd a reward because each unseen position of the agent in the world leads to high intrinsic reward when unseen.

Original: However, in the case of manipulative tasks, we are less interested in the robot visiting all the possible states and more interested in states where the robot interacts with other objects.
Revised: However, in the case of manipulative tasks, we are less interested in the robot visiting all the possible states and more interested in states where the robot interacts with other objects.

Original: In this work, we leverage multimodal inputs that encourage the agent to ﬁnd novel combinations of visual and force/torque modalities.
Revised: In this work, we leverage multimodal inputs that encourage the agent to ﬁnd novel combinations of visual and force/torque modalities.

Original: Self-Supervised Learning via Cross-modality Exploiting multimodality to learn unsupervised representations dates back to at least 1993 [32].
Revised: Self-Supervised Learning via Cross-modality Exploiting multimodality to learn unsupervised representations dates back to at least 1993 [32].

Original: Multimodal signals are naturally suitable for selfsupervised learning, as information from one modality can be used to supervise learning for another
Revised: Multimodal signals are naturally suitable for selfsupervised learning, as information from one modality can be used to supervise learning for another

Original: [Figure]
Revised: [Figure]

Original: modality [33, 34].
Revised: modality [33, 34].

Original: Different modalities typically carry different information, e.g., visual and touch sensory modalities emerge concurrently and often in an interrelated manner during contact-rich manipulation tasks [35].
Revised: Different modalities typically carry different information, e.g., visual and touch sensory modalities emerge concurrently and often in an interrelated manner during contact-rich manipulation tasks [35].

Original: Speciﬁcally, force/torque motor signal has always been a major component in the literature of perception and control [36, 37, 38].
Revised: Speciﬁcally, force/torque motor signal has always been a major component in the literature of perception and control [36, 37, 38].

Original: The most common ways to leverage multimodal signals to learn representations are through vision and language [39, 40].
Revised: The most common ways to leverage multimodal signals to learn representations are through vision and language [39, 40].

Original: Gao et al.
Revised: Gao et al.

Original: [41], Li et al.
Revised: [41], Li et al.

Original: [42] demonstrated a uniﬁed approach to learning representations for prediction tasks using visual and touch data.
Revised: [42] demonstrated a uniﬁed approach to learning representations for prediction tasks using visual and touch data.

Original: Chen et al.
Revised: Chen et al.

Original: [43] learn world-models from multimodal data via a shared latent space.
Revised: [43] learn world-models from multimodal data via a shared latent space.

Original: In robotics and interactive settings, the use of modalities such as tactile sensing [44, 45, 46] is increasingly popular for grasping, manipulation and other externally-speciﬁed tasks [47, 48].
Revised: In robotics and interactive settings, the use of modalities such as tactile sensing [44, 45, 46] is increasingly popular for grasping, manipulation and other externally-speciﬁed tasks [47, 48].

Original: Lee et al.
Revised: Lee et al.

Original: [49] showed the effectiveness of self-supervised training of tactile and visual representations by demonstrating its use on a peg insertion task.
Revised: [49] showed the effectiveness of self-supervised training of tactile and visual representations by demonstrating its use on a peg insertion task.

Original: While the mentioned approaches have used multiple sensory modalities for learning better representations, in this work we demonstrate its utility for allowing agents to explore.
Revised: While the mentioned approaches have used multiple sensory modalities for learning better representations, in this work we demonstrate its utility for allowing agents to explore.

Original: Similar to ours, Dean et al.
Revised: Similar to ours, Dean et al.

Original: [50] use multimodal sensory association (i.e. audio and visual) to compute the intrinsic reward.
Revised: [50] use multimodal sensory association (i.e. audio and visual) to compute the intrinsic reward.

Original: Their curiosity-based formulation allows an agent to efﬁciently explore the environment in settings where audio and visual signals are governed by the same physical processes.
Revised: Their curiosity-based formulation allows an agent to efﬁciently explore the environment in settings where audio and visual signals are governed by the same physical processes.

Original: In addition to the different nature of sensory signals, they use a discriminator that determines whether an observed multimodal pair is novel.
Revised: In addition to the different nature of sensory signals, they use a discriminator that determines whether an observed multimodal pair is novel.

Original: This might not work in our case as touch is a more sparse signal and using a discriminator could lead to ambiguous outcomes.
Revised: This might not work in our case as touch is a more sparse signal and using a discriminator could lead to ambiguous outcomes.

Original:  Proposed Approach
Revised:  Proposed Approach

Original: The goal of our method, Haptics-Based Curiosity (HaC), is to encourage the agent to interact with objects.
Revised: The goal of our method, Haptics-Based Curiosity (HaC), is to encourage the agent to interact with objects.

Original: HaC provides a reward signal for an RL agent to explore the state space of a task that involves interacting with objects.
Revised: HaC provides a reward signal for an RL agent to explore the state space of a task that involves interacting with objects.

Original: The exploration phase is independent of the downstream task, i.e., relying solely on visual and force/torque signals, without a reward signal from the downstream task.
Revised: The exploration phase is independent of the downstream task, i.e., relying solely on visual and force/torque signals, without a reward signal from the downstream task.

Original: Similar to how people spend more time exploring stimuli that are more incongruous [7], HaC guides the agent to focus its experience on different novel cross-modal associations.
Revised: Similar to how people spend more time exploring stimuli that are more incongruous [7], HaC guides the agent to focus its experience on different novel cross-modal associations.

Original: We augment this intrinsic objective with a future visual state prediction objective similar to the one in Pathak et al.
Revised: We augment this intrinsic objective with a future visual state prediction objective similar to the one in Pathak et al.

Original: [9] to avoid getting stuck in undesired inactive conﬁgurations.
Revised: [9] to avoid getting stuck in undesired inactive conﬁgurations.

Original: Note that we sometimes refer to the future state prediction objective in the text as forward dynamics objective.
Revised: Note that we sometimes refer to the future state prediction objective in the text as forward dynamics objective.

Original: In this work, we focus on the cross-modality between vision and touch, but the same idea could be applied to other pairs of sensory domains, such as vision and sound, or touch and acoustics.
Revised: In this work, we focus on the cross-modality between vision and touch, but the same idea could be applied to other pairs of sensory domains, such as vision and sound, or touch and acoustics.

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: The learning problem is formalized as a Markov decision process (MDP) deﬁned by a tuple {S , A , T , R , ⇢ ,  } of states, actions, transition probability, reward, initial state distribution, and discount factor.
Revised: The learning problem is formalized as a Markov decision process (MDP) deﬁned by a tuple {S , A , T , R , ρ, γ } of states, actions, transition probability, reward, initial state distribution, and discount factor.

Original: The goal is to ﬁnd the optimal policy that maximizes the discounted sum of rewards, ⇡ ⇤ =
Revised: The goal is to ﬁnd the optimal policy that maximizes the discounted sum of rewards, π ∗ =

Original: E ⇡ [ P 1 t  t r ( s t , a t )] .
Revised: E π [ (cid:80) ∞ t γ t r ( s t , a t )] .

Original: In our case, each state s t in the trajectory is composed of both visual and corresponding touch features as detailed in the following section.
Revised: In our case, each state s t in the trajectory is composed of both visual and corresponding touch features as detailed in the following section.

Original: We use soft actor- critic policy gradients (SAC) [51] to train our policies, but in principle, our proposed approach is algorithm-agnostic.
Revised: We use soft actor- critic policy gradients (SAC) [51] to train our policies, but in principle, our proposed approach is algorithm-agnostic.

Original: The policy ⇡ is evaluated with an estimation of the soft Q-value:
Revised: The policy π is evaluated with an estimation of the soft Q-value:

Original: [Equation]
Revised: [Equation]

Original: where V ( s t ) = E a t ⇠ ⇡ [ Q ( s t , a t )  log ⇡ ( a t | s t )] is the soft value function.
Revised: where V ( s t ) = E a t ∼ π [ Q ( s t , a t ) − log π ( a t | s t )] is the soft value function.

Original: 3.2 Haptics-based Prediction
Revised: 3.2 Haptics-based Prediction

Original: Our core prediction framework consists of two modules: (i) a haptics control module, which learns to predict expected haptic signal from the visual input, and (ii) a forward dynamics model, which predicts the next latent state from the current latent state and the current action (see Figure 1).
Revised: Our core prediction framework consists of two modules: (i) a haptics control module, which learns to predict expected haptic signal from the visual input, and (ii) a forward dynamics model, which predicts the next latent state from the current latent state and the current action (see Figure 1).

Original: Let the state of the environment s t =
Revised: Let the state of the environment s t =

Original: ( z t , h t ) at time t be composed of a visual feature z t (the encoded visual input) and a haptic signal h t .
Revised: ( z t , h t ) at time t be composed of a visual feature z t (the encoded visual input) and a haptic signal h t .

Original: The touch prediction model consists of a convolutional encoder z t = enc ( x t ) that transforms the image x t into a latent representation z t and a fully-connected decoder ˆ h t
Revised: The touch prediction model consists of a convolutional encoder z t = enc ( x t ) that transforms the image x t into a latent representation z t and a fully-connected decoder ˆ h t

Original: = dec ( z t ) that transforms the latent into a predicted haptic signal ˆ h t .
Revised: = dec ( z t ) that transforms the latent into a predicted haptic signal ˆ h t .

Original: The encoderdecoder is trained with an L2 reconstruction loss, i.e. for every image x t and force/torque sensor h t :
Revised: The encoderdecoder is trained with an L2 reconstruction loss, i.e. for every image x t and force/torque sensor h t :

Original: [Equation] 
Revised: [Equation] [Equation]

Original: A high prediction error on a given image indicates that the agent has had few interactions like this.
Revised: A high prediction error on a given image indicates that the agent has had few interactions like this.

Original: Therefore, to harness this “surprise” to guide exploration, we deﬁne the intrinsic reward at time t during exploration to be proportional to this reconstruction loss.
Revised: Therefore, to harness this “surprise” to guide exploration, we deﬁne the intrinsic reward at time t during exploration to be proportional to this reconstruction loss.

Original: This essentially allows the agent policy to visit under-explored conﬁgurations of the state space by encouraging interactions where the system does not know what the target object “feels” like.
Revised: This essentially allows the agent policy to visit under-explored conﬁgurations of the state space by encouraging interactions where the system does not know what the target object “feels” like.

Original: In addition to efﬁcient exploration, being aware of such incongruity via touch prediction aids learning local regularities in the visual input.
Revised: In addition to efﬁcient exploration, being aware of such incongruity via touch prediction aids learning local regularities in the visual input.

Original: This in turn could assist better generalization to unseen states.
Revised: This in turn could assist better generalization to unseen states.

Original: An overall pipeline of the framework is shown in Figure 1a.
Revised: An overall pipeline of the framework is shown in Figure 1a.

Original: 3.3 Regularization Through Forward Dynamics Model
Revised: 3.3 Regularization Through Forward Dynamics Model

Original: We found empirically (and we demonstrate in the experiments section below) that the surprise stemming from haptic novelty was not enough to cause object-centric interaction.
Revised: We found empirically (and we demonstrate in the experiments section below) that the surprise stemming from haptic novelty was not enough to cause object-centric interaction.

Original: We postulate that by incorporating visual surprise (i.e. the mismatch between predicted forward dynamics and observed dynamics)
Revised: We postulate that by incorporating visual surprise (i.e. the mismatch between predicted forward dynamics and observed dynamics)

Original: [9], we can create an agent that seeks out visual novelty as well as haptic one and thus leads to better state space coverage.
Revised: [9], we can create an agent that seeks out visual novelty as well as haptic one and thus leads to better state space coverage.

Original: To this end, our model is augmented with a forward dynamics model (see Figure-1b)
Revised: To this end, our model is augmented with a forward dynamics model (see Figure-1b)

Original: ˆ z t +1 = fdm ( z t , a t ) that learns to map the latent state z t (obtained from the visual encoder enc ) and action a t at time t to the predicted latent state ˆ z t +1 at the next timestep.
Revised: ˆ z t +1 = fdm ( z t , a t ) that learns to map the latent state z t (obtained from the visual encoder enc ) and action a t at time t to the predicted latent state ˆ z t +1 at the next timestep.

Original: This model is also trained with L2 loss:
Revised: This model is also trained with L2 loss:

Original: [Equation]
Revised: [Equation]

Original: The intrinsic reward is deﬁned as the convex combination of the cross-modal prediction loss (Eq. 2) and the forward dynamics model loss (Eq. 3):
Revised: The intrinsic reward is deﬁned as the convex combination of the cross-modal prediction loss (Eq. 2) and the forward dynamics model loss (Eq. 3):

Original: [Equation]
Revised: [Equation]

Original: where  2 [0 , 1] is a balancing factor.
Revised: where λ ∈ [0 , 1] is a balancing factor.

Original: The effect of the factor  on overall performance is studied in the ablation experiments described in Section 5.1.
Revised: The effect of the factor λ on overall performance is studied in the ablation experiments described in Section 5.1.

Original: 3.4 Training
Revised: 3.4 Training

Original: Learning is divided into two stages: (i) an exploratory step, where the agent performs free exploration following HaC, and (ii) an adaptation step, where the agent is tasked to solve a downstream problem, given a sparse reward.
Revised: Learning is divided into two stages: (i) an exploratory step, where the agent performs free exploration following HaC, and (ii) an adaptation step, where the agent is tasked to solve a downstream problem, given a sparse reward.

Original: During the exploratory step, each trajectory consists of pairs of image and force/torque features, ( z 1 , h 1 ) , ( z 2 , h 2 ) ,..., ( z n , h n ) .
Revised: During the exploratory step, each trajectory consists of pairs of image and force/torque features, ( z 1 , h 1 ) , ( z 2 , h 2 ) ,..., ( z n , h n ) .

Original: These trajectories are used for two purposes: (i) updating the parameters of the prediction model to help shape the representations and (ii) updating the exploration policy based on the intrinsic reward r t .
Revised: These trajectories are used for two purposes: (i) updating the parameters of the prediction model to help shape the representations and (ii) updating the exploration policy based on the intrinsic reward r t .

Original: Note that we use the touch features only to craft the intrinsic rewards and the input to RL agent consists of visual features alone to be comparable to the baselines.
Revised: Note that we use the touch features only to craft the intrinsic rewards and the input to RL agent consists of visual features alone to be comparable to the baselines.

Original: For vision-based curiosity models, Burda et al.
Revised: For vision-based curiosity models, Burda et al.

Original: [8] observed that encoding visual features via a random network constitute a simple and effective strategy compared to learned features.
Revised: [8] observed that encoding visual features via a random network constitute a simple and effective strategy compared to learned features.

Original: In Section 5.1, we investigate the performance of our model in both scenarios, i.e., when the features are learned vs random.
Revised: In Section 5.1, we investigate the performance of our model in both scenarios, i.e., when the features are learned vs random.

Original: The overall optimization problem at this step consists of the policy learning (driven by intrinsic reward), the touch reconstruction loss (Eq. 2), and the forward dynamics loss (Eq. 3).
Revised: The overall optimization problem at this step consists of the policy learning (driven by intrinsic reward), the touch reconstruction loss (Eq. 2), and the forward dynamics loss (Eq. 3).

Original: During the downstream adaptation step, the parameters of the policy network, the Q network and the replay buffer are retained from the exploratory phase.
Revised: During the downstream adaptation step, the parameters of the policy network, the Q network and the replay buffer are retained from the exploratory phase.

Original: The objective of the down-stream task is computed as:
Revised: The objective of the down-stream task is computed as:

Original: [Equation]
Revised: [Equation]

Original: where r et in this phase is task-speciﬁc external sparse reward.
Revised: where r et in this phase is task-speciﬁc external sparse reward.

Original: In both steps, the objectives are optimized with Adam [52].
Revised: In both steps, the objectives are optimized with Adam [52].

Original: Implementation Details : The encoder is a four-layered strided CNN followed by a fully connected network.
Revised: Implementation Details : The encoder is a four-layered strided CNN followed by a fully connected network.

Original: We use LeakyReLU
Revised: We use LeakyReLU

Original: [53] as non-linear activation in all the layers.
Revised: [53] as non-linear activation in all the layers.

Original: The decoder network is a two-layered MLP that maps 256-dimensional visual features to touch vectors.
Revised: The decoder network is a two-layered MLP that maps 256-dimensional visual features to touch vectors.

Original: For a more detailed description of the networks (SAC policy network and HaC networks, including the forward model) and hyper-parameters, please refer to the supplementary material Section 7.3.
Revised: For a more detailed description of the networks (SAC policy network and HaC networks, including the forward model) and hyper-parameters, please refer to the supplementary material Section 7.3.

Original:  Experimental Setting
Revised:  Experimental Setting

Original: Our experiments focus on a tabletop robot manipulation from raw image observations and raw force/torque sensory values, which we refer to as “touch vector”.
Revised: Our experiments focus on a tabletop robot manipulation from raw image observations and raw force/torque sensory values, which we refer to as “touch vector”.

Original: For our experiments, we use a 7-DoF Franka Emika Panda arm with a two-ﬁnger parallel gripper.
Revised: For our experiments, we use a 7-DoF Franka Emika Panda arm with a two-ﬁnger parallel gripper.

Original: Each of the ﬁngers is equipped with a simulated force/torque sensor that measures the joint reaction force applied to it.
Revised: Each of the ﬁngers is equipped with a simulated force/torque sensor that measures the joint reaction force applied to it.

Original: We utilize
Revised: We utilize

Original: PyBullet [54] to simulate the robot arm and haptic sensor 1 .
Revised: PyBullet [54] to simulate the robot arm and haptic sensor 1 .

Original: 4.1 MiniTouch Benchmark
Revised: 4.1 MiniTouch Benchmark

Original: Our proposed benchmark, MiniTouch, consists of four manipulation tasks:
Revised: Our proposed benchmark, MiniTouch, consists of four manipulation tasks:

Original: Playing, Pushing, Opening, Pick-up.
Revised: Playing, Pushing, Opening, Pick-up.

Original: Each of the tasks along with corresponding actions, observations, and rewards is described in detail in the Supplementary Material Section 7.1 and further illustrated in Figure 2.
Revised: Each of the tasks along with corresponding actions, observations, and rewards is described in detail in the Supplementary Material Section 7.1 and further illustrated in Figure 2.

Original: MiniTouch is an active repository 

Revised: MiniTouch is an active repository

Original: and we expect to update the benchmark with new tasks and datasets.
Revised: and we expect to update the benchmark with new tasks and datasets.

Original: The tasks are inspired by Yu et al.
Revised: The tasks are inspired by Yu et al.

Original: [55] but are not based on a proprietary simulator, feature an arm that we have access to for real-world experimentation (in follow-up work), and where the arm is equipped with a haptic sensor.
Revised: [55] but are not based on a proprietary simulator, feature an arm that we have access to for real-world experimentation (in follow-up work), and where the arm is equipped with a haptic sensor.

Original: 4.2 Baseline Comparisons and Metrics
Revised: 4.2 Baseline Comparisons and Metrics

Original: For the task evaluation, we study two versions of our model: (i) HaC-Pure considering the touch vector reconstruction intrinsic reward alone, and (ii) HaC, considering the full intrinsic reward (Eq. 4).
Revised: For the task evaluation, we study two versions of our model: (i) HaC-Pure considering the touch vector reconstruction intrinsic reward alone, and (ii) HaC, considering the full intrinsic reward (Eq. 4).

Original: We compare with several well-known intrinsic exploration baselines based on visual features:
Revised: We compare with several well-known intrinsic exploration baselines based on visual features:

Original: • SAC : The unmodiﬁed Soft Actor-Critic algorithm from Haarnoja et al.
Revised: • SAC : The unmodiﬁed Soft Actor-Critic algorithm from Haarnoja et al.

Original: ICM : SAC augmented with the state-of-the-art visual curiosity approach Intrinsic Curiosity Module (ICM)
Revised: ICM : SAC augmented with the state-of-the-art visual curiosity approach Intrinsic Curiosity Module (ICM)

Original: [12], which uses a visual prediction model to guide exploration.
Revised: [12], which uses a visual prediction model to guide exploration.

Original: • Disagreement : It uses model disagreement as objective for exploration [56, 28].
Revised: • Disagreement : It uses model disagreement as objective for exploration [56, 28].

Original: It leverages variance in the prediction of an ensemble of latent dynamics models as the reward.
Revised: It leverages variance in the prediction of an ensemble of latent dynamics models as the reward.

Original: • RND : Random Network Distillation [57] utilizes a randomly initialized neural network to specify an intrinsic reward for visiting unexplored states in hard exploration problems.
Revised: • RND : Random Network Distillation [57] utilizes a randomly initialized neural network to specify an intrinsic reward for visiting unexplored states in hard exploration problems.

Original: [Figure]
Revised: [Figure]

Original: We built a Pytorch [58] version of these baselines based on their open source code (details in supplementary material).
Revised: We built a Pytorch [58] version of these baselines based on their open source code (details in supplementary material).

Original: We use the following metrics to evaluate our method and the baseline models:
Revised: We use the following metrics to evaluate our method and the baseline models:

Original: Exploration success: measures the percentage of times that the agent attained the goal state in the exploratory phase, i.e. with no external reward.
Revised: Exploration success: measures the percentage of times that the agent attained the goal state in the exploratory phase, i.e. with no external reward.

Original: Higher is better.
Revised: Higher is better.

Original: Success: denotes the percentage of times that the agent attained the goal state during the downstream task phase.
Revised: Success: denotes the percentage of times that the agent attained the goal state during the downstream task phase.

Original: Episode steps: The number of steps required for each episode to succeed.
Revised: Episode steps: The number of steps required for each episode to succeed.

Original: This metric is an indicator of sample efﬁciency.
Revised: This metric is an indicator of sample efﬁciency.

Original: The lesser the number of steps, the faster the agent’s ability to succeed.
Revised: The lesser the number of steps, the faster the agent’s ability to succeed.

Original: Touch-interaction: Amount of interaction the agent’s ﬁngers have with the underlying object.
Revised: Touch-interaction: Amount of interaction the agent’s ﬁngers have with the underlying object.

Original: We measure this by computing the variance of force/torque sensory signal across the whole episode.
Revised: We measure this by computing the variance of force/torque sensory signal across the whole episode.

Original: Object movement: The agent can resort to constantly engaging with the object unnecessarily to satisfy the objective.
Revised: Object movement: The agent can resort to constantly engaging with the object unnecessarily to satisfy the objective.

Original: We, therefore, monitor the variance of door angle (for the Opening task) and the variance of object position (for the remaining tasks) over the course of training.
Revised: We, therefore, monitor the variance of door angle (for the Opening task) and the variance of object position (for the remaining tasks) over the course of training.

Original: A higher movement indicates diverse state space in addition to physical interaction.
Revised: A higher movement indicates diverse state space in addition to physical interaction.

Original:  Results and Discussion
Revised:  Results and Discussion

Original: HaC and baselines were trained on a Panda robot agent [54] for one million steps.
Revised: HaC and baselines were trained on a Panda robot agent [54] for one million steps.

Original: In the exploratory phase of the training, we pre-train our method only with the curiosity-based intrinsic reward.
Revised: In the exploratory phase of the training, we pre-train our method only with the curiosity-based intrinsic reward.

Original: We then progress to the adaptation phase.
Revised: We then progress to the adaptation phase.

Original: Also, note that across all tasks, HaC-Pure is based purely on cross-modal prediction, while HaC includes visual forward prediction reward in addition.
Revised: Also, note that across all tasks, HaC-Pure is based purely on cross-modal prediction, while HaC includes visual forward prediction reward in addition.

Original: Figure 3a and 3b shows results on the basic task of playing with a single object.
Revised: Figure 3a and 3b shows results on the basic task of playing with a single object.

Original: Since single object interaction does not have explicit goal states to evaluate, we instead measure the agent’s ability to constantly engage and play with the underlying object.
Revised: Since single object interaction does not have explicit goal states to evaluate, we instead measure the agent’s ability to constantly engage and play with the underlying object.

Original: HaC-Pure displays four times better interaction with the underlying object when compared to SAC (see Figure 3b).
Revised: HaC-Pure displays four times better interaction with the underlying object when compared to SAC (see Figure 3b).

Original: Note from the plots that there is a trade-off when using HaC and HaC-Pure between constant interaction (i.e. touch interaction performance) and object movement dynamics.
Revised: Note from the plots that there is a trade-off when using HaC and HaC-Pure between constant interaction (i.e. touch interaction performance) and object movement dynamics.

Original: Collecting a variety of such interesting data during the exploratory phase helps the agent in terms of sample efﬁciency while solving the downstream tasks.
Revised: Collecting a variety of such interesting data during the exploratory phase helps the agent in terms of sample efﬁciency while solving the downstream tasks.

Original: Figure 9 shows similar comparison for Open-door task.
Revised: Figure 9 shows similar comparison for Open-door task.

Original: In the following section, we examine the role of the forward objective term on touch interaction and possible ways to encode agent’s observation in our ablation studies.
Revised: In the following section, we examine the role of the forward objective term on touch interaction and possible ways to encode agent’s observation in our ablation studies.

Original: We compare HaC and HaC-Pure with SAC and state-of-the-art vision-based curiosity baselines on the remaining downstream tasks in MiniTouch.
Revised: We compare HaC and HaC-Pure with SAC and state-of-the-art vision-based curiosity baselines on the remaining downstream tasks in MiniTouch.

Original: From Figure 4 it is evident that HaC and HaCPure perform better than SAC in all the tasks and better than the vision-based curiosity models in the majority of the tasks.
Revised: From Figure 4 it is evident that HaC and HaCPure perform better than SAC in all the tasks and better than the vision-based curiosity models in the majority of the tasks.

Original: Using SAC alone hinders the performance and is often unable to solve any of the three tasks.
Revised: Using SAC alone hinders the performance and is often unable to solve any of the three tasks.

Original: This is not surprising since the model is not motivated enough to collect diverse and useful data through interaction.
Revised: This is not surprising since the model is not motivated enough to collect diverse and useful data through interaction.

Original: We hypothesize that HaC-Pure without the visual prediction objective L fdm could potentially bias the solutions towards more physical interaction, which is not necessary for every task.
Revised: We hypothesize that HaC-Pure without the visual prediction objective L fdm could potentially bias the solutions towards more physical interaction, which is not necessary for every task.

Original: For instance, constant interaction in the pushing task could be seen as a hindrance as it does not let the object slide easily towards the target.
Revised: For instance, constant interaction in the pushing task could be seen as a hindrance as it does not let the object slide easily towards the target.

Original: Observe that ICM performs better than HaC-Pure in the pushing task, however, HaC dominates in performance byabout 15%.
Revised: Observe that ICM performs better than HaC-Pure in the pushing task, however, HaC dominates in performance by

Original: Recall that the goal is not just to succeed but to help attain success in a sample efﬁcient

Original: [Figure]
Revised: [Figure]

Original:  manner in fewer steps.
Revised: about 15%. Recall that the goal is not just to succeed but to help attain success in a sample efﬁcient manner in fewer steps.

Original: The results support our hypothesis that cross-modal curiosity enables an RL agent to succeed at an early stage in training and often without any external reward.
Revised: The results support our hypothesis that cross-modal curiosity enables an RL agent to succeed at an early stage in training and often without any external reward.

Original: Similarly, our model outperforms on the opening task without external reward.
Revised: Similarly, our model outperforms on the opening task without external reward.

Original: However, HaC initially has lower success compared to RND but surpasses RND towards the end.
Revised: However, HaC initially has lower success compared to RND but surpasses RND towards the end.

Original: Although HaC and HaC-Pure attain similar success in the pick-up task towards the end of the training, it is compelling to note that HaCPure attains faster convergence.
Revised: Although HaC and HaC-Pure attain similar success in the pick-up task towards the end of the training, it is compelling to note that HaCPure attains faster convergence.

Original: This is because the picking task requires constant touch interaction (where HaC-Pure has an advantage), as opposed to diverse object movement.
Revised: This is because the picking task requires constant touch interaction (where HaC-Pure has an advantage), as opposed to diverse object movement

Original: 5.1 Ablations
Revised: 5.1 Ablations

Original: Forward objective and tuning Lambda Visual forward prediction L fdm plays an important role when it is used in the right proportion.
Revised: Forward objective and tuning Lambda Visual forward prediction L fdm plays an important role when it is used in the right proportion.

Original: Our intrinsic reward is a weighted combination of crossmodal prediction and forward prediction as deﬁned in Eq.4.
Revised: Our intrinsic reward is a weighted combination of crossmodal prediction and forward prediction as deﬁned in Eq.4.

Original: Figure 5 illustrates the model behavior with different levels of emphasis on the forward loss term, with  uniformly sampled between 0 and 1.
Revised: Figure 5 illustrates the model behavior with different levels of emphasis on the forward loss term, with λ uniformly sampled between 0 and 1.

Original: Higher weights indicate that the future prediction dominates force/torque prediction.
Revised: Higher weights indicate that the future prediction dominates force/torque prediction.

Original: This leads  to more object movement but lesser robot’s constant touch-interaction with the underlying objects.[Table]
Revised: This leads [Table] to more object movement but lesser robot’s constant touch-interaction with the underlying objects.

Original: This is useful in tasks such as Pushing.
Revised: This is useful in tasks such as Pushing.

Original: A smaller  value leads to better inactive behavior which is handy in tasks such as grasping.
Revised: A smaller λ value leads to better inactive behavior which is handy in tasks such as grasping.

Original: In our experiments we choose an intermediate value ( e.g. 0.5) that works best for all of our tasks
Revised: In our experiments we choose an intermediate value ( e.g. 0.5) that works best for all of our tasks.

Original: Haptics in future prediction.
Revised: Haptics in future prediction.

Original: The goal of this experiment is to strengthen the argument of crossmodal association.
Revised: The goal of this experiment is to strengthen the argument of crossmodal association.

Original: While conducting experiments, it is important to deduce information of one modality from another modality in a related manner than simply adding another modality on top of visual information.
Revised: While conducting experiments, it is important to deduce information of one modality from another modality in a related manner than simply adding another modality on top of visual information.

Original: We created an additional baseline, ICM+ haptics , where in addition to the visual prediction model we include the haptics-based future prediction model.
Revised: We created an additional baseline, ICM+ haptics , where in addition to the visual prediction model we include the haptics-based future prediction model.

Original: We hypothesize
Revised: We hypothesize

Original: ICM+ haptics to perform better than ICM as it has additional information (haptics).
Revised: ICM+ haptics to perform better than ICM as it has additional information (haptics).

Original: The hapticsbased future prediction model takes a touch vector as input and predicts the touch vector for the next time step.
Revised: The hapticsbased future prediction model takes a touch vector as input and predicts the touch vector for the next time step.

Original: Table 1 compares HaC and ICM+ haptics on MiniTouch tasks and we observe that
Revised: Table 1 compares HaC and ICM+ haptics on MiniTouch tasks and we observe that

Original: ICM+ haptics is better than ICM but compares below HaC. Table 4 in the supplementary compares
Revised: ICM+ haptics is better than ICM but compares below HaC. Table 4 in the supplementary compares

Original: RND based baseline.
Revised: RND based baseline.

Original: Latent features for forward dynamics.
Revised: Latent features for forward dynamics.

Original: Choosing ideal embedding space for decoding the touch vector (Figure 1a) and for predicting future state (Figure. 1b) is important.
Revised: Choosing ideal embedding space for decoding the touch vector (Figure 1a) and for predicting future state (Figure. 1b) is important.

Original: Existing approaches rely on a pretext inverse dynamics (ID) task of predicting the agent’s action given its current and next states [9].
Revised: Existing approaches rely on a pretext inverse dynamics (ID) task of predicting the agent’s action given its current and next states [9].

Original: Another simple yet strong method is to use features from a random but ﬁxed initialization of the encoder [12].
Revised: Another simple yet strong method is to use features from a random but ﬁxed initialization of the encoder [12].

Original: In our work, we learn the features by leveraging the self-supervised pretext task of predicting one modality from the other.
Revised: In our work, we learn the features by leveraging the self-supervised pretext task of predicting one modality from the other.

Original: Figure 6 compares (1) encoder, i.e. learned through crossmodal prediction(HaC) (2) random feature encoder(HaC-RF) (3) encoder learned through ID task (HaC-IDF).
Revised: Figure 6 compares (1) encoder, i.e. learned through crossmodal prediction(HaC) (2) random feature encoder(HaC-RF) (3) encoder learned through ID task (HaC-IDF).

Original: In each case, the decoder network is optimized through touch prediction.
Revised: In each case, the decoder network is optimized through touch prediction.

Original: We observe that the random features variant is stable and effective on both HaC and ICM models.
Revised: We observe that the random features variant is stable and effective on both HaC and ICM models.

Original:  Conclusion
Revised:  Conclusion

Original: Building self-seeking RL agents that can leverage cross-modal associations could be a key research direction in developmental robotics.
Revised: Building self-seeking RL agents that can leverage cross-modal associations could be a key research direction in developmental robotics.

Original: Sense of touch is ubiquitous in human routine.
Revised: Sense of touch is ubiquitous in human routine.

Original: Borrowing insights from how infants learn to explore out of curiosity by touching what they visually perceive, we formulated Haptics-based Curiosity (HaC) that is aimed at encouraging exploration via haptic interaction with the environment.
Revised: Borrowing insights from how infants learn to explore out of curiosity by touching what they visually perceive, we formulated Haptics-based Curiosity (HaC) that is aimed at encouraging exploration via haptic interaction with the environment.

Original: We demonstrated in this work, that by involving additional modalities, the performance of curiosity-based systems on downstream tasks can be increased.
Revised: We demonstrated in this work, that by involving additional modalities, the performance of curiosity-based systems on downstream tasks can be increased.

Original: We observed increased interaction with target objects, and presented evidence that HaC learns to solve the MiniTouch benchmark tasks in an efﬁcient manner while vanilla RL algorithms and vision-based curiosity formulations struggled.
Revised: We observed increased interaction with target objects, and presented evidence that HaC learns to solve the MiniTouch benchmark tasks in an efﬁcient manner while vanilla RL algorithms and vision-based curiosity formulations struggled.

Original: Force/torque sensing is widespread in the lab and industrial robots but while there are plenty of robotic benchmarks, we believe that tactile feedback is an under-explored modality and by releasing our benchmark, we hope to enable future research in this exciting area.
Revised: Force/torque sensing is widespread in the lab and industrial robots but while there are plenty of robotic benchmarks, we believe that tactile feedback is an under-explored modality and by releasing our benchmark, we hope to enable future research in this exciting area.

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to sincerely thank Glen Berseth, Deepak Pathak, Edward Smith and Krishna Murthy for valuable feedback and insightful comments on the paper.  
Revised: We would like to sincerely thank Glen Berseth, Deepak Pathak, Edward Smith and Krishna Murthy for valuable feedback and insightful comments on the paper.  Supplementary Material

Revised: In this Supplementary Material section, we provide additional details concerning various elements which could not be elaborated on in the main paper.

Revised: We begin with a detailed description of the proposed MiniTouch benchmark.

Revised: We outline the action space, state space, and reward structure for each of the tasks.

Revised: This is followed by a closer look into the various aspects of the haptics-control module including architectures, experimental procedures, hyper-parameters, and additional results.

Revised: We include an ablation experiment at the end that investigates the usefulness larger exploration phase while solving each of the tasks.

Revised: 7.1 MiniTouch Tasks

Revised: This section describes the cross-modal benchmark of simulated manipulation tasks, which we make use of and release as part of the paper.

Revised: Unlike these prior simulation benchmarks, we particularly focus on providing a platform where one can use cross-modal information to solve diverse manipulation tasks.

Revised: Existing benchmarks Yu et al.

Revised: [55] do not include touch modality.

Revised: An overview of the tasks in MiniTouch is outlined in Figure 2.

Revised: These tasks are built off Pybullet [54] physics engine and contain different scene setups for each of the four tasks.

Revised: The details of each of these tasks are further expanded.

Revised: All the four tasks are compiled together as a “MiniTouch” benchmark suitable for evaluating interaction-based algorithms.

Revised: Playing: This environment is intended as a toy task to evaluate interaction frequency and does not feature any reward beyond interaction count.

Revised: A cube is placed in a random position on a table at each episode.

Revised: The agent needs to localize and interact with the cube.

Revised: Pushing:

Revised: In this task, the agent needs to push an object placed randomly on a table to a target (visually indicated as a gray cube).

Revised: The object position is sampled uniformly in polar coordinates around the target object (i.e. angle 0 to 360 degrees, distance 10 to 20 centimeters).

Revised: The end effector’s start position is sampled in the same way as the target position.

Revised: In addition, the orientation of the gripper is ﬁxed to be perpendicular to the ground all the time.

Revised: The robot agent succeeds and receives a reward of +25 if the distance between the cube and the target object is less thancentimeters.

Revised: A new episode starts if the agent succeeds.

Revised: The environment also restarts if the cube is placed or pushed beyond a predeﬁned bounding box comprising of acceptable positions on the table ( i.e. positions that can be reached by the robot hand).

Revised: Opening: A cabinet with a door is randomly placed in reach of the agent.

Revised: The goal is to ﬁnd the door handle and open the door.

Revised: The gripper orientation is ﬁxed to point its ﬁngers towards the door, parallel to the ground.

Revised: For this task, the ﬁngers are discretized to be open or closed.

Revised: In addition, a ﬁfth element is added to the action vector to control the yaw (relative rotation of the end-effector) to be able to approach the door.

Revised: The robot succeeds and receives a reward of +25 when the angle of the door opening reaches thirty degrees or higher.

Revised: Similar to the pushing task, a new episode starts if the agent reaches the goal.

Revised: Pick-up: In this environment, the agent needs to grasp and lift a randomly placed object.

Revised: The agent’s goal is to lift the object 5cm above the table.

Revised: The agent receives a reward of +25 upon success.

Revised: The object is placed uniformly randomly on a table.

Revised: Similar to the Opening task, the end effector opening/closing is discretized, meaning when its internal continuous variable is below a threshold, the gripper closes, otherwise it remains open.

Revised: All of the tasks are implemented in the Pybullet physics engine [54], which is a free and open-source library that enables fast simulation.

Revised: 7.2 Code 7.2.1 MiniTouch Library:

Revised: The task environment used in the experiments is packaged and released as a python library 1 that can be easily plugged into the training code.

Revised: Setup instructions, code, and other details can be found in the README ﬁle included in the repository.

Revised: 7.2.2 HaC and baselines:

Revised: We used the following open-source implementations for the baselines.

Revised: We were able to reproduce the results from their papers before attempting to use them as baselines for our model:

Revised: ICM

Revised: https://github.com/openai/large-scale-curiosity

Revised: Disagreement :

Revised: https://github.com/danijar/dreamerv https://github.com/pathak22/exploration-by-disagreement

Revised: RND :

Revised: https

Revised: //github.com/openai/random-network-distillation

Revised: 7.3 Experimental Details

Revised: State space: The input states are a combination of visual and touch vector input.

Revised: The visual input is a grayscale rendering of the scene with dimension 84 × 84 , pre-processed similarly to Mnih et al.

Revised: Image observations are captured from a static camera overlooking each scene.

Revised: The touch vector input is composed of the 3-dimensional end-effector position, 2-dimensional ﬁnger position, ranging from 0 to 1, each denoting how far apart each ﬁnger is, and the 6-dimensional force/torque values.

Revised: In total, the touch vector is 11-dimensional, S ∈ R 11 .

Revised: Action space: Actions are expressed as 4-dimensional continuous vectors.

Revised: The ﬁrst three elements describe the desired offset for the end effector at the next timestep based on its current position.

Revised: The last dimension controls the relative desired distance between the two ﬁngers at the next timestep.

Revised: Training We use SAC [51] as the optimizer for our agent.

Revised: Our training is composed of two phases as described in the main paper.

Revised: (i)Exploration phase, (ii)Downstream task phase.

Revised: In the exploratory phase (curiosity part) agent is trained using our intrinsic reward alone.

Revised: In the task phase, network weights are seeded from the ones in the exploratory phase.

Revised: We also retain the replay buffer from the exploratory phase in the downstream task phase.

Revised: Duration of the exploration phase can be adjusted in the code using a hyper-parameter stop-curiosity .

Revised: We have two hyper-parameters that change between the two phases, (i) α and (ii) the learning rate of the SAC algorithm.

Revised: Details of hyper-parameters used for our experiments are outlined in Table 2 and Table 3.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Convolution

Revised: [Table]

Revised: LeakyReLU

Revised: Convolution

Revised: [Table]

Revised: Convolution

Revised: [Table]

Revised: LeakyReLU

Revised: Convolution

Revised: [Table]

Revised: [Table]

Revised: Fully-connected

Revised: [Table]

Revised: 7.4 Object Interaction

Revised: As touched up in the experiments section of the main paper, Figure 9 depicts the touch interaction and door movement metrics for the Opening task.

Revised: We make a similar observation to the result showed in Figure 3 for the Playing task.

Revised: A higher touch-interaction need not indicate better objectmovement.

Revised: Agent can resort to constantly engaging with the object in a passive manner.

Revised: HaC collects rich interaction data during the exploratory phase and helps the agent in terms of sample efﬁciency while solving the downstream tasks.

Revised: 7.5 Additional Ablations

Revised: Does longer exploration help?

Revised: We observe that having a longer exploratory phase of training with intrinsic reward alone usually beneﬁts the overall performance.

Revised: We can observe from Figurethat HaC attains decent success in the exploratory phase without any external reward on most of the tasks.

Revised: This is because it encourages better associations and a larger collection of interesting conﬁgurations in the replay.

Revised: The effect of the exploratory step is further studied and the results on

Revised: [Table]

Revised: all of the downstream tasks with different duration of exploration are compiled in Figure 10.

Revised: We depict the episode convergence steps and success rate to visualize the trend as expected.

Revised: Touch in Random Network Distillation We created an additional baseline RND+ touch similar to touch in future prediction (ICM+ touch ) baseline.

Revised: In this setting, the input to the random target network and predictor network (that predicts the outcome of target network) consists of both touch and visual feature vector concatenated.

Revised: The baseline beats the vision only version by a margin for both Pushing and picking tasks as shown in the Table.

Revised: 4, however for the Open-door tasks the improvement is not that signiﬁcant.

Revised: Although RND+ touch is comparable with HaC for the

Revised: Pushing task on the success rate, HaC has better sample efﬁciency as measured by episode steps.

Revised: We believe this is because object movement is crucial for the pushing task compared to interaction, and prediction based techniques such as RND could be helpful in such settings.

Revised: Comparisons with Generalization to Novel shapes: It is desirable for an agent to be able to handle diverse shapes in order to be robust across arbitrary manipulation settings.

Revised: We study this using an environment in which an object is sampled from a thousand procedurally generated objects.

Revised: The objects are dissimilar with respect to shape and mass but are sampled from the same generative distribution.

Revised: Out of 1000 different objects, 800 of them are used in the training phase, and we evaluate the agent’s effectiveness on the remaining 200 unseen object shapes.

Revised: Although generalization to novel shapes is not the problem setting our method focuses on.

Revised: We perform this proof of concept ablations to investigate that HaC parameters in the exploratory phase are not just reusable across different tasks with similar shapes (Eg: Playing, Pushing, and Picking), but also can help generalize when applied to distinctive shapes sampled from a similar distribution.

Revised: Figure 11 shows touch interaction and object movement evaluations for a single object exploration task.

Revised: We compare HaC with ICM and Disagreement baselines. .

Revised: The results validate that our model generalizes to unseen object conﬁgurations.

Revised: 7.6 Real world use cases:

Revised: It is possible to deﬁne safety boundaries for a robot arm (e.g. don’t hit the table, don’t move outside the arena boundaries) and have a robot arm autonomously and task-independently collect data for phase 1 of our method.

Revised: In addition to this, we would like to point out that (a) in the pushing and pick-up tasks, only 5-10k steps are required for decent performance on the downstream task and (b) what we call a “step” is however long it takes the inverse kinematic solver to move the arm to the new end effector pose (that was generated by the policy).

Revised: If many of the generated poses cluster together, that dramatically reduces runtime of the curiosity phase making it more suitable to realworld applicability.

Revised: While no real robot data have been used in the experiments, our method is data agnostic.

Revised: We note that the physics engine can be replaced by real-world data as there is no bias introduced in process.

Revised: The networks used for the curiosity prediction task can be easily adapted, if needed, to accommodate inputs from a real robotic platform.

Revised: As a next step and when restrictions are eased, we plan to implement this method on a physical system and replicate our experiments.

Revised: [Figure]