Original: Touch-based Curiosity for Sparse-Reward Tasks
Revised: Haptics-based Curiosity for Sparse-Reward Tasks

Original: Anonymous Author(s)
Revised: Element AI, Montr´eal, 4 CIFAR Fellow

Original: Abstract: Robots in many real-world settings have access to force/torque sensorsin their gripper and tactile sensing is often necessary in tasks that involve contactrich motion.
Revised: Abstract: Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion.

Original: In this work, we leverage surprise from mismatches in touch feedback to guide exploration in hard sparse-reward reinforcement learning tasks.
Revised: In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks.

Original: Ourapproach, Touch-based Curiosity (ToC), learns what visible objects interactionsare supposed to “feel” like.
Revised: Our approach, Haptics-based Curiosity (HaC), learns what visible objects interactions are supposed to “feel” like.

Original: We encourage exploration by rewarding interactionswhere the expectation and the experience do not match.
Revised: We encourage exploration by rewarding interactions where the expectation and the experience do not match.

Original: We test our approach ona range of touch-intensive robot arm tasks (e.g. pushing objects, opening doors),which we also release as part of this work.
Revised: We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work.

Original: Across multiple experiments in asimulated setting, we demonstrate that our method is able to learn these difﬁculttasks through sparse reward and curiosity alone.
Revised: Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difﬁcult tasks through sparse reward and curiosity alone.

Original: We compare our cross-modalapproach to single-modality (touch- or vision-only) approaches as well as othercuriosity-based methods and ﬁnd that our method performs better and is moresample-efﬁcient.
Revised: We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and ﬁnd that our method performs better and is more sample-efﬁcient.

Original:  Introduction
Revised:  Introduction

Original: Most successes in reinforcement learning (RL) come from games [1, 2] or scenarios where thereward is strongly shaped [3, 4].
Revised: Most successes in reinforcement learning (RL) come from games [1, 2] or scenarios where the reward is strongly shaped [3, 4].

Original: In the former the environment is often fully observable and thereward is dense and well-deﬁned.
Revised: In the former, the environment is often fully observable and the reward is dense and well-deﬁned.

Original: In the latter, a large amount of work is required to design usefulreward functions.
Revised: In the latter, a large amount of work is required to design useful reward functions.

Original: While it may be possible to hand-craft dense reward signals for many real-worldtasks, we believe that it is a worthwhile endeavor to investigate learning methods that do not requiredense rewards.
Revised: While it may be possible to hand-craft dense reward signals for many real-world tasks, we believe that it is a worthwhile endeavor to investigate learning methods that do not require dense rewards.

Original: Closely related to the sparse rewards problem is the issue of exploration.
Revised: Closely related to the sparse rewards problem is the issue of exploration.

Original: One reason that traditional
Revised: One reason that traditional

Original: RL agents struggle with sparse-reward problems is a lack of exploration.
Revised: RL agents struggle with sparse-reward problems is a lack of exploration.

Original: An agent may not obtainuseful rewards without an intuitive exploration strategy when rewards are sparse.
Revised: An agent may not obtain useful rewards without an intuitive exploration strategy when rewards are sparse.

Original: Exploration basedon intrinsic curiosity comes naturally to many animals and infants (who start crawling and exploringthe environment at around 9 months [5] and oftentimes even before they can crawl by using theirhands and mouth to touch and probe objects).
Revised: Exploration based on intrinsic curiosity comes naturally to many animals and infants (who start crawling and exploring the environment at around 9 months [5] and oftentimes even before they can crawl by using their hands and mouth to touch and probe objects).

Original: Touch is a local experience and encodes accurategeometrical information while handling objects.
Revised: Touch is a local experience and encodes accurate geometrical information while handling objects.

Original: Experimental studies in infants has suggested thattactile and visual sensory modalities play a central role in systematic learning of tasks related toobject understanding, interaction and manipulation [6, 7].
Revised: Experimental studies in infants has suggested that tactile and visual sensory modalities play a central role in systematic learning of tasks related to object understanding, interaction and manipulation [6, 7].

Original: Ideally, we would like our RL agents to explore the environment in an analogous self-guided fashionto learn the dynamics and object properties, and use this knowledge to solve downstream tasks.
Revised: Ideally, we would like our RL agents to explore the environment in an analogous self-guided fashion to learn the dynamics and object properties, and use this knowledge to solve downstream tasks.

Original: Justas how humans utilize different sensory modalities to explore and understand the world around them,exploration in robots should be more embodied and related to a combination of vision and touch andpotentially other sensor modalities.
Revised: Just as how humans utilize different sensory modalities to explore and understand the world around them, exploration in robots should be more embodied and related to a combination of vision and touch and potentially other sensor modalities.

Original: We believe that building autonomous agents that are self-drivenand seek to explore via multi-modal interaction are crucial to address key issues in developmentalrobotics.
Revised: We believe that building autonomous agents that are self-driven and seek to explore via multi-modal interaction are crucial to address key issues in developmental robotics.

Original: Recent works in RL have focused on a curiosity-driven exploration through prediction-based surprise [8, 9, 10].
Revised: Recent works in RL have focused on a curiosity-driven exploration through prediction-based surprise [8, 9, 10].

Original: In this formulation, a forward dynamics model predicts the future, and if its prediction is incorrect when compared to the real future, the agent is surprised and is thus rewarded.
Revised: In this formulation, a forward dynamics model predicts the future, and if its prediction is incorrect when compared to the real future, the agent is surprised and is thus rewarded.

Original: This encourages the agent to look for novel states while improving its visual forward model in return.
Revised: This encourages the agent to look for novel states while improving its visual forward model in return.

Original: However, this formulation can be practically challenging to optimize since there are manystates that are visually dissimilar but practically irrelevant (e.g. for a pushing task, moving a roboticend-effector without touching the object creates visual novelty but contributes little to task-relatedknowledge).
Revised: However, this formulation can be practically challenging to optimize since there are many states that are visually dissimilar but practically irrelevant (e.g. for a pushing task, moving a robotic end-effector without touching the object creates visual novelty but contributes little to task-related knowledge).

Original: One way to constrain this search space over curious behaviors is by involving anothermodality like touch
Revised: One way to constrain this search space over curious behaviors is by involving another modality like haptics.

Original: In this work, we demonstrate that a self-guided cross-modal exploration policy can help solve sparsereward downstream tasks that existing methods without this curiosity struggle to solve.
Revised: In this work, we demonstrate that a self-guided cross-modal exploration policy can help solve sparsereward downstream tasks that existing methods without this curiosity struggle to solve.

Original: Our methoduses cross-modal consistency (mismatch between visual and touch signal) to guide this exploration.
Revised: Our method uses cross-modal consistency (mismatch between visual and haptic signal) to guide this exploration.

Original: To use self-play knowledge in downstream tasks, we relabel past experiences, providing a densereward signal that allows modern off-policy RL methods to solve the tasks.
Revised: To use self-play knowledge in downstream tasks, we relabel past experiences, providing a dense reward signal that allows modern off-policy RL methods to solve the tasks.

Original: While there are manyexisting methods that use artiﬁcial curiosity/intrinsic motivation, the majority of these methods either rely on strong domain knowledge (e.g. labels of state dimensions in Forestier et al.
Revised: While there are many existing methods that use artiﬁcial curiosity/intrinsic motivation, the majority of these methods either rely on strong domain knowledge (e.g. labels of state dimensions in Forestier et al.

Original: [4], a goalpicking strategy in Andrychowicz et al.
Revised: [4], a goalpicking strategy in Andrychowicz et al.

Original: [11]) or are prone to get stuck in local optima when a singlemeaningless stimulus creates enough surprise to capture the attention of the agent (e.g. noisy-TVexperiment from [12]).
Revised: [11]) or are prone to get stuck in local optima when a single meaningless stimulus creates enough surprise to capture the attention of the agent (e.g. noisy-TV experiment from [12]).

Original: Other approaches depend on unrealistic assumptions and goal conditioning [11].
Revised: Other approaches depend on unrealistic assumptions and goal conditioning [11].

Original: Our method presents a novel approach in the family of prediction-based models [8, 13, 9]and yields better performance on a wide range of robotic manipulation tasks than purely visionbased and touch-based approaches [8, 9].
Revised: Our method presents a novel approach in the family of prediction-based models [8, 13, 9] and yields better performance on a wide range of robotic manipulation tasks than purely visionbased and haptics-based approaches [8, 9].

Original: The tasks are chosen with careful consideration—theycomprise of preliminary robotic manipulations such as grasping, pushing, and pulling.
Revised: The tasks are chosen with careful consideration—they comprise of preliminary robotic manipulations such as grasping, pushing, and pulling.

Original: In this work,we present the following contributions:
Revised: In this work, we present the following contributions:

Original: • A new curiosity method to help solve sparse-reward tasks that use cross-modal consistency(predicting one modality from another) to guide exploration.
Revised: • A new curiosity method to help solve sparse-reward tasks that use cross-modal consistency (predicting one modality from another) to guide exploration.

Original: We implement it in this work as vision and touch modalities, but the formulation of our method does not require any knowledge about the underlying modalities and can thus be applied to other settings.
Revised: We implement it in this work as vision and touch modalities, but the formulation of our method does not require any knowledge about the underlying modalities and can thus be applied to other settings.

Original: • We create and maintain a manipulation benchmark of simulated tasks, MiniTouch , inspiredby Chen et al.
Revised: • We create and maintain a manipulation benchmark of simulated tasks, MiniTouch , inspired by Chen et al.

Original: [14], Andrychowicz et al.
Revised: [14], Andrychowicz et al.

Original: [11], where the robotic arm is equipped with a force/torque sensor.
Revised: [11], where the robotic arm is equipped with a force/torque sensor.

Original: This allows evaluation of models’ performance on different manipula- tion tasks that can leverage cross-modal learning.
Revised: This allows evaluation of models’ performance on different manipulation tasks that can leverage cross-modal learning.

Original: • We validate the performance of our method on MiniTouch environment comprising of fourdownstream tasks
Revised: • We validate the performance of our method on MiniTouch environment comprising of four downstream tasks.

Original: We compare purely vision-based curiosity approaches and standard offpolicy RL algorithms.
Revised: We compare purely vision-based curiosity approaches and standard offpolicy RL algorithms.

Original: Our method improves both performance and sample efﬁciency.
Revised: Our method improves both performance and sample efﬁciency.

Original:  Related Work
Revised:  Related Work

Original: Intrinsic Motivation Intrinsic motivation is an inherent spontaneous tendency to be curious or toseek something novel in order to further enhance one’s skill and knowledge [15, 16, 17].
Revised: Intrinsic Motivation Intrinsic motivation is an inherent spontaneous tendency to be curious or to seek something novel in order to further enhance one’s skill and knowledge [15, 16, 17].

Original: This principle is shown to work well even in the absence of a well-deﬁned goal or objective.
Revised: This principle is shown to work well even in the absence of a well-deﬁned goal or objective.

Original: In reinforcementlearning, intrinsic motivation has been a well-researched topic [18, 19, 20, 21, 22, 23].
Revised: In reinforcement learning, intrinsic motivation has been a well-researched topic [18, 19, 20, 21, 22, 23].

Original: An intuitiveway to perform intrinsic motivation is through the use of “novelty discovery”.
Revised: An intuitive way to perform intrinsic motivation is through the use of “novelty discovery”.

Original: For example, incentivize the RL agent to visit unusual states or states with substantial information gain [24].
Revised: For example, incentivize the RL agent to visit unusual states or states with substantial information gain [24].

Original: In its simplest form, this can be achieved with up-weighting mechanisms such as state visitation counts [25].
Revised: In its simplest form, this can be achieved with up-weighting mechanisms such as state visitation counts [25].

Original: Count-based methods have also been extended to high-dimensional state spaces [26, 12, 27].
Revised: Count-based methods have also been extended to high-dimensional state spaces [26, 12, 27].

Original: Alternative forms of intrinsic motivation include disagreement [28], empowerment [29, 30].
Revised: Alternative forms of intrinsic motivation include disagreement [28], empowerment [29, 30].

Original: Exploratory intrinsic motivation can also be achieved through “curiosity” [18, 31].
Revised: Exploratory intrinsic motivation can also be achieved through “curiosity” [18, 31].

Original: In this setting,an agent is encouraged to visit states with high predictive errors [10, 8, 9] by training a forwarddynamics model that predicts the future state given the current state and action.
Revised: In this setting, an agent is encouraged to visit states with high predictive errors [10, 8, 9] by training a forward dynamics model that predicts the future state given the current state and action.

Original: Instead of makingpredictions in the raw visual space, Pathak et al.
Revised: Instead of making predictions in the raw visual space, Pathak et al.

Original: [9] mapped images to a feature space where relevantinformation is represented via an inverse dynamics model.
Revised: [9] mapped images to a feature space where relevant information is represented via an inverse dynamics model.

Original: Burda et al.
Revised: Burda et al.

Original: [8] demonstrate that randomfeatures are sufﬁcient for many popular RL game benchmarks.
Revised: [8] demonstrate that random features are sufﬁcient for many popular RL game benchmarks.

Original: This approach may work well withtasks that require navigation to ﬁnd a reward because each unseen position of the agent in the worldleads to high intrinsic reward when unseen.
Revised: This approach may work well with tasks that require navigation to ﬁnd a reward because each unseen position of the agent in the world leads to high intrinsic reward when unseen.

Original: However, in the case of manipulative tasks, we are lessinterested in the robot visiting all the possible states and more interested in states where the robotinteracts with other objects.
Revised: However, in the case of manipulative tasks, we are less interested in the robot visiting all the possible states and more interested in states where the robot interacts with other objects.

Original: In this work, we leverage multimodal inputs that encourage the agentto ﬁnd novel combinations of visual and force/torque modalities.
Revised: In this work, we leverage multimodal inputs that encourage the agent to ﬁnd novel combinations of visual and force/torque modalities.

Original: Self-Supervised Learning via Cross-modality Exploiting multimodality to learn unsupervisedrepresentations dates back to at least 1993 [32].
Revised: Self-Supervised Learning via Cross-modality Exploiting multimodality to learn unsupervised representations dates back to at least 1993 [32].

Original: Multimodal signals are naturally suitable for selfsupervised learning, as information from one modality can be used to supervise learning for anothermodality [33, 34].
Revised: Multimodal signals are naturally suitable for selfsupervised learning, as information from one modality can be used to supervise learning for another [Figure] modality [33, 34].

Original: Different modalities typically carry different information, e.g., visual and touchsensory modalities emerge concurrently and often in an interrelated manner during contact-richmanipulation tasks [35].
Revised: Different modalities typically carry different information, e.g., visual and touch sensory modalities emerge concurrently and often in an interrelated manner during contact-rich manipulation tasks [35].

Original: Speciﬁcally, force/torque motor signal has always been a major componentin the literature of perception and control [36, 37, 38].[Figure]
Revised: Speciﬁcally, force/torque motor signal has always been a major component in the literature of perception and control [36, 37, 38].

Original: The most common ways to leverage multimodal signals to learn representations are through visionand language [39, 40].
Revised: The most common ways to leverage multimodal signals to learn representations are through vision and language [39, 40].

Original: Gao et al.
Revised: Gao et al.

Original: [41], Li et al.
Revised: [41], Li et al.

Original: [42] demonstrated a uniﬁed approach to learningrepresentations for prediction tasks using visual and touch data.
Revised: [42] demonstrated a uniﬁed approach to learning representations for prediction tasks using visual and touch data.

Original: Chen et al.
Revised: Chen et al.

Original: [43] learn world-modelsfrom multimodal data via a shared latent space.
Revised: [43] learn world-models from multimodal data via a shared latent space.

Original: In robotics and interactive settings, the use ofmodalities such as tactile sensing [44, 45, 46] is increasingly popular for grasping, manipulation andother externally-speciﬁed tasks [47, 48].
Revised: In robotics and interactive settings, the use of modalities such as tactile sensing [44, 45, 46] is increasingly popular for grasping, manipulation and other externally-speciﬁed tasks [47, 48].

Original: Lee et al.
Revised: Lee et al.

Original: [49] showed the effectiveness of self-supervisedtraining of tactile and visual representations by demonstrating its use on a peg insertion task.
Revised: [49] showed the effectiveness of self-supervised training of tactile and visual representations by demonstrating its use on a peg insertion task.

Original: While the mentioned approaches have used multiple sensory modalities for learning better representations, in this work we demonstrate its utility for allowing agents to explore.
Revised: While the mentioned approaches have used multiple sensory modalities for learning better representations, in this work we demonstrate its utility for allowing agents to explore.

Original: Similar to ours, Deanet al.
Revised: Similar to ours, Dean et al.

Original: [50] use multimodal sensory association (i.e. audio and visual) to compute the intrinsic reward.
Revised: [50] use multimodal sensory association (i.e. audio and visual) to compute the intrinsic reward.

Original: Their curiosity-based formulation allows an agent to efﬁciently explore the environment insettings where audio and visual signals are governed by the same physical processes.
Revised: Their curiosity-based formulation allows an agent to efﬁciently explore the environment in settings where audio and visual signals are governed by the same physical processes.

Original: In addition tothe different nature of sensory signals, they use a discriminator that determines whether an observedmultimodal pair is novel.
Revised: In addition to the different nature of sensory signals, they use a discriminator that determines whether an observed multimodal pair is novel.

Original: This might not work in our case as touch is a more sparse signal and usinga discriminator could lead to ambiguous outcomes.
Revised: This might not work in our case as touch is a more sparse signal and using a discriminator could lead to ambiguous outcomes.

Original:  Proposed Approach
Revised:  Proposed Approach

Original: The goal of our method, Touch-Based Curiosity (ToC), is to encourage the agent to interact withobjects.
Revised: The goal of our method, Haptics-Based Curiosity (HaC), is to encourage the agent to interact with objects.

Original: ToC provides a reward signal for an RL agent to explore the state space of a task thatinvolves interacting with objects.
Revised: HaC provides a reward signal for an RL agent to explore the state space of a task that involves interacting with objects.

Original: The exploration phase is independent of the downstream task, i.e.,relying solely on visual and force/torque signals, without a reward signal from the downstream task.
Revised: The exploration phase is independent of the downstream task, i.e., relying solely on visual and force/torque signals, without a reward signal from the downstream task.

Original: Similar to how people spend more time exploring stimuli that are more incongruous [7], ToC guidesthe agent to focus its experience on different novel cross-modal associations.
Revised: Similar to how people spend more time exploring stimuli that are more incongruous [7], HaC guides the agent to focus its experience on different novel cross-modal associations.

Original: We augment thisintrinsic objective with a future visual state prediction objective similar to the one in Pathak et al.[9] to avoid getting stuck in undesired inactive conﬁgurations.
Revised: We augment this intrinsic objective with a future visual state prediction objective similar to the one in Pathak et al. [9] to avoid getting stuck in undesired inactive conﬁgurations.

Original: Note that we sometimes refer to thefuture state prediction objective in the text as forward dynamics objective.
Revised: Note that we sometimes refer to the future state prediction objective in the text as forward dynamics objective.

Original: In this work, we focuson the cross-modality between vision and touch, but the same idea could be applied to other pairs ofsensory domains, such as vision and sound, or touch and acoustics.
Revised: In this work, we focus on the cross-modality between vision and touch, but the same idea could be applied to other pairs of sensory domains, such as vision and sound, or touch and acoustics.

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: The learning problem is formalized as a Markov decision process (MDP) deﬁned by a tuple{S , A , T , R , ⇢ ,  } of states, actions, transition probability, reward, initial state distribution, and130 discount factor.
Revised: The learning problem is formalized as a Markov decision process (MDP) deﬁned by a tuple {S , A , T , R , ⇢ ,  } of states, actions, transition probability, reward, initial state distribution, and discount factor.

Original: The goal is to ﬁnd the optimal policy that maximizes the discounted sum of rewards, ⇡ ⇤ =
Revised: The goal is to ﬁnd the optimal policy that maximizes the discounted sum of rewards, ⇡ ⇤ =

Original: E ⇡ [
Revised: E ⇡ [

Original: P 1 t  t r ( s t , a t )] .
Revised: P 1 t  t r ( s t , a t )] .

Original: In our case, each state s t in the trajectory is composed of bothvisual and corresponding touch features as detailed in the following section.
Revised: In our case, each state s t in the trajectory is composed of both visual and corresponding touch features as detailed in the following section.

Original: We use soft actorcritic policy gradients (SAC) [51] to train our policies, but in principle, our proposed approach isalgorithm-agnostic.
Revised: We use soft actor- critic policy gradients (SAC) [51] to train our policies, but in principle, our proposed approach is algorithm-agnostic.

Original: The policy ⇡ is evaluated with an estimation of the soft Q-value:
Revised: The policy ⇡ is evaluated with an estimation of the soft Q-value:

Original: [Equation]
Revised: [Equation]

Original: where V ( s t ) = E a t ⇠ ⇡ [ Q ( s t , a t )  log ⇡ ( a t
Revised: where V ( s t ) = E a t ⇠ ⇡ [ Q ( s t , a t )  log ⇡ ( a t

Revised: | s t )] is the soft value function.

Original: 3.2 Touch-based Prediction
Revised: 3.2 Haptics-based Prediction

Original: Our core prediction framework consists of two modules: (i) a touch control module, which learnsto predict expected touch signal from the visual input, and (ii) a forward dynamics model, whichpredicts the next latent state from the current latent state and the current action (see Figure 1).
Revised: Our core prediction framework consists of two modules: (i) a haptics control module, which learns to predict expected haptic signal from the visual input, and (ii) a forward dynamics model, which predicts the next latent state from the current latent state and the current action (see Figure 1).

Original: Letthe state of the environment s t =
Revised: Let the state of the environment s t =

Original: ( z t , h t ) at time t be composed of a visual feature z t (the encodedvisual input) and a touch signal h t .
Revised: ( z t , h t ) at time t be composed of a visual feature z t (the encoded visual input) and a haptic signal h t .

Original: The touch prediction model consists of a convolutional encoderz t = enc ( x t ) that transforms the image x t into a latent representation z t and a fully-connecteddecoder ˆ h t
Revised: The touch prediction model consists of a convolutional encoder z t = enc ( x t ) that transforms the image x t into a latent representation z t and a fully-connected decoder ˆ h t

Original: = dec ( z t ) that transforms the latent into a predicted haptic signal ˆ h t .
Revised: = dec ( z t ) that transforms the latent into a predicted haptic signal ˆ h t .

Original: The encoderdecoder is trained with an L2 reconstruction loss, i.e. for every image x t and force/torque sensor h t :
Revised: The encoderdecoder is trained with an L2 reconstruction loss, i.e. for every image x t and force/torque sensor h t :

Original: [Equation]
Revised: [Equation]

Original: A high prediction error on a given image indicates that the agent has had few interactions like this.
Revised: A high prediction error on a given image indicates that the agent has had few interactions like this.

Original: Therefore, to harness this “surprise” to guide exploration, we deﬁne the intrinsic reward at time tduring exploration to be proportional to this reconstruction loss.
Revised: Therefore, to harness this “surprise” to guide exploration, we deﬁne the intrinsic reward at time t during exploration to be proportional to this reconstruction loss.

Original: This essentially allows the agentpolicy to visit under-explored conﬁgurations of the state space by encouraging interactions wherethe system does not know what the target object “feels” like.
Revised: This essentially allows the agent policy to visit under-explored conﬁgurations of the state space by encouraging interactions where the system does not know what the target object “feels” like.

Original: In addition to efﬁcient exploration,being aware of such incongruity via touch prediction aids learning local regularities in the visualinput.
Revised: In addition to efﬁcient exploration, being aware of such incongruity via touch prediction aids learning local regularities in the visual input.

Original: This in turn could assist better generalization to unseen states.
Revised: This in turn could assist better generalization to unseen states.

Original: An overall pipeline of theframework is shown in Figure 1a.
Revised: An overall pipeline of the framework is shown in Figure 1a.

Original: 3.3 Regularization Through Forward Dynamics Model
Revised: 3.3 Regularization Through Forward Dynamics Model

Original: We found empirically (and we demonstrate in the experiments section below) that the surprise stemming from haptic novelty was not enough to cause object-centric interaction.
Revised: We found empirically (and we demonstrate in the experiments section below) that the surprise stemming from haptic novelty was not enough to cause object-centric interaction.

Original: We postulate that byincorporating visual surprise (i.e. the mismatch between predicted forward dynamics and observeddynamics)
Revised: We postulate that by incorporating visual surprise (i.e. the mismatch between predicted forward dynamics and observed dynamics)

Original: [9], we can create an agent that seeks out visual novelty as well as haptic one and thusleads to better state space coverage.
Revised: [9], we can create an agent that seeks out visual novelty as well as haptic one and thus leads to better state space coverage.

Original: To this end, our model is augmented with a forward dynamicsmodel (see Figure-1b)
Revised: To this end, our model is augmented with a forward dynamics model (see Figure-1b)

Original: ˆ z t +1 = fdm ( z t , a t ) that learns to map the latent state z t (obtained from thevisual encoder enc ) and action a t at time t to the predicted latent state ˆ z t +1 at the next timestep.
Revised: ˆ z t +1 = fdm ( z t , a t ) that learns to map the latent state z t (obtained from the visual encoder enc ) and action a t at time t to the predicted latent state ˆ z t +1 at the next timestep.

Original: This model is also trained with L2 loss:
Revised: This model is also trained with L2 loss:

Original: [Equation]
Revised: [Equation]

Original: The intrinsic reward is deﬁned as the convex combination of the cross-modal prediction loss (Eq. 2)and the forward dynamics model loss (Eq. 3):
Revised: The intrinsic reward is deﬁned as the convex combination of the cross-modal prediction loss (Eq. 2) and the forward dynamics model loss (Eq. 3):

Original: [Equation]
Revised: [Equation]

Original: where  2 [0 , 1] is a balancing factor.
Revised: where  2 [0 , 1] is a balancing factor.

Original: The effect of the factor  on overall performance is studied inthe ablation experiments described in Section 5.1.
Revised: The effect of the factor  on overall performance is studied in the ablation experiments described in Section 5.1.

Original: 3.4 Training
Revised: 3.4 Training

Original: Learning is divided into two stages: (i) an exploratory step, where the agent performs free exploration following ToC, and (ii) an adaptation step, where the agent is tasked to solve a downstreamproblem, given a sparse reward.
Revised: Learning is divided into two stages: (i) an exploratory step, where the agent performs free exploration following HaC, and (ii) an adaptation step, where the agent is tasked to solve a downstream problem, given a sparse reward.

Original: During the exploratory step, each trajectory consists of pairs of image and force/torque features, ( z 1 ,h 1 ) , ( z 2 , h 2 ) ,..., ( z n , h n ) .
Revised: During the exploratory step, each trajectory consists of pairs of image and force/torque features, ( z 1 , h 1 ) , ( z 2 , h 2 ) ,..., ( z n , h n ) .

Original: These trajectories are used for two purposes: (i) updating the parametersof the prediction model to help shape the representations and (ii) updating the exploration policybased on the intrinsic reward r t .
Revised: These trajectories are used for two purposes: (i) updating the parameters of the prediction model to help shape the representations and (ii) updating the exploration policy based on the intrinsic reward r t .

Original: Note that we use the touch features only to craft the intrinsic rewardsand the input to RL agent consists of visual features alone to be comparable to the baselines.
Revised: Note that we use the touch features only to craft the intrinsic rewards and the input to RL agent consists of visual features alone to be comparable to the baselines.

Original: Forvision-based curiosity models, Burda et al.
Revised: For vision-based curiosity models, Burda et al.

Original: [8] observed that encoding visual features via a randomnetwork constitute a simple and effective strategy compared to learned features.
Revised: [8] observed that encoding visual features via a random network constitute a simple and effective strategy compared to learned features.

Original: In Section 5.1,we investigate the performance of our model in both scenarios, i.e., when the features are learnedvs random.
Revised: In Section 5.1, we investigate the performance of our model in both scenarios, i.e., when the features are learned vs random.

Original: The overall optimization problem at this step consists of the policy learning (drivenby intrinsic reward), the touch reconstruction loss (Eq. 2), and the forward dynamics loss (Eq. 3).
Revised: The overall optimization problem at this step consists of the policy learning (driven by intrinsic reward), the touch reconstruction loss (Eq. 2), and the forward dynamics loss (Eq. 3).

Original: During the downstream adaptation step, the parameters of the policy network, the Q network andthe replay buffer are retained from the exploratory phase.
Revised: During the downstream adaptation step, the parameters of the policy network, the Q network and the replay buffer are retained from the exploratory phase.

Original: The objective of the down-stream task iscomputed as:
Revised: The objective of the down-stream task is computed as:

Original: [Equation]
Revised: [Equation]

Original: where r et in this phase is task-speciﬁc external sparse reward.
Revised: where r et in this phase is task-speciﬁc external sparse reward.

Original: In both steps, the objectives areoptimized with Adam [52].
Revised: In both steps, the objectives are optimized with Adam [52].

Original: Implementation Details : The encoder is a four-layered strided CNN followed by a fully connectednetwork.
Revised: Implementation Details : The encoder is a four-layered strided CNN followed by a fully connected network.

Original: We use LeakyReLU
Revised: We use LeakyReLU

Original: [53] as non-linear activation in all the layers.
Revised: [53] as non-linear activation in all the layers.

Original: The decoder network isa two-layered MLP that maps 256-dimensional visual features to touch vectors.
Revised: The decoder network is a two-layered MLP that maps 256-dimensional visual features to touch vectors.

Original: For a more detaileddescription of the networks (SAC policy network and ToC networks, including the forward model)and hyper-parameters, please refer to the supplementary material Section 7.3.
Revised: For a more detailed description of the networks (SAC policy network and HaC networks, including the forward model) and hyper-parameters, please refer to the supplementary material Section 7.3.

Original:  Experimental Setting
Revised:  Experimental Setting

Original: Our experiments focus on a tabletop robot manipulation from raw image observations and rawforce/torque sensory values, which we refer to as “touch vector”.
Revised: Our experiments focus on a tabletop robot manipulation from raw image observations and raw force/torque sensory values, which we refer to as “touch vector”.

Original: For our experiments, we use a7-DoF Franka Emika Panda arm with a two-ﬁnger parallel gripper.
Revised: For our experiments, we use a 7-DoF Franka Emika Panda arm with a two-ﬁnger parallel gripper.

Original: Each of the ﬁngers is equippedwith a simulated force/torque sensor that measures the joint reaction force applied to it.
Revised: Each of the ﬁngers is equipped with a simulated force/torque sensor that measures the joint reaction force applied to it.

Original: We utilize
Revised: We utilize

Original: PyBullet [54] to simulate the robot arm and haptic sensor 1 .
Revised: PyBullet [54] to simulate the robot arm and haptic sensor 1 .

Original: 4.1 MiniTouch Benchmark
Revised: 4.1 MiniTouch Benchmark

Original: Our proposed benchmark, MiniTouch, consists of four manipulation tasks:
Revised: Our proposed benchmark, MiniTouch, consists of four manipulation tasks:

Original: Playing, Pushing, Opening, Pick-up.
Revised: Playing, Pushing, Opening, Pick-up.

Original: Each of the tasks along with corresponding actions, observations, and rewards isdescribed in detail in the Supplementary Material Section 7.1 and further illustrated in Figure 2.
Revised: Each of the tasks along with corresponding actions, observations, and rewards is described in detail in the Supplementary Material Section 7.1 and further illustrated in Figure 2.

Original: MiniTouch is an active repository 

Revised: MiniTouch is an active repository

Original: and we expect to update the benchmark with new tasks anddatasets.
Revised: and we expect to update the benchmark with new tasks and datasets.

Original: The tasks are inspired by Yu et al.
Revised: The tasks are inspired by Yu et al.

Original: [55] but are not based on a proprietary simulator, featurean arm that we have access to for real-world experimentation (in follow-up work), and where thearm is equipped with a haptic sensor.
Revised: [55] but are not based on a proprietary simulator, feature an arm that we have access to for real-world experimentation (in follow-up work), and where the arm is equipped with a haptic sensor.

Original: 4.2 Baseline Comparisons and Metrics
Revised: 4.2 Baseline Comparisons and Metrics

Original: For the task evaluation, we study two versions of our model: (i) ToC-Pure considering the touch vector reconstruction intrinsic reward alone, and (ii) ToC, considering the full intrinsic reward (Eq. 4).
Revised: For the task evaluation, we study two versions of our model: (i) HaC-Pure considering the touch vector reconstruction intrinsic reward alone, and (ii) HaC, considering the full intrinsic reward (Eq. 4).

Original: We compare with several well-known intrinsic exploration baselines based on visual features:
Revised: We compare with several well-known intrinsic exploration baselines based on visual features:

Original: • SAC : The unmodiﬁed Soft Actor-Critic algorithm from Haarnoja et al.
Revised: • SAC : The unmodiﬁed Soft Actor-Critic algorithm from Haarnoja et al.

Original: ICM : SAC augmented with the state-of-the-art visual curiosity approach Intrinsic CuriosityModule (ICM)
Revised: ICM : SAC augmented with the state-of-the-art visual curiosity approach Intrinsic Curiosity Module (ICM)

Original: [12], which uses a visual prediction model to guide exploration.• Disagreement : It uses model disagreement as objective for exploration [56, 28].
Revised: [12], which uses a visual prediction model to guide exploration. • Disagreement : It uses model disagreement as objective for exploration [56, 28].

Original: It leverages variance in the prediction of an ensemble of latent dynamics models as the reward.
Revised: It leverages variance in the prediction of an ensemble of latent dynamics models as the reward.

Original: • RND : Random Network Distillation [57] utilizes a randomly initialized neural network to specify an intrinsic reward for visiting unexplored states in hard exploration problems.We built a Pytorch [
Revised: • RND : Random Network Distillation [57] utilizes a randomly initialized neural network to specify an intrinsic reward for visiting unexplored states in hard exploration problems.

Original: version of these baselines based
Revised: [Figure]

Original:  on their open source code details in sup- plementary material).
Revised: We built a Pytorch [58] version of these baselines based on their open source code (details in supplementary material).

Original: We use the following metrics to evaluate our method and the baseline models:
Revised: We use the following metrics to evaluate our method and the baseline models:

Original: Exploration success: measures the percentage of times that the agent attained the goal state in the exploratory phase, i e. with no external reward.
Revised: Exploration success: measures the percentage of times that the agent attained the goal state in the exploratory phase, i.e. with no external reward.

Original: Higher is better.
Revised: Higher is better.

Original: Success: denotes the percentage of times that the agent attained the goal state during the down- stream task phase.
Revised: Success: denotes the percentage of times that the agent attained the goal state during the downstream task phase.

Original: Episode steps: The number of steps required for each episode to succeed.
Revised: Episode steps: The number of steps required for each episode to succeed.

Original: This metric is an indicatorof sample efﬁciency.
Revised: This metric is an indicator of sample efﬁciency.

Original: The lesser the number of steps, the faster the agent’s ability to succeed.
Revised: The lesser the number of steps, the faster the agent’s ability to succeed.

Original: [Figure]

Original: Touch-interaction: Amount of interaction the agent’s ﬁngers have with the underlying object.
Revised: Touch-interaction: Amount of interaction the agent’s ﬁngers have with the underlying object.

Original: Wemeasure this by computing the variance of force/torque sensory signal across the whole episode.
Revised: We measure this by computing the variance of force/torque sensory signal across the whole episode.

Original: Object movement: The agent can resort to constantly engaging with the object unnecessarily tosatisfy the objective.
Revised: Object movement: The agent can resort to constantly engaging with the object unnecessarily to satisfy the objective.

Original: We, therefore, monitor the variance of door angle (for the Opening task)and the variance of object position (for the remaining tasks) over the course of training.
Revised: We, therefore, monitor the variance of door angle (for the Opening task) and the variance of object position (for the remaining tasks) over the course of training.

Original: A highermovement indicates diverse state space in addition to physical interaction.
Revised: A higher movement indicates diverse state space in addition to physical interaction.

Original:  Results and Discussion
Revised:  Results and Discussion

Original: ToC and baselines were trained on a Panda robot agent [54] for one million steps.
Revised: HaC and baselines were trained on a Panda robot agent [54] for one million steps.

Original: In the exploratoryphase of the training, we pre-train our method only with the curiosity-based intrinsic reward.
Revised: In the exploratory phase of the training, we pre-train our method only with the curiosity-based intrinsic reward.

Original: Wethen progress to the adaptation phase.
Revised: We then progress to the adaptation phase.

Original: Also, note that across all tasks, ToC-Pure is based purely oncross-modal prediction, while ToC includes visual forward prediction reward in addition.
Revised: Also, note that across all tasks, HaC-Pure is based purely on cross-modal prediction, while HaC includes visual forward prediction reward in addition.

Original: Figure 3a and 3b shows results on the basic task of playing with a single object.
Revised: Figure 3a and 3b shows results on the basic task of playing with a single object.

Original: Since singleobject interaction does not have explicit goal states to evaluate, we instead measure the agent’s [Figure] [Figure] [Table]
Revised: Since single object interaction does not have explicit goal states to evaluate, we instead measure the agent’s ability to constantly engage and play with the underlying object.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] ability to constantly engage and play with the underlying object. ToC-Pure displays four timesbetter interaction with the underlying object when compared to SAC (see Figure 3b).
Revised: HaC-Pure displays four times better interaction with the underlying object when compared to SAC (see Figure 3b).

Original: Note from theplots that there is a trade-off when using ToC and ToC-Pure between constant interaction (i.e. touchinteraction performance) and object movement dynamics.
Revised: Note from the plots that there is a trade-off when using HaC and HaC-Pure between constant interaction (i.e. touch interaction performance) and object movement dynamics.

Original: Collecting a variety of such interestingdata during the exploratory phase helps the agent in terms of sample efﬁciency while solving thedownstream tasks.
Revised: Collecting a variety of such interesting data during the exploratory phase helps the agent in terms of sample efﬁciency while solving the downstream tasks.

Original: Figure 8 shows similar comparison for Open-door task.
Revised: Figure 9 shows similar comparison for Open-door task.

Original: In the following section,we examine the role of the forward objective term on touch interaction and possible ways to encodeagent’s observation in our ablation studies.
Revised: In the following section, we examine the role of the forward objective term on touch interaction and possible ways to encode agent’s observation in our ablation studies.

Original: We compare ToC and ToC-Pure with SAC and state-of-the-art vision-based curiosity baselines onthe remaining downstream tasks in MiniTouch.
Revised: We compare HaC and HaC-Pure with SAC and state-of-the-art vision-based curiosity baselines on the remaining downstream tasks in MiniTouch.

Original: From Figure 4 it is evident that ToC and ToCPure perform better than SAC in all the tasks and better than the vision-based curiosity modelsin the majority of the tasks.
Revised: From Figure 4 it is evident that HaC and HaCPure perform better than SAC in all the tasks and better than the vision-based curiosity models in the majority of the tasks.

Original: Using SAC alone hinders the performance and is often unable tosolve any of the three tasks.
Revised: Using SAC alone hinders the performance and is often unable to solve any of the three tasks.

Original: This is not surprising since the model is not motivated enough tocollect diverse and useful data through interaction.
Revised: This is not surprising since the model is not motivated enough to collect diverse and useful data through interaction.

Original: We hypothesize that ToC-Pure without the visualprediction objective L fdm could potentially bias the solutions towards more physical interaction,which is not necessary for every task.
Revised: We hypothesize that HaC-Pure without the visual prediction objective L fdm could potentially bias the solutions towards more physical interaction, which is not necessary for every task.

Original: For instance, constant interaction in the pushing task couldbe seen as a hindrance as it does not let the object slide easily towards the target.
Revised: For instance, constant interaction in the pushing task could be seen as a hindrance as it does not let the object slide easily towards the target.

Original: Observe that ICM performs better than ToC-Pure in the pushing task, however, ToC dominates in performance byabout 15%.
Revised: Observe that ICM performs better than HaC-Pure in the pushing task, however, HaC dominates in performance by about 15%.

Original: Recall that the goal is not just to succeed but to help attain success in a sample efﬁcientmanner in fewer steps.
Revised: Recall that the goal is not just to succeed but to help attain success in a sample efﬁcient [Figure] manner in fewer steps.

Original: The results support our hypothesis that cross-modal curiosity enables an RLagent to succeed at an early stage in training and often without any external reward.
Revised: The results support our hypothesis that cross-modal curiosity enables an RL agent to succeed at an early stage in training and often without any external reward.

Original: Similarly, ourmodel outperforms on the opening task without external reward.
Revised: Similarly, our model outperforms on the opening task without external reward.

Original: However, ToC initially has lowersuccess compared to RND but surpasses RND towards the end.
Revised: However, HaC initially has lower success compared to RND but surpasses RND towards the end.

Original: Although ToC and ToC-Pure attainsimilar success in the pick-up task towards the end of the training, it is compelling to note that ToCPure attains faster convergence.
Revised: Although HaC and HaC-Pure attain similar success in the pick-up task towards the end of the training, it is compelling to note that HaCPure attains faster convergence.

Original: This is because the picking task requires constant touch interaction(where ToC-Pure has an advantage), as opposed to diverse object movement.
Revised: This is because the picking task requires constant touch interaction (where HaC-Pure has an advantage), as opposed to diverse object movement.

Original: 5.1 Ablations
Revised: 5.1 Ablations

Original: Forward objective and tuning Lambda Visual forward prediction L fdm plays an important rolewhen it is used in the right proportion.
Revised: Forward objective and tuning Lambda Visual forward prediction L fdm plays an important role when it is used in the right proportion.

Original: Our intrinsic reward is a weighted combination of crossmodal prediction and forward prediction as deﬁned in Eq.4.
Revised: Our intrinsic reward is a weighted combination of crossmodal prediction and forward prediction as deﬁned in Eq.4.

Original: Figure 5 illustrates the model behaviorwith different levels of emphasis on the forward loss term, with  uniformly sampled between 0 and [Figure]
Revised: Figure 5 illustrates the model behavior with different levels of emphasis on the forward loss term, with  uniformly sampled between 0 and 1.

Original: Higher weights indicate that the future prediction dominates force/torque prediction.
Revised: Higher weights indicate that the future prediction dominates force/torque prediction.

Original: This leadsto more object movement but lesser robot’s constant touch-interaction with the underlying objects.
Revised: This leads to more object movement but lesser robot’s constant touch-interaction with the underlying objects.

Revised: [Table]

Original: This is useful in tasks such as Pushing.
Revised: This is useful in tasks such as Pushing.

Original: A smaller  value leads to better inactive behavior which ishandy in tasks such as grasping.
Revised: A smaller  value leads to better inactive behavior which is handy in tasks such as grasping.

Original: In our experiments we choose an intermediate value ( e.g. 0.5) thatworks best for all of our tasks.
Revised: In our experiments we choose an intermediate value ( e.g. 0.5) that works best for all of our tasks

Original: Touch in future prediction.
Revised: Haptics in future prediction.

Original: The goal of this experiment is to strengthen the argument of crossmodal association.
Revised: The goal of this experiment is to strengthen the argument of crossmodal association.

Original: While conducting experiments, it is important to deduce information of onemodality from another modality in a related manner than simply adding another modality on top ofvisual information.
Revised: While conducting experiments, it is important to deduce information of one modality from another modality in a related manner than simply adding another modality on top of visual information.

Original: We created an additional baseline, ICM+ touch , where in addition to the visualprediction model we include the touch-based future prediction model.
Revised: We created an additional baseline, ICM+ haptics , where in addition to the visual prediction model we include the haptics-based future prediction model.

Original: We hypothesize ICM+ touchto perform better than ICM as it has additional information (touch).
Revised: We hypothesize ICM+ haptics to perform better than ICM as it has additional information (haptics).

Original: The touch-based future prediction model takes a touch vector as input and predicts the touch vector for the next time step.
Revised: The hapticsbased future prediction model takes a touch vector as input and predicts the touch vector for the next time step.

Original: Table 1compares ToC and ICM+ touch on MiniTouch tasks and we observe thatICM+ touch is better than
Revised: Table 1 compares HaC and ICM+ haptics on MiniTouch tasks and we observe that

Original:  ICM but compares below ToC. Table 4 in the supplementary comparesRND based baseline. Robustness to diverse shapes. It is desirable for an agent to be able to handle diverse shapes in orderto be robust across arbitrary manipulation settings. We study this using an environment in whichan object is sampled from a thousand procedurally generated objects.
Revised: ICM+ haptics is better than ICM but compares below HaC. Table 4 in the supplementary compares

Original: The objects are dissimilarwith respect to shape and mass but are sampled from the same generative distribution. Out of 1000different objects, 800 of them are used in the training phase, and we evaluate the agent’s effectivenesson the remaining 200 unseen object shapes. Figure 6b shows touch interaction and object movementevaluations for a single object exploration task. The results validate that our model generalizes tounseen object conﬁgurations (see Section 7.5 in the supplementary material for more results).
Revised: RND based baseline.

Original: Latent features for forward dynamics.
Revised: Latent features for forward dynamics.

Original: Choosing ideal embedding space for decoding the touchvector (Figure 1a) and for predicting future state (Figure. 1b) is important.
Revised: Choosing ideal embedding space for decoding the touch vector (Figure 1a) and for predicting future state (Figure. 1b) is important.

Original: Existing approaches relyon a pretext inverse dynamics (ID) task of predicting the agent’s action given its current and nextstates [9].
Revised: Existing approaches rely on a pretext inverse dynamics (ID) task of predicting the agent’s action given its current and next states [9].

Original: Another simple yet strong method is to use features from a random but ﬁxed initializationof the encoder [12].
Revised: Another simple yet strong method is to use features from a random but ﬁxed initialization of the encoder [12].

Original: In our work, we learn the features by leveraging the self-supervised pretext taskof predicting one modality from the other.
Revised: In our work, we learn the features by leveraging the self-supervised pretext task of predicting one modality from the other.

Original: Figure 6a compares (1) encoder, i.e. learned throughcross-modal prediction(ToC) (2) random feature encoder(ToC-RF) (3) encoder learned through IDtask (ToC-IDF).
Revised: Figure 6 compares (1) encoder, i.e. learned through crossmodal prediction(HaC) (2) random feature encoder(HaC-RF) (3) encoder learned through ID task (HaC-IDF).

Original: In each case, the decoder network is optimized through touch prediction.
Revised: In each case, the decoder network is optimized through touch prediction.

Original: We observe that the random features variant is stable and effective on both ToC and ICM models.
Revised: We observe that the random features variant is stable and effective on both HaC and ICM models.

Original:  Conclusion
Revised:  Conclusion

Original: [Figure] Building self-seeking RL agents that can leverage cross-modal associations could be a key researchdirection in developmental robotics.
Revised: Building self-seeking RL agents that can leverage cross-modal associations could be a key research direction in developmental robotics.

Original: Sense of touch is ubiquitous in human routine.
Revised: Sense of touch is ubiquitous in human routine.

Original: Borrowing insights from how infants learn to explore out of curiosity by touching what they visually perceive, weformulated Touch-based Curiosity (ToC) that is aimed at encouraging exploration via haptic interaction with the environment.
Revised: Borrowing insights from how infants learn to explore out of curiosity by touching what they visually perceive, we formulated Haptics-based Curiosity (HaC) that is aimed at encouraging exploration via haptic interaction with the environment.

Original: We demonstrated in this work, that by involving additional modalities,the performance of curiosity-based systems on downstream tasks can be increased.
Revised: We demonstrated in this work, that by involving additional modalities, the performance of curiosity-based systems on downstream tasks can be increased.

Original: We observedincreased interaction with target objects, and presented evidence that ToC learns to solve the MiniTouch benchmark tasks in an efﬁcient manner while vanilla RL algorithms and vision-based curiosity formulations struggled.
Revised: We observed increased interaction with target objects, and presented evidence that HaC learns to solve the MiniTouch benchmark tasks in an efﬁcient manner while vanilla RL algorithms and vision-based curiosity formulations struggled.

Original: Force/torque sensing is widespread in the lab and industrial robots butwhile there are plenty of robotic benchmarks, we believe that tactile feedback is an under-exploredmodality and by releasing our benchmark, we hope to enable future research in this exciting area. 
Revised: Force/torque sensing is widespread in the lab and industrial robots but while there are plenty of robotic benchmarks, we believe that tactile feedback is an under-explored modality and by releasing our benchmark, we hope to enable future research in this exciting area. Acknowledgements

Revised: We would like to sincerely thank Glen Berseth, Deepak Pathak, Edward Smith and Krishna Murthy for valuable feedback and insightful comments on the paper.