Original: Touch-based Curiosity for Sparse-Reward Tasks
Revised: Touch-based Curiosity for Sparse-Reward Tasks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Abstract: Robots in many real-world settings have access to force/torque sensorsin their gripper and tactile sensing is often necessary in tasks that involve contactrich motion.
Revised: Abstract: Robots in many real-world settings have access to force/torque sensorsin their gripper and tactile sensing is often necessary in tasks that involve contactrich motion.

Original: In this work, we leverage surprise from mismatches in touch feedback to guide exploration in hard sparse-reward reinforcement learning tasks.
Revised: In this work, we leverage surprise from mismatches in touch feedback to guide exploration in hard sparse-reward reinforcement learning tasks.

Original: Ourapproach, Touch-based Curiosity (ToC), learns what visible objects interactionsare supposed to “feel” like.
Revised: Ourapproach, Touch-based Curiosity (ToC), learns what visible objects interactionsare supposed to “feel” like.

Original: We encourage exploration by rewarding interactionswhere the expectation and the experience do not match.
Revised: We encourage exploration by rewarding interactionswhere the expectation and the experience do not match.

Original: We test our approach ona range of touch-intensive robot arm tasks (e.g. pushing objects, opening doors),which we also release as part of this work.
Revised: We test our approach ona range of touch-intensive robot arm tasks (e.g. pushing objects, opening doors),which we also release as part of this work.

Original: Across multiple experiments in asimulated setting, we demonstrate that our method is able to learn these difﬁculttasks through sparse reward and curiosity alone.
Revised: Across multiple experiments in asimulated setting, we demonstrate that our method is able to learn these difﬁculttasks through sparse reward and curiosity alone.

Original: We compare our cross-modalapproach to single-modality (touch- or vision-only) approaches as well as othercuriosity-based methods and ﬁnd that our method performs better and is moresample-efﬁcient.
Revised: We compare our cross-modalapproach to single-modality (touch- or vision-only) approaches as well as othercuriosity-based methods and ﬁnd that our method performs better and is moresample-efﬁcient.

Original:  Introduction
Revised:  Introduction

Original: Most successes in reinforcement learning (RL) come from games [1, 2] or scenarios where thereward is strongly shaped [3, 4].
Revised: Most successes in reinforcement learning (RL) come from games [1, 2] or scenarios where thereward is strongly shaped [3, 4].

Original: In the former the environment is often fully observable and thereward is dense and well-deﬁned.
Revised: In the former the environment is often fully observable and thereward is dense and well-deﬁned.

Original: In the latter, a large amount of work is required to design usefulreward functions.
Revised: In the latter, a large amount of work is required to design usefulreward functions.

Original: While it may be possible to hand-craft dense reward signals for many real-worldtasks, we believe that it is a worthwhile endeavor to investigate learning methods that do not requiredense rewards.
Revised: While it may be possible to hand-craft dense reward signals for many real-worldtasks, we believe that it is a worthwhile endeavor to investigate learning methods that do not requiredense rewards.

Original: Closely related to the sparse rewards problem is the issue of exploration.
Revised: Closely related to the sparse rewards problem is the issue of exploration.

Original: One reason that traditional
Revised: One reason that traditional

Original: RL agents struggle with sparse-reward problems is a lack of exploration.
Revised: RL agents struggle with sparse-reward problems is a lack of exploration.

Original: An agent may not obtainuseful rewards without an intuitive exploration strategy when rewards are sparse.
Revised: An agent may not obtainuseful rewards without an intuitive exploration strategy when rewards are sparse.

Original: Exploration basedon intrinsic curiosity comes naturally to many animals and infants (who start crawling and exploringthe environment at around 9 months [5] and oftentimes even before they can crawl by using theirhands and mouth to touch and probe objects).
Revised: Exploration basedon intrinsic curiosity comes naturally to many animals and infants (who start crawling and exploringthe environment at around 9 months [5] and oftentimes even before they can crawl by using theirhands and mouth to touch and probe objects).

Original: Touch is a local experience and encodes accurategeometrical information while handling objects.
Revised: Touch is a local experience and encodes accurategeometrical information while handling objects.

Original: Developmental research in infants has suggestedthat tactile and visual sensory modalities play a central role in systematic learning of tasks related toobject understanding, interaction and manipulation [6, 7].
Revised: Experimental studies in infants has suggested thattactile and visual sensory modalities play a central role in systematic learning of tasks related toobject understanding, interaction and manipulation [6, 7].

Original: Ideally, we would like our RL agents to explore the environment in an analogous self-guided fashionto learn the dynamics and object properties, and use this knowledge to solve downstream tasks.
Revised: Ideally, we would like our RL agents to explore the environment in an analogous self-guided fashionto learn the dynamics and object properties, and use this knowledge to solve downstream tasks.

Original: Justas how humans utilize different sensory modalities to explore and understand the world around them,exploration in robots should be more embodied and related to a combination of vision and touch andpotentially other sensor modalities.
Revised: Justas how humans utilize different sensory modalities to explore and understand the world around them,exploration in robots should be more embodied and related to a combination of vision and touch andpotentially other sensor modalities.

Original: We believe that building autonomous agents that are self-drivenand seek to explore via multi-modal interaction are crucial to address key issues in developmentalrobotics.
Revised: We believe that building autonomous agents that are self-drivenand seek to explore via multi-modal interaction are crucial to address key issues in developmentalrobotics.

Original: Recent works in RL have focused on a curiosity-driven exploration through prediction-based surprise [8, 9, 10].
Revised: Recent works in RL have focused on a curiosity-driven exploration through prediction-based surprise [8, 9, 10].

Original: In this formulation, a forward dynamics model predicts the future, and if its prediction is incorrect when compared to the real future, the agent is surprised and is thus rewarded.
Revised: In this formulation, a forward dynamics model predicts the future, and if its prediction is incorrect when compared to the real future, the agent is surprised and is thus rewarded.

Original: This encourages the agent to look for novel states while improving its visual forward model in return.
Revised: This encourages the agent to look for novel states while improving its visual forward model in return.

Original: However, this formulation can be practically challenging to optimize since there are manystates that are visually dissimilar but practically irrelevant (e.g. for a pushing task, moving a roboticend-effector without touching the object creates visual novelty but contributes little to task-relatedknowledge).
Revised: However, this formulation can be practically challenging to optimize since there are manystates that are visually dissimilar but practically irrelevant (e.g. for a pushing task, moving a roboticend-effector without touching the object creates visual novelty but contributes little to task-relatedknowledge).

Original: One way to constrain this search space over curious behaviors is by involving anothermodality like touch
Revised: One way to constrain this search space over curious behaviors is by involving anothermodality like touch

Original: In this work, we demonstrate that a self-guided cross-modal exploration policy can help solve sparsereward downstream tasks that existing methods without this curiosity struggle to solve.
Revised: In this work, we demonstrate that a self-guided cross-modal exploration policy can help solve sparsereward downstream tasks that existing methods without this curiosity struggle to solve.

Original: Our methoduses cross-modal consistency (mismatch between visual and touch signal) to guide this exploration.
Revised: Our methoduses cross-modal consistency (mismatch between visual and touch signal) to guide this exploration.

Original: To use self-play knowledge in downstream tasks, we relabel past experiences, providing a densereward signal that allows modern off-policy RL methods to solve the tasks.
Revised: To use self-play knowledge in downstream tasks, we relabel past experiences, providing a densereward signal that allows modern off-policy RL methods to solve the tasks.

Original: While there are manyexisting methods that use artiﬁcial curiosity/intrinsic motivation, the majority of these methods either rely on strong domain knowledge (e.g. labels of state dimensions in Forestier et al.
Revised: While there are manyexisting methods that use artiﬁcial curiosity/intrinsic motivation, the majority of these methods either rely on strong domain knowledge (e.g. labels of state dimensions in Forestier et al.

Original: [4], a goalpicking strategy in Andrychowicz et al.
Revised: [4], a goalpicking strategy in Andrychowicz et al.

Original: [11]) or are prone to get stuck in local optima when a singlemeaningless stimulus creates enough surprise to capture the attention of the agent (e.g. noisy-TVexperiment from [12]).
Revised: [11]) or are prone to get stuck in local optima when a singlemeaningless stimulus creates enough surprise to capture the attention of the agent (e.g. noisy-TVexperiment from [12]).

Original: Other approaches depend on unrealistic assumptions and goal conditioning [11].
Revised: Other approaches depend on unrealistic assumptions and goal conditioning [11].

Original: Our method presents a novel approach in the family of prediction-based models [8, 13, 9]and yields better performance on a wide range of robotic manipulation tasks than purely visionbased and touch-based approaches [8, 9].
Revised: Our method presents a novel approach in the family of prediction-based models [8, 13, 9]and yields better performance on a wide range of robotic manipulation tasks than purely visionbased and touch-based approaches [8, 9].

Original: The tasks are chosen with careful consideration—theycomprise of preliminary robotic manipulations such as grasping, pushing, and pulling.
Revised: The tasks are chosen with careful consideration—theycomprise of preliminary robotic manipulations such as grasping, pushing, and pulling.

Original: In this work,we present the following contributions:
Revised: In this work,we present the following contributions:

Original: • A new curiosity method to help solve sparse-reward tasks that use cross-modal consistency(predicting one modality from another) to guide exploration.
Revised: • A new curiosity method to help solve sparse-reward tasks that use cross-modal consistency(predicting one modality from another) to guide exploration.

Original: We implement it in this work
Revised: We implement it in this work

Original: as vision and touch modalities, but the formulation of our method does not require any
Revised: as vision and touch modalities, but the formulation of our method does not require any

Original: knowledge about the underlying modalities and can thus be applied to other settings.
Revised: knowledge about the underlying modalities and can thus be applied to other settings.

Original: • We create and maintain a manipulation benchmark of simulated tasks, MiniTouch , inspiredby Chen et al.
Revised: • We create and maintain a manipulation benchmark of simulated tasks, MiniTouch , inspiredby Chen et al.

Original: [14], Andrychowicz et al.
Revised: [14], Andrychowicz et al.

Original: [11], where the robotic arm is equipped with a
Revised: [11], where the robotic arm is equipped with a

Original: force/torque sensor.
Revised: force/torque sensor.

Original: This allows evaluation of models’ performance on different manipula-
Revised: This allows evaluation of models’ performance on different manipula-

Original: tion tasks that can leverage cross-modal learning.
Revised: tion tasks that can leverage cross-modal learning.

Original: • We validate the performance of our method on MiniTouch environment comprising
Revised: • We validate the performance of our method on MiniTouch environment comprising

Original: of fourdownstream tasks
Revised: of fourdownstream tasks

Original: We compare purely vision-based curiosity approaches and standard offpolicy RL algorithms.
Revised: We compare purely vision-based curiosity approaches and standard offpolicy RL algorithms.

Original: Our method improves both performance and sample efﬁciency.
Revised: Our method improves both performance and sample efﬁciency.

Original:  Related Work
Revised:  Related Work

Original: Intrinsic Motivation Intrinsic motivation is an inherent spontaneous tendency to be curious or toseek something novel in order to further enhance one’s skill and knowledge [15, 16, 17].
Revised: Intrinsic Motivation Intrinsic motivation is an inherent spontaneous tendency to be curious or toseek something novel in order to further enhance one’s skill and knowledge [15, 16, 17].

Original: This principle is shown to work well even in the absence of a well-deﬁned goal or objective.
Revised: This principle is shown to work well even in the absence of a well-deﬁned goal or objective.

Original: In reinforcementlearning, intrinsic motivation has been a well-researched topic [18, 19, 20, 21, 22, 23].
Revised: In reinforcementlearning, intrinsic motivation has been a well-researched topic [18, 19, 20, 21, 22, 23].

Original: An intuitiveway to perform intrinsic motivation is through the use of “novelty discovery”.
Revised: An intuitiveway to perform intrinsic motivation is through the use of “novelty discovery”.

Original: For example, incentivize the RL agent to visit unusual states or states with substantial information gain [24].
Revised: For example, incentivize the RL agent to visit unusual states or states with substantial information gain [24].

Original: In its simplest form, this can be achieved with up-weighting mechanisms such as state visitation counts [25].
Revised: In its simplest form, this can be achieved with up-weighting mechanisms such as state visitation counts [25].

Original: Count-based methods have also been extended to high-dimensional state spaces [26, 12, 27].
Revised: Count-based methods have also been extended to high-dimensional state spaces [26, 12, 27].

Original: Alternative forms of intrinsic motivation include disagreement [28], empowerment [29, 30].
Revised: Alternative forms of intrinsic motivation include disagreement [28], empowerment [29, 30].

Original: Exploratory intrinsic motivation can also be achieved through “curiosity” [18, 31].
Revised: Exploratory intrinsic motivation can also be achieved through “curiosity” [18, 31].

Original: In this setting,an agent is encouraged to visit states with high predictive errors [10, 8, 9] by training a forwarddynamics model that predicts the future state given the current state and action.
Revised: In this setting,an agent is encouraged to visit states with high predictive errors [10, 8, 9] by training a forwarddynamics model that predicts the future state given the current state and action.

Original: Instead of makingpredictions in the raw visual space, Pathak et al.
Revised: Instead of makingpredictions in the raw visual space, Pathak et al.

Original: [9] mapped images to a feature space where relevantinformation is represented via an inverse dynamics model.
Revised: [9] mapped images to a feature space where relevantinformation is represented via an inverse dynamics model.

Original: Burda et al.
Revised: Burda et al.

Original: [8] demonstrate that randomfeatures are sufﬁcient for many popular RL game benchmarks.
Revised: [8] demonstrate that randomfeatures are sufﬁcient for many popular RL game benchmarks.

Original: This approach may work well withtasks that require navigation to ﬁnd a reward because each unseen position of the agent in the worldleads to high intrinsic reward when unseen.
Revised: This approach may work well withtasks that require navigation to ﬁnd a reward because each unseen position of the agent in the worldleads to high intrinsic reward when unseen.

Original: However, in the case of manipulative tasks, we are lessinterested in the robot visiting all the possible states and more interested in states where the robotinteracts with other objects.
Revised: However, in the case of manipulative tasks, we are lessinterested in the robot visiting all the possible states and more interested in states where the robotinteracts with other objects.

Original: In this work, we leverage multimodal inputs that encourage the agentto ﬁnd novel combinations of visual and force/torque modalities.
Revised: In this work, we leverage multimodal inputs that encourage the agentto ﬁnd novel combinations of visual and force/torque modalities.

Original: Self-Supervised Learning via Cross-modality Exploiting multimodality to learn unsupervisedrepresentations dates back to at least 1993 [32].
Revised: Self-Supervised Learning via Cross-modality Exploiting multimodality to learn unsupervisedrepresentations dates back to at least 1993 [32].

Original: Multimodal signals are naturally suitable for selfsupervised learning, as information from one modality can be used to supervise learning for anothermodality.
Revised: Multimodal signals are naturally suitable for selfsupervised learning, as information from one modality can be used to supervise learning for anothermodality [33, 34].

Original: Different modalities typically carry different information, e.g., visual and touch sensorymodalities emerge concurrently and often in an interrelated manner during contact-rich manipulation tasks [33].
Revised: Different modalities typically carry different information, e.g., visual and touchsensory modalities emerge concurrently and often in an interrelated manner during contact-richmanipulation tasks [35].

Original: Speciﬁcally, force/torque motor signal has always been a major component in theliterature of perception and control [34, 35, 36].
Revised: Speciﬁcally, force/torque motor signal has always been a major componentin the literature of perception and control [36, 37, 38].

Original: [Figure]
Revised: [Figure]

Original: The most common ways to leverage multimodal signals to learn representations are through vision and language[37, 38], or through visual and audio [39, 40].
Revised: The most common ways to leverage multimodal signals to learn representations are through visionand language [39, 40].

Original: Gao et al.
Revised: Gao et al.

Original: [41], Li et al.
Revised: [41], Li et al.

Original: [42]demonstrated a uniﬁed approach to learning representations for prediction tasks using visual andtouch data.
Revised: [42] demonstrated a uniﬁed approach to learningrepresentations for prediction tasks using visual and touch data.

Revised: Chen et al.

Revised: [43] learn world-modelsfrom multimodal data via a shared latent space.

Original: In robotics and interactive settings, the use of additional modalities such as tactile sensing [43, 44, 45] is increasingly popular for grasping and manipulation tasks.
Revised: In robotics and interactive settings, the use ofmodalities such as tactile sensing [44, 45, 46] is increasingly popular for grasping, manipulation andother externally-speciﬁed tasks [47, 48].

Original: Lee et al.
Revised: Lee et al.

Original: [46] showedthe effectiveness of self-supervised training of tactile and visual representations by demonstratingits use on a peg insertion task.
Revised: [49] showed the effectiveness of self-supervisedtraining of tactile and visual representations by demonstrating its use on a peg insertion task.

Original: While the mentioned approaches have used multiple sensory modalities for learning better representations, in this work we demonstrate its utility for allowing agents to explore.
Revised: While the mentioned approaches have used multiple sensory modalities for learning better representations, in this work we demonstrate its utility for allowing agents to explore.

Original: Similar to ours, Deanet al.
Revised: Similar to ours, Deanet al.

Original: [47] use multimodal sensory association (i.e. audio and visual) to compute the intrinsic reward.
Revised: [50] use multimodal sensory association (i.e. audio and visual) to compute the intrinsic reward.

Original: Their curiosity-based formulation allows an agent to efﬁciently explore the environment insettings where audio and visual signals are governed by the same physical processes.
Revised: Their curiosity-based formulation allows an agent to efﬁciently explore the environment insettings where audio and visual signals are governed by the same physical processes.

Original: In addition tothe different nature of sensory signals, they use a discriminator that determines whether an observedmultimodal pair is novel.
Revised: In addition tothe different nature of sensory signals, they use a discriminator that determines whether an observedmultimodal pair is novel.

Original: This might not work in our case as touch is a more sparse signal and usinga discriminator could lead to ambiguous outcomes.
Revised: This might not work in our case as touch is a more sparse signal and usinga discriminator could lead to ambiguous outcomes.

Original:  Proposed Approach
Revised:  Proposed Approach

Original: The goal of our method, Touch-Based Curiosity (ToC), is to encourage the agent to interact withobjects.
Revised: The goal of our method, Touch-Based Curiosity (ToC), is to encourage the agent to interact withobjects.

Original: ToC provides a reward signal for a RL agent to explore the state space of a task thatinvolves interacting with objects.
Revised: ToC provides a reward signal for an RL agent to explore the state space of a task thatinvolves interacting with objects.

Original: The exploration phase is independent of the downstream task, i.e.,relying solely on visual and force/torque signals, without a reward signal from the downstream task.
Revised: The exploration phase is independent of the downstream task, i.e.,relying solely on visual and force/torque signals, without a reward signal from the downstream task.

Original: Similar to how people spend more time exploring stimuli that are more incongruous [7], ToC guidesthe agent to focus its experience on different novel cross-modal associations.
Revised: Similar to how people spend more time exploring stimuli that are more incongruous [7], ToC guidesthe agent to focus its experience on different novel cross-modal associations.

Original: We augment thisintrinsic objective with a future visual state prediction objective similar to the one in Pathak et al.[9] to avoid getting stuck in undesired inactive conﬁgurations.
Revised: We augment thisintrinsic objective with a future visual state prediction objective similar to the one in Pathak et al.[9] to avoid getting stuck in undesired inactive conﬁgurations.

Original: Note that we sometimes refer to thefuture state prediction objective in the text as forward dynamics objective.
Revised: Note that we sometimes refer to thefuture state prediction objective in the text as forward dynamics objective.

Original: In this work, we focuson the cross-modality between vision and touch, but the same idea could be applied to other pairs ofsensory domains, such as vision and sound, or touch and acoustics.
Revised: In this work, we focuson the cross-modality between vision and touch, but the same idea could be applied to other pairs ofsensory domains, such as vision and sound, or touch and acoustics.

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: The learning problem is formalized as a Markov decision process (MDP) deﬁned by a tuple{S , A , T , R , ⇢ ,  } of states, actions, transition probability, reward, initial state distribution, and 130 discount factor.
Revised: The learning problem is formalized as a Markov decision process (MDP) deﬁned by a tuple{S , A , T , R , ⇢ ,  } of states, actions, transition probability, reward, initial state distribution, and 130 discount factor.

Original: The goal is to ﬁnd the optimal policy that maximizes the discounted sum of rewards, ⇡ ⇤ =
Revised: The goal is to ﬁnd the optimal policy that maximizes the discounted sum of rewards, ⇡ ⇤ =

Original: E ⇡ [
Revised: E ⇡ [

Original: P 1 t  t r ( s t , a t )] .
Revised: P 1 t  t r ( s t , a t )] .

Original: In our case, each state s t in the trajectory is composed of bothvisual and corresponding touch features as detailed in the following section.
Revised: In our case, each state s t in the trajectory is composed of bothvisual and corresponding touch features as detailed in the following section.

Original: We use soft actorcritic policy gradients (SAC) [48] to train our policies, but in principle, our proposed approach isalgorithm-agnostic.
Revised: We use soft actorcritic policy gradients (SAC) [51] to train our policies, but in principle, our proposed approach isalgorithm-agnostic.

Original: The policy ⇡ is evaluated with an estimation of the soft Q-value:
Revised: The policy ⇡ is evaluated with an estimation of the soft Q-value:

Original: [Equation]
Revised: [Equation]

Original: (1) where V ( s t ) = E a t ⇠ ⇡ [ Q ( s t , a t )  log ⇡ ( a t| s t )] is the soft value function.
Revised: where V ( s t ) = E a t ⇠ ⇡ [ Q ( s t , a t )  log ⇡ ( a t

Original: 3.2 Touch-based Prediction
Revised: 3.2 Touch-based Prediction

Original: Our core prediction framework consists of two modules: (i) a touch control module, which learnsto predict expected touch signal from the visual input, and (ii) a forward dynamics model, whichpredicts the next latent state from the current latent state and the current action (see Figure 1).
Revised: Our core prediction framework consists of two modules: (i) a touch control module, which learnsto predict expected touch signal from the visual input, and (ii) a forward dynamics model, whichpredicts the next latent state from the current latent state and the current action (see Figure 1).

Original: Letthe state of the environment s t =
Revised: Letthe state of the environment s t =

Original: ( z t , h t ) at time t be composed of a visual feature z t (the encodedvisual input) and a touch signal h t .
Revised: ( z t , h t ) at time t be composed of a visual feature z t (the encodedvisual input) and a touch signal h t .

Original: The touch prediction model consists of a convolutional encoderz t = enc ( x t ) that transforms the image x t into a latent representation z t and a fully-connecteddecoder ˆ h t
Revised: The touch prediction model consists of a convolutional encoderz t = enc ( x t ) that transforms the image x t into a latent representation z t and a fully-connecteddecoder ˆ h t

Original: = dec ( z t ) that transforms the latent into a predicted haptic signal ˆ h t .
Revised: = dec ( z t ) that transforms the latent into a predicted haptic signal ˆ h t .

Original: The encoderdecoder is trained with a L2 reconstruction loss, i.e. for every image x t and force/torque sensor h t :
Revised: The encoderdecoder is trained with an L2 reconstruction loss, i.e. for every image x t and force/torque sensor h t :

Original: [Equation]
Revised: [Equation]

Original: A high prediction error on a given image indicates that the agent has had few interactions like this.
Revised: A high prediction error on a given image indicates that the agent has had few interactions like this.

Original: Therefore, to harness this “surprise” to guide exploration, we deﬁne the intrinsic reward at time tduring exploration to be proportional to this reconstruction loss.
Revised: Therefore, to harness this “surprise” to guide exploration, we deﬁne the intrinsic reward at time tduring exploration to be proportional to this reconstruction loss.

Original: This essentially allows the agentpolicy to visit under-explored conﬁgurations of the state space by encouraging interactions wherethe system does not know what the target object “feels” like.
Revised: This essentially allows the agentpolicy to visit under-explored conﬁgurations of the state space by encouraging interactions wherethe system does not know what the target object “feels” like.

Original: In addition to efﬁcient exploration,being aware of such incongruity via touch prediction aids learning local regularities in the visualinput.
Revised: In addition to efﬁcient exploration,being aware of such incongruity via touch prediction aids learning local regularities in the visualinput.

Original: This in turn could assist better generalization to unseen states.
Revised: This in turn could assist better generalization to unseen states.

Original: An overall pipeline of theframework is shown in Figure 1a.
Revised: An overall pipeline of theframework is shown in Figure 1a.

Original: 3.3 Regularization Through Forward Dynamics Model
Revised: 3.3 Regularization Through Forward Dynamics Model

Original: We found empirically (and we demonstrate in the experiments section below) that the surprise stemming from haptic novelty was not enough to cause object-centric interaction.
Revised: We found empirically (and we demonstrate in the experiments section below) that the surprise stemming from haptic novelty was not enough to cause object-centric interaction.

Original: We postulate that byincorporating visual surprise (i.e. the mismatch between predicted forward dynamics and observeddynamics)
Revised: We postulate that byincorporating visual surprise (i.e. the mismatch between predicted forward dynamics and observeddynamics)

Original: [9], we can create an agent that seeks out visual novelty as well as haptic one and thusleads to better state space coverage.
Revised: [9], we can create an agent that seeks out visual novelty as well as haptic one and thusleads to better state space coverage.

Original: To this end, our model is augmented with a forward dynamicsmodel (see Figure-1b)
Revised: To this end, our model is augmented with a forward dynamicsmodel (see Figure-1b)

Original: ˆ z t +1 = fdm ( z t , a t ) that learns to map the latent state z t (obtained from thevisual encoder enc ) and action a t at time t to the predicted latent state ˆ z t +1 at the next timestep.
Revised: ˆ z t +1 = fdm ( z t , a t ) that learns to map the latent state z t (obtained from thevisual encoder enc ) and action a t at time t to the predicted latent state ˆ z t +1 at the next timestep.

Original: This model is also trained with L2 loss:
Revised: This model is also trained with L2 loss:

Original: [Equation]
Revised: [Equation]

Original: The intrinsic reward is deﬁned as the convex combination of the cross-modal prediction loss (Eq.
Revised: The intrinsic reward is deﬁned as the convex combination of the cross-modal prediction loss (Eq.

Original: 2)and the forward dynamics model loss (Eq. 3):
Revised: 2)and the forward dynamics model loss (Eq. 3):

Original: [Equation]
Revised: [Equation]

Original: where  2 [0 , 1] is a balancing factor.
Revised: where  2 [0 , 1] is a balancing factor.

Original: The effect of the factor  on overall performance is studied inthe ablation experiments described in Section 5.1.
Revised: The effect of the factor  on overall performance is studied inthe ablation experiments described in Section 5.1.

Original: 3.4 Training
Revised: 3.4 Training

Original: Learning is divided into two stages: (i) an exploratory step, where the agent performs free exploration following ToC, and (ii) an adaptation step, where the agent is tasked to solve a downstreamproblem, given a sparse reward.
Revised: Learning is divided into two stages: (i) an exploratory step, where the agent performs free exploration following ToC, and (ii) an adaptation step, where the agent is tasked to solve a downstreamproblem, given a sparse reward.

Original: During the exploratory step, each trajectory consists of pairs of image and force/torque features, ( z 1 ,h 1 ) , ( z 2 , h 2 ) ,..., ( z n , h n ) .
Revised: During the exploratory step, each trajectory consists of pairs of image and force/torque features, ( z 1 ,h 1 ) , ( z 2 , h 2 ) ,..., ( z n , h n ) .

Original: These trajectories are used for two purposes: (i) updating the parametersof the prediction model to help shape the representations and (ii) updating the exploration policybased on the intrinsic reward r t .
Revised: These trajectories are used for two purposes: (i) updating the parametersof the prediction model to help shape the representations and (ii) updating the exploration policybased on the intrinsic reward r t .

Original: Note that we use the touch features only to craft the intrinsicrewards and the input to RL agent consists of visual features alone.
Revised: Note that we use the touch features only to craft the intrinsic rewardsand the input to RL agent consists of visual features alone to be comparable to the baselines.

Original: For vision-based curiositymodels, Burda et al.
Revised: Forvision-based curiosity models, Burda et al.

Original: [8] observed that encoding visual features via a random network constitutea simple and effective strategy compared to learned features.
Revised: [8] observed that encoding visual features via a randomnetwork constitute a simple and effective strategy compared to learned features.

Original: In Section 5.1, we investigate theperformance of our model in both scenarios, i.e., when the features are learned vs random.
Revised: In Section 5.1,we investigate the performance of our model in both scenarios, i.e., when the features are learnedvs random.

Original: Theoverall optimization problem at this step consists of the policy learning (driven by intrinsic reward),the touch reconstruction loss (Eq. 2), and the forward dynamics loss (Eq. 3).
Revised: The overall optimization problem at this step consists of the policy learning (drivenby intrinsic reward), the touch reconstruction loss (Eq. 2), and the forward dynamics loss (Eq. 3).

Original: During the downstreamadaptation step, the parameters of the policy network, the Q network and the replay buffer areretained from the exploratory phase.
Revised: During the downstream adaptation step, the parameters of the policy network, the Q network andthe replay buffer are retained from the exploratory phase.

Original: The objective of the down-stream task is computed as:
Revised: The objective of the down-stream task iscomputed as:

Original: [Equation]
Revised: [Equation]

Original: where r et in this phase is task-speciﬁc external sparse reward.
Revised: where r et in this phase is task-speciﬁc external sparse reward.

Original: In both steps, the objectives areoptimized with Adam [49].
Revised: In both steps, the objectives areoptimized with Adam [52].

Original: Implementation Details : The encoder is a four-layered strided CNN followed by a fully connectednetwork.
Revised: Implementation Details : The encoder is a four-layered strided CNN followed by a fully connectednetwork.

Original: We use LeakyReLU
Revised: We use LeakyReLU

Original: [50] as non-linear activation in all the layers.
Revised: [53] as non-linear activation in all the layers.

Original: The decoder network isa two-layerd MLP that maps 256-dimensional visual features to touch vectors.
Revised: The decoder network isa two-layered MLP that maps 256-dimensional visual features to touch vectors.

Original: For a more detaileddescription of the networks (SAC policy network and ToC networks, including the forward model)and hyper-parameters, please refer to the supplementary material Section 7.2.
Revised: For a more detaileddescription of the networks (SAC policy network and ToC networks, including the forward model)and hyper-parameters, please refer to the supplementary material Section 7.3.

Original:  Experimental Setting
Revised:  Experimental Setting

Original: Our experiments focus on a tabletop robot manipulation from raw image observations and rawforce/torque sensory values, which we refer to as “touch vector”.
Revised: Our experiments focus on a tabletop robot manipulation from raw image observations and rawforce/torque sensory values, which we refer to as “touch vector”.

Original: For our experiments, we use a7-DoF Franka Emika Panda arm with a two-ﬁnger parallel gripper.
Revised: For our experiments, we use a7-DoF Franka Emika Panda arm with a two-ﬁnger parallel gripper.

Original: Each of the ﬁngers is equippedwith a simulated force/torque sensor that measures the joint reaction force applied to it.
Revised: Each of the ﬁngers is equippedwith a simulated force/torque sensor that measures the joint reaction force applied to it.

Original: We utilize
Revised: We utilize

Original: PyBullet [51] to simulate the robot arm and haptic sensor 1 .
Revised: PyBullet [54] to simulate the robot arm and haptic sensor 1 .

Original: 4.1 MiniTouch Benchmark
Revised: 4.1 MiniTouch Benchmark

Original: Our proposed benchmark, MiniTouch, consists of four manipulation tasks:
Revised: Our proposed benchmark, MiniTouch, consists of four manipulation tasks:

Original: Playing, Pushing, Opening, Pick-up.
Revised: Playing, Pushing, Opening, Pick-up.

Original: Each of the tasks along with corresponding actions, observations, and rewards isdescribed in detail in the Supplementary Material Section 7.1 and further illustrated in Figure 7.
Revised: Each of the tasks along with corresponding actions, observations, and rewards isdescribed in detail in the Supplementary Material Section 7.1 and further illustrated in Figure 2.

Original: MiniTouch is an active repository 

Revised: MiniTouch is an active repository

Original: and we expect to update the benchmark with new tasks anddatasets.
Revised: and we expect to update the benchmark with new tasks anddatasets.

Original: The tasks are inspired by Yu et al.
Revised: The tasks are inspired by Yu et al.

Original: [52] but are not based on a proprietary simulator, featurean arm that we have access to for real-world experimentation (in follow-up work), and where thearm is equipped with a haptic sensor.
Revised: [55] but are not based on a proprietary simulator, featurean arm that we have access to for real-world experimentation (in follow-up work), and where thearm is equipped with a haptic sensor.

Original: 4.2 Baseline Comparisons and Metrics
Revised: 4.2 Baseline Comparisons and Metrics

Original: For the task evaluation, we study two versions of our model: (i) ToC-Pure considering the touch vector reconstruction intrinsic reward alone, and (ii) ToC, considering the full intrinsic reward (Eq. 4).
Revised: For the task evaluation, we study two versions of our model: (i) ToC-Pure considering the touch vector reconstruction intrinsic reward alone, and (ii) ToC, considering the full intrinsic reward (Eq. 4).

Original: We compareour model withSAC and several well-known intrinsic exploration  based baselines:
Revised: We compare with several well-known intrinsic exploration baselines based on visual features:

Original: • SAC : The unmodiﬁed Soft Actor-Critic algorithm from Haarnoja et al.
Revised: • SAC : The unmodiﬁed Soft Actor-Critic algorithm from Haarnoja et al.

Original: ICM : SAC augmented with the state-of-the-art visual curiosity approach Intrinsic Curiosity
Revised: ICM : SAC augmented with the state-of-the-art visual curiosity approach Intrinsic CuriosityModule (ICM)

Original: Module (ICM) [12], which uses a visual forward model to guide exploration. • Disagreement : It uses model disagreement as objective for exploration [53, 28].
Revised: [12], which uses a visual prediction model to guide exploration.• Disagreement : It uses model disagreement as objective for exploration [56, 28].

Original: It leverages variance in the prediction of an ensemble of latent dynamics models as the reward.
Revised: It leverages variance in the prediction of an ensemble of latent dynamics models as the reward.

Original: • RND : Random Network Distiallation [54] utilizes a randomly initialized neural network to
Revised: • RND : Random Network Distillation [57] utilizes a randomly initialized neural network to

Original: specify an intrinsic reward for visiting unexplored states in hard exploration problems.
Revised: specify an intrinsic reward for visiting unexplored states in hard exploration problems.

Revised: We built

Revised: a Pytorch [

Revised: version

Original: We built a Pytorch [55] version of these baselines based on their open source code (details in supplementary material).
Revised: of these baselines based on their open source code details

Original:  We use the following metrics to evaluate our method and the baseline models:
Revised: in sup- plementary material). We use the following metrics to evaluate our method and the baseline models:

Original: Exploration success: measures the percentage of times that the agent attained the goal state in the
Revised: Exploration success: measures the percentage of times that the agent attained the goal state in the

Original: exploratory phase, i.e. with no external reward.
Revised: exploratory phase, i e. with no external reward.

Original: Higher is better.
Revised: Higher is better.

Original: Success: denotes the percentage of times that the agent attained the goal state during the down-
Revised: Success: denotes the percentage of times that the agent attained the goal state during the down-

Original: stream task phase.
Revised: stream task phase.

Original: Episode steps: quantiﬁes the number of steps required for each episode to succeed.
Revised: Episode steps: The number of steps required for each episode to succeed.

Original: This metric isan indicator of sample efﬁciency
Revised: This metric is an indicatorof sample efﬁciency.

Original: The lesser the number of steps, the faster the agent’s ability 
Revised: The lesser the number of steps, the faster the agent’s ability to succeed.

Original: tosucceed.
Revised: [Figure]

Original: Touch-interaction: Amount of interaction the agent’s ﬁngers have with the underlying object.
Revised: Touch-interaction: Amount of interaction the agent’s ﬁngers have with the underlying object.

Original: We measure this by computing the variance of force/torque sensory signal across the whole episode.
Revised: Wemeasure this by computing the variance of force/torque sensory signal across the whole episode.

Original: Higher variance indicates better interaction

Original: Object movement: The agent can resort to constantly engaging with the object 
Revised: Object movement: The agent can resort to constantly engaging with the object unnecessarily tosatisfy the objective.

Original: in a

Original: passive manner

Original: in order to satisfy the objective.

Original: We, therefore, compute

Original: the variance of

Original:  door angle (for the Opening task) and the variance of object position (for the remaining tasks) over the course of training
Revised: We, therefore, monitor the variance of door angle (for the Opening task)and the variance of object position (for the remaining tasks) over the course of training.

Original: A

Original: higher motion indicates diverse state space and that the agent does not get stuck
Revised: A highermovement indicates diverse state space in addition to physical interaction.

Original: with

Original: pointless

Original: subtle

Original: movements of the objects.

Original: [Figure]

Original:  Results and Discussion
Revised:  Results and Discussion

Original: ToC and baselines were trained on a Panda robot agent [51] for one million steps.
Revised: ToC and baselines were trained on a Panda robot agent [54] for one million steps.

Original: In the exploratoryphase of the training, we pre-train our method only with the curiosity-based intrinsic reward.
Revised: In the exploratoryphase of the training, we pre-train our method only with the curiosity-based intrinsic reward.

Original: Wethen progress to the adaptation phase.
Revised: Wethen progress to the adaptation phase.

Original: Also, note that across all tasks, ToC-Pure is based purely oncross-modal prediction, while ToC includes visual forward prediction reward in addition.
Revised: Also, note that across all tasks, ToC-Pure is based purely oncross-modal prediction, while ToC includes visual forward prediction reward in addition.

Original: Figure 2 shows results on the basic task of playing with a single object.
Revised: Figure 3a and 3b shows results on the basic task of playing with a single object.

Original: Since single object interaction does not have explicit goal states to evaluate, we instead measure the agent’sability toconstantly engage and play with the underlying object.
Revised: Since singleobject interaction does not have explicit goal states to evaluate, we instead measure the agent’s

Original: ToC-Pure displays four times better interaction with the underlying object when compared to SAC (see Figure 2b).
Revised: [Figure] [Figure]

Original: Also, the plot depictingobject movement metric in Figure 2a indicates that ToC is slightly more dynamic in the interactionin contrast to ToC-Pure.
Revised: [Table] [Figure] [Table] [Figure]

Original: Note from the plots that there is a trade-off when using ToC and ToC-Purebetween constant interaction (i.e. touch interaction performance) and object movement dynamics.
Revised: [Table]

Original: metrics evaluated on the playing task and Open-door task for different forward prediction weightings sampled ⇠ [1 , 0] . Large weight (darker plot) favors object-movement towards the end of training, where as smaller weight improves touch-interaction. Middle value in the sweep range ( e.g. 0.5) balances the trade-off.
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We omit measuring success and episode steps for playing task since success there is equivalent to object-interaction and has no explicit goal.
Revised: ability to constantly engage and play with the underlying object. ToC-Pure displays four timesbetter interaction with the underlying object when compared to SAC (see Figure 3b). Note from theplots that there is a trade-off when using ToC and ToC-Pure between constant interaction (i.e. touchinteraction performance) and object movement dynamics.

Original: Collecting a variety of such interesting data during the exploratory phase helps the agent in terms ofsample efﬁciency while solving the downstream tasks.
Revised: Collecting a variety of such interestingdata during the exploratory phase helps the agent in terms of sample efﬁciency while solving thedownstream tasks.

Original: Figure 8 in the supplemental material sectionshows similar plots for door opening task.
Revised: Figure 8 shows similar comparison for Open-door task.

Original: We also include short video clips displaying qualitativeperformance on each of the tasks in the supplementary material.
Revised: In the following section,we examine the role of the forward objective term on touch interaction and possible ways to encodeagent’s observation in our ablation studies.

Original: We compare ToC and ToC-Pure with SAC and state-of-the-art vision-based curiosity baselines onthe remaining downstream tasks in MiniTouch.
Revised: We compare ToC and ToC-Pure with SAC and state-of-the-art vision-based curiosity baselines onthe remaining downstream tasks in MiniTouch.

Original: From Figure 3 it is evident that ToC and ToC-Pureperform better than SAC in all the tasks and better than the vision-based curiosity models in themajority of the tasks.
Revised: From Figure 4 it is evident that ToC and ToCPure perform better than SAC in all the tasks and better than the vision-based curiosity modelsin the majority of the tasks.

Original: Using SAC alone hinders the performance and is often unable to solve any ofthe three tasks.
Revised: Using SAC alone hinders the performance and is often unable tosolve any of the three tasks.

Original: This is not surprising since the model is not motivated enough to collect diverse anduseful data through interaction.
Revised: This is not surprising since the model is not motivated enough tocollect diverse and useful data through interaction.

Revised: We hypothesize that ToC-Pure without the visualprediction objective L fdm could potentially bias the solutions towards more physical interaction,which is not necessary for every task.

Revised: For instance, constant interaction in the pushing task couldbe seen as a hindrance as it does not let the object slide easily towards the target.

Revised: Observe that

Original: ICM performs better than ToC-Pure in the pushing task, however, ToC dominates in performance by about 15%.
Revised: ICM performs better than ToC-Pure in the pushing task, however, ToC dominates in performance byabout 15%.

Original: Recall that the goal is not just to succeed but tohelp attain success in a sample efﬁcient manner in fewer steps.
Revised: Recall that the goal is not just to succeed but to help attain success in a sample efﬁcientmanner in fewer steps.

Original: The results support our hypothesisthat cross-modal curiosity enables an RL agent to succeeds at an early stage in training and oftenwithout any external reward.
Revised: The results support our hypothesis that cross-modal curiosity enables an RLagent to succeed at an early stage in training and often without any external reward.

Original: Similarly, our model outperforms on the opening task without externalreward.
Revised: Similarly, ourmodel outperforms on the opening task without external reward.

Original: However, ToC initially has lower success compared to RND, but surpasses  towards theend.
Revised: However, ToC initially has lowersuccess compared to RND but surpasses RND towards the end.

Original: Although ToC and ToC-Pure attain similar success in the pick-up task towards the end ofthe training, it is compelling to note that ToC-Pure attains faster convergence.
Revised: Although ToC and ToC-Pure attainsimilar success in the pick-up task towards the end of the training, it is compelling to note that ToCPure attains faster convergence.

Original: This is because thepicking task requires constant touch interaction (where ToC-Pure has an advantage), as opposed todiverse object movement.
Revised: This is because the picking task requires constant touch interaction(where ToC-Pure has an advantage), as opposed to diverse object movement.

Original: In the following section, we examine the role of the forward objectiveterm on touch interaction and possible ways to encode agent’s observation in our ablation studies.

Original: 5.1 Ablations
Revised: 5.1 Ablations

Original: Importance of forward objective. Visual forward prediction L fdm plays a handy role when it isused in the right proportion.
Revised: Forward objective and tuning Lambda Visual forward prediction L fdm plays an important rolewhen it is used in the right proportion.

Original: Intrinsic reward is a weighted combination of cross-modal predictionand forward prediction as deﬁned in Eq.4.
Revised: Our intrinsic reward is a weighted combination of crossmodal prediction and forward prediction as deﬁned in Eq.4.

Original: Figure 4 illustrates theperformance of the model withdifferent levels of emphasis on the forward loss term, with  uniformly sampled between 0 and 1.
Revised: Figure 5 illustrates the model behaviorwith different levels of emphasis on the forward loss term, with  uniformly sampled between 0 and [Figure]

Original: Higher weights indicate that the future prediction dominates force/torque prediction.
Revised: Higher weights indicate that the future prediction dominates force/torque prediction.

Original: We notice thatboth larger and smaller values of  hurt the overall performance. Larger weight leads to more objectmovement but hurts the robot’s constant touch-interaction with the underlying objects, while smallervalue leads to  inactive behavior while satisfying the objective as shown in Figure.4.
Revised: This leadsto more object movement but lesser robot’s constant touch-interaction with the underlying objects. This is useful in tasks such as Pushing. A smaller  value leads to better inactive behavior which ishandy in tasks such as grasping.

Original: [Figure]
Revised: In our experiments we choose an intermediate value ( e.g. 0.5) thatworks best for all of our tasks.

Original: Touch-based future prediction.
Revised: Touch in future prediction.

Original: The goal of this experiment is to strengthen the argument of crossmodal association.
Revised: The goal of this experiment is to strengthen the argument of crossmodal association.

Original: While conducting experiments, it is important to deduce information of onemodality from another modality in a related manner than simply adding another modality on top ofvisual information.
Revised: While conducting experiments, it is important to deduce information of onemodality from another modality in a related manner than simply adding another modality on top ofvisual information.

Original: We created an additionaltouch-based baseline, ToC- future , where in addition tothe visual future prediction model we include the touch-based future prediction model.
Revised: We created an additional baseline, ICM+ touch , where in addition to the visualprediction model we include the touch-based future prediction model.

Revised: We hypothesize ICM+ touchto perform better than ICM as it has additional information (touch).

Original: The touchbased future prediction model takes a touch vector as input and predicts the touch vector for thenext time step.
Revised: The touch-based future prediction model takes a touch vector as input and predicts the touch vector for the next time step.

Original: Table 1 compares ToC and ToC- future on MiniTouch tasks and we observe that 
Revised: Table 1compares ToC and ICM+ touch on MiniTouch tasks and we observe that ICM+ touch is better than

Original: ToC- future is better than ICMin general but compares below ToC. 
Revised: ICM but compares below ToC. Table 4 in the supplementary compares RND based baseline.

Original: Robustness to diverse shapes.
Revised: Robustness to diverse shapes.

Original: It is desirable for an agent to be able to handle diverse shapes in orderto be robust across arbitrary manipulation settings.
Revised: It is desirable for an agent to be able to handle diverse shapes in orderto be robust across arbitrary manipulation settings.

Original: We study this using an environment in whichan object is sampled from a thousand procedurally generated objects.
Revised: We study this using an environment in whichan object is sampled from a thousand procedurally generated objects.

Original: The objects are dissimilarwith respect to shape and mass but are sampled from the same generative distribution.
Revised: The objects are dissimilarwith respect to shape and mass but are sampled from the same generative distribution.

Original: Out of 1000different objects, 800 of them are used in the training phase, and we evaluate the agent’s effectivenesson the remaining 200 unseen object shapes.
Revised: Out of 1000different objects, 800 of them are used in the training phase, and we evaluate the agent’s effectivenesson the remaining 200 unseen object shapes.

Original: Figure 5b shows touch interaction and object movementevaluations for a single object exploration task.
Revised: Figure 6b shows touch interaction and object movementevaluations for a single object exploration task.

Original: The results validate that our model generalizes tounseen object conﬁgurations.
Revised: The results validate that our model generalizes tounseen object conﬁgurations (see Section 7.5 in the supplementary material for more results).

Original: Latent features for forward dynamics.
Revised: Latent features for forward dynamics.

Original: Choosing ideal embedding space for decoding the touchvector (Figure 1a) and for predicting future state (Figure. 1b) is important.
Revised: Choosing ideal embedding space for decoding the touchvector (Figure 1a) and for predicting future state (Figure. 1b) is important.

Original: Existing approachesrely on a pretext inverse dynamics  task of predicting the agent’s action given its current and nextstates [9].
Revised: Existing approaches relyon a pretext inverse dynamics (ID) task of predicting the agent’s action given its current and nextstates [9].

Original: Another simple yet strong method is to use features from a random but ﬁxed initializationof the encoder [12].
Revised: Another simple yet strong method is to use features from a random but ﬁxed initializationof the encoder [12].

Original: In our work, we learn the features by leveraging the self-supervised pretexttask of predicting one modality from the other.
Revised: In our work, we learn the features by leveraging the self-supervised pretext taskof predicting one modality from the other.

Original: Figure.5a compares (1) encoder, i.e. learned throughcross-modal prediction(ToC) (2) random feature encoder(ToC-RF) (3) encoder learned through IDFtask (ToC-IDF).
Revised: Figure 6a compares (1) encoder, i.e. learned throughcross-modal prediction(ToC) (2) random feature encoder(ToC-RF) (3) encoder learned through IDtask (ToC-IDF).

Original: In each case, the decoder network is optimized through touch prediction.
Revised: In each case, the decoder network is optimized through touch prediction.

Original: We observe that the random features variant is stable and effective on both ToC and ICM models.
Revised: We observe that the random features variant is stable and effective on both ToC and ICM models.

Original: Real world validation: While no real robot data have been used in the experiments, our method isdata agnostic.

Original: We note that the physics engine can be replaced by real-world data as there is no biasintroduced in process.

Original: The networks used for the curiosity prediction task can be easily adapted, ifneeded, to accommodate inputs from a real robotic platform.

Original: As a next step and when restrictionsare eased, we plan to implement this method on a physical system and replicate our experiments.

Original:  Conclusion
Revised:  Conclusion

Original:  Building self-seeking RL agents that can leverage cross-modal associations could be a key researchdirection in developmental Robotics.
Revised: [Figure] Building self-seeking RL agents that can leverage cross-modal associations could be a key researchdirection in developmental robotics.

Original: Sense of touch is ubiquitous in human routine.
Revised: Sense of touch is ubiquitous in human routine.

Original: Borrowing insights from how infants learn to explore out of curiosity by touching what they visually perceive, weformulated Touch-based Curiosity (ToC) that is aimed at encouraging exploration via haptic interaction with the environment.
Revised: Borrowing insights from how infants learn to explore out of curiosity by touching what they visually perceive, weformulated Touch-based Curiosity (ToC) that is aimed at encouraging exploration via haptic interaction with the environment.

Original: We demonstrated in this work, that by involving additional modalities,the performance of curiosity-based systems on downstream tasks can be increased.
Revised: We demonstrated in this work, that by involving additional modalities,the performance of curiosity-based systems on downstream tasks can be increased.

Original: We observedincreased interaction with target objects, and presented evidence that ToC learns to solve the MiniTouch benchmark tasks in an efﬁcient manner while vanilla RL algorithms and vision-based curiosity formulations struggled.
Revised: We observedincreased interaction with target objects, and presented evidence that ToC learns to solve the MiniTouch benchmark tasks in an efﬁcient manner while vanilla RL algorithms and vision-based curiosity formulations struggled.

Original: Force/torque sensing is widespread in the lab and industrial robots butwhile there are plenty of robotic benchmarks, we believe that tactile feedback is an under-exploredmodality and by releasing our benchmark, we hope to enable future research in this exciting area.
Revised: Force/torque sensing is widespread in the lab and industrial robots butwhile there are plenty of robotic benchmarks, we believe that tactile feedback is an under-exploredmodality and by releasing our benchmark, we hope to enable future research in this exciting area.