Original: Optical Tactile Sim-to-Real Policy Transfer via Real-to-SimTactile Image Translation
Revised: Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation

Original: Anonymous Author(s) AfﬁliationAddress email Abstract: Simulation has recently become key for deep reinforcement learning to safely and efﬁciently acquire general and complex control policies from visualand proprioceptive inputs.
Revised: Abstract: Simulation has recently become key for deep reinforcement learning to safely and efﬁciently acquire general and complex control policies from visual and proprioceptive inputs.

Original: Tactile information is not usually considered despite itsdirect relation to environment interaction.
Revised: Tactile information is not usually considered despite its direct relation to environment interaction.

Original: In this work, we present a suite of simulated environments tailored towards tactile robotics and reinforcement learning.
Revised: In this work, we present a suite of simulated environments tailored towards tactile robotics and reinforcement learning.

Original: A simple and fast method of simulating optical tactile sensors is provided, wherehigh-resolution contact geometry is represented as depth images.
Revised: A simple and fast method of simulating optical tactile sensors is provided, where high-resolution contact geometry is represented as depth images.

Original: Proximal Policy
Revised: Proximal Policy

Original: Optimisation (PPO) is used to learn successful policies across all considered tasks.
Revised: Optimisation (PPO) is used to learn successful policies across all considered tasks.

Original: A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images.
Revised: A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images.

Original: This policy is implemented withina real-time control loop on a physical robot to demonstrate zero-shot sim-to-realpolicy transfer on several physically-interactive tasks requiring a sense of touch.
Revised: This policy is implemented within a real-time control loop on a physical robot to demonstrate zero-shot sim-to-real policy transfer on several physically-interactive tasks requiring a sense of touch.

Original: Video results are available at https://sites.google.com/view/tactile-sim-to-real .
Revised: Video results: https://sites.google.com/my.bristol.ac.uk/tactile-gym-sim2real/home .

Revised: Code: https://github.com/ac-93/tactile gym .

Original: [Figure]
Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: Learning algorithms have an innate appeal for robotics to enable general, complex behaviours thatwould be difﬁcult to achieve with classic control methods.
Revised: Learning algorithms have an innate appeal for robotics to enable general, complex behaviours that would be difﬁcult to achieve with classic control methods.

Original: Large-scale data is often required forrobot learning, and so physics-based simulation has a vital role to play for data collection.
Revised: Large-scale data is often required for robot learning, and so physics-based simulation has a vital role to play for data collection.

Original: A common approach in learning-based robotics is to simulate data that would be impractical to collect inreality, learn control policies from this data and then transfer learned skills to the physical system
Revised: A common approach in learning-based robotics is to simulate data that would be impractical to collect in reality, learn control policies from this data, and then transfer learned skills to the physical system.

Original: Simulation also offers advantages such as avoiding damage during exploratory training, exploiting privileged information from the simulator and use of open-sourcing for dissemination of research ﬁndings.
Revised: Simulation also offers advantages such as avoiding damage during exploratory training, exploiting privileged information from the simulator, and use of open-sourcing for dissemination of research ﬁndings.

Original: However, physics engines necessarily approximate the real world to reduce computational costs, giving a ‘sim-to-real gap’ that impairs the performance of these policies applied to reality
Revised: However, physics engines necessarily approximate the real world to reduce computational costs, giving a ‘sim-to-real gap’ that impairs the performance of these policies applied to reality.

Original: Reinforcement Learning (RL) robotics research is dominated by the use of proprioception and visionas the sensory inputs.
Revised: Reinforcement Learning (RL) robotics research is dominated by the use of proprioception and vision as the sensory inputs.

Original: Whilst this can lead to complex behaviours [1, 2, 3], other important sourcesof information have been underutilised.
Revised: Whilst this can lead to complex behaviours [1, 2, 3], other important sources of information have been underutilised.

Original: Speciﬁcally, humans use the sense of touch to accomplishcomplex manual tasks, utilising a granularity of detail unavailable to our other senses.
Revised: Speciﬁcally, humans use the sense of touch to accomplish complex manual tasks, utilising a granularity of detail unavailable to our other senses.

Original: Tactile datahas several advantages as the main source of information for learning: it does not suffer from occlusion, particularly for ﬁne manipulation; contact information can  more detailed than visual imagesof an entire scene; and the observation space is constrained (e.g. to tactile images of markers),simplifying the translation of real to simulated data and, likewise, from simulated to real policies.
Revised: Tactile data has several advantages as the main source of information for learning: it does not suffer from occlusion, particularly for ﬁne manipulation; contact information can be more detailed than visual images of an entire scene; and the observation space is constrained (e.g. to tactile images of markers), simplifying the translation of real to simulated data and, likewise, from simulated to real policies.

Original: Here we make the following contributions aimed at bringing together reinforcement learning andtactile robotics: 1) We provide an open-source suite of RL environments tailored to tactile robotics,utilising a simple and fast method of simulating a tactile sensor.
Revised: Here we make the following contributions aimed at bringing together reinforcement learning and tactile robotics: 1) We provide an open-source suite of RL environments tailored to tactile robotics, utilising a simple and fast method of simulating a tactile sensor.

Original: We show that an RL method,
Revised: We show that an RL method,

Original: Proximal Policy Optimisation (PPO) [4], acquires successful policies for all of these tasks.
Revised: Proximal Policy Optimisation (PPO) [4], acquires successful policies for all of these tasks.

Original: 3) Wedemonstrate and validate a data-driven approach for zero-shot sim-to-real policy transfer via imagetranslation between real and simulated tactile images on these tasks.
Revised: We demonstrate and validate a data-driven approach for zero-shot sim-to-real policy transfer via image translation between real and simulated tactile images on these tasks.

Original:  Related Work
Revised:  Related Work

Original: Sim-to-Real Transfer: For computational efﬁciency, physics engines necessarily approximate thereal-world dynamics, leading to a sim-to-real gap where dynamics and visuals differ between simulation and reality.
Revised: Sim-to-Real Transfer: For computational efﬁciency, physics engines necessarily approximate the real-world dynamics, leading to a sim-to-real gap where dynamics and visuals differ between simulation and reality.

Original: This gap can make it difﬁcult to transfer skills learned in simulation to the realworld.
Revised: This gap can make it difﬁcult to transfer skills learned in simulation to the real world.

Original: Several methods have been proposed to mitigate this issue, namely Domain Randomisation ,
Revised: Several methods have been proposed to mitigate this issue, namely Domain Randomisation ,

Original: Network Distillation and Domain Adaption [3].
Revised: Network Distillation and Domain Adaption [3].

Original: In this work we mainly use Domain Adaption , following James et al.
Revised: In this work we mainly use Domain Adaption , following James et al.

Original: [5] to train an imageconditioned generator network that translates between real and simulated images.
Revised: [5] to train an imageconditioned generator network that translates between real and simulated images.

Original: When using vision, Domain Randomisation was necessary on simulated images to train a generator robust enoughto generalise to real visual images.
Revised: When using vision, Domain Randomisation was necessary on simulated images to train a generator robust enough to generalise to real visual images.

Original: Here we exploit the more constrained image space of opticaltactile sensors to train generator networks using a controlled tactile dataset.
Revised: Here we exploit the more constrained image space of optical tactile sensors to train generator networks using a controlled tactile dataset.

Original: We perform some randomisations to make simulated tasks more difﬁcult and RL policies more robust, aiming to keepwithin realistic conditions.
Revised: We perform some randomisations to make simulated tasks more difﬁcult and RL policies more robust, aiming to keep within realistic conditions.

Original: Although privileged information from the simulator is used to constructa reward for improved learning, this information is not part of the observation in any task.
Revised: Although privileged information from the simulator is used to construct a reward for improved learning, this information is not part of the observation in any task.

Original: As weperform zero-shot policy transfer from sim-to-real, Network Distillation is not required.
Revised: As we perform zero-shot policy transfer from sim-to-real, Network Distillation is not required.

Original: Tactile Sim-to-Real: Recently, several works have aimed to bridge between tactile simulation andreality.
Revised: Tactile Sim-to-Real: Recently, several works have aimed to bridge between tactile simulation and reality.

Original: These split into two categories: simulating the physics of sensor deformation via Finite Element (FE) methods or replicating the captured sensor information via image rendering techniques.
Revised: These split into two categories: simulating the physics of sensor deformation via Finite Element (FE) methods or replicating the captured sensor information via image rendering techniques.

Original: A concise overview of the sim-to-real approach applied to non-vision based sensors is given in [6].
Revised: A concise overview of the sim-to-real approach applied to non-vision based sensors is given in [6].

Original: FE methods have seen recent advancements in computational efﬁciency and accuracy.
Revised: FE methods have seen recent advancements in computational efﬁciency and accuracy.

Original: Narang et al.[7] simulated deformations of the BioTac sensor that, in combination with data-driven learning,can accurately regress the sensor output.
Revised: Narang et al. [7] simulated deformations of the BioTac sensor that, in combination with data-driven learning, can accurately regress the sensor output.

Original: Later, a 75-fold improvement in simulation speed wasachieved with GPU-acceleration [6], with each simulated sensor output taking up to 5 seconds whichis still impractical for sample-inefﬁcient methods such as DRL.
Revised: Later, a 75-fold improvement in simulation speed was achieved with GPU-acceleration [6], with each simulated sensor output taking up to 5 seconds which is still impractical for sample-inefﬁcient methods such as DRL.

Original: A similar approach was applied toa marker-based optical tactile sensor for regressing contact force ﬁelds [8, 9].
Revised: A similar approach was applied to a marker-based optical tactile sensor for regressing contact force ﬁelds [8, 9].

Original: Later work [10]achieved zero-shot sim-to-real transfer of a complex policy learned in simulation via RL; however,task-speciﬁc approximation was needed for computational efﬁciency, e.g. assuming planar motion.
Revised: Later work [10] achieved zero-shot sim-to-real transfer of a complex policy learned in simulation via RL; however, task-speciﬁc approximation was needed for computational efﬁciency, e.g. assuming planar motion.

Original: Most comparable to our work, Ding et al.
Revised: Most comparable to our work, Ding et al.

Original: [11] use an elastic deformation approach on the sameoptical tactile sensor(the TacTip) as used here.
Revised: [11] use an elastic deformation approach on the same tactile sensor as used here (a TacTip soft biomimetic optical tactile sensor [12, 13]).

Original: They focussed on supervised learning from simulatedpin positions rather than tactile images, and with domain randomisation could accurately predictedge position and orientation for use in contour following.
Revised: They focused on supervised learning from simulated marker positions rather than tactile images, and with domain randomisation could accurately predict edge position and orientation for use in contour following.

Original: The authors noted that the simulationspeed is restricted by Unity’s collision detection module.
Revised: The authors noted that the simulation speed is restricted by Unity’s collision detection module.

Original: A computationally efﬁcient approach is to render images from speciﬁc optical tactile sensors withrigid-body physics approximating the contact dynamics.
Revised: A computationally efﬁcient approach is to render images from speciﬁc optical tactile sensors with rigid-body physics approximating the contact dynamics.

Original: Depth images from a GelSight tactile sensor were rendered using the Gazebo physics engine with additional Gaussian smoothing, light rendering and image calibration to help match real tactile images [12].
Revised: Depth images from a GelSight tactile sensor were rendered using the Gazebo physics engine with additional Gaussian smoothing, light rendering and image calibration to help match real tactile images [14].

Original: On a supervised classiﬁcationtask, a drop of 39% accuracy was found for direct sim-to-real transfer, reduced to 6.5% with textureaugmentation as a form of domain randomisation.
Revised: On a supervised classiﬁcation task, a drop of 39% accuracy was found for direct sim-to-real transfer, reduced to 6.5% with texture augmentation as a form of domain randomisation.

Original: A similar approach is proposed in [13] with extended image-rendering techniques applicable to a range of GelSight-type sensors on more complexphysically-interactive tasks; however, so far no sim-to-real quantitative results have been reported.
Revised: A similar approach is proposed in [15] with extended image-rendering techniques applicable to a range of GelSight-type sensors on more complex physically-interactive tasks; however, so far no sim-to-real quantitative results have been reported.

Original: Our work here focuses on a high-resolution tactile simulation environment appropriate for RL ofcomplex control policies.
Revised: Our work here focuses on a high-resolution tactile simulation environment appropriate for RL of complex control policies.

Original: By relying on rigid-body approximations to the contact dynamics, ourphysics simulation is signiﬁcantly faster than FE methods without task-speciﬁc approximations.
Revised: By relying on rigid-body approximations to the contact dynamics, our physics simulation is signiﬁcantly faster than FE methods without task-speciﬁc approximations.

Original: Our simulated TacTip images use depth image-rendering techniques like those used for the GelSight[12, 13].
Revised: Our simulated TacTip images use depth image-rendering techniques like those used for the GelSight [14, 15].

Original: However, a key novelty is that our simulated depth images do not closely resemble the real
Revised: However, a key novelty is that our simulated depth images do not closely resemble the real

Original: TacTip images, because depth is non-trivially related to marker shear for the TacTip (and closelyrelated to image shading for the GelSight).
Revised: TacTip images, because depth is non-trivially related to marker shear for the TacTip (and closely related to image shading for the GelSight).

Original: Instead, we close the sim-to-real gap by relying on alater domain adaption phase.
Revised: Instead, we close the sim-to-real gap by relying on a later domain adaption phase.

Original: Hence, we keep the simulation agnostic to the sensor transduction,which should facilitate future application of these methods to other types of tactile sensor.
Revised: Hence, we keep the simulation agnostic to the sensor transduction, which should facilitate future application of these methods to other types of tactile sensor.

Original:  Tactile Simulation
Revised:  Tactile Simulation

Original: In this work, we utilise PyBullet’s synthetic camera rendering to capture depth images within avirtual optical tactile sensor, based on the CAD ﬁles used to 3D print a real TacTip (see e.g. [14,
Revised: In this work, we utilise PyBullet’s synthetic camera rendering to capture depth images within a virtual optical tactile sensor, based on the CAD ﬁles used to 3D print a real TacTip (see e.g. [12,

Original: Fig. 4]).
Revised: Fig. 4]).

Original: When gathering tactile images, we take the difference between the current depth image anda reference depth image taken from when the sensor is not in contact.
Revised: When gathering tactile images, we take the difference between the current depth image and a reference depth image taken from when the sensor is not in contact.

Original: This difference produces apenetration depth map that generalises to arbitrary sensor shapes.
Revised: This difference produces a penetration depth map that generalises to arbitrary sensor shapes.

Original: Noise is removed from the imageby zeroing values below a set tolerance of 10 − 4 , followed by a re-scaling from [0 , max penetration]to [0 , 255] .
Revised: Noise is removed from the image by zeroing values below a set tolerance of 10 − 4 , followed by a re-scaling from [0 , max penetration] to [0 , 255] .

Original: An artiﬁcial border is also overlaid onto the tactile image to bring it closer to real tactileimages and to provide a reference point that transforms with augmentations.
Revised: An artiﬁcial border is also overlaid onto the tactile image to bring it closer to real tactile images and to provide a reference point that transforms with augmentations.

Original: In order to achieve the computational efﬁciency necessary to generate data at large scales, we approximate the soft tip of the tactile sensor with rigid body physics simulation.
Revised: In order to achieve the computational efﬁciency necessary to generate data at large scales, we approximate the soft tip of the tactile sensor with rigid body physics simulation.

Original: We limit the contactstiffness and damping used during collision detection as this allows penetration of objects into thesimulated sensor tip in an approximate manner to the deformation of the real tip.
Revised: We limit the contact stiffness and damping used during collision detection as this allows penetration of objects into the simulated sensor tip in an approximate manner to the deformation of the real tip.

Original: We describe our simulation method in relation to the list of desiderata proposed by Wang et al.
Revised: We describe our simulation method in relation to the list of desiderata proposed by Wang et al.

Original: High Throughput:
Revised: High Throughput:

Original: We use PyBullet’s GPU rendering functionality to offer fast simulation.
Revised: We use PyBullet’s GPU rendering functionality to offer fast simulation.

Original: On a
Revised: On a

Original: PC with an Nvidia 2080Ti, we can achieve up to 1000 fps when rendering single tactile images at128 × 128-resolution.
Revised: PC with an Nvidia 2080Ti, we can achieve up to 1000 fps when rendering single tactile images at 128 × 128-resolution.

Original: Multiple PyBullet physics engines can also run in parallel, so during trainingwe used 10 vectorised environments to increase throughput.
Revised: Multiple PyBullet physics engines can also run in parallel, so during training we used 10 vectorised environments to increase throughput.

Original: Flexible: Whilst this study is predominantly based on the TacTip sensor, we do not attempt to simulate images accurate to any speciﬁc tactile sensor.
Revised: Flexible: Whilst this study is predominantly based on the TacTip sensor, we do not attempt to simulate images accurate to any speciﬁc tactile sensor.

Original: Instead, we simulate only useful tactile features,relying on a later image-translation stage to map from real to simulated images.
Revised: Instead, we simulate only useful tactile features, relying on a later image-translation stage to map from real to simulated images.

Original: We expect similar results are possible with a broad range of other high-resolution optical tactile sensors, includingsensors of the Gelsight type [15, 16, 17], the Soft-Bubble [18] and optical shear-based sensors [19].
Revised: We expect similar results are possible with a broad range of other high-resolution optical tactile sensors, including sensors of the Gelsight type [16, 17, 18], the Soft-Bubble [19] and optical shear-based sensors [20].

Original: Realistic: Instead of aiming to realistically simulate the physical properties of any speciﬁc sensor,which is both difﬁcult and computationally expensive, we simulate only the desired properties ofan idealised tactile sensor, currently focussing only on contact geometry.
Revised: Realistic: Instead of aiming to realistically simulate the physical properties of any speciﬁc sensor, which is both difﬁcult and computationally expensive, we simulate only the desired properties of an idealised tactile sensor, currently focussing only on contact geometry.

Original: Our data-driven approachhelps bridge the sim-to-real gap, so there is no need to generate synthetic images that match imagesfrom a real sensor to high precision.
Revised: Our data-driven approach helps bridge the sim-to-real gap, so there is no need to generate synthetic images that match images from a real sensor to high precision.

Original: Ease of use: The simulation suite and the sim-to-real translation tools will be open-sourced.
Revised: Ease of use: The simulation suite has been open-sourced.

Original: Thesimulation only requires the commonly-used PyBullet, so our approach should have a low barrier ofentry.
Revised: The simulation only requires the commonly-used PyBullet, so our approach should have a low barrier of entry.

Original: Our approach should also extend readily to different tasks and other tactile sensors. 
Revised: Our approach should also extend readily to different tasks and other tactile sensors. [Figure] w

Original:  Reinforcement Learning Environments
Revised:  Reinforcement Learning Environments

Original: Each task is provided with a set of observation spaces to allow for veriﬁcation of the environments,comparisons between tasks, and an examination of multi-modal visuotactile control.
Revised: Each task is provided with a set of observation spaces to allow for veriﬁcation of the environments, comparisons between tasks, and an examination of multi-modal visuotactile control.

Original: Four standardtypes of observation space are considered:
Revised: Four standard types of observation space are considered:

Original: Env State: Comprises state information from the simulator, which is difﬁcult information to collectin the real world.
Revised: Env State: Comprises state information from the simulator, which is difﬁcult information to collect in the real world.

Original: We use this to give baseline performance for a task that is expected to act as anupper limit.
Revised: We use this to give baseline performance for a task that is expected to act as an upper limit.

Original: The information in this state varies between environments but commonly includes toolcenter point (TCP) pose, TCP velocity, goal locations and the current state of the environment.
Revised: The information in this state varies between environments but commonly includes tool center point (TCP) pose, TCP velocity, goal locations and the current state of the environment.

Original: Tactile: Comprises images ( 128 × 128 ) retrieved from the simulated optical tactile sensor attachedto the end effector of the robot arm (Figure 2 right).
Revised: Tactile: Comprises images ( 128 × 128 ) retrieved from the simulated optical tactile sensor attached to the end effector of the robot arm (Figure 2 right).

Original: Where tactile information alone is not sufﬁcientto solve a task, this observation can be extended with state information retrieved from the simulator.[Figure] [Figure]
Revised: Where tactile information alone is not sufﬁcient to solve a task, this observation can be extended with state information retrieved from the simulator.

Original: This can only include information that could be easily and accurately captured in the real world,such as the TCP pose that is available on industrial robotic arms and the goal pose.
Revised: This can only include information that could be easily and accurately captured in the real world, such as the TCP pose that is available on industrial robotic arms and the goal pose.

Original: Visual: Comprises RGB images ( 128 × 128 ) retrieved from a static, simulated camera viewing theenvironment (Figure 2 left).
Revised: Visual: Comprises RGB images ( 128 × 128 ) retrieved from a static, simulated camera viewing the environment (Figure 2 left).

Original: Only a single camera was used, although this could be extended tomultiple cameras.
Revised: Only a single camera was used, although this could be extended to multiple cameras.

Original: As the simulated environment differs visually from the real-world environment,sim-to-real using RGB observations is challenging, requiring an approach like that of [5, 20].
Revised: As the simulated environment differs visually from the real-world environment, sim-to-real using RGB observations is challenging, requiring an approach like that of [5, 21].

Original: Visual + Tactile: Combines the RGB visual and tactile image observations to into a 4-channel RGBT image.
Revised: Visual + Tactile: Combines the RGB visual and tactile image observations to into a 4-channel RGBT image.

Original: This case demonstrates a simple method of multi-modal sensing.
Revised: This case demonstrates a simple method of multi-modal sensing

Original: 4.1 Tactile Exploration Environments
Revised: 4.1 Tactile Exploration Environments

Original: In this set of tasks, the robot interacts with a physical environment that stays static, with the objectiveto learn policies to safely explore physical areas.
Revised: In this set of tasks, the robot interacts with a physical environment that stays static, with the objective to learn policies to safely explore physical areas.

Original: These policies can be used to report informationabout novel objects, simplify human-control methods and improve operational safety of the robots.
Revised: These policies can be used to report information about novel objects, simplify human-control methods and improve operational safety of the robots.

Original: Two exploration environments have been created, for edge and surface following.
Revised: Two exploration environments have been created, for edge and surface following.

Original: More completedetails on these environments are given in Tables 5 and 6 of Appendix C.
Revised: More complete details on these environments are given in Tables 5 and 6 of Appendix C.

Original: Edge Following: The edge-following environment is used to train an agent that maintains its poserelative to an edge whilst traversing the edge towards a goal location.
Revised: Edge Following: The edge-following environment is used to train an agent that maintains its pose relative to an edge whilst traversing the edge towards a goal location.

Original: Previous work has demonstrated that robust 2D contour following can be achieved with supervised learning techniques [21].
Revised: Previous work has demonstrated that robust 2D contour following can be achieved with supervised learning techniques [22].

Original: Here we demonstrate this task can also be completed via sim-to-real reinforcement learning.
Revised: Here we demonstrate this task can also be completed via sim-to-real reinforcement learning.

Original: Surface
Revised: Surface

Original: Following: The surface-following environment is used to train an agent that maintains aset contact penetration of the sensor whilst orientating the TCP normal to an undulating 3D surfacegenerated using OpenSimplex noise [22].
Revised: Following: The surface-following environment is used to train an agent that maintains a set contact penetration of the sensor whilst orientating the TCP normal to an undulating 3D surface generated using OpenSimplex noise [23].

Original: For ease of use, we automatically direct the sensor in thedirection of the goal, analogous to the pose-based tactile servo control methods introduced in [23].
Revised: For ease of use, we automatically direct the sensor in the direction of the goal, analogous to the pose-based tactile servo control methods introduced in [24].

Original: 4.2 Non-prehensile Manipulation Environments
Revised: 4.2 Non-prehensile Manipulation Environments

Original: In this set of tasks, the objective is to learn policies that manipulate external objects in a desiredmanner.
Revised: In this set of tasks, the objective is to learn policies that manipulate external objects in a desired manner.

Original: As our focus is on single sensors in this work, we consider non-prehensile manipulation.
Revised: As our focus is on single sensors in this work, we consider non-prehensile manipulation.

Original: However, the simulation does support rendering of multiple tactile images, which could in principlebe used simulate tactile grippers, manipulators or multiple robot arms in future work.
Revised: However, the simulation does support rendering of multiple tactile images, which could in principle be used simulate tactile grippers, manipulators or multiple robot arms in future work.

Original: Three manipulation tasks are proposed: object rolling, object pushing and object balancing.
Revised: Three manipulation tasks are proposed: object rolling, object pushing and object balancing.

Original: More details on thenon-prehensile manipulation environments are given in Tables 7, 8 and 9 of Appendix C.
Revised: More details on the non-prehensile manipulation environments are given in Tables 7, 8 and 9 of Appendix C.

Original: Object Rolling:
Revised: Object Rolling:

Original: The object-rolling environment requires the manipulation of small spherical objectsinto a goal position within the TCP coordinate frame.
Revised: The object-rolling environment requires the manipulation of small spherical objects into a goal position within the TCP coordinate frame.

Original: The agent must learn a policy to roll the objectfrom a random starting position to a random goal position.
Revised: The agent must learn a policy to roll the object from a random starting position to a random goal position.

Original: In this environment, a ﬂat tactile sensortip is used to simplify the motion needed to maintain contact with a spherical object.
Revised: In this environment, a ﬂat tactile sensor tip is used to simplify the motion needed to maintain contact with a spherical object.

Original: Object Pushing:
Revised: Object Pushing:

Original: In this task, the objective is to push an object to a goal location using the tactileimage, as considered in previous work using supervised learning techniques [24].
Revised: In this task, the objective is to push an object to a goal location using the tactile image, as considered in previous work using supervised learning techniques [25].

Original: Here we considera cube pushed along a randomly-generated trajectory.
Revised: Here we consider a cube pushed along a randomly-generated trajectory.

Original: The initial pose of the cube is randomisedwithin limits and the trajectory generated using OpenSimplex noise.
Revised: The initial pose of the cube is randomised within limits and the trajectory generated using OpenSimplex noise.

Original: Object Balancing: This task is analogous to the well-known 2D inverted pendulum problem [25],where an unstable pole with ﬂat base is balanced on the tip of a sensor that points upwards.
Revised: Object Balancing: This task is analogous to the well-known 2D inverted pendulum problem [26], where an unstable pole with ﬂat base is balanced on the tip of a sensor that points upwards.

Original: Arandom force perturbation is then applied to the object to cause instability.
Revised: A random force perturbation is then applied to the object to cause instability.

Original: The objective is to learna policy that applies planar actions to counteract the rotation of the balanced pole.
Revised: The objective is to learn a policy that applies planar actions to counteract the rotation of the balanced pole.

Original: 4.3 Reinforcement Learning Results
Revised: 4.3 Reinforcement Learning Results

Original: This work uses the Stable-Baselines-3 [26] implementation of PPO to train all tasks (further detailsin Appendix B).
Revised: This work uses the Stable-Baselines-3 [27] implementation of PPO to train all tasks (further details in Appendix B).

Original: For an initial comparison, we train all tasks using all available observation spaces,averaged over 3 seeds.
Revised: For an initial comparison, we train all tasks using all available observation spaces, averaged over 3 seeds.

Original: Training results are given in Figure 3, deterministic evaluation results andmore detailed plots are given in Appendix D.
Revised: Training results are given in Figure 2, deterministic evaluation results and more detailed plots are given in Appendix D.

Original: Environment state observations are used to verify that the environment leads to desired policies.
Revised: Environment state observations are used to verify that the environment leads to desired policies.

Original: As no image rendering is performed, this runs faster than tactile or visual observations; hence, itwas used to ﬁnd stable hyperparameters.
Revised: As no image rendering is performed, this runs faster than tactile or visual observations; hence, it was used to ﬁnd stable hyperparameters.

Original: Although comprised of ideal state information, we ﬁndthat tactile image observations can lead to more sample-efﬁcient and stable training, which is mostclearly visible in the Edge-Following and Object-Rolling environments (Figs 3a, 3c).
Revised: Although comprised of ideal state information, we ﬁnd that tactile image observations can lead to more sample-efﬁcient and stable training, which is most clearly visible in the Edge-Following and Object-Rolling environments (Figs 2a, 2c).

Original: Policies trained exclusively using visual observations tended to perform worst, which we attributeto most tasks being targeted towards tactile data; for example, details of the contact can be obscuredusing a single external camera.
Revised: Policies trained exclusively using visual observations tended to perform worst, which we attribute to most tasks being targeted towards tactile data; for example, details of the contact can be obscured using a single external camera.

Original: The best visual agent performance was for the object pushing task,which is the coarsest manipulation challenge.
Revised: The best visual agent performance was for the object pushing task, which is the coarsest manipulation challenge.

Original: Whilst the object rolling task appears to perform wellwith visual observations, there is a notable performance gap where visual polices do not accuratelylearn the desired behaviour.
Revised: Whilst the object rolling task appears to perform well with visual observations, there is a notable performance gap where visual polices do not accurately learn the desired behaviour.

Original: When combining visual and tactile observations, successful learning cantake place.
Revised: When combining visual and tactile observations, successful learning can take place.

Original: Despite the additional complexity in the observations, agents can learn well with notableimprovements in sample efﬁciency for the Edge-Follow and Object-Push tasks (Figs 3a, 3d).
Revised: Despite the additional complexity in the observations, agents can learn well with notable improvements in sample efﬁciency for the Edge-Follow and Object-Push tasks (Figs 2a, 2d).

Original:  Real-to-Sim Image Translation
Revised:  Real-to-Sim Image Translation

Original: An advantage of using tactile images as the main form of observation is that the image space is moresimplistic (e.g. markers on a uniform background) compared with the variety of visual images froma scene.
Revised: An advantage of using tactile images as the main form of observation is that the image space is more simplistic (e.g. markers on a uniform background) compared with the variety of visual images from a scene.

Original: Vision-based images are affected by features such as changing lighting conditions, shadowsand texture that usually constitute a superﬂuous level of detail when solving a task.
Revised: Vision-based images are affected by features such as changing lighting conditions, shadows and texture that usually constitute a superﬂuous level of detail when solving a task.

Original: Simulating thesevisual features can make the learning more difﬁcult because of the increased complexity of theobservations and the added computation to render these images; conversely, ignoring these visualfeatures would make image translation more difﬁcult because of the greater simulation and realitygap, as they are still prevalent in reality, necessitating techniques such as image randomisation or
Revised: Simulating these visual features can make the learning more difﬁcult because of the increased complexity of the observations and the added computation to render these images; conversely, ignoring these visual features would make image translation more difﬁcult because of the greater simulation and reality gap, as they are still prevalent in reality, necessitating techniques such as image randomisation or

Original: RL task-aware training [5, 20].
Revised: RL task-aware training [5, 21].

Original: These complexities are not present for the real and simulated tactileimages considered here because the camera is conﬁned within the enclosure of the tactile sensor.
Revised: These complexities are not present for the real and simulated tactile images considered here because the camera is conﬁned within the enclosure of the tactile sensor.

Original: Conversely, a disadvantage of using tactile images as the main observations is that the tasks requirethe robot to be in close proximity, or touching, the environment.
Revised: Conversely, a disadvantage of using tactile images as the main observations is that the tasks require the robot to be in close proximity, or touching, the environment.

Original: Whilst this contact may be necessary for some tasks, such as manipulation, it does make exploration in reality a challenge becauseof the self-inﬂicted damage that can occur.
Revised: Whilst this contact may be necessary for some tasks, such as manipulation, it does make exploration in reality a challenge because of the self-inﬂicted damage that can occur.

Original: The approach proposed here is to use a separate data- collection stage, where the tactile sensor explores a series of conﬁgurations in a safe and controlledmanner.
Revised: The approach proposed here is to use a separate datacollection stage, where the tactile sensor explores a series of conﬁgurations in a safe and controlled manner.

Original: As the tactile image space is relatively conﬁned, an efﬁcient exploration of a representativesample of conﬁgurations for a speciﬁc RL task becomes possible.
Revised: As the tactile image space is relatively conﬁned, an efﬁcient exploration of a representative sample of conﬁgurations for a speciﬁc RL task becomes possible.

Original: Whilst some sensor conﬁgurations are not possible to sample in reality, such as large penetrations of the sensor that would causedamage, we aim to train a model that generalises to those unreachable conﬁgurations.
Revised: Whilst some sensor conﬁgura- tions are not possible to sample in reality, such as large penetrations of the sensor that would cause damage, we aim to train a model that generalises to those unreachable conﬁgurations.

Original: Speciﬁcally, in this work we treat the sim-to-real problem as a supervised image-to-image translationproblem.
Revised: Speciﬁcally, in this work we treat the sim-to-real problem as a supervised image-to-image translation problem.

Original: The same data collection procedure is performed in both simulation and reality, producinga data set of real and simulated image pairs.
Revised: The same data collection procedure is performed in both simulation and reality, producing a data set of real and simulated image pairs.

Original: Here we choose to transfer from real-to-sim images,because the real images are richer in information with details such as shear forces that we choose notto model in simulation.
Revised: Here we choose to transfer from real-to-sim images, because the real images are richer in information with details such as shear forces that we choose not to model in simulation.

Original: Generative Adversarial Networks (GANs) are the state of the art for realisticimage generation.
Revised: Generative Adversarial Networks (GANs) are the state of the art for realistic image generation.

Original: Here we use the pix2pix [27] architecture for image-to-image translation, whichuses the U-net [28] architecture for the image conditioned generator and a standard convolutionalnetwork for the discriminator (Appendix E shows this architecture applied to tactile images).
Revised: Here we use the pix2pix [28] architecture for image-to-image translation, which uses the U-net [29] architecture for the image conditioned generator and a standard convolutional network for the discriminator (Appendix E shows this architecture applied to tactile images).

Original: 5.1 Data Collection
Revised: 5.1 Data Collection

Original: Three data sets corresponding to distinct environments are collected for sim-to-real transfer, eachwith 5000 image pairs for training and 2000 pairs for validation.
Revised: Three data sets corresponding to distinct environments are collected for sim-to-real transfer, each with 5000 image pairs for training and 2000 pairs for validation.

Original: We approximate the task environment with a simpliﬁed static environment consisting of 3d-printed components and no specialisthardware.
Revised: We approximate the task environment with a simpliﬁed static environment consisting of 3d-printed components and no specialist hardware.

Original: Per task, these datasets take ∼ 5 hours to collect on physical hardware and ∼ 100 secondsin simulation.
Revised: Per task, these datasets take ∼ 5 hours to collect on physical hardware and ∼ 100 seconds in simulation.

Original: Following a previous investigation of contact-induced shear with the same opticaltactile sensor [29], we deliberately induce shear perturbations by randomly sliding the sensor duringdata collection (details given in Appendix F).
Revised: Following a previous investigation of contact-induced shear with the same optical tactile sensor [30], we deliberately induce shear perturbations by randomly sliding the sensor during data collection (details given in Appendix F).

Original: Past work has shown this step is key to ensuring thatthe trained neural network outputs are insensitive to unavoidable motion-dependent shear duringtask performance (for more details, see [23]).
Revised: Past work has shown this step is key to ensuring that the trained neural network outputs are insensitive to unavoidable motion-dependent shear during task performance (for more details, see [24]).

Original: We do not model any sensor shear in simulation toensure that the real-to-sim image generation is also insensitive to shear on the real sensor.
Revised: We do not model any sensor shear in simulation to ensure that the real-to-sim image generation is also insensitive to shear on the real sensor.

Original: 5.2 Pix2Pix Training
Revised: 5.2 Pix2Pix Training

Original: The only change to the pix2pix architecture was to replace instance normalisation with spectralnormalisation, with other default parameters sufﬁcient for training on tactile images (full details in
Revised: The only change to the pix2pix architecture was to replace instance normalisation with spectral normalisation, with other default parameters sufﬁcient for training on tactile images (full details in

Original: Appendix E).
Revised: Appendix E).

Original: This change reduced droplet artefacts that were otherwise prevalent in our generatedimages.
Revised: This change reduced droplet artefacts that were otherwise prevalent in our generated images.

Original: In addition, excluding the border from the simulated tactile images improved training, asotherwise the GAN focused on generating a realistic border instead of the tactile imprint.
Revised: In addition, excluding the border from the simulated tactile images improved training, as otherwise the GAN focused on generating a realistic border instead of the tactile imprint.

Original: This isimportant because the imprint is the useful component when learning control policies.
Revised: This is important because the imprint is the useful component when learning control policies.

Original: Instead, theborder is re -added from a saved reference image after generation occurs.
Revised: Instead, the border is re-added from a saved reference image after generation occurs.

Original: Accurate tactile image generation is achieved with minimal difference between generated and targetimages (example images for each data set shown in Appendix F, Figure 11).
Revised: Accurate tactile image generation is achieved with minimal difference between generated and target images (example images for each data set shown in Appendix F, Figure 11).

Original: For the edge, surfaceand probe datasets, mean SSIM scores across the full validation set are 0 . 9955 , 0 .
Revised: For the edge, surface and probe datasets, mean SSIM scores across the full validation set are 0 . 9955 , 0 .

Original: 9924 and 0 . 9942respectively.
Revised: 9924 and 0 .respectively.

Original: The source of the small errors are highlighted in the shown SSIM image differences(Figure 11, right).
Revised: The source of the small errors are highlighted in the shown SSIM image differences (Figure 11, right).

Original: The imprint borders in the generated images lack some sharp details generated insimulation, which will likely be due to a slight elastic stretching of the skin of the real tip that is notmodelled in simulation.
Revised: The imprint borders in the generated images lack some sharp details generated in simulation, which will likely be due to a slight elastic stretching of the skin of the real tip that is not modelled in simulation.

Original: An approach such as Gaussian smoothing of the simulated images, as in[12], could reduce this effect, although we did not ﬁnd it necessary to explore in the present work.
Revised: An approach such as Gaussian smoothing of the simulated images, as in [14], could reduce this effect, although we did not ﬁnd it necessary to explore in the present work.

Original: Crucially, the generator can interpret and generate image features useful to training RL policies.
Revised: Crucially, the generator can interpret and generate image features useful to training RL policies.

Original: Forexample, edge orientation and position are accurately captured in the GAN.
Revised: For example, edge orientation and position are accurately captured in the GAN.

Original: Also, the generator cangeneralise to images unseen during training.
Revised: Also, the generator can generalise to images unseen during training.

Original: For example, all training data only had imprints fromone source in each training image; however, during inference, multiple sources can be applied andthe generator still produces realistic outputs (video demonstrations available here ).
Revised: For example, all training data only had imprints from one source in each training image; however, during inference, multiple sources can be applied and the generator still produces realistic outputs (video demonstrations available here ).

Original: Similarly, thegenerator appears to generalise outside the training data for penetration depth, which is importantfor helping to ensure that the sensor can avoid damage by not pressing too far into a surface.
Revised: Similarly, the generator appears to generalise outside the training data for penetration depth, which is important for helping to ensure that the sensor can avoid damage by not pressing too far into a surface.

Original: 5.3 Supervised Learning Comparison
Revised: 5.3 Supervised Learning Comparison

Original: For comparison to the only existing sim-to-real approach for the TacTip [11], we implement a supervised learning task to predict the radial displacement and polar angle of an edge pressed into thesensor (results in Appendix G).
Revised: For comparison to the only existing sim-to-real approach for the TacTip [11], we implement a supervised learning task to predict the radial displacement and polar angle of an edge pressed into the sensor (results in Appendix G).

Original: Domain adaptation with our proposed method of tactile simulationout-performs the previous approach using elastic deformation simulation and domain randomisation,with approximately a 3-fold improvement in Mean Absolute Error (MAE) of the predictions.
Revised: Domain adaptation with our proposed method of tactile simulation out-performs the previous approach using elastic deformation simulation and domain randomisation, with approximately a 3-fold improvement in Mean Absolute Error (MAE) of the predictions.

Original: Real
Revised: Real

Original: Sim
Revised: Sim

Original: Real
Revised: Real

Original: Sim
Revised: Sim

Original: Real
Revised: Real

Original: Sim
Revised: Sim

Original: Real
Revised: Real

Original: Sim
Revised: Sim

Original:  Results: Sim-to-Real Policy Transfer
Revised:  Results: Sim-to-Real Policy Transfer

Original: Successful sim-to-real policy transfer was achieved on 4 of the 5 proposed tasks.
Revised: Successful sim-to-real policy transfer was achieved on 4 of the 5 proposed tasks.

Original: Qualitative demonstration videos are available at https://sites.google.com/view/tactile-sim-to-real
Revised: Qualitative demonstration videos are available at https://sites.google.com/my.bristol.ac.uk/tactile-gym-sim2real/home .

Original: [Table]
Revised: [Table]

Original: Edge Following: The learned edge-following behaviouris evaluated over several novel ﬂat shapes in the environment (Figure 4).
Revised: Edge Following: The learned edge-following behaviour is evaluated over several novel ﬂat shapes in the environment (Figure 3).

Original: The learned policy successfully traversesall objects, generalising to novel features such as curvededges and sharp internal or external corners not presentduring training.
Revised: The learned policy successfully traverses all objects, generalising to novel features such as curved edges and sharp internal or external corners not present during training.

Original: As these shapes are 3D printed, we haveaccess to the ground truth in the CAD ﬁles (accurate tothe precision of the 3D printer).
Revised: As these shapes are 3D printed, we have access to the ground truth in the CAD ﬁles (accurate to the precision of the 3D printer).

Original: We compare the trajectories taken in simulation to those in reality,reporting the distance from each point of the trajectory to the nearest ground truth node (Table 1).
Revised: We compare the trajectories taken in simulation to those in reality, reporting the distance from each point of the trajectory to the nearest ground truth node (Table 1)

Original: The results show that the sensor maintains close proximity to the target edge.
Revised: The results show that the sensor maintains close proximity to the target edge.

Original: The drop in performance on the physical robot seems related to an oscillating behaviour also presentin simulation.
Revised: The drop in performance on the physical robot seems related to an oscillating behaviour also present in simulation.

Original: The movements are exaggerated in reality due to increased latency between capturingan observation and predicting an action.
Revised: The movements are exaggerated in reality due to increased latency between capturing an observation and predicting an action.

Original: A likely cause is the action-range clipping used in the PPOalgorithm, which results in the action extremes being more prevalent.
Revised: A likely cause is the action-range clipping used in the PPO algorithm, which results in the action extremes being more prevalent.

Original: Approaches such as squashingfunctions [30], beta distributions [31] or constraint-based RL [32] could mitigate this artefact.[Table]
Revised: Approaches such as squashing functions [31], beta distributions [32] or constraint-based RL [33] could mitigate this artefact.

Original: [Table]
Revised: [Table]

Original: Surface
Revised: Surface

Original: Following: To evaluate the surface-followingbehaviour, a spherical object is traversed from its centre outwards in a set direction ranging over 360 ° in 45 °intervals (Figure 5).
Revised: Following: To evaluate the surface-following behaviour, a spherical object is traversed from its centre outwards in a set direction ranging over 360 ° in 45 ° intervals (Figure 4).

Original: As with the edge following task, weuse a 3D-printed shape whose ground truth is known fromthe CAD model.
Revised: As with the edge following task, we use a 3D-printed shape whose ground truth is known from the CAD model.

Original: The sensor successfully and accuratelytraverses the object on the physical robot despite texture and frictional forces not modelled in simulation.
Revised: The sensor successfully and accurately traverses the object on the physical robot despite texture and frictional forces not modelled in simulation.

Original: Like the edge following task, the accuracy drops between sim and real, although the evaluatedpolicies still exhibit the desired behaviour.
Revised: Like the edge following task, the accuracy drops between sim and real, although the evaluated policies still exhibit the desired behaviour.

Original: A more extensive qualitative test is given in the supplementary video for the undulating surface shown in Fig.1.
Revised: A more extensive qualitative test is given in the supplementary video for the undulating surface shown in Fig.1.

Original: Object Rolling:
Revised: Object Rolling:

Original: The object-rolling behaviour is tested by manipulating ball bearings (on an extended dimension range of 2 - 8mm diameter) from random initial positions to random goal locations.
Revised: The object-rolling behaviour is tested by manipulating ball bearings (on an extended dimension range of 2 - 8mm diameter) from random initial positions to random goal locations.

Original: During the evaluation, the position of the ball bearing relative to the sensor is obscured, so the imprint of the object is tracked on the tactile image using basic computer vision(blob detection).
Revised: During the evaluation, the position of the ball bearing relative to the sensor is obscured, so the imprint of the object is tracked on the tactile image using basic computer vision (blob detection).

Original: The target is reached when the pixel distance is less than a 5-pixel threshold( ∼ 2 mm ).
Revised: The target is reached when the pixel distance is less than a 5-pixel threshold ( ∼ 2 mm ).

Original: On the physical robot, this is performed for 25 trajectories with each bearing, resulting in 100 consecutive successful trials (subset shown in Figure 6).
Revised: On the physical robot, this is performed for 25 trajectories with each bearing, resulting in 100 consecutive successful trials (subset shown in Figure 5).

Original: To compare with simulated results, we apply the same initial and target positions then perform the same pixel tracking.
Revised: To compare with simulated results, we apply the same initial and target positions then perform the same pixel tracking

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: The sim and real results are similar, albeit withgreater noise in the real trajectories.
Revised: The sim and real results are similar, albeit with greater noise in the real trajectories.

Original: Object Pushing: To evaluate this task wetracked the objects following the method givenin [24].
Revised: Object Pushing: To evaluate this task we tracked the objects following the method given in [25].

Original: Importantly Aruco markers were onlyused for obtaining quantitative results and notas part of the observation.
Revised: Importantly Aruco markers were only used for obtaining quantitative results and not as part of the observation.

Original: This task was themost challenging to achieve accurate perfor- mance, likely due to the approximations of the simulated sensor tip, object being manipulated, andfrictional interactions.
Revised: This task was the most challenging to achieve accurate perfor- mance, likely due to the approximations of the simulated sensor tip, object being manipulated, and frictional interactions.

Original: Despite this, when evaluated in reality, task success was achieved acrossseveral trajectories with objects of different shape to that seen during training (examples in Figure7).
Revised: Despite this, when evaluated in reality, task success was achieved across several trajectories with objects of different shape to that seen during training (examples in Figure 6).

Original: Performance both in simulation and reality was sensitive to initial conditions and perturbations,with compounding errors causing the object to veer off the trajectory.
Revised: Performance both in simulation and reality was sensitive to initial conditions and perturbations, with compounding errors causing the object to veer off the trajectory.

Original: Future work could improverobustness by learning under additional dynamics randomisation, introduction of random force perturbations or by better matching the physical properties of the real objects.
Revised: Future work could improve robustness by learning under additional dynamics randomisation, introduction of random force perturbations or by better matching the physical properties of the real objects.

Original: Object Balancing: The physical task remains an outstanding challenge, because the successfultraining in simulation required physics parameters outside the capabilities of our UR5 robot.
Revised: Object Balancing: The physical task remains an outstanding challenge, because the successful training in simulation required physics parameters outside the capabilities of our UR5 robot.

Original:  Discussion and Future Work
Revised:  Discussion and Future Work

Original: In this work, we demonstrated that zero-shot sim-to-real policy transfer is a viable approach fortactile-based RL agents.
Revised: In this work, we demonstrated that zero-shot sim-to-real policy transfer is a viable approach for tactile-based RL agents.

Original: To learn the policy, we created a fast method for simulating tactile imagesbased on contact geometry.
Revised: To learn the policy, we created a fast method for simulating tactile images based on contact geometry.

Original: From these images, distinct policies were learned for several physicallyinteractive tasks requiring a sense of touch.
Revised: From these images, distinct policies were learned for several physicallyinteractive tasks requiring a sense of touch.

Original: We demonstrated that a data-driven model for real-to-simimage translation can be embedded into the control loop for successful sim-to-real policy transfer.
Revised: We demonstrated that a data-driven model for real-to-sim image translation can be embedded into the control loop for successful sim-to-real policy transfer.

Original: There are several future directions for extending and improving this approach.
Revised: There are several future directions for extending and improving this approach.

Original: The tactile simulation of both the contact dynamics and captured information in the model couldbe improved.
Revised: The tactile simulation of both the contact dynamics and captured information in the model could be improved.

Original: Our contact dynamics model used used rigid bodies with soft contacts as an approximation to the deformation of real tactile sensors.
Revised: Our contact dynamics model used used rigid bodies with soft contacts as an approximation to the deformation of real tactile sensors.

Original: Extending this work with soft-body deformationshould enable a more realistic simulation, which was not pursued in this initial study because ofincreased computational costs and instability issues with the simulators.
Revised: Extending this work with soft-body deformation should enable a more realistic simulation, which was not pursued in this initial study because of increased computational costs and instability issues with the simulators.

Original: The captured informationin the present study focused exclusively on contact geometry, which in principle could be extendedto include global shear information through contact data available in most physics simulators.
Revised: The captured information in the present study focused exclusively on contact geometry, which in principle could be extended to include global shear information through contact data available in most physics simulators.

Original: Texture could be also included using rendering techniques common in photo-realistic image generation.
Revised: Texture could be also included using rendering techniques common in photo-realistic image generation.

Original: Local shear of the sensor would be challenging to simulate, for example during incipient slip, dueto the contact reduction commonly used in physics engines for computational efﬁciency; however,this capability may be possible in soft-body simulations where these local forces are required.
Revised: Local shear of the sensor would be challenging to simulate, for example during incipient slip, due to the contact reduction commonly used in physics engines for computational efﬁciency; however, this capability may be possible in soft-body simulations where these local forces are required.

Original: For translating real to simulated images, improvement could be made to both the GAN and trainingdata.
Revised: For translating real to simulated images, improvement could be made to both the GAN and training data.

Original: We used a conventional pix2pix [27] approach, which can be improved with extended methods[33].
Revised: We used a conventional pix2pix [28] approach, which can be improved with extended methods [34].

Original: In addition, a distinct data set was collected for training the GAN in each learning environment.
Revised: In addition, a distinct data set was collected for training the GAN in each learning environment.

Original: Because of the constrained nature of the tactile image space, a general tactile data set couldpotentially be used train a single image-translation model for policy transfer over multiple tasks.
Revised: Because of the constrained nature of the tactile image space, a general tactile data set could potentially be used train a single image-translation model for policy transfer over multiple tasks.

Original: The environments considered here focused on skills achievable with only a single sensor and wherethe optimal behaviour is obvious for a human to interpret, as needed to verify that RL with a sim-toreal approach is a viable method.
Revised: The environments considered here focused on skills achievable with only a single sensor and where the optimal behaviour is obvious for a human to interpret, as needed to verify that RL with a sim-toreal approach is a viable method.

Original: Future work could focus on tasks where ideal control policies aremore complicated or unknown, so the RL framework can be fully exploited.
Revised: Future work could focus on tasks where ideal control policies are more complicated or unknown, so the RL framework can be fully exploited.

Original: An interesting topicwould be to extend these methods to prehensile tasks such as grasping and dexterous manipulation,where tactile information will be valuable for learning desirable and robust policies.
Revised: An interesting topic would be to extend these methods to prehensile tasks such as grasping and dexterous manipulation, where tactile information will be valuable for learning desirable and robust policies.

Original: Although onlyone sensor was used in this current work, our tactile simulation does support multiple sensors, offering the opportunity to extend to more complex tasks involving hands with multiple tactile ﬁngertips.A Robotics SimulationA.1 Physics Engine
Revised: Although only one sensor was used in this current work, our tactile simulation does support multiple sensors, offering the opportunity to extend to more complex tasks involving hands with multiple tactile ﬁngertips.

Original: Physics-based simulation has played a vital role in the ﬁeld of robotics, enabling rapid prototyping,testing and experimentation.

Original: The accuracy and speed of simulation has scaled with available computation.

Original: Modern methods such as RL have leveraged this to simulate very large data sets that areotherwise impractical to collect from the real world, hence far more complex and general behaviourscan be learnt.

Original: The majority of simulated robotics has focused on rigid-body dynamics and visionsensors.

Original: More recently, a range of speciﬁc environment suites have been introduced to bring simulation closer to reality or to facilitate research in an under-represented direction [34, 35, 36, 37, 38].

Original: Collectively, these works highlight the importance of increasing the breadth and capabilities of simulation software available to researchers.

Original: There are many choices for physics engines when developing a new suite.

Original: Here we choose the

Original: PyBullet [39] for its fast GPU rendering, support for deformable objects [40], fast and reliable kinematics and dynamics solvers, and its demonstrated sim-to-real success in robotics [5].

Original: Importantly,

Original: PyBullet is open source and non-commercialised software which helps to improve accessibility andlower the barrier of entry for RL research.

Original: That said, the tools used here to build our tactile suite areavailable in other physics engines such as Mujoco, Gazebo, Nvidia-isaacgym and Unity ML agents.

Original: A.2 Control

Original: Throughout this work we use Cartesian velocity control where the control input is a desired velocity(twist) speciﬁed by a 6 DoF action constrained to the allowed modes of control of the task.

Original: A controlrate speciﬁes the frequency that new actions can be sent to the robot, with maximum velocity limitsalso imposed.

Original: For most of our experiments, we set a velocity control rate of 10 Hz, a maximumlinear velocity of 10 mm s − 1 and a maximum angular velocity of 5 ° s − 1 . When undergoing a seriesof predeﬁned and random actions we notice no signiﬁcant difference in Tool Centre Point (TCP)pose between simulation and the real robot. Work-space coordinate frames are set in both simulation and reality, speciﬁc to each environment,with each action sent to the robot consisting of a Cartesian move relative to the work frame. [Table]
Revised: Acknowledgements

Original: Therefore, the learned policies can be transferred fromsim-to-real without exactly replicating the simulated task;for example edges and surfaces can be placed in alternative locations provided the work frame is set correctly.

Original: As a consequence, the policy transfers even when thereare notable differences in the simulation, such as mirroredarm conﬁgurations used in this work.

Original: Thus, in principlethe policies could also be transferred to other robot arms,providing the same speed and frequency of control can beachieved.

Original: Near-default hyper-parameters are used in all training(full list in Table 4). Image-based observations use the Atari Nature [41] convolutional layers followed by two256-node fully connected (FC) layers.
Revised: Alex Church was supported by an EPSRC CASE award sponsored by Google DeepMind.

Original: State observationsuse only the FC layers.

Original: For tasks that require both image and state data, the state data is passed through two64-node FC layers and the output concatenated with theﬂattened output of the convolutional layers, which is thenpassed through the ﬁnal FC layers for action and valueprediction.

Original: The convolutional weights are shared for allpolicy and value networks.

Original: Small random image translation augmentations help to improve performance and stabilise training, as proposed in [42, 43].

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: John

Original: D Full Reinforcement Learning Results

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: a t

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: F Image Translation Data Collection

Original: [Figure]

Original: For the edge-following environment, we collecttactile images pressed onto a straight edge, varying the orientations, radial displacements and penetration of the sensor.

Original: A hemispherical sensor tipis used with the tool center point (TCP) locatedcentrally at the end of the sensor.

Original: Relative to the

Original: TCP, the data is gathered over ranges: orientation Rz ∈ [ − 179 ° , 180 ° ] , radial displacement y ∈[ − 6 , 6]mm , and penetration z ∈ [3 . 5 , 5 . 5]mm .

Original: For the surface-following and object-pushing environments, we collect tactile images pressed ontoa ﬂat surface, varying the orientations and penetration, also with a hemispherical tip.

Original: Relative tothe TCP, data is gathered over ranges: orientation{ Rx, Ry } ∈ [ − 15 ° , 15 ° ] , and penetration z ∈ [2 , 5]mm .

Original: For the object-rolling environment, we collect tactile images pressed onto a spherical probe stimulus,using a ﬂat sensor tip appropriate to this environment.

Original: 9 spherical probe stimuli are used, rangingover [2 , 6]mm radius in 0 .

Original: 5mm increments.

Original: The sensor is positioned to contact the probe at randomplacements within a 15mm disk surrounding the centre of the tip.

Original: In this case, shear is not introducedinto the data collection because rolling objects induce negligible motion-dependent shear.

Original: F.1 Full SSIM Scores

Original: We measure the SSIM scores across the validation sets collected for each task, each consisting of2000 image pairs.

Original: For the Edge, Surface and Probe datasets respectively, we found mean scores of[0 .

Original: 995 , 0 . 992 , 0 . 994] , min scores of [0 . 985 , 0 . 982 , 0 . 974] , and max scores of [0 . 999 , 0 . 999 , 0 . 999] .

Original: Whilst high scores are expected due to the relatively sparse target images, this indicates strongperformance in all cases.

Original: G Supervised Learning Comparison

Original: [Table]

Original: [Table]

Original: Whilst we were unable to ﬁnd a sim-to-real re-

Original: [Table]

Original: inforcement learning baseline that does not

Original: re-

Original: [Table]

Original: quire speciﬁc hardware, we can compare to pre-

Original: [Table]

Original: vious work simulating the TacTip sensor for su-

Original: [Table] pervised learning tasks. Ding et al. [11] used
Revised: Lloyd and Nathan Lepora were supported by a Research Leadership Award from the Leverhulme

Original: [Table] an elastic deformation approach to simulate the [Table] TacTip sensor.
Revised: Trust on ‘A biomimetic forebrain for robot touch’ (RL-2016-39).

Original: They focussed on supervised

Original: [Table]

Original: learning from simulated pin positions rather

Original: [Table]

Original: than tactile images, and could accurately pre-

Original: [Table]

Original: dict edge position and orientation, and the ( x, y ) position of a pole pressed into the sensor.

Original: In thissection we draw a comparison between the presented method on their supervised learning tasks.

Original: The tasks are deﬁned as follows:

Original: Task I predicts polar angle θ of an edge pressed into the sensor;

Original: Task II predicts radial displacement r of an edge pressed into the sensor; and Task III predicts the( x, y ) location of the centre of a pole pressed into the sensor.

Original: More details are provided in [11] .

Original: As this is not presented as a standardised benchmark, and we do not have access to the exact setupused, there are some differences in how these tasks have been carried out between the present work

Original: [Table]

Original: and [11].

Original: In particular, we differ notably in Task III, where we use a ﬂat TacTip sensor tip insteadof the original hemispherical tip, because we are using the data, GANs and objects acquired for theobject rolling task which needed a ﬂat tip.

Original: That said, we expect there will only be small differences

Original: [Table]

Original: in accuracy between ﬂat and curved TacTip tips.

Original: For each task we ﬁrst train a convolutional neural network to predict r , θ , x and y using a datasetof 10,000 simulated tactile images per task (taking ∼ 145 seconds to collect).

Original: We then collect adataset of 2000 real sensor images and using GANs trained on edge or probe data, we translate fromreal-to-sim images to create the test datasets of generated images.

Original: The performance of the trainednetworks over the full generated datasets is then measured and compared with the results from [11].

Original: As our simulated tactile data is comprised of images, we use a convolutional neural network (CNN).

Original: To avoid boundary issues over full rotations, we predict a sine/cosine encoding of angle.

Original: The CNNhas 4 convolutional layers (each with kernel: 5, stride: 1, padding: 2, maxpool: 2) and 3 fullyconnected layers (dimensions = 1024, 512, output dim).

Original: Batch normalization is applied only to theconvolutional layers before the ELU [44] activation.

Original: Images of resolution 256 × 256 -pixels wereused.

Original: Table 11 shows a large reduction in Mean Absolute Error (MAE) for both radial displacement andangle prediction in comparison with previous work simulating the TacTip [11].

Original: A 3-fold improvement is found in predicting the angle θ (Task I) and a near 4-fold improvement when predictingradial distance (Task II).

Original: We also ﬁnd a large decrease in MAE from 0.73 mm to 0.059 mm forposition prediction (Task III), although as mentioned above there are some differences in setup.