Original: O BJECT F OLDER : A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Revised: O BJECT F OLDER : A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: Multisensory object-centric perception, reasoning, and interaction havebeen a key research topic in recent years.
Revised: Abstract: Multisensory object-centric perception, reasoning, and interaction havebeen a key research topic in recent years.

Original: However, the progress in these directionsis limited by the small set of objects available—synthetic objects are not realistic enough and are mostly centered around geometry, while real object datasetssuch as YCB are often practically challenging and unstable to acquire due to international shipping, inventory, and ﬁnancial cost.
Revised: However, the progress in these directionsis limited by the small set of objects available—synthetic objects are not realistic enough and are mostly centered around geometry, while real object datasetssuch as YCB are often practically challenging and unstable to acquire due to international shipping, inventory, and ﬁnancial cost.

Original: We present O BJECT F OLDER ,a dataset of 100 virtualized objects that addresses both challenges with two keyinnovations.
Revised: We present O BJECT F OLDER ,a dataset of 100 virtualized objects that addresses both challenges with two keyinnovations.

Original: First, O BJECT F OLDER encodes the visual, auditory, and tactile sensory data for all objects, enabling a number of multisensory object recognitiontasks, beyond existing datasets that focus purely on object geometry.
Revised: First, O BJECT F OLDER encodes the visual, auditory, and tactile sensory data for all objects, enabling a number of multisensory object recognitiontasks, beyond existing datasets that focus purely on object geometry.

Original: Second,
Revised: Second,

Original: O BJECT F OLDER employs a uniform, object-centric, and implicit representationfor each object’s visual textures, acoustic simulations, and tactile readings, making the dataset ﬂexible to use and easy to share.
Revised: O BJECT F OLDER employs a uniform, object-centric, and implicit representationfor each object’s visual textures, acoustic simulations, and tactile readings, making the dataset ﬂexible to use and easy to share.

Original: We demonstrate the usefulnessof our dataset as a testbed for multisensory perception and control by evaluatingit on a variety of benchmark tasks, including instance recognition, cross-sensoryretrieval, 3D reconstruction, and robotic grasping.
Revised: We demonstrate the usefulnessof our dataset as a testbed for multisensory perception and control by evaluatingit on a variety of benchmark tasks, including instance recognition, cross-sensoryretrieval, 3D reconstruction, and robotic grasping.

Original: [Figure]
Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: We perceive the world not as a single giant entity but often through a series of inanimate objects,which exist as bounded wholes and move on connected paths.
Revised: We perceive the world not as a single giant entity but often through a series of inanimate objects,which exist as bounded wholes and move on connected paths.

Original: We interact with these objects throughan array of different sensory systems–vision, audition, touch, smell, taste, and proprioception.
Revised: We interact with these objects throughan array of different sensory systems–vision, audition, touch, smell, taste, and proprioception.

Original: Thesemultisensory inputs shape our daily experiences: we observe the surroundings to avoid obstacles,hear the doorbell ring to realize the guests have arrived, and touch the fabric to sense its degreeof comfort.
Revised: Thesemultisensory inputs shape our daily experiences: we observe the surroundings to avoid obstacles,hear the doorbell ring to realize the guests have arrived, and touch the fabric to sense its degreeof comfort.

Original: Cognitive science studies [1, 2] show that both object representation and multisensoryperception play a crucial role in human cognitive development.
Revised: Cognitive science studies [1, 2] show that both object representation and multisensoryperception play a crucial role in human cognitive development.

Original: Object-centric learning has shown great potential lately in visual reasoning [3], representing physical scenes [4], modeling multi-agent interactions [5], and improving generalization to unseen object compositions [6].
Revised: Object-centric learning has shown great potential lately in visual reasoning [3], representing physical scenes [4], modeling multi-agent interactions [5], and improving generalization to unseen object compositions [6].

Original: These prior methods, however, often do not model the full spectrum of physicalobject properties and sensory modes, including their 3D shape, texture, material, sound, and feel.
Revised: These prior methods, however, often do not model the full spectrum of physicalobject properties and sensory modes, including their 3D shape, texture, material, sound, and feel.

Original: While there has been signiﬁcant progress by “looking”—recognizing objects based on glimpses oftheir visual appearance or 3D shape—objects in the world are often modeled as silent and untouchable entities.
Revised: While there has been signiﬁcant progress by “looking”—recognizing objects based on glimpses oftheir visual appearance or 3D shape—objects in the world are often modeled as silent and untouchable entities.

Original: We identify that this is due to the lack of rich object datasets.
Revised: We identify that this is due to the lack of rich object datasets.

Original: Existing object datasets have contributed to most signiﬁcant progress in image recognition [7, 8] and shape modeling [9, 10], but othersensory information is minimally considered.
Revised: Existing object datasets have contributed to most signiﬁcant progress in image recognition [7, 8] and shape modeling [9, 10], but othersensory information is minimally considered.

Original: Datasets for robotic manipulation research [11, 12]consist of a selected number of real-world household objects, but these objects are expensive andoften unstable to acquire due to international shipping and inventory.
Revised: Datasets for robotic manipulation research [11, 12]consist of a selected number of real-world household objects, but these objects are expensive andoften unstable to acquire due to international shipping and inventory.

Original: Moreover, it is nontrivial tovirtualize these objects and their multiple sensory modes, limiting their usability for developingembodied AI agents [13, 14, 15, 16, 17, 18, 19, 20].
Revised: Moreover, it is nontrivial tovirtualize these objects and their multiple sensory modes, limiting their usability for developingembodied AI agents [13, 14, 15, 16, 17, 18, 19, 20].

Original: Our goal is therefore to create a dataset of 3D objects that are 1) easily accessible to the communityas a standard benchmark, 2) high-quality in terms of visual textures, and 3) augmented with realisticauditory and tactile sensory data.
Revised: Our goal is therefore to create a dataset of 3D objects that are 1) easily accessible to the communityas a standard benchmark, 2) high-quality in terms of visual textures, and 3) augmented with realisticauditory and tactile sensory data.

Original: Towards this end, we introduce O BJECT F OLDER —a multisensorydataset of implicitly represented Object Files .
Revised: Towards this end, we introduce O BJECT F OLDER —a multisensorydataset of implicitly represented Object Files .

Original: The concept of an object ﬁle can be traced backto Kahneman et al.
Revised: The concept of an object ﬁle can be traced backto Kahneman et al.

Original: [21], where it is deﬁned as the sensory information that has been received aboutthe object at a particular location.
Revised: [21], where it is deﬁned as the sensory information that has been received aboutthe object at a particular location.

Original: In a similar spirit, we model and represent each object (or itsintrinsics) using an implicit neural representation, which through querying with extrinsic parameterswe can obtain images of the object from different viewpoints, impact sounds of the object at eachposition, and tactile sensing of the object at every surface location.
Revised: In a similar spirit, we model and represent each object (or itsintrinsics) using an implicit neural representation, which through querying with extrinsic parameterswe can obtain images of the object from different viewpoints, impact sounds of the object at eachposition, and tactile sensing of the object at every surface location.

Original: These different sensory data ofvision, audio, and touch can be regarded as contemporary multisensory object ﬁles.
Revised: These different sensory data ofvision, audio, and touch can be regarded as contemporary multisensory object ﬁles.

Original: Speciﬁcally, we collect 100 high quality 3D objects from online repositories.
Revised: Speciﬁcally, we collect 100 high quality 3D objects from online repositories.

Original: For each object, wemodel its visual appearance at random object pose, lighting condition, and camera viewpoint; weperform modal analysis [22] based on its shape, size, and material type to calculate its characteristic vibration modes for audio simulation; and we simulate its touch readings at all surface locationsusing DIGIT [23, 24]–a vision-based tactile sensor.
Revised: For each object, wemodel its visual appearance at random object pose, lighting condition, and camera viewpoint; weperform modal analysis [22] based on its shape, size, and material type to calculate its characteristic vibration modes for audio simulation; and we simulate its touch readings at all surface locationsusing DIGIT [23, 24]–a vision-based tactile sensor.

Original: Based on recent success on neural implicit representations [25, 26], we design a deep neural network that consists of three sub-networks–VisionNet,
Revised: Based on recent success on neural implicit representations [25, 26], we design a deep neural network that consists of three sub-networks–VisionNet,

Original: AudioNet, and TouchNet, which encode the visual, auditory, and tactile sensory data for each object,respectively.
Revised: AudioNet, and TouchNet, which encode the visual, auditory, and tactile sensory data for each object,respectively.

Original: See Fig.
Revised: See Fig.

Original: Together, they constitute an Object File that contains the complete multisensory proﬁle for each object instance.
Revised: Together, they constitute an Object File that contains the complete multisensory proﬁle for each object instance.

Original: Furthermore, we demonstrate the usefulness of our dataseton four benchmark tasks leveraging multisensory data, including instance recognition, cross-modalretrieval, 3D reconstruction, and robotic grasping.
Revised: Furthermore, we demonstrate the usefulness of our dataseton four benchmark tasks leveraging multisensory data, including instance recognition, cross-modalretrieval, 3D reconstruction, and robotic grasping.

Original: Our main contributions are threefold: First, we introduce O BJECT F OLDER , a dataset that makesmultisensory learning with vision, audio, and touch easily accessible to the research community;
Revised: Our main contributions are threefold: First, we introduce O BJECT F OLDER , a dataset that makesmultisensory learning with vision, audio, and touch easily accessible to the research community;

Original: Second, all objects in our dataset are compatible to different robotic virtual environments and will bemade publicly available as a standard testbed for robotic multisensory learning; Third, we evaluateon a suite of benchmark tasks that require multisensory data to facilitate future work in this direction.
Revised: Second, all objects in our dataset are compatible to different robotic virtual environments and will bemade publicly available as a standard testbed for robotic multisensory learning; Third, we evaluateon a suite of benchmark tasks that require multisensory data to facilitate future work in this direction.

Original:  Related Work
Revised:  Related Work

Original: Object-Centric Datasets Many image datasets exist for object recognition, such as ImageNet [7],
Revised: Object-Centric Datasets Many image datasets exist for object recognition, such as ImageNet [7],

Original: MS COCO [27], ObjectNet [8], and OpenImages [28].
Revised: MS COCO [27], ObjectNet [8], and OpenImages [28].

Original: These datasets consist of 2D images, while
Revised: These datasets consist of 2D images, while

Original: O BJECT F OLDER is a dataset of 3D objects.
Revised: O BJECT F OLDER is a dataset of 3D objects.

Original: ModelNet [9] and ShapeNet [10] are two large-scaledatasets of 3D models, but the collected 3D CAD models either contain no or low-quality visualtextures, making them unsuitable for real-world applications.
Revised: ModelNet [9] and ShapeNet [10] are two large-scaledatasets of 3D models, but the collected 3D CAD models either contain no or low-quality visualtextures, making them unsuitable for real-world applications.

Original: For research on object manipulation,datasets are constructed using real-world household objects such as YCB [11] and BigBIRD [12].
Revised: For research on object manipulation,datasets are constructed using real-world household objects such as YCB [11] and BigBIRD [12].

Original: Different from them, our goal is to construct a dataset of 3D objects, where realistic visual, auditory,and tactile sensory data can be easily obtained for each object in virtual environments.
Revised: Different from them, our goal is to construct a dataset of 3D objects, where realistic visual, auditory,and tactile sensory data can be easily obtained for each object in virtual environments.

Original: Implicit Neural Representations Coordinate-based multi-layer perceptions (MLP) have recentlybeen adopted as continuous, memory-efﬁcient implicit representations for a variety of visual signals such as 3D shape [29, 30], scenes [31, 32], and object appearance [25, 26] with the help ofclassic volume rendering techniques [33].
Revised: Implicit Neural Representations Coordinate-based multi-layer perceptions (MLP) have recentlybeen adopted as continuous, memory-efﬁcient implicit representations for a variety of visual signals such as 3D shape [29, 30], scenes [31, 32], and object appearance [25, 26] with the help ofclassic volume rendering techniques [33].

Original: We also use MLP as a compact neural representation torepresent each 3D object.
Revised: We also use MLP as a compact neural representation torepresent each 3D object.

Original: Differently, apart from visual appearance, our mutisensory implicit neuralrepresentation also encodes object-centric auditory and tactile data of the object.
Revised: Differently, apart from visual appearance, our mutisensory implicit neuralrepresentation also encodes object-centric auditory and tactile data of the object.

Original: Multisensory Perception for Robotics Recent work shows growing interests in using audio ortouch in conjunction with vision for robotic tasks.
Revised: Multisensory Perception for Robotics Recent work shows growing interests in using audio ortouch in conjunction with vision for robotic tasks.

Original: Audio is used to recognize object instances [34] or terrain types [35]; estimate the ﬂow and amount of granular materials [36] or liquid height [37] in robotic scooping and pouring tasks; study its synergies with the motion of a robot [38]; andbridge the simulation-to-reality gap for tasks that involve stochastic dynamics [39].
Revised: Audio is used to recognize object instances [34] or terrain types [35]; estimate the ﬂow and amount of granular materials [36] or liquid height [37] in robotic scooping and pouring tasks; study its synergies with the motion of a robot [38]; andbridge the simulation-to-reality gap for tasks that involve stochastic dynamics [39].

Original: Tactile sensingis more local compared to vision and audio, and it is shown to improve sample efﬁciency on a peginsertion task [40]; beneﬁt robotic grasping [41, 42, 43]; augment vision to enhance 3D shape reconstruction [44].
Revised: Tactile sensingis more local compared to vision and audio, and it is shown to improve sample efﬁciency on a peginsertion task [40]; beneﬁt robotic grasping [41, 42, 43]; augment vision to enhance 3D shape reconstruction [44].

Original: Different from the above work that focuses on a particular task that leverages eitherauditory or tactile data, we construct an object-centric dataset of all three modalities, democratizingmultisensory learning research using vision, audio, and touch.
Revised: Different from the above work that focuses on a particular task that leverages eitherauditory or tactile data, we construct an object-centric dataset of all three modalities, democratizingmultisensory learning research using vision, audio, and touch.

Original:  O BJECT F OLDER
Revised:  O BJECT F OLDER

Original: We introduce O BJECT F OLDER , a dataset of 100 implicitly represented 3D objects with vision, audio, and touch.
Revised: We introduce O BJECT F OLDER , a dataset of 100 implicitly represented 3D objects with vision, audio, and touch.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, each object is represented by an Object File , a compact neuralnetwork that contains all the visual, acoustic, and tactile proﬁles for the object.
Revised: 3, each object is represented by an Object File , a compact neuralnetwork that contains all the visual, acoustic, and tactile proﬁles for the object.

Original: Querying it with extrinsic parameters we can obtain visual appearance of the object from different viewpoints, impactsound of the object at each position, and tactile reading of the object at every surface location.
Revised: Querying it with extrinsic parameters we can obtain visual appearance of the object from different viewpoints, impactsound of the object at each position, and tactile reading of the object at every surface location.

Original: In thefollowing, we ﬁrst describe the source of our objects and the annotations (Sec. 3.1).
Revised: In thefollowing, we ﬁrst describe the source of our objects and the annotations (Sec. 3.1).

Original: Then we presentthe simulation pipeline and how we encode the sensory data using implicit neural representationsfor vision (Sec. 3.2), audio (Sec. 3.3), and touch (Sec. 3.4), respectively.
Revised: Then we presentthe simulation pipeline and how we encode the sensory data using implicit neural representationsfor vision (Sec. 3.2), audio (Sec. 3.3), and touch (Sec. 3.4), respectively.

Original: 3.1 Objects
Revised: 3.1 Objects

Original: We collect 100 high quality 3D objects from online repositories including: 20 objects from 3D
Revised: We collect 100 high quality 3D objects from online repositories including: 20 objects from 3D

Original: Model Haven 1 , 28 objects from the YCB dataset 2 , and 52 objects from Google Scanned Objects 3 .
Revised: Model Haven 1 , 28 objects from the YCB dataset 2 , and 52 objects from Google Scanned Objects 3 .

Original: We select objects that are of realistic visual textures, approximately homogeneous material property,and with a material type that is mappable to one of the following categories: ceramic, glass, wood,plastic, iron, polycarbonate, and steel.
Revised: We select objects that are of realistic visual textures, approximately homogeneous material property,and with a material type that is mappable to one of the following categories: ceramic, glass, wood,plastic, iron, polycarbonate, and steel.

Original: We annotate each object with the material type, which willbe used in audio simulation in Sec. 3.3.
Revised: We annotate each object with the material type, which willbe used in audio simulation in Sec. 3.3.

Original: The dataset contains common household objects of diversecategories such as bowl, mug, cabinet, television, shelf, fork, and spoon.
Revised: The dataset contains common household objects of diversecategories such as bowl, mug, cabinet, television, shelf, fork, and spoon.

Original: See Supp.
Revised: See Supp.

Original: for details.
Revised: for details.

Original: 3.2 Vision
Revised: 3.2 Vision

Original: Simulation: We use Blender’s Cycles path tracer [45] to render images.
Revised: Simulation: We use Blender’s Cycles path tracer [45] to render images.

Original: For each object, we ﬁrstnormalize it into a unit cube and use a point light source at a random location on a unit sphere.
Revised: For each object, we ﬁrstnormalize it into a unit cube and use a point light source at a random location on a unit sphere.

Original: Wethen render images of the object on a white background from camera viewpoints randomly sampledon a full sphere.
Revised: Wethen render images of the object on a white background from camera viewpoints randomly sampledon a full sphere.

Original: See Fig.
Revised: See Fig.

Original: 2 for some example views of objects in our dataset.
Revised: 2 for some example views of objects in our dataset.

Original: VisionNet: After rendering images from different camera viewpoints under varied lighting conditions, we use VisionNet to encode the visual appearance for each object.
Revised: VisionNet: After rendering images from different camera viewpoints under varied lighting conditions, we use VisionNet to encode the visual appearance for each object.

Original: Following prior workon object-centric neural radiance ﬁelds [26], we represent each object as a 7D object-centric neural scattering function whose input is a 3D location x = ( x, y, z ) in the object coordinate frameand the lighting condition at that location ( ω i , ω o ) ,
Revised: Following prior workon object-centric neural radiance ﬁelds [26], we represent each object as a 7D object-centric neural scattering function whose input is a 3D location x = ( x, y, z ) in the object coordinate frameand the lighting condition at that location ( ω i , ω o ) ,

Original: where ω i =
Revised: where ω i =

Original: ( φ i , θ i ) , ω
Revised: ( φ i , θ i ) , ω

Original: o = ( φ o , θ o ) denote the incoming and outgoing light directions, respectively.
Revised: o = ( φ o , θ o ) denote the incoming and outgoing light directions, respectively.

Original: The output is the volume densityσ and fraction of the incoming light that is scattered in the outgoing direction ρ
Revised: The output is the volume densityσ and fraction of the incoming light that is scattered in the outgoing direction ρ

Original: = ( ρ r , ρ g , ρ b ) .
Revised: = ( ρ r , ρ g , ρ b ) .

Original: VisionNet approximates this continuous 7D object-centric representation with an MLP network
Revised: VisionNet approximates this continuous 7D object-centric representation with an MLP network

Original: F v : ( x, y, z, θ i , φ i , θ o , φ o ) −→ ( σ, ρ r , ρ g , ρ b ) that maps each input 7D coordinate to its correspond-
Revised: F v : ( x, y, z, θ i , φ i , θ o , φ o ) −→ ( σ, ρ r , ρ g , ρ b ) that maps each input 7D coordinate to its correspond-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: u m
Revised: u m

Original: [Figure]
Revised: [Figure]

Original: ende
Revised: ende

Original: [Figure]
Revised: [Figure]

Original: (", $
Revised: (", $

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: $ ,
Revised: $ ,

Original: [Equation]
Revised: [Equation]

Original: Object File (Intrinsics)
Revised: Object File (Intrinsics)

Original: TouchNet
Revised: TouchNet

Original: VisionNet
Revised: VisionNet

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: ⨂ ⨂ 3D coordinates
Revised: ⨂ ⨂ 3D coordinates

Original: Light directions
Revised: Light directions

Original: [Figure]
Revised: [Figure]

Original: Extrinsics density
Revised: Extrinsics density

Original: [Figure]
Revised: [Figure]

Original: p
Revised: p

Original: [Figure]
Revised: [Figure]

Original: ing volume density and fraction coefﬁcient for each RGB channel.
Revised: ing volume density and fraction coefﬁcient for each RGB channel.

Original: The amount of light scattered ata point x can be obtained as follows:
Revised: The amount of light scattered ata point x can be obtained as follows:

Original: [Equation]
Revised: [Equation]

Original: where S is a unit sphere and f ρ evaluates the fraction of light incoming from direction ω
Revised: where S is a unit sphere and f ρ evaluates the fraction of light incoming from direction ω

Original: i at the pointthat scatters out in direction ω
Revised: i at the pointthat scatters out in direction ω

Original: o .
Revised: o .

Original: Then we use classic volume rendering [33] to render the color ofany ray passing through the object.
Revised: Then we use classic volume rendering [33] to render the color ofany ray passing through the object.

Original: The expected color C ( r ) of camera ray r ( t ) = x 0 + tω o can beobtained as follows:
Revised: The expected color C ( r ) of camera ray r ( t ) = x 0 + tω o can beobtained as follows:

Original: [Equation]
Revised: [Equation]

Original: T ( t ) denotes the accumulated transmittance along the ray from t n to t ; t n and t f are the near and farintegration bounds; and σ ( r ( t )) denotes the volume density at location r ( t ) .
Revised: T ( t ) denotes the accumulated transmittance along the ray from t n to t ; t n and t f are the near and farintegration bounds; and σ ( r ( t )) denotes the volume density at location r ( t ) .

Original: Similar to [25], we alsouse stratiﬁed sampling, positional encoding, and hierarchical volume sampling to increase renderingquality and efﬁciency.
Revised: Similar to [25], we alsouse stratiﬁed sampling, positional encoding, and hierarchical volume sampling to increase renderingquality and efﬁciency.

Original: See [25, 26] for details.
Revised: See [25, 26] for details.

Original: 3.3 Audio
Revised: 3.3 Audio

Original: Simulation: The goal of audio simulation is to realistically model the impact sound at each position of the object based on its shape, size, material type, external force, and the contact location.
Revised: Simulation: The goal of audio simulation is to realistically model the impact sound at each position of the object based on its shape, size, material type, external force, and the contact location.

Original: Following the standard approach in engineering and acoustics, we use linear modal analysis forphysics-based rigid-body sound synthesis [22, 46].
Revised: Following the standard approach in engineering and acoustics, we use linear modal analysis forphysics-based rigid-body sound synthesis [22, 46].

Original: Firstly, we convert the surface mesh of an object into a volumetric hexahedron mesh, which represents the original shape with N voxels.
Revised: Firstly, we convert the surface mesh of an object into a volumetric hexahedron mesh, which represents the original shape with N voxels.

Original: The key to modal analysis is to solve the following lineardeformation equation for a 3D linear elastic dynamics model:
Revised: The key to modal analysis is to solve the following lineardeformation equation for a 3D linear elastic dynamics model:

Original: [Equation]
Revised: [Equation]

Original: where u ∈ R 3 N denotes the displacement of elemental nodes in 3D; M , K ∈ R 3 N × 3 N denotethe mass and stiffness matrices of the system; C = α M + β K stands for Rayleigh damping; andf ∈ R 3 N represents the external force applied to the object that stimulates the vibration.
Revised: where u ∈ R 3 N denotes the displacement of elemental nodes in 3D; M , K ∈ R 3 N × 3 N denotethe mass and stiffness matrices of the system; C = α M + β K stands for Rayleigh damping; andf ∈ R 3 N represents the external force applied to the object that stimulates the vibration.

Original: Based onthe material type of the object, we obtain its density value ρ , Young’s Modulus E , Poisson’s ratio r ,and Rayleigh damping parameters α and β .
Revised: Based onthe material type of the object, we obtain its density value ρ , Young’s Modulus E , Poisson’s ratio r ,and Rayleigh damping parameters α and β .

Original: The scale of the object and material parameters ρ, E, r are used to build the mass and stiffness matrices M and K .
Revised: The scale of the object and material parameters ρ, E, r are used to build the mass and stiffness matrices M and K .

Original: See Supp.
Revised: See Supp.

Original: for how we map each materialtype to these parameters.
Revised: for how we map each materialtype to these parameters.

Original: The above equation can be decoupled into the following form through generalized eigenvalue decomposition KU = ΛMU :
Revised: The above equation can be decoupled into the following form through generalized eigenvalue decomposition KU = ΛMU :

Original: [Equation]
Revised: [Equation]

Original: where u = Uq and Λ is a diagonal matrix.
Revised: where u = Uq and Λ is a diagonal matrix.

Original: The solution to the above equation is a bank of dampedsinusoidal waves with each wave representing a mode signal:
Revised: The solution to the above equation is a bank of dampedsinusoidal waves with each wave representing a mode signal:

Original: [Equation]
Revised: [Equation]

Original: where ω i is the frequency of the mode,
Revised: where ω i is the frequency of the mode,

Original: c i is the dampling coefﬁcient, a i is the excited amplitude, andθ i is the initial phase.
Revised: c i is the dampling coefﬁcient, a i is the excited amplitude, andθ i is the initial phase.

Original: Assuming the object is stationary originally, we get the following solution:
Revised: Assuming the object is stationary originally, we get the following solution:

Original: [Equation]
Revised: [Equation]

Original: λ i represents the i th mode eigenvalue.
Revised: λ i represents the i th mode eigenvalue.

Original: We simulate the vibration modes excited by unit forcesf 1 , f 2 , f 3 for all vertexes and use the obtained modes signals for training AudioNet as detailed next.
Revised: We simulate the vibration modes excited by unit forcesf 1 , f 2 , f 3 for all vertexes and use the obtained modes signals for training AudioNet as detailed next.

Original: AudioNet:
Revised: AudioNet:

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, for each force direction, we train a separate branch to encode thecorresponding modes signals at all vertexes.
Revised: 3, for each force direction, we train a separate branch to encode thecorresponding modes signals at all vertexes.

Original: We obtain the audio spectrogram 4 for each modessignal and train a MLP to predict the complex spectrogram of dimension F × T × 2 , where F and
Revised: We obtain the audio spectrogram 4 for each modessignal and train a MLP to predict the complex spectrogram of dimension F × T × 2 , where F and

Original: T are the frequency and time dimensions.
Revised: T are the frequency and time dimensions.

Original: Particularly, each branch takes the spatial coordinates( x, y, z ) 5 and the spectrogram location ( f, t ) as input, and predict the real and imagninary part ofthe complex number for every location in the audio spectrogram.
Revised: Particularly, each branch takes the spatial coordinates( x, y, z ) 5 and the spectrogram location ( f, t ) as input, and predict the real and imagninary part ofthe complex number for every location in the audio spectrogram.

Original: The modes signal can then berecovered from the spectrogram using inverse short-time Fourier transform (ISTFT).
Revised: The modes signal can then berecovered from the spectrogram using inverse short-time Fourier transform (ISTFT).

Original: We also usepositional encoding as in VisionNet.
Revised: We also usepositional encoding as in VisionNet.

Original: During test time, we directly use vibration modes for sound synthesis following [46].
Revised: During test time, we directly use vibration modes for sound synthesis following [46].

Original: Any externalforce at a vertex can be decomposed into a linear combination of unit forces along the three orthogonal directions:
Revised: Any externalforce at a vertex can be decomposed into a linear combination of unit forces along the three orthogonal directions:

Original: f = k 1 f 1 + k 2 f 

Revised: f = k 1 f 1 + k 2 f

Original: + k 3 f 3 .
Revised: + k 3 f 3 .

Original: The amplitudes of modes a excited by f can be decomposedinto a linear combination of the amplitudes excited by the unit force: a = k 1 a 1 + k 2 a 

Revised: The amplitudes of modes a excited by f can be decomposedinto a linear combination of the amplitudes excited by the unit force: a = k 1 a 1 + k 2 a

Original: + k 3 a 3 ,where a 1 , a 2 , a 3 denote the amplitudes of modes excited by unit forces f 1 , f 2 , f 3 for a given vertex.
Revised: + k 3 a 3 ,where a 1 , a 2 , a 3 denote the amplitudes of modes excited by unit forces f 1 , f 2 , f 3 for a given vertex.

Original: Because modal analysis is performed on a volumetric hexahedron mesh, for a vertex in the originalpolygon mesh, we ﬁnd its closest four vertexes in the hexahedron and average their modes signals.
Revised: Because modal analysis is performed on a volumetric hexahedron mesh, for a vertex in the originalpolygon mesh, we ﬁnd its closest four vertexes in the hexahedron and average their modes signals.

Original: Prior work [22, 47] have demonstrated that such physics-based rigid-body sound synthesis pipelinerenders sound that transfers well to real scenes and the objects’ shape, material, and other auditoryattributes can be successfully inferred from the simulated sound.
Revised: Prior work [22, 47] have demonstrated that such physics-based rigid-body sound synthesis pipelinerenders sound that transfers well to real scenes and the objects’ shape, material, and other auditoryattributes can be successfully inferred from the simulated sound.

Original: To further demonstrate the realismof our audio simulation, we also perform a user study on acoustic ﬁdelity that asks participants todistinguish between the real and simulated audios.
Revised: To further demonstrate the realismof our audio simulation, we also perform a user study on acoustic ﬁdelity that asks participants todistinguish between the real and simulated audios.

Original: Results show that our simulated audio is preferredby 42% of the total responses, comparing closely to real audio recordings.
Revised: Results show that our simulated audio is preferredby 42% of the total responses, comparing closely to real audio recordings.

Original: See Supp.
Revised: See Supp.

Original: for details.
Revised: for details.

Original: 3.4 Touch
Revised: 3.4 Touch

Original: Simulation: We leverage a state-of-the-art touch simulator TACTO [24] for touch simulation.
Revised: Simulation: We leverage a state-of-the-art touch simulator TACTO [24] for touch simulation.

Original: TACTO is a vision-based touch simulator and uses ray-tracing to simulate high-quality tactile signals.
Revised: TACTO is a vision-based touch simulator and uses ray-tracing to simulate high-quality tactile signals.

Original: It can simulate realistic rendering for both contact surfaces and shadows with different contactgeometry.
Revised: It can simulate realistic rendering for both contact surfaces and shadows with different contactgeometry.

Original: See [24] for details and a comparison of the simulated and real tactile readings.
Revised: See [24] for details and a comparison of the simulated and real tactile readings.

Original: Particularly, we use DIGIT [23] touch sensor and its corresponding camera, lights, and gel mesh conﬁgurations in simulation.
Revised: Particularly, we use DIGIT [23] touch sensor and its corresponding camera, lights, and gel mesh conﬁgurations in simulation.

Original: We choose DIGIT for its compactness, high-resolution of its representation, andpotential applications for robotic in-hand manipulation.
Revised: We choose DIGIT for its compactness, high-resolution of its representation, andpotential applications for robotic in-hand manipulation.

Original: Other touch sensors such as Gelsight [48],
Revised: Other touch sensors such as Gelsight [48],

Original: BioTac [49], or Ominitact [50] can also be potentially used, and the design of the touch sensor touse is orthogonal to our work.
Revised: BioTac [49], or Ominitact [50] can also be potentially used, and the design of the touch sensor touse is orthogonal to our work.

Original: To obtain touch simulations, we place each object at rest and move
Revised: To obtain touch simulations, we place each object at rest and move

Original: DIGIT to touch each object vertex in vertex normal direction.
Revised: DIGIT to touch each object vertex in vertex normal direction.

Original: We constrain the force to range withina small threshold such that the rendered tactile images do not vary signiﬁcantly with different forces.
Revised: We constrain the force to range withina small threshold such that the rendered tactile images do not vary signiﬁcantly with different forces.

Original: We use the RGB tactile image that contains the local contact geometry as our touch representation.
Revised: We use the RGB tactile image that contains the local contact geometry as our touch representation.

Original: TouchNet:
Revised: TouchNet:

Original: For each object, we obtain a RGB tactile image of dimension W × H × 3 that capturesthe local geometry information for every vertex on the surface of the polygon mesh.
Revised: For each object, we obtain a RGB tactile image of dimension W × H × 3 that capturesthe local geometry information for every vertex on the surface of the polygon mesh.

Original: TouchNet takes
Revised: TouchNet takes

Original: [Figure]
Revised: [Figure]

Original: the spatial coordinate ( x, y, z ) of the vertex and the spatial location ( w, h ) in the tactile image asinput, and predicts the per-pixel value for the three channels of the tactile RGB image.
Revised: the spatial coordinate ( x, y, z ) of the vertex and the spatial location ( w, h ) in the tactile image asinput, and predicts the per-pixel value for the three channels of the tactile RGB image.

Original: The othersettings are similar to VisionNet and AudioNet.
Revised: The othersettings are similar to VisionNet and AudioNet.

Original: Implicit
Revised: Implicit

Original: Original
Revised: Original

Original: Vision 7.2 MB ∞
Revised: Vision 7.2 MB ∞

Original: Audio 6.3 MB 12.6 GB
Revised: Audio 6.3 MB 12.6 GB

Original: Touch 2.1 MB 2.9 GB
Revised: Touch 2.1 MB 2.9 GB

Original: Fig.
Revised: Fig.

Original: 4 shows some examples of the visual, auditory, and tactile sensory data obtained from Object Files.
Revised: 4 shows some examples of the visual, auditory, and tactile sensory data obtained from Object Files.

Original: The renderedimages, impact sounds, and touch readings well match theground-truth simulations.
Revised: The renderedimages, impact sounds, and touch readings well match theground-truth simulations.

Original: See Supp.
Revised: See Supp.

Original: for a quantitative comparison on the accuracy of the implicit representations.
Revised: for a quantitative comparison on the accuracy of the implicit representations.

Original: Moreover, using implicit neural representations is much more storage efﬁcient compared to the original data format.
Revised: Moreover, using implicit neural representations is much more storage efﬁcient compared to the original data format.

Original: Table 1shows a comparison between the storage size with and without using implicit representations.
Revised: Table 1shows a comparison between the storage size with and without using implicit representations.

Original: Object File in the form ofan implicit neural network is a much more compact representation of the multisensory data for eachobject.
Revised: Object File in the form ofan implicit neural network is a much more compact representation of the multisensory data for eachobject.

Original: Note that VisionNet models an object-centric neural scattering function [26] and thus cangeneralize to any camera viewpoint under arbitrary lighting conditions, while rendering and savingimages for all scenarios takes inﬁnite space theoretically.
Revised: Note that VisionNet models an object-centric neural scattering function [26] and thus cangeneralize to any camera viewpoint under arbitrary lighting conditions, while rendering and savingimages for all scenarios takes inﬁnite space theoretically.

Original:  Experiments
Revised:  Experiments

Original: O BJECT F OLDER is a potential testbed for many perception and control tasks.
Revised: O BJECT F OLDER is a potential testbed for many perception and control tasks.

Original: Next, we evaluateon four benchmark tasks including instance recognition, cross-sensory retrieval, 3D reconstruction,and robotic grasping to demonstrate the usefulness of the dataset and the value of multisensoryperception.
Revised: Next, we evaluateon four benchmark tasks including instance recognition, cross-sensory retrieval, 3D reconstruction,and robotic grasping to demonstrate the usefulness of the dataset and the value of multisensoryperception.

Original: We show the key results below, and see Supp.
Revised: We show the key results below, and see Supp.

Original: for the detailed experiments setup.
Revised: for the detailed experiments setup.

Original: 4.1 Multisensory Instance Recognition
Revised: 4.1 Multisensory Instance Recognition

Original: Methods
Revised: Methods

Original: Acc.
Revised: Acc.

Original: Chance 1.
Revised: Chance 1.

Original: Vision (V) 94.
Revised: Vision (V) 94.

Original: Audio (
Revised: Audio (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Identifying the object instance that is interacted with is fundamentalfor many robotic applications.
Revised: Identifying the object instance that is interacted with is fundamentalfor many robotic applications.

Original: In this task, we want to identifyan object based on either its visual appearance, impact sound, orlocal contact geometry.
Revised: In this task, we want to identifyan object based on either its visual appearance, impact sound, orlocal contact geometry.

Original: We are interested in ﬁnding out the amountof information that is useful to recognize the object instance fromeach modality.
Revised: We are interested in ﬁnding out the amountof information that is useful to recognize the object instance fromeach modality.

Original: For the vision modality, we train a ResNet-18 [51]network that takes an RGB image of the object as input and predictsthe instance label of the object.
Revised: For the vision modality, we train a ResNet-18 [51]network that takes an RGB image of the object as input and predictsthe instance label of the object.

Original: The settings are the same for audioand touch except that the input is either a magnitude spectrogram ofthe impact sound the object generates or a local tactile RGB imagefrom a random location on the object surface.
Revised: The settings are the same for audioand touch except that the input is either a magnitude spectrogram ofthe impact sound the object generates or a local tactile RGB imagefrom a random location on the object surface.

Original: Table 2 shows the results.
Revised: Table 2 shows the results.

Original: The accuracy for a random classiﬁer is1%.
Revised: The accuracy for a random classiﬁer is1%.

Original: The vision classiﬁer achieves high accuracy, because an imageof the object normally captures informative appearance cues thatdistinguish the object instance.
Revised: The vision classiﬁer achieves high accuracy, because an imageof the object normally captures informative appearance cues thatdistinguish the object instance.

Original: Under some camera viewpoints, the vision classiﬁer can mistakenlyclassify the object.
Revised: Under some camera viewpoints, the vision classiﬁer can mistakenlyclassify the object.

Original: Surprisingly, the audio classiﬁer achieves the best results.
Revised: Surprisingly, the audio classiﬁer achieves the best results.

Original: This conﬁrms that ouraudio simulation pipeline well models the shape, size, and material property of the object instance,so that a single impact sound can be used to recognize the object.
Revised: This conﬁrms that ouraudio simulation pipeline well models the shape, size, and material property of the object instance,so that a single impact sound can be used to recognize the object.

Original: Touch is good at capturing localgeometry of an object, but often does not contain sufﬁcient discriminative cues to recognize theobject especially from a single touch.
Revised: Touch is good at capturing localgeometry of an object, but often does not contain sufﬁcient discriminative cues to recognize theobject especially from a single touch.

Original: Therefore, the touch classiﬁer performs worse comparedto vision and audio.
Revised: Therefore, the touch classiﬁer performs worse comparedto vision and audio.

Original: Classiﬁers with multisensory input are more robust, and combining all threemodalities leads to the best result.
Revised: Classiﬁers with multisensory input are more robust, and combining all threemodalities leads to the best result.

Original: 4.2 Cross-Sensory Retrieval
Revised: 4.2 Cross-Sensory Retrieval

Original: Cross-sensory retrieval plays a crucial role in machine perception to understand the relationshipsbetween different modalities.
Revised: Cross-sensory retrieval plays a crucial role in machine perception to understand the relationshipsbetween different modalities.

Original: Fig.
Revised: Fig.

Original: 5 shows our framework for learning cross-sensory embeddingsfor two modalities.
Revised: 5 shows our framework for learning cross-sensory embeddingsfor two modalities.

Original: Taking cross-sensory learning with vision and audio as an example, we designa two-stream network that takes an RGB image and an audio spectrogram of the same object asinput for each stream.
Revised: Taking cross-sensory learning with vision and audio as an example, we designa two-stream network that takes an RGB image and an audio spectrogram of the same object asinput for each stream.

Original: The two modalities have disjoint pathways in the early layers to capture themodality-speciﬁc features.
Revised: The two modalities have disjoint pathways in the early layers to capture themodality-speciﬁc features.

Original: A ﬁnal modality-agnostic network is used to map the shared embeddingto the label space for training with cross-entropy loss.
Revised: A ﬁnal modality-agnostic network is used to map the shared embeddingto the label space for training with cross-entropy loss.

Original: We map the features of both modalities fromthe last hidden state to a cross-sensory embedding space through a triplet loss.
Revised: We map the features of both modalities fromthe last hidden state to a cross-sensory embedding space through a triplet loss.

Original: We either sample animage or audio from another object as the negative example, and pull the embeddings of the sameobject to be closer in the joint embedding space and push the embeddings from different objectsapart.
Revised: We either sample animage or audio from another object as the negative example, and pull the embeddings of the sameobject to be closer in the joint embedding space and push the embeddings from different objectsapart.

Original: The full model is trained jointly with both the triplet loss and the cross-entropy loss.
Revised: The full model is trained jointly with both the triplet loss and the cross-entropy loss.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Retrieved Chance CCA [52]
Revised: Retrieved Chance CCA [52]

Original: Ours
Revised: Ours

Original: Vision Audio 0.
Revised: Vision Audio 0.

Original: Touch 0.
Revised: Touch 0.

Original: Audio Vision 0.
Revised: Audio Vision 0.

Original: Touch 0.
Revised: Touch 0.

Original: Touch Vision 0.
Revised: Touch Vision 0.

Original: Audio 0.
Revised: Audio 0.

Original: Table 3 shows the results for cross-sensory retrieval on the test set, where we retrieve the sensorydata of a different modality by querying from the shared embedding space.
Revised: Table 3 shows the results for cross-sensory retrieval on the test set, where we retrieve the sensorydata of a different modality by querying from the shared embedding space.

Original: We use cosine similarityto measure the distances between embeddings and retrieve the sample with the smallest distance.
Revised: We use cosine similarityto measure the distances between embeddings and retrieve the sample with the smallest distance.

Original: We report mean Average Precision (mAP) which jointly considers the ranking information and theprecision, and compare with random retrieval and a state-of-the-art cross-sensory retrieval methodbased on canonical correlation analysis (CCA) [52].
Revised: We report mean Average Precision (mAP) which jointly considers the ranking information and theprecision, and compare with random retrieval and a state-of-the-art cross-sensory retrieval methodbased on canonical correlation analysis (CCA) [52].

Original: Our models perform more reliable retrievals,and the results show that the learned cross-sensory embeddings are effective and contain ﬁne-grainedinformation about objects to correctly retrieve sensory data from the matched object.
Revised: Our models perform more reliable retrievals,and the results show that the learned cross-sensory embeddings are effective and contain ﬁne-grainedinformation about objects to correctly retrieve sensory data from the matched object.

Original: Similar toinstance recognition, we observe vision and audio provide more reliable global information aboutthe object compared to touch.
Revised: Similar toinstance recognition, we observe vision and audio provide more reliable global information aboutthe object compared to touch.

Original: 4.3 Audio-Visual 3D Reconstruction
Revised: 4.3 Audio-Visual 3D Reconstruction

Original: Robots often need to build a mental model of the 3D shape of objects based on just a single glimpseor the sounds they make during interaction.
Revised: Robots often need to build a mental model of the 3D shape of objects based on just a single glimpseor the sounds they make during interaction.

Original: In this section, we investigate the potential of 3D reconstruction using either a single image, an impact sound, or their combination.
Revised: In this section, we investigate the potential of 3D reconstruction using either a single image, an impact sound, or their combination.

Original: We leverage Occupancy
Revised: We leverage Occupancy

Original: Network [29], which implicitly represents the 3D surface as the continuous decision boundary of adeep neural network classiﬁer, as a testbed for this task.
Revised: Network [29], which implicitly represents the 3D surface as the continuous decision boundary of adeep neural network classiﬁer, as a testbed for this task.

Original: Fig.
Revised: Fig.

Original: 6 illustrates our I MAGE +A UDIO 2M ESHframework for audio-visual 3D shape reconstruction.
Revised: 6 illustrates our I MAGE +A UDIO 2M ESHframework for audio-visual 3D shape reconstruction.

Original: The image and audio embeddings are fusedthrough a fusion layer into an audio-visual feature, then combined with coordinate-conditioned feature maps through Conditional Batch-Normalization to predict occupancy probabilities p .
Revised: The image and audio embeddings are fusedthrough a fusion layer into an audio-visual feature, then combined with coordinate-conditioned feature maps through Conditional Batch-Normalization to predict occupancy probabilities p .

Original: We alsotest two of its simpliﬁed variants I MAGE 2M ESH , which represents existing single image 3D reconstruction approaches that predict the 3D shape from a single view of the object, and A UDIO 2M ESH ,which performs 3D reconstruction purely from audio.
Revised: We alsotest two of its simpliﬁed variants I MAGE 2M ESH , which represents existing single image 3D reconstruction approaches that predict the 3D shape from a single view of the object, and A UDIO 2M ESH ,which performs 3D reconstruction purely from audio.

Original: Methods
Revised: Methods

Original: IoU ↑
Revised: IoU ↑

Original: Chamfer-
Revised: Chamfer-

Original: L 1 ↓
Revised: L 1 ↓

Original: Normal Consistency ↑
Revised: Normal Consistency ↑

Original: A VERAGE 0.
Revised: A VERAGE 0.

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: 2M
Revised: 2M

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: 2M
Revised: 2M

Original: [Table]
Revised: [Table]

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: +A
Revised: +A

Original: [Table]
Revised: [Table]

Original: 2M
Revised: 2M

Original: [Table]
Revised: [Table]

Original: We evaluate on a heldout test set with standard metrics: IoU, Chamfer-
Revised: We evaluate on a heldout test set with standard metrics: IoU, Chamfer-

Original: L 1 distance, and Normal
Revised: L 1 distance, and Normal

Original: Consistency with respect to the ground-truth mesh.
Revised: Consistency with respect to the ground-truth mesh.

Original: Table 4 shows the results.
Revised: Table 4 shows the results.

Original: Compared to the simple A VERAGE baseline that averages the results obtained using the ground-truth mesh of eachof the 100 objects as the prediction, we can see that 3D reconstruction from a single image oran impact sound performs much better.
Revised: Compared to the simple A VERAGE baseline that averages the results obtained using the ground-truth mesh of eachof the 100 objects as the prediction, we can see that 3D reconstruction from a single image oran impact sound performs much better.

Original: Augmenting traditional single image 3d reconstruction withaudio achieves the best performance by leveraging the additional acoustic spatial cues.
Revised: Augmenting traditional single image 3d reconstruction withaudio achieves the best performance by leveraging the additional acoustic spatial cues.

Original: Furthermore,as shown in Fig. 7, we apply our I MAGE 2M ESH model trained on objects in O BJECT F OLDER toreal-world images.
Revised: Furthermore,as shown in Fig. 7, we apply our I MAGE 2M ESH model trained on objects in O BJECT F OLDER toreal-world images.

Original: Our model generalizes reasonably well to real image, demonstrating the realismof the objects in our dataset.
Revised: Our model generalizes reasonably well to real image, demonstrating the realismof the objects in our dataset.

Original: The last column shows a typical failure case, where the teapot image isvery visually different from objects in our dataset.
Revised: The last column shows a typical failure case, where the teapot image isvery visually different from objects in our dataset.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 4.4 Robotic Grasping with Vision and Touch
Revised: 4.4 Robotic Grasping with Vision and Touch

Original: Next we show how we combine vision and touch for prediction of grasp stability.
Revised: Next we show how we combine vision and touch for prediction of grasp stability.

Original: The goal is touse a robotic arm to successfully grasp and hold an object between the robot’s left and right ﬁngers.
Revised: The goal is touse a robotic arm to successfully grasp and hold an object between the robot’s left and right ﬁngers.

Original: For touch data, we infer the RGB tactile image from the corresponding Object File based on thegrasp contact positions of the left and right grippers on the object; For vision data, we obtain imagesfrom an externally mounted camera’s viewpoint at the grasping moment.
Revised: For touch data, we infer the RGB tactile image from the corresponding Object File based on thegrasp contact positions of the left and right grippers on the object; For vision data, we obtain imagesfrom an externally mounted camera’s viewpoint at the grasping moment.

Original: We label the data as either“Success” or “Failure” based on whether the object can be held between the robot’s ﬁngers afterbeing lifted.
Revised: We label the data as either“Success” or “Failure” based on whether the object can be held between the robot’s ﬁngers afterbeing lifted.

Original: Fig.
Revised: Fig.

Original: 8 shows an example of both a successful grasp and a failure case.
Revised: 8 shows an example of both a successful grasp and a failure case.

Original: We collect data for 10 objects from O BJECT F OLDER , and then train binary classiﬁers under differentsample sizes to predict the grasp outcome based on either the vision signal, the touch signal, or theircombination.
Revised: We collect data for 10 objects from O BJECT F OLDER , and then train binary classiﬁers under differentsample sizes to predict the grasp outcome based on either the vision signal, the touch signal, or theircombination.

Original: This experiment is similar to that of [43, 24] except that we obtain touch readingsdirectly by querying our implicit representation networks.
Revised: This experiment is similar to that of [43, 24] except that we obtain touch readingsdirectly by querying our implicit representation networks.

Original: Fig.
Revised: Fig.

Original: 9 shows the results.
Revised: 9 shows the results.

Original: We observe thatit takes signiﬁcantly less amount of data to reach high accuracy when learning from touch comparedto vision.
Revised: We observe thatit takes signiﬁcantly less amount of data to reach high accuracy when learning from touch comparedto vision.

Original: Leveraging both visual and tactile data achieves the best accuracy.
Revised: Leveraging both visual and tactile data achieves the best accuracy.

Original: Additionally, we train apolicy for touch-based robotic grasping using TRPO [53].
Revised: Additionally, wetrain a policy for touch-based robotic grasping using TRPO [53].

Original: We achieve a success rate of 75% duringtesting, while that of a random grasping policy is 50% .
Revised: We achieve a success rate of75 . 5% during testing, while that of a random grasping policy is 53 . 0% .

Original: To further demonstrate the potential of usingour dataset on robot manipulation tasks, we also perform an experiment on an object manipulationtask (reach) using Meta-World [54] with three of our objects (cup, bowl, dice).
Revised: To further demonstrate thepotential of using our dataset on robot manipulation tasks, we also perform an experiment on anobject manipulation task (reach) using Meta-World [54] with three of our objects (cup, bowl, dice).

Original: We achieve 100%success rate for each object.
Revised: We achieve 100% success rate for each object.

Original: See Supp.
Revised: See Supp.

Original: for details.
Revised: for details.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: We presented O BJECT F OLDER , a dataset of objects with implicit visual, auditory, and tactile representations.
Revised: We presented O BJECT F OLDER , a dataset of objects with implicit visual, auditory, and tactile representations.

Original: It will facilitate research with vision, audio, and touch, and enable multisensory simulation of objects in robotic virtual environments.
Revised: It will facilitate research with vision, audio, and touch, and enable multisensory simulation of objects in robotic virtual environments.

Original: Through evaluation on an array of benchmarktasks, we show the usefulness of our dataset for both perception and control.
Revised: Through evaluation on an array of benchmarktasks, we show the usefulness of our dataset for both perception and control.

Original: As future work, weplan to explore richer object states with more accurate and ﬁne-grained physics, additional sensorymodalities, and textual descriptions.
Revised: As future work, weplan to explore richer object states with more accurate and ﬁne-grained physics, additional sensorymodalities, and textual descriptions.