Original: O BJECT F OLDER : A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Revised: O BJECT F OLDER : A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations

Original: Abstract: Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years.
Revised: Abstract: Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years.

Original: However, the progress in these directions is limited by the small set of objects available—synthetic objects are not realistic enough and are mostly centered around geometry, while real object datasets such as YCB are often practically challenging and unstable to acquire due to international shipping, inventory, and ﬁnancial cost.
Revised: However, the progress in these directions is limited by the small set of objects available—synthetic objects are not realistic enough and are mostly centered around geometry, while real object datasets such as YCB are often practically challenging and unstable to acquire due to international shipping, inventory, and ﬁnancial cost.

Original: We present O BJECT F OLDER , a dataset of 100 virtualized objects that addresses both challenges with two key innovations.
Revised: We present O BJECT F OLDER , a dataset of 100 virtualized objects that addresses both challenges with two key innovations.

Original: First, O BJECT F OLDER encodes the visual, auditory, and tactile sensory data for all objects, enabling a number of multisensory object recognition tasks, beyond existing datasets that focus purely on object geometry.
Revised: First, O BJECT F OLDER encodes the visual, auditory, and tactile sensory data for all objects, enabling a number of multisensory object recognition tasks, beyond existing datasets that focus purely on object geometry.

Original: Second,
Revised: Second,

Original: O BJECT F OLDER employs a uniform, object-centric, and implicit representation for each object’s visual textures, acoustic simulations, and tactile readings, making the dataset ﬂexible to use and easy to share.
Revised: O BJECT F OLDER employs a uniform, object-centric, and implicit representation for each object’s visual textures, acoustic simulations, and tactile readings, making the dataset ﬂexible to use and easy to share.

Original: We demonstrate the usefulness of our dataset as a testbed for multisensory perception and control by evaluating it on a variety of benchmark tasks, including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping.
Revised: We demonstrate the usefulness of our dataset as a testbed for multisensory perception and control by evaluating it on a variety of benchmark tasks, including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping.

Original: [Figure]
Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: We perceive the world not as a single giant entity but often through a series of inanimate objects, which exist as bounded wholes and move on connected paths.
Revised: We perceive the world not as a single giant entity but often through a series of inanimate objects, which exist as bounded wholes and move on connected paths.

Original: We interact with these objects through an array of different sensory systems–vision, audition, touch, smell, taste, and proprioception.
Revised: We interact with these objects through an array of different sensory systems–vision, audition, touch, smell, taste, and proprioception.

Original: These multisensory inputs shape our daily experiences: we observe the surroundings to avoid obstacles, hear the doorbell ring to realize the guests have arrived, and touch the fabric to sense its degree of comfort.
Revised: These multisensory inputs shape our daily experiences: we observe the surroundings to avoid obstacles, hear the doorbell ring to realize the guests have arrived, and touch the fabric to sense its degree of comfort.

Original: Cognitive science studies [1, 2] show that both object representation and multisensory perception play a crucial role in human cognitive development.
Revised: Cognitive science studies [1, 2] show that both object representation and multisensory perception play a crucial role in human cognitive development.

Original: Object-centric learning has shown great potential lately in visual reasoning [3], representing physical scenes [4], modeling multi-agent interactions [5], and improving generalization to unseen object compositions [6].
Revised: Object-centric learning has shown great potential lately in visual reasoning [3], representing physical scenes [4], modeling multi-agent interactions [5], and improving generalization to unseen object compositions [6].

Original: These prior methods, however, often do not model the full spectrum of physical object properties and sensory modes, including their 3D shape, texture, material, sound, and feel.
Revised: These prior methods, however, often do not model the full spectrum of physical object properties and sensory modes, including their 3D shape, texture, material, sound, and feel.

Original: While there has been signiﬁcant progress by “looking”—recognizing objects based on glimpses of their visual appearance or 3D shape—objects in the world are often modeled as silent and untouchable entities.
Revised: While there has been signiﬁcant progress by “looking”—recognizing objects based on glimpses of their visual appearance or 3D shape—objects in the world are often modeled as silent and untouchable entities.

Original: We identify that this is due to the lack of rich object datasets.
Revised: We identify that this is due to the lack of rich object datasets.

Original: Existing object datasets have contributed to most signiﬁcant progress in image recognition [7, 8] and shape modeling [9, 10], but other sensory information is minimally considered.
Revised: Existing object datasets have contributed to most signiﬁcant progress in image recognition [7, 8] and shape modeling [9, 10], but other sensory information is minimally considered.

Original: Datasets for robotic manipulation research [11, 12] consist of a selected number of real-world household objects, but these objects are expensive and often unstable to acquire due to international shipping and inventory.
Revised: Datasets for robotic manipulation research [11, 12] consist of a selected number of real-world household objects, but these objects are expensive and often unstable to acquire due to international shipping and inventory.

Original: Moreover, it is nontrivial to virtualize these objects and their multiple sensory modes, limiting their usability for developing embodied AI agents [13, 14, 15, 16, 17, 18, 19, 20, 21].
Revised: Moreover, it is nontrivial to virtualize these objects and their multiple sensory modes, limiting their usability for developing embodied AI agents [13, 14, 15, 16, 17, 18, 19, 20, 21].

Original: Our goal is therefore to create a dataset of 3D objects that are 1) easily accessible to the community as a standard benchmark, 2) high-quality in terms of visual textures, and 3) augmented with realistic auditory and tactile sensory data.
Revised: Our goal is therefore to create a dataset of 3D objects that are 1) easily accessible to the community as a standard benchmark, 2) high-quality in terms of visual textures, and 3) augmented with realistic auditory and tactile sensory data.

Original: Towards this end, we introduce O BJECT F OLDER —a multisensory dataset of implicitly represented Object Files .
Revised: Towards this end, we introduce O BJECT F OLDER —a multisensory dataset of implicitly represented Object Files .

Original: The concept of an object ﬁle can be traced back to Kahneman et al.
Revised: The concept of an object ﬁle can be traced back to Kahneman et al.

Original: [22], where it is deﬁned as the sensory information that has been received about the object at a particular location.
Revised: [22], where it is deﬁned as the sensory information that has been received about the object at a particular location.

Original: In a similar spirit, we model and represent each object (or its intrinsics) using an implicit neural representation, which through querying with extrinsic parameters we can obtain images of the object from different viewpoints, impact sounds of the object at each position, and tactile sensing of the object at every surface location.
Revised: In a similar spirit, we model and represent each object (or its intrinsics) using an implicit neural representation, which through querying with extrinsic parameters we can obtain images of the object from different viewpoints, impact sounds of the object at each position, and tactile sensing of the object at every surface location.

Original: These different sensory data of vision, audio, and touch can be regarded as contemporary multisensory object ﬁles.
Revised: These different sensory data of vision, audio, and touch can be regarded as contemporary multisensory object ﬁles.

Original: Speciﬁcally, we collect 100 high quality 3D objects from online repositories.
Revised: Speciﬁcally, we collect 100 high quality 3D objects from online repositories.

Original: For each object, we model its visual appearance at random object pose, lighting condition, and camera viewpoint; we perform modal analysis [23] based on its shape, size, and material type to calculate its characteristic vibration modes for audio simulation; and we simulate its touch readings at all surface locations using DIGIT [24, 25]–a vision-based tactile sensor.
Revised: For each object, we model its visual appearance at random object pose, lighting condition, and camera viewpoint; we perform modal analysis [23] based on its shape, size, and material type to calculate its characteristic vibration modes for audio simulation; and we simulate its touch readings at all surface locations using DIGIT [24, 25]–a vision-based tactile sensor.

Original: Based on recent success on neural implicit representations [26, 27], we design a deep neural network that consists of three sub-networks–VisionNet,
Revised: Based on recent success on neural implicit representations [26, 27], we design a deep neural network that consists of three sub-networks–VisionNet,

Original: AudioNet, and TouchNet, which encode the visual, auditory, and tactile sensory data for each object, respectively.
Revised: AudioNet, and TouchNet, which encode the visual, auditory, and tactile sensory data for each object, respectively.

Original: See Fig.
Revised: See Fig.

Original: Together, they constitute an Object File that contains the complete multisensory proﬁle for each object instance.
Revised: Together, they constitute an Object File that contains the complete multisensory proﬁle for each object instance.

Original: Furthermore, we demonstrate the usefulness of our dataset on four benchmark tasks leveraging multisensory data, including instance recognition, cross-modal retrieval, 3D reconstruction, and robotic grasping.
Revised: Furthermore, we demonstrate the usefulness of our dataset on four benchmark tasks leveraging multisensory data, including instance recognition, cross-modal retrieval, 3D reconstruction, and robotic grasping.

Original: Our main contributions are threefold: First, we introduce O BJECT F OLDER , a dataset that makes multisensory learning with vision, audio, and touch easily accessible to the research community;
Revised: Our main contributions are threefold: First, we introduce O BJECT F OLDER , a dataset that makes multisensory learning with vision, audio, and touch easily accessible to the research community;

Original: Second, all objects in our dataset are compatible to different robotic virtual environments and will be made publicly available as a standard testbed for robotic multisensory learning; Third, we evaluate on a suite of benchmark tasks that require multisensory data to facilitate future work in this direction.
Revised: Second, all objects in our dataset are compatible to different robotic virtual environments and will be made publicly available as a standard testbed for robotic multisensory learning; Third, we evaluate on a suite of benchmark tasks that require multisensory data to facilitate future work in this direction.

Original:  Related Work
Revised:  Related Work

Original: Object-Centric Datasets Many image datasets exist for object recognition, such as ImageNet [7],
Revised: Object-Centric Datasets Many image datasets exist for object recognition, such as ImageNet [7],

Original: MS COCO [28], ObjectNet [8], and OpenImages [29].
Revised: MS COCO [28], ObjectNet [8], and OpenImages [29].

Original: These datasets consist of 2D images, while
Revised: These datasets consist of 2D images, while

Original: O BJECT F OLDER is a dataset of 3D objects.
Revised: O BJECT F OLDER is a dataset of 3D objects.

Original: ModelNet [9] and ShapeNet [10] are two large-scale datasets of 3D models, but the collected 3D CAD models either contain no or low-quality visual textures, making them unsuitable for real-world applications.
Revised: ModelNet [9] and ShapeNet [10] are two large-scale datasets of 3D models, but the collected 3D CAD models either contain no or low-quality visual textures, making them unsuitable for real-world applications.

Original: For research on object manipulation, datasets are constructed using real-world household objects such as YCB [11] and BigBIRD [12].
Revised: For research on object manipulation, datasets are constructed using real-world household objects such as YCB [11] and BigBIRD [12].

Original: Different from them, our goal is to construct a dataset of 3D objects, where realistic visual, auditory, and tactile sensory data can be easily obtained for each object in virtual environments.
Revised: Different from them, our goal is to construct a dataset of 3D objects, where realistic visual, auditory, and tactile sensory data can be easily obtained for each object in virtual environments.

Original: Implicit Neural Representations Coordinate-based multi-layer perceptrons (MLP) have recently been adopted as continuous, memory-efﬁcient implicit representations for a variety of visual signals such as 3D shape [30, 31], scenes [32, 33], and object appearance [26, 27] with the help of classic volume rendering techniques [34].
Revised: Implicit Neural Representations Coordinate-based multi-layer perceptrons (MLP) have recently been adopted as continuous, memory-efﬁcient implicit representations for a variety of visual signals such as 3D shape [30, 31], scenes [32, 33], and object appearance [26, 27] with the help of classic volume rendering techniques [34].

Original: We also use MLP as a compact neural representation to represent each 3D object.
Revised: We also use MLP as a compact neural representation to represent each 3D object.

Original: Differently, apart from visual appearance, our mutisensory implicit neural representation also encodes object-centric auditory and tactile data of the object.
Revised: Differently, apart from visual appearance, our mutisensory implicit neural representation also encodes object-centric auditory and tactile data of the object.

Original: Multisensory Perception for Robotics Recent work shows growing interests in using audio or touch in conjunction with vision for robotic tasks.
Revised: Multisensory Perception for Robotics Recent work shows growing interests in using audio or touch in conjunction with vision for robotic tasks.

Original: Audio is used to recognize object instances [35] or terrain types [36]; estimate the ﬂow and amount of granular materials [37] or liquid height [38] in robotic scooping and pouring tasks; study its synergies with the motion of a robot [39]; and bridge the simulation-to-reality gap for tasks that involve stochastic dynamics [40].
Revised: Audio is used to recognize object instances [35] or terrain types [36]; estimate the ﬂow and amount of granular materials [37] or liquid height [38] in robotic scooping and pouring tasks; study its synergies with the motion of a robot [39]; and bridge the simulation-to-reality gap for tasks that involve stochastic dynamics [40].

Original: Tactile sensing is more local compared to vision and audio, and it is shown to improve sample efﬁciency on a peg insertion task [41]; beneﬁt robotic grasping [42, 43, 44]; augment vision to enhance 3D shape reconstruction [45].
Revised: Tactile sensing is more local compared to vision and audio, and it is shown to improve sample efﬁciency on a peg insertion task [41]; beneﬁt robotic grasping [42, 43, 44]; augment vision to enhance 3D shape reconstruction [45].

Original: Different from the above work that focuses on a particular task that leverages either auditory or tactile data, we construct an object-centric dataset of all three modalities, democratizing multisensory learning research using vision, audio, and touch.
Revised: Different from the above work that focuses on a particular task that leverages either auditory or tactile data, we construct an object-centric dataset of all three modalities, democratizing multisensory learning research using vision, audio, and touch.

Original:  O BJECT F OLDER
Revised:  O BJECT F OLDER

Original: We introduce O BJECT F OLDER , a dataset of 100 implicitly represented 3D objects with vision, audio, and touch.
Revised: We introduce O BJECT F OLDER , a dataset of 100 implicitly represented 3D objects with vision, audio, and touch.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, each object is represented by an Object File , a compact neural network that contains all the visual, acoustic, and tactile proﬁles for the object.
Revised: 3, each object is represented by an Object File , a compact neural network that contains all the visual, acoustic, and tactile proﬁles for the object.

Original: Querying it with extrinsic parameters we can obtain visual appearance of the object from different viewpoints, impact sound of the object at each position, and tactile reading of the object at every surface location.
Revised: Querying it with extrinsic parameters we can obtain visual appearance of the object from different viewpoints, impact sound of the object at each position, and tactile reading of the object at every surface location.

Original: In the following, we ﬁrst describe the source of our objects and the annotations (Sec. 3.1).
Revised: In the following, we ﬁrst describe the source of our objects and the annotations (Sec. 3.1).

Original: Then we present the simulation pipeline and how we encode the sensory data using implicit neural representations for vision (Sec. 3.2), audio (Sec. 3.3), and touch (Sec. 3.4), respectively.
Revised: Then we present the simulation pipeline and how we encode the sensory data using implicit neural representations for vision (Sec. 3.2), audio (Sec. 3.3), and touch (Sec. 3.4), respectively.

Original: 3.1 Objects
Revised: 3.1 Objects

Original: We collect 100 high quality 3D objects from online repositories including: 20 objects from 3D
Revised: We collect 100 high quality 3D objects from online repositories including: 20 objects from 3D

Original: Model Haven 2 , 28 objects from the YCB dataset 3 , and 52 objects from Google Scanned Objects 4 .
Revised: Model Haven 2 , 28 objects from the YCB dataset 3 , and 52 objects from Google Scanned Objects 4 .

Original: We select objects that are of realistic visual textures, approximately homogeneous material property, and with a material type that is mappable to one of the following categories: ceramic, glass, wood, plastic, iron, polycarbonate, and steel.
Revised: We select objects that are of realistic visual textures, approximately homogeneous material property, and with a material type that is mappable to one of the following categories: ceramic, glass, wood, plastic, iron, polycarbonate, and steel.

Original: We annotate each object with the material type, which will be used in audio simulation in Sec. 3.3.
Revised: We annotate each object with the material type, which will be used in audio simulation in Sec. 3.3.

Original: The dataset contains common household objects of diverse categories such as bowl, mug, cabinet, television, shelf, fork, and spoon.
Revised: The dataset contains common household objects of diverse categories such as bowl, mug, cabinet, television, shelf, fork, and spoon.

Original: See Supp.
Revised: See Supp.

Original: for details.
Revised: for details.

Original: 3.2 Vision
Revised: 3.2 Vision

Original: Simulation: We use Blender’s Cycles path tracer [46] to render images.
Revised: Simulation: We use Blender’s Cycles path tracer [46] to render images.

Original: For each object, we ﬁrst normalize it into a unit cube and use a point light source at a random location on a unit sphere.
Revised: For each object, we ﬁrst normalize it into a unit cube and use a point light source at a random location on a unit sphere.

Original: We then render images of the object on a white background from camera viewpoints randomly sampled on a full sphere.
Revised: We then render images of the object on a white background from camera viewpoints randomly sampled on a full sphere.

Original: See Fig.
Revised: See Fig.

Original: 2 for some example views of objects in our dataset.
Revised: 2 for some example views of objects in our dataset.

Original: VisionNet: After rendering images from different camera viewpoints under varied lighting conditions, we use VisionNet to encode the visual appearance for each object.
Revised: VisionNet: After rendering images from different camera viewpoints under varied lighting conditions, we use VisionNet to encode the visual appearance for each object.

Original: Following prior work on object-centric neural radiance ﬁelds [27], we represent each object as a 7D object-centric neural scattering function whose input is a 3D location x = ( x, y, z ) in the object coordinate frame and the lighting condition at that location ( ω i , ω o ) , where ω i =
Revised: Following prior work on object-centric neural radiance ﬁelds [27], we represent each object as a 7D object-centric neural scattering function whose input is a 3D location x = ( x, y, z ) in the object coordinate frame and the lighting condition at that location ( ω i , ω o ) , where ω i =

Original: ( φ i , θ i ) , ω
Revised: ( φ i , θ i ) , ω

Original: o = ( φ o , θ o ) denote the incoming and outgoing light directions, respectively.
Revised: o = ( φ o , θ o ) denote the incoming and outgoing light directions, respectively.

Original: The output is the volume density σ and fraction of the incoming light that is scattered in the outgoing direction ρ
Revised: The output is the volume density σ and fraction of the incoming light that is scattered in the outgoing direction ρ

Original: = ( ρ r , ρ g , ρ b ) .
Revised: = ( ρ r , ρ g , ρ b ) .

Original: VisionNet approximates this continuous 7D object-centric representation with an MLP network
Revised: VisionNet approximates this continuous 7D object-centric representation with an MLP network

Original: F v : ( x, y, z, θ i , φ i , θ o , φ o ) −→ ( σ, ρ r , ρ g , ρ b ) that maps each input 7D coordinate to its correspond-
Revised: F v : ( x, y, z, θ i , φ i , θ o , φ o ) −→ ( σ, ρ r , ρ g , ρ b ) that maps each input 7D coordinate to its correspond-

Original: [Figure]
Revised: [Figure]

Original: (", $
Revised: (", $

Original: [Figure]
Revised: [Figure]

Original: (∅
Revised: (∅

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Object File (Intrinsics)
Revised: Object File (Intrinsics)

Original: TouchNet
Revised: TouchNet

Original: VisionNet
Revised: VisionNet

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Figure]
Revised: [Figure]

Original: 3D coordinates
Revised: 3D coordinates

Original: Light directions
Revised: Light directions

Original: [Figure]
Revised: [Figure]

Original: Extrinsics density
Revised: Extrinsics density

Original: [Figure]
Revised: [Figure]

Original: ing volume density and fraction coefﬁcient for each RGB channel.
Revised: ing volume density and fraction coefﬁcient for each RGB channel.

Original: The amount of light scattered at a point x can be obtained as follows:
Revised: The amount of light scattered at a point x can be obtained as follows:

Original: [Equation]
Revised: [Equation]

Original: where S is a unit sphere, L ( x , ω i ) denotes the amount of light scattered at point x along direction ω
Revised: where S is a unit sphere, L ( x , ω i ) denotes the amount of light scattered at point x along direction ω

Original: i , and f ρ evaluates the fraction of light incoming from direction ω
Revised: i , and f ρ evaluates the fraction of light incoming from direction ω

Original: i at the point that scatters out in direction ω
Revised: i at the point that scatters out in direction ω

Original: o .
Revised: o .

Original: Then we use classic volume rendering [34] to render the color of any ray passing through the object.
Revised: Then we use classic volume rendering [34] to render the color of any ray passing through the object.

Original: The expected color C ( r ) of camera ray r ( t ) = x 0 + tω o can be obtained as follows:
Revised: The expected color C ( r ) of camera ray r ( t ) = x 0 + tω o can be obtained as follows:

Original: [Equation]
Revised: [Equation]

Original: T ( t ) denotes the accumulated transmittance along the ray from t n to t ; t n and t f are the near and far integration bounds; and σ ( r ( t )) denotes the volume density at location r ( t ) .
Revised: T ( t ) denotes the accumulated transmittance along the ray from t n to t ; t n and t f are the near and far integration bounds; and σ ( r ( t )) denotes the volume density at location r ( t ) .

Original: Similar to [26], we also use stratiﬁed sampling, positional encoding, and hierarchical volume sampling to increase rendering quality and efﬁciency.
Revised: Similar to [26], we also use stratiﬁed sampling, positional encoding, and hierarchical volume sampling to increase rendering quality and efﬁciency.

Original: See [26, 27] for details.
Revised: See [26, 27] for details.

Original: 3.3 Audio
Revised: 3.3 Audio

Original: Simulation: The goal of audio simulation is to realistically model the impact sound at each position of the object based on its shape, size, material type, external force, and the contact location.
Revised: Simulation: The goal of audio simulation is to realistically model the impact sound at each position of the object based on its shape, size, material type, external force, and the contact location.

Original: Following the standard approach in engineering and acoustics, we use linear modal analysis for physics-based rigid-body sound synthesis [23, 47].
Revised: Following the standard approach in engineering and acoustics, we use linear modal analysis for physics-based rigid-body sound synthesis [23, 47].

Original: Firstly, we convert the surface mesh of an object into a volumetric hexahedron mesh, which represents the original shape with N voxels.
Revised: Firstly, we convert the surface mesh of an object into a volumetric hexahedron mesh, which represents the original shape with N voxels.

Original: The key to modal analysis is to solve the following linear deformation equation for a 3D linear elastic dynamics model:
Revised: The key to modal analysis is to solve the following linear deformation equation for a 3D linear elastic dynamics model:

Original: [Equation]
Revised: [Equation]

Original: where u ∈ R 3 N denotes the displacement of elemental nodes in 3D; M , K ∈ R 3 N × 3 N denote the mass and stiffness matrices of the system; C = α M + β K stands for Rayleigh damping; and f ∈ R 3 N represents the external force applied to the object that stimulates the vibration.
Revised: where u ∈ R 3 N denotes the displacement of elemental nodes in 3D; M , K ∈ R 3 N × 3 N denote the mass and stiffness matrices of the system; C = α M + β K stands for Rayleigh damping; and f ∈ R 3 N represents the external force applied to the object that stimulates the vibration.

Original: Based on the material type of the object, we obtain its density value ρ , Young’s Modulus E , Poisson’s ratio r , and Rayleigh damping parameters α and β .
Revised: Based on the material type of the object, we obtain its density value ρ , Young’s Modulus E , Poisson’s ratio r , and Rayleigh damping parameters α and β .

Original: The scale of the object and material parameters ρ, E, r are used to build the mass and stiffness matrices M and K .
Revised: The scale of the object and material parameters ρ, E, r are used to build the mass and stiffness matrices M and K .

Original: See Supp.
Revised: See Supp.

Original: for how we map each material type to these parameters.
Revised: for how we map each material type to these parameters.

Original: The above equation can be decoupled into the following form through generalized eigenvalue decomposition KU = ΛMU :
Revised: The above equation can be decoupled into the following form through generalized eigenvalue decomposition KU = ΛMU :

Original: [Equation]
Revised: [Equation]

Original: where u = Uq and Λ is a diagonal matrix.
Revised: where u = Uq and Λ is a diagonal matrix.

Original: The solution to the above equation is a bank of damped sinusoidal waves with each wave representing a mode signal:
Revised: The solution to the above equation is a bank of damped sinusoidal waves with each wave representing a mode signal:

Original: [Equation]
Revised: [Equation]

Original: where ω i is the frequency of the mode,
Revised: where ω i is the frequency of the mode,

Original: c i is the dampling coefﬁcient, a i is the excited amplitude, and θ i is the initial phase.
Revised: c i is the damping coefﬁcient, a i is the excited amplitude, and θ i is the initial phase.

Original: Assuming the object is stationary originally, we get the following solution:
Revised: Assuming the object is stationary originally, we get the following solution:

Original: [Equation]
Revised: [Equation]

Original: λ i represents the i th mode eigenvalue.
Revised: λ i represents the i th mode eigenvalue.

Original: We simulate the vibration modes excited by unit forces f 1 , f 2 , f 3 for all vertexes and use the obtained modes signals for training AudioNet as detailed next.
Revised: We simulate the vibration modes excited by unit forces f 1 , f 2 , f 3 for all vertexes and use the obtained modes signals for training AudioNet as detailed next.

Original: AudioNet:
Revised: AudioNet:

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, for each force direction, we train a separate branch to encode the corresponding modes signals at all vertexes.
Revised: 3, for each force direction, we train a separate branch to encode the corresponding modes signals at all vertexes.

Original: We obtain the audio spectrogram 5 for each modes signal and train a MLP to predict the complex spectrogram of dimension F × T × 2 , where F and
Revised: We obtain the audio spectrogram 5 for each modes signal and train a MLP to predict the complex spectrogram of dimension F × T × 2 , where F and

Original: T are the frequency and time dimensions.
Revised: T are the frequency and time dimensions.

Original: Particularly, each branch takes the spatial coordinates ( x, y, z ) 6 and the spectrogram location ( f, t ) as input, and predict the real and imagninary part of the complex number for every location in the audio spectrogram.
Revised: Particularly, each branch takes the spatial coordinates ( x, y, z ) 6 and the spectrogram location ( f, t ) as input, and predict the real and imaginary part of the complex number for every location in the audio spectrogram.

Original: The modes signal can then be recovered from the spectrogram using inverse short-time Fourier transform (ISTFT).
Revised: The modes signal can then be recovered from the spectrogram using inverse short-time Fourier transform (ISTFT).

Original: We also use positional encoding as in VisionNet.
Revised: We also use positional encoding as in VisionNet.

Original: During test time, we directly use vibration modes for sound synthesis following [47].
Revised: During test time, we directly use vibration modes for sound synthesis following [47].

Original: Any external force at a vertex can be decomposed into a linear combination of unit forces along the three orthogonal directions:
Revised: Any external force at a vertex can be decomposed into a linear combination of unit forces along the three orthogonal directions:

Original: f = k 1 f 1 + k 2 f 

Revised: f = k 1 f 1 + k 2 f

Original: + k 3 f 3 .
Revised: + k 3 f 3 .

Original: The amplitudes of modes a excited by f can be decomposed into a linear combination of the amplitudes excited by the unit force: a = k 1 a 1 + k 2 a 

Revised: The amplitudes of modes a excited by f can be decomposed into a linear combination of the amplitudes excited by the unit force: a = k 1 a 1 + k 2 a

Original: + k 3 a 3 , where a 1 , a 2 , a 3 denote the amplitudes of modes excited by unit forces f 1 , f 2 , f 3 for a given vertex.
Revised: + k 3 a 3 , where a 1 , a 2 , a 3 denote the amplitudes of modes excited by unit forces f 1 , f 2 , f 3 for a given vertex.

Original: Because modal analysis is performed on a volumetric hexahedron mesh, for a vertex in the original polygon mesh, we ﬁnd its closest four vertexes in the hexahedron and average their modes signals.
Revised: Because modal analysis is performed on a volumetric hexahedron mesh, for a vertex in the original polygon mesh, we ﬁnd its closest four vertexes in the hexahedron and average their modes signals.

Original: Prior work [23, 48] have demonstrated that such physics-based rigid-body sound synthesis pipeline renders sound that transfers well to real scenes and the objects’ shape, material, and other auditory attributes can be successfully inferred from the simulated sound.
Revised: Prior work [23, 48] have demonstrated that such physics-based rigid-body sound synthesis pipeline renders sound that transfers well to real scenes and the objects’ shape, material, and other auditory attributes can be successfully inferred from the simulated sound.

Original: To further demonstrate the realism of our audio simulation, we also perform a user study on acoustic ﬁdelity that asks participants to distinguish between the real and simulated audios.
Revised: To further demonstrate the realism of our audio simulation, we also perform a user study on acoustic ﬁdelity that asks participants to distinguish between the real and simulated audios.

Original: Results show that our simulated audio is preferred by 42% of the total responses, comparing closely to real audio recordings.
Revised: Results show that our simulated audio is preferred by 42% of the total responses, comparing closely to real audio recordings.

Original: See Supp.
Revised: See Supp.

Original: for details.
Revised: for details.

Original: 3.4 Touch
Revised: 3.4 Touch

Original: Simulation: We leverage a state-of-the-art touch simulator TACTO [25] for touch simulation.
Revised: Simulation: We leverage a state-of-the-art touch simulator TACTO [25] for touch simulation.

Original: TACTO is a vision-based touch simulator and uses ray-tracing to simulate high-quality tactile signals.
Revised: TACTO is a vision-based touch simulator and uses ray-tracing to simulate high-quality tactile signals.

Original: It can simulate realistic rendering for both contact surfaces and shadows with different contact geometry.
Revised: It can simulate realistic rendering for both contact surfaces and shadows with different contact geometry.

Original: See [25] for details and a comparison of the simulated and real tactile readings.
Revised: See [25] for details and a comparison of the simulated and real tactile readings.

Original: Particularly, we use DIGIT [24] touch sensor and its corresponding camera, lights, and gel mesh conﬁgurations in simulation.
Revised: Particularly, we use DIGIT [24] touch sensor and its corresponding camera, lights, and gel mesh conﬁgurations in simulation.

Original: We choose DIGIT for its compactness, high-resolution of its representation, and potential applications for robotic in-hand manipulation.
Revised: We choose DIGIT for its compactness, high-resolution of its representation, and potential applications for robotic in-hand manipulation.

Original: Other touch sensors such as Gelsight [49],
Revised: Other touch sensors such as Gelsight [49],

Original: BioTac [50], or Ominitact [51] can also be potentially used, and the design of the touch sensor to use is orthogonal to our work.
Revised: BioTac [50], or Ominitact [51] can also be potentially used, and the design of the touch sensor to use is orthogonal to our work.

Original: To obtain touch simulations, we place each object at rest and move
Revised: To obtain touch simulations, we place each object at rest and move

Original: DIGIT to touch each object vertex in vertex normal direction.
Revised: DIGIT to touch each object vertex in vertex normal direction.

Original: We constrain the force to range within a small threshold such that the rendered tactile images do not vary signiﬁcantly with different forces.
Revised: We constrain the force to range within a small threshold such that the rendered tactile images do not vary signiﬁcantly with different forces.

Original: We use the RGB tactile image that contains the local contact geometry as our touch representation.
Revised: We use the RGB tactile image that contains the local contact geometry as our touch representation.

Original: TouchNet:
Revised: TouchNet:

Original: For each object, we obtain a RGB tactile image of dimension W × H × 3 that captures the local geometry information for every vertex on the surface of the polygon mesh.
Revised: For each object, we obtain a RGB tactile image of dimension W × H × 3 that captures the local geometry information for every vertex on the surface of the polygon mesh.

Original: TouchNet takes
Revised: TouchNet takes

Original: [Figure]
Revised: [Figure]

Original: the spatial coordinate ( x, y, z ) of the vertex and the spatial location ( w,
Revised: the spatial coordinate ( x, y, z ) of the vertex and the spatial location ( w,

Original: ) in the tactile as input, and predicts the per-pixel value for the three channels of the tactile RGB image.
Revised: ) in the tactile as input, and predicts the per-pixel value for the three channels of the tactile RGB image.

Original: The other settings are similar to VisionNet and AudioNet.
Revised: The other settings are similar to VisionNet and AudioNet.

Original: Implicit
Revised: Implicit

Original: Original
Revised: Original

Original: [Table]
Revised: [Table]

Original: Touch
Revised: Touch

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 4 shows some examples of the visual, auditory, and tactile sensory data obtained from Object Files.
Revised: 4 shows some examples of the visual, auditory, and tactile sensory data obtained from Object Files.

Original: The rendered images, impact sounds, and touch readings well match the ground-truth simulations.
Revised: The rendered images, impact sounds, and touch readings well match the ground-truth simulations.

Original: See Supp.
Revised: See Supp.

Original: for a quantitative comparison on the accuracy of the implicit representations.
Revised: for a quantitative comparison on the accuracy of the implicit representations.

Original: Moreover, using implicit neural representations is much more storage efﬁcient compared to the original data format.
Revised: Moreover, using implicit neural representations is much more storage efﬁcient compared to the original data format.

Original: Tableshows a comparison between the storage size with and without using implicit representations.
Revised: Tableshows a comparison between the storage size with and without using implicit representations.

Original: Object File in the form of an implicit neural network is a much more compact representation of the multisensory data for each object.
Revised: Object File in the form of an implicit neural network is a much more compact representation of the multisensory data for each object.

Original: Note that VisionNet models an object-centric neural scattering function [27] and thus can generalize to any camera viewpoint under arbitrary lighting conditions, while rendering and saving images for all scenarios takes inﬁnite space theoretically.
Revised: Note that VisionNet models an object-centric neural scattering function [27] and thus can generalize to any camera viewpoint under arbitrary lighting conditions, while rendering and saving images for all scenarios takes inﬁnite space theoretically.

Original:  Experiments
Revised:  Experiments

Original: O BJECT F OLDER is a potential testbed for many perception and control tasks.
Revised: O BJECT F OLDER is a potential testbed for many perception and control tasks.

Original: Next, we evaluate on four benchmark tasks including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping to demonstrate the usefulness of the dataset and the value of multisensory perception.
Revised: Next, we evaluate on four benchmark tasks including instance recognition, cross-sensory retrieval, 3D reconstruction, and robotic grasping to demonstrate the usefulness of the dataset and the value of multisensory perception.

Original: We show the key results below, and see Supp.
Revised: We show the key results below, and see Supp.

Original: for the detailed experiments setup.
Revised: for the detailed experiments setup.

Original: 4.1 Multisensory Instance Recognition
Revised: 4.1 Multisensory Instance Recognition

Original: Methods
Revised: Methods

Original: [Table]
Revised: [Table]

Original: Chance
Revised: Chance

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Touch
Revised: Touch

Original: [Table] [Table]
Revised: [Table] [Table]

Original: V + A 99.
Revised: V + A 99.

Original: [Table]
Revised: [Table]

Original: Identifying the object instance that is interacted with is fundamental for many robotic applications.
Revised: Identifying the object instance that is interacted with is fundamental for many robotic applications.

Original: In this task, we want to identify an object based on either its visual appearance, impact sound, or local contact geometry.
Revised: In this task, we want to identify an object based on either its visual appearance, impact sound, or local contact geometry.

Original: We are interested in ﬁnding out the amount of information that is useful to recognize the object instance from each modality.
Revised: We are interested in ﬁnding out the amount of information that is useful to recognize the object instance from each modality.

Original: For the vision modality, we train a ResNet-18 [52] network that takes an RGB image of the object as input and predicts the instance label of the object.
Revised: For the vision modality, we train a ResNet-18 [52] network that takes an RGB image of the object as input and predicts the instance label of the object.

Original: The settings are the same for audio and touch except that the input is either a magnitude spectrogram of the impact sound the object generates or a local tactile RGB image from a random location on the object surface.
Revised: The settings are the same for audio and touch except that the input is either a magnitude spectrogram of the impact sound the object generates or a local tactile RGB image from a random location on the object surface.

Original: Table 2 shows the results.
Revised: Table 2 shows the results.

Original: The accuracy for a random classiﬁer is 1%.
Revised: The accuracy for a random classiﬁer is 1%.

Original: The vision classiﬁer achieves high accuracy, because an image of the object normally captures informative appearance cues that distinguish the object instance.
Revised: The vision classiﬁer achieves high accuracy, because an image of the object normally captures informative appearance cues that distinguish the object instance.

Original: Under some camera viewpoints, the vision classiﬁer can mistakenly classify the object.
Revised: Under some camera viewpoints, the vision classiﬁer can mistakenly classify the object.

Original: Surprisingly, the audio classiﬁer achieves the best results.
Revised: Surprisingly, the audio classiﬁer achieves the best results.

Original: This conﬁrms that our audio simulation pipeline well models the shape, size, and material property of the object instance, so that a single impact sound can be used to recognize the object.
Revised: This conﬁrms that our audio simulation pipeline well models the shape, size, and material property of the object instance, so that a single impact sound can be used to recognize the object.

Original: Touch is good at capturing local geometry of an object, but often does not contain sufﬁcient discriminative cues to recognize the object especially from a single touch.
Revised: Touch is good at capturing local geometry of an object, but often does not contain sufﬁcient discriminative cues to recognize the object especially from a single touch.

Original: Therefore, the touch classiﬁer performs worse compared to vision and audio.
Revised: Therefore, the touch classiﬁer performs worse compared to vision and audio.

Original: Classiﬁers with multisensory input are more robust, and combining all three modalities leads to the best result.
Revised: Classiﬁers with multisensory input are more robust, and combining all three modalities leads to the best result.

Original: 4.2 Cross-Sensory Retrieval
Revised: 4.2 Cross-Sensory Retrieval

Original: Cross-sensory retrieval plays a crucial role in machine perception to understand the relationships between different modalities.
Revised: Cross-sensory retrieval plays a crucial role in machine perception to understand the relationships between different modalities.

Original: Fig.
Revised: Fig.

Original: 5 shows our framework for learning cross-sensory embeddings for two modalities.
Revised: 5 shows our framework for learning cross-sensory embeddings for two modalities.

Original: Taking cross-sensory learning with vision and audio as an example, we design a two-stream network that takes an RGB image and an audio spectrogram of the same object as input for each stream.
Revised: Taking cross-sensory learning with vision and audio as an example, we design a two-stream network that takes an RGB image and an audio spectrogram of the same object as input for each stream.

Original: The two modalities have disjoint pathways in the early layers to capture the modality-speciﬁc features.
Revised: The two modalities have disjoint pathways in the early layers to capture the modality-speciﬁc features.

Original: A ﬁnal modality-agnostic network is used to map the shared embedding to the label space for training with cross-entropy loss.
Revised: A ﬁnal modality-agnostic network is used to map the shared embedding to the label space for training with cross-entropy loss.

Original: We map the features of both modalities from the last hidden state to a cross-sensory embedding space through a triplet loss.
Revised: We map the features of both modalities from the last hidden state to a cross-sensory embedding space through a triplet loss.

Original: We either sample an image or audio from another object as the negative example, and pull the embeddings of the same object to be closer in the joint embedding space and push the embeddings from different objects apart.
Revised: We either sample an image or audio from another object as the negative example, and pull the embeddings of the same object to be closer in the joint embedding space and push the embeddings from different objects apart.

Original: The full model is trained jointly with both the triplet loss and the cross-entropy loss.
Revised: The full model is trained jointly with both the triplet loss and the cross-entropy loss.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Retrieved Chance
Revised: Retrieved Chance

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Ours
Revised: Ours

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Touch
Revised: Touch

Original: [Table]
Revised: [Table]

Original: Audio
Revised: Audio

Original: [Table]
Revised: [Table]

Original: Table 3 shows the results for cross-sensory retrieval on the test set, where we retrieve the sensory data of a different modality by querying from the shared embedding space.
Revised: Table 3 shows the results for cross-sensory retrieval on the test set, where we retrieve the sensory data of a different modality by querying from the shared embedding space.

Original: We use cosine similarity to measure the distances between embeddings and retrieve the sample with the smallest distance.
Revised: We use cosine similarity to measure the distances between embeddings and retrieve the sample with the smallest distance.

Original: We report mean Average Precision (mAP) which jointly considers the ranking information and the precision, and compare with random retrieval and a state-of-the-art cross-sensory retrieval method based on canonical correlation analysis (CCA) [53].
Revised: We report mean Average Precision (mAP) which jointly considers the ranking information and the precision, and compare with random retrieval and a state-of-the-art cross-sensory retrieval method based on canonical correlation analysis (CCA) [53].

Original: Our models perform more reliable retrievals, and the results show that the learned cross-sensory embeddings are effective and contain ﬁne-grained information about objects to correctly retrieve sensory data from the matched object.
Revised: Our models perform more reliable retrievals, and the results show that the learned cross-sensory embeddings are effective and contain ﬁne-grained information about objects to correctly retrieve sensory data from the matched object.

Original: Similar to instance recognition, we observe vision and audio provide more reliable global information about the object compared to touch.
Revised: Similar to instance recognition, we observe vision and audio provide more reliable global information about the object compared to touch.

Original: 4.3 Audio-Visual 3D Reconstruction
Revised: 4.3 Audio-Visual 3D Reconstruction

Original: Robots often need to build a mental model of the 3D shape of objects based on just a single glimpse or the sounds they make during interaction.
Revised: Robots often need to build a mental model of the 3D shape of objects based on just a single glimpse or the sounds they make during interaction.

Original: In this section, we investigate the potential of 3D reconstruction using either a single image, an impact sound, or their combination.
Revised: In this section, we investigate the potential of 3D reconstruction using either a single image, an impact sound, or their combination.

Original: We leverage Occupancy
Revised: We leverage Occupancy

Original: Network [30], which implicitly represents the 3D surface as the continuous decision boundary of a deep neural network classiﬁer, as a testbed for this task.
Revised: Network [30], which implicitly represents the 3D surface as the continuous decision boundary of a deep neural network classiﬁer, as a testbed for this task.

Original: Fig.
Revised: Fig.

Original: 6 illustrates our I MAGE +A UDIO 2M ESH framework for audio-visual 3D shape reconstruction.
Revised: 6 illustrates our I MAGE +A UDIO 2M ESH framework for audio-visual 3D shape reconstruction.

Original: The image and audio embeddings are fused through a fusion layer into an audio-visual feature, then combined with coordinate-conditioned feature maps through Conditional Batch-Normalization to predict occupancy probabilities p .
Revised: The image and audio embeddings are fused through a fusion layer into an audio-visual feature, then combined with coordinate-conditioned feature maps through Conditional Batch-Normalization to predict occupancy probabilities p .

Original: We also test two of its simpliﬁed variants I MAGE 2M ESH , which represents existing single image 3D reconstruction approaches that predict the 3D shape from a single view of the object, and A UDIO 2M ESH , which performs 3D reconstruction purely from audio.
Revised: We also test two of its simpliﬁed variants I MAGE 2M ESH , which represents existing single image 3D reconstruction approaches that predict the 3D shape from a single view of the object, and A UDIO 2M ESH , which performs 3D reconstruction purely from audio.

Original: [Table]
Revised: [Table]

Original: We evaluate on a held out test set with standard metrics: IoU, Chamfer-
Revised: We evaluate on a held out test set with standard metrics: IoU, Chamfer-

Original: L 1 distance, and Normal
Revised: L 1 distance, and Normal

Original: Consistency with respect to the ground-truth mesh.
Revised: Consistency with respect to the ground-truth mesh.

Original: Table 4 shows the results.
Revised: Table 4 shows the results.

Original: Compared to the simple A VERAGE baseline that averages the results obtained using the ground-truth mesh of each of the 100 objects as the prediction, we can see that 3D reconstruction from a single image or an impact sound performs much better.
Revised: Compared to the simple A VERAGE baseline that averages the results obtained using the ground-truth mesh of each of the 100 objects as the prediction, we can see that 3D reconstruction from a single image or an impact sound performs much better.

Original: Augmenting traditional single image 3d reconstruction with audio achieves the best performance by leveraging the additional acoustic spatial cues.
Revised: Augmenting traditional single image 3d reconstruction with audio achieves the best performance by leveraging the additional acoustic spatial cues.

Original: Furthermore, as shown in Fig. 7, we apply our I MAGE 2M ESH model trained on objects in O BJECT F OLDER to real-world images.
Revised: Furthermore, as shown in Fig. 7, we apply our I MAGE 2M ESH model trained on objects in O BJECT F OLDER to real-world images.

Original: Our model generalizes reasonably well to real image, demonstrating the realism of the objects in our dataset.
Revised: Our model generalizes reasonably well to real image, demonstrating the realism of the objects in our dataset.

Original: The last column shows a typical failure case, where the teapot image is very visually different from objects in our dataset.
Revised: The last column shows a typical failure case, where the teapot image is very visually different from objects in our dataset.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4.4 Robotic Grasping with Vision and Touch
Revised: 4.4 Robotic Grasping with Vision and Touch

Original: Next we show how we combine vision and touch for prediction of grasp stability.
Revised: Next we show how we combine vision and touch for prediction of grasp stability.

Original: The goal is to use a robotic arm to successfully grasp and hold an object between the robot’s left and right ﬁngers.
Revised: The goal is to use a robotic arm to successfully grasp and hold an object between the robot’s left and right ﬁngers.

Original: For touch data, we infer the RGB tactile image from the corresponding Object File based on the grasp contact positions of the left and right grippers on the object; For vision data, we obtain images from an externally mounted camera’s viewpoint at the grasping moment.
Revised: For touch data, we infer the RGB tactile image from the corresponding Object File based on the grasp contact positions of the left and right grippers on the object; For vision data, we obtain images from an externally mounted camera’s viewpoint at the grasping moment.

Original: We label the data as either “Success” or “Failure” based on whether the object can be held between the robot’s ﬁngers after being lifted.
Revised: We label the data as either “Success” or “Failure” based on whether the object can be held between the robot’s ﬁngers after being lifted.

Original: Fig.
Revised: Fig.

Original: 8 shows an example of both a successful grasp and a failure case.
Revised: 8 shows an example of both a successful grasp and a failure case.

Original: We collect data for 10 objects from O BJECT F OLDER , and then train binary classiﬁers under different sample sizes to predict the grasp outcome based on either the vision signal, the touch signal, or their combination.
Revised: We collect data for 10 objects from O BJECT F OLDER , and then train binary classiﬁers under different sample sizes to predict the grasp outcome based on either the vision signal, the touch signal, or their combination.

Original: This experiment is similar to that of [44, 25] except that we obtain touch readings directly by querying our implicit representation networks.
Revised: This experiment is similar to that of [44, 25] except that we obtain touch readings directly by querying our implicit representation networks.

Original: Fig.
Revised: Fig.

Original: 9 shows the results.
Revised: 9 shows the results.

Original: We observe that it takes signiﬁcantly less amount of data to reach high accuracy when learning from touch compared to vision.
Revised: We observe that it takes signiﬁcantly less amount of data to reach high accuracy when learning from touch compared to vision.

Original: Leveraging both visual and tactile data achieves the best accuracy.
Revised: Leveraging both visual and tactile data achieves the best accuracy.

Original: Additionally, we train a policy for touch-based robotic grasping using TRPO [54].
Revised: Additionally, we train a policy for touch-based robotic grasping using TRPO [54].

Original: We achieve a success rate of 75 .
Revised: We achieve a success rate of 75 .

Original: 5% during testing, while that of a random grasping policy is 53 . 0% .
Revised: 5% during testing, while that of a random grasping policy is 53 . 0% .

Original: To further demonstrate the potential of using our dataset on robot manipulation tasks, we also perform an experiment on an object manipulation task (reach) using Meta-World [55] with three of our objects (cup, bowl, dice).
Revised: To further demonstrate the potential of using our dataset on robot manipulation tasks, we also perform an experiment on an object manipulation task (reach) using Meta-World [55] with three of our objects (cup, bowl, dice).

Original: We achieve 100% success rate for each object.
Revised: We achieve 100% success rate for each object.

Original: See Supp.
Revised: See Supp.

Original: for details.
Revised: for details.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: We presented O BJECT F OLDER , a dataset of objects with implicit visual, auditory, and tactile representations.
Revised: We presented O BJECT F OLDER , a dataset of objects with implicit visual, auditory, and tactile representations.

Original: It will facilitate research with vision, audio, and touch, and enable multisensory simulation of objects in robotic virtual environments.
Revised: It will facilitate research with vision, audio, and touch, and enable multisensory simulation of objects in robotic virtual environments.

Original: Through evaluation on an array of benchmark tasks, we show the usefulness of our dataset for both perception and control.
Revised: Through evaluation on an array of benchmark tasks, we show the usefulness of our dataset for both perception and control.

Original: As future work, we plan to explore richer object states with more accurate and ﬁne-grained physics, additional sensory modalities, and textual descriptions.
Revised: As future work, we plan to explore richer object states with more accurate and ﬁne-grained physics, additional sensory modalities, and textual descriptions.

Original: Acknowledgements: We thank Michelle Guo, Xutong Jin, Shaoxiong Wang, Huazhe Xu and
Revised: Acknowledgements: We thank Michelle Guo, Xutong Jin, Shaoxiong Wang, Huazhe Xu and

Original: Samuel Clarke for helpful discussions on experiments setup and Michael Lingelbach and Jingwei
Revised: Samuel Clarke for helpful discussions on experiments setup and Michael Lingelbach and Jingwei

Original: Ji for suggestions on paper drafts.
Revised: Ji for suggestions on paper drafts.

Original: The work is in part supported by Toyota Research Institute (TRI), Samsung Global Research Outreach (GRO) program, ARMY MURI grant W911NF-15-10479, NSF CCRI #2120095, Amazon Research Award (ARA), Autodesk, Qualcomm and Stanford
Revised: The work is in part supported by Toyota Research Institute (TRI), Samsung Global Research Outreach (GRO) program, ARMY MURI grant W911NF-15-10479, NSF CCRI #2120095, Amazon Research Award (ARA), Autodesk, Qualcomm and Stanford

Original: Institute for Human-Centered AI (HAI).
Revised: Institute for Human-Centered AI (HAI).