Original: 3D N EURAL E MBEDDING L IKELIHOOD FOR R OBUST S IM - TO -R EAL T RANSFER IN I NVERSE G RAPHICS
Revised: 3D N EURAL E MBEDDING L IKELIHOOD FOR R OBUST S IM - TO -R EAL T RANSFER IN I NVERSE G RAPHICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A central challenge in 3D scene perception via inverse graphics is robustly modeling the gap between 3D graphics and real-world data.
Revised: A central challenge in 3D scene perception via inverse graphics is robustly modeling the gap between 3D graphics and real-world data.

Original: We propose a novel 3D
Revised: We propose a novel 3D

Original: Neural Embedding Likelihood (3DNEL) over RGB-D images to address this gap.
Revised: Neural Embedding Likelihood (3DNEL) over RGB-D images to address this gap.

Original: 3DNEL uses neural embeddings to predict 2D-3D correspondences from RGB and combines this with depth in a principled manner.
Revised: 3DNEL uses neural embeddings to predict 2D-3D correspondences from RGB and combines this with depth in a principled manner.

Original: 3DNEL is trained entirely from synthetic images and generalizes to real-world data.
Revised: 3DNEL is trained entirely from synthetic images and generalizes to real-world data.

Original: To showcase this capability, we develop a multi-stage inverse graphics pipeline that uses 3DNEL for 6D object pose estimation from real RGB-D images.
Revised: To showcase this capability, we develop a multi-stage inverse graphics pipeline that uses 3DNEL for 6D object pose estimation from real RGB-D images.

Original: Our method outperforms the previous state-of-the-art in sim-to-real pose estimation on the YCB-Video dataset, and improves robustness, with significantly fewer large-error predictions.
Revised: Our method outperforms the previous state-of-the-art in sim-to-real pose estimation on the YCB-Video dataset, and improves robustness, with significantly fewer large-error predictions.

Original: Unlike existing bottom-up, discriminative approaches that are specialized for pose estimation, 3DNEL adopts a probabilistic generative formulation that jointly models multi-object scenes.
Revised: Unlike existing bottom-up, discriminative approaches that are specialized for pose estimation, 3DNEL adopts a probabilistic generative formulation that jointly models multi-object scenes.

Original: This generative formulation enables easy extension of 3DNEL to additional tasks like object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.
Revised: This generative formulation enables easy extension of 3DNEL to additional tasks like object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is a widespread need for models that bridge the gap between 3D graphics and real RGB-D data.
Revised: There is a widespread need for models that bridge the gap between 3D graphics and real RGB-D data.

Original: Accurate simulation environments exist in domains such as autonomous driving, augmented reality, and robotic manipulation, yet robust 3D scene perception remains a central bottleneck.
Revised: Accurate simulation environments exist in domains such as autonomous driving, augmented reality, and robotic manipulation, yet robust 3D scene perception remains a central bottleneck.

Original: “Inverse graphics” is an appealing approach to 3D scene understanding that treats scene perception as the inverse problem to 3D graphics.
Revised: “Inverse graphics” is an appealing approach to 3D scene understanding that treats scene perception as the inverse problem to 3D graphics.

Original: However, in practice these methods have been outperformed by more bottom-up, discriminative approaches, especially those using deep learning (LeCun et al., 2015).
Revised: However, in practice these methods have been outperformed by more bottom-up, discriminative approaches, especially those using deep learning (LeCun et al., 2015).

Original: A key challenge in inverse graphics is modeling the gap between rendered images and observed real-world images.
Revised: A key challenge in inverse graphics is modeling the gap between rendered images and observed real-world images.

Original: This paper aims to address this gap with 3D Neural
Revised: This paper aims to address this gap with 3D Neural

Original: Embedding Likelihood (3DNEL), a likelihood model of RGB-D images that is trained entirely from synthetic data and generalizes to real-world data.
Revised: Embedding Likelihood (3DNEL), a likelihood model of RGB-D images that is trained entirely from synthetic data and generalizes to real-world data.

Original: 3DNEL uses learned neural embeddings to predict dense 2D-3D correspondences from RGB and combines this with 3D information from depth in a principled way.
Revised: 3DNEL uses learned neural embeddings to predict dense 2D-3D correspondences from RGB and combines this with 3D information from depth in a principled way.

Original: To showcase 3DNEL’s capabilities in sim-to-real transfer, we develop a multi-stage inverse graphics pipeline (MSIGP) that uses 3DNEL for 6D object pose estimation.
Revised: To showcase 3DNEL’s capabilities in sim-to-real transfer, we develop a multi-stage inverse graphics pipeline (MSIGP) that uses 3DNEL for 6D object pose estimation.

Original: We demonstrate that 3DNEL can be applied to real RGB-D images without having to train on any real data.
Revised: We demonstrate that 3DNEL can be applied to real RGB-D images without having to train on any real data.

Original: Our 3DNEL MSIGP consists of (1) a coarse enumerative procedure that generates pose hypotheses and an initial estimate of the 3D scene and (2) an iterative Markov Chain Monte Carlo (MCMC) process that finetunes the 3D scene.
Revised: Our 3DNEL MSIGP consists of (1) a coarse enumerative procedure that generates pose hypotheses and an initial estimate of the 3D scene and (2) an iterative Markov Chain Monte Carlo (MCMC) process that finetunes the 3D scene.

Original: We empirically evaluate 3DNEL MSIGP on the popular YCB-Video (YCB-V) dataset (Xiang et al., 2018).
Revised: We empirically evaluate 3DNEL MSIGP on the popular YCB-Video (YCB-V) dataset (Xiang et al., 2018).

Original: 3DNEL MSIGP outperforms the previous state-of-the-art (SOTA) SurfEMB (Haugaard & Buch, 2022) in sim-to-real 6D pose estimation, albeit at the cost of increased computation.
Revised: 3DNEL MSIGP outperforms the previous state-of-the-art (SOTA) SurfEMB (Haugaard & Buch, 2022) in sim-to-real 6D pose estimation, albeit at the cost of increased computation.

Original: It is also significantly more robust: we show over 50% reduction in high-error pose predictions compared to SurfEMB.
Revised: It is also significantly more robust: we show over 50% reduction in high-error pose predictions compared to SurfEMB.

Original: Extensive ablation studies illustrate the source of performance improvements.
Revised: Extensive ablation studies illustrate the source of performance improvements.

Original: Existing approaches for 6D pose estimation are predominantly discriminative and bottom-up, and are specialized to the specific task of 6D pose estimation.
Revised: Existing approaches for 6D pose estimation are predominantly discriminative and bottom-up, and are specialized to the specific task of 6D pose estimation.

Original: In contrast, 3DNEL adopts a probabilistic generative formulation which extends beyond just pose estimation.
Revised: In contrast, 3DNEL adopts a probabilistic generative formulation which extends beyond just pose estimation.

Original: To demonstrate the value of 3DNEL’s generative formulation, we present additional experiments on 3DNEL’s easy extension to object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.
Revised: To demonstrate the value of 3DNEL’s generative formulation, we present additional experiments on 3DNEL’s easy extension to object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: 3D Inverse Graphics
Revised: 3D Inverse Graphics

Original: Our method follows a long line of work in the “analysis-by-synthesis” paradigm that treats perception as the inverse problem to computer graphics (Kersten & Yuille, 1996; Yuille & Kersten, 2006; Lee & Mumford, 2003; Kersten et al., 2004; Mansinghka et al., 2013; Kulkarni et al., 2015).
Revised: Our method follows a long line of work in the “analysis-by-synthesis” paradigm that treats perception as the inverse problem to computer graphics (Kersten & Yuille, 1996; Yuille & Kersten, 2006; Lee & Mumford, 2003; Kersten et al., 2004; Mansinghka et al., 2013; Kulkarni et al., 2015).

Original: While elegant and conceptually appealing, robustly modeling the gap between 3D graphics and real-world data, especially using appearance information, remains a central challenge in 3D inverse graphics.
Revised: While elegant and conceptually appealing, robustly modeling the gap between 3D graphics and real-world data, especially using appearance information, remains a central challenge in 3D inverse graphics.

Original: Our key observation is that, dense 2D-3D correspondences, widely used in many recent 6D pose estimation methods (Hodan et al., 2020;
Revised: Our key observation is that, dense 2D-3D correspondences, widely used in many recent 6D pose estimation methods (Hodan et al., 2020;

Original: Li et al., 2019; He et al., 2020; Tremblay et al., 2018; Florence et al., 2018; Haugaard & Buch, 2022), provide a natural way to model appearance information in 3D inverse graphics, and can be combined with depth information into a unified probabilistic generative model to effectively bridge the sim-to-real gap.
Revised: Li et al., 2019; He et al., 2020; Tremblay et al., 2018; Florence et al., 2018; Haugaard & Buch, 2022), provide a natural way to model appearance information in 3D inverse graphics, and can be combined with depth information into a unified probabilistic generative model to effectively bridge the sim-to-real gap.

Original: Sim-to-Real Transfer Most computer vision systems require annotated real-world data for training (Hodan et al., 2017; Brachmann et al., 2014; Xiang et al., 2018;
Revised: Sim-to-Real Transfer Most computer vision systems require annotated real-world data for training (Hodan et al., 2017; Brachmann et al., 2014; Xiang et al., 2018;

Original: Krizhevsky et al., 2017) in order to achieve strong performance on real-world data at test time, since without it the “sim-to-real gap” is too difficult to overcome.
Revised: Krizhevsky et al., 2017) in order to achieve strong performance on real-world data at test time, since without it the “sim-to-real gap” is too difficult to overcome.

Original: Practically, it is tedious and expensive to collect and annotate real data.
Revised: Practically, it is tedious and expensive to collect and annotate real data.

Original: Recent advances in photorealistic rendering and physics-based simulations (Hodan et al., 2018;
Revised: Recent advances in photorealistic rendering and physics-based simulations (Hodan et al., 2018;

Original: Tremblay et al., 2018; Denninger et al., 2019) have resulted in strong performance of models trained entirely on synthetic data.
Revised: Tremblay et al., 2018; Denninger et al., 2019) have resulted in strong performance of models trained entirely on synthetic data.

Original: SurfEMB (Haugaard & Buch, 2022) is one such model that obtains SOTA performance in 6D pose estimation, outperforming numerous approaches that use real data for training.
Revised: SurfEMB (Haugaard & Buch, 2022) is one such model that obtains SOTA performance in 6D pose estimation, outperforming numerous approaches that use real data for training.

Original: 3DNEL builds upon SurfEMB’s learned dense 2D-3D correspondences, and expands previous work (Gothoskar et al., 2021) on probabilistic modeling of real depth data.
Revised: 3DNEL builds upon SurfEMB’s learned dense 2D-3D correspondences, and expands previous work (Gothoskar et al., 2021) on probabilistic modeling of real depth data.

Original: It combines RGB and depth information into a unified probabilistic generative model, and improves SurfEMB nontrivially in both accuracy and robustness to achieve new SOTA in sim-to-real 6D pose estimation.
Revised: It combines RGB and depth information into a unified probabilistic generative model, and improves SurfEMB nontrivially in both accuracy and robustness to achieve new SOTA in sim-to-real 6D pose estimation.

Original: 6D Object Pose Estimation 6D object pose estimation aims to infer the rigid SE (3) transformation (position and orientation) of an object in the camera frame, given an image observation.
Revised: 6D Object Pose Estimation 6D object pose estimation aims to infer the rigid SE (3) transformation (position and orientation) of an object in the camera frame, given an image observation.

Original: Discriminatively trained deep learning approaches (Xiang et al., 2018; Li et al., 2018; Deng et al., 2019;
Revised: Discriminatively trained deep learning approaches (Xiang et al., 2018; Li et al., 2018; Deng et al., 2019;

Original: He et al., 2020; Sundermeyer et al., 2018) have, in general, outperformed more traditional feature or template matching methods (Besl & McKay, 1992; Rusinkiewicz & Levoy, 2001; Lowe, 1999; Rothganger et al., 2006; Collet et al., 2011).
Revised: He et al., 2020; Sundermeyer et al., 2018) have, in general, outperformed more traditional feature or template matching methods (Besl & McKay, 1992; Rusinkiewicz & Levoy, 2001; Lowe, 1999; Rothganger et al., 2006; Collet et al., 2011).

Original: With the advent of RGB-D cameras, there has been a growing interest in leveraging both the appearance information from RGB and the geometric shape information from depth.
Revised: With the advent of RGB-D cameras, there has been a growing interest in leveraging both the appearance information from RGB and the geometric shape information from depth.

Original: Existing methods either use depth to post process estimations from
Revised: Existing methods either use depth to post process estimations from

Original: RGB (Xiang et al., 2018; Haugaard & Buch, 2022), or fuse learned features from both RGB and real, noisy depth (Wang et al., 2019;
Revised: RGB (Xiang et al., 2018; Haugaard & Buch, 2022), or fuse learned features from both RGB and real, noisy depth (Wang et al., 2019;

Original: He et al., 2021).
Revised: He et al., 2021).

Original: 3DNEL differs from prior work in two important ways.
Revised: 3DNEL differs from prior work in two important ways.

Original: (1) 3DNEL combines RGB and depth information in a principled probabilistic model, which enables superior sim-to-real transfer.
Revised: (1) 3DNEL combines RGB and depth information in a principled probabilistic model, which enables superior sim-to-real transfer.

Original: (2) 3DNEL jointly models multi-object scenes using a probabilistic generative formulation, allowing easy extension to tasks beyond 6D pose estimation.
Revised: (2) 3DNEL jointly models multi-object scenes using a probabilistic generative formulation, allowing easy extension to tasks beyond 6D pose estimation.

Original:  M ETHODS 3. P RELIMINARIES
Revised:  M ETHODS 3. P RELIMINARIES

Original: Likelihood for 3D Inverse Graphics 3D inverse graphics formulates the perception problem as searching for the 3D scene description that can be rendered by a graphics engine to best reconstruct the input image.
Revised: Likelihood for 3D Inverse Graphics 3D inverse graphics formulates the perception problem as searching for the 3D scene description that can be rendered by a graphics engine to best reconstruct the input image.

Original: A central challenge in applying the 3D inverse graphics approach to real images is robustly modeling the “gap” between rendered and real images.
Revised: A central challenge in applying the 3D inverse graphics approach to real images is robustly modeling the “gap” between rendered and real images.

Original: In this paper we aim to develop a likelihood P ( Observed RGB-D Image | 3D scene description ) that can combine shape and appearance information in a principled way to robustly assess how well an observed RGB-D image is explained by a 3D scene description.
Revised: In this paper we aim to develop a likelihood P ( Observed RGB-D Image | 3D scene description ) that can combine shape and appearance information in a principled way to robustly assess how well an observed RGB-D image is explained by a 3D scene description.

Original: A naive approach is to render the 3D scene description to an RGB-D image and define the likelihood as a noise model that directly compares the rendered and real RGB-D images.
Revised: A naive approach is to render the 3D scene description to an RGB-D image and define the likelihood as a noise model that directly compares the rendered and real RGB-D images.

Original: Recent work 3DP3 (Gothoskar et al., 2021) demonstrates promising performance of this approach when applied to depth images.
Revised: Recent work 3DP3 (Gothoskar et al., 2021) demonstrates promising performance of this approach when applied to depth images.

Original: However, it is much more challenging to specify a sensible noise model operating directly on RGB images.
Revised: However, it is much more challenging to specify a sensible noise model operating directly on RGB images.

Original: Intuitively, the “gap” between rendered and real depth images is mainly due to small spatial displacements, yet the “gap” between rendered and real RGB images comes from a variety of different factors.
Revised: Intuitively, the “gap” between rendered and real depth images is mainly due to small spatial displacements, yet the “gap” between rendered and real RGB images comes from a variety of different factors.

Original: In addition, a principled combination of RGB and depth information in a unified likelihood remains an open problem.
Revised: In addition, a principled combination of RGB and depth information in a unified likelihood remains an open problem.

Original: [Figure]
Revised: [Figure]

Original: 3D Scene Description We describe 3D scenes in terms of the objects in the scene and their poses.
Revised: 3D Scene Description We describe 3D scenes in terms of the objects in the scene and their poses.

Original: Each object is associated with a textured mesh, which captures the 3D shape and appearance information of the object.
Revised: Each object is associated with a textured mesh, which captures the 3D shape and appearance information of the object.

Original: The poses of the objects in the scene are represented in SE (3) (rigid transformations in 3D) which define an object’s position and orientation in space.
Revised: The poses of the objects in the scene are represented in SE (3) (rigid transformations in 3D) which define an object’s position and orientation in space.

Original: The 3D scene description is specified as the number N of objects in the scene, their classes t 1 , · · · , t N ∈ { 1 , · · · , M } , and corresponding poses P 1 , · · · , P N ∈ SE (3) .
Revised: The 3D scene description is specified as the number N of objects in the scene, their classes t 1 , · · · , t N ∈ { 1 , · · · , M } , and corresponding poses P 1 , · · · , P N ∈ SE (3) .

Original: We assume uniform prior distributions over object poses (uniform over a bounded volume for position and uniform on SO (3) for orientation).
Revised: We assume uniform prior distributions over object poses (uniform over a bounded volume for position and uniform on SO (3) for orientation).

Original: Noise Model on Depth Information We use the probabilistic model P depth ( c | ˜c ; r ) =
Revised: Noise Model on Depth Information We use the probabilistic model P depth ( c | ˜c ; r ) =

Original: [Equation]
Revised: [Equation]

Original: from 3DP3 as our noise model on depth information.
Revised: from 3DP3 as our noise model on depth information.

Original: P depth is a uniform distribution in a radius- r ball centered at a rendered point ˜c ∈ R 3 , and models the small spatial displacements in the observed point c ∈ R 3 .
Revised: P depth is a uniform distribution in a radius- r ball centered at a rendered point ˜c ∈ R 3 , and models the small spatial displacements in the observed point c ∈ R 3 .

Original: r is a hyperparameter that controls the variance of the noise model.
Revised: r is a hyperparameter that controls the variance of the noise model.

Original: Dense 2D-3D Correspondence Distributions as Noise Models on RGB Information Instead of directly operating on RGB images, we leverage learned dense 2D-3D correspondences to specify the noise model on RGB information.
Revised: Dense 2D-3D Correspondence Distributions as Noise Models on RGB Information Instead of directly operating on RGB images, we leverage learned dense 2D-3D correspondences to specify the noise model on RGB information.

Original: Concretely, we build on SurfEMB (Haugaard & Buch (2022)) and use the proposed dense 2D-3D correspondence distributions over object surfaces.
Revised: Concretely, we build on SurfEMB (Haugaard & Buch (2022)) and use the proposed dense 2D-3D correspondence distributions over object surfaces.

Original: For each object class t ∈ { 1 , · · · , M } , SurfEMB learns two neural embedding models: (1) a query embedding model which maps an RGB image I ∈ { 0 , · · · , 255 } H × W × 3 to a set of query embeddings Q t ∈
Revised: For each object class t ∈ { 1 , · · · , M } , SurfEMB learns two neural embedding models: (1) a query embedding model which maps an RGB image I ∈ { 0 , · · · , 255 } H × W × 3 to a set of query embeddings Q t ∈

Original: R H
Revised: R H

Original: × W ×, one for each 2D pixel location, and (2) a key embedding model g t : R 3 (cid:55)→ R E which maps each 3D location x ∈ R 3 (coordinate in the object frame) on the object surface to a key embedding g t ( x ) ∈ R E .
Revised: × W ×, one for each 2D pixel location, and (2) a key embedding model g t : R 3 (cid:55)→ R E which maps each 3D location x ∈ R 3 (coordinate in the object frame) on the object surface to a key embedding g t ( x ) ∈ R E .

Original: Given a pixel with query embedding q ∈ R E , SurfEMB defines a surface distribution P RGB ( x | q , t ) ∝ exp( q T g t ( x ))
Revised: Given a pixel with query embedding q ∈ R E , SurfEMB defines a surface distribution P RGB ( x | q , t ) ∝ exp( q T g t ( x ))

Original: describing which point x on the object surface the given pixel corresponds to.
Revised: describing which point x on the object surface the given pixel corresponds to.

Original: Importantly, these models can be trained entirely from synthetic data (with photorealistic rendering and physics-based simulations).
Revised: Importantly, these models can be trained entirely from synthetic data (with photorealistic rendering and physics-based simulations).

Original: See Appendix A.1 for a more detailed review of the query and key embedding models and the dense 2D-3D correspondence distributions.
Revised: See Appendix A.1 for a more detailed review of the query and key embedding models and the dense 2D-3D correspondence distributions.

Original: 3. 3D N EURAL E MBEDDING L IKELIHOOD (3DNEL)
Revised: 3. 3D N EURAL E MBEDDING L IKELIHOOD (3DNEL)

Original: Processing 3D Scene Description for 3DNEL evaluation For a given 3D scene description, we use a graphics engine to render it into: (1) A rendered point cloud image ˜C
Revised: Processing 3D Scene Description for 3DNEL evaluation For a given 3D scene description, we use a graphics engine to render it into: (1) A rendered point cloud image ˜C

Original: ∈ R H
Revised: ∈ R H

Original: × W × 3 , where ˜C i,j ∈ R 3 represents the camera frame coordinate at pixel ( i, j ) .
Revised: × W × 3 , where ˜C i,j ∈ R 3 represents the camera frame coordinate at pixel ( i, j ) .

Original: A semantic segmentation map ˜S ∈ { 0 , 1 , · · · , M } H × W where ˜S i,j represents the class to which the pixel ( i, j ) belongs.
Revised: A semantic segmentation map ˜S ∈ { 0 , 1 , · · · , M } H × W where ˜S i,j represents the class to which the pixel ( i, j ) belongs.

Original: Here
Revised: Here

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: represents background.
Revised: represents background.

Original: An object coordinate image ˜X ∈ R H
Revised: An object coordinate image ˜X ∈ R H

Original: × W × 3 where ˜X i,j represents the coordinate at pixel ( i, j ) in the object frame of the object of class ˜S i,j .
Revised: × W × 3 where ˜X i,j represents the coordinate at pixel ( i, j ) in the object frame of the object of class ˜S i,j .

Original: Processing Observed RGB-D Image for 3DNEL Evaluation For an observed RGB image I ∈ { 0 , · · · , 255 } H × W × 3 and depth image, we use the learned query embedding models to obtain M sets of query embeddings Q t ∈ R H × W × E , t ∈ { 1 , · · · , M } , one for each object class, and use camera intrinsics to unproject the depth image into an observed point cloud image
Revised: Processing Observed RGB-D Image for 3DNEL Evaluation For an observed RGB image I ∈ { 0 , · · · , 255 } H × W × 3 and depth image, we use the learned query embedding models to obtain M sets of query embeddings Q t ∈ R H × W × E , t ∈ { 1 , · · · , M } , one for each object class, and use camera intrinsics to unproject the depth image into an observed point cloud image

Original: C ∈ R H
Revised: C ∈ R H

Original: × W × 3 .
Revised: × W × 3 .

Original: 3DNEL evaluation Figure 1 visualizes 3DNEL evaluation using processed 3D scene descriptions and observed RGB-D images.
Revised: 3DNEL evaluation Figure 1 visualizes 3DNEL evaluation using processed 3D scene descriptions and observed RGB-D images.

Original: 3DNEL combines the noise model P depth on depth information from 3DP3 and the dense 2D-3D correspondence distributions P RGB from SurfEMB into a unified likelihood for sim-to-real transfer.
Revised: 3DNEL combines the noise model P depth on depth information from 3DP3 and the dense 2D-3D correspondence distributions P RGB from SurfEMB into a unified likelihood for sim-to-real transfer.

Original: 3DNEL jointly models multiple objects in a scene through a mixture model formulation, resulting in a probabilistic generative model of real RGB-D images.
Revised: 3DNEL jointly models multiple objects in a scene through a mixture model formulation, resulting in a probabilistic generative model of real RGB-D images.

Original: Intuitively, we assess how well each pixel ( i, j ) in the observed point cloud image C is explained by a pixel (˜ i, ˜ j ) in the rendered point cloud image ˜C , by combining the noise model P depth on depth and the noise model P RGB on RGB.
Revised: Intuitively, we assess how well each pixel ( i, j ) in the observed point cloud image C is explained by a pixel (˜ i, ˜ j ) in the rendered point cloud image ˜C , by combining the noise model P depth on depth and the noise model P RGB on RGB.

Original: To jointly model multiple objects in a scene, we assume each pixel ( i, j ) in C can be explained by multiple pixels in ˜C .
Revised: To jointly model multiple objects in a scene, we assume each pixel ( i, j ) in C can be explained by multiple pixels in ˜C .

Original: We formalize this with a mixture model formulation, where the mixture component associated with the rendered pixel (˜ i, ˜ j ) combines P depth and P RGB to assess how well the observed pixel ( i, j ) is explained by the rendered pixel (˜ i, ˜ j ) .
Revised: We formalize this with a mixture model formulation, where the mixture component associated with the rendered pixel (˜ i, ˜ j ) combines P depth and P RGB to assess how well the observed pixel ( i, j ) is explained by the rendered pixel (˜ i, ˜ j ) .

Original: To model background pixels in C , we assume the observed point cloud image C resides in a bounded region of volume B , and introduce a uniform distribution P background ( c ; B )
Revised: To model background pixels in C , we assume the observed point cloud image C resides in a bounded region of volume B , and introduce a uniform distribution P background ( c ; B )

Original: = 1 /B on the bounded region with mixture probability ϵ as an additional mixture component for background modeling.
Revised: = 1 /B on the bounded region with mixture probability ϵ as an additional mixture component for background modeling.

Original: Representing the total number of non-background pixels in the rendered images as ˜ K
Revised: Representing the total number of non-background pixels in the rendered images as ˜ K

Original: =
Revised: =

Original: (cid:80) ˜
Revised: (cid:80) ˜

Original: i, ˜ j 1 [ ˜S ˜ i, ˜ j > 0] , the mixture probability for the mixture component
Revised: i, ˜ j 1 [ ˜S ˜ i, ˜ j > 0] , the mixture probability for the mixture component

Original: Figure 3(e) shows how pose hypotheses far away ground
Revised: Figure 3(e) shows how pose hypotheses far away ground

Original: In Figures 3(b) to (e) , light yellow represents low scores, while dark red associated with rendered pixel (˜ i, ˜ j ) is given by (1 − ϵ ) /
Revised: In Figures 3(b) to (e) , light yellow represents low scores, while dark red associated with rendered pixel (˜ i, ˜ j ) is given by (1 − ϵ ) /

Original: ˜ K .
Revised: ˜ K .

Original: Since the query embedding at a pixel depends on the entire image I , the mixture components are not properly normalized.
Revised: Since the query embedding at a pixel depends on the entire image I , the mixture components are not properly normalized.

Original: This leads to the following energy-based formulation for 3DNEL:
Revised: This leads to the following energy-based formulation for 3DNEL:

Original: [Equation]
Revised: [Equation]

Original: where we denote C i,j by c , ˜C
Revised: where we denote C i,j by c , ˜C

Original: ˜ i, ˜ j by ˜c , ˜S ˜ i, ˜ j by ˜ s , ˜X ˜ i, ˜ j by ˜ x , and Q ti,j by q t .
Revised: ˜ i, ˜ j by ˜c , ˜S ˜ i, ˜ j by ˜ s , ˜X ˜ i, ˜ j by ˜ x , and Q ti,j by q t .

Original: The product is over all observed pixels, and the sum is over all non-background rendered pixels.
Revised: The product is over all observed pixels, and the sum is over all non-background rendered pixels.

Original: ϵ, B and r are hyperparameters that we pick in the experiments.
Revised: ϵ, B and r are hyperparameters that we pick in the experiments.

Original: See Appendix A.2 for more details.
Revised: See Appendix A.2 for more details.

Original: 3. I NFERRING THE 3D SCENE DESCRIPTION FROM RGB-D
Revised: 3. I NFERRING THE 3D SCENE DESCRIPTION FROM RGB-D

Original: Figure 2 gives an overview of 3DNEL MSIGP for 6D object pose estimation.
Revised: Figure 2 gives an overview of 3DNEL MSIGP for 6D object pose estimation.

Original: Enumeration-based Pose Hypotheses Generation We develop a novel spherical voting procedure and a heuristic scoring using the query embeddings and observed point cloud image C defined in Section 3.2, and use them in an enumerative procedure to efficiently generate pose hypotheses.
Revised: Enumeration-based Pose Hypotheses Generation We develop a novel spherical voting procedure and a heuristic scoring using the query embeddings and observed point cloud image C defined in Section 3.2, and use them in an enumerative procedure to efficiently generate pose hypotheses.

Original: We use the object center and n k points sampled using farthest point sampling from the object surface as our keypoints, and discretize the camera frame space into a L x × L y ×
Revised: We use the object center and n k points sampled using farthest point sampling from the object surface as our keypoints, and discretize the camera frame space into a L x × L y ×

Original: L z voxel grid .
Revised: L z voxel grid .

Original: For a given keypoint, our spherical voting procedure aggregates information from the entire image to score how likely the keypoint is present at different voxel locations, and stores the scores in a voxel grid.
Revised: For a given keypoint, our spherical voting procedure aggregates information from the entire image to score how likely the keypoint is present at different voxel locations, and stores the scores in a voxel grid.

Original: Figure 3(a) visualizes spherical voting for the center x ∗ of the mug object: for pixel location ( i, j ) with camera frame coordinate c ∈ R 3 and query embedding q ∈ R E , we identify its most likely corresponding point on the mug surface x = arg max ˜ x P RGB (˜ x | q , t ) , calculate the distance r x = || x − x ∗ || 2 from x to x ∗ in the object frame, and cast votes (Qi et al., 2019) with weight p
Revised: Figure 3(a) visualizes spherical voting for the center x ∗ of the mug object: for pixel location ( i, j ) with camera frame coordinate c ∈ R 3 and query embedding q ∈ R E , we identify its most likely corresponding point on the mug surface x = arg max ˜ x P RGB (˜ x | q , t ) , calculate the distance r x = || x − x ∗ || 2 from x to x ∗ in the object frame, and cast votes (Qi et al., 2019) with weight p

Original: i,j = max ˜ x
Revised: i,j = max ˜ x

Original: P RGB (˜ x | q , t ) towards all points on a sphere of radius r x centered at c .
Revised: P RGB (˜ x | q , t ) towards all points on a sphere of radius r x centered at c .

Original: Figure 3(b) visualizes the 20 top-scoring voxels from the voxel grid for the mug center, and Figure 3(c) visualizes the 20 top-scoring voxels from the voxel grids for the n k keypoints on the mug surface.
Revised: Figure 3(b) visualizes the 20 top-scoring voxels from the voxel grid for the mug center, and Figure 3(c) visualizes the 20 top-scoring voxels from the voxel grids for the n k keypoints on the mug surface.

Original: We coarsely discretize the object pose space.
Revised: We coarsely discretize the object pose space.

Original: We reuse the same camera frame space discretization into a voxel grid, and use the L x × L y ×
Revised: We reuse the same camera frame space discretization into a voxel grid, and use the L x × L y ×

Original: L z voxel centers to discretize the location space.
Revised: L z voxel centers to discretize the location space.

Original: We use a customized procedure (Appendix A.5) to generate n r rotations and discretize the rotation space.
Revised: We use a customized procedure (Appendix A.5) to generate n r rotations and discretize the rotation space.

Original: We use the voxel grid for the object center to identify top-scoring object locations, and score all n r rotations at these locations.
Revised: We use the voxel grid for the object center to identify top-scoring object locations, and score all n r rotations at these locations.

Original: We score a given object pose with the sum of the scores of the voxels the corresponding n k keypoints fall into.
Revised: We score a given object pose with the sum of the scores of the voxels the corresponding n k keypoints fall into.

Original: Figure 3(d) visualizes 3 example top pose hypotheses from the enumerative procedure.
Revised: Figure 3(d) visualizes 3 example top pose hypotheses from the enumerative procedure.

Original: Figure 3(e) visualizes how poses far away from ground truth get low scores from our heuristic scoring.
Revised: Figure 3(e) visualizes how poses far away from ground truth get low scores from our heuristic scoring.

Original: See Appendix A.3 for a formal description of the process.
Revised: See Appendix A.3 for a formal description of the process.

Original: MCMC Finetuning We generate an initial 3D scene description using top scoring pose hypothesis for each object instance in the scene based on the pose hypotheses for its class, and use MCMC, in particular the Metropolis-Hastings (MH) algorithm, to iteratively finetune the 3D scene description.
Revised: MCMC Finetuning We generate an initial 3D scene description using top scoring pose hypothesis for each object instance in the scene based on the pose hypotheses for its class, and use MCMC, in particular the Metropolis-Hastings (MH) algorithm, to iteratively finetune the 3D scene description.

Original: For a scene with N objects, at each iteration, MH proposes to change the pose P i of a single object i , and accepts the proposal with the MH acceptance probability.
Revised: For a scene with N objects, at each iteration, MH proposes to change the pose P i of a single object i , and accepts the proposal with the MH acceptance probability.

Original: See Figure 2 for visualizations.
Revised: See Figure 2 for visualizations.

Original: We use three different proposals, all of which sample a pose’s position from a spherical normal distribution centered at a desired position and the pose’s orientation from a von Mises-Fisher distribution centered at a desired orientation.
Revised: We use three different proposals, all of which sample a pose’s position from a spherical normal distribution centered at a desired position and the pose’s orientation from a von Mises-Fisher distribution centered at a desired orientation.

Original: (1) Pose hypotheses proposals go through the N objects multiple times.
Revised: (1) Pose hypotheses proposals go through the N objects multiple times.

Original: For object i of class t i , the pose hypotheses proposals sequentially go through the generated pose hypotheses for object class t i , sample a pose centered at each of the pose hypotheses, and propose to change the pose of object i to the sampled pose.
Revised: For object i of class t i , the pose hypotheses proposals sequentially go through the generated pose hypotheses for object class t i , sample a pose centered at each of the pose hypotheses, and propose to change the pose of object i to the sampled pose.

Original: ICP proposals iterate over all objects and use ICP to align each object’s 3D model to the observed 3D point cloud with the point cloud from all the other objects masked out, and make a proposal centered at the resulting pose.
Revised: ICP proposals iterate over all objects and use ICP to align each object’s 3D model to the observed 3D point cloud with the point cloud from all the other objects masked out, and make a proposal centered at the resulting pose.

Original: Finally, random walk proposals that randomly perturb object poses.
Revised: Finally, random walk proposals that randomly perturb object poses.

Original: See Appendix A.4 for details.
Revised: See Appendix A.4 for details.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct extensive experiments on YCB-V, a popular benchmark for 6D object pose estimation.
Revised: We conduct extensive experiments on YCB-V, a popular benchmark for 6D object pose estimation.

Original: We establish that 3DNEL MSIGP significantly improves the previous SOTA SurfEMB in sim-toreal 6D object pose estimation, and is more robust in terms of reducing large pose estimation errors and overcoming failures of 2D detectors.
Revised: We establish that 3DNEL MSIGP significantly improves the previous SOTA SurfEMB in sim-toreal 6D object pose estimation, and is more robust in terms of reducing large pose estimation errors and overcoming failures of 2D detectors.

Original: Detailed ablations illustrate the source of performance improvements, and additional experiments on easy extension of 3DNEL to object and camera tracking from video demonstrates the value of 3DNEL’s formulation as a probabilistic generative model.
Revised: Detailed ablations illustrate the source of performance improvements, and additional experiments on easy extension of 3DNEL to object and camera tracking from video demonstrates the value of 3DNEL’s formulation as a probabilistic generative model.

Original: Evaluation We follow the evaluation protocol of the Benchmark for 6D Object Pose Estimation (BOP) challenge (Hodan et al., 2018).
Revised: Evaluation We follow the evaluation protocol of the Benchmark for 6D Object Pose Estimation (BOP) challenge (Hodan et al., 2018).

Original: The task is to estimate the 6D poses of objects in a scene, from a single RGB-D image.
Revised: The task is to estimate the 6D poses of objects in a scene, from a single RGB-D image.

Original: The BOP challenge assumes knowledge of the number of instances of each object class in the scene.
Revised: The BOP challenge assumes knowledge of the number of instances of each object class in the scene.

Original: For a predicted pose, we calculate three error metrics with respect to the ground truth pose: Visible Surface Discrepancy (VSD) (Hoda ˇ n et al., 2016;
Revised: For a predicted pose, we calculate three error metrics with respect to the ground truth pose: Visible Surface Discrepancy (VSD) (Hoda ˇ n et al., 2016;

Original: Hodan et al., 2018), Maximum Symmetry-Aware Surface Distance (MSSD)
Revised: Hodan et al., 2018), Maximum Symmetry-Aware Surface Distance (MSSD)

Original: (Drost et al., 2017), and Maximum Symmetry-Aware Projection Distance (Brachmann et al., 2016).
Revised: (Drost et al., 2017), and Maximum Symmetry-Aware Projection Distance (Brachmann et al., 2016).

Original: Average recalls AR V SD ,
Revised: Average recalls AR V SD ,

Original: AR MSSD , AR MSPD are computed for each error metric across a range of error thresholds.
Revised: AR MSSD , AR MSPD are computed for each error metric across a range of error thresholds.

Original: The aggregate Average Recall (as reported in Table 1) is the average of AR V SD , AR MSSD , and AR MSPD .
Revised: The aggregate Average Recall (as reported in Table 1) is the average of AR V SD , AR MSSD , and AR MSPD .

Original: Training To allow easy comparison with baselines, we use publicly released pretrained SurfEMB models for our experiments.
Revised: Training To allow easy comparison with baselines, we use publicly released pretrained SurfEMB models for our experiments.

Original: These include query models, key models, mask predictors for different object classes, and an additional 2D detector from CosyPose (Labb´e et al., 2020), all trained entirely from synthetic data.
Revised: These include query models, key models, mask predictors for different object classes, and an additional 2D detector from CosyPose (Labb´e et al., 2020), all trained entirely from synthetic data.

Original: While sharing the same underlying models, our 3DNEL MSIGP is significantly different from SurfEMB’s PnP+RANSAC procedure, and demonstrates nontrivial improvements in overall accuracy as well as robustness.
Revised: While sharing the same underlying models, our 3DNEL MSIGP is significantly different from SurfEMB’s PnP+RANSAC procedure, and demonstrates nontrivial improvements in overall accuracy as well as robustness.

Original: We expect additional training tailored towards the 3DNEL formulation can further improve performance, but leave this to future work.
Revised: We expect additional training tailored towards the 3DNEL formulation can further improve performance, but leave this to future work.

Original: 3DNEL MSIGP We follow He et al.
Revised: 3DNEL MSIGP We follow He et al.

Original: and use Ku et al.
Revised: and use Ku et al.

Original: (2018) to fill in missing depth.
Revised: (2018) to fill in missing depth.

Original: Recent strong pose estimation methods commonly rely on 2D detection and mask prediction.
Revised: Recent strong pose estimation methods commonly rely on 2D detection and mask prediction.

Original: Since our spherical voting procedure can robustly generate pose hypotheses by aggregating information from the entire image, 3DNEL MSIGP can work well even without separate 2D detection and mask prediction (Table 1, 3DNEL MSIGP (No 2D Detection) ) while SurfEMB completely fails without these.
Revised: Since our spherical voting procedure can robustly generate pose hypotheses by aggregating information from the entire image, 3DNEL MSIGP can work well even without separate 2D detection and mask prediction (Table 1, 3DNEL MSIGP (No 2D Detection) ) while SurfEMB completely fails without these.

Original: However, empirically we find that we can leverage additional 2D detection and mask prediction to allow spherical voting to aggregate information only from regions of the query embedding images that are likely relevant to the objects, and significantly outperform SurfEMB using identical information.
Revised: However, empirically we find that we can leverage additional 2D detection and mask prediction to allow spherical voting to aggregate information only from regions of the query embedding images that are likely relevant to the objects, and significantly outperform SurfEMB using identical information.

Original: We pick hyperparameters by visually inspecting inference results on a small number of real training images outside the test set.
Revised: We pick hyperparameters by visually inspecting inference results on a small number of real training images outside the test set.

Original: See Appendix A.5 for details.
Revised: See Appendix A.5 for details.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: pose estimation.
Revised: pose estimation.

Original: Results are averaged over 5 runs.
Revised: Results are averaged over 5 runs.

Original: Standard deviation is below 0 .
Revised: Standard deviation is below 0 .

Original: 2% for all setups.
Revised: 2% for all setups.

Original: Our superior performance comes at the cost of increased computation.
Revised: Our superior performance comes at the cost of increased computation.

Original: Empirically we observe our current (highly unoptimized) implementation can be 6x slower than SurfEMB (20 to 30s on static images, 15 to 20s for tracking, on a machine with a single NVIDIA A100 GPU).
Revised: Empirically we observe our current (highly unoptimized) implementation can be 6x slower than SurfEMB (20 to 30s on static images, 15 to 20s for tracking, on a machine with a single NVIDIA A100 GPU).

Original: In Section 5 we discuss this limitation and propose some potential solutions.
Revised: In Section 5 we discuss this limitation and propose some potential solutions.

Original: 4. A CCURACY
Revised: 4. A CCURACY

Original: In Table 1, we report the Average Recall for 3DNEL MSIGP, its various ablations and representative bottom-up baselines.
Revised: In Table 1, we report the Average Recall for 3DNEL MSIGP, its various ablations and representative bottom-up baselines.

Original: In the first section we compare with two baselines trained entirely on synthetic data: the current SOTA SurfEMB (Haugaard & Buch, 2022) which uses depth to heuristically refine pose estimates with RGB information, and FFB6D (He et al., 2021) which is a bottom-up approach that fuses appearance information from RGB with geometry information from depth.
Revised: In the first section we compare with two baselines trained entirely on synthetic data: the current SOTA SurfEMB (Haugaard & Buch, 2022) which uses depth to heuristically refine pose estimates with RGB information, and FFB6D (He et al., 2021) which is a bottom-up approach that fuses appearance information from RGB with geometry information from depth.

Original: 3DNEL formulates a probabilistic 3D likelihood over depth data and appearance information from RGB in the form of 2D-3D correspondence distributions, and significantly outperforms SurfEMB, FFB6D.
Revised: 3DNEL formulates a probabilistic 3D likelihood over depth data and appearance information from RGB in the form of 2D-3D correspondence distributions, and significantly outperforms SurfEMB, FFB6D.

Original: In the second section, we evaluate various ablations of 3DNEL MSIGP.
Revised: In the second section, we evaluate various ablations of 3DNEL MSIGP.

Original: We first investigate the importance of principled combination of RGB and depth.
Revised: We first investigate the importance of principled combination of RGB and depth.

Original: We evaluate 3DNEL MSIGP (No RGB in
Revised: We evaluate 3DNEL MSIGP (No RGB in

Original: Likelihood) , where we drop P RGB in Equation 1, and 3DNEL MSIGP (No Depth in Likelihood) where we replace the 3D ball 1 [ || C i,j − ˜ C
Revised: Likelihood) , where we drop P RGB in Equation 1, and 3DNEL MSIGP (No Depth in Likelihood) where we replace the 3D ball 1 [ || C i,j − ˜ C

Original: ˜
Revised: ˜

Original: i, ˜ j || 2 ≤ r ] in Equation 1 with a 3 × 3 2D patch.
Revised: i, ˜ j || 2 ≤ r ] in Equation 1 with a 3 × 3 2D patch.

Original: Both are substantially worse.
Revised: Both are substantially worse.

Original: We next evaluate the importance of different components in 3DNEL MSIGP.
Revised: We next evaluate the importance of different components in 3DNEL MSIGP.

Original: We evaluate 3DNEL MSIGP (SurfEMB initialization + MCMC) , which replaces the enumerationbased pose hypotheses generation with SurfEMB predictions and does MCMC finetuning with 3DNEL.
Revised: We evaluate 3DNEL MSIGP (SurfEMB initialization + MCMC) , which replaces the enumerationbased pose hypotheses generation with SurfEMB predictions and does MCMC finetuning with 3DNEL.

Original: This leads to 2 .
Revised: This leads to 2 .

Original: 73% improvements over SurfEMB but does not outperform 3DNEL MSIGP, illustrating the effectiveness of MCMC finetuning and the value of having multiple pose hypothe-
Revised: 73% improvements over SurfEMB but does not outperform 3DNEL MSIGP, illustrating the effectiveness of MCMC finetuning and the value of having multiple pose hypothe-

Original: [Table]
Revised: [Table]

Original: 3DNEL MSIGP (Initialization only) , which uses initialiations from the enumerative as pose estimates.
Revised: 3DNEL MSIGP (Initialization only) , which uses initialiations from the enumerative as pose estimates.

Original: This is substantially worse, the of finetuning.
Revised: This is substantially worse, the of finetuning.

Original: We
Revised: We

Original: 3DNEL MSIGP (No 2D Detection) , which uses spherical voting to aggregate evidence from entire embedding images to generate pose hypotheses.
Revised: 3DNEL MSIGP (No 2D Detection) , which uses spherical voting to aggregate evidence from entire embedding images to generate pose hypotheses.

Original: This uses much less information than SurfEMB (no 2D detection and mask prediction), yet are not far off, demonstrating the effectiveness of our enumeration-based pose hypotheses generation.
Revised: This uses much less information than SurfEMB (no 2D detection and mask prediction), yet are not far off, demonstrating the effectiveness of our enumeration-based pose hypotheses generation.

Original: In the last section we contextualize 3DNEL MSIGP’s performance in relation to methods that depend on real data.
Revised: In the last section we contextualize 3DNEL MSIGP’s performance in relation to methods that depend on real data.

Original: 3DNEL MSIGP outperforms several methods, despite no exposure to real data before test time.
Revised: 3DNEL MSIGP outperforms several methods, despite no exposure to real data before test time.

Original: We also observe that previous bottom-up approaches that combine RGB and depth information rely heavily on annotated real data.
Revised: We also observe that previous bottom-up approaches that combine RGB and depth information rely heavily on annotated real data.

Original: FFB6D, for example, has average recall of 85 .
Revised: FFB6D, for example, has average recall of 85 .

Original: but this drops to 75 .
Revised: but this drops to 75 .

Original: 8% without real data.
Revised: 8% without real data.

Original: We hypothesize this is because these methods essentially overfit to the characteristics of noisy depth and struggle to generalize to real RGB-D images without having seen them at training.
Revised: We hypothesize this is because these methods essentially overfit to the characteristics of noisy depth and struggle to generalize to real RGB-D images without having seen them at training.

Original: In contrast, as a result of our principled probabilistic modeling of depth information, we can transfer from simulated data to real data without sacrificing performance.
Revised: In contrast, as a result of our principled probabilistic modeling of depth information, we can transfer from simulated data to real data without sacrificing performance.

Original: 4. R OBUSTNESS
Revised: 4. R OBUSTNESS

Original: Figure 4 contains a more detailed comparison between SurfEMB and 3DNEL MSIGP.
Revised: Figure 4 contains a more detailed comparison between SurfEMB and 3DNEL MSIGP.

Original: Across all
Revised: Across all

Original: YCB-V test images, there are 4123 object instances.
Revised: YCB-V test images, there are 4123 object instances.

Original: Each point on the scatter plots corresponds to an object instance, and the point’s x and y coordinates are the pose prediction error of 3DNEL MSIGP and SurfEMB, respectively.
Revised: Each point on the scatter plots corresponds to an object instance, and the point’s x and y coordinates are the pose prediction error of 3DNEL MSIGP and SurfEMB, respectively.

Original: Points above the dashed line correspond to object instances for which 3DNEL MSIGP had a lower prediction error than SurfEMB.
Revised: Points above the dashed line correspond to object instances for which 3DNEL MSIGP had a lower prediction error than SurfEMB.

Original: Figure 4(a) shows the scatter plot for all 4123 predictions and Figure 4(c) shows the scatter plots for 6 representative object classes.
Revised: Figure 4(a) shows the scatter plot for all 4123 predictions and Figure 4(c) shows the scatter plots for 6 representative object classes.

Original: Figure 4(b) shows, across a range of error thresholds, the number of pose predictions with error above that threshold.
Revised: Figure 4(b) shows, across a range of error thresholds, the number of pose predictions with error above that threshold.

Original: 3DNEL MSIGP significantly and consistently reduces the the number of high-error pose predictions, highlighting 3DNEL MSIGP’s robustness compared to bottom-up approaches.
Revised: 3DNEL MSIGP significantly and consistently reduces the the number of high-error pose predictions, highlighting 3DNEL MSIGP’s robustness compared to bottom-up approaches.

Original: We find many instances, especially concentrated within a few object classes, for which SurfEMB is systematically making bad pose predictions.
Revised: We find many instances, especially concentrated within a few object classes, for which SurfEMB is systematically making bad pose predictions.

Original: This is primarily a result of the noisy neural embeddings, which SurfEMB’s PnP+RANSAC procedure does not handle well.
Revised: This is primarily a result of the noisy neural embeddings, which SurfEMB’s PnP+RANSAC procedure does not handle well.

Original: 3DNEL MSIGP is able to correctly estimate the object’s pose as a result of enumeration-based pose hypotheses generation, which can robustly handle noisy neural embeddings and identify a variety of pose hypotheses.
Revised: 3DNEL MSIGP is able to correctly estimate the object’s pose as a result of enumeration-based pose hypotheses generation, which can robustly handle noisy neural embeddings and identify a variety of pose hypotheses.

Original: 3DNEL MSIGP also improves over SurfEMB on test images where the 2D object detector that SurfEMB relies on fails.
Revised: 3DNEL MSIGP also improves over SurfEMB on test images where the 2D object detector that SurfEMB relies on fails.

Original: SurfEMB uses a neural 2D object detector and only makes pose predictions within the predicted bounding box.
Revised: SurfEMB uses a neural 2D object detector and only makes pose predictions within the predicted bounding box.

Original: In contrast, 3DNEL MSIGP considers the full image using spherical voting and therefore can make reasonable pose estimates in spite of missing or wrong 2D detections.
Revised: In contrast, 3DNEL MSIGP considers the full image using spherical voting and therefore can make reasonable pose estimates in spite of missing or wrong 2D detections.

Original: Out of the 4123 object instances, there are 174 object instances for which there is no 2D detection and therefore no prediction from SurfEMB.
Revised: Out of the 4123 object instances, there are 174 object instances for which there is no 2D detection and therefore no prediction from SurfEMB.

Original: Figure 5 and Figure A3 in the appendix visualize 3DNEL MSIGP and SurfEMB’s pose predictions for representative YCB-V test images.
Revised: Figure 5 and Figure A4 in the appendix visualize 3DNEL MSIGP and SurfEMB’s pose predictions for representative YCB-V test images.

Original: 4. E XTENSION TO O BJECT AND C AMERA T RACKING FROM V IDEO
Revised: 4. E XTENSION TO O BJECT AND C AMERA T RACKING FROM V IDEO

Original: We demonstrate that, due to its probabilistic generative formulation, we can easily extend 3DNEL to support object and camera tracking from video using principled probabilistic inference in the same model without task specific retraining.
Revised: We demonstrate that, due to its probabilistic generative formulation, we can easily extend 3DNEL to support object and camera tracking from video using principled probabilistic inference in the same model without task specific retraining.

Original: We extend the our single-frame 3DNEL to the multi-timestep setup by introducing a simple dynamics prior that samples the object pose at time t + 1 uniformly from the space of poses with position at most 3cm away from the object pose at time t .
Revised: We extend the our single-frame 3DNEL to the multi-timestep setup by introducing a simple dynamics prior that samples the object pose at time t + 1 uniformly from the space of poses with position at most 3cm away from the object pose at time t .

Original: For object tracking, we initialize the object poses at the first frame using our single-frame prediction, and apply MCMC finetuning with ICP and random walk proposals for subsequent frames, taking into account the dynamics prior.
Revised: For object tracking, we initialize the object poses at the first frame using our single-frame prediction, and apply MCMC finetuning with ICP and random walk proposals for subsequent frames, taking into account the dynamics prior.

Original: For camera tracking, we initialize object poses at the first frame to ground truth annotations to avoid introducing systematic errors.
Revised: For camera tracking, we initialize object poses at the first frame to ground truth annotations to avoid introducing systematic errors.

Original: We again apply MCMC finetuning with ICP and random walk proposals.
Revised: We again apply MCMC finetuning with ICP and random walk proposals.

Original: However, we further assume we know the scene is static and only the camera moves, which translates into jointly updating the poses of all objects in a scene.
Revised: However, we further assume we know the scene is static and only the camera moves, which translates into jointly updating the poses of all objects in a scene.

Original: Results in Table 2 show that the same inference procedure can readily handle such extensions, taking into account the dynamics prior for object tracking and the knowledge of a static scene for camera tracking within the same probabilistic generative model.
Revised: Results in Table 2 show that the same inference procedure can readily handle such extensions, taking into account the dynamics prior for object tracking and the knowledge of a static scene for camera tracking within the same probabilistic generative model.

Original: We observe comprehensive improvements over single frame predictions, except for object tracking in scene 59 where we make large errors in the first frame with our single frame predictions and are not able to recover in later frames.
Revised: We observe comprehensive improvements over single frame predictions, except for object tracking in scene 59 where we make large errors in the first frame with our single frame predictions and are not able to recover in later frames.

Original: To demonstrate the benefits of modeling uncertainty, a key advantage enabled by 3DNEL’s probabilistic formulation, we additionally generatesimple 2-object synthetic videos, with one object moving across the scene and getting completely occluded by the other static object before reappearing.
Revised: Finally, to highlight the benefits of modeling uncertainty, a key advantage enabled by 3DNEL’s probabilistic formulation, we consider a challenging object tracking setup with highly occluded objects. We generate synthetic videos containing two YCB objects, in which one object moves across the scene and becomes fully ocluded by the other static object before reappearing.

Original: This is a challenging setup for  the 3DNEL tracking extension that rely only on point estimates.
Revised: Methods that maintain only a single object pose estimation for each object tend to lose track of the moving object once it becomes occluded.

Original: However,we observe that by  properly modeling uncertainty using 3DNEL’s posterior distribution on object poses through MCMC, we  achieve successful tracking.
Revised: However, by leveraging Sequential Monte Carlo (Liu & Chen, 1998) to properly model the uncertainty in the pose of the moving object when it is occluded, we are able to achieve reliable tracking in this challenging setup.

Original: See Appendix A.7 for details.
Revised: See Appendix A.7 for details.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: This paper presents 3DNEL, a 3D likelihood for RGB-D images based on neural embeddings, and demonstrated SOTA performance in sim-to-real 6D object pose estimation on YCB-V.
Revised: This paper presents 3DNEL, a 3D likelihood for RGB-D images based on neural embeddings, and demonstrated SOTA performance in sim-to-real 6D object pose estimation on YCB-V.

Original: However, the current work has several limitations.
Revised: However, the current work has several limitations.

Original: Since 3DNEL is formulated to explain the whole scene, inference with 3DNEL sometimes moves highly occluded objects to explain visible backgrounds.
Revised: Since 3DNEL is formulated to explain the whole scene, inference with 3DNEL sometimes moves highly occluded objects to explain visible backgrounds.

Original: In addition, certain objects are difficult to capture with depth cameras because of their reflective surfaces, which can lead to degraded pose estimation due to the importance of depth cues in 3DNEL.
Revised: In addition, certain objects are difficult to capture with depth cameras because of their reflective surfaces, which can lead to degraded pose estimation due to the importance of depth cues in 3DNEL.

Original: We expect a background model and an outlier model that can properly fall back to use only RGB data when depth is missing will help address these issues.
Revised: We expect a background model and an outlier model that can properly fall back to use only RGB data when depth is missing will help address these issues.

Original: Finally, while 3DNEL improves pose estimation accuracy and robustness, this comes at the cost of increased computation.
Revised: Finally, while 3DNEL improves pose estimation accuracy and robustness, this comes at the cost of increased computation.

Original: This is due in part to the lack of performance engineering efforts, and to the non-standard nature of our inference pipeline, which lacks highly optimized software implementations.
Revised: This is due in part to the lack of performance engineering efforts, and to the non-standard nature of our inference pipeline, which lacks highly optimized software implementations.

Original: In future work, we plan to additionally leverage batched GPU rendering and compilation, which will enable a large number of
Revised: In future work, we plan to additionally leverage batched GPU rendering and compilation, which will enable a large number of

Original: MCMC proposals to be evaluated in parallel and therefore significantly speed up our pipeline.
Revised: MCMC proposals to be evaluated in parallel and therefore significantly speed up our pipeline.

Original: A A PPENDIX A. R EVIEW OF S URF EMB
Revised: A A PPENDIX A. R EVIEW OF S URF EMB

Original: Our noise model on RGB information build on SurfEMB (Haugaard & Buch, 2022) which learns neural embeddings to establish dense 2D-3D correspondences via contrastive learning.
Revised: Our noise model on RGB information build on SurfEMB (Haugaard & Buch, 2022) which learns neural embeddings to establish dense 2D-3D correspondences via contrastive learning.

Original: For each object class t ∈ { 1 , · · · , M } , SurfEMB learns a pair of neural networks: the query model f t : { 0 , · · · , 255 } H × W × 3 (cid:55)→ R H
Revised: For each object class t ∈ { 1 , · · · , M } , SurfEMB learns a pair of neural networks: the query model f t : { 0 , · · · , 255 } H × W × 3 (cid:55)→ R H

Original: × W × E and the key model g t : R 3 (cid:55)→ R E .
Revised: × W × E and the key model g t : R 3 (cid:55)→ R E .

Original: The query model transforms an observed RGB image I into query embeddings Q t , while the key model transforms a rendered object coordinate image ˜X into a set of key embeddings.
Revised: The query model transforms an observed RGB image I into query embeddings Q t , while the key model transforms a rendered object coordinate image ˜X into a set of key embeddings.

Original: For a given 2D pixel location on the observed image with query embedding q , SurfEMB specifies a surface correspondence distribution P RGB (˜ x | q , t ) ∝ exp( q T g t (˜ x )) for each object class t .
Revised: For a given 2D pixel location on the observed image with query embedding q , SurfEMB specifies a surface correspondence distribution P RGB (˜ x | q , t ) ∝ exp( q T g t (˜ x )) for each object class t .

Original: To normalize this surface correspondence distribution, for each object class t , we subsample uniformly across the object’s surface to get a set Z t of surface 3D coordinates in object frame.
Revised: To normalize this surface correspondence distribution, for each object class t , we subsample uniformly across the object’s surface to get a set Z t of surface 3D coordinates in object frame.

Original: Then, given a pixel with query embedding q , we calculate the probability that this pixel corresponds to a surface point
Revised: Then, given a pixel with query embedding q , we calculate the probability that this pixel corresponds to a surface point

Original: ˜x ∈
Revised: ˜x ∈

Original: Z t on object class t as:
Revised: Z t on object class t as:

Original: [Equation]
Revised: [Equation]

Original: A. M ORE DETAILS ON THE ENERGY - BASED FORMULATION OF 3DNEL
Revised: A. M ORE DETAILS ON THE ENERGY - BASED FORMULATION OF 3DNEL

Original: Since we are working with an energy-based formulation (Equation 1), to make the probability distribution properly defined we need to make sure the normalization constant, i.e. the sum of the energy function over all I and C (cid:88)
Revised: Since we are working with an energy-based formulation (Equation 1), to make the probability distribution properly defined we need to make sure the normalization constant, i.e. the sum of the energy function over all I and C (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: is finite and well-defined.
Revised: is finite and well-defined.

Original: For RGB images of size H × W , since each pixel has only 256 values, there are at most 256 H × W ×
Revised: For RGB images of size H × W , since each pixel has only 256 values, there are at most 256 H × W ×

Original: RGB images of size H × W which is a finite number.
Revised: RGB images of size H × W which is a finite number.

Original: Since the value of the energy function is less than 1 for any given I and C , summing over a finite number of I and integrating over a bounded region for C gives us a finite normalization constant, making the probability distribution well-defined.
Revised: Since the value of the energy function is less than 1 for any given I and C , summing over a finite number of I and integrating over a bounded region for C gives us a finite normalization constant, making the probability distribution well-defined.

Original: A. D ETAILS ON E NUMERATION - BASED POSE HYPOTHESES GENERATION
Revised: A. D ETAILS ON E NUMERATION - BASED POSE HYPOTHESES GENERATION

Original: In Algorithm 1 we present a detailed description of our enumeration-based pose hypotheses generation process that is introduced in Section 3.3.
Revised: In Algorithm 1 we present a detailed description of our enumeration-based pose hypotheses generation process that is introduced in Section 3.3.

Original: In practice we also optionally apply non-max suppression in the TopPositions function in the algorithm to make sure the promising positions we identify are well spread out to cover different parts of the image and better represent uncertainty.
Revised: In practice we also optionally apply non-max suppression in the TopPositions function in the algorithm to make sure the promising positions we identify are well spread out to cover different parts of the image and better represent uncertainty.

Original: A. D ETAILS ON MCMC FINETUNING
Revised: A. D ETAILS ON MCMC FINETUNING

Original: Our pose hypotheses proposals go through the N objects present in a scene multiple times.
Revised: Our pose hypotheses proposals go through the N objects present in a scene multiple times.

Original: For object i of class t i , the pose hypotheses proposals sequentially go through all pose hypotheses for class t i .
Revised: For object i of class t i , the pose hypotheses proposals sequentially go through all pose hypotheses for class t i .

Original: For each one of these pose hypotheses, the pose hypotheses proposals sample a pose centered at the given pose hypothesis, and propose to change the pose of object i to the sampled pose.
Revised: For each one of these pose hypotheses, the pose hypotheses proposals sample a pose centered at the given pose hypothesis, and propose to change the pose of object i to the sampled pose.

Original: Going through the objects more times help improve performance, at the cost of increased computation.
Revised: Going through the objects more times help improve performance, at the cost of increased computation.

Original: In practice we find going through all objects twice strikes a good balance between performance and speed.
Revised: In practice we find going through all objects twice strikes a good balance between performance and speed.

Original: A. D ETAILS ON INFERENCE PIPELINE IMPLEMENTATION A.5. D ISCRETIZING THE ROTATION SPACE
Revised: A. D ETAILS ON INFERENCE PIPELINE IMPLEMENTATION A.5. D ISCRETIZING THE ROTATION SPACE

Original: /* Basic setups */
Revised: /* Basic setups */

Original: Object-specific: A set of surface points Z for object class t , query model f t , key model g t , n k keypoints with object frame coordinates x ∗ 1 , · · · , x ∗ n k ∈ R 3 .
Revised: Object-specific: A set of surface points Z for object class t , query model f t , key model g t , n k keypoints with object frame coordinates x ∗ 1 , · · · , x ∗ n k ∈ R 3 .

Original: Spatial discretization: Camera frame coordinates of the center of the boundary voxel y ∈ R 3 , size of the voxel grid ( L x , L y , L z ) , diameter of the voxels d > 0 .
Revised: Spatial discretization: Camera frame coordinates of the center of the boundary voxel y ∈ R 3 , size of the voxel grid ( L x , L y , L z ) , diameter of the voxels d > 0 .

Original: Orientation discretization: n r representative orientations R 1 , · · · , R n r ∈
Revised: Orientation discretization: n r representative orientations R 1 , · · · , R n r ∈

Original: SO
Revised: SO

Original: Parameters: Number of pose hypotheses to generate n p , number of top positions n t .
Revised: Parameters: Number of pose hypotheses to generate n p , number of top positions n t .

Original: 1 /* Enumeration-based pose hypotheses generation */
Revised: 1 /* Enumeration-based pose hypotheses generation */

Original: Input
Revised: Input

Original: Output
Revised: Output

Original: 2 Q ← f t
Revised: 2 Q ← f t

Original: I
Revised: I

Original: ; // Get query embeddings from the image I 3 V 0 ← Voting ( Q , C , (0 , 0 , 0) ); // Aggregation for object center (0 , 0 , 0) 4 for i ← 1 to n k do //
Revised: ; // Get query embeddings from the image I 3 V 0 ← Voting ( Q , C , (0 , 0 , 0) ); // Aggregation for object center (0 , 0 , 0) 4 for i ← 1 to n k do //

Original: Aggregation for n k keypoints.
Revised: Aggregation for n k keypoints.

Original: V i ← Voting ( Q , C , x ∗ i ); // V i ∈
Revised: V i ← Voting ( Q , C , x ∗ i ); // V i ∈

Original: R L x × L y ×
Revised: R L x × L y ×

Original: L z / * Identify top positions based on V 0 ’s largest entires.
Revised: L z / * Identify top positions based on V 0 ’s largest entires.

Original: * / 6 l 1 , · · · , l n t ← TopPositions ( V 0 ); // l 1 , · · · , l n t ∈ R7 for i ← 1 to n t , j ← 1 to n r do
Revised: * / 6 l 1 , · · · , l n t ← TopPositions ( V 0 ); // l 1 , · · · , l n t ∈ R7 for i ← 1 to n t , j ← 1 to n r do

Original: s i,j ← Scoring ( l i , R j , V 1 , · · · , V n k ); //
Revised: s i,j ← Scoring ( l i , R j , V 1 , · · · , V n k ); //

Original: Heuristic pose scoring 9 P t 1 , · · · , P tn p ← RankByScore ( l 1 , · · · , l n t , s i,j , i = 1 , · · · , n t , j = 1 , · · · , n r ); 10 return P t 1 , · · · , P tn p ; 11 /* Voting and heuristic scoring */ 12 def Voting ( Q , C , x ∗ ) //
Revised: Heuristic pose scoring 9 P t 1 , · · · , P tn p ← RankByScore ( l 1 , · · · , l n t , s i,j , i = 1 , · · · , n t , j = 1 , · · · , n r ); 10 return P t 1 , · · · , P tn p ; 11 /* Voting and heuristic scoring */ 12 def Voting ( Q , C , x ∗ ) //

Original: Voting-based evidence aggregation
Revised: Voting-based evidence aggregation

Original: V ← 0 ; //
Revised: V ← 0 ; //

Original: Initialize V ∈ R L x × L y ×
Revised: Initialize V ∈ R L x × L y ×

Original: L z to all 0 array for i ← 1 to H , j ← 1 to W do x ← arg max ˜ x P RGB (
Revised: L z to all 0 array for i ← 1 to H , j ← 1 to W do x ← arg max ˜ x P RGB (

Original: ˜ x | Q i,j , Z t , t ) , p i,j ← max ˜ x P RGB (˜ x | Q i,j , Z t , t ) ; for u ← 1 to L x , v ← 1 to L y , w ← 1 to L z do c ← ( y 1 + ( u − 1) d, y 

Revised: ˜ x | Q i,j , Z t , t ) , p i,j ← max ˜ x P RGB (˜ x | Q i,j , Z t , t ) ; for u ← 1 to L x , v ← 1 to L y , w ← 1 to L z do c ← ( y 1 + ( u − 1) d, y

Original: + ( v − 1) d, y 3 + ( w − 1) d ) ;
Revised: + ( v − 1) d, y 3 + ( w − 1) d ) ;

Original: if || C i,j − c || 2 ≈ || x − x ∗ || 2 then
Revised: if || C i,j − c || 2 ≈ || x − x ∗ || 2 then

Original: V u,v,w = V u,v,w + p
Revised: V u,v,w = V u,v,w + p

Original: i,j return V 21 def Scoring ( l, R , V 1 , · · · , V n k ) //
Revised: i,j return V 21 def Scoring ( l, R , V 1 , · · · , V n k ) //

Original: Heuristic pose scoring s ← 0 ; // Initialize score to for i ← 1 to n k do x ← R x ∗ i + l ; // Location of x ∗ i in world frame for pose l, R ( u, v, w ) ← Round [( x − y ) /d ] ; // Identify corresponding voxel of x s = s + V iu,v,w ; //
Revised: Heuristic pose scoring s ← 0 ; // Initialize score to for i ← 1 to n k do x ← R x ∗ i + l ; // Location of x ∗ i in world frame for pose l, R ( u, v, w ) ← Round [( x − y ) /d ] ; // Identify corresponding voxel of x s = s + V iu,v,w ; //

Original: Heuristic scoring return s
Revised: Heuristic scoring return s

Original: We discretize SO (3) into 6400 representative orientations.
Revised: We discretize SO (3) into 6400 representative orientations.

Original: We generate these orientations by first picking 200 points roughly uniformly on the unit sphere using the Fibonacci sphere.
Revised: We generate these orientations by first picking 200 points roughly uniformly on the unit sphere using the Fibonacci sphere.

Original: Therepresentative orientations are genearted by first rotating the axis (0 , 0 , 1 . 0) to point to one of the 200 points, followed by one of 32 in-plane rotations around the axis.
Revised: Therepresentative orientations are genearted by first rotating the axis (0 , 0 , 1 . 0) to point to one of the 200 points, followed by one of 32 in-plane rotations around the axis.

Original: A.5. U SING ADDITIONAL 2D DETECTION AND MASK PREDCITION FROM S URF EMB
Revised: A.5. U SING ADDITIONAL 2D DETECTION AND MASK PREDCITION FROM S URF EMB

Original: In the best performing setup, we leverage the same 2D detector used in SurfEMB as part of the pose hypotheses generation process.
Revised: In the best performing setup, we leverage the same 2D detector used in SurfEMB as part of the pose hypotheses generation process.

Original: Although our spherical voting procedure can robustly aggregate information from the entire image to generate pose hypotheses, as demonstrated by the competitive performance of 3DNEL MSIGP (No 2D detection) (Abalations in Table 1), in practice the query embedding images for many objects are very noisy and tend to hurt performance.
Revised: Although our spherical voting procedure can robustly aggregate information from the entire image to generate pose hypotheses, as demonstrated by the competitive performance of 3DNEL MSIGP (No 2D detection) (Abalations in Table 1), in practice the query embedding images for many objects are very noisy and tend to hurt performance.

Original: Empirically,  we observe that by additionally using 2D detections, we can focus the spherical voting process on regions of the observed image that is likely relevant for the objects, and further improve performance.
Revised: Empirically, [Figure] we observe that by additionally using 2D detections, we can focus the spherical voting process on regions of the observed image that is likely relevant for the objects, and further improve performance.

Original: For each object class in the scene, there can be multiple 2D detections.
Revised: For each object class in the scene, there can be multiple 2D detections.

Original: For each 2D detection, we do spherical voting just within the detector crop and generate 80 pose hypotheses per detector crop.
Revised: For each 2D detection, we do spherical voting just within the detector crop and generate 80 pose hypotheses per detector crop.

Original: When there is a missing 2D detection, we obtain the query embeddings by upsampling the input RGB image by 1.5x, and do spherical voting on the whole image.
Revised: When there is a missing 2D detection, we obtain the query embeddings by upsampling the input RGB image by 1.5x, and do spherical voting on the whole image.

Original: In such cases, when we identify top positions, we additionally do non-max suppression with a filter size of 10 to spread the top-scoring positions out.
Revised: In such cases, when we identify top positions, we additionally do non-max suppression with a filter size of 10 to spread the top-scoring positions out.

Original: For each such top-scoring position we identify top 2 orientation, and we consider all top-scoring positions and generate in total 30 pose hypotheses.
Revised: For each such top-scoring position we identify top 2 orientation, and we consider all top-scoring positions and generate in total 30 pose hypotheses.

Original: SurfEMB’s query embedding model was trained on image crops from 2D detections rescaled to 224 × 224 , but can be applied to RGB images of any size.
Revised: SurfEMB’s query embedding model was trained on image crops from 2D detections rescaled to 224 × 224 , but can be applied to RGB images of any size.

Original: Empirically we observe that due to SurfEMB’s rescaling of the 2D detection crops to 224 × 224 , the query embeddings for some objects are only informative when we properly rescale the input RGB image.
Revised: Empirically we observe that due to SurfEMB’s rescaling of the 2D detection crops to 224 × 224 , the query embeddings for some objects are only informative when we properly rescale the input RGB image.

Original: To account for this, we estimate the best scale by scoring the top pose hypotheses (using our heuristic scoring as described in
Revised: To account for this, we estimate the best scale by scoring the top pose hypotheses (using our heuristic scoring as described in

Original: Section 3.3) from each 2D detector crop and pick the scale of the highest-scoring 2D detector crop.
Revised: Section 3.3) from each 2D detector crop and pick the scale of the highest-scoring 2D detector crop.

Original: Concretely, when there are multiple 2D detections for an object class, we score the corresponding scales with the mean heuristic score of the top 5 pose hypotheses from that detector crop, and use the highest scoring scale to obtain the query embeddings for the object class.
Revised: Concretely, when there are multiple 2D detections for an object class, we score the corresponding scales with the mean heuristic score of the top 5 pose hypotheses from that detector crop, and use the highest scoring scale to obtain the query embeddings for the object class.

Original: When there is no 2D detections we fall back to a default scale of 1 . 5 .
Revised: When there is no 2D detections we fall back to a default scale of 1 . 5 .

Original: A.5. I MPLEMENTATION DETAILS AND HYPERPARAMETERS
Revised: A.5. I MPLEMENTATION DETAILS AND HYPERPARAMETERS

Original: We use OpenGL for rendering, and use Taichi (Hu et al., 2019) for both spherical voting and 3DNEL evaluation.
Revised: We use OpenGL for rendering, and use Taichi (Hu et al., 2019) for both spherical voting and 3DNEL evaluation.

Original: A Python implementation reproducing the results reported in the paper will be released upon acceptance.
Revised: A Python implementation reproducing the results reported in the paper will be released upon acceptance.

Original: As we describe in the main text, we pick hyperparameters by visually inspecting detection results on a small number of real training images that are outside the test set.
Revised: As we describe in the main text, we pick hyperparameters by visually inspecting detection results on a small number of real training images that are outside the test set.

Original: We select n k = 8 keypoints from the surface of each object class, and use y = ( − 350 . 0 , − 210 . 0 , 530 . 0) , L x = 129 , L y = 87 , L z = 168 and d = 5 . 0 to make the voxel grids large enough to cover all the keypoints that can be present in the camera frame.
Revised: We select n k = 8 keypoints from the surface of each object class, and use y = ( − 350 . 0 , − 210 . 0 , 530 . 0) , L x = 129 , L y = 87 , L z = 168 and d = 5 . 0 to make the voxel grids large enough to cover all the keypoints that can be present in the camera frame.

Original: Here the units for the values in y and d are mm.
Revised: Here the units for the values in y and d are mm.

Original: We use r = 5 . 0 in evaluating 3DNEl for all our experiments.
Revised: We use r = 5 . 0 in evaluating 3DNEl for all our experiments.

Original: We implement 3DNEL with Taichi to leverage powerful GPUs.
Revised: We implement 3DNEL with Taichi to leverage powerful GPUs.

Original: When identifying points in the rendered point cloud ˜C (organized as an H × W × 3 image) that is within distance r from a point ( i, j ) in the observed point cloud C , to further speed up 3DNEL evaluation, we only look at the points from the patch of size (10 , 10) centered at ( i, j ) .
Revised: When identifying points in the rendered point cloud ˜ C (organized as an H × W × 3 image) that is within distance r from a point ( i, j ) in the observed point cloud C , to further speed up 3DNEL evaluation, we only look at the points from the patch of size (10 , 10) centered at ( i, j ) .

Original: For MCMC finetuning, we go through all pose hypotheses twice in the pose hypotheses proposals.
Revised: For MCMC finetuning, we go through all pose hypotheses twice in the pose hypotheses proposals.

Original: We go through all objects 5 times in the ICP proposals.
Revised: We go through all objects 5 times in the ICP proposals.

Original: We go through all objects 50 times in the random walk proposals.
Revised: We go through all objects 50 times in the random walk proposals.

Original: Each time we go through an object in the random walk proposals, we propose 7 types of random pose perturbations.
Revised: Each time we go through an object in the random walk proposals, we propose 7 types of random pose perturbations.

Original: A. A DDITIONAL ROBUSTNESS RESULTS
Revised: A. A DDITIONAL ROBUSTNESS RESULTS

Original: In Figure A1 we include additional robustness results.
Revised: In Figure A1 we include additional robustness results.

Original: A. B ENEFITS OF MODELING UNCERTAINTY : AN ADDITIONAL TRACKING EXPERIMENT
Revised: A. B ENEFITS OF MODELING UNCERTAINTY : AN ADDITIONAL TRACKING EXPERIMENT

Original: To hightlight the benefits of modeling uncertainty, a key advantage enabled by 3DNEL’s probabilistic formulation, we present an additional experiment on object tracking in a synthetic video containing two  objects, with one object being static and the other object moving across the scene.
Revised: To highlight the benefits of modeling uncertainty, a key advantage enabled by 3DNEL’s probabilistic formulation, we present an additional experiment on object tracking in a synthetic video containing two YCB objects, the cracker box and the sugar box. The cracker box remains static while the sugar box moves across the scene.

Original: The moving object gets fully occluded by the static object before reappearing.
Revised: The sugar box gets fully occluded by the cracker box before reappearing.

Original: We visualize an example  synthetic video  in Figure A2.Due to the simple and synthetic nature of the data, we use a variant of the 3DNEL formulation that only uses depth information for this proof-of-concept experiment, and relies on simple local random walk proposals for MCMC inference. [Figure]
Revised: We generated this video in PyBullet (Coumans & Bai, 2016–2021) using the textured mesh models from YCB-V dataset for the cracker box and sugar box. Some example frames from the synthetic video are shown in Figure A2.

Original: This is a challenging setup for methods that do not model uncertainty but only rely on a single point estimate, as such methods tend to lose track once the moving object becomes severely occluded.
Revised: This is a challenging setup for methods that do not model uncertainty but only rely on a single estimation of the object pose, as such methods tend to lose track once the moving object becomes severely occluded.

Original: We observe that the current 3DNEL extension to  object  tracking indeed fails.
Revised: We find that when 3DNEL is used to only produce a single estimation of the object pose, tracking indeed fails.

Original: However, but leveraging
Revised: However, when we apply 3DNEL for tracking using Sequential Monte Carlo, we are able to approximate the posterior distribution over the object’s pose at each timestep and capture the uncertainty in the pose of the moving object when it gets occluded (rather than a single estimation of the object pose). We find that this can help us accurately trace the object and regain track of the moving object when it reappears from occlusion. For Sequential Monte Carlo, we work in a particle filtering setup, and assume a simple dynamics model in which the object’s translation is sampled from a multivariate Gaussian centered at the translation from the previous timestep, and the orientation is sampled from a Von-Mises-Fisher distribution centered at the orientation from the previous timestep.

Original: MCMC inference to maintain a full posterior distribution on the object poses (represented as multiple particles in a particle filtering setup), we are able to properly model uncertainty and achieve successful tracking: the posterior distribution would spread out across the scene when the moving object becomes highly occluded (since there is a high-level of uncertainty on where the object is), which allows the system to continue tracking the object when it reappears.
Revised: In Figure A3, we visualize the tracking results and the inferred posterior distribution (approximated via particles in a particle filtering setup) at different timesteps. When the sugar box becomes occluded, we see the posterior distribution on the pose of the sugar box spread out across the many positions the sugar box could be at behind the cracker box. When the sugar box eventually reappears we can regain track of its pose.

Original:  Note that for this experiment, we leverage an improved pure JAX implementation that can support a large number of 3DNEL evaluations in parallel.
Revised: On the other hand, methods that maintain only a single object pose estimation will almost surely lose track once the moving object gets occluded. Note that for this experiment, we leverage an improved pure JAX implementation of the 3DNEL likelihood. We implemented a JAX depth image renderer, which enables a large number of 3DNEL evaluations in parallel.

Original: This allows us to achieve 10 frames-per-second tracking speed for 480 × 640 images on a machine with a single NVIDIA A100 GPU.
Revised: This allows us to achieve real-time tracking performance for 480 ×images on a machine with a single NVIDIA A100 GPU.

Original: Of course the involved data are synthetic and much simpler.
Revised: Of course the involved data are synthetic and much simpler.

Original: However this is a promising early step towards significantly speeding up the 3DNEL inference process.
Revised: However this is a promising early step towards significantly speeding up the 3DNEL inference process.

Original: We are additionally putting together an experiment where we generate similar videos but using

Original: YCB objects, and do inference with the full 3DNEL formulation combining both RGB and depth information.

Original: We will update the results as soon as we have them ready.

Original: A. A DDITIONAL VISUALIZATIONS OF S URF EMB AND 3DNEL MSIGP PREDICTIONS ON YCB-V TEST IMAGES
Revised: A. A DDITIONAL VISUALIZATIONS OF S URF EMB AND 3DNEL MSIGP PREDICTIONS ON YCB-V TEST IMAGES

Original: In Figure A3 we include additional visualizations of SurfEMB and 3DNEL MSIGP predictions on
Revised: In Figure A4 we include additional visualizations of SurfEMB and 3DNEL MSIGP predictions on

Original: YCB-V test images.
Revised: YCB-V test images.