Original: LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery
Revised: LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery

Original: Abstract
Revised: Abstract

Original: Creating high-quality articulated 3D models of animals is challenging either via manual creation or using 3D scanning tools.
Revised: Creating high-quality articulated 3D models of animals is challenging either via manual creation or using 3D scanning tools.

Original: Therefore, techniques to reconstruct articulated 3D objects from 2D images are crucial and highly useful.
Revised: Therefore, techniques to reconstruct articulated 3D objects from 2D images are crucial and highly useful.

Original: In this work, we propose a practical problem setting to estimate 3D pose and shape of animals given only a few (10-30) in-the-wild images of a particular animal species (say, horse).
Revised: In this work, we propose a practical problem setting to estimate 3D pose and shape of animals given only a few (10-30) in-the-wild images of a particular animal species (say, horse).

Original: Contrary to existing works that rely on pre-defined template shapes, we do not assume any form of 2D or 3D ground-truth annotations, nor do we leverage any multi-view or temporal information.
Revised: Contrary to existing works that rely on pre-defined template shapes, we do not assume any form of 2D or 3D ground-truth annotations, nor do we leverage any multi-view or temporal information.

Original: Moreover, each input image ensemble can contain animal instances with varying poses, backgrounds, illuminations, and textures.
Revised: Moreover, each input image ensemble can contain animal instances with varying poses, backgrounds, illuminations, and textures.

Original: Our key insight is that 3D parts have much simpler shape compared to the overall animal and that they are robust w.r.t.
Revised: Our key insight is that 3D parts have much simpler shape compared to the overall animal and that they are robust w.r.t.

Original: animal pose articulations.
Revised: animal pose articulations.

Original: Following these insights, we propose LASSIE, a novel optimization framework which discovers 3D parts in a self-supervised manner with minimal user intervention.
Revised: Following these insights, we propose LASSIE, a novel optimization framework which discovers 3D parts in a self-supervised manner with minimal user intervention.

Original: A key driving force behind LASSIE is the enforcing of 2D-3D part consistency using self-supervisory deep features.
Revised: A key driving force behind LASSIE is the enforcing of 2D-3D part consistency using self-supervisory deep features.

Original: Experiments on Pascal-Part and self-collected in-the-wild animal datasets demonstrate considerably better 3D reconstructions as well as both 2D and 3D part discovery compared to prior arts.
Revised: Experiments on Pascal-Part and self-collected in-the-wild animal datasets demonstrate considerably better 3D reconstructions as well as both 2D and 3D part discovery compared to prior arts.

Original: Project page: https://chhankyao.github.io/lassie/
Revised: Project page: https://chhankyao.github.io/lassie/

Original:  Introduction
Revised:  Introduction

Original: The last decade has witnessed significant advances in estimating human body pose and shape from images [26, 2, 15, 19, 18, 34, 17].
Revised: The last decade has witnessed significant advances in estimating human body pose and shape from images [26, 2, 15, 19, 18, 34, 17].

Original: Much progress is fueled by the availability of rich datasets [14, 29, 39] with 3D human scans in a variety of shapes and poses.
Revised: Much progress is fueled by the availability of rich datasets [14, 29, 39] with 3D human scans in a variety of shapes and poses.

Original: In contrast, 3D scanning wild animals is quite challenging and existing animal datasets such as SMALR [48] rely on man-made animal shapes.
Revised: In contrast, 3D scanning wild animals is quite challenging and existing animal datasets such as SMALR [48] rely on man-made animal shapes.

Original: Manually creating realistic animal models for a wide diversity of animals (e g. mammals and birds) is also challenging and time-consuming.
Revised: Manually creating realistic animal models for a wide diversity of animals (e g. mammals and birds) is also challenging and time-consuming.

Original: In this work, we propose to automatically estimate 3D shape and articulation (pose) of animals from only a sparse set of image ensemble (collection) without using any 2D or 3D ground-truth annotations.
Revised: In this work, we propose to automatically estimate 3D shape and articulation (pose) of animals from only a sparse set of image ensemble (collection) without using any 2D or 3D ground-truth annotations.

Original: Our problem setting is highly practical as each image ensemble consists of only a few (10-30) in-the-wild images of a specific animal species.
Revised: Our problem setting is highly practical as each image ensemble consists of only a few (10-30) in-the-wild images of a specific animal species.

Original: Fig.shows sample zebra images which forms our input.
Revised: Fig.shows sample zebra images which forms our input.

Original: In addition, we assume a human-specified 3D skeleton (left in Fig.
Revised: In addition, we assume a human-specified 3D skeleton (left in Fig.

Original: 1) that can be quite rough with incorrect bone lengths and this mainly provides the connectivity of animal parts.
Revised: 1) that can be quite rough with incorrect bone lengths and this mainly provides the connectivity of animal parts.

Original: For instance, we use the same 3D skeleton for all the quadruped animals despite considerable shape variations across species, e.g., giraffe vs. zebra vs. elephant.
Revised: For instance, we use the same 3D skeleton for all the quadruped animals despite considerable shape variations across species, e.g., giraffe vs. zebra vs. elephant.

Original: Manually specifying a rough skeleton is a trivial task which only takes a few minutes.
Revised: Manually specifying a rough skeleton is a trivial task which only takes a few minutes.

Original: Estimating articulated 3D shapes from in-the-wild image ensemble is a highly ambiguous and challenging problem.
Revised: Estimating articulated 3D shapes from in-the-wild image ensemble is a highly ambiguous and challenging problem.

Original: There are several varying factors across images: backgrounds; lighting; camera
Revised: There are several varying factors across images: backgrounds; lighting; camera

Original: [Figure]
Revised: [Figure]

Original: The part-based representation also allows applications like texture and pose transfer, animation, etc viewpoints; animal shape, pose and texture, etc.
Revised: The part-based representation also allows applications like texture and pose transfer, animation, etc viewpoints; animal shape, pose and texture, etc.

Original: In addition, our highly practical problem setting does not allow access to any 3D animal models, per-image annotations such as keypoints and silhouettes, or multi-view images.
Revised: In addition, our highly practical problem setting does not allow access to any 3D animal models, per-image annotations such as keypoints and silhouettes, or multi-view images.

Original: In this work, we present LASSIE, an optimization technique to L earn A rticulated S hapes from S parse
Revised: In this work, we present LASSIE, an optimization technique to L earn A rticulated S hapes from S parse

Original: I mage E nsemble.
Revised: I mage E nsemble.

Original: Our key observation is that shapes are composed of 3D parts with the following characteristics: 1) 3D parts are geometrically and semantically consistent across different instances.
Revised: Our key observation is that shapes are composed of 3D parts with the following characteristics: 1) 3D parts are geometrically and semantically consistent across different instances.

Original: Shapes of 3D parts remain relatively constant w.r.t.
Revised: Shapes of 3D parts remain relatively constant w.r.t.

Original: changes in animal poses.
Revised: changes in animal poses.

Original: Despite the complex shape of overall animal body, 3D parts are usually made of simple convex shapes.
Revised: Despite the complex shape of overall animal body, 3D parts are usually made of simple convex shapes.

Original: Following these observations, LASSIE optimizes 3D parts and their articulation instead of directly optimizing the overall animal shape.
Revised: Following these observations, LASSIE optimizes 3D parts and their articulation instead of directly optimizing the overall animal shape.

Original: Specifically, LASSIE optimizes a 3D skeleton and part shapes that are shared across instances; while also optimizing instance-specific camera viewpoint, pose articulation, and surface texture.
Revised: Specifically, LASSIE optimizes a 3D skeleton and part shapes that are shared across instances; while also optimizing instance-specific camera viewpoint, pose articulation, and surface texture.

Original: To enforce semantic part consistency across different images, we leverage deep features from a vision transformer [10] (DINO-ViT [6]) trained in a self-supervised fashion.
Revised: To enforce semantic part consistency across different images, we leverage deep features from a vision transformer [10] (DINO-ViT [6]) trained in a self-supervised fashion.

Original: Recent methods [1, 38] demonstrate good 2D feature correspondences in DINO-ViT features and we make use of these correspondences for 3D part discovery.
Revised: Recent methods [1, 38] demonstrate good 2D feature correspondences in DINO-ViT features and we make use of these correspondences for 3D part discovery.

Original: To regularize part shapes, we constrain the optimized parts to a latent shape manifold that is learned using 3D geometric primitives (spheres, cylinders, cones etc.).
Revised: To regularize part shapes, we constrain the optimized parts to a latent shape manifold that is learned using 3D geometric primitives (spheres, cylinders, cones etc.).

Original: Being self-supervised at the image level, LASSIE can easily generalize to a wide range of animal species with minimal user intervention.
Revised: Being self-supervised at the image level, LASSIE can easily generalize to a wide range of animal species with minimal user intervention.

Original: We conduct extensive experiments on image ensembles from the Pascal-Part Dataset [7] with quantitative evaluations using 2D ground-truth part segmentation and keypoints.
Revised: We conduct extensive experiments on image ensembles from the Pascal-Part Dataset [7] with quantitative evaluations using 2D ground-truth part segmentation and keypoints.

Original: We also collect in-the-wild web image ensembles with more animal species and annotate 2D keypoints for evaluation.
Revised: We also collect in-the-wild web image ensembles with more animal species and annotate 2D keypoints for evaluation.

Original: Both qualitative and quantitative results demonstrate considerably better 3D reconstructions compared to the prior arts that use even more supervision.
Revised: Both qualitative and quantitative results demonstrate considerably better 3D reconstructions compared to the prior arts that use even more supervision.

Original: In addition, estimation of explicit 3D parts enables easy editing and manipulation of animal parts.
Revised: In addition, estimation of explicit 3D parts enables easy editing and manipulation of animal parts.

Original: Fig.
Revised: Fig.

Original: 1 shows sample 3D reconstructions with discovered parts from LASSIE.
Revised: 1 shows sample 3D reconstructions with discovered parts from LASSIE.

Original: The main contributions of this work are:
Revised: The main contributions of this work are:

Original: • To the best of our knowledge, this is the first work that recovers 3D articulated shapes from in-the-wild image ensembles without using any pre-defined animal shape models or instance-specific annotations like keypoints or silhouettes.
Revised: • To the best of our knowledge, this is the first work that recovers 3D articulated shapes from in-the-wild image ensembles without using any pre-defined animal shape models or instance-specific annotations like keypoints or silhouettes.

Original: • We propose a novel optimization framework called LASSIE, where we incorporate several useful properties of mid-level 3D part representation.
Revised: • We propose a novel optimization framework called LASSIE, where we incorporate several useful properties of mid-level 3D part representation.

Original: LASSIE is category-agnostic and can easily generalize to a wide range of animal species.
Revised: LASSIE is category-agnostic and can easily generalize to a wide range of animal species.

Original: • LASSIE can produce high-quality 3D articulated shapes with state-of-the-art results on existing benchmarks as well as self-collected in-the-wild web image ensembles.
Revised: • LASSIE can produce high-quality 3D articulated shapes with state-of-the-art results on existing benchmarks as well as self-collected in-the-wild web image ensembles.

Original:  Related Work
Revised:  Related Work

Original: Animal shape and pose estimation.
Revised: Animal shape and pose estimation.

Original: Estimating 3D pose and shape of animal bodies from in-the-wild images is highly ill-posed and challenging due to large variations across different classes, instances, viewpoints, articulations, etc.
Revised: Estimating 3D pose and shape of animal bodies from in-the-wild images is highly ill-posed and challenging due to large variations across different classes, instances, viewpoints, articulations, etc.

Original: Most existing mesh reconstruction work [21, 24, 11, 36, 44] focus more on compact shapes like birds and cars, and thus cannot handle articulations of animal bodies.
Revised: Most existing mesh reconstruction work [21, 24, 11, 36, 44] focus more on compact shapes like birds and cars, and thus cannot handle articulations of animal bodies.

Original: Recent articulation-aware methods, on the other hand, deal with a simplified scenario by assuming a pre-defined class-level statistical model, instance-specific images, or accessible human annotations.
Revised: Recent articulation-aware methods, on the other hand, deal with a simplified scenario by assuming a pre-defined class-level statistical model, instance-specific images, or accessible human annotations.

Original: For instance, Zuffi et al.
Revised: For instance, Zuffi et al.

Original: [49] build a statistical shape model, SMAL, for common quadrupedal animals, similar to the SMPL [26] model for human bodies.
Revised: [49] build a statistical shape model, SMAL, for common quadrupedal animals, similar to the SMPL [26] model for human bodies.

Original: Follow-up work either optimize the
Revised: Follow-up work either optimize the

Original: SMAL parameters based on human annotated images [48] or train a neural network on large-scale image datasets to directly regress the parameters [47, 35].
Revised: SMAL parameters based on human annotated images [48] or train a neural network on large-scale image datasets to directly regress the parameters [47, 35].

Original: However, the output shapes are limited by the SMAL shape space or training images which contain one or few animal classes.
Revised: However, the output shapes are limited by the SMAL shape space or training images which contain one or few animal classes.

Original: Kulkarni et al.
Revised: Kulkarni et al.

Original: [20] propose A-CSM technique that align a known template mesh with skinning weights to input 2D images.
Revised: [20] propose A-CSM technique that align a known template mesh with skinning weights to input 2D images.

Original: Other recent approaches [41, 40, 42] exploit the dense temporal correspondence in video frames to reconstruct articulated shapes.
Revised: Other recent approaches [41, 40, 42] exploit the dense temporal correspondence in video frames to reconstruct articulated shapes.

Original: In contrast to existing methods, we deal with a novel and practical problem setting: learn without any pre-defined shape model, instance-level human annotations, or temporal information to leverage.
Revised: In contrast to existing methods, we deal with a novel and practical problem setting: learn without any pre-defined shape model, instance-level human annotations, or temporal information to leverage.

Original: Part discovery from image collections.
Revised: Part discovery from image collections.

Original: Deep feature factorization (DFF) [9] shows that one could automatically obtain consistent part segments by clustering intermediate deep features of a classification network trained on ImageNet.
Revised: Deep feature factorization (DFF) [9] shows that one could automatically obtain consistent part segments by clustering intermediate deep features of a classification network trained on ImageNet.

Original: Inspired by DFF, SCOPS [13] learns a 2D part segmentation network from image collections, which can be used as semantic supervision for 3D reconstruction [24].
Revised: Inspired by DFF, SCOPS [13] learns a 2D part segmentation network from image collections, which can be used as semantic supervision for 3D reconstruction [24].

Original: Choudury et al.
Revised: Choudury et al.

Original: [8] proposed to use contrastive learning for 2D part discovery.
Revised: [8] proposed to use contrastive learning for 2D part discovery.

Original: Lathuilière et al.
Revised: Lathuilière et al.

Original: [22] exploit motion cues in videos for part discovery.
Revised: [22] exploit motion cues in videos for part discovery.

Original: Recently, Amir et al.
Revised: Recently, Amir et al.

Original: [1] demonstrate that clustering self-supervisedly learned vision transformers (ViT) such as DINO [6] features can provide good 2D part segmentations.
Revised: [1] demonstrate that clustering self-supervisedly learned vision transformers (ViT) such as DINO [6] features can provide good 2D part segmentations.

Original: In this work, we make use of DINO features for 3D part discovery instead of 2D segmentation.
Revised: In this work, we make use of DINO features for 3D part discovery instead of 2D segmentation.

Original: In the 3D domain, Tulsiani et al.
Revised: In the 3D domain, Tulsiani et al.

Original: [37] use volumetric cuboids as part abstractions to learn 3D reconstruction.
Revised: [37] use volumetric cuboids as part abstractions to learn 3D reconstruction.

Original: Mandikal et al.
Revised: Mandikal et al.

Original: [28] predict part-segmented 3D reconstructions from a single image.
Revised: [28] predict part-segmented 3D reconstructions from a single image.

Original: In addition, Luo et al.
Revised: In addition, Luo et al.

Original: [27] and Paschalidou et al.
Revised: [27] and Paschalidou et al.

Original: [31, 32] learn to form object parts by clustering 3D points.
Revised: [31, 32] learn to form object parts by clustering 3D points.

Original: Considering that most 3D approaches require ground-truth 3D shape of a whole object or its parts as supervision, Yao et al.
Revised: Considering that most 3D approaches require ground-truth 3D shape of a whole object or its parts as supervision, Yao et al.

Original: [43] propose to discover and reconstruct 3D parts automatically by learning a primitive shape prior.
Revised: [43] propose to discover and reconstruct 3D parts automatically by learning a primitive shape prior.

Original: Most of these works assume some form of supervision in terms of 3D shape or camera viewpoint.
Revised: Most of these works assume some form of supervision in terms of 3D shape or camera viewpoint.

Original: In this paper, we aim to discover 3D parts of articulated animals from image ensembles without any 2D or 3D annotations, which, to the best of our knowledge, is unexplored in the literature.
Revised: In this paper, we aim to discover 3D parts of articulated animals from image ensembles without any 2D or 3D annotations, which, to the best of our knowledge, is unexplored in the literature.

Original: 3D reconstruction from sparse images.
Revised: 3D reconstruction from sparse images.

Original: Optimizing a 3D scene or object from multi-view images is one of the widely-studied problems in computer vision.
Revised: Optimizing a 3D scene or object from multi-view images is one of the widely-studied problems in computer vision.

Original: The majority of recent breakthroughs are based on the powerful Neural Radiance Field (NeRF) [30] representation.
Revised: The majority of recent breakthroughs are based on the powerful Neural Radiance Field (NeRF) [30] representation.

Original: Given a set of multi-view images, NeRF can learn a neural volume from which one can render high-quality novel views.
Revised: Given a set of multi-view images, NeRF can learn a neural volume from which one can render high-quality novel views.

Original: Closely related to our work, NeRS [45] optimizes a neural surface representation from a sparse set of image collections and also only requires rough camera initialization.
Revised: Closely related to our work, NeRS [45] optimizes a neural surface representation from a sparse set of image collections and also only requires rough camera initialization.

Original: In this work, we use a neural surface representation to represent animal parts.
Revised: In this work, we use a neural surface representation to represent animal parts.

Original: Existing methods assume multi-view images captured in the same illumination setting and also the object appearance (texture) to be same across images.
Revised: Existing methods assume multi-view images captured in the same illumination setting and also the object appearance (texture) to be same across images.

Original: Another set of work [3, 46, 5] propose neural reflectance decomposition on image collections captured in varying illuminations.
Revised: Another set of work [3, 46, 5] propose neural reflectance decomposition on image collections captured in varying illuminations.

Original: But they work on rigid objects with ground-truth segmentation and known camera poses.
Revised: But they work on rigid objects with ground-truth segmentation and known camera poses.

Original: A concurrent work, SAMURAI [4] jointly reasons about camera pose along with shape and materials of rigid objects from image collections.
Revised: A concurrent work, SAMURAI [4] jointly reasons about camera pose along with shape and materials of rigid objects from image collections.

Original: In contrast, the input for our method consists of in-the-wild animal images with varying textures, viewpoints and pose articulations and captured in different environments.
Revised: In contrast, the input for our method consists of in-the-wild animal images with varying textures, viewpoints and pose articulations and captured in different environments.

Original:  Approach
Revised:  Approach

Original: Given a generic 3D skeleton and few images of an articulated animal species, LASSIE optimizes the camera viewpoint and articulation for each instance as well as the resting canonical skeleton and part shapes that are shared across all instances.
Revised: Given a generic 3D skeleton and few images of an articulated animal species, LASSIE optimizes the camera viewpoint and articulation for each instance as well as the resting canonical skeleton and part shapes that are shared across all instances.

Original: We introduce our neural part representation in Section 3.
Revised: We introduce our neural part representation in Section 3.

Original: and the optimization framework in Section 3.2.
Revised: and the optimization framework in Section 3.2.

Original: An overview of LASSIE is illustrated in Fig.
Revised: An overview of LASSIE is illustrated in Fig.

Original: 3.1 Articulated shapes with neural part surfaces
Revised: 3.1 Articulated Shapes with Neural Part Surfaces

Original: 3D skeleton.
Revised: 3D skeleton.

Original: Constructing a template mesh or statistical shape model for specific animal classes requires either the ground-truth 3D shapes or extensive human annotation, which are both hard to obtain in real-world scenarios.
Revised: Constructing a template mesh or statistical shape model for specific animal classes requires either the ground-truth 3D shapes or extensive human annotation, which are both hard to obtain in real-world scenarios.

Original: A 3D skeleton, on the other hand, is easy to annotate since it
Revised: A 3D skeleton, on the other hand, is easy to annotate since it

Original: Prior MLP ℱ !
Revised: Prior MLP ℱ !

Original: Unit sphere
Revised: Unit sphere

Original: Part outputs
Revised: Part outputs

Original: Textured outputs
Revised: Textured outputs

Original: Part MLPs { ℱ "∆ } (Δx, Δy, Δz) (u,v)
Revised: Part MLPs { ℱ "∆ } (Δx, Δy, Δz) (u,v)

Original: Primitive part encodings { e i }
Revised: Primitive part encodings { e i }

Original: Camera-space articulation
Revised: Camera-space articulation

Original: Primitive parts (x, y, z)
Revised: Primitive parts (x, y, z)

Original: Input images
Revised: Input images

Original: Sample texture
Revised: Sample texture

Original: DINO features ℒ $"% ℒ $&’ … …
Revised: DINO features ℒ $"% ℒ $&’ … …

Original: Canonical articulation 𝜋 R, t { e i } { ℱ "∆ }
Revised: Canonical articulation 𝜋 R, t { e i } { ℱ "∆ }

Original: Resting pose
Revised: Resting pose

Original: Neural Part Representation
Revised: Neural Part Representation

Original: LASSIE optimization the linear or neural blend skinning models [40, 42] where bones are unconstrained Gaussian components, our skeleton-based representation allows us to regularize bone transformations,connected
Revised: LASSIE optimization the linear or neural blend skinning models [40, ] where bones are unconstrained Gaussian components, our skeleton-based representation allows us to regularize bone transformations,

Original: Neural part surfaces.
Revised: Neural part surfaces.

Original: Given a skeleton which specifies the 3D joints and bones, we represent the articulated shape as a composition of several neural parts.
Revised: Given a skeleton which specifies the 3D joints and bones, we represent the articulated shape as a composition of several neural parts.

Original: Specifically, each part is defined as a neural surface wrapped around a skeleton bone.
Revised: Specifically, each part is defined as a neural surface wrapped around a skeleton bone.

Original: Motivated by object representation in NeRS [45], we represent part surface as a multi-layer perceptron (MLP) network that predicts the 3D deformation of a given continuous UV coordinate on the unit sphere.
Revised: Motivated by object representation in NeRS [45], we represent part surface as a multi-layer perceptron (MLP) network that predicts the 3D deformation of a given continuous UV coordinate on the unit sphere.

Original: A key advantage of modeling parts instead of whole animal is that part shapes are usually of simple convex shapes and can be represented well with a deformed sphere.
Revised: A key advantage of modeling parts instead of whole animal is that part shapes are usually of simple convex shapes and can be represented well with a deformed sphere.

Original: Compared to explicit mesh representation, the neural representation enables efficient mesh regularization on the output shapes while producing high resolution surfaces as MLPs can work with meshes of arbitrary vertex resolution.
Revised: Compared to explicit mesh representation, the neural representation enables efficient mesh regularization on the output shapes while producing high resolution surfaces as MLPs can work with meshes of arbitrary vertex resolution.

Original: Each part surface is first reconstructed in the canonical space, then scaled by the corresponding bone length, rotated by the bone orientation, and centered at the bone centroid.
Revised: Each part surface is first reconstructed in the canonical space, then scaled by the corresponding bone length, rotated by the bone orientation, and centered at the bone centroid.

Original: This ensures that the parts are connected under various bone transformations and also easy to repose.
Revised: This ensures that the parts are connected under various bone transformations and also easy to repose.

Original: Formally, let X ∈ R 3 × m denote the m uniformly distributed 3D vertices on a unit sphere.
Revised: Formally, let X ∈ R 3 × m denote the m uniformly distributed 3D vertices on a unit sphere.

Original: Further let R i ∈
Revised: Further let R i ∈

Original: R 3 × 3 , t i ∈ R 3 and s
Revised: R 3 × 3 , t i ∈ R 3 and s

Original: i ∈
Revised: i ∈

Original: R denote the global rotation, translation and scaling of the i th part ( i ∈ { 0 , · · · , b } ), where b denotes the number of parts.
Revised: R denote the global rotation, translation and scaling of the i th part ( i ∈ { 0 , · · · , b } ), where b denotes the number of parts.

Original: We first deform the sphere to obtain part shape using the part MLP, X → F i ( X ) ; followed by global transformation to obtain part vertices V i ∈ R 3 × m in the global coordinate frame:
Revised: We first deform the sphere to obtain part shape using the part MLP, X → F i ( X ) ; followed by global transformation to obtain part vertices V i ∈ R 3 × m in the global coordinate frame:

Original: [Equation]
Revised: [Equation]

Original: Latent part prior.
Revised: Latent part prior.

Original: Although fitting neural part surfaces on sparse images can produce reasonable outputs that are faithful to the input view, we observe that part shapes tend to be non-uniform or unrealistic from other novel views.
Revised: Although fitting neural part surfaces on sparse images can produce reasonable outputs that are faithful to the input view, we observe that part shapes tend to be non-uniform or unrealistic from other novel views.

Original: To further regularize the part shapes, we decompose the part MLP
Revised: To further regularize the part shapes, we decompose the part MLP

Original: F i into a prior MLP, F p and a part deformation MLP, F ∆ i .
Revised: F i into a prior MLP, F p and a part deformation MLP, F ∆ i .

Original: The prior MLP is trained to produce the base shapes that are close to geometric primitives such as spheres, cylinders, etc.
Revised: The prior MLP is trained to produce the base shapes that are close to geometric primitives such as spheres, cylinders, etc.

Original: The deformation
Revised: The deformation

Original: MLP is then used to provide additional deformations on top of the base shapes.
Revised: MLP is then used to provide additional deformations on top of the base shapes.

Original: Fig.
Revised: Fig.

Original: 2 illustrates these two MLPs.
Revised: 2 illustrates these two MLPs.

Original: As shown in the figure, the prior MLP takes as input the surface coordinate as well as a d -dimensional latent shape code e i ∈ R d .
Revised: As shown in the figure, the prior MLP takes as input the surface coordinate as well as a d -dimensional latent shape code e i ∈ R d .

Original: Both the latent space and the prior MLP are trained using geometric primitive shapes.
Revised: Both the latent space and the prior MLP are trained using geometric primitive shapes.

Original: Concretely, we train a Variational Auto-Encoder (VAE) which takes a primitive mesh as input, predicts the latent shape encodings and reconstructs the primitive through a conditional neural surface decoder which forms our prior MLP.
Revised: Concretely, we train a Variational Auto-Encoder (VAE) which takes a primitive mesh as input, predicts the latent shape encodings and reconstructs the primitive through a conditional neural surface decoder which forms our prior MLP.

Original: The primitive meshes in this work are randomly sampled geometric primitives (spheres, ellipsoids, cylinders, and cones).
Revised: The primitive meshes in this work are randomly sampled geometric primitives (spheres, ellipsoids, cylinders, and cones).

Original: With the use of prior and deformation MLPs, Eq. 1 now becomes:
Revised: With the use of prior and deformation MLPs, Eq. 1 now becomes:

Original: [Equation]
Revised: [Equation]

Original: where e i denotes the latent shape encodings of part i .
Revised: where e i denotes the latent shape encodings of part i .

Original: 3.2 Discovering 3D Neural Parts from Image Ensemble
Revised: 3.2 Discovering 3D Neural Parts from Image Ensemble

Original: Optimization setting.
Revised: Optimization setting.

Original: The input to our optimization is a sparse set (typically 20-30) of n in-the-wild images { I j } nj =1 along with a generic/rough 3D skeleton P ∈ R p × 3 that has p joints and b bones/parts.
Revised: The input to our optimization is a sparse set (typically 20-30) of n in-the-wild images { I j } nj =1 along with a generic/rough 3D skeleton P ∈ R p × 3 that has p joints and b bones/parts.

Original: The only image-level annotations for optimization are from the self-supervisory DINO [6] features.
Revised: The only image-level annotations for optimization are from the self-supervisory DINO [6] features.

Original: All the instances in the input image ensemble are of same species, but could have varying appearance due to texture, pose, camera angle, lighting, etc.
Revised: All the instances in the input image ensemble are of same species, but could have varying appearance due to texture, pose, camera angle, lighting, etc.

Original: We also assume that the entire animal body is visible in each image without truncated or occluded by another object although some degree of self-occlusion is allowed.
Revised: We also assume that the entire animal body is visible in each image without truncated or occluded by another object although some degree of self-occlusion is allowed.

Original: Let j denote the index over images j ∈ { 1 , ., n } and i denote the index over parts i ∈ { 1 , .
Revised: Let j denote the index over images j ∈ { 1 , ., n } and i denote the index over parts i ∈ { 1 , .

Original: , b } .
Revised: , b } .

Original: For each image, LASSIE optimizes the camera viewpoint π j = ( R 0 , t 0 ) and part rotations R j ∈ R b × 3 × 3 .
Revised: For each image, LASSIE optimizes the camera viewpoint π j = ( R 0 , t 0 ) and part rotations R j ∈ R b × 3 × 3 .

Original: In addition, LASSIE optimizes these variables for each of the i th part that are shared across all the animal instances: bone length scaling s i ∈ R , latent shape encodings e i ∈ R d and part deformation MLPs F ∆ i .
Revised: In addition, LASSIE optimizes these variables for each of the i th part that are shared across all the animal instances: bone length scaling s i ∈ R , latent shape encodings e i ∈ R d and part deformation MLPs F ∆ i .

Original: That is, we assume that part shapes are shared across all instances whereas their articulation (pose transformation) is instance dependent.
Revised: That is, we assume that part shapes are shared across all instances whereas their articulation (pose transformation) is instance dependent.

Original: Although this may not be strictly valid in practice, this provides an important constraint for our highly ill-posed optimization problem.
Revised: Although this may not be strictly valid in practice, this provides an important constraint for our highly ill-posed optimization problem.

Original: Analysis by synthesis.
Revised: Analysis by synthesis.

Original: Since we do not have access to any form of 3D supervision, we use analysisby-synthesis to drive the optimization.
Revised: Since we do not have access to any form of 3D supervision, we use analysisby-synthesis to drive the optimization.

Original: That is, we use differentiable rendering to render the optimized 3D articulated shape and define loss functions w.r.t.
Revised: That is, we use differentiable rendering to render the optimized 3D articulated shape and define loss functions w.r.t.

Original: input images.
Revised: input images.

Original: Using Eq. 2, we first obtain the complete articulated shape for each image as a combination of different part meshes.
Revised: Using Eq. 2, we first obtain the complete articulated shape for each image as a combination of different part meshes.

Original: We then use a differentiable mesh renderer [25] to project the 3D points onto 2D image space via the learnable camera viewpoints.
Revised: We then use a differentiable mesh renderer [25] to project the 3D points onto 2D image space via the learnable camera viewpoints.

Original: A key challenge is that we do not have access to any from of 2D annotations such as keypoints or part segmentation.
Revised: A key challenge is that we do not have access to any from of 2D annotations such as keypoints or part segmentation.

Original: Self-supervisory deep features to the rescue.
Revised: Self-supervisory deep features to the rescue.

Original: Since different instances in the ensemble can have different texture, we do not model and render image pixel values.
Revised: Since different instances in the ensemble can have different texture, we do not model and render image pixel values.

Original: Instead, we rely on learned deep features for our self-supervisory analysis-by-synthesis framework.
Revised: Instead, we rely on learned deep features for our self-supervisory analysis-by-synthesis framework.

Original: Recent works [1, 12] demonstrate that deep features from DINO [6] vision transformer are semantically descriptive, robust to appearance variations, and higher-resolution compared to CNN features [1, 38].
Revised: Recent works [1, 12] demonstrate that deep features from DINO [6] vision transformer are semantically descriptive, robust to appearance variations, and higher-resolution compared to CNN features [1, 38].

Original: In this work, we use the key features from the last layer of the DINO network as semantic visual features for each image.
Revised: In this work, we use the key features from the last layer of the DINO network as semantic visual features for each image.

Original: We assume that these DINO features are relatively similar for same parts across different instances.
Revised: We assume that these DINO features are relatively similar for same parts across different instances.

Original: In addition to using DINO features for consistent semantic features across images, we also compute 2D parts and foreground animal mask following similar steps as in [1].
Revised: In addition to using DINO features for consistent semantic features across images, we also compute 2D parts and foreground animal mask following similar steps as in [1].

Original: Concretely, we take the class tokens from the last layer, compute the mean attention of class tokens as saliency maps, and sample the keys of image patches with high saliency scores.
Revised: Concretely, we take the class tokens from the last layer, compute the mean attention of class tokens as saliency maps, and sample the keys of image patches with high saliency scores.

Original: We then collect the salient keys from all images and apply an off-the-shelf K-means clustering algorithm to form c semantic part clusters.
Revised: We then collect the salient keys from all images and apply an off-the-shelf K-means clustering algorithm to form c semantic part clusters.

Original: Finally, we obtain foreground silhouettes by simply thresholding the pixels with minimum distance to cluster centroids.
Revised: Finally, we obtain foreground silhouettes by simply thresholding the pixels with minimum distance to cluster centroids.

Original: Fig.
Revised: Fig.

Original: 3 shows the DINO part clusters on sample images from different image ensembles.
Revised: 3 shows the DINO part clusters on sample images from different image ensembles.

Original: We observe that using 4 clusters works well in practice.
Revised: We observe that using 4 clusters works well in practice.

Original: With the use of self-supervised features (keys) and rough silhouettes, we alleviate the dependency ofground-truth segmentation mask and keypoint annotations in our optimization framework.
Revised: With the use of self-supervised features (keys) and rough silhouettes, we alleviate the dependency of mask and keypoint annotations in our optimization framework.

Original: Silhouette Loss.
Revised: Silhouette loss.

Original: We compute the silhouette loss L mask by rendering all part surfaces with a differentiable renderer [25] and comparing with the pseudo ground-truth foreground masks obtained via clustering DINO features: L mask = (cid:80) j
Revised: We compute the silhouette loss L mask by rendering all part surfaces with a differentiable renderer [25] and comparing with the pseudo ground-truth foreground masks obtained via clustering DINO features: L mask = (cid:80) j

Original: ∥ M j − ˆ M j ∥ 2 , where M j and ˆ M j are the rendered and pseudo ground-truth silhouettes of instance j , respectively.
Revised: ∥ M j − ˆ M j ∥ 2 , where M j and ˆ M j are the rendered and pseudo ground-truth silhouettes of instance j , respectively.

Original: Although these silhouette-based losses encourage the overall shape to match the 2D silhouettes, their supervisory signal is often too coarse to resolve the ambiguity caused by different camera viewpoints and pose articulations.
Revised: Although these silhouette-based losses encourage the overall shape to match the 2D silhouettes, their supervisory signal is often too coarse to resolve the ambiguity caused by different camera viewpoints and pose articulations.

Original: Therefore, we design a novel semantic consistency loss to more densely supervise the reconstruction.
Revised: Therefore, we design a novel semantic consistency loss to more densely supervise the reconstruction.

Original: 2D-3D Semantic Consistency.
Revised: 2D-3D semantic consistency.

Original: We propose a novel strategy to make use of the semantically consistent 2D DINO features for 3D part discovery.
Revised: We propose a novel strategy to make use of the semantically consistent 2D DINO features for 3D part discovery.

Original: At a high-level, we impose the 2D feature consistency across the images via learned 3D parts.
Revised: At a high-level, we impose the 2D feature consistency across the images via learned 3D parts.

Original: For this, we propose an Expectation-Maximization (EM) like optimization strategy where we alternatively estimate the 3D part features (E-step) from image ensemble features and then use these estimated 3D part features to update the parts (M-step).
Revised: For this, we propose an Expectation-Maximization (EM) like optimization strategy where we alternatively estimate the 3D part features (E-step) from image ensemble features and then use these estimated 3D part features to update the parts (M-step).

Original: More formally, let K j ∈ R h
Revised: More formally, let K j ∈ R h

Original: ×
Revised: ×

Original: w × f denote the f -dimensional DINO key features from j th image and
Revised: w × f denote the f -dimensional DINO key features from j th image and

Original: Q ∈ R m × f denote the corresponding features on the 3D shape with total m vertices.
Revised: Q ∈ R m × f denote the corresponding features on the 3D shape with total m vertices.

Original: In the Estep, we update Q by projecting the 3D coordinates onto all the 2D image spaces via differentiable rendering under the current estimated shape and camera viewpoints.
Revised: In the Estep, we update Q by projecting the 3D coordinates onto all the 2D image spaces via differentiable rendering under the current estimated shape and camera viewpoints.

Original: We aggregate the 2D image features from the image ensemble corresponding to each 3D vertex to estimate Q .
Revised: We aggregate the 2D image features from the image ensemble corresponding to each 3D vertex to estimate Q .

Original: In the M-step, we optimize the 3D parts and camera viewpoints using a 2D-3D semantic consistency loss L sem .
Revised: In the M-step, we optimize the 3D parts and camera viewpoints using a 2D-3D semantic consistency loss L sem .

Original: To define L sem , we discretize the foreground 2D coordinates in each image { p | ˆ M j ( p ) = 1 } .
Revised: To define L sem , we discretize the foreground 2D coordinates in each image { p | ˆ M j ( p ) = 1 } .

Original: Likewise, we sample a set of points { v ∈ V j } on the 3D part surfaces and obtain their 2D projected coordinates π j ( v ) .
Revised: Likewise, we sample a set of points { v ∈ V j } on the 3D part surfaces and obtain their 2D projected coordinates π j ( v ) .

Original: L sem is then defined as the Chamfer distance in a high-dimensional space:
Revised: L sem is then defined as the Chamfer distance in a high-dimensional space:

Original: [Equation]
Revised: [Equation]

Original: For each instance j , the distance D between an image point p and 3D surface point v is defined as:
Revised: For each instance j , the distance D between an image point p and 3D surface point v is defined as:

Original: [Equation]
Revised: [Equation]

Original: where α is a scalar weighting for semantic distance.
Revised: where α is a scalar weighting for semantic distance.

Original: In effect, L sem optimizes the 3D part shapes such that the aggregated 3D point features in the E-step would project closer to the similar pixel features in the image ensemble.
Revised: In effect, L sem optimizes the 3D part shapes such that the aggregated 3D point features in the E-step would project closer to the similar pixel features in the image ensemble.

Original: We alternate between E and M steps every other iteration in the optimization process.
Revised: We alternate between E and M steps every other iteration in the optimization process.

Original: In Eq.
Revised: In Eq.

Original: 4, the image coordinates p and semantic features K, Q are fixed in each optimization iteration, hence L sem can effectively push the surface points towards their corresponding 2D coordinates by minimizing the overall distance.
Revised: 4, the image coordinates p and semantic features K, Q are fixed in each optimization iteration, hence L sem can effectively push the surface points towards their corresponding 2D coordinates by minimizing the overall distance.

Original: Since this EM-style optimization is prone to local-minima as we jointly optimize camera viewpoints, shapes and pose articulations, we find that a good initialization of 3D features is important.
Revised: Since this EM-style optimization is prone to local-minima as we jointly optimize camera viewpoints, shapes and pose articulations, we find that a good initialization of 3D features is important.

Original: For this, we manually map each part index i to one of the DINO clusters (4 in our experiments) and then initialize the 3D features Q by assigning the average 2D cluster features to the corresponding 3D part vertices.
Revised: For this, we manually map each part index i to one of the DINO clusters (4 in our experiments) and then initialize the 3D features Q by assigning the average 2D cluster features to the corresponding 3D part vertices.

Original: This manual 3D part to 2D cluster assignment takes minimal effort since it is category-level (all instances share the same 3D parts).
Revised: This manual 3D part to 2D cluster assignment takes minimal effort since it is category-level (all instances share the same 3D parts).

Original: Alternatively, one could roughly initialize the camera viewpoints like in recent works such as
Revised: Alternatively, one could roughly initialize the camera viewpoints like in recent works such as

Original: NeRS [45] which allows the computation of initial Q features from 2D features.
Revised: NeRS [45] which allows the computation of initial Q features from 2D features.

Original: Nonetheless, 2D-3D assignment is easier and faster compared to camera viewpoint initialization as 2D-3D assignment only needs to be done once per image ensemble whereas camera initialization is required for each instance in the ensemble.
Revised: Nonetheless, 2D-3D assignment is easier and faster compared to camera viewpoint initialization as 2D-3D assignment only needs to be done once per image ensemble whereas camera initialization is required for each instance in the ensemble.

Original: To make the optimization process fully automatic, we also propose to use simple heuristics for the 2D-3D part assignment using intuitions like animal head are usually above the torso and legs at the bottom.
Revised: To make the optimization process fully automatic, we also propose to use simple heuristics for the 2D-3D part assignment using intuitions like animal head are usually above the torso and legs at the bottom.

Original: We refer to the LASSIE optimization with this heuristic-based 3D feature initialization as ‘LASSIE-heuristic’ and describe the details in the supplemental material.
Revised: We refer to the LASSIE optimization with this heuristic-based 3D feature initialization as ‘LASSIE-heuristic’ and describe the details in the supplemental material.

Original: Comparison to related methods.
Revised: Comparison to related methods.

Original: Our use of 2D-3D semantic consistency is closely related to prior works, UMR [24] and BANMo [42].
Revised: Our use of 2D-3D semantic consistency is closely related to prior works, UMR [24] and BANMo [42].

Original: At a high-level, LASSIE and UMR both start from 2D part segmentations and then lift them onto 3D. A key difference is that LASSIE directly discovers parts in 3D using 2D feature consistency, whereas UMR uses 2D part consistency via canonical UV maps.
Revised: At a high-level, LASSIE and UMR both start from 2D part segmentations and then lift them onto 3D. A key difference is that LASSIE directly discovers parts in 3D using 2D feature consistency, whereas UMR uses 2D part consistency via canonical UV maps.

Original: More specifically, UMR uses 2D parts (not 2D features) and 2D UV maps as common canonical part representations.
Revised: More specifically, UMR uses 2D parts (not 2D features) and 2D UV maps as common canonical part representations.

Original: Instead, LASSIE directly optimizes 3D part features with feature based loss (not 2D parts) and LASSIE does not use UV maps like in UMR.
Revised: Instead, LASSIE directly optimizes 3D part features with feature based loss (not 2D parts) and LASSIE does not use UV maps like in UMR.

Original: We only use 2D parts for feature initialization and not during optimization.
Revised: We only use 2D parts for feature initialization and not during optimization.

Original: That is, we discover parts directly in 3D with skeleton constraints, without much reliance on intermediate 2D part discovery.
Revised: That is, we discover parts directly in 3D with skeleton constraints, without much reliance on intermediate 2D part discovery.

Original: Hence, LASSIE is less sensitive to the issues in 2D part segmentations compared to UMR.
Revised: Hence, LASSIE is less sensitive to the issues in 2D part segmentations compared to UMR.

Original: BANMo, on the other hand, learns a canonical feature embedding by enforcing the consistency between feature matching and geometric warping, which is made tractable by the dense temporal correspondence (optical flow) between video frames.
Revised: BANMo, on the other hand, learns a canonical feature embedding by enforcing the consistency between feature matching and geometric warping, which is made tractable by the dense temporal correspondence (optical flow) between video frames.

Original: The semantic loss in BanMo enforces the 2D-3D cycle consistency between 2D coordinates and canonical 3D points.
Revised: The semantic loss in BanMo enforces the 2D-3D cycle consistency between 2D coordinates and canonical 3D points.

Original: Considering that our image ensemble is sparse and un-correlated, we coarsely initialize and regularize the 3D surface features at the part-level.
Revised: Considering that our image ensemble is sparse and un-correlated, we coarsely initialize and regularize the 3D surface features at the part-level.

Original: The proposed semantic loss allows us to first localize the 3D parts then refine the detailed part shapes in our challenging setting.
Revised: The proposed semantic loss allows us to first localize the 3D parts then refine the detailed part shapes in our challenging setting.

Original: Regularization.
Revised: Regularization.

Original: Our skeleton-based representation allows us to conveniently impose pose prior or regularization.
Revised: Our skeleton-based representation allows us to conveniently impose pose prior or regularization.

Original: Specifically, we calculate a pose prior loss L pose to minimize the bone rotation deviations from the resting pose as: L pose =
Revised: Specifically, we calculate a pose prior loss L pose to minimize the bone rotation deviations from the resting pose as: L pose =

Original: (cid:80)
Revised: (cid:80)

Original: j ∥
Revised: j ∥

Original: R j − ¯ R ∥ 2 , where R j is the bone rotations of instance j and ¯ R denotes the bone rotations of shared resting pose.
Revised: R j − ¯ R ∥ 2 , where R j is the bone rotations of instance j and ¯ R denotes the bone rotations of shared resting pose.

Original: We also impose a regularization
Revised: We also impose a regularization

Original: L ang on joint angles on the animal legs to limit their sideway rotations:
Revised: L ang on joint angles on the animal legs to limit their sideway rotations:

Original: [Equation]
Revised: [Equation]

Original: where R i,y and R i,z are the rotations of bone i with respect to y and z axes respectively.
Revised: where R i,y and R i,z are the rotations of bone i with respect to y and z axes respectively.

Original: Since the skeleton faces the + z direction in the canonical space ( + x right, + y up, + z out), minimizing the y and z -axis rotations essentially constrains the sideway movements of the bones.
Revised: Since the skeleton faces the + z direction in the canonical space ( + x right, + y up, + z out), minimizing the y and z -axis rotations essentially constrains the sideway movements of the bones.

Original: Note that both L pose and L ang are generic to all quadrupeds in our experiments, which we find crucial to avoid unrealistic poses.
Revised: Note that both L pose and L ang are generic to all quadrupeds in our experiments, which we find crucial to avoid unrealistic poses.

Original: Finally, we apply common mesh regularization terms to encourage smooth surfaces, including the Laplacian loss L lap and surface normal loss L norm .
Revised: Finally, we apply common mesh regularization terms to encourage smooth surfaces, including the Laplacian loss L lap and surface normal loss L norm .

Original: L lap regularizes 3D surfaces by pushing each vertex towards the center of its neighbors, and L norm encourages neighboring mesh faces to have similar normal vectors.
Revised: L lap regularizes 3D surfaces by pushing each vertex towards the center of its neighbors, and L norm encourages neighboring mesh faces to have similar normal vectors.

Original: Note that we apply regularization losses to each part surface individually since the part shapes should be primitive-like and usually do not have sharp or pointy surfaces.
Revised: Note that we apply regularization losses to each part surface individually since the part shapes should be primitive-like and usually do not have sharp or pointy surfaces.

Original: Optimization and textured outputs.
Revised: Optimization and textured outputs.

Original: The overall optimization objective is:
Revised: The overall optimization objective is:

Original: [Equation]
Revised: [Equation]

Original: where { λ i } are weighting hyper-parameters.
Revised: where { λ i } are weighting hyper-parameters.

Original: We first pre-train the part prior MLP F p with 3D geometric primitives and freeze it during the optimization on a given image ensemble.
Revised: We first pre-train the part prior MLP F p with 3D geometric primitives and freeze it during the optimization on a given image ensemble.

Original: For each image ensemble of an animal species, we perform multi-stage optimization on the camera, pose, and shape parameters until convergence.
Revised: For each image ensemble of an animal species, we perform multi-stage optimization on the camera, pose, and shape parameters until convergence.

Original: That is, we update the camera viewpoints and fix the rest first, then optimize the bone transformations, and finally the latent part codes as well as part deformation
Revised: That is, we update the camera viewpoints and fix the rest first, then optimize the bone transformations, and finally the latent part codes as well as part deformation

Original: MLPs.
Revised: MLPs.

Original: In each iteration, we first update the semantic features of 3D surfaces, then use the updated features to update 3D surfaces, forming an EM-style optimization.
Revised: In each iteration, we first update the semantic features of 3D surfaces, then use the updated features to update 3D surfaces, forming an EM-style optimization.

Original: We implement the framework in
Revised: We implement the framework in

Original: PyTorch [33] and update all the learnable parameters using an Adam optimizer [16].
Revised: PyTorch [33] and update all the learnable parameters using an Adam optimizer [16].

Original: To obtain the final textured outputs, we densely sample vertices from the optimized neural surfaces and directly sample the colors of visible vertices from individual images via 2D projection, where the visibility information is obtained from naive rasterization.
Revised: To obtain the final textured outputs, we densely sample vertices from the optimized neural surfaces and directly sample the colors of visible vertices from individual images via 2D projection, where the visibility information is obtained from naive rasterization.

Original: For the invisible (self-occluded) vertices, we assign the color by finding their left-right symmetries or nearest visible neighbors.
Revised: For the invisible (self-occluded) vertices, we assign the color by finding their left-right symmetries or nearest visible neighbors.

Original: More implementation details can be found in the supplemental material.
Revised: More implementation details can be found in the supplemental material.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: Considering the novel problem of sparse-image optimization, we select a subset of the
Revised: Considering the novel problem of sparse-image optimization, we select a subset of the

Original: Pascal-Part dataset [7] and collect web images of additional animal classes to evaluate LASSIE and related prior arts.
Revised: Pascal-Part dataset [7] and collect web images of additional animal classes to evaluate LASSIE and related prior arts.

Original: The Pascal-Part dataset contains diverse images of several animal classes which are annotated with 2D part segmentation masks.
Revised: The Pascal-Part dataset contains diverse images of several animal classes which are annotated with 2D part segmentation masks.

Original: We automatically select images of horse, cow, or sheep with only one object covering 1-50% area of the entire image, and find the keypoints by calculating the centers/corners of ground-truth part masks.
Revised: We automatically select images of horse, cow, or sheep with only one object covering 1-50% area of the entire image, and find the keypoints by calculating the centers/corners of ground-truth part masks.

Original: In addition, for analysis on more diverse animal categories, we collect sparse image ensembles (with CC-licensed images) of some other animals from the internet and manually annotate the 2D keypoints for evaluation.
Revised: In addition, for analysis on more diverse animal categories, we collect sparse image ensembles (with CC-licensed images) of some other animals from the internet and manually annotate the 2D keypoints for evaluation.

Original: The classes include quadrupeds like zebra, tiger, giraffe, elephant, as well as bipeds like kangaroo and penguin.
Revised: The classes include quadrupeds like zebra, tiger, giraffe, elephant, as well as bipeds like kangaroo and penguin.

Original: We filter out the images where the animal body is heavily occluded or truncated, resulting in roughly 30 images per category.
Revised: We filter out the images where the animal body is heavily occluded or truncated, resulting in roughly 30 images per category.

Original: The LASSIE optimization and evaluations are performed on each image ensemble separately.
Revised: The LASSIE optimization and evaluations are performed on each image ensemble separately.

Original: The self-collected data does not contain personally identifiable information or offensive content, and will be released to public.
Revised: The self-collected data does not contain personally identifiable information or offensive content, and will be released to public.

Original: Baselines.
Revised: Baselines.

Original: Due to the lack of prior works on our problem setting (sparse image optimization for articulated animal shapes), we mainly compare LASSIE with learning-based mesh reconstruction methods.
Revised: Due to the lack of prior works on our problem setting (sparse image optimization for articulated animal shapes), we mainly compare LASSIE with learning-based mesh reconstruction methods.

Original: Among these methods, we find 3D Safari [47] and A-CSM [20] to be most comparable to LASSIE since they also model articulation for animal classes of our interest.
Revised: Among these methods, we find 3D Safari [47] and A-CSM [20] to be most comparable to LASSIE since they also model articulation for animal classes of our interest.

Original: Other recent mesh reconstruction methods, on the other hand, either cannot handle articulations [21, 24, 11, 36, 44] or assume different inputs [23, 40, 42].
Revised: Other recent mesh reconstruction methods, on the other hand, either cannot handle articulations [21, 24, 11, 36, 44] or assume different inputs [23, 40, 42].

Original: Note that both 3D Safari and A-CSM are trained on large-scale image sets while LASSIE optimizes on a sparse image ensemble.
Revised: Note that both 3D Safari and A-CSM are trained on large-scale image sets while LASSIE optimizes on a sparse image ensemble.

Original: To evaluate their methods on our dataset, we use the released models trained on the closest animal classes.
Revised: To evaluate their methods on our dataset, we use the released models trained on the closest animal classes.

Original: For instance, the zebra model of 3D Safari and the horse model of A-CSM are used to evaluate on similar quadrupeds.
Revised: For instance, the zebra model of 3D Safari and the horse model of A-CSM are used to evaluate on similar quadrupeds.

Original: Visual Comparisons.
Revised: Visual comparisons.

Original: Fig.
Revised: Fig.

Original: 3 shows the qualitative results of LASSIE against 3D Safari and A-CSM on the self-collected animal images.
Revised: 3 shows the qualitative results of LASSIE against 3D Safari and A-CSM on the self-collected animal images.

Original: The 3D Safari [47] model is trained on zebra images and does
Revised: The 3D Safari [47] model is trained on zebra images and does

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: not generalize well to other animal classes.
Revised: not generalize well to other animal classes.

Original: A-CSM [20] assumes high-quality skinned model of an object category which enables detailed shapes of animal bodies.
Revised: A-CSM [20] assumes high-quality skinned model of an object category which enables detailed shapes of animal bodies.

Original: However, its outputs do not align well with the given 2D images.
Revised: However, its outputs do not align well with the given 2D images.

Original: Our results demonstrate that LASSIE can effectively learn from sparse image ensemble to discover 3D articulated parts that are high-quality, faithful to input images, and semantically consistent across instances.
Revised: Our results demonstrate that LASSIE can effectively learn from sparse image ensemble to discover 3D articulated parts that are high-quality, faithful to input images, and semantically consistent across instances.

Original: Note that our 3D reconstructions also improve the 2D part segmentation from DINO-ViT feature clustering [1].
Revised: Note that our 3D reconstructions also improve the 2D part segmentation from DINO-ViT feature clustering [1].

Original: Keypoint transfer metrics.
Revised: Keypoint transfer metrics.

Original: As there is no ground-truth 3D annotation in our datasets, we quantitatively evaluate using 2D keypoint transfer between each pair of images which is a standard practice [47, 20].
Revised: As there is no ground-truth 3D annotation in our datasets, we quantitatively evaluate using 2D keypoint transfer between each pair of images which is a standard practice [47, 20].

Original: We map a given set of 2D keypoints on a source image onto the 3D part surfaces, and then project them to a target image using the optimized camera viewpoints, shapes and pose articulations.
Revised: We map a given set of 2D keypoints on a source image onto the 3D part surfaces, and then project them to a target image using the optimized camera viewpoints, shapes and pose articulations.

Original: Since this keypoint transfer goes through the 2D-to-3D and 3D-to-2D mappings, a successful keypoint transfer requires accurate 3D reconstruction on both the source and target images.
Revised: Since this keypoint transfer goes through the 2D-to-3D and 3D-to-2D mappings, a successful keypoint transfer requires accurate 3D reconstruction on both the source and target images.

Original: We calculate the percentage of correct keypoints (PCK) under two different thresholds: 0.1 × max ( h, w ) and 0.05 × max ( h, w ) , where h and w are image height and width, respectively.
Revised: We calculate the percentage of correct keypoints (PCK) under two different thresholds: 0.1 × max ( h, w ) and 0.05 × max ( h, w ) , where h and w are image height and width, respectively.

Original: Table 1 shows that LASSIE achieves higher PCK on the Pascal-part [7] images and most other classes in our in-the-wild image collections.
Revised: Table 1 shows that LASSIE achieves higher PCK on the Pascal-part [7] images and most other classes in our in-the-wild image collections.

Original: We also show ablative results of LASSIE without using the semantic feature consistency loss L sem or part prior MLP F p .
Revised: We also show ablative results of LASSIE without using the semantic feature consistency loss L sem or part prior MLP F p .

Original: Note that L sem is essential to fit the skeleton and part surfaces faithfully to the input images, and F p provides effective regularization on part surfaces to generate realistic shapes from various viewpoints.
Revised: Note that L sem is essential to fit the skeleton and part surfaces faithfully to the input images, and F p provides effective regularization on part surfaces to generate realistic shapes from various viewpoints.

Original: LASSIE-heuristic achieves slightly lower PCK
Revised: LASSIE-heuristic, by using

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original:  but still performs favorably against prior arts overall.
Revised: automatic 3D feature initialization, achieves slightly lower PCK but still performs favorably against prior arts overall.

Original: Example visual results of keypoint transfer are shown in Fig.
Revised: Example visual results of keypoint transfer are shown in Fig.

Original: [Table]

Original: 2D overall/part IOU metrics.
Revised: 2D overall/part IOU metrics.

Original: Our part-based representation can generate realistic 3D shapes and can even improve 2D part discovery compared to prior arts.
Revised: Our part-based representation can generate realistic 3D shapes and can even improve 2D part discovery compared to prior arts.

Original: In Table 2 and Fig. 4, we show the qualitative and quantitative comparisons against prior 3D works as well as 2D co-part discovery methods of SCOPS [13] and DINO clustering [1].
Revised: In Table 2 and Fig. 4, we show the qualitative and quantitative comparisons against prior 3D works as well as 2D co-part discovery methods of SCOPS [13] and DINO clustering [1].

Original: For SCOPS [13] and DINO clustering [1], we set the number of parts N p = 4 (which we find to be optimal for these techniques) and manually assign each discovered part to the best matched part in the Pascal-part annotations.
Revised: For SCOPS [13] and DINO clustering [1], we set the number of parts N p = 4 (which we find to be optimal for these techniques) and manually assign each discovered part to the best matched part in the Pascal-part annotations.

Original: Note that higher part IOU against Pascal-part annotations may not strictly indicate better part segmentation since the self-discovered parts need not correspond to human annotations.
Revised: Note that higher part IOU against Pascal-part annotations may not strictly indicate better part segmentation since the self-discovered parts need not correspond to human annotations.

Original: Compared to prior arts on articulated shape reconstruction, LASSIE results match the Pascal-part segmentation masks better and achieves higher overall IOU.
Revised: Compared to prior arts on articulated shape reconstruction, LASSIE results match the Pascal-part segmentation masks better and achieves higher overall IOU.

Original: On co-part segmentation, our 3D reconstruction captures the semantic parts in 2D while being more geometrically refined than DINO clustering.
Revised: On co-part segmentation, our 3D reconstruction captures the semantic parts in 2D while being more geometrically refined than DINO clustering.

Original: Moreover, our results can separate parts with similar semantic features, e.g. , 4 animal legs, and thus resulting in much higher part IOU.
Revised: Moreover, our results can separate parts with similar semantic features, e.g. , 4 animal legs, and thus resulting in much higher part IOU.

Original: Part transfer.
Revised: Part transfer.

Original: While keypoint transfer and part IOU are commonly used in the literature, we observe that they are either sparse, biased towards certain body parts ( e.g. , animal faces), or not directly comparable due to part mismatch.
Revised: While keypoint transfer and part IOU are commonly used in the literature, we observe that they are either sparse, biased towards certain body parts ( e.g. , animal faces), or not directly comparable due to part mismatch.

Original: To address this issue, we propose a part transfer metric for better evaluation of 3D reconstruction consistency across image ensemble.
Revised: To address this issue, we propose a part transfer metric for better evaluation of 3D reconstruction consistency across image ensemble.

Original: This part transfer metric is similar to keypoint transfer but uses part segmentations.
Revised: This part transfer metric is similar to keypoint transfer but uses part segmentations.

Original: That is, we densely transfer the part
Revised: That is, we densely transfer the part

Original: [Figure]
Revised: [Figure]

Original: segmentation in a source image to a target image through forward and backward mapping between the 2D pixels and the canonical 3D surfaces.
Revised: segmentation in a source image to a target image through forward and backward mapping between the 2D pixels and the canonical 3D surfaces.

Original: A pixel is considered transferred correctly if and only if it is mapped to the same 2D part in the target image as where it belongs in the source image.
Revised: A pixel is considered transferred correctly if and only if it is mapped to the same 2D part in the target image as where it belongs in the source image.

Original: Similar to PCK in keypoint transfer evaluation, we calculate the percentage of correct pixels (PCP) for each source-target pair.
Revised: Similar to PCK in keypoint transfer evaluation, we calculate the percentage of correct pixels (PCP) for each source-target pair.

Original: Since the part segmentation only covers the visible surfaces and not the occluded regions, we ignore the pixels mapped to a self-occluded surface when calculating the PCP metric.
Revised: Since the part segmentation only covers the visible surfaces and not the occluded regions, we ignore the pixels mapped to a self-occluded surface when calculating the PCP metric.

Original: The quantitative results are shown in Table 2, which demonstrate the favorable performance of LASSIE against prior arts.
Revised: The quantitative results are shown in Table 2, which demonstrate the favorable performance of LASSIE against prior arts.

Original: Applications.
Revised: Applications.

Original: Our 3D neural part representation not only produces more faithful and realistic shapes but also enables various applications like part manipulation, texture transfer, motion re-targeting, etc.
Revised: Our 3D neural part representation not only produces more faithful and realistic shapes but also enables various applications like part manipulation, texture transfer, motion re-targeting, etc.

Original: Specifically, we can generate a new shape by swapping or interpolating certain parts, transfer the sampled surface texture from one object class to another using the same base skeleton, or specify the bone rotations to produce desirable re-posed shape or animation.
Revised: Specifically, we can generate a new shape by swapping or interpolating certain parts, transfer the sampled surface texture from one object class to another using the same base skeleton, or specify the bone rotations to produce desirable re-posed shape or animation.

Original: We show some example results of pose transfer and texture transfer between different animal classes in Fig.
Revised: We show some example results of pose transfer and texture transfer between different animal classes in Fig.

Original: Limitations.
Revised: Limitations.

Original: LASSIE relies heavily on the 3D skeleton and strong pose and shape regularization.
Revised: LASSIE relies heavily on the 3D skeleton and strong pose and shape regularization.

Original: As the first attempt to address this novel and highly ill-posed problem, we find the proposed pose and shape constraints essential to produce good quality results and avoid unrealistic outputs.
Revised: As the first attempt to address this novel and highly ill-posed problem, we find the proposed pose and shape constraints essential to produce good quality results and avoid unrealistic outputs.

Original: Moreover,
Revised: Moreover,

Original: LASSIE can not handle image collections with heavy truncation, occlusions, or extreme pose variations since we leverage self-supervisory DINO-ViT features on sparse image ensemble as our main supervision.
Revised: LASSIE can not handle image collections with heavy truncation, occlusions, or extreme pose variations since we leverage self-supervisory DINO-ViT features on sparse image ensemble as our main supervision.

Original: For instance, partial-body or noisy DINO features can cause ambiguities in camera pose and part localization.
Revised: For instance, partial-body or noisy DINO features can cause ambiguities in camera pose and part localization.

Original: Finally, we observe that LASSIE currently struggles with a) highly articulated parts like elephant trucks and b) fluffy animals that appear with more instance variations and ambiguous articulations.
Revised: Finally, we observe that LASSIE currently struggles with a) highly articulated parts like elephant trucks and b) fluffy animals that appear with more instance variations and ambiguous articulations.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we study a novel and practical problem of articulated shape optimization of animals from sparse image collections in-the-wild.
Revised: In this paper, we study a novel and practical problem of articulated shape optimization of animals from sparse image collections in-the-wild.

Original: Instead of relying on pre-defined shape models or human annotations like keypoints or segmentation masks, we propose a neural part representation based on a generic 3D skeleton, which is robust to appearance variations and generalizes well across different animal classes.
Revised: Instead of relying on pre-defined shape models or human annotations like keypoints or segmentation masks, we propose a neural part representation based on a generic 3D skeleton, which is robust to appearance variations and generalizes well across different animal classes.

Original: In LASSIE, our key insight is to reason about animal parts instead of whole shape as parts allow imposing several constraints.
Revised: In LASSIE, our key insight is to reason about animal parts instead of whole shape as parts allow imposing several constraints.

Original: We leverage self-supervisory dense ViT features to provide both silhouette and semantic part consistency for supervision.
Revised: We leverage self-supervisory dense ViT features to provide both silhouette and semantic part consistency for supervision.

Original: Both quantitative and qualitative results on Pascal-Part and our own in-the-wild image ensembles show that LASSIE can effectively learn from few in-the-wild images and produce high-quality results.
Revised: Both quantitative and qualitative results on Pascal-Part and our own in-the-wild image ensembles show that LASSIE can effectively learn from few in-the-wild images and produce high-quality results.

Original: Our part-based representation also enables various applications such as pose interpolation, texture transfer, and animation.
Revised: Our part-based representation also enables various applications such as pose interpolation, texture transfer, and animation.

Original: We hope that this work will facilitate future research efforts on learning, inferring, and manipulating articulated shapes from image ensembles in the wild.
Revised: We hope that this work will facilitate future research efforts on learning, inferring, and manipulating articulated shapes from image ensembles in the wild.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] Since the scope of our paper is within 3D reconstruction of animal bodies, there is no potential negative social impact that we are aware of.
Revised: [No] Since the scope of our paper is within 3D reconstruction of animal bodies, there is no potential negative social impact that we are aware of.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] In our optimization framework, the random seed initialization does not affect the outputs much since the most sensitive parameters (e.g. camera, bone scaling, pose) are not randomly initialized.
Revised: [No] In our optimization framework, the random seed initialization does not affect the outputs much since the most sensitive parameters (e.g. camera, bone scaling, pose) are not randomly initialized.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] 5.
Revised: [Yes] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]