Original: LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery
Revised: LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Creating high-quality articulated 3D models of animals is challenging either viamanual creation or using 3D scanning tools.
Revised: Creating high-quality articulated 3D models of animals is challenging either via manual creation or using 3D scanning tools.

Original: Therefore, techniques to reconstructarticulated 3D objects from 2D images are crucial and highly useful.
Revised: Therefore, techniques to reconstruct articulated 3D objects from 2D images are crucial and highly useful.

Original: In this work,we propose a practical problem setting of estimating 3Dshape and pose  of animalsgiven only a few ( ≈ 30) in-the-wild images of a particular animal species (say,horse).
Revised: In this work, we propose a practical problem setting to estimate 3D pose and shape of animals given only a few (10-30) in-the-wild images of a particular animal species (say, horse).

Original: Contrary to existing works that rely on pre-defined template shapes, we donot assume any form of 2D or 3D ground-truth annotations, nor do we assume anymulti-view or temporal information.
Revised: Contrary to existing works that rely on pre-defined template shapes, we do not assume any form of 2D or 3D ground-truth annotations, nor do we leverage any multi-view or temporal information.

Original: Our input image ensemble can have animalinstances with varying poses, backgrounds, illuminations andalso textures.
Revised: Moreover, each input image ensemble can contain animal instances with varying poses, backgrounds, illuminations, and textures.

Original: Ourkey insight is that 3D parts have much more simplistic shapes compared to theoverall animal and that the part shapes are robust w.r.t.
Revised: Our key insight is that 3D parts have much simpler shape compared to the overall animal and that they are robust w.r.t.

Original: animal pose articulations.
Revised: animal pose articulations.

Original: Using these insights, We propose LASSIE, a novel optimization framework thatdiscovers 3D parts in a self-supervised manner using minimal user intervention.
Revised: Following these insights, we propose LASSIE, a novel optimization framework which discovers 3D parts in a self-supervised manner with minimal user intervention.

Original: A key driving force behind LASSIE is the enforcing of 2D-3D part consistencyusing self-supervisory deep features.
Revised: A key driving force behind LASSIE is the enforcing of 2D-3D part consistency using self-supervisory deep features.

Original: Experiments on Pascal Part and self-collectedin-the-wild animal datasets demonstrate considerably better 3D reconstructions aswell as both 2D and 3D part discovery compared to prior art.
Revised: Experiments on Pascal-Part and self-collected in-the-wild animal datasets demonstrate considerably better 3D reconstructions as well as both 2D and 3D part discovery compared to prior arts. Project page: https://chhankyao.github.io/lassie/

Original:  Introduction
Revised:  Introduction

Original: The last decade has witnessed significant advances in estimating human body pose and shape fromimages [21, 2, 12, 16, 15, 29, 14].
Revised: The last decade has witnessed significant advances in estimating human body pose and shape from images [25, 2, 14, 18, 17, 33, 16].

Original: Much progress is fueled by the availability of rich datasets [11, 24,33] with 3D human scans in a variety of shapes and poses.
Revised: Much progress is fueled by the availability of rich datasets [13, 28, 38] with 3D human scans in a variety of shapes and poses.

Original: In contrast, 3D scanning wild animals isquite challenging and existing animal datasets such as SMALR [40] rely on man-made animal shapes.
Revised: In contrast, 3D scanning wild animals is quite challenging and existing animal datasets such as SMALR [46] rely on man-made animal shapes.

Original: Manually creating realistic animal models for a wide diversity of animals (e g. mammals and birds)is alsoquite challenging and time-consuming.
Revised: Manually creating realistic animal models for a wide diversity of animals (e g. mammals and birds) is also challenging and time-consuming.

Original: In this work, we propose to automatically estimate3D shape and articulation (pose) of animals from only a sparse set of image ensemble (collection)without using any 2D or 3D ground-truth annotations.
Revised: In this work, we propose to automatically estimate 3D shape and articulation (pose) of animals from only a sparse set of image ensemble (collection) without using any 2D or 3D ground-truth annotations.

Original: Our problem setting is highly practical as eachimage ensemble consists of only a few (10-30) in-the-wild images of a specific animal species.
Revised: Our problem setting is highly practical as each image ensemble consists of only a few (10-30) in-the-wild images of a specific animal species.

Original: Fig. 1shows sample zebra images which forms our input.
Revised: Fig.shows sample zebra images which forms our input.

Original: In addition, we assume a human-specified 3Dskeleton (Fig. 1(left)) that can be quite rough with incorrect bone lengths and this mainly providesthe connectivity of animal parts.
Revised: In addition, we assume a human-specified 3D skeleton (left in Fig. 1) that can be quite rough with incorrect bone lengths and this mainly provides the connectivity of animal parts.

Original: For instance,  all the quadruped animals share a same 3D skeletondespite considerable shape variations across species (e.g., giraffe vs. zebra vs. elephant).
Revised: For instance, we use the same 3D skeleton for all the quadruped animals despite considerable shape variations across species, e.g., giraffe vs. zebra vs. elephant.

Original: Manuallyspecifying a rough skeleton is a trivial task which only takes a few minutes.
Revised: Manually specifying a rough skeleton is a trivial task which only takes a few minutes.

Original: Estimating articulated 3D shapes from in-the-wild image ensemble is a highly ambiguous andchallenging problem.
Revised: Estimating articulated 3D shapes from in-the-wild image ensemble is a highly ambiguous and challenging problem.

Original: There are several varying factors across images: backgrounds; lighting; cameraviewpoints; animal shape, pose and texture, etc.
Revised: There are several varying factors across images: backgrounds; lighting; camera

Original: In addition, our highly practical problem setting doesnot allow access to any 3D animal models, per-image annotations such as keypoints and silhouettes,or multi-view images.

Original: [Figure]
Revised: [Figure]

Original: The part-based representation also allows applications like texture and pose transfer, animation, etc. 
Revised: The part-based representation also allows applications like texture and pose transfer, animation, etc. viewpoints; animal shape, pose and texture, etc.

Revised: In addition, our highly practical problem setting does not allow access to any 3D animal models, per-image annotations such as keypoints and silhouettes, or multi-view images.

Original: In this work, we present LASSIE, an optimization technique to Learn Articulated Shapes from 
Revised: In this work, we present LASSIE, an optimization technique to L earn A rticulated S hapes from S parse

Original: Sparse Image Ensemble.
Revised: I mage E nsemble.

Original: Our key observation is that shapes are composed of 3D parts with thefollowing characteristics: 1. 3D parts are geometrically and semantically consistent across differentinstances.
Revised: Our key observation is that shapes are composed of 3D parts with the following characteristics: 1) 3D parts are geometrically and semantically consistent across different instances.

Original:  Shapes of 3D parts remain relatively constant w.r.t.
Revised: 2) Shapes of 3D parts remain relatively constant w.r.t.

Original: changes in animal poses.
Revised: changes in animal poses.

Original: Despitecomplex overall animal shape, 3D parts are usually made of simple(mostly) convex geometricshapes.
Revised: Despite the complex shape of overall animal body, 3D parts are usually made of simple convex shapes.

Original: Following these observations, LASSIE optimizes 3D parts and their articulation insteadof directly optimizing the overall animal shape.
Revised: Following these observations, LASSIE optimizes 3D parts and their articulation instead of directly optimizing the overall animal shape.

Original: Specifically, LASSIE optimizes a 3D skeletonand part shapes that are shared across instances; while also optimizing instance-specific cameraviewpoint, pose articulation and surface texture.
Revised: Specifically, LASSIE optimizes a 3D skeleton and part shapes that are shared across instances; while also optimizing instance-specific camera viewpoint, pose articulation, and surface texture.

Original: To enforce semantic part consistency across differentimages, we leverage deep features from a vision transformer [8] (DINO-ViT [5]) that is trainedin a self-supervised fashion.
Revised: To enforce semantic part consistency across different images, we leverage deep features from a vision transformer [9] (DINO-ViT [5]) trained in a self-supervised fashion.

Original: Recent works [1] demonstrate good 2D feature correspondences in DINO-ViT features and we make use of these correspondences for 3D part discovery.
Revised: Recent methods [1, 37] demonstrate good 2D feature correspondences in DINO-ViT features and we make use of these correspondences for 3D part discovery.

Original: To regularizepart shapes, we constrain the optimized parts to a latent shape manifold that is learned using 3Dgeometric primitives (spheres, cylinders, cones etc.).
Revised: To regularize part shapes, we constrain the optimized parts to a latent shape manifold that is learned using 3D geometric primitives (spheres, cylinders, cones etc.).

Original: Being completely self-supervised, LASSIE caneasily generalize to a wide range of animal species with minimal user intervention.
Revised: Being self-supervised at the image level, LASSIE can easily generalize to a wide range of animal species with minimal user intervention.

Original: We conduct extensive experiments on image ensembles from Pascal-part dataset [6] with quantitativeevaluations using 2D ground-truth part segmentations and keypoints.
Revised: We conduct extensive experiments on image ensembles from the Pascal-Part Dataset [6] with quantitative evaluations using 2D ground-truth part segmentation and keypoints.

Original: We also collect in-the-wildweb image ensembles with more animal species and annotate 2D keypoints for evaluations.
Revised: We also collect in-the-wild web image ensembles with more animal species and annotate 2D keypoints for evaluation.

Original: Bothqualitative and quantitative results demonstrate considerably better 3D reconstructions compared tothe prior art that use even more supervision.
Revised: Both qualitative and quantitative results demonstrate considerably better 3D reconstructions compared to the prior arts that use even more supervision.

Original: In addition, estimation of explicit 3D parts enables easyediting and manipulation of animal parts.
Revised: In addition, estimation of explicit 3D parts enables easy editing and manipulation of animal parts.

Original: Fig.
Revised: Fig.

Original: 1 shows sample 3D reconstructions with discoveredparts from LASSIE.
Revised: 1 shows sample 3D reconstructions with discovered parts from LASSIE.

Original: The main contributions of this work are:
Revised: The main contributions of this work are:

Original: • To the best of our knowledge, this is the first work that recovers 3D articulated shapes from inthe-wild image ensembles without using any pre-defined animal shape models or instance-specificannotations like keypoints or silhouettes.
Revised: • To the best of our knowledge, this is the first work that recovers 3D articulated shapes from in-the-wild image ensembles without using any pre-defined animal shape models or instance-specific annotations like keypoints or silhouettes.

Original:  We propose a novel optimization framework called LASSIE, where we incorporate several usefulproperties of mid-level 3D part representation.
Revised: • We propose a novel optimization framework called LASSIE, where we incorporate several useful properties of mid-level 3D part representation.

Original: LASSIE is category-agnostic and can easily generalize to  wide range of animal species.
Revised: LASSIE is category-agnostic and can easily generalize to a wide range of animal species.

Original: 65 • LASSIE can produce high-quality 3D articulated shapes with state-of-the-art results on existing benchmarks as well as self-collected in-the-wild web image ensembles
Revised: • LASSIE can produce high-quality 3D articulated shapes with state-of-the-art results on existing benchmarks as well as self-collected in-the-wild web image ensembles.

Original:  Related Work
Revised:  Related Work

Original: Animal shape and pose estimation.
Revised: Animal shape and pose estimation.

Original: Estimating 3D pose and shape of animal bodies from in-the-wildimages is highly ill-posed and challenging due to large variations across different classes, instances,viewpoints, articulations, etc.
Revised: Estimating 3D pose and shape of animal bodies from in-the-wild images is highly ill-posed and challenging due to large variations across different classes, instances, viewpoints, articulations, etc.

Revised: Most existing mesh reconstruction work [20, 23, 10, 35, 43] focus more on compact shapes like birds and cars, and thus cannot handle articulations of animal bodies.

Original: Most prior methods deal with a simplified scenario by assuming apre-defined class-level statistical model, instance-specific images, or accessible human annotations.
Revised: Recent articulation-aware methods, on the other hand, deal with a simplified scenario by assuming a pre-defined class-level statistical model, instance-specific images, or accessible human annotations.

Original: For instance, Zuffi et al.
Revised: For instance, Zuffi et al.

Original: [41] build a statistical shape model, SMAL, for common quadrupedal animals, similar to the SMPL [21] model for human bodies.
Revised: [47] build a statistical shape model, SMAL, for common quadrupedal animals, similar to the SMPL [25] model for human bodies.

Original: Follow-up works either optimize the
Revised: Follow-up work either optimize the

Original: SMAL parameters based on human annotated images [40] or train  neural network on large-scaleimage datasets to directly regress the parameters [39, 30].
Revised: SMAL parameters based on human annotated images [46] or train a neural network on large-scale image datasets to directly regress the parameters [45, 34].

Original: However, the output shapes are limitedby the SMAL shape space or training images which contain one or few animal classes.
Revised: However, the output shapes are limited by the SMAL shape space or training images which contain one or few animal classes.

Original: Kulkarni etal.
Revised: Kulkarni et al.

Original: [17] propose A-CSM technique that align a known template mesh with skinning weights to input2D images.
Revised: [19] propose A-CSM technique that align a known template mesh with skinning weights to input 2D images.

Original: Other recent works [35, 34, 36] exploit the dense temporal correspondence in videoframes to reconstruct articulated shapes.
Revised: Other recent approaches [40, 39, 41] exploit the dense temporal correspondence in video frames to reconstruct articulated shapes.

Original: In contrasted to existing methods, we deal with a novel andpractical problem setting without any pre-defined shape model, instance-level human annotations, ortemporal information to leverage.
Revised: In contrast to existing methods, we deal with a novel and practical problem setting: learn without any pre-defined shape model, instance-level human annotations, or temporal information to leverage.

Original: Part discovery from image collections.
Revised: Part discovery from image collections.

Original: Deep feature factorization (DFF) [7] shows that onecould automatically obtain consistent part segments by clustering intermediate deep features of aclassification network trained on ImageNet.
Revised: Deep feature factorization (DFF) [8] shows that one could automatically obtain consistent part segments by clustering intermediate deep features of a classification network trained on ImageNet.

Original: Inspired by DFF, SCOPS [10] learn a 2D part segmentation network from image collections, which can be used as semantic supervision for 3Dreconstruction [19].
Revised: Inspired by DFF, SCOPS [12] learns a 2D part segmentation network from image collections, which can be used as semantic supervision for 3D reconstruction [23]. Choudury et al. [7] proposed to use contrastive learning for 2D part discovery.

Original: Lathuilière et al.
Revised: Lathuilière et al.

Original: [18] exploit motion cues in videos for part discovery.
Revised: [21] exploit motion cues in videos for part discovery.

Original: Recently, Amir et al.
Revised: Recently, Amir et al.

Original: [1] demonstrate that clustering self-supervisedly learned vision transformers (ViT) such as DINO [5] features can provide good 2D part segmentations.
Revised: [1] demonstrate that clustering self-supervisedly learned vision transformers (ViT) such as DINO [5] features can provide good 2D part segmentations.

Original: In this work, we make use of DINOfeatures for 3D part discovery instead of 2D segmentation.
Revised: In this work, we make use of DINO features for 3D part discovery instead of 2D segmentation.

Original: In the 3D domain, Tulsiani et al.
Revised: In the 3D domain, Tulsiani et al.

Original: [31] usevolumetric cuboids as part abstractions to learn 3D reconstruction.
Revised: [36] use volumetric cuboids as part abstractions to learn 3D reconstruction.

Original: Mandikal et al.
Revised: Mandikal et al.

Original: [23] predict partsegmented 3D reconstructions from a single image.
Revised: [27] predict part-segmented 3D reconstructions from a single image.

Original: In addition, Luo et al.
Revised: In addition, Luo et al.

Original: [22] and Paschalidou etal.
Revised: [26] and Paschalidou et al.

Original: [26, 27] learn to form object parts by clustering 3D points.
Revised: [30, 31] learn to form object parts by clustering 3D points.

Original: Considering that most 3D approachesrequire ground-truth 3D shape of a whole object or its parts as supervision, Yao et al.
Revised: Considering that most 3D approaches require ground-truth 3D shape of a whole object or its parts as supervision, Yao et al.

Original: [37] propose todiscover and reconstruct 3D parts automatically by learning a primitive shape prior.
Revised: [42] propose to discover and reconstruct 3D parts automatically by learning a primitive shape prior.

Original: Most of theseworks assume some form of supervision in terms of 3D shape or camera viewpoint.
Revised: Most of these works assume some form of supervision in terms of 3D shape or camera viewpoint.

Original: In this paper,we aim to discover 3D parts of articulated animals from image ensembles without any 2D or 3Dannotations, which, to the best of our knowledge, is unexplored in the literature.3D reconstruction from image collections.
Revised: In this paper, we aim to discover 3D parts of articulated animals from image ensembles without any 2D or 3D annotations, which, to the best of our knowledge, is unexplored in the literature. 3D reconstruction from sparse images.

Original: Optimizing a 3D scene or object from multi-view imagesis one of the widely-studied problems in computer vision.
Revised: Optimizing a 3D scene or object from multi-view images is one of the widely-studied problems in computer vision.

Original: The majority of recent breakthroughs arebased on the powerful Neural Radiance Field (NeRF) [25] representation.
Revised: The majority of recent breakthroughs are based on the powerful Neural Radiance Field (NeRF) [29] representation.

Original: Given a set of multi-viewimages, NeRF can learn a neural volume from which one can render high-quality novel views.
Revised: Given a set of multi-view images, NeRF can learn a neural volume from which one can render high-quality novel views.

Original: Closelyrelated to our work is NeRS [38], which optimizes a neural surface representation from a sparseset of image collections and also only requires rough camera initialization.
Revised: Closely related to our work, NeRS [44] optimizes a neural surface representation from a sparse set of image collections and also only requires rough camera initialization.

Original: In this work, we useneural surface representations to represent animal parts.
Revised: In this work, we use a neural surface representation to represent animal parts.

Original: Existing methods assume multi-view imagescaptured in the same illumination setting and also the object appearance (texture) to be same acrossimages.
Revised: Existing methods assume multi-view images captured in the same illumination setting and also the object appearance (texture) to be same across images.

Original: Another set of works [3, 4] propose neural reflectance decomposition on image collectionscaptured in varying illuminations.
Revised: Another set of work [3, 4] propose neural reflectance decomposition on image collections captured in varying illuminations.

Original: But they work on rigid objects with ground-truth segmentation andknown camera poses.
Revised: But they work on rigid objects with ground-truth segmentation and known camera poses.

Original: In contrast, the input for our method consists of in-the-wild animal imageswith varying textures, viewpoint and pose articulations and captured in different environments.
Revised: In contrast, the input for our method consists of in-the-wild animal images with varying textures, viewpoints and pose articulations and captured in different environments.

Original:  Approach
Revised:  Approach

Original: Given a generic 3D skeleton and few images of an articulated animal species, LASSIE optimizesthe camera viewpoint and articulation for each instance as well as the resting canonical skeleton andpart shapes that are shared across all instances.
Revised: Given a generic 3D skeleton and few images of an articulated animal species, LASSIE optimizes the camera viewpoint and articulation for each instance as well as the resting canonical skeleton and part shapes that are shared across all instances.

Original: Wefirst introduce our neural part representation in 
Revised: We introduce our neural part representation in Section 3.

Original: Section 3.1 and theLASSIE optimization framework in Section 3.2.
Revised: and the optimization framework in Section 3.2.

Original: An overview of our method isillustrated in Fig.
Revised: An overview of LASSIE is illustrated in Fig.

Original: 3.1 Articulated shapes with neural part surfaces
Revised: 3.1 Articulated shapes with neural part surfaces

Original: 3D skeleton.
Revised: 3D skeleton.

Original: Constructing a template mesh or statistical shape model for specific animal classesrequires either the ground-truth 3D shapes or extensive human annotations, which are  hard toobtain in real-world scenarios.
Revised: Constructing a template mesh or statistical shape model for specific animal classes requires either the ground-truth 3D shapes or extensive human annotation, which are both hard to obtain in real-world scenarios.

Original: A 3D skeleton, on the other hand, is easy to annotate since it onlycontains a set of sparse joints connected by bones.
Revised: A 3D skeleton, on the other hand, is easy to annotate since it only contains a set of sparse joints connected by bones.

Original: Furthermore, it generalizes well across classes,and one can effortlessly modify certain joints or bones to adapt toa distinct animal types (e.g.,mammals vs. birds).
Revised: Furthermore, it generalizes well across classes, and one can effortlessly modify certain joints or bones to adapt to distinct animal types (e.g.,

Original: Assuming a simple and generic 3D skeleton is given, we propose a part-basedrepresentation for articulated shapes. A skeleton can be defined by a set of 3D joints P ∈ R p × 3and bones B = { ( P i , P j ) } b in a tree structure, where p and b = p − 1 are numbers of joints andbones/parts, respectively. For instance, we use the same base skeleton for all four-legged animals inour experiments, which specifies the connection of 16 joints and the initial joint coordinates in the Primitive decoder ℱ !
Revised: Prior MLP ℱ !

Original: Unit sphere
Revised: Unit sphere

Original: Part outputs
Revised: Part outputs

Original: Textured outputs
Revised: Textured outputs

Original: Part decoders { ℱ "∆ } (Δx, Δy, Δz) (u,v)
Revised: Part MLPs { ℱ "∆ } (Δx, Δy, Δz) (u,v)

Original: Primitive part encodings { e i }
Revised: Primitive part encodings { e i }

Original: Camera-space articulation
Revised: Camera-space articulation

Original: Primitive parts (x, y, z)
Revised: Primitive parts (x, y, z)

Original: Input images
Revised: Input images

Original: Sample texture
Revised: Sample texture

Original: DINO features ℒ $"% ℒ $&’ … …
Revised: DINO features ℒ $"% ℒ $&’ … …

Original: Canonical articulation 𝜋 R, t { e i } { ℱ "∆ }
Revised: Canonical articulation 𝜋 R, t { e i } { ℱ "∆ }

Original: Resting pose
Revised: Resting pose

Original: [Figure]
Revised: [Figure]

Original: Part Representation
Revised: Representation

Original: LASSIE

Original: [Figure]
Revised: [Figure]

Revised: mammals vs. birds).

Original:  root-relative 3D space.
Revised: Assuming a simple and generic 3D skeleton is given, we propose a part-based representation for articulated shapes. A skeleton can be defined by a set of 3D joints P ∈ R p ×and bones B = { ( P i , P j ) } b in a tree structure, where p and b = p − 1 are numbers of joints and bones/parts, respectively. For instance, we use the same base skeleton for all four-legged animals in our experiments. The base skeleton specifies the connection of 16 joints and the initial joint coordinates in the root-relative 3D space.

Original: The skeleton is shared for all instances in the same class and optimizable byscaling the bone lengths and rotating the bones with respect to the parent joints.
Revised: The skeleton is shared by all instances in the same class and can be optimized by scaling the bone lengths and rotating the bones with respect to the parent joints.

Original: Unlike the linearor neural blend skinning models [34, 36] where bones are unconstrained Gaussian components, ourskeleton-based representation allows us to regularize bone transformations, discover meaningful andconnected parts, and produce realistic shape animations.
Revised: Unlike the linear or neural blend skinning models [39, 41] where bones are unconstrained Gaussian components, our skeleton-based representation allows us to regularize bone transformations, discover meaningful and connected parts, and produce realistic shape animations.

Original: Neural part surfaces.
Revised: Neural part surfaces.

Original: Given a skeleton which specifies the 3D joints and bones, we represent thearticulated shape as a composition of several neural parts.
Revised: Given a skeleton which specifies the 3D joints and bones, we represent the articulated shape as a composition of several neural parts.

Original: Specifically, each part is defined as aneural surface wrapped around a skeleton bone.
Revised: Specifically, each part is defined as a neural surface wrapped around a skeleton bone.

Original: Motivated by object representation in NeRS [38],we represent part surface as a multi-layer perceptron (MLP)  that predicts the 3D deformation ofa given continuous UV coordinate on the unit sphere.
Revised: Motivated by object representation in NeRS [44], we represent part surface as a multi-layer perceptron (MLP) network that predicts the 3D deformation of a given continuous UV coordinate on the unit sphere.

Original: One of the key advantages of modellingparts instead of whole animal is that part shapes are usually  simple convex shapes and can berepresented well with a deformed sphere.
Revised: A key advantage of modeling parts instead of whole animal is that part shapes are usually of simple convex shapes and can be represented well with a deformed sphere.

Original: Compared to explicit mesh representation, the neuralrepresentation enables efficient mesh regularization on the output shapes while producing highresolution surfaces as MLPs can work with meshes of arbitrary vertex resolution.
Revised: Compared to explicit mesh representation, the neural representation enables efficient mesh regularization on the output shapes while producing high resolution surfaces as MLPs can work with meshes of arbitrary vertex resolution.

Original: Each part surfaceis first reconstructed in the canonical space, then scaled by the corresponding bone length, rotatedby the bone orientation, and centered at the bone center.
Revised: Each part surface is first reconstructed in the canonical space, then scaled by the corresponding bone length, rotated by the bone orientation, and centered at the bone centroid.

Original: This ensures that the parts are connectedunder various bone transformations and also easy to repose.
Revised: This ensures that the parts are connected under various bone transformations and also easy to repose.

Original: Formally, let X ∈ R 3 × m denotes them uniformly distributed 3D vertices on a unit sphere.
Revised: Formally, let X ∈ R 3 × m denote the m uniformly distributed 3D vertices on a unit sphere.

Original: Further let R i ∈
Revised: Further let R i ∈

Original: R 3 × 3 , t i ∈ R 3 and s
Revised: R 3 × 3 , t i ∈ R 3 and s

Original: i ∈ Rdenote the global rotation, translation and scaling of the i th part ( i ∈ { 0 , · · · , b } ), where k denotesthe number of parts.
Revised: i ∈ R denote the global rotation, translation and scaling of the i th part ( i ∈ { 0 , · · · , b } ), where b denotes the number of parts.

Original: We first deform the sphere to obtain part shape using the part MLP, X → F i ( X ) ;followed by global transformation to obtain part vertices V i ∈ R 3 × m in the global coordinate frame:
Revised: We first deform the sphere to obtain part shape using the part MLP, X → F i ( X ) ; followed by global transformation to obtain part vertices V i ∈ R 3 × m in the global coordinate frame:

Original: [Equation]
Revised: [Equation]

Original: Latent part prior.
Revised: Latent part prior.

Original: Although fitting neural part surfaces on sparse images can produce reasonableoutputs that are faithful to the input view, we observe that part shapes tend to be non-uniform orunrealistic from  novel views.
Revised: Although fitting neural part surfaces on sparse images can produce reasonable outputs that are faithful to the input view, we observe that part shapes tend to be non-uniform or unrealistic from other novel views.

Original: To further regularize the part shapes, we decompose the part MLP F iinto a prior MLP, F p and  part deformation MLP, F ∆ i .
Revised: To further regularize the part shapes, we decompose the part MLP F i into a prior MLP, F p and a part deformation MLP, F ∆ i .

Original: The prior MLP is trained to produce the baseshapes that are close to geometric primitives such as spheres, cylinders etc.
Revised: The prior MLP is trained to produce the base shapes that are close to geometric primitives such as spheres, cylinders, etc.

Original: The deformation MLP isthen used to provide additional deformations on top of the base shapes.
Revised: The deformation MLP is then used to provide additional deformations on top of the base shapes.

Original: Fig.
Revised: Fig.

Original: 2 illustrates these two MLPs.
Revised: 2 illustrates these two MLPs.

Original: As shown in the figure, the prior MLPalso takes  a d -dimensional latent shape code e i ∈ R das input in addition to the input coordinates.
Revised: As shown in the figure, the prior MLP takes as input the surface coordinate as well as a d -dimensional latent shape code e i ∈ R d .

Original: Both the latent space and the primitive MLP are trainedusing geometric primitive shapes.
Revised: Both the latent space and the prior MLP are trained using geometric primitive shapes.

Original: Concretely, we train a Variational Auto-Encoder (VAE) whichtakes a primitive mesh as input, predicts the latent shape encodings, and reconstructs the primitivesthrough a conditional neural surface decoder which forms our primitive MLP.
Revised: Concretely, we train a Variational Auto-Encoder (VAE) which takes a primitive mesh as input, predicts the latent shape encodings and reconstructs the primitive through a conditional neural surface decoder which forms our prior MLP.

Original: The primitive meshesare the randomly sampled geometric primitives (spheres, ellipsoids, cylinders and cones).
Revised: The primitive meshes in this work are randomly sampled geometric primitives (spheres, ellipsoids, cylinders, and cones).

Original: With theuse of prior and deformation MLPs, Eq. 1 now becomes:
Revised: With the use of prior and deformation MLPs, Eq. 1 now becomes:

Original: [Equation]
Revised: [Equation]

Original: where e i denotes the latent shape encodings of part i .
Revised: where e i denotes the latent shape encodings of part i .

Original: 3.2 Discovering 3D Neural 
Revised: 3.2 Discovering 3D Neural Parts from Image Ensemble

Original: Parts from Image Ensemble

Original: Optimization setting.
Revised: Optimization setting.

Original: The input to our optimization is a sparse set (typically 20-30) of n in-the-wildimages { I j } nj =1 along with a generic/rough 3D skeleton P ∈ R p × 3 that has p joints and b bones/parts.
Revised: The input to our optimization is a sparse set (typically 20-30) of n in-the-wild images { I j } nj =1 along with a generic/rough 3D skeleton P ∈ R p × 3 that has p joints and b bones/parts.

Original: No other form of 2D or 3D annotationsare available for optimization.
Revised: The only image-level annotations for optimization are from the self-supervisory DINO [5] features.

Original: All the instances in the inputimage ensemble are of same species, but could have varying texture (appearance) due to  pose, cameraangle, and lighting.
Revised: All the instances in the input image ensemble are of same species, but could have varying appearance due to texture, pose, camera angle, lighting, etc.

Original: We also assume that the entire animal  is visible in each image without truncatedor occluded by another object although some  self-occlusion is allowed.
Revised: We also assume that the entire animal body is visible in each image without truncated or occluded by another object although some degree of self-occlusion is allowed.

Original: Let j denote the indexover images j ∈ { 1 , ., n } and i denote the index over parts i ∈ { 1 , .
Revised: Let j denote the index over images j ∈ { 1 , ., n } and i denote the index over parts i ∈ { 1 , .

Original: , b } .
Revised: , b } .

Original: For each image, LASSIEoptimizes the camera viewpoint π j = ( R 0 , t 0 ) , part rotations R j ∈ R b × 3 × 3 and translations.
Revised: For each image, LASSIE optimizes the camera viewpoint π j = ( R 0 , t 0 ) and part rotations R j ∈ R b × 3 × 3 .

Original: Inaddition, LASSIE optimizes these variables for each of the i th part that are shared across all theanimal instances: bone length scaling s i ∈ R , latent shape encodings e i ∈ R d and part deformation MLPs F ∆ i .
Revised: In addition, LASSIE optimizes these variables for each of the i th part that are shared across all the animal instances: bone length scaling s i ∈ R , latent shape encodings e i ∈ R d and part deformation MLPs F ∆ i .

Original: In other words, we assume that part shapes are shared across all instances whereas theirarticulation (pose transformation) is instance dependent.
Revised: That is, we assume that part shapes are shared across all instances whereas their articulation (pose transformation) is instance dependent.

Original: Although this may not be strictly valid inpractice, this provides an important constraint for our highly ill-posed optimization problem.
Revised: Although this may not be strictly valid in practice, this provides an important constraint for our highly ill-posed optimization problem.

Original: Analysis by synthesis.
Revised: Analysis by synthesis.

Original: Since we do not have access to any form of 3D supervision, we use analysisby-synthesis to drive the optimization.
Revised: Since we do not have access to any form of 3D supervision, we use analysisby-synthesis to drive the optimization.

Original: That is, we use differentiable rendering to render2D featuresfrom the optimized 3D articulated shape and define loss functions w.r.t.
Revised: That is, we use differentiable rendering to render the optimized 3D articulated shape and define loss functions w.r.t.

Original: input images.
Revised: input images.

Original: Using Eq. 2, wefirst obtain the complete articulated shape for each image as a combination of different part meshes.
Revised: Using Eq. 2, we first obtain the complete articulated shape for each image as a combination of different part meshes.

Original: We then use  differentiable mesh rendering [20] to project the 3D points onto 2D image space usingthe learnable camera viewpoints.
Revised: We then use a differentiable mesh renderer [24] to project the 3D points onto 2D image space via the learnable camera viewpoints.

Original: A key challenge is that we do noteven have access to any from of2D annotations such as keypoints or part segmentations.
Revised: A key challenge is that we do not have access to any from of 2D annotations such as keypoints or part segmentation.

Original: Self-supervisory deep features to the rescue.
Revised: Self-supervisory deep features to the rescue.

Original: Since different instances in the ensemble can havedifferent texture, we do not model and render image pixel values.
Revised: Since different instances in the ensemble can have different texture, we do not model and render image pixel values.

Original: Instead, we rely on learned deepfeatures for our self-supervisory analysis-by-synthesis framework.
Revised: Instead, we rely on learned deep features for our self-supervisory analysis-by-synthesis framework.

Original: Recent works [1, 9] demonstratethat deep features from DINO [5] vision transformer are semantically descriptive, robust to appearancevariations, and higher-resolution compared to CNN features [1, 32].
Revised: Recent work [1, 11] demonstrate that deep features from DINO [5] vision transformer are semantically descriptive, robust to appearance variations, and higher-resolution compared to CNN features [1, 37].

Original: In this work, we use the keyfeatures from the last layer of the DINO network as semantic visual features for each image.
Revised: In this work, we use the key features from the last layer of the DINO network as semantic visual features for each image.

Original: Weassume that these DINO features are relatively similar for same parts across different instance.
Revised: We assume that these DINO features are relatively similar for same parts across different instances.

Original: Inaddition to using DINO features as providing consistent semantic features across images, we alsocompute 2D parts and foreground animal mask following similar steps as in [1].
Revised: In addition to using DINO features for consistent semantic features across images, we also compute 2D parts and foreground animal mask following similar steps as in [1].

Original: Concretely, we takethe class tokens from the last layer, compute the mean attention of class tokens as saliency maps,and sample the keys of image patches with high saliency scores.
Revised: Concretely, we take the class tokens from the last layer, compute the mean attention of class tokens as saliency maps, and sample the keys of image patches with high saliency scores.

Original: We then collect the salient keysfrom all images and apply an off-the-shelf K-means clustering algorithm to form c semantic partclusters.
Revised: We then collect the salient keys from all images and apply an off-the-shelf K-means clustering algorithm to form c semantic part clusters.

Original: We then obtain foreground silhouettes by simply thresholding the pixels with minimumdistance to cluster centroids.
Revised: Finally, we obtain foreground silhouettes by simply thresholding the pixels with minimum distance to cluster centroids.

Original: Fig.
Revised: Fig.

Original: 3 shows the DINO part clusters on sample images from differentimage ensembles.
Revised: 3 shows the DINO part clusters on sample images from different image ensembles.

Original: We observe that using 4 clusters works well in practice.
Revised: We observe that using 4 clusters works well in practice.

Original: The use of self-supervisedfeatures (keys) and rough silhouettes, we alleviate the dependency of ground-truth segmentation maskand keypoint annotations in our optimization framework.
Revised: With the use of self-supervised features (keys) and rough silhouettes, we alleviate the dependency of ground-truth segmentation mask and keypoint annotations in our optimization framework.

Original: It is still non-trivial to use these 2D featuresfor 3D part discovery.

Original: Next, we explain their use in our silhouette and semantic consistency losses.

Original: Silhouette Loss.
Revised: Silhouette Loss.

Original: We compute the silhouette loss L mask by rendering all part surfaces with adifferentiable renderer [20] and comparing with the pseudo ground-truth foreground masks obtainedvia clustering DINO features: L mask = (cid:80) j
Revised: We compute the silhouette loss L mask by rendering all part surfaces with a differentiable renderer [24] and comparing with the pseudo ground-truth foreground masks obtained via clustering DINO features: L mask = (cid:80) j

Original: ∥ M j − ˆ M j ∥ 2 , where M j and ˆ M j are respectively therendered and pseudo ground-truth silhouettes of instance j .
Revised: ∥ M j − ˆ M j ∥ 2 , where M j and ˆ M j are the rendered and pseudo ground-truth silhouettes of instance j , respectively.

Original: Although these silhouette based lossesencourage the overall shape to match the 2D silhouettes, the supervisory signal is often too coarse toresolve the ambiguities caused by different camera viewpoints and pose articulations.
Revised: Although these silhouette-based losses encourage the overall shape to match the 2D silhouettes, their supervisory signal is often too coarse to resolve the ambiguity caused by different camera viewpoints and pose articulations.

Original: Therefore, wedesign a novel semantic consistency loss to more densely supervise the reconstruction.2D-3D Semantic Consistency.
Revised: Therefore, we design a novel semantic consistency loss to more densely supervise the reconstruction. 2D-3D Semantic Consistency.

Original: We propose a novel strategy to make use of the semantically consistent  DINO featuresin 2D for 3D part discovery.
Revised: We propose a novel strategy to make use of the semantically consistent 2D DINO features for 3D part discovery.

Original: At a high-level, we impose the 2D feature consistencyacross the images via learned 3D parts.
Revised: At a high-level, we impose the 2D feature consistency across the images via learned 3D parts.

Original: For this, we propose an Expectation-Maximization (EM)like optimization strategy where we alternatively estimate the 3D part features (E-step) from imageensemble features and then use these estimated 3D part features to update the parts (M-step).
Revised: For this, we propose an Expectation-Maximization (EM) like optimization strategy where we alternatively estimate the 3D part features (E-step) from image ensemble features and then use these estimated 3D part features to update the parts (M-step).

Original: Moreformally, let K j ∈ R h
Revised: More formally, let K j ∈ R h

Original: ×
Revised: ×

Original: w × f denotes the f -dimensional DINO key features from j th image and
Revised: w × f denote the f -dimensional DINO key features from j th image and

Original: Q ∈ R m × f denotes the corresponding features on the 3D shape with total m vertices.
Revised: Q ∈ R m × f denote the corresponding features on the 3D shape with total m vertices.

Original: In the Estep, we update Q by projecting the 3D coordinates onto all the 2D image spaces via differentiablerendering under the current estimated shape and camera viewpoints.
Revised: In the Estep, we update Q by projecting the 3D coordinates onto all the 2D image spaces via differentiable rendering under the current estimated shape and camera viewpoints.

Original: We aggregate the 2D imagefeatures from the image ensemble corresponding to each 3D vertex to estimate Q .
Revised: We aggregate the 2D image features from the image ensemble corresponding to each 3D vertex to estimate Q .

Original: In the M-step, we optimize the 3D parts and camera viewpoints using a 2D-3D semantic consistency loss L sem .
Revised: In the M-step, we optimize the 3D parts and camera viewpoints using a 2D-3D semantic consistency loss L sem .

Original: Todefine L sem , we discretize the foreground 2D coordinates in each image { p | ˆ M j ( p ) = 1 } .
Revised: To define L sem , we discretize the foreground 2D coordinates in each image { p | ˆ M j ( p ) = 1 } .

Original: Likewise,we sample a set of points { v ∈ V j } on the 3D part surfaces and obtain their 2D projected coordinatesπ j ( v ) .
Revised: Likewise, we sample a set of points { v ∈ V j } on the 3D part surfaces and obtain their 2D projected coordinates π j ( v ) .

Original: L sem is then defined as the Chamfer distance in a high-dimensional space:
Revised: L sem is then defined as the Chamfer distance in a high-dimensional space:

Original: [Equation]
Revised: [Equation]

Original: For each instance j , the distance D between a image point p and 3D surface point v is defined as:
Revised: For each instance j , the distance D between an image point p and 3D surface point v is defined as:

Original: [Equation]
Revised: [Equation]

Original: where α is a scalar weighting for semantic distance.
Revised: where α is a scalar weighting for semantic distance.

Original: In effect, L sem optimizes the 3D part shapessuch that the aggregated 3D points features in the E-step would project closer to the similar pixelfeatures in the image ensemble.
Revised: In effect, L sem optimizes the 3D part shapes such that the aggregated 3D point features in the E-step would project closer to the similar pixel features in the image ensemble.

Original: We alternate between E and M steps every other iteration in theoptimization process.
Revised: We alternate between E and M steps every other iteration in the optimization process.

Original: In Eq.
Revised: In Eq.

Original: 4, the image coordinates p and semantic features K, Q are fixed in eachoptimization iteration, hence L sem can effectively push the surface points towards their corresponding2D coordinates by minimizing the overall distance.
Revised: 4, the image coordinates p and semantic features K, Q are fixed in each optimization iteration, hence L sem can effectively push the surface points towards their corresponding 2D coordinates by minimizing the overall distance.

Original: Since this EM-style optimization is prone tolocal-minima as we jointly optimize camera viewpoints, shapes and pose articulations, we find thatgood initialization of 3D features is important.
Revised: Since this EM-style optimization is prone to local-minima as we jointly optimize camera viewpoints, shapes and pose articulations, we find that a good initialization of 3D features is important.

Original: For this, we manually map each part index i toone of the DINO clusters (4 in our experiments) and then initialize the 3D features Q by assigningthe average 2D cluster features to the corresponding 3D part vertices.
Revised: For this, we manually map each part index i to one of the DINO clusters (4 in our experiments) and then initialize the 3D features Q by assigning the average 2D cluster features to the corresponding 3D part vertices.

Original: This manual 3D part to 2Dcluster assignment takes minimal effort since it is category-level (all instances share the same 3Dparts).
Revised: This manual 3D part to 2D cluster assignment takes minimal effort since it is category-level (all instances share the same 3D parts).

Original: Alternatively, one could roughly initialize the camera viewpoints like in recent works such as
Revised: Alternatively, one could roughly initialize the camera viewpoints like in recent works such as

Original: NeRS [38] which allows the computation of initial Q features from 2D features.
Revised: NeRS [44] which allows the computation of initial Q features from 2D features.

Original: Nonetheless, 2D-3Dassignment is easier and faster compared to camera viewpoint initialization as 2D-3D assignmentneeds to be done once per image ensemble whereas camera initialization needs to done for eachinstance in the ensemble.
Revised: Nonetheless, 2D-3D assignment is easier and faster compared to camera viewpoint initialization as 2D-3D assignment only needs to be done once per image ensemble whereas camera initialization is required for each instance in the ensemble.

Revised: To make the optimization process fully automatic, we also propose to use simple heuristics for the 2D-3D part assignment using intuitions like animal head are usually above the torso and legs at the bottom.

Revised: We refer to the LASSIE optimization with this heuristic-based 3D feature initialization as ‘LASSIE-heuristic’ and describe the details in the supplemental material.

Revised: Comparison to related methods.

Revised: Our use of 2D-3D semantic consistency is closely related to prior works ,UMR [23] and BanMo [41].

Original: Regularizations.
Revised: At a high-level, LASSIE and UMR both start from 2D part segmentations and then lift them onto 3D. A key difference is that LASSIE directly discovers parts in 3D using 2D feature consistency, whereas UMR uses 2D part consistency via canonical UV maps. More specifically, UMR uses 2D parts (not 2D features) and 2D UV maps as common canonical part representations. Instead, LASSIE directly optimizes 3D part features with feature based loss (not 2D parts) and LASSIE does not use UV maps like in UMR. We only use 2D parts for feature initialization and not during optimization.

Revised: That is, we discover parts directly in 3D with skeleton constraints, without much reliance on intermediate 2D part discovery.

Revised: Hence, LASSIE is less sensitive to the issues in 2D part segmentations compared to UMR.

Revised: BanMo, on the other hand, learns a canonical feature embedding by enforcing the consistency between feature matching and geometric warping, which is made tractable by the dense temporal correspondence (optical flow) between video frames.

Revised: The semantic loss in BanMo enforces the 2D-3D cycle consistency between 2D coordinates and canonical 3D points.

Revised: Considering that our image ensemble is sparse and un-correlated, we coarsely initialize and regularize the 3D surface features at the part-level.

Revised: The proposed semantic loss allows us to first localize the 3D parts then refine the detailed part shapes in the challenging setting.

Revised: Regularization.

Original: Our skeleton-based representation allows us to conveniently impose pose prioror regularization.
Revised: Our skeleton-based representation allows us to conveniently impose pose prior or regularization.

Original: Specifically, we calculate a pose prior loss L pose to minimize the bone rotationdeviations from the resting pose as: L pose =
Revised: Specifically, we calculate a pose prior loss L pose to minimize the bone rotation deviations from the resting pose as: L pose =

Original: (cid:80)
Revised: (cid:80)

Original: j ∥
Revised: j ∥

Original: R j − ¯ R ∥ 2 , where R j is the bone rotations ofinstance j and ¯ R denotes the bone rotations of shared resting pose.
Revised: R j − ¯ R ∥ 2 , where R j is the bone rotations of instance j and ¯ R denotes the bone rotations of shared resting pose.

Original: We also impose a regularization
Revised: We also impose a regularization

Original: L ang on joint angles on the animal legs to limit their sideway rotations:
Revised: L ang on joint angles on the animal legs to limit their sideway rotations:

Original: [Equation]
Revised: [Equation]

Original: where R i,y and R i,z are the rotations of bone i with respect to y and z axes respectively.
Revised: where R i,y and R i,z are the rotations of bone i with respect to y and z axes respectively.

Original: Since theskeleton faces the + z direction in the canonical space ( + x right, + y up, + z out), minimizing they and z -axis rotations essentially constrain the sideway movements of the bones.
Revised: Since the skeleton faces the + z direction in the canonical space ( + x right, + y up, + z out), minimizing the y and z -axis rotations essentially constrains the sideway movements of the bones.

Revised: Note that both L pose and L ang are generic to all quadrupeds in our experiments, which we find crucial to avoid unrealistic poses.

Original: Finally, we applycommon mesh regularization terms to encourage smooth surfaces, including the Laplacian loss L lapand surface normal loss L norm .
Revised: Finally, we apply common mesh regularization terms to encourage smooth surfaces, including the Laplacian loss L lap and surface normal loss L norm .

Original: L lap regularizes 3D surfaces by pushing each vertex to the center ofits neighbors, and L norm encourages neighboring mesh faces to have similar normal vectors.
Revised: L lap regularizes 3D surfaces by pushing each vertex towards the center of its neighbors, and L norm encourages neighboring mesh faces to have similar normal vectors.

Original: Notethat we apply regularization losses to each part surface individually since the part shapes should beprimitive-like and usually do not have sharp or pointy surfaces.
Revised: Note that we apply regularization losses to each part surface individually since the part shapes should be primitive-like and usually do not have sharp or pointy surfaces.

Original: Optimization and textured outputs.
Revised: Optimization and textured outputs.

Original: The overall optimization objective is:
Revised: The overall optimization objective is:

Original: [Equation]
Revised: [Equation]

Original: where { λ i } are weighting hyper-parameters.
Revised: where { λ i } are weighting hyper-parameters.

Original: We first pre-train the part prior MLP F p with 3Dgeometric primitives and freeze it during the optimization on a given image ensemble.
Revised: We first pre-train the part prior MLP F p with 3D geometric primitives and freeze it during the optimization on a given image ensemble.

Original: For eachimage ensemble of an animal species, we perform alternative optimization on the camera, pose, andshape parameters until convergence.
Revised: For each image ensemble of an animal species, we perform multi-stage optimization on the camera, pose, and shape parameters until convergence.

Original: That is, we update the camera viewpoints and fix the rest first,then optimize the bone transformations, and finally the latent part codes as well as part deformation
Revised: That is, we update the camera viewpoints and fix the rest first, then optimize the bone transformations, and finally the latent part codes as well as part deformation

Original: MLPs.
Revised: MLPs.

Revised: In each iteration, we first update the semantic features of 3D surfaces, then use the updated features to update 3D surfaces, forming an EM-style optimization.

Original: All the optimizable parametersare updated by an Adam optimizer [13] implemented in PyTorch [28].
Revised: We implement the framework in PyTorch [32] and update all the learnable parameters by an Adam optimizer [15].

Original: To obtain the final textured outputs, we densely sample vertices from the optimizedneural surfaces and directly sample the colors of visible vertices from individual images via 2Dprojection, where the visibility information is obtained from naive rasterization.
Revised: To obtain the final textured outputs, we densely sample vertices from the optimized neural surfaces and directly sample the colors of visible vertices from individual images via 2D projection, where the visibility information is obtained from naive rasterization.

Original: For the invisible(self-occluded) vertices, we assign the color by finding their left-right symmetries or nearest visibleneighbors.
Revised: For the invisible (self-occluded) vertices, we assign the color by finding their left-right symmetries or nearest visible neighbors.

Original: More implementation details can be found in the supplemental material.
Revised: More implementation details can be found in the supplemental material.

Original: [Figure]

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: Considering the novel problemsetting of sparse-image optimization, we select a subset ofthe Pascal-part dataset [6] and collect web images of additional animal classes to evaluate LASSIEand related prior art.
Revised: Considering the novel problem of sparse-image optimization, we select a subset of the Pascal-Part dataset [6] and collect web images of additional animal classes to evaluate LASSIE and related prior arts.

Original: The Pascal-part dataset contains diverse images of several animal classes whichare annotated with 2D part segmentation masks.
Revised: The Pascal-Part dataset contains diverse images of several animal classes which are annotated with 2D part segmentation masks.

Original: We automatically select images of horse, cow, orsheep with only one object covering 1-50% area of the entire image.
Revised: We automatically select images of horse, cow, or sheep with only one object covering 1-50% area of the entire image, and find the keypoints by calculating the centers/corners of ground-truth part masks.

Original: In addition, for analysis on morediverse animal categories, we collect sparse image ensembles (with CC-licensed images) of someother animals from the internet and  annotate the 2D keypoints for evaluation.
Revised: In addition, for analysis on more diverse animal categories, we collect sparse image ensembles (with CC-licensed images) of some other animals from the internet and manually annotate the 2D keypoints for evaluation.

Original: The classes includequadrupeds like zebra, tiger, giraffe, elephant, as well as bipeds like kangaroo and penguin.
Revised: The classes include quadrupeds like zebra, tiger, giraffe, elephant, as well as bipeds like kangaroo and penguin.

Original: Wefilter the examples where objects are heavily occluded or truncated, resulting in about 30 imagesper category.
Revised: We filter out the images where the animal body is heavily occluded or truncated, resulting in roughly 30 images per category.

Original: The LASSIE optimization and evaluation are performed on each image ensembleseparately.
Revised: The LASSIE optimization and evaluations are performed on each image ensemble separately.

Revised: The self-collected data does not contain personally identifiable information or offensive content, which will be released to public.

Original: Baselines.
Revised: Baselines.

Original: We mainly compare our results with  3D Safari [39] and A-CSM [17] as they also learn 3Darticulated shapes from animal images and there exists no closer framework to our novel scenario.
Revised: Due to the lack of prior works on our problem setting (sparse image optimization for articulated animal shapes), we mainly compare LASSIE with learning-based mesh reconstruction methods. Among these methods, we find 3D Safari [45] and A-CSM [19] are most comparable to LASSIE since they also model articulation for animal classes of our interest.

Revised: Other recent mesh reconstruction methods, on the other hand, either cannot handle articulations [20, 23, 10, 35, 43] or assume different inputs [22, 39, 41].

Original: Wenote that both 3D Safari and A-CSM are trained on large-scale image sets while LASSIE optimizeson a sparse image ensemble.
Revised: Note that both 3D Safari and A-CSM are trained on large-scale image sets while LASSIE optimizes on a sparse image ensemble.

Original: To evaluate their methods on our dataset, we use the  models trainedon the closest animal classes.
Revised: To evaluate their methods on our dataset, we use the released models trained on the closest animal classes.

Original: For instance, the zebra models of 3D Safari and the horse model of A-CSM are used to evaluate on similar quadrupeds.
Revised: For instance, the zebra model of 3D Safari and the horse model of A-CSM are used to evaluate on similar quadrupeds.

Original: Visual Comparisons.
Revised: Visual Comparisons.

Original: Fig.
Revised: Fig.

Original: 3 shows the qualitative results of LASSIE against 3D Safari and A-CSMon the self-collected animal images.
Revised: 3 shows the qualitative results of LASSIE against 3D Safari and A-CSM on the self-collected animal images.

Original: The 3D Safari [39] model is trained on zebra images and doesnot generalize well to other animal classes.
Revised: The 3D Safari [45] model is trained on zebra images and does [Figure] [Table] not generalize well to other animal classes.

Original: A-CSM [17] assume high-quality skinned model of anobject category is able to produce detailed shapes of animal bodies which.
Revised: A-CSM [19] assumes high-quality skinned model of an object category which enables detailed shapes of animal bodies.

Original: However, they do notalign well  the given 2D images.
Revised: However, its outputs do not align well with the given 2D images.

Original: Our results demonstrate that LASSIE can effectively learn fromsparse image ensemble to discover 3D articulated parts that are high-quality, faithful to input images,and semantically consistent across instances.
Revised: Our results demonstrate that LASSIE can effectively learn from sparse image ensemble to discover 3D articulated parts that are high-quality, faithful to input images, and semantically consistent across instances.

Original: Note that our 3D reconstructions also improve the 2Dpart segmentation from DINO-ViT feature clustering [1].
Revised: Note that our 3D reconstructions also improve the 2D part segmentation from DINO-ViT feature clustering [1].

Original: Keypoint transfer metrics.
Revised: Keypoint transfer metrics.

Original: As there exist no ground-truth 3D annotations in our datasets, wequantitatively evaluate using 2D keypoint transfer between each pair of images which is a standardpractice [39, 17].
Revised: As there is no ground-truth 3D annotation in our datasets, we quantitatively evaluate using 2D keypoint transfer between each pair of images which is a standard practice [45, 19].

Original: We map a given set of 2D keypoint on a source image onto the 3D part surfaces,and then project them to a target image using the optimized camera viewpoints, shapes and posearticulations.
Revised: We map a given set of 2D keypoints on a source image onto the 3D part surfaces, and then project them to a target image using the optimized camera viewpoints, shapes and pose articulations.

Original: Since this keypoint transfer goes through the 2D-to-3D and 3D-to-2D mappings,a successful keypoint transfer requires accurate 3D reconstruction on both the source and target [Table] [Figure] [Table]
Revised: Since this keypoint transfer goes through the 2D-to-3D and 3D-to-2D mappings, a successful keypoint transfer requires accurate 3D reconstruction on both the source and target images.

Original: LASSIE images. We calculate the percentage of correct keypoints (PCK) under two different thresholds:0.1 × max ( h, w ) and 0.05 × max ( h, w ) , where h and w are image height and width respectively.
Revised: We calculate the percentage of correct keypoints (PCK) under two different thresholds: 0.1 × max ( h, w ) and 0.05 × max ( h, w ) , where h and w are image height and width, respectively.

Original: Table 1 shows that LASSIE achieves higher PCK on the Pascal-part [6] images and most other classesin our in-the-wild image collections.
Revised: Table 1 shows that LASSIE achieves higher PCK on the Pascal-part [6] images and most other classes in our in-the-wild image collections.

Original: We also show ablative results of LASSIE without using thesemantic feature consistency loss L sem or part prior MLP F p .
Revised: We also show ablative results of LASSIE without using the semantic feature consistency loss L sem or part prior MLP F p .

Original: Note that L sem is essential to fit theskeleton and part surfaces faithfully to the input images, and F p provides effective regularization onpart surfaces to generate realistic shapes from various viewpoints. 
Revised: Note that L sem is essential to fit the skeleton and part surfaces faithfully to the input images, and F p provides effective regularization on part surfaces to generate realistic shapes from various viewpoints. LASSIE-heuristic achieves slightly lower PCK [Figure] [Figure]

Revised: [Table]

Revised: but still performs favorably against prior arts overall.

Original: Example visual results of keypointtransfer are shown in Fig.
Revised: Example visual results of keypoint transfer are shown in Fig.

Original: 5.2D overall/part IOU metrics.
Revised: 2D overall/part IOU metrics.

Original: Our part-based representation can generate realistic 3D shapes andcan even improve 2D part discovery compared to prior art.
Revised: Our part-based representation can generate realistic 3D shapes and can even improve 2D part discovery compared to prior arts.

Original: In Table 2 and Figure 4, we show thequalitative and quantitative comparisons against prior 3D works as well as 2D co-part discoverymethods of SCOPS [10] and DINO clustering [1].
Revised: In Table 2 and Fig. 4, we show the qualitative and quantitative comparisons against prior 3D works as well as 2D co-part discovery methods of SCOPS [12] and DINO clustering [1].

Original: For SCOPS [10] and DINO clustering [1], weset the number of parts N p = 4 (which we find to be optimal for these techniques) and manuallyassign each discovered part to the best matched part in the Pascal-part annotations.
Revised: For SCOPS [12] and DINO clustering [1], we set the number of parts N p = 4 (which we find to be optimal for these techniques) and manually assign each discovered part to the best matched part in the Pascal-part annotations.

Original: Note that higherpart IOU against Pascal-part annotations may not strictly indicate better part segmentation since theself-discovered parts need not correspond to human annotations.
Revised: Note that higher part IOU against Pascal-part annotations may not strictly indicate better part segmentation since the self-discovered parts need not correspond to human annotations.

Original: Compared to prior arts on articulatedshape reconstruction, LASSIE results match the Pascal-part segmentation mask better and achieveshigher overall IOU.
Revised: Compared to prior arts on articulated shape reconstruction, LASSIE results match the Pascal-part segmentation mask better and achieves higher overall IOU.

Original: On co-part segmentation, our 3D reconstruction captures the semantic parts in2D while being more geometrically refined than DINO clustering.
Revised: On co-part segmentation, our 3D reconstruction captures the semantic parts in 2D while being more geometrically refined than DINO clustering.

Original: Moreover, our results can separateparts with similar semantic features, e.g. , 4 animal legs, and thus resulting in much higher part IOU.
Revised: Moreover, our results can separate parts with similar semantic features, e.g. , 4 animal legs, and thus resulting in much higher part IOU.

Original: Part transfer.
Revised: Part transfer.

Original: While keypoint transfer and part IOU are commonly used in the literature, weobserve that they are either sparse, biased towards certain body parts ( e.g. , animal faces), or notdirectly comparable due to part mismatch.
Revised: While keypoint transfer and part IOU are commonly used in the literature, we observe that they are either sparse, biased towards certain body parts ( e.g. , animal faces), or not directly comparable due to part mismatch.

Original: To address this, we propose a part transfer metric forbetter evaluation of 3D reconstruction consistency across image ensemble.
Revised: To address this issue, we propose a part transfer metric for better evaluation of 3D reconstruction consistency across image ensemble.

Original: This part transfer metricis similar to keypoint transfer but uses part segmentations.
Revised: This part transfer metric is similar to keypoint transfer but uses part segmentations.

Original: That is, we densely transfer the partsegmentation in a source image to a target image through forward and backward mapping between [Table] [Figure] [Table]
Revised: That is, we densely transfer the part

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] the 2D pixels and the canonical 3D surfaces.
Revised: segmentation in a source image to a target image through forward and backward mapping between the 2D pixels and the canonical 3D surfaces.

Original: A pixel is considered transferred correctly if and only ifit is mapped to the same 2D part in the target image as where it belongs in the source image.
Revised: A pixel is considered transferred correctly if and only if it is mapped to the same 2D part in the target image as where it belongs in the source image.

Original: Similarto PCK in keypoint transfer evaluation, we calculate the percentage of correct pixels (PCP) for eachsource-target pair.
Revised: Similar to PCK in keypoint transfer evaluation, we calculate the percentage of correct pixels (PCP) for each source-target pair.

Original: Since the part segmentation only covers the visible surfaces and not the occludedregions, we ignore the pixels mapped to a self-occluded surface when calculating the PCP metric.
Revised: Since the part segmentation only covers the visible surfaces and not the occluded regions, we ignore the pixels mapped to a self-occluded surface when calculating the PCP metric.

Original: Thequantitative results are shown in Table 2, which demonstrate the favorable performance of LASSIEagainst prior arts.
Revised: The quantitative results are shown in Table 2, which demonstrate the favorable performance of LASSIE against prior arts.

Original: Applications.
Revised: Applications.

Original: Our 3D neural part representation not only produces more faithful and realistic shapesbut also enables various applications like part manipulation, texture transfer, animation (re-posing),etc.
Revised: Our 3D neural part representation not only produces more faithful and realistic shapes but also enables various applications like part manipulation, texture transfer, motion re-targeting, etc.

Original: Specifically, we can generate a new shape by swapping or interpolating certain parts, transfer thesampled surface texture from one object class to another using the same base skeleton, or specify thebone rotations to produce desirable re-posed shape or animation.
Revised: Specifically, we can generate a new shape by swapping or interpolating certain parts, transfer the sampled surface texture from one object class to another using the same base skeleton, or specify the bone rotations to produce desirable re-posed shape or animation.

Original: We show some example results ofpose transfer and texture transfer between different animal classes in Figure 6.
Revised: We show some example results of pose transfer and texture transfer between different animal classes in Fig.

Original: Limitations.
Revised: Limitations.

Revised: LASSIE relies heavily on the 3D skeleton and strong pose and shape regularization.

Revised: As the first attempt to address this novel and highly ill-posed problem, we find the proposed pose and shape constraints essential to produce good quality results and avoid unrealistic outputs.

Original:  LASSIE can not handle image collections with heavy  occlusions, truncations, orextreme pose variations since we leverage self-supervised DINO-ViT features on sparse imageensemble as our main supervision.
Revised: Moreover, LASSIE can not handle image collections with heavy truncation, occlusions, or extreme pose variations since we leverage self-supervisory DINO-ViT features on sparse image ensemble as our main supervision.

Original: Moreover, we require category-level annotations of 3D skeletonand 2D-3D part mapping for semantic feature initialization, which forms important future work.
Revised: For instance, partial-body or noisy DINO features can cause ambiguities in camera pose and part localization. Finally, we observe that LASSIE currently struggles with a) highly articulated parts like elephant trucks and b) fluffy animals that appear with more instance variations and ambiguous articulations.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we study a novel and practical problem of articulated shape optimization of animalsfrom sparse image collections in-the-wild.
Revised: In this paper, we study a novel and practical problem of articulated shape optimization of animals from sparse image collections in-the-wild.

Original: Instead of relying on pre-defined shape models or humanannotations like keypoints or segmentation masks, we propose a neural part representation based on ageneric 3D skeleton, which is robust to appearance variations and generalizes well across differentanimal classes.
Revised: Instead of relying on pre-defined shape models or human annotations like keypoints or segmentation masks, we propose a neural part representation based on a generic 3D skeleton, which is robust to appearance variations and generalizes well across different animal classes.

Original: In LASSIE, our key insight is to reason about animal parts instead of whole shape asparts allow imposing several constraints.
Revised: In LASSIE, our key insight is to reason about animal parts instead of whole shape as parts allow imposing several constraints.

Original: We leverage self-supervisory dense ViT features to provideboth silhouette and semantic part consistency for supervision.
Revised: We leverage self-supervisory dense ViT features to provide both silhouette and semantic part consistency for supervision.

Original: Both quantitative and qualitative resultson Pascal-Part and our own in-the-wild image ensembles show that LASSIE can effectively learnfrom few in-the-wild images and produce high-quality results.
Revised: Both quantitative and qualitative results on Pascal-Part and our own in-the-wild image ensembles show that LASSIE can effectively learn from few in-the-wild images and produce high-quality results.

Original: Our part-based representation alsoallows various applications such as pose interpolation, texture transfer, and 
Revised: Our part-based representation also enables various applications such as pose interpolation, texture transfer, and animation.

Revised: We hope that this work will facilitate future research efforts on learning, inferring, and manipulating articulated shapes in this novel setting.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?  • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [No] Since the scope of our paper is within 3D reconstruction of animal bodies, there is no potential negative social impact that we are aware of.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [No] In our optimization framework, the random seed initialization does not affect the outputs much since the most sensitive parameters (e.g. camera, bone scaling, pose) are not randomly initialized.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] 
Revised: [Yes] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]