Geometry-aware Two-scale PIFu Representation for Human Reconstruction
Abstract
Although PIFu-based 3D human reconstruction methods are popular, the quality of recovered details is still unsatisfactory.
In a sparse ( e.g. , 3 RGBD sensors) capture setting, the depth noise is typically amplified in the PIFu representation, resulting in flat facial surfaces and geometry-fallible bodies.
In this paper, we propose a novel geometry-aware two-scale PIFu for 3D human reconstruction from sparse, noisy inputs.
Our key idea is to exploit the complementary properties of depth denoising and 3D reconstruction, for learning a two-scale PIFu representation to reconstruct high-frequency facial details and consistent bodies separately.
To this end, we first formulate depth denoising and 3D reconstruction as a multi-task learning problem.
The depth denoising process enriches the local geometry information of the reconstruction features, while the reconstruction process enhances depth denoising with global topology information.
We then propose to learn the two-scale PIFu representation using two MLPs based on the denoised depth and geometry-aware features.
Extensive experiments demonstrate the effectiveness of our approach in reconstructing facial details and bodies of different poses and its superiority over state-of-the-art methods.
Introduction
Three-dimensional human reconstruction, which aims to obtain a dense surface geometry from singleview or multi-view human images, is a fundamental topic in computer vision and computer graphics.
While reconstructing high-fidelity 3D human models is possible using commercial multi-view/stereo software under the customized studio setting [14, 58, 81, 29, 85], it is highly desirable to lift the studio setting constraint, which may be inaccessible to most users.
Low-cost RGBD sensors have recently become popular in 3D human reconstruction, and tracking-based methods are developed to fuse the depth data from RGBD sensors for reconstruction [66].
During fusion, the estimation of non-rigid human body deformation is essential to improve the reconstruction quality [65, 101].
However, it is technically challenging to ensure the stability of the depth fusion algorithm, due to occlusions and severe noise in the depth data.
Recently, learning-based 3D human reconstruction methods have significantly simplified the capture setting.
The parametric human model [60, 69, 67] is introduced to reduce the modeling difficulties from complex poses.
After training on images-to-model pairs, methods [46, 105] may even reconstruct 3D human shapes from single images.
However, these methods typically only obtain minimally clothed human bodies.
As detail requirements increase, the focus of learning-based human reconstruction methods has been shifted to the implicit representation, e.g. , pixel-aligned implicit function (PIFu) [73, 74, 40].
PIFu-based methods [73] can reconstruct human bodies with different types of details ( e.g. , hair and clothing) without utilizing predefined templates.
However, they often produce topology errors in the reconstructed human models, especially in the regions that are invisible or where the input depth is highly noisy ( e.g. , hairs).
While PIFuHD [74] partially alleviates this problem by synthesizing the body’s normal maps at both front and back sides, the details reconstructed from the synthesized normals may not be consistent with the target.
Hence, some methods [40, 100] resort to the multi-view feature fusion scheme to reduce these topology errors.
Although implicit methods develop fast, we observe that the quality of their recovered details is still unsatisfactory under the sparse capture settings ( e.g. , 3 RGBD sensors).
Due to the less overlapping between sparse views, the input depth noise is typically amplified, increasing the difficulty of performing stable reconstruction.
In addition, due to the ineffective fusion of RGB and depth features, these methods may not easily reconstruct high-frequency details.
As a result, we often observe flat or incorrect facial surfaces, body geometries with topology errors, as shown in Fig.
1(b,c,d).
[Figure]
Our method learns the geometry-aware two-scale PIFu representation, which can produce vivid facial/ and accurate
In this paper, we propose a geometry-aware two-scale PIFu representation for reconstructing digital humans from sparse, noisy inputs.
Our method is based on two observations.
First, depth denoising and 3D reconstruction are complementary to each other.
The former task preserves local geometric fidelity, while the latter task provides global topology guidance.
Second, while a function of high complexity may not be easy to express, it is much easier to approximate it piecewise ( e.g. , in two parts).
Inspired by these observations, we first formulate depth denoising and 3D reconstruction as a multi-task learning problem.
The two tasks can work together to further improve the reconstruction quality.
We encode RGB and depth separately, and use fused features to perform depth denoising, enriching deep features with local geometric information.
Based on the denoised depth and geometryaware features, we propose using two MLPs to represent the PIFus for the face (particular region) and the body, respectively.
This separate modeling increases the network capacity for handling details at different scales.
As shown in Fig.
1(e, f), our method can produce results with high-fidelity body and facial details for different actions.
Our main contributions are:
1) We propose a novel geometry-aware PIFu method for digital human reconstruction from sparse, noisy RGBD images.
Our approach exploits the complementary properties of depth denoising and 3D reconstruction to learn robust geometry information while suppressing the input noise.
We propose to learn a two-scale PIFu representation based on geometry-aware features, by using two individual MLPs for the face and the body separately.
The two-scale formulation enhances the network capacity in producing high-frequency facial details and smooth body surfaces.
3) Extensive experiments demonstrate that the proposed method can produce high-quality digital human reconstruction results, based on noisy depth maps taken by three Azure Kinect sensors.
Related Works
Tracking-based Human Reconstruction methods [50, 66, 109, 65, 30, 19, 49, 18, 101, 33, 79, 100] track human motions and infer the non-rigid deformations of the references to reconstruct 3d human mesh in a temporal fusion manner.
The references are typically parameterized as the 3D poses/skeletons [24, 98, 99, 92, 101, 33, 34, 55] and/or parametric body models [7, 102, 101, 45] ( e.g. ,SMPL [60, 69, 45]).
Some methods combine tracking with segmentation [59] or optical flow estimation [63] to help compute the references for reconstruction.
To tackle the occlusion and large motion problems, some methods develop high-end systems for the dense capture of human performance, consisting of a large number (up to 100) of RGBD sensors [14, 58, 81] or custom color lights [29, 85] ( e.g. , 1,200 individually controllable light sources in the acquisition setup in [85]).
Learning-based Human Reconstruction methods [108, 77, 2, 23, 88, 64, 106, 83, 93, 100] leverage the neural 3D representation for reconstructing the geometry and/or texture details.
Some methods obtain 3D human meshes from a single RGB [77] or RGBD [88] image by incorporating the SMPL template [80], which are ineffective for deformed human bodies.
Some approaches reconstruct continuous results from RGB [91] or RGBD [107] videos, skeletal motions [32], but are limited by garments types and cannot obtain facial expression details.
Other methods adopt the image-toimage translation pipeline to regress the 3D mesh via 2D estimations of intermediate textures [2], silhouettes [64], and depths [23], while some approaches jointly exploit 2D and 3D information, e.g. , body joints and per-pixel shading information [108], and 2D/3D poses and segmentation map [83].
These methods typically suffer from over-smoothed surfaces in occluded regions.
Recently, the pixel-aligned implicit function (PIFu) [73] has attracted much attention for 3D human reconstruction due to its effective implicit representation, and PIFuHD [74] estimates normal maps to reduce the geometry errors in the occluded regions based on PIFu.
Due to its success, many methods promote PIFu with voxel-alignment [104, 40, 37], deformation field [38, 41], real-time approach [51], illumination [3], monocular fusion [54], sparse-view temporal fusion [100], or apply it for point clouds based human reconstruction [13, 4, 62].
Typically, PIFu-based methods lack fine facial details in the sparse capture setting due to its all-in-one implicit 3D representation learning.
To tackle this problem, JIFF [9] employs the facial 3DMM [5] model as a shape prior, enhancing pixel-aligned features for detailed geometry information. However, their face reconstruction capacity is constrained by the 3DMM parameters, so that JIFF may not handle dense facial geometry well. In this paper, we propose the geometry-aware two-scale PIFu representation, which utilizes high-resolution RGB images to obtain the face geometric details and explicitly fuses the face and body occupancy fields for high-quality full-body reconstructed results.
Depth Image Denoising is essential for depth images captured from consumer-level depth sensors ( e.g. , Azure Kinect).
Traditional filtering-based methods [47, 10, 17, 71, 36, 76, 35] enhance depth data from various sensors.
Color or infrared images are used to help depth image denoising [16, 96, 56, 68, 57, 22, 95, 61, 35].
These methods typically assume the intensity image to be edge-aligned with the depth image, and they tend to produce artifacts when this assumption is violated.
Some methods [28, 75, 66, 15, 39, 11, 65, 31] fuse multiple frames to refine the depth images, but they tend to produce over-smoothed depth images.
Learning-based depth image denoising methods are proposed based on dictionary-learning [48] and deep-learning [27, 53, 52, 44, 93, 78].
Recently, Yan et al.
[93] propose the DDRNet for deep denoising using fused geometry and color images.
Their method tends to produce inconsistent results.
Sterzentsenko et al.
[78] propose a self-supervised method that leverages photometric supervision from a differentiable rendering model to smooth the depth noise.
However, the photometric supervision lacks 3D geometry information and their approach tends to produce homogeneous results.
In this paper, we propose to formulate depth denoising and 3D reconstruction as a multi-task learning problem.
The global topology information facilitates the depth denoising significantly.
Monocular Face Reconstruction methods aim to reconstruct personalized faces from monocular data.
The parametric face model, e.g. , 3D morphable model (3DMM) [6, 5, 72] or multi-linear blend shapes [8, 84, 25], are typically used in conventional methods.
However, these methods may not reconstruct accurate or dense facial geometry.
Deep-learning-based methods [42, 97, 21, 26] that incorporate face landmarks or the 3DMM model for face reconstruction also tend to produce coarse reconstructions results.
Other methods [82, 43, 21, 70, 103] estimate the facial dense shape instead of the low-dimensional template parameters.
All these approaches only reconstruct faces.
In this paper, we apply the PIFu representation and extend it to two scales for modeling human bodies and faces.
Our formulation enables producing vivid facial expressions with accurate body reconstructions.
Proposed Method
We propose the geometry-aware two-scale PIFu method, to address the problem of PIFu-based approaches that reconstruct flat facial and geometry-fallible body surfaces in the sparse capture setting.
First, to handle the noise issue of the sparse capture setting, we propose to formulate the depth denoising and the body reconstruction processes in the multi-task learning manner to exploit their complementary properties.
Second, although the face only occupies a small proportion of the whole
[Figure]
human model, it typically contains more high-frequency ( e.g. , vivid expressions) than other parts and plays a vital role in assessing the reconstruction fidelity.
To this end, we propose the two-scale PIFu representation to allocate more network capacity for face reconstruction.
As illustrated in Fig.
2, our network contains three parts: (1) the Geometry-aware PIFu-Body , F b that predicts the body occupancy field O b and the refined depth maps from the noisy RGBD images of N ( i.e. , three) perspective views; (2) the High-resolution PIFu-Face , F f which obtains the fine-grained face occupancy field O f , using refined depth map from F b and the high-resolution RGB image of the front view as inputs; (3) the light-weight Face-to-body Fusion , W that reconstructs the full human model by fusing the face and body occupancy fields ( i.e. , O b and O f ).
3.1 Geometry-aware PIFu-Body: F b
PIFu-based methods predict 3D occupancy fields in an implicit manner, enabling image-aligned features to be aware of global topological information, which helps suppress depth noise.
For example, in Fig.
3(e), the holes can be filled using only geometric supervision.
However, this depth map tends to be over-smoothed, and the reconstructed surface also lacks details (Fig. 3(f)).
On the other hand, with only depth supervision, image-to-image depth denoising can fill holes and add details.
However, the obtained depth contains incorrect details ( e.g. , face, hands, and clothing regions shown in Fig. 3(g) and Fig. 9(b)), when lacking of 3D geometric guidance.
Based on these observations, we formulate depth denoising and PIFu-based occupancy estimation in a multi-task learning manner.
It exploits the global topological information of the 3D occupancy field to guide the denoising process, and the local high-frequency details of refined depth to improve occupancy estimation (Fig. 3(h,i,j) and
Fig.
9(c,h)).
[Figure]
Back
Formulation of F b .
Given the triplet ( T ) of RGB ( I ), depth ( D ) images, body binary masks ( M ) from N perspective views, and the query point X ∈ R 3 as inputs, we formulate the F b to predict both the body occupancy value σ b ∈ [0 , 1] and the refined depth maps D rf , as:
[Equation]
where ψ g ( · ) is a mapping function that encodes T into multi-scale Geometry-aware features, x i =
π i ( X ) ∈ R 2 , is the 2D projection of point X at i -th view, and z i ∈ R is the depth of X in the local coordinate system of the i -th view.
C i ( X ) =
[ z i , p i ( X )] where p i ( X ) =
T ( z i − B
( D irf , x i )) ∈ [ − δ p , δ p ] , is the truncated PSDF value as in [100].
B ( · , x i ) is the sampling function to obtain pixel-aligned 2D features B ( ψ g ( T i ) , x i ) and depth information C i ( X ) , which are processed by the multi-view feature aggregation module A and further fed into the implicit function M b for occupancy querying.
Meanwhile, the decoder D ( · ) processes features ψ g ( T i ) for depth denoising.
Geometry-aware Features: ψ g ( T i ) .
The geometry-aware mapping ψ g ( · ) aims to exploit the complementary properties of depth denoising and occupancy field estimation.
Hence, it is expected to fuse the multi-modal RGB-D inputs effectively.
To handle their modal discrepancy, we use two independent HRNets [87] (Fig. 2) to process the RGB ( I ) and depth ( D ) inputs respectively, where { I , D } will be pre-processed to filter out the background via the masks, i.e. , M ( I , D ) .
To fuse the RGB and depth backbone features, we propose a novel Cross Attention Module (CAM) on the highest level of backbone features, and use the CAM output feature to guide the fusion of lower levels in a level-wise manner.
We also propose a novel Geometry Aware Module (GAM) to enrich the CAM output features with high-frequency information.
The enhanced features form the features ψ g ( T i ) .
[Figure]
Cross Attention Module (CAM) .
The CAM aims to fuse the RGB and depth features by computing their non-local correlations.
Since depth is typically noisier than RGB information, we first compute the self-attention map based on the RGB feature and then use it to reweight both RGB and depth features before they are fused.
The architecture of CAM is shown in Fig. 4, of which the implementation is based on the non-local model [89] but we extend it to handle the RGB-D fusion.
Specifically, the fused feature Y can be written as:
[Equation]
(2) where R r and R d represent two ResCBAMs
[90] used for calculating the local attentions for fusion, ∥ is the channel-wise concatenation and ⊗ denotes the matmul operation.
F r and F d indicate the
RGB and depth backbone features output by Layer
(Fig. 2).
κ ( F r ) computes the non-local feature affinity map as κ ( F r )
= softmax ( θ ( F r ) T ⊗ ϕ
( F r )) , where θ and ϕ are learnable linear embedding functions.
g r , g d are two functions to compute the value features of F r and F d .
[Figure]
Geometry Aware Module (GAM) .
The GAM aims to enrich the fused features of CAM with high-frequency information for reconstructing geometric details.
To this end, we propose to calculate the depth contrasted feature between a local region and its surroundings to capture highfrequency depth variations, as shown in Fig.
Specifically, we split out the RGB and depth features: F ′ r , F ′ d , according to the channels number, and calculate the contrasted feature for F ′ d as:
[Equation]
where f l denotes the local convolution with a 1x1 kernel and f g denotes the context convolution with a 3x3 kernel(dilate rate= x ).
G is a global average pooling operation.
∥ is the concatenation operation.
The second item after ∥ represents the local depth feature of the pixels relative to the global feature
F ′ d .
As a result, C d amplifies high-frequency signals at depth transitions, benefiting the prediction of these details ( e.g. , hairs in Fig. 1).
For F ′ r , we maintain its information through local convolutions.
3.2 High-resolution PIFu-Face: F f
Face regions typically have more high-frequency components than the body ( e.g. , mouth v.s. soft clothing) (Fig.
6(a,b,f)).
Even enhanced with depth denoising, the all-in-one PIFu ( F ) still struggles to represent high-frequency facial details ( e.g. , nose & mouth in Fig.
6(c,f)).
Hence, we propose to express the implicit function F in a piece-wise manner ( i.e. , F b and F f ) to reduce the complexity of joint occupancy estimation while producing vivid facial and body details.
Specifically, we propose to learn the high-frequency facial PIFu representation ( σ f ∈ [0 , 1] ) conditioned on the high-resolution face image ( I f cropped from the frontal view) and the corresponding denoised facial depth map D frf . [Figure]
Formulation of F f .
Given the above inputs (denoted as T f = { I f , D frf , M f } ), we formulate the
F f along the query point X f ∈ R 3 within the face regions as: [Equation]
[Equation]
where H f denotes the feature extractor for facial images T f .
The function U ↑ ( · ) up-samples T f to the same resolution as T .
M f is the implicit function for querying O f .
C f
( X f ) is defined the same as C i ( X ) , but we use the up-sampled masked facial depth M f ( U ↑ ( D frf )) to compute p f ( X ) .
Facial Points Selection .
To determine the facial points for inferring σ f , we select the points among all the querying points of which the projection x f ∈ R 2 falls inside the bounding-box R f of D frf and absolute PSDF value is less than α as X f (Fig. 7(b)).
We set a flag v f ( · ) to mark the facial points as:
[Equation]
else
[Equation]
3.3 Face-to-body Occupancy Fields Fusion: W
[Figure]
Simply merging the reconstructed face and body ( i.e. , replacing σ b with σ f for the facial points) would result in the discontinuity artifacts at the stitching (Fig. 6(e), 1st row).
To address this issue, we propose to fuse O b and O f via adaptive weights calculated in 3D space.
As shown in Fig.
6(d), we compute a 2D fusion weight map ( M ef ) in the x - y plane by eroding edges of the facial mask M f .
Along the z axis, we compute the weights through a Gaussian distribution model of the PSDF values.
Then, we formulate the joint probability distribution of the final fusion weight ω as:
[Equation]
where P ( · , · ) denotes the function to calculate the PSDF value, i.e. , P ( D frf , X f )
=
z f − B ( D f rf , x f ) and the parameter β is set to 1 e 3 in default.
In Eq.
6, the first term yields smaller values from the center of the face region to its boundaries, which improves the smoothness around the stitching.
The second term emphasizes the occupancy values computed by F f before and after the face surface.
Hence, we can leverage ω to fuse the σ b and σ f as: W ( σ b , σ f , ω )
= ω · σ f
+ (1 − ω ) · σ b .
3.4 Loss Function
We adopt the extended Binary Cross Entropy (BCE) loss [74] to supervise the predicted occupancy values σ b and σ f on the sampled body and facial points ˜X
= [ X b , X f ] , which can be written as:
[Equation]
where σ ∗ b , σ ∗ f are the ground-truth occupancy values for X b and X f .
S 0 and S 1 denote the sampled sets.
L B represents the BCE loss, µ 0 and µ 1 are the weights to balance PIFu-Body and PIFu-Face.
Regularization Term .
We propose a Regularization Loss ( L reg ) to reduce the artifacts in depthjumping regions during multi-views aggregation, as:
[Equation]
where S j ∈ S 0 is the set of points projected on the depth-jumping regions (refer to the supplemental for details of obtaining these regions).
The parameter ϵ is a small random uniform 3D perturbation.
n ( X b , T ) ∈ R 3 is the normal vector at X b , defined as ∇ X b F b ( X b , T ) / ∥∇ X b F b ( X b , T )
∥ 2 .
Eq.encourages the normals of X b to be consistent with those of the points sampled in its neighborhood, hence enhancing the smoothness along the stitching boundaries.
Depth Denoising Term .
We penalize the per-pixel difference between D rf and the rendered groundtruth depth map D gt .
We also penalize the error of calculated normal maps to prevent D rf from becoming blurred.
The loss for depth denoising is formulated as:
[Equation]
where d srf ( p ) denotes the s -scale (S=4, the scale of ψ g ( T ) )
predicted depth value of D rf in pixel p .
n rf ∈ R 3 is the normal vector of N rf in pixel p , where N rf is the normal map computed from D rf . d sgt ( p ) and n gt are the corresponding ground-truth value and vector.
L 1 and L 2 are the smooth Lloss and L 2 loss.
The weights ρ D , ρ N and λ s are used for balancing different loss terms.
The whole loss function can be defined as L =
L σ + λ reg · L reg + λ D · L D , where the weights λ reg ,
λ D are the balance terms.
Refer to the supplemental for detailed implementation information.
Experiments
Datasets and Evaluation Metrics .
We use the THuman2.0 [100] dataset which containshigh-quality 3D human scans to train and validate our network.
We split the dataset into training and test sets with a ratio of 4:1.
We rotate each scan along the yaw axis and render RGBD body portraits at every 6-degree rotation with CUDA acceleration.
For PIFu-Face, we pretrain F f using the FaceScape [94] dataset, which contains 3D head models of different people and expressions.
We crop the face regions and make the cropped meshes watertight (Fig. 6(a)).
For the input raw depth maps, following [20], we synthesize the sensor noise on D gt to obtain D .
For σ ∗ b and σ ∗ f , we follow the sampling strategy of PIFuHD [74] to sample body and facial points (Fig. 7(a)), and compute their occupancy values as the ground-truth labels.
We adopt the Point-to-Surface (P2S) distance(cm) and Chamfer distance(cm) for mesh, L 2 ( 1 e − 1 ) and
Cosine distance ( 1 e − 3 ) for normals as the metrics to measure the errors between the reconstructed and the ground-truth surfaces.
Lower metric values indicate better performance.
4.1 Comparisons with the State-of-the-arts
We compare our method with six state-of-the-arts human reconstruction methods, including the
Multi-view RGBD-PIFu [73], PIFuHD [74], StereoPIFu [40], IPNet [4], DoubleFusion [101] and
Function4D [100].
Among them, the DoubleFusion [101] and Function4D [100] are trackingbased, Multi-view RGBD-PIFu [73], PIFuHD (single RGB image) [74], StereoPIFu (stereo RGB images) [40] are PIFu-based, and the IPNet [4] is implicit function based.
[Figure]
[Figure]
Quantitative Comparisons.
Tab. 1 reports the quantitative comparisons on our test set between the proposed method and four PIFu (or implicit function)-based approaches.
Our method ranks in the first place under all metrics, exceeding the second-best results with a 16.72 % reduction in Chamfer and P2S distances, a 9.50 % reduction in L 2 and Cosine distances.
The
PIFuHD [74] reconstructs 3D human bodies from a single RGB image, which cannot handle the topology errors, resulting in the low reconstruction accuracy.
The IPNet [4] receives the fused point clouds as inputs, but it still cannot handle the significant noise issues in depths.
It also has to fit the SMPL, which may not handle the complex poses.
The StereoPIFu [40] can produce reasonable results from the front view but cannot handle the topological errors hidden in other perspectives.
Obtaining stereo pairs from multiple views may be a solution, but it significantly increases the computational cost introduced by its 3D voxel features.
For RGBD-PIFu [73], we use multi-view RGBDs as inputs to handle the topology errors.
However, it still fails to produce reliable results ( e.g. , artifacts in hairs, over-smoothed faces) when depth maps contain larger noise.
In contrast, our method achieves state-of-the-art performance by learning the geometry-aware two-scale PIFu representation.
Qualitative Comparisons.
Fig.
8 shows the reconstruction results of five existing methods and our method on our captured real data.
Although refined depths D irf can be fused ( e.g. , TSDF-Fusion in [66]) to obtain 3D human models (Fig. 8(b)), the reconstructed results contain large holes and lowquality regions due to sparse inputs and multi-view inconsistencies.
The multi-view RGBD-PIFu and the PIFuHD methods often suffer from floating geometry (Fig. 8(c)) and topology errors (Fig. 8(d)) due to non-negligible depth noise and the lack of other view information.
For IPNet, even if we take the fused point clouds (from 3 frames) as inputs, the topological errors are still obvious (Fig. 8(e)), and facial details are missing.
We also test the DoubleFusion [101] on our captured data.
As shown in Fig. 8(f), due to the low quality of the original depths and the sparse embed-graph of the whole body, this volumetric fusion-based approach tends to smooth out regions where we expect to see the high-frequency information ( e.g. , faces).
In addition, when a pose differs significantly from the initial pose, the reconstructed mesh tends to be over-stretched (blue boxes).
The Function4D [100] tracks the former and latter frames for the current frame and fuses the multi-frame point clouds to produce new depth maps.
When the motion changes are not small, their method is not easy to reconstruct reasonable high-frequency details (flat faces, hairs in Fig. 8(g)).
Our approach leverages the depth denoising to produce robust depth ( e.g. , hairs in Fig. 8(h)) for the PIFu-based reconstruction process.
Moreover, our two-scale PIFu represents the face and body separately to produce vivid details.
4.2 Ablation Study
[Figure]
Multi-task Formulation.
To evaluate the effectiveness of our multi-task design, we compare our geometry-aware PIFu with a sequential model that first performs the depth denoising and then the human reconstruction using two individual networks.
The two networks are trained individually.
For depth denoising, we use an ablation version of our full model by removing the
PIFu-body MLP and the occupancy supervision.
For human reconstruction, we remove the depth supervision and compute the trunc-TSDF from the input raw depth maps.
As shown in Tab. 3, the performance of the sequential pipeline is lower than our model on all five metrics, which illustrates that our geometry-aware features benefit both two tasks.
Besides, comparing the refined depths and normals of the sequential model (Fig. 9(b)) with our model (Fig. 9(c)), we find solely depth denoising tends to produce incorrect geometric details ( e.g. , error depths and normals), as shown in Fig.
9(b,d), which further results in geometric errors (Fig. 9(g)) in the reconstruction process.
[Figure] [Figure] Depth & Occupancy Supervision. Tab. 2(1st row) shows that by removing the depth denoising task, the performance drops by 14.84 % and 18.98 % in terms of Mesh and Normal metrics.
The decoder D ( · ) produces over-smoothed depths and the reconstructed results also lack details, as shown in Fig.
3(e,f) and Fig. 10(c).
On the other hand, by removing the 3D supervision, the denoised depth often incorporates incorrect details ( e.g. , face, hands and clothing regions); See Fig. 3(g) and Fig. 9(b).
This experiment further verifies the effectiveness of our multi-task formulation.
CAM & GAM.
The 2nd-to-4th rows of Tab.
2 evaluate our CAM and GAM modules for RGBD fusion and geometry-aware enrichment, respectively.
By replacing the CAM with the common combination of ASPP [12] and ResCBAM [90] ( i.e. , AR in Fig. 2) or removing the GAM, both the performances drop.
We also observe that using AR without GAM yields worse performance on average than removing the GAM.
Fig.
10(d,e,f) illustrates the differences, where we can see that these two modules help reconstruct the high-frequency details ( e.g. , clothes folds of Fig.
10(h,j)).
PIFu-Face: F f .
The visual comparisons in Fig.
6(c,f) show that the PIFu-Face, i.e. , F f significantly improves the ability of our method to reconstruct high-frequency facial details.
Face-to-body Fusion: W .
The visual comparison in Fig. 6(e) shows that our weighted face-to-body fusion can eliminate the artifacts generated by simply merging the reconstructed face and body.
Regularization loss: L reg .
Fig.
10(g) shows that without the proposed L reg , our method tends to produce jagged noise on stitched boundaries where depth may not be consistent. [Figure] e) (f) (g) (h) (i)
4.3 More Experimental Results
Geometry and Texture .
As shown in Fig. 11, we provide more of our 3d geometric models along with corresponding textured results. We can see that the textured results are of high quality and well aligned with our reconstructed geometries. Moreover, our method is able to handle the loose dressing and respond to the boundary of eyeglasses wearing to some extent (Fig.
11(c,d,e,f)).
[Figure]
[Figure]
Limitation for Hand Details .
As shown in Fig.
12, it remains challenging for our method to generate highquality hand details when the hand is occluded, or the hand pose is complex.
In our two-scale PIFu, we model the face surface from only the front view since the front image contains enough details for the face reconstruction.
In contrast, hands have more degrees of movement freedom, which makes them harder to locate and reconstruct independently via extending additional scales based on PIFu from only one view.
Combining hand parametric models with our two-scale PIFu may be a feasible solution for hand modeling.
Conclusion
In this paper, we propose the geometry-aware two-scale PIFu representation to reconstruct the digital human body with fine-grained facial details.
The first novelty is that we formulate the depth denoising and implicit occupancy estimation in a multi-task learning manner, to exploit their complementary properties.
The second novelty lies in that we formulate the two-scale PIFu via two MLPs to represent the face and body separately, to reduce the complexity of modeling high-frequency facial details.
Finally, a lightweight face-to-body fusion scheme fuses the face and body occupancy fields to generate reliable reconstruction results of high fidelity.
For the limitation of modeling high-quality hand details, we are interested in investigating this topic in the future and exploring a light-weight approach. Acknowledgments : We thank all the reviewers for their constructive comments. Weiwei Xu is supported by NSFC (No. 61732016).
This work was partially supported by a GRF grant from RGC of Hong Kong (Ref.: 11205620). This paper is supported by the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Yes, we discuss this in Sec. 4. and Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Yes, we discuss this in the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We explain the implementation details in the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All four metrics are computed as the average of 5 runs with different degrees of input depth noise, as stated in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] This information is provided in the supplemental material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] The THuman2.0 and FaceScape datasets that we used are encrypted to prevent unauthorized access.
We have submitted a request form and a request mail to obtain the datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] Our captured data contains personal information but with permission, as stated in the supplemental material.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]