R ECOVERING G EOMETRIC I NFORMATION WITH L EARNED T EXTURE P ERTURBATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Regularization is used to avoid overﬁtting when training a neural network; unfortunately, this reduces the attainable level of detail hindering the ability to capture high-frequency information present in the training data.
Even though various approaches may be used to re-introduce high-frequency detail, it typically does not match the training data and is often not time coherent.
In the case of network inferred cloth, these sentiments manifest themselves via either a lack of detailed wrinkles or unnaturally appearing and/or time incoherent surrogate wrinkles.
Thus, we propose a general strategy whereby high-frequency information is procedurally embedded into low-frequency data so that when the latter is smeared out by the network the former still retains its high-frequency detail.
We illustrate this approach by learning texture coordinates which when smeared do not in turn smear out the high-frequency detail in the texture itself but merely smoothly distort it.
Notably, we prescribe perturbed texture coordinates that are subsequently used to correct the over-smoothed appearance of inferred cloth, and correcting the appearance from multiple camera views naturally recovers lost geometric information.
I NTRODUCTION
[Figure]
Since neural networks are trained to generalize to unseen data, regularization is important for reducing overﬁtting, see e.g. Goodfellow et al.
(2016); Scholkopf & Smola (2001).
However, regularization also removes some of the high variance characteristic of much of the physical world.
Even though high-quality ground truth data can be collected or generated to reﬂect the desired complexity of the outputs, regularization will inevitably smooth network predictions.
Rather than attempting to directly infer highfrequency features, we alternatively propose to learn a low-frequency space in which such features can be embedded.
We focus on the speciﬁc task of adding highfrequency wrinkles to virtual clothing, noting that the idea of learning a low-frequency embedding may be generalized to other tasks.
Because cloth wrinkles/folds are high-frequency features, existing deep neural networks (DNNs) trained to infer cloth shape tend to predict overly smooth meshes Alldieck et al.
(2019a)
; Danˇeˇrek et al.
; Guan et al.
; Gundogdu et al.
Jin et al.
; Lahner et al.
; Natsume et al.
; Santesteban et al.
Wang et al.
; Patel et al.
Rather than attempting to amend such errors directly, we perturb texture so that the rendered cloth mesh appears to more closely match the ground truth.
See Figure 1.
Then given texture perturbations from at least two unique camera views, 3D geometry can be accurately reconstructed Hartley & Sturm (1997) to recover high-frequency wrinkles.
Similarly, for AR/VR applications, correcting visual appearance from two views (one for each eye) is enough to allow the viewer to accurately discern 3D geometry.
Our proposed texture coordinate perturbations are highly dependent on the camera view.
Thus, we demonstrate that one can train a separate texture sliding neural network (TSNN) for each of a ﬁnite number of cameras laid out into an array and use nearby networks to interpolate results valid for any view enveloped by the array.
Although an approach similar in spirit might be pursued for various lighting conditions, this limitation is left as future work since there are a great deal of applications where the light is ambient/diffuse/non-directional/etc.
In such situations, this further complication may be ignored without signiﬁcant repercussion.
R ELATED W ORK
Cloth: While physically-based cloth simulation has matured as a ﬁeld over the last few decades
Baraff & Witkin (1998); Baraff et al. (2003); Bridson et al.
; Selle et al.
(2008), datadriven methods are attractive for many applications.
There is a rich body of work in reconstructing cloth from multiple views or 3D scans, see e.g. Bradley et al.
(2008b)
; Franco et al.
; Vlasic et al.
More recently, optimization-based methods have been used to generate higher resolution reconstructions Huang et al.
(2015); Pons-Moll et al.
; Wu et al.
; Yang et al.
Some of the most interesting work focuses on reconstructing the body and cloth separately B˘alan &
Black (2008); Neophytou & Hilton (2014);
Yang et al.
; Zhang et al.
With advances in deep learning, one can aim to reconstruct 3D cloth meshes from single views.
A number of approaches reconstruct a joint cloth/body mesh from a single RGB image Alldieck et al.
(2019a;b);
Natsume et al.
; Onizuka et al.
Saito et al.
(2019; 2020), RGB-D image Yu et al.
(2019), or video Alldieck et al.
(2018a;b); Habermann et al.
; Xu et al.
To reduce the dimensionality of the output space, DNNs are often trained to predict the pose/shape parameters of human body models such as SCAPE Anguelov et al.
(2005) or SMPL Loper et al.
(2015) (see also Pavlakos et al.
Habermann et al.
; Natsume et al.
; Varol et al.
(2018) leverage predicted pose information to infer shape.
When only the garment shape is predicted, a number of recent works output predictions in UV space to represent geometric information as pixels
Danˇeˇrek et al.
; Jin et al.
; Lahner et al.
(2018), although others Gundogdu et al.
Santesteban et al.
; Patel et al.
(2020) deﬁne loss functions directly in terms of the 3D cloth vertices.
Wrinkles and Folds: Cloth realism can be improved by introducing wrinkles and folds.
In the graphics community, researchers have explored both procedural and data-driven methods for generating wrinkles De Aguiar et al. (2010);
Guan et al.
; Hahn et al.
; Müller & Chentanez (2010);
Rohmer et al. (2010); Wang et al. (2010).
Other works add real-world wrinkles as a postprocessing step to improve smooth captured cloth: Popa et al.
(2009) extracts the edges of cloth folds and then applies space-time deformations, Robertini et al.
(2014) solves for shape deformations directly by optimizing over all frames of a video sequence.
Recently, Lahner et al.
(2018) used a conditional
Generative Adversarial Network Mirza & Osindero (2014) to generate normal maps as proxies for wrinkles on captured cloth.
Geometry: More broadly, deep learning on 3D meshes falls under the umbrella of geometric deep learning , which was coined by Bronstein et al.
(2017) to characterize learning in non-Euclidean domains.
Scarselli et al.
(2008) was one of the earliest works in this area and introduced the notion of a Graph Neural Network (GNN) in relation to CNNs.
Subsequent works similarly extend the
CNN architecture to graphs and manifolds Boscaini et al.
; Maron et al.
; Masci et al.
; Monti et al.
Kostrikov et al.
(2018) introduces a latent representation that explicitly incorporates the Dirac operator to detect principal curvature directions.
Tan et al.
(2018) trains a mesh generative model to generate novel meshes outside an original dataset.
Returning to the speciﬁc application of virtual cloth, Jin et al.
(2020) embeds a non-Euclidean cloth mesh into a Euclidean pixel space, making it possible to directly use CNNs to make non-Euclidean predictions.
M ETHODS
We deﬁne texture sliding as the changing of texture coordinates on a per-camera basis such that any point which is visible from some stereo pair of cameras can be triangulated back to its ground truth position.
Other stereo reconstruction techniques can also be used in place of triangulation because the images we generate are consistent with the ground truth geometry.
See e.g. Bradley et al.
(2008a);
Hartley & Sturm (1997); Seitz et al.
3. P ER -V ERTEX D ISCRETIZATION
Since the cloth mesh is discretized into vertices and triangles, we take a per-vertex, not a per-point, approach to texture sliding.
Our proposed method (see Section 4.1) computes per-vertex texture coordinates on the inferred cloth that match those of the ground truth as seen by the camera under consideration.
Then during 3D reconstruction, barycentric interpolation is used to ﬁnd the subtriangle locations of the texture coordinates corresponding to ground truth cloth vertices.
This assumes linearity, which is only valid when the triangles are small enough to capture the inherent nonlinearities in a piecewise linear sense; moreover, folds and wrinkles can create signiﬁcant nonlinearity.
See
Figure 2.
3. O CCLUSION B OUNDARIES
Accurate 3D reconstruction requires that a vertex of the ground truth mesh be visible from at least two cameras and that camera projections of the vertex to the inferred cloth exist and are valid.
However, occlusions can derail these assumptions.
First, consider things from the standpoint of the inferred cloth.
For a given camera view, some inferred cloth triangles will not contain any visible pixels, and we denote a vertex as occluded when none of its incident triangles contain any visible pixels.
Although we do not assign perturbed texture coordinates to occluded vertices (i.e. they keep their original texture coordinates, or a perturbation of zero), we do aim to keep the texture coordinate perturbation function smooth (see Section 4.2).
In addition, there will be so called non-occluded vertices in the inferred cloth that do not project to visible pixels of the ground truth cloth.
This often occurs near silhouette boundaries where the inferred cloth silhouette is sometimes wider than the ground truth cloth silhouette.
These vertices are also treated as occluded, similar to those around the back side of the cloth behind the silhouette, essentially treating some extra vertices near occlusion boundaries as also being occluded.
See Figure 3a.
Next, consider things from the standpoint of the ground truth cloth.
For example, consider the case where all the cameras are in the front, and vertices on the back side of the ground truth cloth are not visible from any camera.
The best one can do in reconstructing these occluded vertices is to use the inferred cloth vertex positions; however, care should be taken near occlusion boundaries to smoothly taper between our texture sliding 3D reconstruction and the inferred cloth prediction.
A simple approach is to extrapolate/smooth the geometric difference between our texture sliding 3D reconstruction and the inferred cloth prediction to occluded regions of the mesh.
Once again, the deﬁnition of occluded vertices needs to be broadened for silhouette consideration.
Not only will vertices not visible from at least two cameras have to be considered occluded, but vertices that don’t project to the interior of an inferred cloth triangle with valid texture coordinate perturbations will also have to be considered occluded.
See Figure 3b.
Figure 2: Consider an extreme case, where the inferred cloth has a quite large triangle (shown in red).
That triangle should encompass the nonlinear texture region outlined in yellow (shown in pattern space).
Note: the yellow curve was generated by sampling the ground truth cloth’s texture coordinates along the projected edges of the red triangle.
The linearity assumption implied by barycentric interpolation instead uses the region outlined in green.
[Figure]
can fail near silhouettes of the inferred and ground truth cloth meshes, in which case smoothness assumptions are used (see Section 4.2).
In (a), inferred triangles with at least one vertex falling outside the silhouette of the ground truth mesh are colored red.
In (b), ground truth triangles with at least one vertex falling outside the silhouette of the inferred mesh are colored blue.
D ATASET G ENERATION
Let C
=
{ X, T } be a cloth triangulated surface with n vertices X ∈ R 3 n and texture coordinates
T ∈ R 2 n .
We assume that mesh connectivity remains ﬁxed throughout.
The ground truth cloth mesh
C G ( θ ) =
{ X G ( θ ) , T G } depends on the pose θ .
Given a pre-trained DNN (we use the network from
Jin et al.
(2020)), the inferred cloth C N ( θ ) =
{ X N ( θ ) , T G } is also a function of the pose θ .
Our objective is to replace the ground truth texture coordinates T G with perturbed texture coordinates
T N ( θ, v ) , i.e. to compute C (cid:48)
N ( θ, v ) =
{ X N ( θ ) , T N ( θ, v ) } where T N ( θ, v ) depends on both the pose θ and the view v .
Even though T N ( θ, v ) is in principle valid for all v using interpolation (see Section 6.3), training data T N ( θ, v p ) is only required for a ﬁnite number of camera views v p .
For each camera p , we also only require training data for ﬁnite number of poses θ k , i.e. we require T N ( θ k , v p ) , which is computed from T G using X G ( θ k ) , X N ( θ k ) , and v p .
4. T EXTURE C OORDINATE P ROJECTION
We project texture coordinates to the inferred cloth vertices X N ( θ k ) from the ground truth cloth mesh C G ( θ k ) using ray intersection.
For each inferred cloth vertex in X N ( θ k ) , we cast a ray from camera p ’s aperture through the vertex and ﬁnd the ﬁrst intersection with the ground truth mesh C G ( θ k ) ; subsequently, T G is barycentrically interpolated to the point of intersection and assigned to the inferred cloth vertex as its T N ( θ k , v p ) value.
See Figure 4.
Rays are only cast for inferred cloth vertices that have at least one incident triangle with a nonzero area subregion visible to camera p .
Also, a ground truth texture coordinate value is only assigned to an inferred cloth vertex when the point of intersection with the ground truth mesh is visible to camera p .
We store and learn texture coordinate displacements d v p ( θ k )
=
T N ( θ k , v p ) − T G .
After this procedure, any remaining vertices of the inferred cloth that have not been assigned d v p ( θ k ) values are treated as occluded and handled via smoothness considerations as discussed in Section 4.2.
4. O CCLUSION H ANDLING
Some vertices of the inferred cloth mesh remain unassigned with d v p ( θ k )
=
after executing the algorithm outlined in Section 4.1.
This creates a discontinuity in d v p ( θ k ) which excites high frequencies that require a more complex network architecture to capture.
In order to alleviate demands on the network, we smooth d v p ( θ k ) as follows.
First, we use the Fast Marching Method on triangulated surfaces Kimmel & Sethian (1998) to generate a signed distance ﬁeld.
Then, we extrapolate d v p ( θ k ) normal to the distance ﬁeld into the unassigned region, see e.g. Osher & Fedkiw (2002).
Finally, a bit of averaging is used to provide smoothness, while keeping the assigned values of d v p ( θ k ) unchanged.
Alternatively, one could solve a Poisson equation as in Cong et al.
(2015) while using the assigned d v p ( θ k ) as Dirichlet boundary conditions.
N ETWORK A RCHITECTURE
A separate texture sliding neural network (TSNN) is trained for each camera p ; thus, we drop the v p notation in this section.
The loss is deﬁned over all poses θ k in the training set
[Equation]
(1) to minimize the difference between the desired displacements d ( θ k ) and predicted displacements ˆ d ( θ k ) . The inferred cloth data we chose to correct are predictions of the T-shirt meshes from Jin et al.
(2020), each of which contains about 3,000 vertices.
The dataset spans about 10,000 different poses generated from a scanned garment using physically-based simulation, and includes texture coordinates for the garment mesh.
To improve the resolution, we up-sampled each cloth mesh by subdividing each triangle into four subtriangles.
Notably, our texture sliding approach can be used to augment the results of any dataset for which ground truth and inferred training examples are available.
Moreover, it is trivial to increase the resolution of any such dataset simply by subdividing triangles.
Note that perturbations of the subdivided geometry are unnecessary, as we merely desire more sample points (to address Figure 2).
Finally, we applied an 80-10-10 training-validation-test set split.
Similar to Jin et al.
, the displacements d ( θ k ) are stored as pixel-based cloth images for the front and back sides of the T-shirt, though we still output per-vertex texture coordinate displacements in UV space.
See Figure 5 for an overview of the network architecture.
Given input joint transformation matrices of shape 1 × 1 × 90 , TSNN applies a series of transpose convolution, batch normalization, and
ReLU activation layers to upsample the input to 512 × 512 × 4 .
The ﬁrst two dimensions of the output tensor represent the predicted displacements for the front side of the T-shirt, and the remaining two dimensions represent those for the back side.
E XPERIMENTS
In Section 6.1, we quantify the data generation approach of Section 4 and highlight the advantages of mesh subdivision for up-sampling.
In Section 6.2, we evaluate the predictions made by our trained texture sliding neural network (TSNN).
In Section 6.3, we demonstrate the interpolation of texture sliding results to novel views between a ﬁnite number of cameras.
Finally, in Section 6.4, we use multi-view texture sliding to reconstruct 3D geometry.
6. D ATASET G ENERATION AND E VALUATION
We aim to have the material coordinates of the cloth be in the correct locations as viewed by multiple cameras, so that the material can be accurately 3D reconstructed with point-wise accuracy.
As such, our error metric is a bit more stringent than that commonly used because our aim is to reproduce the actual material behavior, not merely to mimic its look (e.g. , by perturbing normal vectors to create shading consistent with wrinkles in spite of the cloth being smooth, as in Lahner et al.
In order to elucidate this, consider a two-step approach where one ﬁrst approximates a smooth cloth mesh and then perturbs that mesh to add wrinkles (similar to Santesteban et al.
In order to preserve area and achieve the correct material behavior, material in the vicinity of a newly forming wrinkle should slide laterally towards that wrinkle as it is formed.
Merely non-physically stretching the material in order to create a wrinkle may look plausible, but does not admit the correct material behavior.
In fact, the texture would be unrealistically stretched as well, although this is less apparent visually when using simple textures.
Since texture coordinates provide a proxy surface parameterization for material coordinates, we measure texture coordinate errors in a per-pixel fashion comparing between the ground truth and inferred cloth at the center of each pixel.
Figure 6a shows results typical for cloth inferred using the network from Jin et al.
(2020), and Figure 6b shows the highly improved results obtained on the same inferred geometry using our texture sliding approach (with 1 level of subdivision).
Note that the vast majority of the errors in Figure 6b occur near the wrinkles where the nonlinearities illustrated in
Figure 2 are most prevalent.
In Figure 6c, we deform the vertices of the inferred cloth mesh so that they lie exactly on the ground truth mesh in order to mimic a two-step approach (as discussed above).
Note how our error metric captures the still rather large errors in the material coordinates (and thus cloth vertex positions) in spite of the mesh in Figure 6c appearing to have the same wrinkles and folds as the ground truth mesh.
Figure 7 compares the local compression and extension energies of
[Figure]
match the in a
[Figure]
ground cloth mesh the ground truth mesh (Figure 7a), the inferred cloth mesh (Figure 7b), and the result of this two-step process (Figure 7c).
In spite of the untextured mesh in Figure 7c bearing visual similarity to the ground truth in Figure 7a, it still has rather large errors in deformation energy.
Figure 8 illustrates the efﬁcacy of subdividing the cloth mesh to get more samples for texture sliding.
The particular ground truth cloth wrinkle shown in Figure 8e is not captured by the inferred cloth geometry shown in Figure 8a.
The texture sliding result shown in Figure 8b better represents the ground truth cloth.
Figures 8c and 8d show how subdividing the inferred cloth mesh one and two times (respectively) progressively alleviates errors emanating from the linearity assumption illustrated in Figure 2.
Table 1 shows quantitative results comparing the inferred cloth to texture sliding with and without subdivision.
[Table]
[Figure]
6. N ETWORK T RAINING AND I NFERENCE
[Figure] [Figure]
The network was trained using the Adam optimizer Kingma & Ba (2014) with a 10 − 3 learning rate in PyTorch Paszke et al.
As mentioned earlier, we subdivided the mesh triangles once.
Figure 9 shows a typical prediction on a test set example, including the per-pixel errors in predicted texture coordinates.
While the TSNN is able to recover the majority of the shirt, it struggles near wrinkles.
Figure 10 highlights a particular wrinkle comparing the inferred cloth (Figure 10a) and the results of the TSNN before (Figure 10b) and after (Figure 10c) subdivision to the ground truth (Figure 10d).
Table 2 shows quantitative results comparing the inferred cloth to TSNN results with and without subdivision.
[Table]
[Figure]
is
6. I NTERPOLATING TO N OVEL V IEWS
Given a ﬁnite number of camera views v p one can specify a new view enveloped by the array using a variety of interpolation methods.
For the sake of demonstration we take a simple approach assuming that one can interpolate via v = (cid:80) p w p v p and then use these same weights to compute
T N ( θ k , v ) =
(cid:80) p w p T N
θ k , v p ) .
This same equation is also used for ˆ T N ( θ k , v ) .
Figure 11 shows the results obtained by linearly interpolating between two camera views.
Note how the largest errors appear near areas occluded by wrinkles where one (or both) of the cameras has no valid texture sliding results and instead uses the inferred cloth textures.
This can be alleviated by using more cameras placed closer together.
as a function of the chosen novel view.
6. 3D R ECONSTRUCTION
In order to reconstruct the 3D position of a vertex of the ground truth mesh, we take the usual approach of ﬁnding rays that pass through that vertex and the camera aperture for a number of cameras.
Then given at least two rays, one can triangulate a 3D point that is minimal distance from all the rays.
We can do this without solving the typical image to image correspondence problem because we know the ground truth texture coordinates for any given vertex.
Thus, we merely have to ﬁnd the ray that passes through the camera aperture and the ground truth texture coordinate for the vertex under consideration.
To ﬁnd a ground truth texture coordinate on a texture corrected inferred cloth mesh C (cid:48) N ( θ k , v ) , or ˆ C (cid:48)
N ( θ k , v ) , we ﬁrst ﬁnd the triangle containing that texture coordinate.
This can be done quickly by using a hierarchical bounding box structure where the base level boxes around each triangle are deﬁned using the min/max texture coordinates at the three vertices.
Then one can write the barycentric interpolation formula that interpolates the triangle vertex texture coordinates to obtain the given ground truth texture coordinate, and subsequently invert the matrix to solve for the weights.
These weights determine the sub-triangle position of the vertex under consideration (taking care to note that different answers are obtained in 3D space versus screen space, since the camera projection is nonlinear).
Figure 12 shows the 3D reconstruction of a test set example using texture sliding (Figure 12c) and the TSNN (Figure 12d).
To remove reconstruction noise generated by network inference
[Figure] [Figure]
errors in Figure 12d, we used the postprocess from Geng et al.
(2020); although, there are many other smoothing options in the literature that one might also consider.
Figure 13 compares the per-pixel errors and local compression/extension energies of Figures 12c and 12d.
D ISCUSSION AND F UTURE W ORK
There are many disparate applications for clothing including for example video games, AR/VR,
Hollywood special effects, virtual try-on and shopping, scene acquisition and understanding, and even bullet proof vests and soft armor.
Various scenarios deﬁne accuracy or ﬁdelity in vastly different ways.
So while it is typical to state that one cares about more than just the visual appearance (or “graphics”), often those aiming for predictive capability still make concessions.
For example, wherein
Santesteban et al.
(2019) proposes a network that well predicts wrinkles mapped to new body types, the discussion in Lahner et al.
(2018) implies that the horizontal wrinkles predicted by Santesteban et al.
(2019) are more characteristic of inaccurate physical simulation than real-world behavior.
Instead, Lahner et al.
(2018) strives for more vertical wrinkles to better match their data, but they accomplish this by predicting lighting to match an image while accepting overly smooth geometry.
And as we have shown in Figure 7c, predicting the correct geometry still allows for rather large errors in the deformation (see Geng et al.
In light of this, we state the problem of most interest to us: Our aim is to study the efﬁcacy of using deep neural networks to aid in the modeling of material behavior, especially for those materials for which predictive methods do not currently exist because of various unknowns including friction, material parameters (for cloth and body), etc.
Given this goal, we focus on the accurate prediction of material coordinates, which are a super set of deformation, geometry, lighting, visual plausibility, etc.
As demonstrated by the remarkably accurate 3D reconstruction in Figure 12c (see 13a), our approach to encoding high frequency wrinkles into lower frequency texture coordinates (i.e. texture sliding) works quite well.
It can be used as a post-process to any existing neural network to capture lost details (as long as ground truth and inferred training examples are available); moreover, we showed that trivial subdivision could be used to increase the sampling resolution to limit linearization artifacts.
One needs to take care when training the texture sliding neural network (TSNN) since inference errors can cause reconstruction noise.
Thus, as future work, we plan on experimenting with the network architecture, the size of the image used in the CNN, the smoothing methods near occlusion boundaries, the amount of subdivision, etc.
In addition, it would be interesting to consider more savvy multiview 3D reconstruction methods (particularly ones that employ DNNs; then, one might train the whole process end-to-end).