AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Structured representations such as keypoints are widely used in pose transfer,conditional image generation, animation, and 3D reconstruction.
However, theirsupervised learning requires expensive annotation for each target domain.
Wepropose a self-supervised method that learns to disentangle object structure fromthe appearance with a graph of 2D keypoints linked by straight edges.
Boththe keypoint location and their pairwise edge weights are learned, given only acollection of images depicting the same object class.
The graph is interpretable,for example, AutoLink recovers the human skeleton topology when applied toimages showing people.
Our key ingredients are i) an encoder that predicts keypointlocations in an input image, ii) a shared graph as a latent variable that links the samepairs of keypoints in every image, iii) an intermediate edge map that combines thelatent graph edge weights and keypoint locations in a soft, differentiable manner,and iv) an inpainting objective on randomly masked images.
Although simpler,
AutoLink outperforms existing self-supervised methods on the established keypointand pose estimation benchmarks and paves the way for structure-conditionedgenerative models on more diverse datasets.
Introduction
Object structure representations are widely used in modern computer graphics and computer visiontechniques, including keypoints for image generation [60, 61, 89] and skeletons for 3D reconstruction[22, 37, 111, 77, 91].
However, the structure is usually supervised on large annotated datasets [54, 3, 2,64] or via hand-crafted parametric models [58, 76, 102, 73, 4, 52].
Neither approach generalizes wellto new domains and both require additional manual annotation whenever more detail is needed [81].
Our goal is to reconstruct the keypoint locations of an object by learning from an unlabelled imagecollection, thereby sidestepping the generalization problem.
Our key idea is to leverage that the sameobject shares the same topology by introducing an explicit graph that links the same pairs of keypointsin all instances.
By contrast, existing self-supervised keypoint learning methods model objects asa set of independent parts.
Consistency over different instances of the same object is encouragedby either enforcing parts to follow hand-crafted image transformations [93, 115, 38, 59, 33, 55] orby adding implicit bias in the network architecture that encodes such spatial equivariances [31, 30].
Only [39, 85] use an explicit skeleton representation, but both require predeﬁned topology, rely onvideo input, and [39] is trained in a CycleGAN setting that still requires manually labeled examples.
We propose a simple yet effective method to learn both the keypoints and their links withoutsupervision in terms of a sparse graph serving two purposes.
First, the graph acts as a bottleneck that
[Figure]
au t oen c ode r t ha t t r an s f e r s po s e c ond i t i
ona l G A
N ( [Figure] ) applications can associates observations across training images.
We enforce the same topology across instances of the same class by learning a shared graph with a single set of edge weights.
In the absence of labels, we train AutoLink with only an autoencoder reconstruction objective.
Since the graph bottleneckshould not model appearance, we additionally feed the decoder with the input image after maskingthe majority of its pixels.
In turn, it is important for the disentangling that the heavily masked image models appearance without leaking structural information.
This is the case as inpainting methods are unable to infer the original image precisely from only sparse pixel colors [117, 112] unless conditioning on structure representations, such as edge maps [100, 68, 51, 49, 27].
Therefore, by forcing the autoencoder to reconstruct the original image, the detector converges to generate representative structures of images.
We demonstrate on 4 benchmarks that the trained detector has a signiﬁcantly improved keypointlocalization accuracy and on 6 additional datasets that it applies to a broader set of images spanningportraits, persons, animals, hands and ﬂowers, which we attribute to the explicit modeling of linksin the graph.
Figure 1 shows the diverse set of image domains it applies to, including challengingtextures and uncontrolled background, how both skeleton representations as well as object outlinesare learned by varying the number of keypoints, and exempliﬁes applications to controlled imagegeneration.
The code and the pre-trained models will be released upon publication.
Ethics - Risks.
The estimated keypoints and edges could be abused for deep fakes as the drivingsignal for generative models or for unwanted surveillance applications.
However, our method workstowards improved generality, including objects and animals, and does not improve upon supervisedmodels that already exist in high detail for humans.
Beneﬁts.
Since our method is entirely selfsupervised, it can be applied to a diverse set of persons, objects, animals, or situations that have notyet been labeled.
Related Work
Most representation learning methods focus on generic feature vectors for entire images to initializedeep networks for improved object classiﬁcation [98, 10, 72, 29, 28].
By contrast, our methodintroduces explicit object structure and we review the most related approaches in the following.
Self-supervised Keypoints Detection.
The most common idea to discover keypoints in an unsupervised manner is to rely on the notion that keypoints move as the image changes.
Various losses havebeen designed to enforce that the keypoints follow a known transformation. For instance, the transformation can be the view change in a multi-view recording [92, 80, 79] or the natural motion in videos[19, 86, 48, 67, 16, 46, 39].
When only single images are available, artiﬁcial image deformation isapplied, either from randomized [93, 115, 38, 59] or learned [97, 106] transformations within a predeﬁned deformation space that is usually tuned for each dataset.
However, the learned keypoints maymodel the background [115, 86] and struggle with large pose variation [33] as image deformations donot separate foreground from background and are bound.
Most models leverage multi-branch networkarchitectures to encode the structure and appearance separately and utilize multiple losses that needto be balanced.
By contrast, we do not apply artiﬁcial transformations, use a single branch, and asingle loss which eases and stabilizes training.
To overcome the need for artiﬁcial image deformation,
He et al.
[30, 31] exploit GANs to generate images along with corresponding keypoints and later usethem to train a detector.
However, this leads to even more complex network architectures and comeswith instabilities in GAN training, limiting their applicability to complex objects like human bodies.
Skeleton Representations.
Bone maps representing the keypoints connectivity are, for example,used in the supervised bottom-up approaches for human pose estimation. They learn an afﬁnityﬁeld from manual labels that connects joints with bone maps [8] or via explicit offsets [75].
Weuse a similar edge map representation but learn both the location and linking from scratch withoutannotations.
In the weakly-supervised setting, Jakab et al.
[39] exploit CycleGAN [118] to translatebetween image and edge maps.
The graph connectivity is predeﬁned to the human skeleton andedges are supervised by a large dataset of unpaired ground truth object edges, which can come from adifferent dataset but are manually annotated.
Schmidtke et al.
[85] overcome the manual labeling bydeforming a template skeleton.
However, they both require the known connectivity of the keypointsand require videos for training while ours learns both the keypoints and connectivity from a collectionof single images.
Noguchi et al.
[71] heuristically generate the skeleton by linking the centers of thelearned Signed Distance Field [63] for parts.
However, they require videos without the backgroundof the same object, and the learned skeleton does not generalize to other objects of the same class.
Object Sketch Learning.
Sketches are made of strokes drawn by a pen.
It is a concise and abstractrepresentation, which can be used in object recognition [110, 105] and image retrieval [94, 109, 104].
There are two common sketch representations used in neural models [103]. The ﬁrst one is ablack-white raster image [95, 84, 108], often used for image-to-image translation [36, 118, 74, 43]. Alternatively, a sketch can also be represented as a sequence of points (pen coordinates) [26, 20, 83],which is usually used by recurrent generation models [26, 11, 7, 42, 15, 24].
This graph representationis similar to ours.
However, instead of learning to mimic human drawings, ours directly predicts boththe keypoints and their connectivity on real natural images.
Structure-enhanced Image Inpainting.
When key parts are missing in an image, e.g., eyes onfaces or arms of humans, it is hard for inpainting networks to imagine the content accurately fromscratch.
Recently researchers try to use structural cues to guide the networks.
These methods usuallyhave two stages, where structure cues are detected in the ﬁrst stage, and images are generated basedon the structural cues in the second stage.
The structural cues can be edges [68, 51, 40, 101, 6],segmentation masks [116, 53, 27, 90], landmarks [49, 113, 107], low-frequency subband [96, 78],and foreground contours [100].
These structural cues are detected from the masked images andsupervised by the ground truth calculated from the original images.
Our reconstruction objectivecan be seen as such two-stage inpainting, but self-supervised and with the image edges replacedwith the learned graph edge representation.
Method
We leverage that the objects in the dataset share the same topology and can be represented as agraph that connects keypoints by a set of edges.
To learn the keypoints and edges, we design anautoencoder that aims to accurately reconstruct the input image, with the graph as the intermediate [Figure]
representation. To encode the input image into a graph, we detect the keypoints and create an edgeheatmap based on learnable edge weights.
To mostly obtain appearance information, we mask out themajority of the image, which reduces the structure information to a very low level.
The edge heatmapis combined with the masked image to reconstruct the original image.
Since the missing structure isimportant to reconstruct the original image, the network is forced to learn the structure of the objectin a self-supervised manner.
Figure 2 shows an overview of our method.
Formally, given an image I 2 R H ⇥ W ⇥ 3 with height H and width W we aim to learn a set of keypoints{ k i } Ki =1 ,
where k i
[  1 , 1] ⇥
[  1 , 1] ⇢ R 2 is the normalized keypoint coordinate, and K is the number of keypoints.
We use a ResNet with upsampling [99] to detect keypoints.
Afterward, we draw a differentiable edge [66] between each pair of keypoints.
The edge map S 2 R H ⇥ W is concatenated alongthe channel dimension with the randomly masked image I m 2 R H ⇥ W ⇥ 3 and fed into a UNet [82]to obtain the reconstructed image I 0 .
The detailed network architectures can be found in Appendix D.
3.1 Image Structure Representation
In this section, we introduce the generation of keypoints and the edge map from the image.
Let
H 2 R H ⇥ W ⇥ K be the K heatmaps generated by a ResNet with upsampling [99] from the image I .
The keypoint k
i is calculated by the differentiable soft-argmax function,
[Equation]
where p 2 [  1 , 1] ⇥ [  1 , 1] is the normalized pixel coordinates.
Given two keypoints k i , k j , a differentiable edge map [66] S ij is deﬁned as
[Equation]
where  is a hyperparameter controlling the thickness of the edge, and d ij ( p ) is the L 2 distancebetween the pixel p and the edge drawn by keypoints k i and k j ,
[Equation]
We assign a weight w ij > 0 to each edge, which is enforced to be positive by SoftPlus [18].
Thisweight is learned during training and shared across all object instances in a dataset.
Finally, we takethe maximum at each pixel of the heatmaps to obtain the ﬁnal edge map S 2 R H ⇥ W ,
[Equation]
Taking the maximum at each pixel avoids the entanglement of the edge weights and the convolutionkernel weights, which is further explained in Section 4.4. L a t e n t K e ypo i n t GAN [ 30 ] GAN S e g []
O u r s
O u r s ( e dg e s )
3.2 Image Reconstruction
The masked image I m is generated by ﬁrst uniformly dividing the image I into a 16 ⇥ 16 grid, andrandomly masking out 80% of the grid cells, similar to [28].
We concatenate the masked image withthe edge map and feed it into a UNet decoder [82] to reconstruct the original image,
[Equation]
(5) where  means concatenation along the channel dimension, and ↵ is a learnable parameter thatcompensates for the change of the edge weight magnitude during training.
↵ is initialized to 1.
Wefound this parameter to be helpful in training stability.
Different to [39], we have no ground truth forthe edge map.
Our edge map is an unobserved latent variable.
Thus we only minimize the differenceof the original image and the reconstructed image by the perceptual loss [41], [Equation]
[Equation]
where N is the number of examples and  is the feature extractor.
The perceptual loss is believed tomeasure the structure similarity [41, 23, 17], and leads to more robust training [38, 39].
3.3 Implementation Details
We use the Adam optimizer [47] with a learning rate of 10  4 with  1 = 0 .
9 ,  2 = 0 .
The batchsize is 64.
We train for 20k iterations.
It takes 3 hours to train on a single V100 GPU.
All images areresized to 128 ⇥ 128 .
The learning rate for the edge weights is multiplied by 512 due to the smallgradient of SoftPlus [18] when the value is close to 0.
To show the robustness of our model, wereport all experiments on the sampling strategy of masking 80% of the 16 ⇥ 16 patches.
We performexperiments with the same edge thickness of  2 = 5 e  5 for all benchmark datasets.
We train for 10times and report the mean and the standard deviation of the evaluation metrics.
Although it alreadyoutperforms other work in most experiments, we also tune thicknesses to each individual dataset, asothers did for their hyperparameters, which further improve the results.
The tuned thicknesses can befound in Appendix B.
The only other hyperparameter is the number of keypoints, which we rangefrom 4 to 32 and set to that of the established benchmarks for quantitative comparisons.
Experiments
In this section, we compare our results to the related methods, showing that our model is simple yet effective.
Besides, we perform a number of ablation studies on hyperparameters and algorithm variants,exhibiting the robustness of our model and justifying the necessity of every model component.
[Table] by [88] and the sign ? means being reported by [5].
The number of keypoints are K =for Human3.6m and DeepFashion and K = 10 for Taichi.
[Table]
4.1 Datasets and Evaluation Metrics
CelebA-aligned [57] contains 200k celebrity faces aligned in center.
We follow [93] splitting it intothree subsets: CelebA training set without MAFL (160k images), MAFL training set (19k), MAFLtest set (1k).
We train our network on the CelebA training set without MAFL.
To quantitativelyevaluate the consistency of our predicted keypoints, we follow [93] training a linear regressionwithout bias from our detected keypoints to the ground truth keypoints on the MAFL training set andreporting the mean L 2 error normalized by inter-ocular distance on the MAFL test set.
CelebA-in-the-wild [57] contains celebrity faces in unconstrained conditions.
We follow [33] andﬁrst split it into three subsets as for CelebA-aligned, and then remove the images where a face coversless than 30% of the area, which results in 45,609 images for model training, 5,379 with keypointlabels for regression, and 283 for testing.
The evaluation metric is the same as CelebA-aligned.
Human3.6m [35] contains human activity videos in static backgrounds.
We follow [115] consideringsix activities (direction, discussion, posing, waiting, greeting, walking), and using subjects 1, 5, 6, 7,8, 9 for training and 11 for testing.
This results in 796,648 images for training and 87,975 images fortesting.
The evaluation metric is the regressed (without bias) mean L 2 error normalized by the imagesize.
We remove the background as in [115, 59] to make a fair comparison to others.
To underlinethe robustness against structured backgrounds we also report the numbers including background.
DeepFashion [56] contains 53k in-shop clothes images.
We follow [59] only keeping the full-bodyimages.
We use 10604 images for training and 1179 images for testing as in [5].
We use the keypointsgenerated by AlphaPose [21] as the ground truth.
The evaluation metric is Percentage of Correct Keypoints of d=6 pixels in resolution 256 ⇥ 256 .
Taichi [87] contains 3049 training videos and 285 test videos of people performing Tai-Chi, withvarious appearance of foreground and background.
We follow [88] using 5000 and 300 images (notcontained in training data) for training a linear regression and for testing, respectively.
The evaluationmetric, mean average error (MAE), is calculated as the sum of the L 2 error in resolution 256 ⇥ 256 .
Flower [70], 11k Hands [1], Horses [118], and Zebras [118] are used for qualitative experiments.
VoxCeleb2 [13] and AFHQ [12] are used for pose transfer and conditional image generation,respectively.
Horses and Zebras are extracted from the CycleGAN dataset [118] by removing theimages with multiple horses and aligning them to face left.
The train/test split of Flower follows [9].
All the other datasets follow their ofﬁcial train/test split.
4.2 Qualitative Analysis
We qualitatively compare our detected keypoints with other methods and show the examples of thelearned edges in Figure 3.
For visualization purposes, we scale the edge weights to obtain visibleedges.
We use the same number of keypoints as the previous method [31] for a fair comparison,which are 8 for CelebA-in-the-Wild, 16 for DeepFashion, and 10 for Taichi.
We will discuss moreon the choice of the number of keypoints in Ablation Study 4.4.
As shown in Figure 3, our modelnot only detects consistent keypoints but also learns reasonable edges, such as human skeletons in
DeepFashion and Taichi.
For example, the feet are clearly connected with the corresponding knees,and there is no edge between the left and right hands.
We show 105 images with detected keypointsand visualized graph structure for each dataset in Figure 9-21 in the Appendix, demonstrating thatour model works on various classes of objects of diverse appearance and complex backgrounds.
4.3 Quantitative Analysis
We compare the keypoint detection results with other methods in Table 1 (CelebA), and Table 2(Human3.6, DeepFashion, Taichi).
Our simple model outperforms other methods in all benchmarks, With very few keypoints, the model only models a very basic shape, such as the box on the face, and sometimes cannot fully capture the structure.
For example, with K=4, only legs are modeled on humans. With abundant keypoints, it is able to model details. [Table] except for the most constrained setting CelebA-aligned where all methods perform well and theresults are comparable.
The results on CelebA in Table 1 conﬁrms that our model is more robustto poses in the wild.
Since self-supervised part segmentation methods are usually more robust onwild faces [55, 33, 31], we also include them for comparison, demonstrating the robustness of ourmodel over existing baselines.
The results on Human3.6m and DeepFashion in Table 2 show thecapability of our model to detect keypoints on human bodies of either similar or diverse appearances.
The results on Taichi in Table 2 demonstrates the general applicability of our model to human bodiesof diverse pose in complex backgrounds.
4.4 Ablation Tests
In this section, we analyze the hyperparameters and demonstrate the robustness of our model. Wealso discuss the possible variants of our model and show the superiority of our design. Number of Keypoints & Thickness . We show ablation test results on the different numbers ofkeypoints and edge thicknesses in Figure 4.
The exact numbers can be found in Table 4 in Appendix B. Our model shows very strong robustness to edge thickness. The accuracy remains state-of-the-artwhile the thickness of the edges varies by multiple orders of magnitude. On the other hand, with theincreasing number of keypoints the accuracy increases.
This is expected since more keypoints areable to capture structure in more detail, as shown in Figure 4.4.
Masking Strategy . In our standard setting, the image is divided into 16 ⇥ 16 patches and 80% ofthe patches are randomly masked. We investigate how the patch size and masking ratio affect themodel performance. Figure 5 shows that a too low masking ratio enables the network to directlyinfer the structure from the masked image which is undesired in our case.
In these cases, the networkwould not choose to infer a set of compact keypoints from the original image.
Figure 5 illustrates thatthe patch size cannot be too small ( 4 ⇥ 4 ) or too large ( 64 ⇥ 64 ).
Although in some cases, such as
CelebA-in-the-Wild, a different masking strategy gives better results (4.14 vs 5.24), we choose toreport a uniﬁed strategy that we mask 80% of 16 ⇥ 16 patches for simplicity and conciseness.
Variants of Edge Heatmap Generation .
Besides the heatmap generation described in Method 3, wetest four more ways to generate the edge maps: 1) we deﬁne the thickness as a globally learnable
[Figure]
parameter; 2) we learn each edge thickness independently as a learnable parameter; 3) we treat eachedge heatmap as an independent channel of the feature map, instead of making them a single channelheatmap with Equation 4; 4) instead of using the edge heatmap, we generate Gaussian heatmapsfor the keypoints and use Equation 4 to combine them in a single channel heatmap.
The results arelisted in Table 3. Overall, these variants have worse performance. Although in some cases, the modelhas slightly better results on speciﬁc datasets, the performance boost does not hold in general. Weobserve that with only keypoints without edges, the model may degenerate, as shown in Figure 7a.
Interestingly, assigning each edge a different channel performs worse than simply combining all edgesinto a single channel.
We believe it is caused by entangling the edge weights with the convolutionkernel weights.
As visualized in Figure 7b, there exists dummy edges that do not model the objectstructure.
In addition, we tried to remove the learnable ↵ in Equation 5, ﬁxing ↵ = 1 , but the overallperformance decreases as shown in Table 3.
Does Texture Matter?
As shown in Figure 1a, the horse and the zebra share similar shape structuresbut only one is textured.
The striped texture not having a signiﬁcant impact on the learned structureshows that our model primarily learns the structure instead of texture features.
What if the model is trained on the structured background?
We tested on Human3.6m withbackground, where all images are taken in a single room.
The error is 5.02.
As shown in Figure 7c,our model captures the entrance on the background.
It is expected since we assume the foregroundobject is structured.
If we apply spectral clustering [69] on the learned graph, the keypoints are clearlydivided into two clusters.
Limitations and Future Work
As shown before, if the background is highly structured, the keypoints will appear on the background.
We can use a post graph clustering to separate the object from the background.
Similar to previous 2Dself-supervised methods [59, 88, 31], our model cannot model occlusion well.
We show in Figure 7dleft that the occluded right arm becomes the back when the person turns to the left.
In addition, as forall other methods, the model cannot distinguish left and right side of the human body as shown in Figure 7d right.
We believe it is necessary to model the structure in 3D to solve these problems.
Conclusion
We presented a simple approach for learning a spatial graph representation from unlabelled imagecollections by reconstructing a masked images.
The crucial part is our learnable graph designmodeling the relationship between different keypoints.
It is simpler than existing alternatives andopens up a path for image understanding, image editing and learning 3D models from 2D images.