Original: E MPIRICAL S TUDY OF P RE - TRAININGA B ACKBONE FOR 3D H UMAN P OSE AND S HAPE E STIMATIONAnonymous authors Paper under double-blind review
Revised: R ETHINKING S ELF -S UPERVISED V ISUAL R EPRESENTATION L EARNING IN P RE - TRAINING FOR 3D H UMAN P OSE AND S HAPE E STIMATION

Original: A BSTRACT
Revised: A BSTRACT

Original: We empirically study unexplored, yet must-know baselines of pre-training a backbone for 3D human pose and shape estimation (3DHPSE).

Original: Recently, a few selfsupervised representation learning (SSL) methods have been reported to outperform the ImageNet classification pre-training for vision tasks such as object detection.
Revised: Recently, a few self-supervised representation learning (SSL) methods have outperformed the ImageNet classification pre-training for vision tasks such as object detection.

Original: However, its effects on 3DHPSE are open to question, whose target is fixed to a single class, the human.
Revised: However, its effects on 3D human body pose and shape estimation (3DHPSE) are open to question, whose target is fixed to a unique class, the human, and has an inherent task gap with SSL.

Original: In this regard, we inspect the effectiveness of SSLon 3DHPSE and investigate two other pre-training approaches that have received relatively less attention.
Revised: We empirically study and analyze the effects of SSL and compare it with other pre-training alternatives for 3DHPSE.

Original: They are 2D annotation-based pre-training and synthetic data pre-training. Similar to the motivation of SSL  to benefit from unlabeled data, they have potential advantages to exploit data with less data collection cost compared with real 3D data.
Revised: The alternatives are 2D annotation-based pre-training and synthetic data pre-training, which share the motivation of SSL that aims to reduce the labeling cost. They have been widely utilized as a source of weak-supervision or fine-tuning, but have not been remarked as a pre-training source.

Original: SSL methods underperform the conventional ImageNet classification pre-training on multiple 3DHPSE benchmarks by 7.7% on average.
Revised: SSL methods underperform the conventional ImageNet classification pre-training on multiple 3DHPSE benchmarks by 7.7% on average.

Original: In contrast, despite a much less amount of pre-training data, the 2D annotationbased pre-training improves accuracy on all benchmarks and shows faster convergence during fine-tuning.
Revised: In contrast, despite a much less amount of pre-training data, the 2D annotation-based pre-training improves accuracy on all benchmarks and shows faster convergence during fine-tuning.

Original: In the semi-supervised setting, the improvement increases up to 8.2%, while SSL decreases accuracy by 10.7%, and synthetic data pre-training shows 0.2% decreased accuracy compared with the classification pretraining.

Original: Our observations would make the community carefully think about the current SSL-based pre-training trend for 3DHPSE and diversify research on pretraining approaches.
Revised: Our observations challenge the naive application of the current SSL pre-training to 3DHPSE and relight the value of other data types in the pre-training aspect.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transferring the knowledge contained in one task and dataset to solve other downstream tasks ( i.e. , transfer learning) has proven very successful in a range of computer vision tasks (Girshick et al., 2014; Carreira & Zisserman, 2017; He et al., 2017).
Revised: Transferring the knowledge contained in one task and dataset to solve other downstream tasks ( i.e. , transfer learning) has proven very successful in a range of computer vision tasks (Girshick et al., 2014; Carreira & Zisserman, 2017; He et al., 2017).

Original: In practice, transfer learning is done by pretraining a backbone (He et al., 2016) on source data to learn better  representations for the target task.
Revised: In practice, transfer learning is done by pretraining a backbone (He et al., 2016) on source data to learn better visual representations for the target task.

Original: The ImageNet classification has been the de facto pre-training paradigm in computer vision, and the 3D human  pose and shape estimation1 (3DHPSE) literature has followed this.
Revised: The ImageNet classification has been the de facto pre-training paradigm in computer vision, and the 3D human body pose and shape estimation (3DHPSE) literature has followed this.

Original: Recently, self-supervised representation learning (SSL) has gained popularity in the community (Chen et al., 2020a;
Revised: Recently, self-supervised representation learning (SSL) has gained popularity in the interest of reducing labeling costs (Chen et al., 2020a;

Original: Grill et al., 2020;
Revised: Grill et al., 2020;

Original: He et al., 2020; Caron et al., 2020; H´enaff et al., 2021).
Revised: He et al., 2020; Caron et al., 2020; H´enaff et al., 2021).

Original: It pre-trains a backbone using unlabeled arbitrary object images and fine-tunes the backbone on target tasks.
Revised: SSL pre-trains a backbone using unlabeled arbitrary object images and fine-tunes the backbone on target tasks.

Original: MoCo (He et al., 2020) and DetCon (H´enaff et al., 2021) reported to surpass the ImageNet classification pre-training for downstream tasks like object detection and instance segmentation on arbitrary class objects.
Revised: MoCo (He et al., 2020) and DetCon (H´enaff et al., 2021) surpassed the ImageNet classification pre -training for downstream tasks like object detection and instance segmentation on arbitrary class objects.

Original: Motivated by them, PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) targeted asingle class, the human hand, and pre-trained a backbone on hand data without 3D labels.
Revised: Motivated by them, PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) targeted a human hand and pre-trained a backbone on hand data without 3D labels.

Original: They showed the accuracy improvement for 3D hand pose and shape estimation from the controlled setting (Zimmermann et al., 2019), compared with random initialization (no pre-training) and the ImageNet classification pre-training.
Revised: They showed the accuracy improvement for 3D hand pose and shape estimation from the controlled setting (Zimmermann et al., 2019), compared with random initialization (no pre-training) and the ImageNet classification pre-training.

Original: While the results of PeCLR and HanCo are promising for 3DHPSE, they require labeled data with bounding boxes of a hand. The results are still meaningful, since a bounding box label is more affordable option than a 3D label.
Revised: While the results of PeCLR and HanCo are promising for 3DHPSE, the amounts of labeled hand data, which is fine-tuning data, are significantly smaller ( ∼ 64K) than that of the commonly used labeled body data ( ∼ 480K).

Original: However, they also intrigue investigation and a comparison with other relatively inexpensive label-based pre-training.
Revised: Also, the total training (pre- [Figure] training&fine-tuning) time of the different approaches is not matched, which is critical to the final accuracy (He et al., 2019). Last, they require labeled data with bounding boxes of a hand.

Original: This paper questions the effectiveness of SSL  for 3DHPSEand investigates other pre-training approaches, and compares them with SSL.
Revised: This paper questions the effectiveness of SSL pre-training for 3DHPSE with thorough experimental comparisons with alternatives.

Original: The experiments are organized in two steps.
Revised: The experiments are organized in three steps.

Original: First, we[Figure] compare state-of-the-art SSL methods that pre-train on ImageNet with the ImageNet classification pre-training.
Revised: First, we compare state-of-the-art SSL methods that pre-train on ImageNet with the ImageNet classification pretraining.

Original: The motivation is to verify whether purely unlabeled data-based SSL can outperform the de facto paradigm of classification pre-training for 3DHPSE.

Original: Different from downstream tasks like object detection and instance segmentation, the SSL methods are outperformed by the classification pre-training in three 3DHPSE benchmarks with 7.7% margin on average.
Revised: Different from downstream tasks like object detection and instance segmentation, the SSL methods are outperformed by the classification pre-training in three 3DHPSE benchmarks with 7.7% margin on average.

Original: Interestingly, the accuracy of SSL is comparable to or even worse than the random initialization baseline.
Revised: Interestingly, the accuracy of SSL is comparable to or even worse than the random initialization baseline.

Original: The results imply that  representations learned by SSL could be detrimental for 3DHPSE.
Revised: The results imply that general visual representations learned by SSL could be detrimental to 3DHPSE.

Original: Second, we apply SSL methods on human data following PeCLR (Spurr et al., 2021) and
Revised: Second, we explore the reasons behind the current SSL methods’ disappointing performance in depth by contriving a new pre-training approach.

Original: HanCo (Zimmermann et al., 2021) and compare them with 2D annotation-based pre-training and synthetic data pre-training. 2D annotation-based pre-training and synthetic data pre-training are worth investigating in that they share the motivation of SSL, which is to benefit from data 2 with less collection cost. Acquiring real 3D data requires a specially equipped Motion Capture (MoCap) studio, which is an extremely expensive setup.
Revised: Modern SSL pre-training methods (Chen et al., 2020a; He et al., 2020) have two unfavorable factors on 3DHPSE; 1) they learn inconsistent representations for the same class instances (Khosla et al., 2020), which hinders learning high-level priors about a specific class, and 2) SSL pre-training has an instance-level learning characteristic, which has an inherent task gap with 3DHPSE that requires understanding of the fine-level semantic information, the human joints. We combined an SSL approach with 2D joint labels, which we call

Original: Real 2D  data  and synthetic3D data of humans are relatively inexpensive to obtain (Rong et al., 2019;
Revised: JointCon, to experimentally validate the two factors’ effects. Third, we apply SSL methods on human data following PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) and compare them with 2D annotation-based pre-training and synthetic data pre-training. 2D annotation-based pre-training and synthetic data pre-training are worth investigating in that they share the motivation of SSL, which is to benefit from data 1 with less collection cost (Rong et al., 2019;

Original: Patel et al., 2021).
Revised: Patel et al., 2021).

Original: Also, their pose and appearance diversity has clear advantages against MoCap data, whose poses and appearances are restricted and highly different from in-the-wild test inputs.

Original: If we can transfer knowledge about humans in real 2D datasets and synthetic 3D datasets to the 3DHPSE task, 3DHPSE can be brought closer to accurate inference on in-the-wild images.

Original: In our experiments on human data, 2D annotation-based pre-training shows the highest accuracy and the fastest convergence speed among different pre-training approaches.
Revised: In our experiments on human data, 2D annotationbased pre-training shows the highest accuracy and the fastest convergence speed among different pre-training approaches.

Original: Compared with the classification baseline, its final accuracy is increased by 3.1% on 3DPW (von Marcard et al., 2018) and 1.9% on Human3.6M (Ionescu et al., 2014).
Revised: Compared with the classification baseline, its final accuracy is increased by 3.1% on 3DPW (von Marcard et al., 2018) and 1.9% on Human3.6M (Ionescu et al., 2014).

Original: In 3DPW, the convergence speed is approximately 2 × faster.
Revised: In 3DPW, the convergence speed is approximately 2 × faster.

Original: In the semi-supervised setting, the accuracy improvement increases to 9.9% on 3DPW and 7.1% on Human3.6M. We assume rich pose and appearance information learned from the 2D pose data is the key to these improvements as expected.
Revised: In the semi-supervised setting, the accuracy improvement increases to 9.9% on 3DPW and 7.1% on Human3.6M. We assume rich pose and appearance information learned from the 2D pose data is the key to these improvements as expected.

Original: Synthetic data pre-training produces higher errors than the classification baseline.
Revised: Synthetic data pre-training produces higher errors than the classification baseline.

Original: We conjecture that a domain gap between real and synthetic data interrupts efficient transfer learning.
Revised: We conjecture that a domain gap between real and synthetic data interrupts efficient transfer learning.

Original: Finally, SSL on human data also underperforms the classification pre-training.
Revised: Finally, SSL on human data also underperforms the classification pre-training.

Original: The overall results of SSL suggest that the current stage of SSL may not be enough to benefit 3DHPSE, which essentially requires high-level understanding of human geometry.
Revised: The overall results of SSL suggest that the current stage of SSL may not be enough to benefit 3DHPSE, which essentially requires high-level understanding of human geometry.

Original: Our main empirical results are summarized in Figure 1.
Revised: Our main empirical results are summarized in Figure 1.

Original: The current SSL that pre-trains on unlabeled arbitrary object images is not effective for 3DHPSE.
Revised: The current SSL that pre-trains on unlabeled arbitrary object images is not effective for 3DHPSE.

Original: Despite the least amount of pre-training data, 2D annotation-based pre-training provides the best result.
Revised: Despite the least amount of pre-training data, 2D annotation-based pre-training provides the best result.

Original: We expect our findings tomake the community rethink about pre-training techniques for 3DHPSE.
Revised: This paper has two significant empirical contributions; 1) we provide novel experimental evidence and discussion points for people to rethink the naive application of SSL pre-training paradigm in 3DHPSE, and 2) we relight the value of other data types that have received relatively less attention in the pre-training aspect.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: 3D human pose and shape estimation.
Revised: 3D human pose and shape estimation.

Original: We focus on reviewing the body, not hand literature, where extensive works have been addressed for 3D pose and shape estimation from in-the-wild images.
Revised: We focus on reviewing the body, not hand literature, where extensive works have been addressed for 3D pose and shape estimation from in-the-wild images.

Original: HMR (Kanazawa et al., 2018) proposed an end-to-end trainable human mesh recovery system that introduced adversarial loss to leverage MoCap data without images.
Revised: HMR (Kanazawa et al., 2018) proposed an end-to-end trainable human mesh recovery system that introduced adversarial loss to leverage MoCap data without images.

Original: GraphCMR (Kolotouros et al., 2019b) designed a graph convolutional network that takes the rest pose human mesh and image features as input and predicts mesh vertex coordinates.
Revised: GraphCMR (Kolotouros et al., 2019b) designed a graph convolutional network that takes the rest pose human mesh and image features as input and predicts mesh vertex coordinates.

Original: SPIN (Kolotouros et al., 2019a) combined a neural network regressor (Kanazawa et al., 2018) and an iterative fitting framework (Bogo et al., 2016).
Revised: SPIN (Kolotouros et al., 2019a) combined a neural network regressor (Kanazawa et al., 2018) and an iterative fitting framework (Bogo et al., 2016).

Original: HKMR (Georgakis et al., 2020) leveraged hierarchical kinematic priors of a human body at a network design level.

Original: I2L-MeshNet (Moon & Lee, 2020) introduced a lixel -based 1D heatmap to estimate mesh vertex coordinates in a fully convolutional manner with reasonable computational cost.
Revised: I2L-MeshNet (Moon & Lee, 2020) introduced a lixel -based 1D heatmap to estimate mesh vertex coordinates.

Original: Pose2Mesh (Choi et al., 2020) proposed a graph convolutional network that recovers 3D human pose and mesh from a 2D human pose.
Revised: Pose2Mesh (Choi et al., 2020) proposed a graph convolutional network that recovers 3D human pose and mesh from a 2D human pose.

Original: Pose2Pose (Moon et al., 2022a) designed a joint-based regressor that overcomes limitations of the conventional global average pooling-based regressor, such as the inclusion of unnecessary features.
Revised: VIBE (Kocabas et al., 2020) and

Original: VIBE (Kocabas et al., 2020) extended HMR to video input.
Revised: TCMR (Choi et al., 2021) extended HMR to video input.

Original: METRO (Lin et al., 2021) improved GraphCMR (Kolotouros et al., 2019b) with a transformer architecture.
Revised: METRO (Lin et al., 2021) improved GraphCMR (Kolotouros et al., 2019b) with a transformer architecture.

Original: ROMP (Sun et al., 2021) introduced a bottom-up method for multi-person 3D mesh recovery.

Original: PARE (Kocabas et al., 2021) introduced a part-guided attention mechanism for mesh parameter regression.
Revised: PARE (Kocabas et al., 2021) introduced a part-guided attention mechanism for mesh parameter regression.

Original: PyMAF (Zhang et al., 2021a) used mesh-aligned  features to iteratively refine prediction.
Revised: PyMAF (Zhang et al., 2021a) used mesh-aligned image features to iteratively refine prediction.

Revised: 3DCrowdNet ( ? ) resolved the inter-person occlusion issue in crowded scenes with a joint-based regressor (Moon et al., 2022a) and 2D pose guidance for image features.

Original: Most of the current methods maintain a network architecture that comprises a backbone and a 3D mesh regressor.
Revised: Most of the current methods maintain a network architecture that comprises a backbone and a 3D mesh regressor.

Original: The backbones are mostly initialized with weights pre-trained by the ImagNet classification (He et al., 2016) during training.
Revised: The backbones are mostly initialized with weights pre-trained by the ImagNet classification (He et al., 2016) during training.

Original: Human dataset.
Revised: Human dataset.

Original: Human datasets for 3DHPSE can be broadly categorized into three types; Motion capture (MoCap) dataset, in-the-wild 2D dataset, and synthetic dataset.
Revised: Human datasets for 3DHPSE can be broadly categorized into three types; Motion capture (MoCap) dataset, in-the-wild 2D dataset, and synthetic dataset.

Original: MoCap datasets (Ionescu et al., 2014; Mehta et al., 2017) provide accurate 3D joint labels with images captured from a controlled multi-view studio.
Revised: MoCap datasets (Ionescu et al., 2014; Mehta et al., 2017) provide accurate 3D joint labels with images captured from a controlled multi-view studio.

Original: In-the-wild 2D datasets (Andriluka et al., 2014; Johnson & Everingham, 2010) contain manually annotated 2D labels, which are usually 2D joints.
Revised: In-the-wild 2D datasets (Andriluka et al., 2014; Johnson & Everingham, 2010) contain manually annotated 2D labels, which are usually 2D joints.

Original: MSCOCO
Revised: MSCOCO

Original: (Lin et al., 2014) is the most widely used dataset that has rich 2D annotations, including part segmentation and
Revised: (Lin et al., 2014) is the most widely used dataset that has rich 2D annotations, including part segmentation and

Original: DensePose (G ¨ uler et al., 2018).
Revised: DensePose (G ¨ uler et al., 2018).

Original: Synthetic datasets (Varol et al., 2017; Patel et al., 2021) render 3D human avatars on synthetic background images.
Revised: Synthetic datasets (Varol et al., 2017; Patel et al., 2021) render 3D human avatars on synthetic background images.

Original: Human poses from MoCap data and appearances from real scan data are exploited.
Revised: Human poses from MoCap data and appearances from real scan data are exploited.

Original: Recent works (Patel et al., 2021; Baradel et al.,  2021) have shown that fine-tuning on synthetic human data can improve accuracy on real-world benchmarks.
Revised: Recent works (Patel et al., 2021; Baradel et al., 2021; Cai et al., 2021) have shown that fine-tuning on synthetic human data can improve accuracy on real-world benchmarks.

Original: 3DPW (von Marcard et al., 2018) is an in-the-wild 3D human benchmark with 3D body pose and mesh annotations.
Revised: 3DPW (von Marcard et al., 2018) is an in-the-wild 3D human benchmark with 3D body pose and mesh annotations.

Original: Since few in-the-wild 3D human datasets exist, evaluation on 3DPW is the current best way to measure the performance of 3DHPSE methods on in-the-wild images.
Revised: Since few in-the-wild 3D human datasets exist, evaluation on 3DPW is the current best way to measure the performance of 3DHPSE methods on in-the-wild images.

Revised: The concurrent work of (Pang et al., 2022) provides experimental results on pre-training on three datasets (classification on ImageNet, 2D pose estimation on MPII (Andriluka et al., 2014) and

Revised: MSCOCO).

Revised: However, the effects of SSL and synthetic data are still unexplored.

Revised: Considering that

Revised: SSL has shown a powerful impact on other vision tasks, the extensive experiments and analysis on the current SSL methods distinguish our work from (Pang et al., 2022).

Original: Self-supervised representation learning.
Revised: Self-supervised representation learning.

Original: Recently, contrastive learning-based methods are showing state-of-the-art performance among self-supervised approaches.
Revised: Recently, contrastive learning-based methods are showing state-of-the-art performance among self-supervised approaches.

Original: The contrastive learning’s fundamental idea is to pull together an anchor and a “positive” sample in embedding space, and to push apart the anchor from many “negative” samples (Khosla et al., 2020).
Revised: The contrastive learning’s fundamental idea is to pull together an anchor and a “positive” sample in embedding space, and to push apart the anchor from many “negative” samples (Khosla et al., 2020).

Original: Since this strategy can be applied to unlabeled training data, assuming a positive pair from data augmentations of the same sample, the research community has endeavored to use the learned representation for downstream transfer tasks.
Revised: Since this strategy can be applied to unlabeled training data, assuming a positive pair from data augmentations of the same sample, the research community has endeavored to use the learned representation for downstream transfer tasks.

Original: In practice, a backbone is pre-trained on a large-scale classification dataset (Russakovsky et al., 2015; Mahajan et al., 2018) without using labels, and fine-tunedfor classification, object detection, or instance segmentation.
Revised: In practice, a backbone is pre-trained on a large-scale classification dataset (Rus- [Figure] sakovsky et al., 2015; Mahajan et al., 2018) without using labels, and fine-tuned classification, object detection, or instance segmentation.

Original: MoCo (He et al., 2020; Chen et al., 2020c) interpreted the contrastive learning as building a dynamic and large dictionary of embeddings with a queue and a moving-averaged key encoder.
Revised: MoCo (He et al., 2020; Chen et al., 2020c) interpreted the contrastive learning as building a dynamic and large dictionary of embeddings with a queue and a moving-averaged key encoder.

Original: SimCLR (Chen et al., 2020a;b) introduced a nonlinear projection layer and proved that augmentation during pre-training should be stronger than supervised learning.
Revised: SimCLR (Chen et al., 2020a;b) introduced a nonlinear projection layer and proved that augmentation during pre-training should be stronger than supervised learning.

Original: SwAV (Caron et al., 2020), SiamSiam (Chen & He, 2020), and BYOL (Grill et al., 2020) eliminated the requirements for negative samples, while maintaining the Siamese architecture.
Revised: SwAV (Caron et al., 2020), SiamSiam (Chen & He, 2020), and BYOL (Grill et al., 2020) eliminated the requirements for negative samples, while maintaining the Siamese architecture.

Original: DetCon (H´enaff et al., 2021) proposed a new contrastive objective that is based on unsupervised mask generation.
Revised: DetCon (H ´ enaff et al., 2021) proposed a new contrastive objective that is based on unsupervised mask generation.

Original: PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) are the recent works that adapted
Revised: PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) are the recent works that adapted

Original: SSL to 3D hand pose and shape estimation.
Revised: SSL to 3D hand pose and shape estimation.

Original: PeCLR proposed a contrastive objective equivari- [Figure] ant to geometric transformations ( e.g. , rotation and translation), which models the transformations in a latent vector level similar to (Rhodin et al., 2018) and NSD (Rhodin et al., 2019).
Revised: PeCLR proposed a contrastive objective equivariant to geometric transformations ( e.g. , rotation and translation), which models the transformations in a latent vector level similar to (Rhodin et al., 2018) and NSD (Rhodin et al., 2019).

Original: It showed accuracy improvements over the random initialization baseline, but trained on a small-scale dataset ( ∼ 32K) (Zimmermann et al., 2019) during fine-tuning, although more hand data, for example, MSCOCO (150K) and YT3D (47K), exists.
Revised: It showed accuracy improvements over the random initialization baseline, but trained on a small-scale dataset ( ∼ 32K) (Zimmermann et al., 2019) during fine-tuning, although more hand data, for example, MSCOCO (150K) and YT3D (47K), exists.

Original: HanCo applied MoCo (He et al., 2020) on a background-augmented unlabeled hand dataset, but the improvement was marginal and like PeCLR, experiments were done in the controlled setting, not the in-the-wild environment.
Revised: HanCo applied MoCo (He et al., 2020) on a background-augmented unlabeled hand dataset, but the improvement was marginal and like PeCLR, experiments were done in the controlled setting, not the in-the-wild environment.

Original:  C ONVENTION OF 3D HUMAN POSE AND SHAPE ESTIMATION
Revised:  C ONVENTION OF 3D HUMAN POSE AND SHAPE ESTIMATION

Original: Architecture.
Revised: Architecture.

Original: A 3DHPSE network consists of a backbone and a human mesh regressor as depicted in Figure 2.
Revised: A 3DHPSE network consists of a backbone and a human mesh regressor, as depicted in Figure 2.

Original: A backbone (He et al., 2016; Sun et al., 2019) extracts image features from a given human-centered image.
Revised: A backbone (He et al., 2016; Sun et al., 2019) extracts image features from a given human-centered image.

Original: The features are fed to a human mesh regressor that estimates a mesh defined by human models, such as SMPL (Loper et al., 2015).
Revised: The features are fed to a human mesh regressor that estimates a mesh defined by human models, such as SMPL (Loper et al., 2015).

Original: Pre-training.
Revised: Pre-training.

Original: Almost all 3DHPSE methods pre-train their backbones by ImageNet (Russakovsky et al., 2015) classification.
Revised: Almost all 3DHPSE methods pre-train their backbones by ImageNet (Russakovsky et al., 2015) classification.

Original: They take pre-trained weights from the open source ( e.g. , torchvision (Paszke et al., 2017)) and initialize their backbones with them in practice.
Revised: They take pre-trained weights from the open source ( e.g. , torchvision (Paszke et al., 2017)) and initialize their backbones with them in practice.

Original: Initializing a backbone’s weights with the ImageNet classification-pre-trained weights is known to expedite training convergence and to bring better accuracy than random initialization.
Revised: Initializing a backbone’s weights with the ImageNet classification-pre-trained weights is known to expedite training convergence and to bring better accuracy than random initialization.

Original: Fine-tuning.
Revised: Fine-tuning.

Original: During fine-tuning, a pre-trained backbone and a human mesh regressor are trained in an end-to-end manner. MoCapdataset and in-the-wild 2D datasets are used.
Revised: During fine-tuning, a pre-trained backbone and a human mesh regressor are trained in an end-to-end manner by mix-using MoCap and in-the-wild 2D datasets.

Original: 2D annotations of in-the-wild 2D datasets weakly supervise the 3D mesh prediction (Kanazawa et al., 2018), and the 3D pseudo-mesh labels generated from 2D joints (Pavlakos et al., 2019; Joo et al., 2021; Moon et al., 2022b) directly supervise the 3D output.
Revised: 2D annotations of inthe-wild 2D datasets weakly supervise the 3D mesh prediction (Kanazawa et al., 2018), and the 3D pseudo-mesh labels generated from 2D joints (Pavlakos et al., 2019; Joo et al., 2021; Moon et al., 2022b) directly supervise the 3D output.

Original: The mixed-use of the MoCap and the in-the-wild 2D datasets has enabled reasonable performance on in-the-wild 3DHPSE, despite the scarce in-the-wild 3D training data.
Revised: The mixed use of the MoCap and the in-the-wild 2D datasets has enabled reasonable performance on in-the-wild 3DHPSE, despite the scarce in-the-wild 3D training data.

Original:  E XPERIMENT 4. S ETTING
Revised:  E XPERIMENT 4. S ETTING

Original: We adopt ResNet-50 (He et al., 2016) as a backbone and an SMPL (Loper et al., 2015) mesh as an estimation target.
Revised: We adopt ResNet-50 (He et al., 2016) as a backbone and a SMPL (Loper et al., 2015) mesh as an estimation target.

Original: During fine-tuning on 3DHPSE, we mainly use PARE (Kocabas et al., 2021) as a human mesh regressor and Human3.6M (Ionescu et al., 2014) and
Revised: During fine-tuning on 3DHPSE, we mainly use PARE (Kocabas et al., 2021) as a human mesh regressor and Human3.6M (Ionescu et al., 2014) and

Original: MSCOCO (Lin et al., 2014) as training datasets.
Revised: MSCOCO (Lin et al., 2014) as training datasets.

Original: For the semi-supervised setting, we exploit 10% of Human3.6M and 10% of MSCOCO data.
Revised: For the semi-supervised setting, we exploit 10% of Human3.6M and 10% of MSCOCO data.

Original: 3DPW (von Marcard et al., 2018), Human36M (Ionescu et al., 2014), andMuPoTS-
Revised: 3DPW (von Marcard et al., 2018), Human36M (Ionescu et al., 2014), and

Original: [Table]
Revised: MuPoTS-3D (Mehta et al., 2018) are used as evaluation benchmarks. We report PA-MPJPE for 3DPW, Human36M, and 3DPCK for MuPoTS-3D following the convention. PA-MPJPE stands for the Procrustes-aligned mean per-joint position error in millimeters. 3DPCK indicates the percentage of correct 3D keypoints.

Original: Pre-training.
Revised: Pre-training.

Original: For the SSL pre-training on ImageNet (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021), we use the publicly released pre-trained ResNet50 weights.
Revised: For the SSL pre-training on ImageNet (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021), we use the publicly released pre-trained ResNet50 weights.

Original: SSL pre-training on ImageNet typically requires much larger computational costs to be effective than the classification baseline.
Revised: SSL pre-training on ImageNet typically requires much larger computational costs to be effective than the classification baseline.

Original: For pre-training on human datasets, we fix the pre-training length (total number of training epochs) at 140 epochs regardless of methods, which is longer than those of 3DHPSE adapted SSL papers (100 epochs) (Spurr et al., 2021; Zimmermann et al., 2021) and the same as that of (Xiao et al., 2018).
Revised: For pre-training on human datasets, we fix the pre-training length (total number of training epochs) at 140 epochs regardless of methods, which is longer than those of 3DHPSE adapted SSL papers (100 epochs) (Spurr et al., 2021; Zimmermann et al., 2021) and the same as that of (Xiao et al., 2018).

Original: To pre-train the proposed 2D pose-driven alternatives, we decay the learning rate by 10 × at 90 and 120 epochs, starting from the initial value of 10 −following (Xiao et al., 2018).[Table] MSCOCO part segmenetation est. DensePose est.
Revised: To pre-train the proposed 2D pose-driven alternatives, we decay the learning rate by 10 × at 90 and 120 epochs, starting from the initial value of 10 −following (Xiao et al., 2018).

Original: 2D pose est.

Original: Fine-tuning.
Revised: Fine-tuning.

Original: For fine-tuning, we train a 3DHPSE network for 120 epochs and decay the learning rate by 10 × at 90 epochs, which is originally 10 − 4 .
Revised: For fine-tuning, we train a 3DHPSE network for 120 epochs and decay the learning rate by 10 × at 90 epochs, which is originally 10 − 4 .

Original: This schedule empirically showed full convergence for various networks initialized with different pre-trained weights.
Revised: This schedule empirically showed full convergence for various networks initialized with different pre-trained weights.

Original: For the case of training from scratch ( i.e. , random initialization), following the spirit of (He et al., 2019), we extend the total training length to 240 epochs and decay the learning rate by 10 × at 180 epochs from 10 − 4 .
Revised: For the case of training from scratch ( i.e. , random initialization), following the spirit of (He et al., 2019), we extend the total training length to 240 epochs and decay the learning rate by 10 × at 180 epochs from 10 − 4 .

Original: It roughly matches the total training length of the pre-training counterparts (140 epochs + 120 epochs) and empirically showed full convergence.
Revised: It roughly matches the total training length of the pre-training counterparts (140 epochs + 120 epochs) and empirically showed full convergence.

Original: 4. P RE - TRAINING ON I MAGE N ET
Revised: 4. P RE - TRAINING ON I MAGE N ET

Original: We first examine the effects of the current state-of-the-art SSL methods (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021) that pre-train a backbone on ImageNet.
Revised: We first examine the effects of the current state-of-the-art SSL methods (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021) that pre-train a backbone on ImageNet.

Original: As summarized in Table 1 and Figure 3, when full fine-tuning data is used, they show approximately 2 × slower convergence and 7.7% higher errors (PA-MPJPE) than the ImageNet classification pre-training (He et al., 2016).
Revised: As summarized in Table 1 and Figure 3, when full fine-tuning data is used, they show approximately 2 × slower convergence and 7.7% higher errors (PA-MPJPE) than the ImageNet classification pre-training (He et al., 2016).

Original: Interestingly, the final accuracy is even worse than the random initialization baseline, though the convergence is faster.
Revised: Interestingly, the final accuracy is even worse than the random initialization baseline, though the convergence is faster.

Original: In the semi-supervised setting, SSL methods outperform the random initialization baseline by 2.5%, but still the classification pre-training provides the best overall accuracy.
Revised: In the semi-supervised setting, SSL methods outperform the random initialization baseline by 2.5%, but still the classification pretraining provides the best overall accuracy.

Original: We analyze the results that seemingly oppose the fact that SSL surpasses the random initialization and classification baselines in object detection and instance segmentation (Grill et al., 2020; H´enaff et al., 2021) by answering three questions.
Revised: We analyze the results that seemingly oppose the fact that SSL surpasses the random initialization and classification baselines in object detection and instance segmentation (Grill et al., 2020; H´enaff et al., 2021) by answering three questions.

Original: (i) Why SSL is worse than the random initialization baseline, when full fine-tuning data is used?
Revised: (i) Why SSL is worse than the random initialization baseline, when full fine-tuning data is used?

Original: We conjecture that a data domain gap is one of the reasons for the different results.
Revised: We conjecture that a data domain gap is one of the reasons for the different results.

Original: While both object detection and instance segmentation target localization of arbitrary class objects, 3DHPSE only targets a single class, the human.
Revised: While both object detection and instance segmentation target localization of arbitrary class objects, 3DHPSE only targets a single class, the human.

Original: For inference on arbitrary class objects, learning a wide range of general features unlimited to labels of a dataset could be advantageous in the generalization aspect (Tendle & Hasan, 2021).
Revised: For inference on arbitrary class objects, learning a wide range of general features unlimited to labels of a dataset could be advantageous in the generalization aspect (Tendle & Hasan, 2021).

Original: However, for 3DHPSE, a backbone network is preferred to learn more about human features rather than features of arbitrary objects, given the limited learning capacity.
Revised: However, for 3DHPSE, a backbone network is preferred to learn more about human features rather than features of arbitrary objects, given the limited learning capacity.

Original: Transferring knowledge about arbitrary objects could distract a network from learning necessary human features for 3DHPSE. (ii) Why the classification pre-training outperforms the random initialization baseline, while SSL does not, when full fine-tuning data is used?
Revised: Transferring knowledge about arbitrary objects could distract a network from learning necessary human features for 3DHPSE. (ii) Why the classification pre-training outperforms the random initialization baseline, while SSL does not, when full finetuning data is used?

Original: The classification pre-training makes a backbone  learn high-level semantic representations, such as the global structure of objects, that could be beneficially transferred to inference on humans.
Revised: The classification pre-training makes a backbone to learn high-level semantic representations, such as the global structure of objects, that could be beneficially transferred to inference on humans.

Original: For example, (Yosinski et al., 2015) showed that AlexNet (Krizhevsky et al., 2017) trained on ImageNet can recognize important features of human faces, although ImageNet has no labels of human faces.
Revised: For example, (Yosinski et al., 2015) showed that AlexNet (Krizhevsky et al., 2017) trained on ImageNet can recognize important features of human faces, although ImageNet has no labels of human faces.

Original: On the contrary, visual representations learned by SSL are likely to lack high-level information that can be found in objects with the same class.
Revised: On the contrary, visual representations learned by SSL are likely to lack high- [Table] trained on ImageNet by different pre-training methods. level information that can be found in objects with the same class.

Original: Instead, SSL focuses on learning representations that are invariant to data augmentations, which are performed on a single object sample.
Revised: Instead, SSL focuses on learning representations that are invariant to data augmentations, which are performed on a single object sample.

Original: These representations can be inconsistent over instances of the same class (Khosla et al., 2020).
Revised: These representations can be inconsistent over instances of the same class (Khosla et al., 2020).

Original: Considering that 3DHPSE essentially requires high-level priors that are shared across different human samples, inconsistent representations learned by SSL could be detrimental for 3DHPSE. (iii) Why SSL is effective in the semi-supervised setting, compared with the random initialization?
Revised: Considering that 3DHPSE essentially requires high-level priors that are shared across different human samples, inconsistent representations learned by SSL could be detrimental to 3DHPSE. (iii) Why SSL is effective in the semi-supervised setting, compared with the random initialization?

Original: Inconsistent representations learned by SSL could be a problem for 3DHPSE, but it does not indicate their representations are obsolete.
Revised: Inconsistent representations learned by SSL could be a problem for 3DHPSE, but it does not indicate their representations are obsolete.

Original: A convolutional neural network requires sufficient training data to extract meaningful low-level features ( e.g. , textures) from images.
Revised: A convolutional neural network requires sufficient training data to extract meaningful low-level features ( e.g. , textures) from images.

Original: In this perspective, insufficient training images deprive a network of learning high-level representations, which in turn harms 3DHPSE.
Revised: In this perspective, insufficient training images deprive a network of learning high-level representations, which in turn harms 3DHPSE.

Original: In such a circumstance, SSL can step in to provide a sufficient amount of training images to a network and to make it learn necessary features.
Revised: In such a circumstance, SSL can step in to provide a sufficient amount of training images to a network and to make it learn necessary features.

Revised: 4. A NALYSIS OF SSL FOR 3DHPSE.

Revised: In this section, we provide an in-depth investigation of the disappointing performance of SSL pretraining observed in the above experiments.

Revised: For the analysis, we devise a new pre-training approach,

Revised: JointCon, that combines the SSL approach with the 2D annotation-based approach.

Revised: The JointCon extracts joint-level features by sampling image features based on GT 2D joint locations, following

Original: 4. P RE - TRAINING ON A HUMAN DATASET
Revised: Pose2Pose (Moon et al., 2022a). We apply contrastive learning in the joint-level features to pull [Table] together an anchor with “positive samples” and push apart the anchor from “negative samples”.

Revised: We design three variations of JointCon as below: (a) JointCon(I) sets “positive samples” if they are from the same image and “negative samples” if they are from different images.

Revised: (b) JointCon(J) sets “positive samples” if they belong to the same joint label and “negative samples” if they belong to different joint labels.

Revised: Note that it treats joint-level features of the same joint class from different images as “positive samples”.

Revised: (c) JointCon(I+J) sets “positive samples” if they are from the same image and the same joint label and “negative samples” otherwise.

Revised: Table 2 shows that JointCon(J) outperforms the other variations in all three benchmarks.

Revised: The accuracy gap between JointCon(J) and JointCon(I+J) proves that it is important to learn consistent representations across instances of the same class.

Revised: Contrasting representations of the instances in the same class ( i.e. , objects or joints in the same class) hinder a backbone from learning useful highlevel priors and degrades the performance of 3DHPSE.

Original: {+As a result, the ImageNet classification pretraining rather learns consistent representations of the instances in the same class and achieves better accuracy than SSL, as shown in Table 
+}
Revised: {+As a result, the ImageNet classification pretraining rather learns consistent representations of the instances in the same class and achieves better accuracy than SSL, as shown in Table 
+}

Revised: The other important observation is that both JointCon(J) and JointCon(I+J) show better accuracy than JointCon(I).

Revised: The accuracy boost by only adding joint information to the SSL system demonstrates that the current SSL’s poor performance is not coming from architecture, data augmentation, or losses, but from the instance-level learning characteristic.

Revised: 4. P RE - TRAINING ON A HUMAN DATASET

Original: We investigate three pre-training approaches, SSL, 2D annotation-based pre-training, and synthetic data pre-training.
Revised: We investigate three pre-training approaches, SSL, 2D annotation-based pre-training, and synthetic data pre-training.

Original: First of all, given the results of Section 4.2, we apply SSL on a human dataset following (Spurr et al., 2021; Zimmermann et al., 2021).
Revised: First of all, given the results of Section 4.2, we apply SSL on a human dataset following (Spurr et al., 2021; Zimmermann et al., 2021).

Original: PeCLR (Spurr et al., 2021) that adapted
Revised: PeCLR (Spurr et al., 2021) that adapted

Original: SimCLR (Chen et al., 2020a;b) to a 3D hand pose and shape estimation, MoCo v2 (Chen et al., 2020c), the method used by HanCo (Zimmermann et al., 2021), and Swav (Caron et al., 2020), which is not based on contrastive learning, are experimented.
Revised: SimCLR (Chen et al., 2020a;b) to a 3D hand pose and shape estimation, MoCo v2 (Chen et al., 2020c), the method used by HanCo (Zimmermann et al., 2021), and Swav (Caron et al., 2020), which is not based on contrastive learning, are experimented.

Original: MSCOCO
Revised: MSCOCO

Original: (Lin et al., 2014) is used as a pre-training dataset.
Revised: (Lin et al., 2014) is used as a pre-training dataset.

Original: For synthetic data pre-training, we pre-train PARE (Kocabas et al., 2021) on AGORA (Patel et al., 2021) and SURREAL (Varol et al., 2017).
Revised: For synthetic data pre-training, we pre-train PARE (Kocabas et al., 2021) on AGORA (Patel et al., 2021) and SURREAL (Varol et al., 2017).

Original: For 2D annotation-based pretraining, we experiment with part segmentation estimation, DensePose (G¨uler et al., 2018) estimation, and 2D pose estimation on MSCOCO.
Revised: For 2D annotation-based pretraining, we experiment with part segmentation estimation, DensePose (G¨uler et al., 2018) estimation, and 2D pose estimation on MSCOCO.

Original: As shown in Table 2 and Figure 4, the SSL methods bring faster convergence, but the final accuracy becomes on par with the random initialization baseline.
Revised: As shown in Table 3 and Figure 4, the SSL methods bring faster convergence, but the final accuracy becomes on par with the random initialization baseline.

Original: It is also lower than the traditional classification pre-training.
Revised: It is also lower than the traditional classification pre-training.

Original: SSL appears to be effective only when significantly less fine-tuning data (10% Human3.6M + 10% MSCOCO, < 50 K) is used.
Revised: SSL appears to be effective only when significantly less fine-tuning data (10% Human3.6M + 10% MSCOCO, < 50 K) is used.

Original: The results coincide with the experimental results of PeCLR and HanCo; PeCLR was fine-tuned on ∼ 32K labeled data (Zimmermann et al., 2019) and HanCo was fine-tuned on ∼ 64K labeled data (Zimmermann et al., 2021).Synthetic data [Figure]
Revised: The results coincide with the experimental results of PeCLR and HanCo; PeCLR was fine-tuned on ∼ 32K labeled data (Zimmermann et al., 2019), and HanCo was fine-tuned on ∼ 64K labeled data (Zimmermann et al., 2021).

Original:  pre-training shows a similar tendency with SSL, though it outperforms SSL.
Revised: Synthetic data pre-training shows a similar tendency with SSL, though it outperforms SSL.

Original: On the contrary, 2D annotation-based pre-training improves accuracy against the random initialization and classification baselines on all benchmarks.
Revised: On the contrary, 2D annotation-based pre-training improves accuracy against the random initialization and classification baselines on all benchmarks.

Original: In the semi-supervised setting, the improvement increases up to 12.4% compared with thethe random initialization.
Revised: In the semi-supervised setting, the improvement increases up to 12.4% compared with the random initialization.

Original: The convergence speed is much faster than both random initialization and classification, especially in the semi-supervised setting.
Revised: The convergence speed is much faster than both random initialization and classification, especially in the semi-supervised setting.

Original: We think the results of SSL on human data support the statement in Section 4.2 that representations learned by SSL are hard to embed high-level information related to humans. 
Revised: We think the results of SSL on human data support the statement in Section 4.2 that representations learned by SSL are hard to embed high-level information related to humans. Synthetic data

Original: Synthetic data pre-training has potential to benefit from rich pose and appearance diversity, but a domain gap problem from highly different image appearances between synthetic and real images seems to remain.
Revised: [Table] pre-training has potential to benefit from rich pose and appearance diversity, but a domain gap problem from highly different image appearances between synthetic and real images seems to remain.

Original: Compared with SSL and synthetic data pre-training, 2D annotation-based pre-training appears to effectively transfer high-level priors of humans, such as body articulation, to 3DHPSE.
Revised: Compared with SSL and synthetic data pre-training, 2D annotation-based pre-training appears to effectively transfer high-level priors of humans, such as body articulation, to 3DHPSE.

Original: Well aligned 3D body pose and robustness to occlusion in Figure 5 verify the effectiveness of 2D annotationbased pre-training.
Revised: Well-aligned 3D body pose and robustness to occlusion in Figure 5 verify the effectiveness of 2D annotationbased pre-training.

Original: Figure 6 visualizes each human part’s feature activation of PARE (Kocabas et al., 2021) to further inspect how representations learned by each pre-training approach affect the human mesh regressor’s understanding of human geometry.
Revised: Figure 6 visualizes each human part’s feature activation of PARE (Kocabas et al., 2021) to further inspect how representations learned by each pre-training approach affect the human mesh regressor’s understanding of human geometry.

Original: In Table 3, we experiment with different human mesh regressors (Kolotouros et al., 2019a;

Original: Zhang et al., 2021a; Moon et al., 2022a; Moon & Lee, 2020) that have distinct architectures and achieved state-of-the-art accuracy recently.

Original: The best performing methods of SSL, synthetic data pre-training, and 2D annotation-based pre-training are used to compare.

Original: Overall, the same tendency is observed.

Original: SSL and synthetic data pre-training underperform the classification baseline, while 2D annotationbased pre-training outperforms it in multiple 3DHPSE benchmarks.

Original: This shows that our observations are not limited to a single 3DHPSE method, but can generalize to different methods.

Original: Only

Original: I2L-MeshNet (Moon & Lee, 2020) gives the different tendency, and the reason is discussed in Section D of Appendix.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our empirical observations propose that we should explore 2D annotation-based pre-training instead of SSL for 3DHPSE.
Revised: Our empirical observations propose that we should explore 2D annotation-based pre-training instead of SSL for 3DHPSE.

Original: However, SSL may still have an advantage over 2D annotation-based pretraining in the aspect of labeling costs.
Revised: However, SSL may still have an advantage over 2D annotation-based pre- [Figure] [Figure] training in the aspect of labeling costs.

Original: In this regard, we answer two questions that could help future 3DHPSE researchers to choose their pre-training strategies.
Revised: In this regard, we answer two questions that could help future 3DHPSE researchers to choose their pre-training strategies.

Original: (i) Does SSL entail a zero cost?
Revised: (i) Does SSL entail a zero cost?

Original: No.
Revised: No.

Original: It is not true in the aspect of data collection.
Revised: It is not true in the aspect of data collection.

Original: Collecting, cleaning, and curating data for SSL demand resources (Russakovsky et al., 2015; He et al., 2019;
Revised: Collecting, cleaning, and curating data for SSL demand resources (Russakovsky et al., 2015; He et al., 2019;

Original: Kotar et al., 2021).
Revised: Kotar et al., 2021).

Original: More importantly, the current SSL methods for 3DHPSE (Spurr et al., 2021;
Revised: More importantly, the current SSL methods for 3DHPSE (Spurr et al., 2021;

Original: Zimmermann et al., 2021) assume a human to be centered in an input image, which is an unrealistic setting for in-the-wild images without bounding box labels.
Revised: Zimmermann et al., 2021) assume a human to be centered in an input image, which is an unrealistic setting for in-the-wild images without bounding box labels.

Original: Thus, we should consider the labeling cost of bounding boxes when using SSL (Purushwalkam & Gupta, 2020; Goyal et al., 2021; ElNouby et al., 2021).
Revised: Thus, we should consider the labeling cost of bounding boxes when using SSL (Purushwalkam & Gupta, 2020; Goyal et al., 2021; ElNouby et al., 2021).

Original: Further discussion is provided in Section B of Appendix.
Revised: Further discussion is provided in Section B of Appendix.

Original: (ii) Is SSL efficient than 2D annotation-based pre-training?
Revised: (ii) Is SSL efficient than 2D annotation-based pre-training?

Original: No. SSL pre-trains on massive unlabeled data, which involves high computational costs.
Revised: No. SSL pre-trains on massive unlabeled data, which involves high computational costs.

Original: It often results in order of magnitude more[Table] [Figure] computation than the supervised counterparts (H´enaff et al., 2021).
Revised: It often results in order of magnitude more computation than the supervised counterparts (H ´ enaff et al., 2021).

Original: In addition, due to double feature encoders and complex feature contrasting mechanism, SSL takes more time than 2D annotationbased pre-training to pre-train on the same amount of pre-training data.
Revised: In addition, due to double feature encoders and complex feature contrasting mechanism, SSL takes more time than 2D annotationbased pre-training to pre-train on the same amount of pre-training data.

Original: Pre-training ResNet-50 (He et al., 2016) with PeCLR (Spurr et al., 2021) , MoCo v2 (Chen et al., 2020c), and the 2D pose estimator on MSCOCO took 110, 226, and 14 hours respectively, with the four RTX 2080Ti GPUs.
Revised: Pre-training ResNet-50 (He et al., 2016) with PeCLR (Spurr et al., 2021) , MoCo v2 (Chen et al., 2020c), and the 2D pose estimator on MSCOCO took 110, 226, and 14 hours respectively, with the four RTX 2080Ti GPUs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have investigated different approaches of pre-training a 3DHPSE backbone.
Revised: We have investigated different approaches for pre-training a 3DHPSE backbone.

Original: SSL, which has recently become the major trend in the community, has been thoroughly inspected.
Revised: SSL, which has recently become the major trend in the community, has been thoroughly inspected.

Original: 2D annotationbased pre-training and synthetic data pre-training have also been experimented, since they share the similar motivation of reducing labeling costs and transferring useful representations to 3DHPSE.
Revised: 2D annotationbased pre-training and synthetic data pre-training have also been experimented, since they share the similar motivation of reducing labeling costs and transferring useful representations to 3DHPSE.

Original: We experimented with multiple methods of each approach on multiple benchmarks to not draw a conclusion valid to a limited setting.
Revised: We experimented with multiple methods of each approach on multiple benchmarks to not draw a conclusion valid in a limited setting.

Original: Qualitative comparisons are provided to better understand how different pre-training approaches affect 3DHPSE.

Original: Finally, we verified that our observations on the three pre-training approaches are preserved regardless of a 3DHPSE mesh regressor.
Revised: Finally, we verified that our observations on the three pretraining approaches are preserved regardless of a 3DHPSE mesh regressor.

Original: Our empirical results show that 1) SSL is yet to replace the de facto paradigm of the ImageNet classification pretraining, and 2) 2D annotation-based pre-training can effectively transfer the knowledge to 3DHPSE, despite the least amount of pre-training data. 
Revised: Our empirical results show that 1) SSL is yet to replace the de facto paradigm of the ImageNet classification pre-training, and 2) 2D annotation-based pre-training can effectively transfer the knowledge to 3DHPSE, despite the least amount of pre-training data. Acknowledgements.

Revised: This work was supported in part by the IITP grants [No.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University), No.2022-0-00156, No.

Revised: 2021-0-02068, and No.2022-0-00156], and the NRF grant [No. 2021M3A9E4080782] funded by the Korea government (MSIT), and AIRS Company in Hyundai Motor Company & Kia Corporation through HMC/KIA-SNU AI Consortium Fund.

Revised: A PPENDIX

Original: We hope our findings, analysis, and discussion tobenefit the community.
Revised: In this supplementary material, we provide more experiments and discussions that could not be included in the main text due to the lack of pages. The contents are summarized below: A - Effects of SSL and 2D annotation-based pre-training on different amounts of data; InstaVariety(246K) (Kanazawa et al., 2019) and MPII(25K) (Andriluka et al., 2014) are experimented in addition to MSCOCO(150K) (Lin et al., 2014).

Revised: B - Discussion about labeling costs of SSL and 2D annotation-based pre-training C - Effects of different pre-training approaches, when fine-tuned on different sizes of

Revised: human

Revised: data

Revised: (30% and 60% of (Human3.6M (Ionescu et al., 2014) + MSCOCO (Lin et al., 2014)))

Revised: D - Effects of different human

Revised: mesh regressors

Original: A E FFECTS OF PRE - TRAINING ON DIFFERENT AMOUNTS OF DATA
Revised: A E FFECTS OF PRE - TRAINING ON DIFFERENT AMOUNTS OF DATA

Original: SSL is known to require large-scale pre-training data to be effective (He et al., 2020).
Revised: SSL is known to require large-scale pre-training data to be effective (He et al., 2020).

Original: In this regard, we apply PeCLR (Spurr et al., 2021) and MoCo v2 (Chen et al., 2020c) on InstaVariety(246K) (Kanazawa et al., 2019), which is 1 . 64 × bigger than MSCOCO(150K) (Lin et al., 2014).
Revised: In this regard, we apply PeCLR (Spurr et al., 2021) and MoCo v2 (Chen et al., 2020c) on InstaVariety(246K) (Kanazawa et al., 2019), which is 1 . 64 × bigger than MSCOCO(150K) (Lin et al., 2014).

Original: Interestingly, pre-training on InstaVariety consistently shows worse accuracy in all benchmarks as shown in Table 4.
Revised: Interestingly, pre-training on InstaVariety consistently shows worse accuracy in all benchmarks as shown in Table 4.

Original: If we use less fine-tuning data, the accuracy gap is increased, except PeCLR on
Revised: If we use less fine-tuning data, the accuracy gap is increased, except PeCLR on

Original: Human3.6M. We think the seemingly contradictory results come from the noisy bounding box used to crop a human from an image.
Revised: Human3.6M. We think the seemingly contradictory results come from the noisy bounding box used to crop a human from an image.

Original: The bounding box of a human in InstaVariety is calculated by measuring minimum and maximum x, y locations of OpenPose (Cao et al., 2017)’s
Revised: The bounding box of a human in InstaVariety is calculated by measuring minimum and maximum x, y locations of OpenPose (Cao et al., 2017)’s

Original: 2D pose estimation.
Revised: 2D pose estimation.

Original: Since the estimated values inevitably have errors, the bounding box obtained from them may not produce a human-centered image.
Revised: Since the estimated values inevitably have errors, the bounding box obtained from them may not produce a human-centered image.

Original: Thus, the input distribution could be severely different during pre-training and fine-tuning, which could lead to the counter-intuitive results.
Revised: Thus, the input distribution could be severely different during pre-training and fine-tuning, which could lead to the counter-intuitive results.

Original: We also provide the results of 2D pose estimation pre-training on MPII(25K)
Revised: We also provide the results of 2D pose estimation pre-training on MPII(25K)

Original: (Andriluka et al., 2014).
Revised: (Andriluka et al., 2014).

Original: Despite much less pre-training data, 2D pose estimation pre-training achieves comparable accuracy to that of pre-training MSCOCO.
Revised: Despite much less pre-training data, 2D pose estimation pre-training achieves comparable accuracy to that of pre-training MSCOCO.

Original: This shows the cost-effectiveness of 2D pose estimation pre-training, which is further discussed in the next section.
Revised: This shows the cost-effectiveness of 2D pose estimation pre-training, which is further discussed in the next section.

Original: [Table]
Revised: [Table]

Original: B D ISCUSSION ABOUT LABELING COSTS OF SSL AND 2D ANNOTATION - BASED PRE - TRAINING
Revised: B D ISCUSSION ABOUT LABELING COSTS OF SSL AND 2D ANNOTATION - BASED PRE - TRAINING

Original: Some could value SSL on bounding box-labeled human data more than 2D annotation-based pretraining, despite the experimental results of the main text.
Revised: Some could value SSL on bounding box-labeled human data more than 2D annotation-based pretraining, despite the experimental results of the main text.

Original: For example, assuming less fine-tuning data like the main text’s Table 6, MoCo v2 (Chen et al., 2020c) may be a more attractive option than the 2D pose estimation pre-training, considering that bounding box annotations take less cost than 2D pose annotations.
Revised: For example, assuming less fine-tuning data like the main text’s Table 6, MoCo v2 (Chen et al., 2020c) may be a more attractive option than the 2D pose estimation pre-training, considering that bounding box annotations take less cost than 2D pose annotations.

Original: In this context, we analyze the cost-effectiveness of different pre-training approaches by comparing the annotation cost in terms of annotation time.
Revised: In this context, we analyze the cost-effectiveness of different pre-training approaches by comparing the annotation cost in terms of annotation time.

Original: Table 5 supports that the 2D pose estimation pre-training is much more cost-effective, given the similar annotation time.
Revised: Table 5 supports that the 2D pose estimation pre-training is much more cost-effective, given the similar annotation time.

Original: The state-of-the-art object annotation paper (Papadopoulos et al., 2017) reported 7 seconds per one bounding box annotation.
Revised: The state-of-the-art object annotation paper (Papadopoulos et al., 2017) reported 7 seconds per one bounding box annotation.

Original: We assume that the same time would take for a human bounding box.
Revised: We assume that the same time would take for a human bounding box.

Original: Liu et al.
Revised: Liu et al.

Original: (Liu & Ferrari, 2017) reported 1.5 seconds per one human key point.
Revised: (Liu & Ferrari, 2017) reported 1.5 seconds per one human key point.

Original: Since a person in MPII (Andriluka et al., 2014) images have 14 key points, we can assume that the annotation time is 21 seconds per person.
Revised: Since a person in MPII (Andriluka et al., 2014) images have 14 key points, we can assume that the annotation time is 21 seconds per person.

Original: Cormier et al.
Revised: Cormier et al.

Original: (Cormier et al., 2021) reported 42.
Revised: (Cormier et al., 2021) reported 42.

Original: seconds per bounding box and pose of one person.
Revised: seconds per bounding box and pose of one person.

Original: We take the number of Cormier et al.
Revised: We take the number of Cormier et al.

Original: (Cormier et al., 2021) in Table
Revised: (Cormier et al., 2021) in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C F INE - TUNING ON DIFFERENT SIZES OF HUMAN DATA
Revised: C F INE - TUNING ON DIFFERENT SIZES OF HUMAN DATA

Original: We explore  effects of different pre-training approaches in different semi-supervised settings by varying sizes of fine-tuning data.
Revised: We explore the effects of different pre-training approaches in different semi-supervised settings by varying sizes of fine-tuning data.

Original: Experimental results of using 10%, 30%, 60%, and 100% fine-tuning data of (Human3.6M (Ionescu et al., 2014) + MSCOCO (Lin et al., 2014)) are shown in Tableand Figure 7.
Revised: Experimental results of using 10%, 30%, 60%, and 100% finetuning data of (Human3.6M (Ionescu et al., 2014) + MSCOCO (Lin et al., 2014)) are shown in Table 6 and Figure 7.

Original: SSL starts  surpass the random initialization baseline when fine-tuning data is reduced to 30%.
Revised: SSL starts to surpass the random initialization baseline when fine-tuning data is reduced to 30%.

Original: Synthetic data pre
Revised: Synthetic data pre

Original: -training also shows a similar tendency.
Revised: -training also shows a similar tendency.

Original: Only 2D annotation-based pre-training consistently outperforms the random initialization baseline, and the accuracy margin is enlarged as the fine-tuning data reduces.
Revised: Only 2D annotationbased pre-training consistently outperforms the random initialization baseline, and the accuracy margin is enlarged as the fine-tuning data reduces.

Original: D E FFECTS OF DIFFERENT HUMAN MESH REGRESSORS IN THE SEMI - SUPERVISED SETTING

Original: As shown in Table 7, the general tendency is similar to the full supervision setting in Table 3 of the main text.

Original: The notable result is that I2L-MeshNet (Moon & Lee, 2020) produces overall the same accuracy regardless of the pre-training approaches in both Table 3 and Table 7.

Original: We conjecture that the different tendency of I2L-MeshNet comes from its architecture and target representation.

Original: While other human mesh regressors estimate the SMPL (Loper et al., 2015) parameters with MLP layers at the end of their networks, I2L-MeshNet directly regresses 2.5D locations of mesh vertices in a fully convolutional manner.

Original: The 2.5D representation expresses the xy locations of mesh vertices in the image pixel space and the z location as root joint-relative depth.

Original: As a result, the accuracy of

Original: I2L-MeshNet depends on the segmentation of a human from a cropped human-centric image.

Original: This partially posits the problem of 3DHPSE to the foreground estimation, which may less demand on learning complex priors about humans, such as 3D human body articulation.

Original: [Table]
Revised: [Table]

Original: ImageNet (labeled) classification 57.
Revised: ImageNet (labeled) classification 57.

Original: MSCOCO (unlabeled)
Revised: MSCOCO (unlabeled)

Original: PeCLR (Spurr et al., 2021) 63.
Revised: PeCLR (Spurr et al., 2021) 63.

Original: MoCo v2 (Chen et al., 2020c) 60.
Revised: MoCo v2 (Chen et al., 2020c) 60.

Original: AGORA 3DHPSE 60.
Revised: AGORA 3DHPSE 60.

Original: SURREAL 57.
Revised: SURREAL 57.

Original: MSCOCO
Revised: MSCOCO

Original: DensePose est.
Revised: DensePose est.

Original: 2D pose est.
Revised: 2D pose est.

Original: H36M+
Revised: H36M+

Original: MSCOCO (10%) - random init.
Revised: MSCOCO (10%) - random init.

Original: ImageNet (labeled) classification 63.
Revised: ImageNet (labeled) classification 63.

Original: MSCOCO (unlabeled)
Revised: MSCOCO (unlabeled)

Original: PeCLR (Spurr et al., 2021) 71.
Revised: PeCLR (Spurr et al., 2021) 71.

Original: MoCo v2 (Chen et al., 2020c) 66.
Revised: MoCo v2 (Chen et al., 2020c) 66.

Original: AGORA 3DHPSE 64.
Revised: AGORA 3DHPSE 64.

Original: SURREAL 62.
Revised: SURREAL 62.

Original: MSCOCO
Revised: MSCOCO

Original: DensePose est.
Revised: DensePose est.

Original: 2D pose est.
Revised: 2D pose est.

Original: [Figure]
Revised: [Figure]

Revised: D E FFECTS OF DIFFERENT HUMAN MESH REGRESSORS

Revised: In Table 7 and Table 8, we experiment with different human mesh regressors (Kolotouros et al., 2019a;

Revised: Zhang et al., 2021a; Moon et al., 2022a; Moon & Lee, 2020) that have distinct architectures and achieved state-of-the-art accuracy recently.

Revised: The best-performing methods of SSL, synthetic data pre-training, and 2D annotation-based pre-training are used to compare.

Original: [Table]
Revised: Overall, the same tendency is observed. SSL and synthetic data pre-training underperform the classification baseline, while 2D annotation-based pre-training outperforms it in multiple 3DHPSE benchmarks. This shows that our observations are not limited to a single 3DHPSE method but can generalize to different methods. The notable result is that I2L-MeshNet (Moon & Lee, 2020) produces overall the same accuracy regardless of the pre-training approaches.

Revised: We conjecture that the different tendency of I2LMeshNet comes from its architecture and target representation.

Revised: While other human mesh regressors estimate the SMPL (Loper et al., 2015) parameters with MLP layers at the end of their networks,

Revised: I2L-MeshNet directly regresses 2.5D locations of mesh vertices in a fully convolutional manner.

Revised: The 2.5D representation expresses the xy locations of mesh vertices in the image pixel space and the z location as root joint-relative depth.

Revised: As a result, the accuracy of I2L-MeshNet depends on the segmentation of a human from a cropped human-centric image.

Revised: This partially posits the problem of 3DHPSE to the foreground estimation, which may less demand on learning complex priors about humans, such as 3D human body articulation.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]