Original: R ETHINKING S ELF -S UPERVISED V ISUAL R EPRESENTATION L EARNING IN P RE - TRAINING FOR 3D H UMAN P OSE AND S HAPE E STIMATION
Revised: R ETHINKING S ELF -S UPERVISED V ISUAL R EPRESENTATION L EARNING IN P RE - TRAINING FOR 3D H UMAN P OSE AND S HAPE E STIMATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, a few self-supervised representation learning (SSL) methods have outperformed the ImageNet classification pre-training for vision tasks such as object detection.
Revised: Recently, a few self-supervised representation learning (SSL) methods have outperformed the ImageNet classification pre-training for vision tasks such as object detection.

Original: However, its effects on 3D human body pose and shape estimation (3DHPSE) are open to question, whose target is fixed to a unique class, the human, and has an inherent task gap with SSL.
Revised: However, its effects on 3D human body pose and shape estimation (3DHPSE) are open to question, whose target is fixed to a unique class, the human, and has an inherent task gap with SSL.

Original: We empirically study and analyze the effects of SSL and further compare it with other pre-training alternatives for 3DHPSE.
Revised: We empirically study and analyze the effects of SSL and further compare it with other pre-training alternatives for 3DHPSE.

Original: The alternatives are 2D annotation-based pre-training and synthetic data pretraining, which share the motivation of SSL that aims to reduce the labeling cost.
Revised: The alternatives are 2D annotation-based pre-training and synthetic data pretraining, which share the motivation of SSL that aims to reduce the labeling cost.

Original: They have been widely utilized as a source of weak-supervision or fine-tuning, but have not been remarked as a pre-training source.
Revised: They have been widely utilized as a source of weak-supervision or fine-tuning, but have not been remarked as a pre-training source.

Original: SSL methods underperform the conventional ImageNet classification pre-training on multiple 3DHPSE benchmarks by 7.7% on average.
Revised: SSL methods underperform the conventional ImageNet classification pre-training on multiple 3DHPSE benchmarks by 7.7% on average.

Original: In contrast, despite a much less amount of pre-training data, the 2D annotation-based pre-training improves accuracy on all benchmarks and shows faster convergence during fine-tuning.
Revised: In contrast, despite a much less amount of pre-training data, the 2D annotation-based pre-training improves accuracy on all benchmarks and shows faster convergence during fine-tuning.

Original: Our observations challenge the naive application of the current SSL pre-training to 3DHPSE and relight the value of other data types in the pre-training aspect.
Revised: Our observations challenge the naive application of the current SSL pre-training to 3DHPSE and relight the value of other data types in the pre-training aspect.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transferring the knowledge contained in one task and dataset to solve other downstream tasks ( i.e. , transfer learning) has proven very successful in a range of computer vision tasks (Girshick et al., 2014; Carreira & Zisserman, 2017; He et al., 2017).
Revised: Transferring the knowledge contained in one task and dataset to solve other downstream tasks ( i.e. , transfer learning) has proven very successful in a range of computer vision tasks (Girshick et al., 2014; Carreira & Zisserman, 2017; He et al., 2017).

Original: In practice, transfer learning is done by pretraining a backbone (He et al., 2016) on source data to learn better visual representations for the target task.
Revised: In practice, transfer learning is done by pretraining a backbone (He et al., 2016) on source data to learn better visual representations for the target task.

Original: The ImageNet classification has been the de facto pre-training paradigm in computer vision, and the 3D human body pose and shape estimation (3DHPSE) literature has followed this.
Revised: The ImageNet classification has been the de facto pre-training paradigm in computer vision, and the 3D human body pose and shape estimation (3DHPSE) literature has followed this.

Original: Recently, self-supervised representation learning (SSL) has gained popularity in the interest of reducing labeling costs (Chen et al., 2020a;
Revised: Recently, self-supervised representation learning (SSL) has gained popularity in the interest of reducing labeling costs (Chen et al., 2020a;

Original: Grill et al., 2020;
Revised: Grill et al., 2020;

Original: He et al., 2020; Caron et al., 2020; H´enaff et al., 2021).
Revised: He et al., 2020; Caron et al., 2020; H´enaff et al., 2021).

Original: SSL pre-trains a backbone using unlabeled arbitrary object images and fine-tunes the backbone on target tasks.
Revised: SSL pre-trains a backbone using unlabeled arbitrary object images and fine-tunes the backbone on target tasks.

Original: MoCo (He et al., 2020) and DetCon (H´enaff et al., 2021) surpassed the
Revised: MoCo (He et al., 2020) and DetCon (H´enaff et al., 2021) surpassed the

Original: ImageNet classification pre
Revised: ImageNet classification pre

Original: -training for downstream tasks like object detection and instance segmentation on arbitrary class objects.
Revised: -training for downstream tasks like object detection and instance segmentation on arbitrary class objects.

Original: Motivated by them, PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) targeted a human hand and pre-trained a backbone on hand data without 3D labels.
Revised: Motivated by them, PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) targeted a human hand and pre-trained a backbone on hand data without 3D labels.

Original: They showed the accuracy improvement for 3D hand pose and shape estimation from the controlled setting (Zimmermann et al., 2019), compared with random initialization (no pre-training) and the ImageNet classification pre-training.
Revised: They showed the accuracy improvement for 3D hand pose and shape estimation from the controlled setting (Zimmermann et al., 2019), compared with random initialization (no pre-training) and the ImageNet classification pre-training.

Original: While the results of PeCLR and HanCo are promising for 3DHPSE, they have limited practical lessons.
Revised: While the results of PeCLR and HanCo are promising for 3DHPSE, they have limited practical lessons.

Original: For example, the amounts of labeled hand data, which is fine-tuning data, are significantly smaller ( ∼ 64K) than that of the commonly used labeled body data ( ∼ 480K).
Revised: For example, the amounts of labeled hand data, which is fine-tuning data, are significantly smaller ( ∼ 64K) than that of the commonly used labeled body data ( ∼ 480K).

Original: Also, the total training (pre-training&fine-tuning) time of the different
Revised: Also, the total training (pre-training&fine-tuning) time of the different

Original: [Figure]
Revised: [Figure]

Original: approaches is not matched, which is critical to the final accuracy (He et al., 2019).
Revised: approaches is not matched, which is critical to the final accuracy (He et al., 2019).

Original: Last, they require labeled data with bounding boxes of a hand.
Revised: Last, they require labeled data with bounding boxes of a hand.

Original: This paper questions the effectiveness of SSL pre-training for 3DHPSE by thoroughly comparing with alternatives in multiple aspects ( i.e. final accuracy, convergence speed, and cost-effectiveness).
Revised: This paper questions the effectiveness of SSL pre-training for 3DHPSE by thoroughly comparing with alternatives in multiple aspects ( i.e. final accuracy, convergence speed, and cost-effectiveness).

Original: We perform experiments by fixing the fine-tuning task to 3DHPSE and changing the pre-training approach.
Revised: We perform experiments by fixing the fine-tuning task to 3DHPSE and changing the pre-training approach.

Original: The experiments are organized in three steps.
Revised: The experiments are organized in three steps.

Original: First, we compare state-of-the-art SSL methods, pre-trained on ImageNet, with the ImageNet classification pre-training.
Revised: First, we compare state-of-the-art SSL methods, pre-trained on ImageNet, with the ImageNet classification pre-training.

Original: Different object detection and instance segmentation, the SSL methods are outperformed by the classification pretraining in three 3DHPSE benchmarks with 7.7% margin on average.
Revised: Different object detection and instance segmentation, the SSL methods are outperformed by the classification pretraining in three 3DHPSE benchmarks with 7.7% margin on average.

Original: Interestingly, the accuracy of SSL is comparable to or even worse than the random initialization baseline.
Revised: Interestingly, the accuracy of SSL is comparable to or even worse than the random initialization baseline.

Original: The results imply general visual representations learned by SSL could be detrimental to 3DHPSE.
Revised: The results imply general visual representations learned by SSL could be detrimental to 3DHPSE.

Original: Second, we explore the reasons behind the current SSL methods’ disappointing performance in depth by contriving a new pre-training approach.
Revised: Second, we explore the reasons behind the current SSL methods’ disappointing performance in depth by contriving a new pre-training approach.

Original: Modern SSL pre-training methods (Chen et al., 2020a; He et al., 2020) have two unfavorable factors on 3DHPSE; 1) they learn inconsistent representations for the same class instances as argued by (Khosla et al., 2020), which hinders learning high-level priors about a specific class, and 2) SSL pre-training has an instance-level learning characteristic ( i.e. , a single attribute per image), which has an inherent task gap with 3DHPSE that requires understanding of the fine-level semantic information (i.e., multiple attributes per image), the human joints.
Revised: Modern SSL pre-training methods (Chen et al., 2020a; He et al., 2020) have two unfavorable factors on 3DHPSE; 1) they learn inconsistent representations for the same class instances as argued by (Khosla et al., 2020), which hinders learning high-level priors about a specific class, and 2) SSL pre-training has an instance-level learning characteristic ( i.e. , a single attribute per image), which has an inherent task gap with 3DHPSE that requires understanding of the fine-level semantic information (i.e., multiple attributes per image), the human joints.

Original: We combine an SSL approach with 2D joint labels, which we call JointCon, to experimentally validate the two factors’ effects.
Revised: We combine an SSL approach with 2D joint labels, which we call JointCon, to experimentally validate the two factors’ effects.

Original: JointCon contrasts local image features of human joints instead of global image features of images.
Revised: JointCon contrasts local image features of human joints instead of global image features of images.

Original: Third, we compare SSL methods with 2D annotation-based pre-training and synthetic data pretraining on human data following PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021), and discuss cost-effectiveness in Section 5 and B. 2D annotation-based pre-training and synthetic data pre-training are worth investigating in that they share the motivation of SSL, which is to benefit from data 1 with less collection cost (Rong et al., 2019; Patel et al., 2021).
Revised: Third, we compare SSL methods with 2D annotation-based pre-training and synthetic data pretraining on human data following PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021), and discuss cost-effectiveness in Section 5 and B. 2D annotation-based pre-training and synthetic data pre-training are worth investigating in that they share the motivation of SSL, which is to benefit from data 1 with less collection cost (Rong et al., 2019; Patel et al., 2021).

Original: In our experiments on human data, 2D annotation-based pre-training shows the highest accuracy and the fastest convergence speed among different pre-training approaches.
Revised: In our experiments on human data, 2D annotation-based pre-training shows the highest accuracy and the fastest convergence speed among different pre-training approaches.

Original: Compared with the classification baseline, its final accuracy is increased by 3.1% on 3DPW (von Marcard et al., 2018) and 1.9% on Human3.6M (Ionescu et al., 2014).
Revised: Compared with the classification baseline, its final accuracy is increased by 3.1% on 3DPW (von Marcard et al., 2018) and 1.9% on Human3.6M (Ionescu et al., 2014).

Original: In 3DPW, the convergence speed is approximately 2 × faster.
Revised: In 3DPW, the convergence speed is approximately 2 × faster.

Original: In the semi-supervised setting, the accuracy improvement increases to 9.9% on 3DPW and 7.1% on
Revised: In the semi-supervised setting, the accuracy improvement increases to 9.9% on 3DPW and 7.1% on

Original: Human3.6M. We assume rich pose and appearance information learned from the 2D pose data is the key to these improvements as expected.
Revised: Human3.6M. We assume rich pose and appearance information learned from the 2D pose data is the key to these improvements as expected.

Original: Synthetic data pre-training produces higher errors than the classification baseline.
Revised: Synthetic data pre-training produces higher errors than the classification baseline.

Original: We conjecture that a domain gap between real and synthetic data inter- rupts efficient transfer learning.
Revised: We conjecture that a domain gap between real and synthetic data inter- rupts efficient transfer learning.

Original: Finally, SSL on human data also underperforms the classification pre-training.
Revised: Finally, SSL on human data also underperforms the classification pre-training.

Original: The overall results of SSL suggest that the current stage of SSL may not be enough to benefit 3DHPSE, which essentially requires high-level understanding of human kinematic structure.
Revised: The overall results of SSL suggest that the current stage of SSL may not be enough to benefit 3DHPSE, which essentially requires high-level understanding of human kinematic structure.

Original: Our main empirical results are summarized in Figure 1.
Revised: Our main empirical results are summarized in Figure 1.

Original: The current SSL that pre-trains on unlabeled arbitrary object images is not effective for 3DHPSE.
Revised: The current SSL that pre-trains on unlabeled arbitrary object images is not effective for 3DHPSE.

Original: Despite the least amount of pre-training data, 2D annotation-based pre-training provides the best result.
Revised: Despite the least amount of pre-training data, 2D annotation-based pre-training provides the best result.

Original: This paper has two significant empirical contributions; 1) we provide novel experimental evidence and discussion points for people to rethink the naive application of SSL pre-training paradigm in 3DHPSE, and 2) we relight the value of other data types that have received relatively less attention in the pre-training aspect.
Revised: This paper has two significant empirical contributions; 1) we provide novel experimental evidence and discussion points for people to rethink the naive application of SSL pre-training paradigm in 3DHPSE, and 2) we relight the value of other data types that have received relatively less attention in the pre-training aspect.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: 3D human pose and shape estimation.
Revised: 3D human pose and shape estimation.

Original: We focus on reviewing the body, not hand literature, where extensive works have been addressed for 3D pose and shape estimation from in-the-wild images.
Revised: We focus on reviewing the body, not hand literature, where extensive works have been addressed for 3D pose and shape estimation from in-the-wild images.

Original: HMR (Kanazawa et al., 2018) proposed an end-to-end trainable human mesh recovery system that introduced adversarial loss to leverage MoCap data without images.
Revised: HMR (Kanazawa et al., 2018) proposed an end-to-end trainable human mesh recovery system that introduced adversarial loss to leverage MoCap data without images.

Original: GraphCMR (Kolotouros et al., 2019b) designed a graph convolutional network that takes the rest pose human mesh and image features as input and predicts mesh vertex coordinates.
Revised: GraphCMR (Kolotouros et al., 2019b) designed a graph convolutional network that takes the rest pose human mesh and image features as input and predicts mesh vertex coordinates.

Original: SPIN (Kolotouros et al., 2019a) combined a neural network regressor (Kanazawa et al., 2018) and an iterative fitting framework (Bogo et al., 2016).
Revised: SPIN (Kolotouros et al., 2019a) combined a neural network regressor (Kanazawa et al., 2018) and an iterative fitting framework (Bogo et al., 2016).

Original: I2L-MeshNet (Moon & Lee, 2020) introduced a lixel -based 1D heatmap to estimate mesh vertex coordinates.
Revised: I2L-MeshNet (Moon & Lee, 2020) introduced a lixel -based 1D heatmap to estimate mesh vertex coordinates.

Original: Pose2Mesh (Choi et al., 2020) proposed a graph convolutional network that recovers 3D human pose and mesh from a 2D human pose.
Revised: Pose2Mesh (Choi et al., 2020) proposed a graph convolutional network that recovers 3D human pose and mesh from a 2D human pose.

Original: VIBE (Kocabas et al., 2020) and
Revised: VIBE (Kocabas et al., 2020) and

Original: TCMR (Choi et al., 2021) extended HMR to video input.
Revised: TCMR (Choi et al., 2021) extended HMR to video input.

Original: METRO (Lin et al., 2021) improved
Revised: METRO (Lin et al., 2021) improved

Original: GraphCMR (Kolotouros et al., 2019b) with a transformer architecture.
Revised: GraphCMR (Kolotouros et al., 2019b) with a transformer architecture.

Original: PARE (Kocabas et al., 2021) introduced a part-guided attention mechanism for mesh parameter regression.
Revised: PARE (Kocabas et al., 2021) introduced a part-guided attention mechanism for mesh parameter regression.

Original: PyMAF (Zhang et al., 2021a) used mesh-aligned image features to iteratively refine prediction.
Revised: PyMAF (Zhang et al., 2021a) used mesh-aligned image features to iteratively refine prediction.

Original: 3DCrowdNet (Choi et al., 2022) resolved the inter-person occlusion issue in crowded scenes with 2D pose guidance for image features and a joint-based regressor (Moon et al., 2022a).
Revised: 3DCrowdNet (Choi et al., 2022) resolved the inter-person occlusion issue in crowded scenes with 2D pose guidance for image features and a joint-based regressor (Moon et al., 2022a).

Original: Human dataset.
Revised: Human dataset.

Original: Human datasets for 3DHPSE can be broadly categorized into three types; Motion capture (MoCap) dataset, in-the-wild 2D dataset, and synthetic dataset.
Revised: Human datasets for 3DHPSE can be broadly categorized into three types; Motion capture (MoCap) dataset, in-the-wild 2D dataset, and synthetic dataset.

Original: MoCap datasets (Ionescu et al., 2014; Mehta et al., 2017) provide accurate 3D joint labels with images captured from a controlled multi-view studio.
Revised: MoCap datasets (Ionescu et al., 2014; Mehta et al., 2017) provide accurate 3D joint labels with images captured from a controlled multi-view studio.

Original: In-the-wild 2D datasets (Andriluka et al., 2014; Johnson & Everingham, 2010) contain manually annotated 2D labels, which are usually 2D joints.
Revised: In-the-wild 2D datasets (Andriluka et al., 2014; Johnson & Everingham, 2010) contain manually annotated 2D labels, which are usually 2D joints.

Original: MSCOCO
Revised: MSCOCO

Original: (Lin et al., 2014) is the most widely used dataset that has rich 2D annotations, including part segmentation and
Revised: (Lin et al., 2014) is the most widely used dataset that has rich 2D annotations, including part segmentation and

Original: DensePose (G¨uler et al., 2018).
Revised: DensePose (G¨uler et al., 2018).

Original: Synthetic datasets (Varol et al., 2017; Patel et al., 2021) render 3D human avatars on synthetic background images.
Revised: Synthetic datasets (Varol et al., 2017; Patel et al., 2021) render 3D human avatars on synthetic background images.

Original: Human poses from MoCap data and appearances from real scan data are exploited.
Revised: Human poses from MoCap data and appearances from real scan data are exploited.

Original: Recent works (Patel et al., 2021; Baradel et al., 2021; Cai et al., 2021) have shown that fine-tuning on synthetic human data can improve accuracy on real-world benchmarks.
Revised: Recent works (Patel et al., 2021; Baradel et al., 2021; Cai et al., 2021) have shown that fine-tuning on synthetic human data can improve accuracy on real-world benchmarks.

Original: 3DPW (von Marcard et al., 2018) is an in-the-wild human benchmark with 3D body pose and mesh annotations.
Revised: 3DPW (von Marcard et al., 2018) is an in-the-wild human benchmark with 3D body pose and mesh annotations.

Original: Since few in-the-wild 3D human datasets exist, evaluation on 3DPW is the current best way to evaluate 3DHPSE methods on in-the-wild images.
Revised: Since few in-the-wild 3D human datasets exist, evaluation on 3DPW is the current best way to evaluate 3DHPSE methods on in-the-wild images.

Original: The concurrent work of (Pang et al., 2022) provides experimental results on pre-training on three datasets (classification on ImageNet, 2D pose estimation on MPII (Andriluka et al., 2014) and
Revised: The concurrent work of (Pang et al., 2022) provides experimental results on pre-training on three datasets (classification on ImageNet, 2D pose estimation on MPII (Andriluka et al., 2014) and

Original: MSCOCO).
Revised: MSCOCO).

Original: However, the effects of SSL and synthetic data are still unexplored.
Revised: However, the effects of SSL and synthetic data are still unexplored.

Original: Considering that
Revised: Considering that

Original: SSL has shown a powerful impact on other vision tasks, the extensive experiments and analysis on the current SSL methods distinguish our work from (Pang et al., 2022).
Revised: SSL has shown a powerful impact on other vision tasks, the extensive experiments and analysis on the current SSL methods distinguish our work from (Pang et al., 2022).

Original: Self-supervised representation learning.
Revised: Self-supervised representation learning.

Original: Recently, contrastive learning-based methods are showing state-of-the-art performance among self-supervised approaches.
Revised: Recently, contrastive learning-based methods are showing state-of-the-art performance among self-supervised approaches.

Original: The contrastive learning’s fundamental idea is to pull together an anchor and a “positive” sample in embedding space, and to push apart the anchor from many “negative” samples (Khosla et al., 2020).
Revised: The contrastive learning’s fundamental idea is to pull together an anchor and a “positive” sample in embedding space, and to push apart the anchor from many “negative” samples (Khosla et al., 2020).

Original: Since this strategy can be applied to unlabeled training data, assuming a positive pair from data augmentations of the same sample, the research community has endeavored to use the learned representation for downstream transfer tasks.
Revised: Since this strategy can be applied to unlabeled training data, assuming a positive pair from data augmentations of the same sample, the research community has endeavored to use the learned representation for downstream transfer tasks.

Original: In practice, a backbone is pre-trained on a large-scale classification dataset (Russakovsky et al., 2015; Mahajan et al., 2018) without using labels, and fine-tuned for classification, object detection, or instance segmentation.
Revised: In practice, a backbone is pre-trained on a large-scale classification dataset (Russakovsky et al., 2015; Mahajan et al., 2018) without using labels, and fine-tuned for classification, object detection, or instance segmentation.

Original: [Figure]
Revised: [Figure]

Original: MoCo (He et al., 2020; Chen et al., 2020c) interpreted the contrastive learning as building a dynamic and large dictionary of embeddings with a queue and a moving-averaged key encoder.
Revised: MoCo (He et al., 2020; Chen et al., 2020c) interpreted the contrastive learning as building a dynamic and large dictionary of embeddings with a queue and a moving-averaged key encoder.

Original: SimCLR (Chen et al., 2020a;b) introduced a nonlinear projection layer and proved that augmentation during pre-training should be stronger than supervised learning.
Revised: SimCLR (Chen et al., 2020a;b) introduced a nonlinear projection layer and proved that augmentation during pre-training should be stronger than supervised learning.

Original: SwAV (Caron et al., 2020), SiamSiam (Chen & He, 2020), and BYOL (Grill et al., 2020) eliminated the requirements for negative samples, while maintaining the Siamese architecture.
Revised: SwAV (Caron et al., 2020), SiamSiam (Chen & He, 2020), and BYOL (Grill et al., 2020) eliminated the requirements for negative samples, while maintaining the Siamese architecture.

Original: DetCon (H´enaff et al., 2021) proposed a new contrastive objective that is based on unsupervised mask generation.
Revised: DetCon (H´enaff et al., 2021) proposed a new contrastive objective that is based on unsupervised mask generation.

Original: PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) are the recent works that adapted
Revised: PeCLR (Spurr et al., 2021) and HanCo (Zimmermann et al., 2021) are the recent works that adapted

Original: SSL to 3D hand pose and shape estimation.
Revised: SSL to 3D hand pose and shape estimation.

Original: PeCLR proposed a contrastive objective equivariant to geometric transformations ( e.g. , rotation and translation), which models the transformations in a latent vector level similar to (Rhodin et al., 2018) and NSD (Rhodin et al., 2019).
Revised: PeCLR proposed a contrastive objective equivariant to geometric transformations ( e.g. , rotation and translation), which models the transformations in a latent vector level similar to (Rhodin et al., 2018) and NSD (Rhodin et al., 2019).

Original: It showed accuracy improvements over the random initialization baseline, but trained on a small-scale dataset ( ∼ 32K) (Zimmermann et al., 2019) during fine-tuning, although more hand data, for example, MSCOCO (150K) and YT3D (47K), exists.
Revised: It showed accuracy improvements over the random initialization baseline, but trained on a small-scale dataset ( ∼ 32K) (Zimmermann et al., 2019) during fine-tuning, although more hand data, for example, MSCOCO (150K) and YT3D (47K), exists.

Original: HanCo applied MoCo (He et al., 2020) on a background-augmented unlabeled hand dataset, but the improvement was marginal and like PeCLR, experiments were done in the controlled setting, not the in-the-wild environment.
Revised: HanCo applied MoCo (He et al., 2020) on a background-augmented unlabeled hand dataset, but the improvement was marginal and like PeCLR, experiments were done in the controlled setting, not the in-the-wild environment.

Original:  C ONVENTION OF 3D HUMAN POSE AND SHAPE ESTIMATION
Revised:  C ONVENTION OF 3D HUMAN POSE AND SHAPE ESTIMATION

Original: Architecture.
Revised: Architecture.

Original: A 3DHPSE network consists of a backbone and a human mesh regressor, as depicted in Figure 2.
Revised: A 3DHPSE network consists of a backbone and a human mesh regressor, as depicted in Figure 2.

Original: A backbone (He et al., 2016; Sun et al., 2019) extracts image features from a given human-centered image.
Revised: A backbone (He et al., 2016; Sun et al., 2019) extracts image features from a given human-centered image.

Original: The features are fed to a human mesh regressor that estimates a mesh defined by human models, such as SMPL (Loper et al., 2015).
Revised: The features are fed to a human mesh regressor that estimates a mesh defined by human models, such as SMPL (Loper et al., 2015).

Original: Pre-training.
Revised: Pre-training.

Original: Almost all 3DHPSE methods pre-train their backbones by ImageNet (Russakovsky et al., 2015) classification.
Revised: Almost all 3DHPSE methods pre-train their backbones by ImageNet (Russakovsky et al., 2015) classification.

Original: They take pre-trained weights from the open source ( e.g. , torchvision (Paszke et al., 2017)) and initialize their backbones with them in practice.
Revised: They take pre-trained weights from the open source ( e.g. , torchvision (Paszke et al., 2017)) and initialize their backbones with them in practice.

Original: Initializing a backbone’s weights with the ImageNet classification-pre-trained weights is known to expedite training convergence and to bring better accuracy than random initialization.
Revised: Initializing a backbone’s weights with the ImageNet classification-pre-trained weights is known to expedite training convergence and to bring better accuracy than random initialization.

Original: Fine-tuning.
Revised: Fine-tuning.

Original: During fine-tuning, a pre-trained backbone and a human mesh regressor are trained in an end-to-end manner by mix-using MoCap and in-the-wild 2D datasets.
Revised: During fine-tuning, a pre-trained backbone and a human mesh regressor are trained in an end-to-end manner by mix-using MoCap and in-the-wild 2D datasets.

Original: 2D annotations of inthe-wild 2D datasets weakly supervise the 3D mesh prediction (Kanazawa et al., 2018), and the 3D pseudo-mesh labels generated from 2D joints (Pavlakos et al., 2019; Joo et al., 2021; Moon et al., 2022b) directly supervise the 3D output.
Revised: 2D annotations of inthe-wild 2D datasets weakly supervise the 3D mesh prediction (Kanazawa et al., 2018), and the 3D pseudo-mesh labels generated from 2D joints (Pavlakos et al., 2019; Joo et al., 2021; Moon et al., 2022b) directly supervise the 3D output.

Original: The mixed use of the MoCap and the in-the-wild 2D datasets has enabled reasonable performance on in-the-wild 3DHPSE, despite the scarce in-the-wild 3D training data.
Revised: The mixed use of the MoCap and the in-the-wild 2D datasets has enabled reasonable performance on in-the-wild 3DHPSE, despite the scarce in-the-wild 3D training data.

Original:  E XPERIMENT 4. S ETTING
Revised:  E XPERIMENT 4. S ETTING

Original: We adopt ResNet-50 (He et al., 2016) as a backbone and a SMPL (Loper et al., 2015) mesh as an estimation target.
Revised: We adopt ResNet-50 (He et al., 2016) as a backbone and a SMPL (Loper et al., 2015) mesh as an estimation target.

Original: During fine-tuning on 3DHPSE, we mainly use PARE (Kocabas et al., 2021) as a human mesh regressor and Human3.6M (Ionescu et al., 2014) and
Revised: During fine-tuning on 3DHPSE, we mainly use PARE (Kocabas et al., 2021) as a human mesh regressor and Human3.6M (Ionescu et al., 2014) and

Original: MSCOCO (Lin et al., 2014) as training datasets.
Revised: MSCOCO (Lin et al., 2014) as training datasets.

Original: For the semi-supervised setting, we exploit 10% of Human3.6M and 10% of MSCOCO data.
Revised: For the semi-supervised setting, we exploit 10% of Human3.6M and 10% of MSCOCO data.

Original: 3DPW (von Marcard et al., 2018), Human36M (Ionescu et al., 2014), and
Revised: 3DPW (von Marcard et al., 2018), Human36M (Ionescu et al., 2014), and

Original: MuPoTS-3D (Mehta et al., 2018) are used as evaluation benchmarks.
Revised: MuPoTS-3D (Mehta et al., 2018) are used as evaluation benchmarks.

Original: We report PA-MPJPE for 3DPW, Human36M, and 3DPCK for MuPoTS-3D following the convention.
Revised: We report PA-MPJPE for 3DPW, Human36M, and 3DPCK for MuPoTS-3D following the convention.

Original: PA-MPJPE stands for the Procrustes-aligned mean per-joint position error in millimeters.
Revised: PA-MPJPE stands for the Procrustes-aligned mean per-joint position error in millimeters.

Original: 3DPCK indicates the percentage of correct 3D keypoints.
Revised: 3DPCK indicates the percentage of correct 3D keypoints.

Original: Pre-training.
Revised: Pre-training.

Original: For the SSL pre-training on ImageNet (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021), we use the publicly released pre-trained ResNet50 weights.
Revised: For the SSL pre-training on ImageNet (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021), we use the publicly released pre-trained ResNet50 weights.

Original: SSL pre-training on ImageNet typically requires much larger computational costs to be effective than the classification baseline.
Revised: SSL pre-training on ImageNet typically requires much larger computational costs to be effective than the classification baseline.

Original: For pre-training on human datasets, we fix the pre-training length (total number of training epochs) at 140 epochs regardless of methods, which is longer than those of 3DHPSE adapted SSL papers (100 epochs) (Spurr et al., 2021; Zimmermann et al., 2021) and the same as that of (Xiao et al., 2018).
Revised: For pre-training on human datasets, we fix the pre-training length (total number of training epochs) at 140 epochs regardless of methods, which is longer than those of 3DHPSE adapted SSL papers (100 epochs) (Spurr et al., 2021; Zimmermann et al., 2021) and the same as that of (Xiao et al., 2018).

Original: To pre-train the proposed 2D pose-driven alternatives, we decay the learning rate by 10 × at 90 and 120 epochs, starting from the initial value of 10 −following (Xiao et al., 2018).
Revised: To pre-train the proposed 2D pose-driven alternatives, we decay the learning rate by 10 × at 90 and 120 epochs, starting from the initial value of 10 −following (Xiao et al., 2018).

Original: Fine-tuning.
Revised: Fine-tuning.

Original: For fine-tuning, we train a 3DHPSE network for 120 epochs and decay the learning rate by 10 × at 90 epochs, which is originally 10 − 4 .
Revised: For fine-tuning, we train a 3DHPSE network for 120 epochs and decay the learning rate by 10 × at 90 epochs, which is originally 10 − 4 .

Original: This schedule empirically showed full convergence for various networks initialized with different pre-trained weights.
Revised: This schedule empirically showed full convergence for various networks initialized with different pre-trained weights.

Original: For the case of training from scratch ( i.e. , random initialization), following the spirit of (He et al., 2019), we extend the total training length to 240 epochs and decay the learning rate by 10 × at 180 epochs from 10 − 4 .
Revised: For the case of training from scratch ( i.e. , random initialization), following the spirit of (He et al., 2019), we extend the total training length to 240 epochs and decay the learning rate by 10 × at 180 epochs from 10 − 4 .

Original: It roughly matches the total training length of the pre-training counterparts (140 epochs + 120 epochs) and empirically showed full convergence.
Revised: It roughly matches the total training length of the pre-training counterparts (140 epochs + 120 epochs) and empirically showed full convergence.

Original: 4. P RE - TRAINING ON I MAGE N ET
Revised: 4. P RE - TRAINING ON I MAGE N ET

Original: We first examine the effects of the current state-of-the-art SSL methods (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021) that pre-train a backbone on ImageNet.
Revised: We first examine the effects of the current state-of-the-art SSL methods (Chen et al., 2020b; Chen & He, 2020; Caron et al., 2020; Chen et al., 2020c; H´enaff et al., 2021) that pre-train a backbone on ImageNet.

Original: As summarized in Table 1 and Figure 3, when full fine-tuning data is used, they show approximately 2 × slower convergence and 7.7% higher errors (PA-MPJPE) than the ImageNet classification pre-training (He et al., 2016).
Revised: As summarized in Table 1 and Figure 3, when full fine-tuning data is used, they show approximately 2 × slower convergence and 7.7% higher errors (PA-MPJPE) than the ImageNet classification pre-training (He et al., 2016).

Original: Interestingly, the final accuracy is even worse than the random initialization baseline, though the convergence is faster.
Revised: Interestingly, the final accuracy is even worse than the random initialization baseline, though the convergence is faster.

Original: In the semi-supervised setting,
Revised: In the semi-supervised setting,

Original: SSL methods outperform the random initialization baseline by 2.5%, but still the classification pretraining provides the best overall accuracy.
Revised: SSL methods outperform the random initialization baseline by 2.5%, but still the classification pretraining provides the best overall accuracy.

Original: We analyze the results that seemingly oppose the fact that SSL surpasses the random initialization and classification baselines in object detection and instance segmentation (Grill et al., 2020; H´enaff et al., 2021) by answering three questions.
Revised: We analyze the results that seemingly oppose the fact that SSL surpasses the random initialization and classification baselines in object detection and instance segmentation (Grill et al., 2020; H´enaff et al., 2021) by answering three questions.

Original: (i) Why SSL is worse than the random initialization baseline, when full fine-tuning data is used?
Revised: (i) Why SSL is worse than the random initialization baseline, when full fine-tuning data is used?

Original: We conjecture that a data domain gap is one of the reasons for the different results.
Revised: We conjecture that a data domain gap is one of the reasons for the different results.

Original: While both object detection and instance segmentation target localization of arbitrary class objects, 3DHPSE only targets a single class, the human.
Revised: While both object detection and instance segmentation target localization of arbitrary class objects, 3DHPSE only targets a single class, the human.

Original: For inference on arbitrary class objects, learning a wide range of general features unlimited to labels of a dataset could be advantageous in the generalization aspect (Tendle & Hasan, 2021).
Revised: For inference on arbitrary class objects, learning a wide range of general features unlimited to labels of a dataset could be advantageous in the generalization aspect (Tendle & Hasan, 2021).

Original: However, for 3DHPSE, a backbone network is preferred to learn more about human features rather than features of arbitrary objects, given the limited learning capacity.
Revised: However, for 3DHPSE, a backbone network is preferred to learn more about human features rather than features of arbitrary objects, given the limited learning capacity.

Original: Transferring knowledge about arbitrary objects could distract a network from learning necessary human features for 3DHPSE. (ii) Why the classification pre-training outperforms the random initialization baseline, while SSL does not, when full finetuning data is used?
Revised: Transferring knowledge about arbitrary objects could distract a network from learning necessary human features for 3DHPSE. (ii) Why the classification pre-training outperforms the random initialization baseline, while SSL does not, when full finetuning data is used?

Original: The classification pre-training makes a backbone to learn high-level semantic representations, such as the global structure of objects, that could be beneficially transferred to inference on humans.
Revised: The classification pre-training makes a backbone to learn high-level semantic representations, such as the global structure of objects, that could be beneficially transferred to inference on humans.

Original: For example, (Yosinski et al., 2015) showed that AlexNet (Krizhevsky et al., 2017) trained on ImageNet can recognize important features of human faces, although ImageNet has no labels of human faces.
Revised: For example, (Yosinski et al., 2015) showed that AlexNet (Krizhevsky et al., 2017) trained on ImageNet can recognize important features of human faces, although ImageNet has no labels of human faces.

Original: On the contrary, visual representations learned by SSL are likely to lack high-
Revised: On the contrary, visual representations learned by SSL are likely to lack high-

Original: [Table]
Revised: [Table]

Original: trained on ImageNet by different pre-training methods.
Revised: trained on ImageNet by different pre-training methods.

Original: level information that can be found in objects with the same class.
Revised: level information that can be found in objects with the same class.

Original: Instead, SSL focuses on learning representations that are invariant to data augmentations, which are performed on a single object sample.
Revised: Instead, SSL focuses on learning representations that are invariant to data augmentations, which are performed on a single object sample.

Original: These representations can be inconsistent over instances of the same class (Khosla et al., 2020).
Revised: These representations can be inconsistent over instances of the same class (Khosla et al., 2020).

Original: Considering that 3DHPSE essentially requires high-level priors that are shared across different human samples, inconsistent representations learned by SSL could be detrimental to 3DHPSE.
Revised: Considering that 3DHPSE essentially requires high-level priors that are shared across different human samples, inconsistent representations learned by SSL could be detrimental to 3DHPSE.

Original: (iii) Why SSL is effective in the semi-supervised setting, compared with the random initialization?
Revised: (iii) Why SSL is effective in the semi-supervised setting, compared with the random initialization?

Original: Inconsistent representations learned by SSL could be a problem for 3DHPSE, but it does not indicate their representations are obsolete.
Revised: Inconsistent representations learned by SSL could be a problem for 3DHPSE, but it does not indicate their representations are obsolete.

Original: A convolutional neural network requires sufficient training data to extract meaningful low-level features ( e.g. , textures) from images.
Revised: A convolutional neural network requires sufficient training data to extract meaningful low-level features ( e.g. , textures) from images.

Original: In this perspective, insufficient training images deprive a network of learning high-level representations, which in turn harms 3DHPSE.
Revised: In this perspective, insufficient training images deprive a network of learning high-level representations, which in turn harms 3DHPSE.

Original: In such a circumstance, SSL can step in to provide a sufficient amount of training images to a network and to make it learn necessary features.
Revised: In such a circumstance, SSL can step in to provide a sufficient amount of training images to a network and to make it learn necessary features.

Original: 4. A NALYSIS OF SSL FOR 3DHPSE.
Revised: 4. A NALYSIS OF SSL FOR 3DHPSE.

Original: We further investigate the reasons for the SSL’s disappointing performance discussed in Section 4.2, with experimental evidence.
Revised: We further investigate the reasons for the SSL’s disappointing performance discussed in Section 4.2, with experimental evidence.

Original: We devise a new pre-training approach, JointCon, that combines the
Revised: We devise a new pre-training approach, JointCon, that combines the

Original: SSL approach with the 2D annotation-based approach.
Revised: SSL approach with the 2D annotation-based approach.

Original: JointCon extracts joint-level features (Moon et al., 2022a) by sampling image features based on GT 2D joint locations.
Revised: JointCon extracts joint-level features (Moon et al., 2022a) by sampling image features based on GT 2D joint locations.

Original: It applies contrastive learning in the joint-level features to pull together an anchor with “positive samples” and push
Revised: It applies contrastive learning in the joint-level features to pull together an anchor with “positive samples” and push

Original: [Table]
Revised: [Table]

Original: apart the anchor from “negative samples”.
Revised: apart the anchor from “negative samples”.

Original: We design three variations of JointCon as below: 1)
Revised: We design three variations of JointCon as below: 1)

Original: JointCon(I) defines “positive samples” as joint-level features extracted from the same image and “negative samples” as those from different images.
Revised: JointCon(I) defines “positive samples” as joint-level features extracted from the same image and “negative samples” as those from different images.

Original: 2) JointCon(J) defines “positive samples” as joint-level features extracted from the same joint label and “negative samples” as those from different labels.
Revised: 2) JointCon(J) defines “positive samples” as joint-level features extracted from the same joint label and “negative samples” as those from different labels.

Original: Note that it treats joint-level features of the same joint class from different images as “positive samples”.
Revised: Note that it treats joint-level features of the same joint class from different images as “positive samples”.

Original: 3) JointCon(I+J) defines “positive samples” as joint-level features extracted from the same image and the same joint label, and “negative samples” otherwise.
Revised: 3) JointCon(I+J) defines “positive samples” as joint-level features extracted from the same image and the same joint label, and “negative samples” otherwise.

Original: Table 2 shows that JointCon(J) outperforms the other variations in all three benchmarks.
Revised: Table 2 shows that JointCon(J) outperforms the other variations in all three benchmarks.

Original: The accuracy gap between JointCon(J) and JointCon(I+J) proves that it is important to learn consistent representations across instances of the same class.
Revised: The accuracy gap between JointCon(J) and JointCon(I+J) proves that it is important to learn consistent representations across instances of the same class.

Original: Contrasting representations of the instances in the same class ( i
Revised: Contrasting representations of the instances in the same class ( i

Original: [Table]
Revised: [Table]

Original: e. , objects or joints in the same class) hinders a backbone from learning useful high-level priors and degrades the performance of 3DHPSE.
Revised: e. , objects or joints in the same class) hinders a backbone from learning useful high-level priors and degrades the performance of 3DHPSE.

Original: As a result, the ImageNet classification pre-training rather learns consistent representations of the instances in the same class and achieves better accuracy than SSL, as shown in Table 

Revised: As a result, the ImageNet classification pre-training rather learns consistent representations of the instances in the same class and achieves better accuracy than SSL, as shown in Table

Original: The other important observation is that both
Revised: The other important observation is that both

Original: JointCon(J) and JointCon(I+J) outperform JointCon(I).
Revised: JointCon(J) and JointCon(I+J) outperform JointCon(I).

Original: The accuracy boost by simply adding joint information to the SSL system demonstrates that the current SSL’s low performance is due to the instance-level learning characteristic rather than network architecture, data augmentation, or losses.
Revised: The accuracy boost by simply adding joint information to the SSL system demonstrates that the current SSL’s low performance is due to the instance-level learning characteristic rather than network architecture, data augmentation, or losses.

Original: 4. P RE - TRAINING ON A HUMAN DATASET
Revised: 4. P RE - TRAINING ON A HUMAN DATASET

Original: We investigate three pre-training approaches, SSL, 2D annotation-based pre-training, and synthetic data pre-training.
Revised: We investigate three pre-training approaches, SSL, 2D annotation-based pre-training, and synthetic data pre-training.

Original: First of all, given the results of Section 4.2, we apply SSL on a human dataset following (Spurr et al., 2021; Zimmermann et al., 2021).
Revised: First of all, given the results of Section 4.2, we apply SSL on a human dataset following (Spurr et al., 2021; Zimmermann et al., 2021).

Original: PeCLR (Spurr et al., 2021) that adapted
Revised: PeCLR (Spurr et al., 2021) that adapted

Original: SimCLR (Chen et al., 2020a;b) to a 3D hand pose and shape estimation, MoCo v2 (Chen et al., 2020c), the method used by HanCo (Zimmermann et al., 2021), and Swav (Caron et al., 2020), which is not based on contrastive learning, are experimented.
Revised: SimCLR (Chen et al., 2020a;b) to a 3D hand pose and shape estimation, MoCo v2 (Chen et al., 2020c), the method used by HanCo (Zimmermann et al., 2021), and Swav (Caron et al., 2020), which is not based on contrastive learning, are experimented.

Original: MSCOCO
Revised: MSCOCO

Original: (Lin et al., 2014) is used as a pre-training dataset.
Revised: (Lin et al., 2014) is used as a pre-training dataset.

Original: For synthetic data pre-training, we pre-train PARE (Kocabas et al., 2021) on AGORA (Patel et al., 2021) and SURREAL (Varol et al., 2017).
Revised: For synthetic data pre-training, we pre-train PARE (Kocabas et al., 2021) on AGORA (Patel et al., 2021) and SURREAL (Varol et al., 2017).

Original: For 2D annotation-based pretraining, we experiment with part segmentation estimation, DensePose (G¨uler et al., 2018) estimation, and 2D pose estimation on MSCOCO.
Revised: For 2D annotation-based pretraining, we experiment with part segmentation estimation, DensePose (G¨uler et al., 2018) estimation, and 2D pose estimation on MSCOCO.

Original: As shown in Table 3 and Figure 4, the SSL methods bring faster convergence, but the final accuracy becomes on par with the random initialization baseline.
Revised: As shown in Table 3 and Figure 4, the SSL methods bring faster convergence, but the final accuracy becomes on par with the random initialization baseline.

Original: It is also lower than the traditional classification pre-training.
Revised: It is also lower than the traditional classification pre-training.

Original: SSL appears to be effective only when significantly less fine-tuning data (10% Human3.6M + 10% MSCOCO, < 50 K) is used.
Revised: SSL appears to be effective only when significantly less fine-tuning data (10% Human3.6M + 10% MSCOCO, < 50 K) is used.

Original: The results coincide with the experimental results of PeCLR and HanCo; PeCLR was fine-tuned on ∼ 32K labeled data (Zimmermann et al., 2019), and HanCo was fine-tuned on ∼ 64K labeled data (Zimmermann et al., 2021).
Revised: The results coincide with the experimental results of PeCLR and HanCo; PeCLR was fine-tuned on ∼ 32K labeled data (Zimmermann et al., 2019), and HanCo was fine-tuned on ∼ 64K labeled data (Zimmermann et al., 2021).

Original: Synthetic data pre-training shows a similar tendency with SSL, though it outperforms SSL.
Revised: Synthetic data pre-training shows a similar tendency with SSL, though it outperforms SSL.

Original: On the contrary, 2D annotation-based pre-training improves accuracy against the random initialization and classification baselines on all benchmarks.
Revised: On the contrary, 2D annotation-based pre-training improves accuracy against the random initialization and classification baselines on all benchmarks.

Original: In the semi-supervised setting, the improvement increases up to 12.4% compared with the random initialization.
Revised: In the semi-supervised setting, the improvement increases up to 12.4% compared with the random initialization.

Original: The convergence speed is much faster than both random initialization and classification, especially in the semi-supervised setting.
Revised: The convergence speed is much faster than both random initialization and classification, especially in the semi-supervised setting.

Original: [Table]
Revised: [Table]

Original: We think the results of SSL on human data support the statement in Section 4.2 that representations learned by SSL are hard to embed high-level information related to humans.
Revised: We think the results of SSL on human data support the statement in Section 4.2 that representations learned by SSL are hard to embed high-level information related to humans.

Original: Synthetic data pre-training has potential to benefit from rich pose and appearance diversity, but a domain gap problem from highly different image appearances between synthetic and real images seems to remain.
Revised: Synthetic data pre-training has potential to benefit from rich pose and appearance diversity, but a domain gap problem from highly different image appearances between synthetic and real images seems to remain.

Original: Compared with SSL and synthetic data pre-training, 2D annotation-based pre-training appears to effectively transfer high-level priors of humans, such as body articulation, to 3DHPSE.
Revised: Compared with SSL and synthetic data pre-training, 2D annotation-based pre-training appears to effectively transfer high-level priors of humans, such as body articulation, to 3DHPSE.

Original: Well-aligned 3D body pose and robustness to occlusion in Figure 5 verify the effectiveness of 2D annotationbased pre-training.
Revised: Well-aligned 3D body pose and robustness to occlusion in Figure 5 verify the effectiveness of 2D annotationbased pre-training.

Original: Figure 6 visualizes each human part’s feature activation of PARE (Kocabas et al., 2021) to further inspect how representations learned by each pre-training approach affect the human mesh regressor’s understanding of human geometry.
Revised: Figure 6 visualizes each human part’s feature activation of PARE (Kocabas et al., 2021) to further inspect how representations learned by each pre-training approach affect the human mesh regressor’s understanding of human geometry.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our empirical observations propose that we should explore 2D annotation-based pre-training instead of SSL for 3DHPSE.
Revised: Our empirical observations propose that we should explore 2D annotation-based pre-training instead of SSL for 3DHPSE.

Original: However, SSL may still have an advantage over 2D annotation-based pre-
Revised: However, SSL may still have an advantage over 2D annotation-based pre-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: training in the aspect of labeling costs.
Revised: training in the aspect of labeling costs.

Original: In this regard, we answer two questions that could help future 3DHPSE researchers to choose their pre-training strategies.
Revised: In this regard, we answer two questions that could help future 3DHPSE researchers to choose their pre-training strategies.

Original: (i) Does SSL entail a zero cost?
Revised: (i) Does SSL entail a zero cost?

Original: No.
Revised: No.

Original: It is not true in the aspect of data collection.
Revised: It is not true in the aspect of data collection.

Original: Collecting, cleaning, and curating data for SSL demand resources (Russakovsky et al., 2015; He et al., 2019;
Revised: Collecting, cleaning, and curating data for SSL demand resources (Russakovsky et al., 2015; He et al., 2019;

Original: Kotar et al., 2021).
Revised: Kotar et al., 2021).

Original: More importantly, the current SSL methods for 3DHPSE (Spurr et al., 2021;
Revised: More importantly, the current SSL methods for 3DHPSE (Spurr et al., 2021;

Original: Zimmermann et al., 2021) assume a human to be centered in an input image, which is an unrealistic setting for in-the-wild images without bounding box labels.
Revised: Zimmermann et al., 2021) assume a human to be centered in an input image, which is an unrealistic setting for in-the-wild images without bounding box labels.

Original: Thus, we should consider the labeling cost of bounding boxes when using SSL (Purushwalkam & Gupta, 2020; Goyal et al., 2021; ElNouby et al., 2021).
Revised: Thus, we should consider the labeling cost of bounding boxes when using SSL (Purushwalkam & Gupta, 2020; Goyal et al., 2021; ElNouby et al., 2021).

Original: Further discussion is provided in Section B of Appendix.
Revised: Further discussion is provided in Section B of Appendix.

Original: (ii) Is SSL efficient than 2D annotation-based pre-training?
Revised: (ii) Is SSL efficient than 2D annotation-based pre-training?

Original: No. SSL pre-trains on massive unlabeled data, which involves high computational costs.
Revised: No. SSL pre-trains on massive unlabeled data, which involves high computational costs.

Original: It often results in order of magnitude more computation than the supervised counterparts (H ´ enaff et al., 2021).
Revised: It often results in order of magnitude more computation than the supervised counterparts (H ´ enaff et al., 2021).

Original: In addition, due to double feature encoders and complex feature contrasting mechanism, SSL takes more time than 2D annotationbased pre-training to pre-train on the same amount of pre-training data.
Revised: In addition, due to double feature encoders and complex feature contrasting mechanism, SSL takes more time than 2D annotationbased pre-training to pre-train on the same amount of pre-training data.

Original: Pre-training ResNet-50 (He et al., 2016) with PeCLR (Spurr et al., 2021) , MoCo v2 (Chen et al., 2020c), and the 2D pose estimator on MSCOCO took 110, 226, and 14 hours respectively, with the four RTX 2080Ti GPUs.
Revised: Pre-training ResNet-50 (He et al., 2016) with PeCLR (Spurr et al., 2021), MoCo v2 (Chen et al., 2020c), and the 2D pose estimator on MSCOCO took 110, 226, and 14 hours respectively, with the four RTX 2080Ti GPUs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have investigated different approaches for pre-training a 3DHPSE backbone.
Revised: We have investigated different approaches for pre-training a 3DHPSE backbone.

Original: SSL, which has recently become the major trend in the community, has been thoroughly inspected.
Revised: SSL, which has recently become the major trend in the community, has been thoroughly inspected.

Original: 2D annotationbased pre-training and synthetic data pre-training have also been experimented, since they share the similar motivation of reducing labeling costs and transferring useful representations to 3DHPSE.
Revised: 2D annotationbased pre-training and synthetic data pre-training have also been experimented, since they share the similar motivation of reducing labeling costs and transferring useful representations to 3DHPSE.

Original: We experimented with multiple methods of each approach on multiple benchmarks to not draw a conclusion valid in a limited setting.
Revised: We experimented with multiple methods of each approach on multiple benchmarks to not draw a conclusion valid in a limited setting.

Original: Please also refer to Section D to confirm that our observations on the three pre-training approaches are preserved regardless of a 3DHPSE mesh regressor.
Revised: Please also refer to Section D to confirm that our observations on the three pre-training approaches are preserved regardless of a 3DHPSE mesh regressor.

Original: Our empirical results show that 1) SSL is yet to replace the de facto paradigm of the ImageNet classification pre-training, and 2) 2D annotation-based pre-training can effectively transfer the knowledge to 3DHPSE, despite the least amount of pre-training data.
Revised: Our empirical results show that 1) SSL is yet to replace the de facto paradigm of the ImageNet classification pre-training, and 2) 2D annotation-based pre-training can effectively transfer the knowledge to 3DHPSE, despite the least amount of pre-training data.

Original: We believe our findings, analysis, and discussion will significantly influence future research on pre-training for 3DHPSE.
Revised: We believe our findings, analysis, and discussion will significantly influence future research on pre-training for 3DHPSE.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: This work was supported in part by the IITP grants [No.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University), No.2022-0-00156, No.
Revised: This work was supported in part by the IITP grants [No.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University), No.2022-0-00156, No.

Original: 2021-0-02068, and No.2022-0-00156], and the NRF grant [No. 2021M3A9E4080782] funded by the Korea government (MSIT), and AIRS Company in Hyundai Motor Company & Kia Corporation through HMC/KIA-SNU AI Consortium Fund.
Revised: 2021-0-02068, and No.2022-0-00156], and the NRF grant [No. 2021M3A9E4080782] funded by the Korea government (MSIT), and AIRS Company in Hyundai Motor Company & Kia Corporation through HMC/KIA-SNU AI Consortium Fund.

Original: A PPENDIX
Revised: A PPENDIX

Original: In this supplementary material, we provide more experiments and discussions that could not be included in the main text due to the lack of pages.
Revised: In this supplementary material, we provide more experiments and discussions that could not be included in the main text due to the lack of pages.

Original: The contents are summarized below:
Revised: The contents are summarized below:

Original: A - Effects of
Revised: A - Effects of

Original: SSL and 2D
Revised: SSL and 2D

Original: annotation-
Revised: annotation-

Original: based pre training on different
Revised: based pre training on different

Original: amounts of data;
Revised: amounts of data;

Original: InstaVa
Revised: InstaVa

Original: - riety(246K) (Kanazawa et
Revised: - riety(246K) (Kanazawa et

Original: al.
Revised: al.

Original: , 2019) and MPII(25K) (Andriluka
Revised: , 2019) and MPII(25K) (Andriluka

Original: et al.
Revised: et al.

Original: are experimented in addition to MSCOCO(150K)
Revised: are experimented in addition to MSCOCO(150K)

Original: (Lin et al., 2014).
Revised: (Lin et al., 2014).

Original: B - Discussion about labeling costs of SSL and 2D annotation-based pre-training C - Effects of different pre-training approaches, when fine-tuned on different sizes of human data (30% and 60% of (Human3.6M (Ionescu et al., 2014) + MSCOCO (Lin et al.,
Revised: B - Discussion about labeling costs of SSL and 2D annotation-based pre-training C - Effects of different pre-training approaches, when fine-tuned on different sizes of human data (30% and 60% of (Human3.6M (Ionescu et al., 2014) + MSCOCO (Lin et al.,

Original: D - Effects of different human mesh regressors.
Revised: D - Effects of different human mesh regressors.

Original: A E FFECTS OF PRE - TRAINING ON DIFFERENT AMOUNTS OF DATA
Revised: A E FFECTS OF PRE - TRAINING ON DIFFERENT AMOUNTS OF DATA

Original: SSL is known to require large-scale pre-training data to be effective (He et al., 2020).
Revised: SSL is known to require large-scale pre-training data to be effective (He et al., 2020).

Original: In this regard, we apply PeCLR (Spurr et al., 2021) and MoCo v2 (Chen et al., 2020c) on InstaVariety(246K) (Kanazawa et al., 2019), which is 1 . 64 × bigger than MSCOCO(150K) (Lin et al., 2014).
Revised: In this regard, we apply PeCLR (Spurr et al., 2021) and MoCo v2 (Chen et al., 2020c) on InstaVariety(246K) (Kanazawa et al., 2019), which is 1 . 64 × bigger than MSCOCO(150K) (Lin et al., 2014).

Original: Interestingly, pre-training on InstaVariety consistently shows worse accuracy in all benchmarks as shown in Table 4.
Revised: Interestingly, pre-training on InstaVariety consistently shows worse accuracy in all benchmarks as shown in Table 4.

Original: If we use less fine-tuning data, the accuracy gap is increased, except PeCLR on
Revised: If we use less fine-tuning data, the accuracy gap is increased, except PeCLR on

Original: Human3.6M. We think the seemingly contradictory results come from the noisy bounding box used to crop a human from an image.
Revised: Human3.6M. We think the seemingly contradictory results come from the noisy bounding box used to crop a human from an image.

Original: The bounding box of a human in InstaVariety is calculated by measuring minimum and maximum x, y locations of OpenPose (Cao et al., 2017)’s
Revised: The bounding box of a human in InstaVariety is calculated by measuring minimum and maximum x, y locations of OpenPose (Cao et al., 2017)’s

Original: 2D pose estimation.
Revised: 2D pose estimation.

Original: Since the estimated values inevitably have errors, the bounding box obtained from them may not produce a human-centered image.
Revised: Since the estimated values inevitably have errors, the bounding box obtained from them may not produce a human-centered image.

Original: Thus, the input distribution could be severely different during pre-training and fine-tuning, which could lead tothe counter-intuitive results.
Revised: Thus, the input distribution could be severely different during pre-training and fine-tuning, which could lead to counter-intuitive results.

Original: We also provide the results of 2D pose estimation pre-training on MPII(25K)
Revised: We also provide the results of 2D pose estimation pre-training on MPII(25K)

Original: (Andriluka et al., 2014).
Revised: (Andriluka et al., 2014).

Original: Despite much less pre-training data, 2D pose estimation pre-training achieves comparable accuracy to that of pre-training MSCOCO.
Revised: Despite much less pre-training data, 2D pose estimation pre-training achieves comparable accuracy to that of pre-training MSCOCO.

Original: This shows the cost-effectiveness of 2D pose estimation pre-training, which is further discussed in the next section.
Revised: This shows the cost-effectiveness of 2D pose estimation pre-training, which is further discussed in the next section.

Original: [Table]
Revised: [Table]

Original: B D ISCUSSION ABOUT LABELING COSTS OF SSL AND 2D ANNOTATION - BASED PRE - TRAINING
Revised: B D ISCUSSION ABOUT LABELING COSTS OF SSL AND 2D ANNOTATION - BASED PRE - TRAINING

Original: Some could value SSL on bounding box-labeled human data more than 2D annotation-based pretraining, despite the experimental results of the main text.
Revised: Some could value SSL on bounding box-labeled human data more than 2D annotation-based pretraining, despite the experimental results of the main text.

Original: For example, assuming less fine-tuning data like the main text’s Table 6, MoCo v2 (Chen et al., 2020c) may be a more attractive option than the 2D pose estimation pre-training, considering that bounding box annotations take less cost than 2D pose annotations.
Revised: For example, assuming less fine-tuning data like the main text’s Table 6, MoCo v2 (Chen et al., 2020c) may be a more attractive option than the 2D pose estimation pre-training, considering that bounding box annotations take less cost than 2D pose annotations.

Original: In this context, we analyze the cost-effectiveness of different pre-training approaches by comparing the annotation cost in terms of annotation time.
Revised: In this context, we analyze the cost-effectiveness of different pre-training approaches by comparing the annotation cost in terms of annotation time.

Original: Table 5 supports that the 2D pose estimation pre-training is much more cost-effective, given the similar annotation time.
Revised: Table 5 supports that the 2D pose estimation pre-training is much more cost-effective, given the similar annotation time.

Original: The state-of-the-art object annotation paper (Papadopoulos et al., 2017) reported 7 seconds per one bounding box annotation.
Revised: The state-of-the-art object annotation paper (Papadopoulos et al., 2017) reported 7 seconds per one bounding box annotation.

Original: We assume that the same time would take for a human bounding box.
Revised: We assume that the same time would take for a human bounding box.

Original: Liu et al.
Revised: Liu et al.

Original: (Liu & Ferrari, 2017) reported 1.5 seconds per one human key point.
Revised: (Liu & Ferrari, 2017) reported 1.5 seconds per one human key point.

Original: Since a person in MPII (Andriluka et al., 2014) images have 14 key points, we can assume that the annotation time is 21 seconds per person.
Revised: Since a person in MPII (Andriluka et al., 2014) images have 14 key points, we can assume that the annotation time is 21 seconds per person.

Original: Cormier et al.
Revised: Cormier et al.

Original: (Cormier et al., 2021) reported 42.
Revised: (Cormier et al., 2021) reported 42.

Original: seconds per bounding box and pose of one person.
Revised: seconds per bounding box and pose of one person.

Original: We take the number of Cormier et al.
Revised: We take the number of Cormier et al.

Original: (Cormier et al., 2021) in Table
Revised: (Cormier et al., 2021) in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C F INE - TUNING ON DIFFERENT SIZES OF HUMAN DATA
Revised: C F INE - TUNING ON DIFFERENT SIZES OF HUMAN DATA

Original: We explore the effects of different pre-training approaches in different semi-supervised settings by varying sizes of fine-tuning data.
Revised: We explore the effects of different pre-training approaches in different semi-supervised settings by varying sizes of fine-tuning data.

Original: Experimental results of using 10%, 30%, 60%, and 100% finetuning data of (Human3.6M (Ionescu et al., 2014) + MSCOCO (Lin et al., 2014)) are shown in
Revised: Experimental results of using 10%, 30%, 60%, and 100% finetuning data of (Human3.6M (Ionescu et al., 2014) + MSCOCO (Lin et al., 2014)) are shown in

Original: Table 6 and Figure 7.
Revised: Table 6 and Figure 7.

Original: SSL starts to surpass the random initialization baseline when fine-tuning data is reduced to 30%.
Revised: SSL starts to surpass the random initialization baseline when fine-tuning data is reduced to 30%.

Original: Synthetic data pre
Revised: Synthetic data pre

Original: -training also shows a similar tendency.
Revised: -training also shows a similar tendency.

Original: Only 2D annotationbased pre-training consistently outperforms the random initialization baseline, and the accuracy margin is enlarged as the fine-tuning data reduces.
Revised: Only 2D annotationbased pre-training consistently outperforms the random initialization baseline, and the accuracy margin is enlarged as the fine-tuning data reduces.

Original: [Table]
Revised: [Table]

Original: ImageNet
Revised: ImageNet

Original: [Table]
Revised: [Table]

Original: labeled) classification 57.
Revised: labeled) classification 57.

Original: MSCOCO (unlabeled)
Revised: MSCOCO (unlabeled)

Original: PeCLR (Spurr et al., 2021) 63.
Revised: PeCLR (Spurr et al., 2021) 63.

Original: MoCo v2 (Chen et al., 2020c) 60.
Revised: MoCo v2 (Chen et al., 2020c) 60.

Original: AGORA 3DHPSE 60.
Revised: AGORA 3DHPSE 60.

Original: SURREAL 57.
Revised: SURREAL 57.

Original: MSCOCO
Revised: MSCOCO

Original: DensePose est.
Revised: DensePose est.

Original: 2D pose est.
Revised: 2D pose est.

Original: H36M+
Revised: H36M+

Original: MSCOCO
Revised: MSCOCO

Original: [Table]
Revised: [Table]

Original: - random init.
Revised: - random init.

Original: ImageNet
Revised: ImageNet

Original: [Table]
Revised: [Table]

Original: labeled) classification 63.
Revised: labeled) classification 63.

Original: MSCOCO (unlabeled)
Revised: MSCOCO (unlabeled)

Original: PeCLR (Spurr et al., 2021) 71.
Revised: PeCLR (Spurr et al., 2021) 71.

Original: MoCo v2 (Chen et al., 2020c) 66.
Revised: MoCo v2 (Chen et al., 2020c) 66.

Original: AGORA 3DHPSE 64.
Revised: AGORA 3DHPSE 64.

Original: SURREAL 62.
Revised: SURREAL 62.

Original: MSCOCO
Revised: MSCOCO

Original: DensePose est.
Revised: DensePose est.

Original: 2D pose est.
Revised: 2D pose est.

Original: [Figure]
Revised: [Figure]

Original: D E FFECTS OF DIFFERENT HUMAN MESH REGRESSORS
Revised: D E FFECTS OF DIFFERENT HUMAN MESH REGRESSORS

Original: In Table 7 and Table 8, we experiment with different human mesh regressors (Kolotouros et al., 2019a;
Revised: In Table 7 and Table 8, we experiment with different human mesh regressors (Kolotouros et al., 2019a;

Original: Zhang et al., 2021a; Moon et al., 2022a; Moon & Lee, 2020) that have distinct architectures and achieved state-of-the-art accuracy recently.
Revised: Zhang et al., 2021a; Moon et al., 2022a; Moon & Lee, 2020) that have distinct architectures and achieved state-of-the-art accuracy recently.

Original: The best-performing methods of SSL, synthetic data pre-training, and 2D annotation-based pre-training are used to compare.
Revised: The best-performing methods of SSL, synthetic data pre-training, and 2D annotation-based pre-training are used to compare.

Original: Overall, the same tendency is observed.
Revised: Overall, the same tendency is observed.

Original: SSL and synthetic data pre-training underperform the classification baseline, while 2D annotation-based pre-training outperforms it in multiple 3DHPSE benchmarks.
Revised: SSL and synthetic data pre-training underperform the classification baseline, while 2D annotation-based pre-training outperforms it in multiple 3DHPSE benchmarks.

Original: This shows that our observations are not limited to a single 3DHPSE method but can generalize to different methods.
Revised: This shows that our observations are not limited to a single 3DHPSE method but can generalize to different methods.

Original: The notable result is that I2L-MeshNet (Moon & Lee, 2020) produces overall the same accuracy regardless of the pre-training approaches.
Revised: The notable result is that I2L-MeshNet (Moon & Lee, 2020) produces overall the same accuracy regardless of the pre-training approaches.

Original: We conjecture that the different tendency of I2LMeshNet comes from its architecture and target representation.
Revised: We conjecture that the different tendency of I2LMeshNet comes from its architecture and target representation.

Original: While other human mesh regressors estimate the SMPL (Loper et al., 2015) parameters with MLP layers at the end of their networks,
Revised: While other human mesh regressors estimate the SMPL (Loper et al., 2015) parameters with MLP layers at the end of their networks,

Original: I2L-MeshNet directly regresses 2.5D locations of mesh vertices in a fully convolutional manner.
Revised: I2L-MeshNet directly regresses 2.5D locations of mesh vertices in a fully convolutional manner.

Original: The 2.5D representation expresses the xy locations of mesh vertices in the image pixel space and the z location as root joint-relative depth.
Revised: The 2.5D representation expresses the xy locations of mesh vertices in the image pixel space and the z location as root joint-relative depth.

Original: As a result, the accuracy of I2L-MeshNet depends on the segmentation of a human from a cropped human-centric image.
Revised: As a result, the accuracy of I2L-MeshNet depends on the segmentation of a human from a cropped human-centric image.

Original: This partially posits the problem of 3DHPSE to the foreground estimation, which may less demand on learning complex priors about humans, such as 3D human body articulation.
Revised: This partially posits the problem of 3DHPSE to the foreground estimation, which may less demand on learning complex priors about humans, such as 3D human body articulation.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]