PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models
Abstract
We introduce a new synthetic data generator PSPHDRI + that proves to be a superior pre-training alternative to ImageNet and other large-scale synthetic data counterparts.
We demonstrate that pre-training with our synthetic data will yield a more general model that performs better than alternatives even when tested on out-of-distribution (OOD) sets.
Furthermore, using ablation studies guided by person keypoint estimation metrics with an off-the-shelf model architecture, we show how to manipulate our synthetic data generator to further improve model performance
Introduction
Supervised pre-training has accelerated success in computer vision applications.
There remain questions about which type of data is best suited for pre-training models that are specialized to solve one task.
For human-centric computer vision, researchers have established large-scale human-labeled datasets (Lin et al., 2014; Andriluka et al., 2014b; Li et al., 2019;
Milan et al., 2016; Johnson & Everingham, 2010;
Zhang et al., 2019).
These datasets are hard to create and label and are increasingly scrutinized for labeling and data bias, ethics, legality, and safety issues.
Recently, researchers have started considering synthetic data alternatives to mitigate those issues (Wood et al., 2021;
Fabbri et al., 2018;
Hu et al., 2019; 2021; Fabbri et al., 2021; Bak et al., 2018;
Pishchulin et al., 2011; Varol et al., 2017; Kviatkovsky et al., 2021; Hassan et al., 2021; Ros et al., 2016; Gaidon et al., 2016; Dosovitskiy et al., 2017; Richter et al., 2017;
Wrenninge & Unger, 2018; Roberts & Paczan, 2020; Li et al., 2021; Morrical et al., 2021).
However, synthetic data generators are challenging to create; so researchers have focused on leveraging game environments such as GTA V to render synthetic data and labels.
Most of these datasets have pre-rendered frames, and researchers cannot manipulate the data generator.
Hence, the barrier of entry into simulation-ready and user-friendly synthetic data generators is still high for computer vision and AI researchers.
Consequently, exploiting synthetic data to its full potential is yet to be realized.
Motivated by these limitations, we present a privacy-preserving, ethically sourced, and fully manipulable synthetic data generator for human-centric computer vision named PSP-HDRI + , which is built upon PeopleSansPeople (Ebadi et al., 2021)
(PSP) in Unity 1 .
We demonstrate a strategy by which synthetic data can help surpass benchmark model performance.
We explore two main tasks, namely human detection and keypoint localization.
Having experimented with multiple training methodologies, we have consistently found pre-training with synthetic data and fine-tuning on real data through transfer learning to be the most successful strategy across many domains and tasks.
We validate the above propositions with model performance on in-distribution and OOD benchmarks in our extensive studies.
This work argues that pre-training with synthetic humancentric data helps generalize model performance in the real world.
These performance gains hold even when our synthetic dataset is not tuned to create a 1:1 replica (digital twin) of the real world.
We show that even using na ¨ ıvely generated human-centric data makes it possible to pre-train models that perform better than ImageNet and other large-scale data.
The performance gains from synthetic pre-training are substantial in the few-shot learning and small data regiments.
Related Work
Models pre-trained using large-scale human-labeled datasets such as ImageNet (Deng et al., 2009), MS COCO (Lin et al., 2014), PASCAL VOC (Everingham et al., 2010), NYUDepth V2 (Silberman et al., 2012), and SUN RGB-D (Song et al., 2015) have enabled rapid progress across many computer vision tasks.
The pre-training usually expedites the training process by leveraging the already-learned representations.
The task-specific nature of the pre-training stage tends to encode representations that are necessary to solve that specific task.
If the fine-tuning stage involves solving a different task, the pre-learned features may harm or reduce the model’s ability to perform at its full potential and capacity on the downstream tasks (Goyal et al., 2022).
Furthermore, research has shown that metric sensitivity is a problem with universal generalist pre-training datasets such as ImageNet (Kynk¨a¨anniemi et al., 2022), which can trickle down into the kinds of learned representations.
Recently, in (Madan et al., 2020) the authors argued that data diversity improves OOD performance but degrades in-distribution performance.
Research has shown that pre-training without natural images (with fractals) can help in many natural image tasks (Kataoka et al., 2020).
Therefore, it is necessary to ask, how much of the universal generalist pre-training data is paramount to better fine-tuning?
Do the labeling inaccuracies in such datasets have a detrimental effect on transfer learning?
More importantly, can superior alternatives be found that are easy to create and label?
Therefore, we are motivated to understand whether na ¨ ıvely generated task-specific synthetic data of any size can replace universal generalist pre-training.
PSP-HDRI 3.1 Baseline Environment Design
We use PeopleSansPeople (Ebadi et al., 2021)
(PSP) as our baseline data generator.
PSP is a parametric data generator, created in the Unity game engine, and contains simulationready and fully rigged 3D human assets, a diverse animation library for humans, a parameterized lighting and camera system, and generates synthetic RGB images with ground truth annotations of 2D/3D bounding box, human keypoints, and semantic/instance segmentation.
PSP leverages Domain
Randomization (Tobin et al., 2017) where aspects of the simulation environment are randomized to introduce variations in the generated synthetic data.
These variations are necessary to increase the generalization of models trained with the synthetic data to the real or other domains.
The
Unity Perception package provides a domain randomization framework (Borkman et al., 2021), which allows for diversifying the synthetic dataset with a large number of parametric variations, using a “randomizer” paradigm.
During the simulation, the randomizers act on predefined Unity scene components (e.g., lighting, camera, environment, human assets’ placement and orientation, human assets’ clothing and pose, etc.).
The randomizers can use various sampling techniques to randomly select parameters for each parametric attribute of the scene components.
More information can be found in (Ebadi et al., 2021; Borkman et al., 2021).
3.2 Bridging the Visual Quality and Label Gaps
The existing domain gap between the synthetic and other types of data – whether real or synthetic – poses challenges for OOD performance of models trained solely on synthetic data.
Some factors that will exacerbate the domain gap are content, material, and texture quality.
These however, are very hard and expensive to source and usually require hours of extensive work by artists.
PSP uses low-resolution
COCO images as background textures.
Whilst the background regions in PSP seem to affect the visual quality of the generated images more than the foreground regions, other factors such as simulated sensor noise and sensor type, asset geometry, asset quality, material textures, etc. also do contribute to the overall perceived quality.
To increase the visual quality, we complement PSP with High Dynamic
Range Image (HDRI) backgrounds from Poly Haven 2 as skyboxes, which will enable high-quality rendered backgrounds for our dataset.
These skyboxes contribute to ambient scene lighting and global illumination.
We also use a combination of scene lighting intensity and Sun randomization (which simulates the time of the day and day of the year) to capture more realistic and diverse lighting settings.
We spawn the human assets and other random shapes (occluders and distractors) at the scene’s center, and the HDRI skybox background wraps around the scene.
As with PSP, we use primitive 3D game objects as occluders and distractors, albeit with randomized high-quality HDRI textures.
We modified the camera randomization to orbit around the scene and capture a diverse range of perspectives of the human assets and take advantage of all viewpoints in the
HDRI skyboxes.
To this end, we have produced a new version of PSP, dubbed PSP-HDRI.
We performed additional ablation studies to guide the design of our data generator for better pre-training and OOD generalization, which we call
PSP-HDRI + .
Some examples are shown in fig.
Our data generator produces sub-pixel-perfect labels.
However, real datasets rarely have very accurate labeling as human labeling is a subjective task and prone to the errors and negligence of human annotators.
Further, there exist inevitable label distribution discrepancies between different domains.
In PSP-HDRI + , we experimented with crude label adaptation for bounding boxes and keypoints to overcome the label gap.
We use the bounding box and keypoint annotations from the COCO training set as our comparison baseline.
Our criteria for bounding box label adaptation are: firstly, remove boxes smaller than the smallest box in COCO that has keypoint annotations; and secondly, remove boxes whose size to image size ratio is less than or equal to the same ratio in COCO.
This strategy effectively removes boxes that are too small.
For keypoint label adaptation, we ensure that for each box area range, the probability of having annotations for each keypoint matches between the synthetic and COCO, by randomly removing surplus keypoint annotations.
Note that COCO has six box area ranges.
This keypoint adaptation statistically matched the labeling inaccuracies of human annotators and the label dis- tribution in the COCO dataset.
As a result, we observed improvements in large-scale human-annotated benchmark datasets.
Experiments and Results
We generated our synthetic datasets using domain randomization with na¨ıve (random uniform) sampling from the set of our data generator parameters; we also generated our synthetic data from three random generation seeds in order to obtain a fair comparison.
For our experiments we considered the tasks of human detection and keypoint localization.
To obtain a suite of benchmarks, we used three pre-training strategies: (1) random initialization (no pre– training), (2) pre-training with ImageNet, (3) and pre-training with various sizes of synthetic data. We then fine-tuned all these models on various sizes of real data.
We use the same training regiment without any hyperparameter tuning for all our models.
Lastly, we compared the performance of these models on in-distribution and OOD sets to establish the generalization and applicability of each model across a wide range of real-world examples.
For our real data training, we used the COCO person training dataset, divided into overlapping sets of 641 , 6411 , 32057 , and 64115 images.
The COCO person-val2017 and test-dev2017 have 2693 and 20288 images, respectively.
We also used the MPII Human Pose dataset (Andriluka et al., 2014a) divided into 16712 training and 696 validation sets.
We compare our synthetic data to another largescale synthetic dataset generated from scenes of the GTA V game with crowds of people walking in them, called MOTSynth (Fabbri et al., 2021); we randomly selectedtraining and 5008 validation images from MOTSynth to obtain a fair comparison.
For our OOD tests we used the following datasets with their respective image numbers:
CrowdPose (Li et al., 2019)
Trainval ( 12000 images), Leeds
Sports Pose (Johnson & Everingham, 2010) ( 10000 images),
Occluded Humans (Zhang et al., 2019) ( 4731 images), and
MOT17 (Milan et al., 2016) ( 5316 images).
4.1 Training Strategy
For all our experiments, we use the Detectron2 Keypoint R-CNN R50-FPN variant (He et al., 2017) with
ResNet-50 (He et al., 2016) plus Feature Pyramid Network (FPN) (Lin et al., 2017) backbones.
We trained our models from scratch (random weight initialization) with Group
Normalization (GN) (Wu & He, 2018; Wu et al., 2019;
He et al., 2019).
Similar to (Ebadi et al., 2021), for all our models, we use a learning rate annealing strategy, where we reduce the learning rate when the validation keypoint
Average Precision (AP) metric has stopped improving.
Our models benefited from reducing the learning rate by a factor of 10 × once learning has stagnated based on a threshold (epsilon) for several epochs (patience period).
Every time the patience period ends, we reduce the learning rate, and halve epsilon and the next patience period.
We perform the learning rate reduction three times.
Every time the learning rate is reduced, we restore the weights from the model checkpoint that achieves the highest metrics on the validation.
Thus we ensure that the last model checkpoint is also the best performing one.
For both pre-training and fine-tuning experiments, we set the initial learning rate to 0 .
02 , the initial patience to 38 epochs, and the initial epsilon to 5 .
We perform a linear warm-up period of 1000 iterations at the start of training, where we slowly increase the learning rate to the initial learning rate.
The weight decay is 0 . 0001 , and momentum is 0 .
We use 8 NVIDIA Tesla V100 GPUs on synchronized SGD with a mini-batch size of 2 images per GPU; the mean pixel value and standard deviation from ImageNet is used for image normalization.
We do not change the default augmentations used by Detectron2 and perform the evaluation every two epochs.
Additionally, we fix the model seed to improve reproducibility.
When we train on real data, we also evaluate real data from the same distribution.
Likewise, we evaluate on synthetic data from the same distribution when we train on synthetic sets.
4.2 Pre-Training Benchmarks
Tab. 1 shows a comparison between models with no pretraining, ImageNet, and synthetic pre-training.
Unsurprisingly, we find that ImageNet pre-training improves the model performance over training from scratch for any real data size.
Interestingly, even small sets of 4 .
9 × 10 3 synthetic images obtain performance that is better or on par with ImageNet pre-training, with larger effects for few-shot transfer.
While we observe improvements with pre-training on all sizes of synthetic data, our largest set of 245 × 10 3 images achieves the best performance.
In (Ebadi et al., 2021) the authors showed positive trends for much larger sets of synthetic data, and we do expect the numbers on tab.
1 to improve if more synthetic data is used for pre-training.
Further, we measure the generalization ability of models with no pre-training, ImageNet, and synthetic pre-training on a wide range of OOD sets for the tasks of human detection and keypoint localization.
In tab.
2 all our models are fine-tuned with the entire COCO person training data.
We observe only marginal improvements with ImageNet pre-training over training from scratch.
Additionally, on average, MOTSynth pre-trained models are inferior to those of a model pre-trained with a similarly-sized PSP-HDRI dataset of 49 × 10 3 images.
We also reconfirm that OOD generalization of even 4 .
9 × 10 3 synthetic images for pretraining is on par with ImageNet pre-training.
The best overall performance is also still achieved with larger synthetic pre-training sets.
We hypothesize that since PSP-HDRI synthetic data is task-specific, it contains the necessary signals and representations needed for fine-tuning and better generalization on human-centric tasks.
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Table] [Table]
[Table]
4.3 Zero-Shot Ablation Studies
We performed a set of ablation studies using metrics obtained from a suite of OOD datasets to guide our data generator design choices.
In tab.
3 we list the zero-shot performance of models trained with approximatelysynthetic images in each row.
Beyond PSP-HDRI, we explored the effect of the following: bounding box label adaptation; bounding box + keypoint label adaptation; using no occluder/distractor objects; using high-quality 3D models from Poly Haven 3 instead of our primitive 3D game objects as occluders/distractors; disabling the clothing texture randomization (shader graph randomizer); using Subpixel
Morphological Anti-Aliasing (SMAA) which gives graphics a smoother appearance; and lastly, using only simple animations, such as walking, running, and standing idle, instead of PSP-HDRI’s diverse animation library.
Interestingly, we observed performance boosts over the PSPHDRI baseline with every modification, except when we removed our occluder/distractor objects from the scene.
We then combined all the modifications that hinted toward positive trends together to form the PSP-HDRI + set, which includes: “box + kpt adapt.”
, “Poly Haven occluders”, “no shadergrpah”, “SMAA”, and “simple anims”.
PSP-HDRI
+ obtains the best overall zero-shot performance.
We also trained PSP-HDRI + with random crop augmentation during training, which improved zero-shot performance on the
MOTSynth and MOT17.
The MOTSynth trained model performs exceedingly well on MPII val and Crowdpose
Trainval.
We also report its performance on in-distribution
MOTSynth val set, which is unsurprisingly better than the rest.
The MOTSynth dataset was designed to expedite pretraining for the MOT17 challenge, hence its better performance on the MOT17 set.
However, on the larger and more diverse COCO test-dev2017 and COCO person-val2017, the
PSP-HDRI
+ performs better than MOTSynth.
4.4 Comparison with Another Synthetic Counterpart
Lastly, we pre-train our models on PSP-HDRI + and a similarly-sized set from MOTSynth.
We fine-tuned either of them on two real datasets COCO and MPII and tested them in-distribution (COCO test-dev2017, person-val2017, and MPII val) and
OOD (Crowdpose Trainval, Leeds Sports,
Occluded Humans, MOT17) as listed in tab.
As before, models pre-trained on PSP-HDRI + outperform those pre- trained on MOTsynth when tested on in-distribution test sets (COCO test-dev2017, person-val2017, and MPII val).
Furthermore, the OOD generalization of PSP-HDRI + trained models also tends to be more robust than MOTSynth.
Interestingly, a model pre-trained on MOTSynth and then fine-tuned on COCO (second row) is only marginally better than a model pre-trained on PSP-HDRI
+ (first row) when tested on MOTSynth again.
When tested on MOT17, the model pre-trained on MOTSynth and then fine-tuned on
COCO data (second row) performs on par with a model trained only on MOTSynth (bbox AP of 32 . 46 vs. 32 . 75 ).
Conversely, the model pre-trained on PSP-HDRI + and finetuned on COCO (first row) benefits from a more than double improvement on bounding box AP when tested on MOT (bbox AP of 32 . 01 vs. 13 . 97 ).
This result indicates that after fine-tuning, the model forgets the MOTSynth pre-training.
It is worth noting that the entire MOTSynth dataset hassequences where the background is primarily static.
In PSPHDRI + , we use 510 HDRI backgrounds, but the camera randomizer allows us to capture unique and diverse perspectives from these spherical images and the human assets.
Additionally, PSP-HDRI + has more diverse human poses, the image quality is improved with SMAA, the lighting randomization produces unique and diverse light settings for our scenes and provides image augmentation out-of-the-box, the occluder objects play an important role in making the model more robust in cases of partially or almost entirely occluded foreground objects, and finally our label adaptation can statistically match the target label distributions, further reducing the visual quality and label gaps.
Conclusions
We introduced a new synthetic data generator PSP-HDRI + and showed that for human-centric computer vision, it provides a superior pre-training data compared with other common alternatives.
We have also identified a training strategy whereby we obtained the best pre-training and fine-tuning results without the need for hyper-parameter search.
This is made possible by automatic learning-rate annealing that adapts to the model performance on a validation set and is scaled based on the size of training data.
We demonstrated that our open-source, privacy-preserving, ethically sourced, and fully manipulable human-centric synthetic data generator has the potential for improvement beyond its out-of-thebox capabilities and is an excellent choice for meta-learning and sim2real research.
A Appendix: More examples from PSP-HDRI +
B Appendix: Pose Diversity
In our data generator we used a set of animations derived from human motion capture clips to create a reasonably diverse set of poses for our human models.
In order to quantify the pose diversity in our generated dataset, we used a technique from (Ebadi et al., 2021) where keypoint annotations from all the annotated person instances are used, provided that the torso of the character has annotations (hips and shoulders).
Then all the keypoints are aligned such that the mid-hip point is at (0 , 0) coordinates on a 2D axis.
The keypoint distances are scaled according to the length of each torso, in order to make all the skeletons roughly the same size.
If we then plot each keypoint individually, we obtain the heatmaps shown in fig.
B.2.
We opted to show only the representative keypoints that belong to the extremities of the human, as those will have the largest dispalcement.
Note that PSP-HDRI (in blue) shows a more symmetrical pattern with larger footprint compared with COCO (in red).
Most people in COCO are captured from the frontal view, hence the asymmetrical heatmaps.
For PSP-HDRI + (in purple) since we only used simple animations, we observe a smaller footprint for each keypoint location variation.
The MOTSynth dataset (in green) does not have facial keypoints, hence the nose keypoint has no information.
For the rest of the keypoints, we observe a larger footprint for the PSP-HDRI + compared with MOTSynth; meaning that our animations are still more diverse than those of MOTSynth with people walking around in scenes of GTA V game.
[Figure]
[Figure]