  Existing datasets for training pedestrian detectors in images suffer from
limited appearance and pose variation. The most challenging scenarios are
rarely included because they are too difficult to capture due to safety
reasons, or they are very unlikely to happen. The strict safety requirements in
assisted and autonomous driving applications call for an extra high detection
accuracy also in these rare situations. Having the ability to generate people
images in arbitrary poses, with arbitrary appearances and embedded in different
background scenes with varying illumination and weather conditions, is a
crucial component for the development and testing of such applications. The
contributions of this paper are three-fold. First, we describe an augmentation
method for controlled synthesis of urban scenes containing people, thus
producing rare or never-seen situations. This is achieved with a data generator
(called DummyNet) with disentangled control of the pose, the appearance, and
the target background scene. Second, the proposed generator relies on novel
network architecture and associated loss that takes into account the
segmentation of the foreground person and its composition into the background
scene. Finally, we demonstrate that the data generated by our DummyNet improve
performance of several existing person detectors across various datasets as
well as in challenging situations, such as night-time conditions, where only a
limited amount of training data is available. In the setup with only day-time
data available, we improve the night-time detector by $17\%$ log-average miss
rate over the detector trained with the day-time data only.
