  We introduce a new method for generating color images from sketches or edge
maps. Current methods either require some form of additional user-guidance or
are limited to the "paired" translation approach. We argue that segmentation
information could provide valuable guidance for sketch colorization. To this
end, we propose to leverage semantic image segmentation, as provided by a
general purpose panoptic segmentation network, to create an additional
adversarial loss function. Our loss function can be integrated to any baseline
GAN model. Our method is not limited to datasets that contain segmentation
labels, and it can be trained for "unpaired" translation tasks. We show the
effectiveness of our method on four different datasets spanning scene level
indoor, outdoor, and children book illustration images using qualitative,
quantitative and user study analysis. Our model improves its baseline up to 35
points on the FID metric. Our code and pretrained models can be found at
https://github.com/giddyyupp/AdvSegLoss.
