  Both generative learning and discriminative learning have recently witnessed
remarkable progress using Deep Neural Networks (DNNs). For structured input
synthesis and structured output prediction problems (e.g., layout-to-image
synthesis and image semantic segmentation respectively), they often are studied
separately. This paper proposes deep consensus learning (DCL) for joint
layout-to-image synthesis and weakly-supervised image semantic segmentation.
The former is realized by a recently proposed LostGAN approach, and the latter
by introducing an inference network as the third player joining the two-player
game of LostGAN. Two deep consensus mappings are exploited to facilitate
training the three networks end-to-end: Given an input layout (a list of object
bounding boxes), the generator generates a mask (label map) and then use it to
help synthesize an image. The inference network infers the mask for the
synthesized image. Then, the latent consensus is measured between the mask
generated by the generator and the one inferred by the inference network. For
the real image corresponding to the input layout, its mask also is computed by
the inference network, and then used by the generator to reconstruct the real
image. Then, the data consensus is measured between the real image and its
reconstructed image. The discriminator still plays the role of an adversary by
computing the realness scores for a real image, its reconstructed image and a
synthesized image. In experiments, our DCL is tested in the COCO-Stuff dataset.
It obtains compelling layout-to-image synthesis results and weakly-supervised
image semantic segmentation results.
