On Boosting Semantic Street Scene Segmentation with Weak Supervision Training convolutional networks for semantic segmentation requires perpixel ground truth labels, which are very time consuming and hence costly to obtain. Therefore, in this work, we research and develop a hierarchical deep network architecture and the corresponding loss for semantic segmentation that can be trained from weak supervision, such as bounding boxes or image level labels, as well as from strong perpixel supervision. We demonstrate that the hierarchical structure and the simultaneous training on strong (perpixel) and weak (bounding boxes) labels, even from separate datasets, constantly increases the performance against perpixel only training. Moreover, we explore the more challenging case of adding weak imagelevel labels. We collect street scene images and weak labels from the immense Open Images dataset to generate the OpenScapes dataset, and we use this novel dataset to increase segmentation performance on two established perpixel labeled datasets, Cityscapes and Vistas. We report performance gains up to 13.2% mIoU on crucial street scene classes, and inference speed of 20 fps on a Titan V GPU for Cityscapes at 512 x 1024 resolution. Our network and OpenScapes dataset are shared with the research community.

Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions Semantic segmentation and instance level segmentation made substantial progress in recent years due to the emergence of deep neural networks (DNNs). A number of deep architectures with Convolution Neural Networks (CNNs) were proposed that surpass the traditional machine learning approaches for segmentation by a large margin. These architectures predict the directly observable semantic category of each pixel by usually optimizing a crossentropy loss. In this work we push the limit of semantic segmentation towards predicting semantic labels of directly visible as well as occluded objects or objects parts, where the networkxe2x80x99s input is a single depth image. We group the semantic categories into one background and multiple foreground object groups, and we propose a modification of the standard crossentropy loss to cope with the settings. In our experiments we demonstrate that a CNN trained by minimizing the proposed loss is able to predict semantic categories for visible and occluded object parts without requiring to increase the network size (compared to a standard segmentation task). The results are validated on a newly generated dataset (augmented from SUNCG) dataset.