Abstract-Semantic segmentation tasks can be well modeled by Markov Random Field (MRF). This paper addresses semantic segmentation by incorporating high-order relations and mixture of label contexts into MRF. Unlike previous works that optimized MRFs using iterative algorithm, we solve MRF by proposing a Convolutional Neural Network (CNN), namely Deep Parsing Network (DPN), which enables deterministic end-to-end computation in a single forward pass. Specifically, DPN extends a contemporary CNN to model unary terms and additional layers are devised to approximate the mean field (MF) algorithm for pairwise terms. It has several appealing properties. First, different from the recent works that required many iterations of MF during back-propagation, DPN is able to achieve high performance by approximating one iteration of MF. Second, DPN represents various types of pairwise terms, making many existing models as its special cases. Furthermore, pairwise terms in DPN provide a unified framework to encode rich contextual information in high-dimensional data, such as images and videos. Third, DPN makes MF easier to be parallelized and speeded up, thus enabling efficient inference. DPN is thoroughly evaluated on standard semantic image/video segmentation benchmarks, where a single DPN model yields state-of-the-art segmentation accuracies on PASCAL VOC 2012, Cityscapes dataset and CamVid dataset. Index Terms-Semantic image/video segmentation, Markov random field, convolutional neural network Since pixels in natural images or videos generally exhibit strong correlation, jointly modeling label distribution in all locations is desirable. To capture these contextual information, Markov random field (MRF) and conditional random field (CRF) [17] are commonly used as classic frameworks for semantic segmentation. They model the joint distribution of labels by defining both unary term and pairwise terms. Unary term reflects the per-pixel confidence of assigning labels while pairwise terms capture the inter-pixel constraints. Most previous studies focus on designing pairwise terms that possess strong expressive power. For example, Kr€ ahenb€ uhl et al. [18] attained accurate segmentation boundary by inferring on a fully-connected graph. Vineet et al. [19] extended [18] by defining both high-order and long-range terms between pixels. Global or local semantic contexts between labels were also investigated by [20] . However, their performance are limited by the relatively shallow models (e.g., SVM or Adaboost) used as unary term. As deep learning gradually takes over in many image recognition fields [21] , researchers have also explored the possibility of designing effective deep architecture for semantic segmentation. For instance, Long et al. [6] transformed fully-connected layers of CNN into convolutional layers, making accurate per-pixel classification possible using contemporary CNN architectures that were pre-trained on ImageNet [22] . Chen et al. [23] improved [6] by feeding the outputs of CNN into a MRF with simple pairwise potentials, but it treated CNN and MRF as separate components. A recent advance was made in joint training CNN and MRF by passing the error of MRF inference backward into CNN [24] . Nonetheless, an iterative inference of MRF such as the mean field algorithm (MF) [25] is required for each training image during the back-propagation (BP). Zheng et al. [26] further showed that the procedure of MF inference can be represented as a Recurrent Neural Network (RNN), but their computational costs are similar to that of [24] . We observed that a direct combination of CNN and MRF as above is inefficient, posing challenges on both optimization difficulty and inference speed. Since CNN typically has millions of parameters and MRF typically has thousands of latent variables, they are cumbersome to jointly optimize and infer. Even worse, incorporating complex pairwise terms into a MRF becomes impractical, limiting the performance of the entire system. In this study, we propose a novel Deep Parsing Network (DPN), which is an end-toend system enabling jointly training of CNN and complex pairwise terms. DPN has several appealing properties: The authors are with the
Taking one step further, the Deep Parsing Network (DPN) REF is designed to approximate the mean-field inference for MRFs in one pass.
Deep Learning Markov Random Field for Semantic Segmentation
IEEE Transactions on Pattern Analysis and Machine Intelligence
IEEE Transactions on Pattern Analysis and Machine Intelligence
Computer Science Medicine