  Semantic segmentation is a basic but non-trivial task in computer vision.
Many previous work focus on utilizing affinity patterns to enhance segmentation
networks. Most of these studies use the affinity matrix as a kind of feature
fusion weights, which is part of modules embedded in the network, such as
attention models and non-local models. In this paper, we associate affinity
matrix with labels, exploiting the affinity in a supervised way. Specifically,
we utilize the label to generate a multi-scale label affinity matrix as a
structural supervision, and we use a square root kernel to compute a non-local
affinity matrix on output layers. With such two affinities, we define a novel
loss called Affinity Regression loss (AR loss), which can be an auxiliary loss
providing pair-wise similarity penalty. Our model is easy to train and adds
little computational burden without run-time inference. Extensive experiments
on NYUv2 dataset and Cityscapes dataset demonstrate that our proposed method is
sufficient in promoting semantic segmentation networks.
