E-CRF: E MBEDDED C ONDITIONAL R ANDOM F IELD FOR B OUNDARY - CAUSED C LASS W EIGHTS C ONFUSION IN S EMANTIC S EGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Modern semantic segmentation methods devote much effect to adjusting image feature representations to improve the segmentation performance in various ways, such as architecture design, attention mechnism, etc.
However, almost all those methods neglect the particularity of class weights (in the classiﬁcation layer) in segmentation models.
In this paper, we notice that the class weights of categories that tend to share many adjacent boundary pixels lack discrimination, thereby limiting the performance.
We call this issue B oundary-caused C lass W eights
C onfusion ( BCWC ).
We try to focus on this problem and propose a novel method named E mbedded C onditional R andom F ield ( E-CRF ) to alleviate it.
E-CRF innovatively fuses the CRF into the CNN network as an organic whole for more effective end-to-end optimization.
The reasons are two folds.
It utilizes CRF to guide the message passing between pixels in high-level features to purify the feature representation of boundary pixels, with the help of inner pixels belonging to the same object.
More importantly, it enables optimizing class weights from both scale and direction during backpropagation.
We make detailed theoretical analysis to prove it.
Besides, superpixel is integrated into E-CRF and served as an auxiliary to exploit the local object prior for more reliable message passing.
Finally, our proposed method yields impressive results on ADE20K, Cityscapes, and Pascal Context datasets.
Code will be available.
I NTRODUCTION
Semantic segmentation plays an important role in practical applications such as autonomous driving, image editing, etc .
Nowadays, numerous CNN-based methods Chen et al.
Fu et al.
Ding et al.
(2019) have been proposed.
They attempt to adjust the image feature representation of the model itself to recognize each pixel correctly.
However, almost all those methods neglect the particularity of class weights (in the classiﬁcation layer) that play an important role in distinguishing pixel categories in segmentation models.
Hence, it is critical to keep class weights discriminative.
Unfortunately, CNN models have the natural defect for this.
Generally speaking, most discriminative higher layers in the CNN network always have the larger receptive ﬁeld, thus pixels around the boundary may obtain confusing features from both sides.
As a result, these ambiguous boundary pixels will mislead the optimization direction of the model and make the class weights of such categories that tend to share adjacent pixels indistinguishable.
For the convenience of illustration, we call this issue as B oundary-caused C lass W eights C onfusion ( BCWC ).
We take DeeplabV3+ Chen et al.
(2018a)
as an example to train on ADE20K Zhou et al.
(2017) dataset.
Then, we count the number of adjacent pixels for each class pair and ﬁnd a corresponding category that has the most adjacent pixels for each class.
Fig.1(a) shows the similarity of the class weight between these pairs in descending order according to the number of adjacent pixels.
It is clear that if two categories share more adjacent pixels, their class weights tend to be more similar, which actually indicates that BCWC makes class representations lack discrimination and damages the overall segmentation performance.
Previous works mainly aim to improve boundary pixel segmentation, but they seldom explicitly take class weights confusion i.e ., BCWC, into consideration 1 .
[Figure]
+
[Figure]
F p
[Figure]
C l a ss w e i g h t
s i m il a r i t y
Considering the inherent drawback of CNN networks mentioned before, delving into the relationship between raw pixels becomes a potential alternative to eliminate the BCWC problem, and Conditional
Random Field ( CRF ) Chen et al.
(2014) stands out.
It is generally known that pixels of the same object tend to share similar characteristics in the local area.
Intuitively, CRF utilizes the local consistency between original image pixels to reﬁne the boundary segmentation results with the help of inner pixels of the same object.
CRF makes some boundary pixels that are misclassiﬁed by the CNN network quite easy to be recognized correctly.
But these CRF-based methods Chen et al.
Zhen et al.
(2020a) only adopt CRF as an ofﬂine post-processing module, we call it
Vanilla-CRF , to reﬁne the ﬁnal segmentation results.
They are incapable of relieving BCWC problem as CRF and the CNN network are treated as two totally separate modules.
Based on Chen et al.
(2014; 2017a), Lin et al.
; Arnab et al.
; Zheng et al.
(2015) go a step further to unify the segmentation model and CRF in a single pipeline for end-to-end training.
We call it Joint-CRF for simplicity.
Same as Vanilla-CRF , Joint-CRF inclines to rectify those misclassiﬁed boundary pixels via increasing the prediction score of the associated category, which means it still operates on the object class probabilities .
But it can alleviate the BCWC problem to some extent as the probability score reﬁned by CRF directly involves in the model backpropagation.
Afterwards, the disturbing gradients caused by those pixels will be relieved, which will promote the class representation learning.
However, as shown in Fig.3, the effectiveness of Joint-CRF is restricted as it only optimizes the scale of the gradient and lacks the ability to optimize class representations effectively due to the defective design.
More theoretical analysis can be found in Sec.3.3.
To overcome the aforementioned drawbacks, in this paper, we present a novel approach named
E mbedded CRF ( E-CRF ) to address the BCWC problem more effectively.
The superiority of E-CRF lies in two main aspects.
On the one hand, by fusing CRF mechanism into the segmentation model,
E-CRF utilizes the local consistency among original image pixels to guide the message passing of high-level features.
Each pixel pair that comes from the same object tends to obtain higher message passing weights.
Therefore, the feature representation of the boundary pixels can be puriﬁed by the corresponding inner pixels from the same object.
In turn, those pixels will further contribute to the discriminative class representation learning.
On the other hand, it extends the fashion of optimizing class weights from one perspective ( i.e ., scale) to two ( i.e ., scale and direction) during backpropagation.
In Sec.3.3, we prove theoretically that E-CRF outperforms other CRF-based methods on eliminating the BCWC problem by optimizing both direction and scale of the disturbing gradient of class weights.
However, during this process, the noise information can also have a direct inﬂuence on the class weights (likely to hinder the optimization for the BCWC problem).
In addition, E-CRF adopts superpixel Ren & Malik (2003) as an auxiliary and leverage its local prior to [Figure] suppress the noise and further strengthen the reliability of the message passing to the boundary pixels.
Superpixel groups adjacent pixels that share similar characteristics to form a block.
It is prone to achieve clear and smooth boundaries and increases the potential for higher segmentation performance.
In E-CRF, we average the deep feature representation of all inner pixels in the same superpixel block and then add this local object prior to each pixel back to enhance the representation of boundary pixels.
In this work, we explicitly propose the BCWC problem in semantic segmentation and an effective approach to alleviate it.
We conduct extensive experiments on three challenging semantic segmentation benchmarks, i.e ., ADE20K Zhou et al.
, Cityscapes Cordts et al.
(2016), and Pascal
Context Mottaghi et al.
(2014), and yeild impressive results.
For example, E-CRF outperforms baselines (DeeplabV3+ Chen et al.
(2018a) with ResNet-
He et al.
by 1.42 % mIoU on ADE20K and 0.92 % mIoU on Cityscapes in single scale.
In addition, we make an exhaustive theoretical analysis in Sec.3.3 to prove the effectiveness of E-CRF.
R ELATED W ORK
Semantic Segmentation.
Fully convolutional network (FCN)
Long et al.
(2015) based methods have made great progress in semantic segmentation by leveraging the powerful convolutional features of classiﬁcation networks He et al.
; Huang et al.
(2017) pre-trained on large-scale data Russakovsky et al.
There are several model variants proposed to enhance contextual aggregation.
For example, DeeplabV2 Chen et al. (2017a) and DeeplabV3 Chen et al.
(2017b) take advantage of the astrous spatial pyramid pooling (ASPP) to embed contextual information, which consists of parallel dilated convolutions with different dilated rates to broaden the receptive ﬁeld.
Inspired by the encoder-decoder structures Ronneberger et al.
; Ding et al.
(2018), DeeplabV3+ Chen et al.
(2018a) adds a decoder upon DeeplabV3 to reﬁne the segmentation results especially along object boundaries.
With the success of self-attention mechanism in natural language processing,
Non-local Wang et al.
ﬁrst adopts self-attention mechanism as a module for computer vision tasks, such as video classiﬁcation, object detection and instance segmentation.
A 2 NetChen et al.
(2018b) proposes the double attention block to distribute and gather informative global features from the entire spatio-temporal space of the images.
Conditional Random Fields.
Fully connected CRFs have been used for semantic image labeling in
Payet & Todorovic (2010); Toyoda & Hasegawa (2008), but inference complexity in fully connected models has restricted their application to sets of hundreds of image regions or fewer.
To address this issue, densely connected pairwise potentials Krähenbühl & Koltun (2011) facilitate interactions between all pairs of image pixels based on a mean ﬁeld approximation to the CRF distribution.
Chen et al. Chen et al.
(2014) show further improvements by post-processing the results of a CNN with a
CRF.
Subsequent works Lin et al.
; Arnab et al.
; Zheng et al.
(2015) have taken this idea further by incorporating a CRF as layers within a deep network and then learning parameters of both the CRF and CNN together via backpropagation.
In terms of enhancements to conventional
CRF models, Ladicky et al.
Ladický et al.
(2010) propose using an off-the-shelf object detector to provide additional cues for semantic segmentation.
Superpixel.
Superpixel Ren & Malik (2003) is pixels with similar characteristics that are grouped together to form a large block.
Since its introduction in 2003, there have been many mature algorithms Achanta et al.
; Weikersdorfer et al.
; Van den Bergh et al.
Owing to their representational and computational efﬁciency, superpixels are widely-used in computer vision algorithms such as target detection Shu et al.
; Yan et al. (2015), semantic segmentation Gould et al.
; Sharma et al.
; Gadde et al.
(2016), and saliency estimation He et al.
Perazzi et al.
Yan et al. Yan et al.
(2015) convert object detection problem into superpixel labeling problem and conducts an energy function considering appearance, spatial context and numbers of labels.
Gadde et al. Gadde et al.
(2016) use superpixels to change how information is stored in the higher level of a CNN.
In He et al.
, superpixels are taken as input and contextual information is recovered among superpixels, which enables large context to be involved in analysis.
M ETHOD 3. R EVISITING C ONDITIONAL R ANDOM F IELD (CRF)
CRF is a typical discriminative model suitable for prediction tasks where contextual information or the state of the neighbors affects the current prediction.
Nowadays, it is widely adopted in the semantic segmentation ﬁeld Chen et al.
Ding et al.
CRF utilizes the correlation between original image pixels to reﬁne the segmentation results by modeling this problem as the maximum a posteriori (MAP) inference in a conditional random ﬁeld (CRF), deﬁned over original image pixels.
In practice, the most common way is to approximate CRF as a message passing procedure among pixels and it can be formulated as:
[Equation]
where Y i and Y ∗ i are deﬁned as the classiﬁcation scores of CNN model and CRF respectively for pixel i , Z i is the normalization factor known as the partition function, and ψ u ( i ) is a unary function which often adopts Y i as the default value.
G is the associated pixel set with pixel i .
For example,
DenseCRF Krähenbühl & Koltun (2011) takes all other pixels except pixel i itself as the set G .
Moreover, the pairwise function ψ p ( i, j ) is deﬁned to measure the message passing weight from pixel j to pixel i .
It is formulated as:
[Equation]
where µ ( i, j ) is a label compatibility function that introduces the co-occurrent probability for a speciﬁc label pair assignment at pixel i and j , while k ( f i , f j ) is a set of hand-designed Gaussian kernels, f i and f j are feature vectors of pixel i and j in any arbitrary feature space, such as RGB images.
w ( m ) is the corresponding linear combination weight for each Gaussian kernel.
When dealing with multi-class image segmentation, M =2 is a common setting.
Then, k ( f i , f j ) is carefully designed as contrast-sensitive two-kernel potentials, deﬁned in terms of color vectors ( I i , I j ) and position coordinates ( p i , p j ) for pixel i and j respectively:
[Equation]
The appearance kernel is inspired by the observation that nearby pixels with similar colors are more likely to share the same class.
θ α and θ β are scale factors to control the degree of these two elements, i.e ., similarity and distance between two pixels.
Apart from this, the smoothness kernel further removes the inﬂuence of some small isolated regions Krähenbühl & Koltun (2011) and θ γ is the associated scale factor.
Notably, all these parameters are learnable during the model training.
Unfortunately, current CRF-based methods Chen et al.
(2014; 2017a)
; Lin et al. (2015); Liu et al.
(2015) for semantic segmentation always adopt CRF as a post-processing module.
For example,
Vanilla-CRF Chen et al.
(2014; 2017a) utilizes CRF to reﬁne segmentation scores ofﬂine, which has no impacts on BCWC since the CNN network and CRF are treated as two separate modules.
Joint-CRF Lin et al.
; Liu et al.
; Lin et al.
(2016) works in a similar way although CRF involves in the backpropagation of CNN networks, restricting its ability to relieve BCWC.
3. E MBEDDED CRF
To solve the BCWC problem in a more intrinsical way, we propose a novel method named Embedded
CRF (E-CRF) to tackle the tough problem via fusing the CRF mechanism into the CNN network as an organic whole for more effective end-to-end training.
An overview of E-CRF can be found in
Fig.2 and we formulate its core function based on Eq.(1) as:
[Equation]
Speciﬁcally, the ﬁrst two terms are analogous to Eq.(1) but we perform CRF mechanism on the high-level features.
F i stands for the original output of feature extractors for pixel i , ψ fu ( i ) and ψ fp ( i, j ) play the same role as they do in Eq.(1).
ψ fu ( i ) takes F i as the default value.
In addition, we reformulate ψ fp ( i, j ) to perform message passing between pixel pairs in the high-level feature:
[Equation]
It is worth noting that k ( f i , f j ) is no longer hand-designed Gaussian kernels as it is in Eq.(1) but simple convolution operators instead to make the whole model more ﬂexible for end-to-end training and optimization.
Experiments in Sec.4 prove this modiﬁcation is a more suitable choice:
[Equation]
where [ x, y ] denotes the concatenation operator.
Different from Eq.(3), we normalize the input image
I into the range [0 , 1] to eliminate the scale variance between pixels and we replace original absolute position coordinates p with cosine position embeddings Vaswani et al.
(2017) to make it more compatible with CNN networks.
E-CRF encodes the appearance and position of pixels into more discriminative tokens via the ﬂexible convolution operation, then the dot product is adopted to measure the similarity between pixel pairs.
As indicated in Eq.(6), E-CRF intends to make nearby pixel pairs that share same appearance to achieve higher k ( f i , f j ) .
Its intention is the same as Eq.(3).
Correspondingly, we also adjust µ f ( i, j ) as the feature compatibility to measure the co-occurrent probability of F i and F j :
[Equation]
Another component in Eq.(4) is F Si .
It relies on the superpixel algorithm Ren & Malik (2003);
Weikersdorfer et al.
; Van den Bergh et al.
; Gadde et al.
(2016) to divide the whole image
I into several non-overlapping blocks.
Pixels in the same superpixel block tend to share the same characteristics.
Thus, we adopt this local object prior to achieve the more effective message passing between pixels in the high-level feature space.
Concretely, we design F Si as:
[Equation]
Q is the associated superpixel block that contains pixel i and ψ f s ( l ) devotes the re-weighting factor for the deep features of pixel l in Q .
We adopt ψ f s ( l ) = 1 n and n is the total number of pixels in
Q .
F Si serves as a supplement in Eq.(4) to add the local object prior to each pixel back, which increases the reliability of message passing in E-CRF.
What’s more, superpixel Gould et al.
Sharma et al.
; Gadde et al.
(2016) always tends to generate clearer and smoother boundary segmentation results than traditional CNN networks or CRFs do, which also increases the potential for more accurate segmentation results.
Detailed experiments can be found in Sec.4.
2 W 1 W *1 W 2 W 1 W *1 W 2 W 1 W *1 W 1  (a) (b) (c) 2  3  k F k F k F * k F j F
W  W 
W  3. H OW E-CRF R ELIEVES
[Figure] In this section, without loss of generality, we take the simple two-class segmentation problem as an example to dive into the principle of E-CRF from the perspective of gradient descent.
Suppose F k is the feature vector for a foreground boundary pixel k and its prediction probability is P k .
Then, the typical cross-entropy loss L k can be deﬁned as:
[Equation]
where W is the class weight matrix and its gradient variation ∇ W can be formulated as:
[Equation]
Through Eq.(9), (10) and (11), class weights in the next iteration will be updated 2 :
[Equation]
As shown in Eq.(12), the direction of the gradient descent in pixel k keeps the same as F k while the magnitude of the gradient is decided by P k .
What happens if we integrate the CRF into the segmentation pipeline?
As we have discussed in Sec.1, Vanilla-CRF has nothing to do with the optimization process of CNN networks, while if we adopt Joint-CRF, ∇ W can be reformulated as:
[Equation]
where ˆ P k is the reﬁned score by CRF, w j is the message passing weight from pixel j to pixel k and
P j is the original score of pixel j .
In general, boundary pixel k is hard to classify correctly due to the confusing features from both sides.
Thus the original probability P k is always small.
In contrast, other inner pixels of the same object are easy to recognize and tend to achieve a higher probability.
Consequently, ˆ P k is usually larger than P k and disturbing gradients caused by pixel k will be relieved to some extent, which makes inter-class distance further as shown in Fig.3(b).
However, Eq.(13) only adjusts the scale of the gradient descent while the direction still keeps the same as F k , which weakens its effects for better representation learning.
When it comes to our proposed E-CRF, ∇ W can be further deﬁned as:
[Equation]
where F ∗ k is the reﬁned feature representations by E-CRF, and P ∗ k is the reﬁned score which is analogous to ˆ P k in Eq.(13).
Comparing with Joint-CRF, it is clear that E-CRF not only changes the scale of the gradient descent but also adjusts its optimization direction. The optimization process is directly applied to the class weight matrix (in the ﬁnal layer), which opens up room for more discriminative class weights.
In other words, we can adjust the class weight from both the scale and direction to make the class weights more discriminative to decrease the class weights similarity (or class weights confusion).
As depicted in Fig.3(c), assume W 1 is the class weight vector that pixel k belongs to, while W 2 is the other one which has a higher co-occurrent probability with Win the same image.
E-CRF designs an effective message passing procedure to purify the feature representation of boundary pixels assisted by inner pixels from the same object ( F j in Fig.3(c)).
In this way, it relieves the inﬂuence of disturbing gradients and makes the inter-class distance between
W 1 ( W ∗ 1 ) and W 2 further, which means more discriminative feature representations.
E XPERIMENT 4. I MPLEMENTATION D ETAILS
We follow the previous works Chen et al.
; He et al.
(2019b)
; Chen et al.
(2018a)
and perform experiments on three challenging semantic segmentation benchmarks, i.e ., ADE20K Zhou et al.
(2017), Cityscapes Cordts et al.
and
Pascal Context Mottaghi et al.
Due to the space limit, a detailed description of these three datasets can be found in our Appendix A .
We adopt
DeeplabV3+ Chen et al.
(2018a)
with ResNet He et al.
(2016) pretrained on ImageNet Russakovsky et al.
(2015) as our baseline to implement E-CRF.
The detailed information follows standard settings in
Chen et al.
(2014; 2018a)
and we add it into our Appendix A .
Specially, we employ SLIC Achanta et al.
(2012), a common superpixel segmentation algorithm, to divide each image of ADE20K, Cityscapes and Pascal Context into 200, 600, and 200 blocks respectively.
Note that the superpixel is generated ofﬂine.
To verify the effectiveness of our approach for semantic segmentation, we adopt two common metrics in our experiments, i.e ., class-wise mIoU to measure the overall segmentation performance and 1-pixel boundary F-score Takikawa et al.
(2020) to measure the boundary segmentation performance.
4. A BLATION S TUDY 4.2. C OMPARISONS WITH R ELATED M ETHODS
[Table]
As shown in Table 1, we compare our proposed E-CRF with other traditional CRF-based methods, i.e ., Vanilla-CRF and Joint-CRF.
First of all, it is clear that all the CRF-based methods outperform the baseline model by a large margin, which well veriﬁes the main claim in Chen et al.
(2014; 2017a);
Lin et al. (2015); Liu et al.
; Lin et al. (2016) that CRF is beneﬁcial to boundary segmentation (F-score).
What’s more, E-CRF achieves the best result among all those methods, which surpasses the baseline model with up to 1.48% mIoU and 2.20% F-score improvements.
E-CRF fuses the CRF mechanism into the CNN network as an organic whole.
It relieves the disturbing gradients caused by the BCWC problem and adjusts the feature representations to boost the overall segmentation performance and the boundary segmentation.
Fig.1(a) also proves that E-CRF can decrease the interclass similarity consistently which results in more discriminative feature representations.
Experiments on Cityscapes dataset can be found in our Appendix A .
4.2. A BLATION ON M ESSAGE P ASSING S TRATEGIES
As we have discussed in Sec.3.2, two message passing components, i.e ., pairwise module ψ fp and superpixel-based module ψ fs , play vital roles in our proposed E-CRF.
Table 2 shows that ψ fp and ψ fs can boost the overall segmentation performance on ADE20K val dataset with up to 1.19% mIoU and 1.25% mIoU gains when integrated into the baseline model respectively.
Moreover, if we fuse them as a whole into E-CRF, they can further promote the segmentation performance by up to 1.48% mIoU improvements.
We also compare with Non-local Wang et al.
(2018), another famous attention-based message passing method, into our experiments for comprehensive comparisons even though it actually has different design concepts from ours.
Unfortunately, we ﬁnd that although Non-local achieves improvements over the baseline, it is still inferior to our E-CRF.
[Table]
4.2. A BLATION OF S UPERPIXEL N UMBERS
We follow standard settings in our paper and take DeeplabV3+ based on ResNet-50 as the baseline model to present the performance of E-CRF under different superpixel numbers.
Detailed comparisons on ADE20K dataset are reported in Table 3 and SP denotes S uper P ixel.
As shown in Table 3, different numbers of superpixels indeed affect the performance of E-CRF.
[Table]
Intuitively, when the number of superpixels is 200, E-CRF acquires the best performance as it achieves a better trade-off between the superpixel purity and the long-range dependency.
Moreover, it is worth noting that when the pairwise message passing strategy ( i.e ., ψ fp ) is also adopted in E-CRF, it becomes more robust to the different numbers of superpixels that may introduce noise, as our adaptive message passing mechanism (including ψ fp and ψ fs ) can be compatible with the variance.
4.2. D IFFERENT B OUNDARY R EFINEMENT
We consider three typical methods inlcuding Segﬁx Yuan et al.
(2020), DecoupleSegNet Li et al.
(2020b), and ABL 5 Wang et al.
They reﬁne boundary segmentation via post-processing, improving boundary representation, and adding boundary allignment loss respectively.
But all of them ignore the existence of BCWC issue, which may restrict their capability.
In Table 4, we use DeeplabV3+ with ResNet101 as our baseline method. For Segﬁx, we use the ofﬁcial code (It uses HRNet as backbone) to boost the performance of DeeplabV3+.
For DecoupleSegNet which is constructed based on DeeplabV3+, we also use the ofﬁcial code (It uses ResNet101 as backbone). All the models are trained on ADE20K for 80K iterations with batch size set to 16. When testing, we adopt the single-scale testing strategy (i.e., raw image) because using a single scale (i.e., raw images) when comparing with baselines or performing ablation studies is a traditional default setting in the semantic segmentation ﬁeld. The goal is to eliminate the effect of other elements (e.g., image augmentation).
[Table]
It is observed that all the methods enhance the performance and E-CRF achieves the highest mIoU and F-score.
We speculate that this is because E-CRF has explicitly considered
the
BCWC problem and optimizes the class weights from both scale and direction aspects while reﬁning boundary representation.
It also indicates the importance of obtaining distinguishable class weights in semantic segmentation.
4. C OMPARISONS ON C OMPUTATIONAL C OSTS
We take DeeplabV3+ based on ResNet101 as the baseline model to perform the training time comparisons.
Image size is set to 512 × 512 and all the experiments are conducted on 8 GeForce
RTX 2080Ti GPUs with two images per GPU.
The FLOPs, parameter size, and inference FPS are also reported in Table 5.
We can ﬁnd that our proposed E-CRF brings negligible extra costs over the baseline model.
The cost difference between E-CRF and Joint-CRF is marginal.
We also measure the time consuming of superpixel method ( 5ms ), which is much smaller than that of inference ( 55ms ).
[Table]
C OMPARISONS WITH SOTA M ETHODS
In this research, we mainly focus on the Boundary-caused Class Weight Confusion (BCWC) in CNN models.
Hence, in this section, we choose CNN-based methods for fair comparisons.
ADE20K.
We ﬁrst compare our E-CRF (ResNet101 as backbone) with existing methods on the
ADE20K val set.
We follow standard settings in Huang et al.
; Yuan et al.
Zhu et al.
(2021) to adopt multi-scale testing and left-right ﬂipping strategies.
Results are presented in Table 6.
It is shown that E-CRF outperforms existing approaches.
Segmentation visualization is presented in our Appendix
Cityscapes.
To verify the generalization of our method, we perform detailed comparisons with other SOTA methods on Cityscapes val and test set.
Multi-scale testing and left-right ﬂipping strategies are also adopted.
The results with ResNet101 as backbone are reported in Table 6.
Remarkably, our algorithm achieves 82.74% mIoU in val set and outperforms previous methods by a large margin.
Pascal Context.
To further verify the generalization of E-CRF (ResNet101 as backbone), we compare our method with other SOTA method on Pascal Context dataset as shown in Table 6.
We adopt multi-scale testing and left-right ﬂipping strategies as well.
The result suggests the superiority of our method.
D ISCUSSION
In this section, we want to discuss the difference between E-CRF and three related works including PCGrad Yu et al. (2020b), OCNet Yuan & Wang (2018), and SegFix Yuan et al.
Difference between Projecting Conﬂicting Gradients (PCGrad) and E-CRF: PCGrad is designed to mitigate a key optimization issue in multi-task learning caused by conﬂicting gradients, where
[Table]
gradients for different tasks point away from one another as measured by a negative inner product.
If two gradients are conﬂicting, PCGrad alters the gradients by projecting each onto the normal plane of the other, preventing the interfering components of the gradient from being applied to the network.
The idea behind PCGrad is simple and the method is effective.
PCGrad is a task-level gradient optimization method, mainly focusing on conﬂicting gradients caused by multiple tasks during training (e.g., in semantic segmentation and depth estimation).
E-CRF is a ﬁner-grained pixel-level gradient optimization method.
E-CRF mainly aims at mitigating the boundary-caused class weights confusion in semantic segmentation via adjusting class weights from both scale and direction.
Difference between OCNet and E-CRF: OCNet uses self-attention to implement the object context pooling module.
The object context pooling estimates the context representation of each pixel i by aggregating the representations of the selected subset of pixels based on the estimated dense relation matrix.
Further, OCNet combines context pooling module with the conventional multi-scale context schemes including PPM and ASPP.
In E-CRF, we follow the idea behind Conditional Random Filed and embed it from logit space to deep-feature space.
We instance the unary function and reformulate the pairwise function with a convolutional-based kernel.
The kernel takes raw image RGB value and relative position embeddings as inputs (See Eq. 6), which is different from self-attention that takes extracted deep features as inputs.
We also maintain one special term in CRF called label compatibility and transfer it to feature compatibility (See Eq. 7).
Such is missing in self-attention.
Besides, we do not measure the similarity between superpixel center and boundary pixels.
We simply leverage the local prior in superpixel and use it to guide deep feature averaging.
The motivation is to suppress noise information.
Finally, the motivation between OCNet and E-CRF is different.
OCNet mainly focuses on integrating as much object context as possible while E-CRF explicitly targets on the
BCWC problem and optimizes class weights from both scale and direction.
Difference between SegFix and E-CRF: SegFix ﬁrst encodes input image and predicts a boundary map and a direction map.
Then SegFix uses the predicted boundary map and offset map derived from the direction map to correct the wrongly classiﬁed boundary pixels via internal points with high conﬁdence.
SegFix is beneﬁcial for reﬁning boundary segmentation and is served as a postprocessing method, thus lacking the capability to alleviate BCWC problem.
Our method is derived from traditional CRF (a post-processing method) and can be regarded as a plug-and-play module.
It can be easily integrated with other methods.
Besides, our method extends the optimization ﬂexibility for BCWC problem.
Empirically, we have compared Segﬁx and E-CRF based on DeeplabV3+ in
Table 4 (Please see Sec.4.2.4).
E-CRF produces 46.02% for mIoU on ADE20K, outperforming
SegFix (45.62%) by 0.4%. This indicates the importance of alleviating BCWC problem.  C ONCLUSION AND F UTURE W ORKS In this paper, we focus on the particularity of class weights in semantic segmentation and explicitly consider an important issue , named as Boundary-caused Class Weights Confusion (BCWC).
We dive deep into it and propose a novel method, E-CRF , via combining CNN network with CRF as an organic whole to alleviate BCWC from two aspects ( i e ., scale and direction).
In addition, we make an exhaustive theoretical analysis to prove the effectiveness of E-CRF.
Eventually, our proposed method achieves new results on ADE20K, Cityscapes and Pascal Context datasets.
There are two important directions for future research.
In this work, we use SLIC, a common algorithm for fast implementation.
There exist many other superpixel algorithms such as graphicalbased Felzenszwalb & Huttenlocher (2004) and CNN-based Jampani et al.
(2018) that may give better boundary results for objects.
Therefore how these different methods inﬂuence the performance in our framework is interesting.
Besides, We ﬁnd that transformer-based networks suffer from BCWC issue as well and make a preliminary exploration.
More works are expected to focus on this issue.
A A PPENDIX A. F ORMULA D ERIVATION
Firstly, we give the gradient equation of ∇ W mentioned in this paper:
[Equation]
According to it, we present the derivative of each term respectively:
[Equation]
and
[Equation]
[Equation]
So after integrating Eq.
(17), Eq.
(18), and Eq.
(19), Eq.
(16) can be formulated as:
[Equation]
Further, it is worth noting that P k is given by:
[Equation]
To simplify Eq.
(20), we take Eq.
(21) into account.
Thus, Eq.
(20) is formulated as:
[Equation]
[Equation]
Finally, the class weights W ∗ in the next iteration will be updated:
[Equation]
A. E XPERIMENT S ETUP
ADE20K ADE20K Zhou et al.
(2017) is one of the most challenging benchmarks, containingﬁne-grained semantic concepts and a variety of scenes with 1,038 image-level labels.
There are 20210 images in training set and 2000 images in validation set.
Cityscapes Cityscapes Cordts et al.
(2016) has 5,000 images captured from 50 different cities.
Each image has 2048 × 1024 pixels, which have high quality pixel-level labels of 19 semantic classes.
There are 2,975 images in training set, 500 images in validation set and 1,525 images in test set.
We do not use coarse data in our experiments.
Pascal Context PASCAL Context Mottaghi et al.
(2014) is a challenging scene understanding dataset, which provides the semantic labels for the images.
There are 4, 998 images for training and 5,images for validation on PASCAL Context dataset.
In our experiment, the 59 most frequent categories are used for training.
Implementation Details.
The initial learning rate is set as 0.01 for both datasets.
We employ a poly learning rate strategy where the initial learning rate is multiplied by (1 − iter/totaliter )
.after each iteration.
We set training time to 80000 iterations for ADE20K and Pascal Context, and 180 epochs for Cityscapes.
Momentum and weight decay coefﬁcients are set as 0.9 and 0.0005, respectively.
For data augmentation, we apply the common scale (0.5 to 2.0), cropping and ﬂipping of the image to augment the training data.
Input size for ADE20K dataset is set to 512 × 512, and 480 × 480 is for Pascal Context while input size for Cityscapes dataset is set to 832 × 832.
The syncBN Peng et al.
(2018) is adopted in all experiments, and batch size on ADE20K and Pascal
Context is set to 16 and it is set to 8 for Cityscapes.
A. C OMPARISONS WITH R ELATED M ETHODS ON C ITYSCAPES
To further evaluate our proposed method
[Table] we thoroughly compare our approach with baseline and other traditional CRF-based methods, i [Table] e ., Vanilla-CRF and Joint-CRF on Cityscapes dataset.
As shown in Table [Table] E-CRF achieves the best result among all those methods, which outperforms the baseline model by 0.92% in mIoU and 3.81% in F-score respectively.
Obviously, our method is more effective than both Vanilla-CRF and Joint-CRF.
[Table]
A. P AIRWISE M ESSAGE P ASSING V ISUALIZATION
E-CRF takes an equivalent transformation to replace hand-designed Gaussian kernels in VanillaCRF with simple convolution operators for more ﬂexible end-to-end optimization.
The convolution operation involves two aspects.
One is the appearance similarity and the other one is the relative position between pixels.
As depicted in Fig.4(a), we take a pixel k in the stool for an example and show its relationship with other pixels in the image.
Fig.4(b) and Fig.4(c) show its appearance similarity and cosine position embedding with other pixels respectively.
It is clear that pixels share similar colors or close to the target pixel k tend to be highlighted.
Subsequently, in Fig.4(d), we directly visualize the results of our pairwise message passing module ψ fp deﬁned in Eq.(5).
We can ﬁnd that ψ fp becomes concentrated on the most relevant pixels compared with the pixel k , which veriﬁes the reliability of our pairwise message passing design.
[Figure]
B BCWC IN T RANSFORMER
Are transformer-based models also suffering from Boundary-caused Class Weight Confusion?
Is our method effective to transformer-based models?
To answer these questions, we make a preliminary exploration in this section.
B. O BSERVATIONS ON ADE20K
[Figure]
Following the same idea in Fig.1(a) of this paper, we take Segformer Xie et al.
(2021) (a transformer-based segmentation model) as an example to train on ADE20K Zhou et al.
(2017) dataset.
We count the number of adjacent pixels for each class pair and ﬁnd a corresponding category that has the most adjacent pixels for each class.
Then, we calculate the similarity of their class weights and depict it in Fig.
X-axis stands for the number of adjacent pixels for each class pair in descending order, and Yaxis represents the similarity of their class weights.
Blue line denotes Segformer while orange line denotes ECRF based on Segformer.
As shown in Fig. 6, two categories that share more adjacent pixels are inclined to have more similar class weights, while E-CRF effectively decreases the similarity between adjacent categories and makes their class weights more discriminative.
These observations on transformer-based model are quite similar to previous results in CNN-based models.
Apparently, transformer-based models are also suffering from Boundary-caused Class Weight Confusion.
B. E FFECTIVENESS ON T RANSFORMER
To evaluate the effectiveness of our method, we take Segformer Xie et al.
(2021) (based on MiT-B5) as our transformer baseline and incoporate E-CRF into it.
Experiments are conducted on ADE20K and Cityscapes datasets.
Similarly, we also compare our method with other traditinal CRF-based methods, i.e ., Vanilla-CRF and Joint-CRF 7 .
As shown in Table 8, E-CRF achieves the best result among all those methods, which surpasses the baseline model with up to 1.01% mIoU and 3.81%
F-score improvements.
By E-CRF relieves the disturbing gradients caused by the BCWC problem boost the overall segmentation performance and the boundary segmentation.
Fig.6 also proves that
E-CRF can decrease the inter-class similarity consistently which results in more discriminative feature representations.
[Table]
B. C OMPARISONS WITH SOTA M ETHODS
To further verify the effectiveness, we compare our methods with other transformer-based SOTA methods with similar number of parameters (except SETR) for fair comparisons in both ADE20K and Cityscapes datasets.
Multi-scale testing and left-right ﬂipping strategies are adopted.
As shown in Table 9, our method achieves the best results among all the SOTA methods in both ADE20K and
Cityscapes datasets.
Besides, our method also has the smallest number of parameters.
[Table]