S WIN ZS3: Z ERO -S HOT S EMANTIC S EGMENTATION WITH A
S
WIN
T RANSFORMER
Anonymous authors
Paper under double-blind review
A BSTRACT
Zero-shot semantic segmentation (ZS3) aims at learning to classify the never-seen classes with zero training samples.
Convolutional neural networks (CNNs) have recently achieved great success in this task.
However, their limited attention ability constraints existing network architectures to reason based on word embeddings.
In this light of the recent successes achieved by Swin Transformers, we propose SwinZS3, a new framework exploiting the visual embeddings and semantic embeddings on joint embedding space.
The SwinZS3 combines a transformer image encoder with a language encoder.
The image encoder is trained by pixeltext score maps using the dense language-guided semantic prototypes which are computed by the language encoder.
This allows the SwinZS3 could recognize the unseen classes at test time without retraining.
We experiment with our method on the ZS3 standard benchmarks (PASCAL VOC and PASCAL Context) and the results demonstrate the effectiveness of our method by showing the state-of-art performance.
I NTRODUCTION
Semantic segmentation is at the foundation of several high-level computer vision applications such as autonomous driving, medical imaging, and so on.
Recent deep learning has achieved great success in semantic segmentation Chen et al.
Long et al. (2015); Ronneberger et al.
Zhao et al.
(2017b).
However, the fully-supervised semantic segmentation models usually require extensive collections of labeled images with pixel-level annotations.
And it could only handle the pre-deﬁned classes.
Considering the high cost of collecting dense labels, recently weakly supervised semantic(WSSS) segmentation methods have been explored.
WSSS are often based on easily obtaining annotations, such as scribbles Sun et al.
,bounding boxes Dai et al.
(2015), and image-level labels
Hou et al.
Among them, a popular trend is based on network visualization techniques like classiﬁcation activation map generating pseudo ground-truths Zeiler & Fergus (2014); Zhang et al.
However, these methods also require the networks to have labeled images.
On the contrary, humans could recognize novel classes with only descriptions.
With the inspiration of this, some recent methods seek zero-shot semantic segmentation (ZS3) Zhao et al. (2017a);
Bucher et al.
; Gu et al.
; Li et al.
ZS3 beneﬁts from semantic level supervision from texts by exploiting the semantic relationships between the pixels and the associated texts, which makes it enjoy a cheaper source of training data.
The ZS3 methods could be categorized into generative and discriminative methods Baek et al.
Both could predict unseen classes using only language-guided semantic information of the corresponding classes.
For the generative
ZS3 methods Creswell et al.
(2018); Kingma & Welling (2013), segmentation networks are ﬁrst trained with only seen classes labeled data.
Then, they freeze the feature extractor to extract seen classes’ visual features and train a semantic generator network to translate the language embedding to visual space.
By doing this, the semantic generator could generate visual features conditioned on language embedding vectors.
Finally, a classiﬁer is trained to classify the features combined with features produced by the feature extractor on seen classes and generated features produced by the semantic generator from language embeddings on unseen classes.
With generative methods achieve impressive performance in zero-shot semantic segmentation tasks.
The methods are limited by a multi-stage training strategy, and the visual features extracted from the feature extractor not considering the language information during training.
This will cause a seen bias problem towards the visual and generated features.
[Figure]
Improving the network backbone of zero-shot networks is another effective way to reduce the bias problem.
As shown in Figure.1 We argue that a shared shortcoming of previous ZS3 models falls in the reduced receptive ﬁeld of CNNs and less using attention mechanisms for extracting the global relations of visual features conditioned with language semantic information.
The local nature of convolutions leads CNNs to extract visual features missing long-range relationships across the same image.
CNNs based frameworks sometimes fail to extract language-guided activation ﬁelds for lacking the global perceiving attention mechanism.
Recently, transformers Vaswani et al.
(2017) have signiﬁcant breakthroughs in both of natural language processing(NLP) and computer vision(CV) ﬁeld Xie et al.
(2021a)
; Zheng et al.
(2021); Arnab et al.
Dosovitskiy et al.
(2020) (ViT) is the ﬁrst work to apply the transformer architecture to image classiﬁcation.
Moreover, the Liu et al.
(2021) (swin-transformer) presents a new architecture for more general-purpose vision tasks, especially dense predicting.
We argue that the self-attention mechanism beneﬁts the zero-shot semantic segmentation tasks and the semantic information supervisor.
The transformer-based model could capture the global feature relations and the semantic information in visual features by multi-head self-attention(MHSA).
This paper takes this missing step and explores the swin-transformer for ZS3.
It combines convolutional layers and transformer blocks to model the global information guided by pixel-text distances and score maps.
We also improve the decision boundary by modifying the Nearest Neighbor(NN)
Classiﬁer with weighted euclidean distance by score map.
We demonstrate the effectiveness of our approach on standard zero-shot semantic segmentation benchmarks, achieving state-of-the-art performance on PASCAL-VOC Everingham et al.
and
PASCAL-Context Mottaghi et al.
Some methods based on CLIP (Radford et al.
(2021)Xu et al.
(2022)Ding et al.
(2022)Arnab et al.
Xu et al.
(2021)) often claim to be zero-shot learning methods.
However, those methods usually use all classes of images and text labels during training, which will cause supervision leakage.
R ELATED W ORK
Semantic segmentation: Semantic segmentation has made great advancements due to the rise of deep learning.
Most recent state-of-the-art models are based on fully convolutional neural networks
Long et al.
and assume that all the training data have pixel-level annotations.
DeepLab
[Figure]
exhibited distinct performance improvement on the PASCAL VOC2012 Everingham et al.
and
MS-COCO Lin et al.
by using the multiple scales Li et al.
; Zhang et al.
and dilated convolution Chen et al.
Other methods like UNet Ronneberger et al.
and
SegNet Badrinarayanan et al.
(2017) also achieve an impressing performance by different strategies.
The ViT Dosovitskiy et al.
(2020) is the ﬁrst work to use transformer architecture for recognition task.
And swin transformer extends transformer to the dense prediction task and achieved the stateof-the-art performance.
But they all heavily rely on expensive pixel-level segmentation labels, and they assume that training data for all categories is available beforehand.
Recently, lots of weakly supervised semantic segmentation (WSSS) methods which are based on easily obtained annotations such as bounding boxes Dai et al.
(2015), scribbles Sun et al.
and image-level labels Hou et al.
(2018) have been explored.
The key to prevailing pipeline WSSS is to generate the pseudo-labels, especially network visualization techniques like CAM.
And some works use growing strategies to grow the CAM ground-truth regions to the entire objects .
But it is still difﬁcult to get the pseudolabels revealing entire object areas with accurate boundaries for the ill-posed procedure Singh &
Lee (2017); Li et al.
Zero-shot semantic segmentation : The ZS3 networks could be categorized into discriminative and generative methods.
For the discriminative approach, the work of Zhao et al.
(2017a) focuses on hierarchical predicting unseen classes by adopting the discriminative approach.
SPNet Xian et al.
(2019) exploits a semantic embedding space by mapping visual features to ﬁxed semantic ones.
JoEm Baek et al.
(2021) propose to align visual and semantic featuers in joint embedding space.
In contrast to discriminative methods, ZS3Net Bucher et al.
(2019) synthesize visual features by a generative moment matching network (GMMN).
However, the ZS3Net training pipeline consists of three stages that will cause the bias problem.
CSRL Li et al.
(2020) exploit the relations of both seen and unseen classes to preserve them to synthesized visual features.
CaGNet Gu et al.
(2020) proposes to use the channel-wise attention mechanism in dilated convolutional layers for extracting visual features.
Visual-language learning : Recently years, image-language pairs learning is a rapidly growing ﬁeld.
There are some representative works such as CLIP Radford et al.
(2021) and ALIGN Jia et al.
(2021) which are pretrained on hundreds of millions of image-language pairs.
Yang et al.
(2022) presents the uniﬁed contrastive learning method that can leverage both the image-language methods and image-label data.
And our work further extends the method to pixel-level on ZS3.
[Figure]
M ETHOD 3. M OTIVATIONS
Unlike supervised semantic segmentation methods, The unseen classes prototypes of discriminative zero-shot semantic segmentation rely on joint optimization of the visual encoder and language encoder.
Thus, to achieve good performance, this formulation requires the network to perceive the language context structure.
Current network Baek et al.
(2021) adopt traditional convolutional layers for aggregating language information.
However, the intrinsic locality and weak attention of the convolution operator can hardly model long-range and accurate visual-language joint features.
So, we propose to use transformer-based blocks to address the limitation.
Another limitation of ZS3 is the seen bias problem.
Lacking the unseen classes labeled data, it is difﬁcult for the visual encoder to extract distinguishable features.
As shown in Fig.3, modulating the decision boundary could also reduce the bias problem.
We analyze the shortcomings of traditional NN classiﬁers and use the new decision boundary to improve the performance.
Driven by this, we design our SwinZS3 framework.
As shown in Fig.2.
3. O VERVIEW
Following the common practice, we divide classes into seen classes S and unseen classes U .
During the training time, we train our model, which includes a visual feature extractor and a semantic prototype encoder with the seen classes set S only.
Zero-shot semantic segmentation aims to allow the model to recognize both seen classes S and unseen classes U in the test time.
We use the visual extractor to extract visual features and input language embeddings ( word 2 vec ) to the language encoder for obtaining semantic prototypes of corresponding classes.
The visual features will be input into a classiﬁer and supervised by ground-truth labels.
The language prototypes will have a regression loss with the visual features.
And the prototypes’ inter-relationship are transferred from the language embeddings like word 2 vec using the semantic-consistency loss.
And then SwinZS3 calculates pixel-text score maps in a hyper-sphere space for the projected visual features and semantic prototypes.
These score maps are also supervised by the ground-truth labels.
The visual features are also fed into a classiﬁer for being supervised by ground truth labels.
In the following, we describe our framework in detail.
3. T RANSFORMER B ACKBONE
Our framework uses the transformer Liu et al.
(2021) as backbone that an input image is split into non-overlapping h × w patches.
The patches will be projected to h × w tokens.
The multi-head selfattention (MHSA) layer is used for the transformer block to capture the global features information.
In MHSA, the patch tokens are projected to queries Q ∈ R hw × d k , keys K ∈ R hw × d k , and values
V ∈ R hw × d v .
h and w are the shapes of feature maps.
d k and d v denote the dimension of features.
Based on Q , K , and V , outputs X are
[Equation] [Equation]
The MHSA is the core operation of the transformer block.
The transformer backbone’s ﬁnal output is produced by stacking multiple transformer blocks.
3. N ETWORK T RAINING
As shown in Fig.2, our framework consists of four loss terms : cross-entropy loss L ce Misra &
Maaten (2020), pixel-wise regression loss L r , pixel-text score map loss L aux , semantic consistency loss L sc .
The overall loss is ﬁnally formulated as
[Equation]
where λ 1 and λ 2 balance the contributions of different losses.
Cross-entropy loss:
Given the ﬁnal outputs of feature maps υ ∈ R h × w × c , where h , w , and c are the height, width, and the number of channels.
Then, υ will be put into a classiﬁer head f c .
For the zero-shot setting, the classiﬁer could learn the seen classes.
So, we apply the cross-entropy loss
Murphy (2012) that is widely adopted in supervised semantic segmentation on the seen classes set
S as follows:
[Equation]
where N c indicates a set of locations labeled as the class c in ground-truth.
Regression loss: Although the l ce could train the model to a discriminative embedding space on seen classes S .
However, the model is not adaptable to classify the unseen classes U while the classiﬁer head does not learn the unseen classes prototypes.
At test time, we want to use the language prototypes of both seen and unseen classes as the classiﬁer to recognize the dense visual features extracted by the transformer backbone.
For this, the distances of visual features and corresponding language prototypes should be minimized in embedding space.
To address it, we introduced regression loss l r .
As the l ce , we ﬁrst get the ﬁnal outputs visual feature maps υ ∈ R h × w × c .
Then, we get semantic feature maps s ∈ R h × w ×
d
where each pixel s c of s is a word or language embedding and the same class with corresponding visual feature pixel.
Given the language embedding maps, we input them to semantic encoder f s as follows:
[Equation]
where µ ∈ R h × w × c .
We denote each pixel of µ c is a semantic prototype for a class c .
So, the regression loss as follows:
[Equation]
[Equation]
where d () is the euclidean distance metric.
R c means the regions labeled with the same class in the ground truth.
The l r give a promise that the dense visual features and semantic prototypes will be projected to a join embedding space where the pixels of corresponding classes will be close.
But, for
ZS3, there are similar limitations with l ce : The l r deal with pixel-wise visual features and semantic
[Figure]
prototypes independently but ignore explicitly considering the other pixels’ relationship between them.
To address it, we proposed to use the contrastive loss.
Pixel-text score map: In our framework, we use the score map to reduce the seen bias problem in
ZS3.
As shown in Fig.4, to get the discriminative joint embedding space, we compute the pixeltext score maps using the language prototypes µ c ∈ R k × c and the ﬁnal outputs of feature maps υ ∈ R h × w × c by:
[Equation]
(6) where ˆ µ c and ˆ υ are the l 2 normalized version of υ and µ c along the channel dimension.
By the way, the µ c in the score map must use the seen classes prototypes only.
Otherwise, it will make the unseen bias problem more serious.
The score maps characterize the results of visual features pixel and language-guided semantic prototypes matching, which is one of the most crucial parts of our
SwinZS3.
First, we use the score maps to compute an auxiliary segmentation loss:
[Equation]
where τ is a temperature coefﬁcient which we set 0 .
07 and y is the ground truth label.
The auxiliary segmentation loss can make the joint embedding space more discriminative which is beneﬁcial to zero-shot semantic segmentation.
Semantic consistency loss: The semantic consistency loss l sc could transfer the relationship of word 2 vec to the semantic prototypes embedding space.
For adopting the pre-trained word embedding features, l sc could keep the class contextual information.
The l sc deﬁnes the relation of prototypes as follows:
[Equation]
where d () is the distance between two prototypes and τ µ is a temperature parameter.
Then for word embedding space, the relationship could be deﬁned:
[Equation]
So, the semantic consistency loss is deﬁned as follows:
[Equation]
The l sc could distill the word embedding contextual information to the prototypes.
3. N ETWORK I NFERENCE
During the inference, we use the semantic prototypes, which are the outputs of the semantic encoder as a NN classiﬁer Cover & Hart (1967).
We compute the euclidean distances and score maps from individual visual features to language prototypes, and classify each visual feature to the nearest language prototypes as follows:
[Equation]
where d is the euclidean distance metric and s is the score map.
For the unseen classes U still being biased towards the seen classes S , we adapted the work of Baek et al.
(2021), which proposed the
Apollonius circle.
The top 2 nearest language prototypes with individual visual features are d 1 and d 2 .
d 1 is the euclidean and score distance with the language prototype µ
and d 2 is the distance with language prototype µ 2 where we denote c 1 and c 2 as the class of µ 1 and µ 2 .
The decision rule is deﬁned with the Apollonius circle as follows:
[Equation] [Equation]
where
[Equation]
We denote Π as an function whose value 1 if the argument is true, and 0 otherwise.
The γ is an adjustable parameter which could modulate the decision boundary.
E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Training:
For transformer backbone, we use the swin transformer (swin-tiny) proposed in Singh &
Lee (2017), which is a baseline for transformer-baed ZS3 task.
To avoid supervision leakage from unseen classes Xian et al.
, the backbone parameters are initialized with self-supervised model
MoBY Xie et al.
(2021b)
pre-trained on Imagenet.
We use an AdamW optimizer as the optimizer to train SwinZS3.
For the backbone, we set the initial learning rate as 1 × 10 − 4 , and it uses the polynomial scheduler to decay at every iteration.
The other parameters’ learning rate is 10 times the backbone parameters’ learning rate.
The weight decay factor is set as 0 .
For data augmentation, we keep the same setting with Baek et al.
For other parameters ( λ , γ ) , we set λ 1 , λ 2 to 0 .and
the γ is 0 .
Dataset split: We perform experiments on PASCAL VOC and PASCAL Context.
The PASCALVOC2012 dataset contains 1464 training images and 1449 validation images with a total of 21 categories (20 object categories and background).
The PASCAL Context dataset contains 4998 training and 5105 validation samples of 60 classes with 59 different categories and a single background class.
Following the common practice, we adopt the 10582 augmented training samples for PASCAL VOC.
For the zero-shot semantic segmentation network, we divide Pascal-VOC2012 training samples according to N-seen and 20-N unseen classes.
For example, we choose cow and motorbike as unseen categories.
Then we ﬁlter out those samples with cow and motorbike labels and train the segmentation network using the remaining samples.
During the training time, the segmentation model should keep the mIOU of unseen classes 0 .
We follow the experiment settings provided by ZS3Net which dividing the Pascal-VOC 2012 training samples 20 object classes into four splits (1) 18-2 classes ( cow , motorbike ), (2) 16-4 classes ( cat , sofa ), (3) 14-6 classes ( boat , fence ), and
[Figure]
[Table]
{+Deeplabv
+}
+ ! ! !
[Table]
{+Deeplabv
+}
+ ! ! !
[Table] {+Deeplabv
 + ! ! ! ! [Table]+}
SwinZS ! ! !
[Table]
SwinZS ! ! !
[Table]
[Table]
(4) 12-8 classes ( bird , tvmonitor ).
Each split contains previous unseen classes gradually.
Then the model is evaluated on the full 1449 validation images.
Evaluation metrics : We use the mean intersection-over-union ( mIoU ) as evaluation metrics Long et al. (2015).
In detail, we separately count the metrics of the seen classes and the unseen classes which are denoted by mIoU s and mIoU u .
We also adopt the harmonic mean ( hIoU ) of mIoU s and mIoU u for the arithmetic mean might be dominated by mIoU s .
4. A BLATION E XPERIMENT AND R ESULTS
Ablation study:
In the table 1, we present an ablation analysis on two aspects: (a) CNNs vs Transformer (b) Whether to use the score map ( l aux ) to modulate the decision boundary.
For cross-entropy loss and regression loss are crucial to recognize the unseen classes, we choose the Deeplabv3+ with l ce , l r and l sc as our baseline.
For the ﬁrst row, we report the baseline IoU scores without l aux .
Then, we compare swin transformer (swin-tiny) backbone with Deeplabv3+ and transformer backbone gives gain of hIoU 1 . 0 over the baseline.
The second and third rows show that the l aux respectively give gain of mIoU u 3 . 0 and 3 .
1 over the baseline and 1 . 9 , 2 . 1 over the SwinZS3 baseline.
This shows a signiﬁcant improvement for the ZS3 and gives a demonstration of the effectiveness of the two methods.
Finally, we combine the transformer and score maps, and report the best mIoU scores.
Comparison to state-of-the-Arts: As showed in table 2, we compare our approach with other state-of-art methods on PASCAL VOC and Context.
We report the best scores of different split [Table] settings.
And the other reported mIoU are from Baek et al.
For PASCAL Context dataset, the comparison shows that : we outperform the second best method JoEm by large margins, for the 6-split setting, we outperform 3 . 0 in mIoU u and 3 . 1 in hIoU .
It is remarkably outperforming for the ZS3.
We outperform the best generative ZS3 method GSRL Li et al.
and give a mIoU u gain of 4 . 4 and hIoU gain of 4 . 2 , which shows the effectiveness and convenience of the discriminative method.
And the CSRL has to be retrained when the novel unseen classes are added.
However, our framework adopts one-stage training strategy (2) We achieve state-of-the-art performance on almost all the zero-shot settings like unseen-2,4,6,8 splits on mIoU u and hIoU .
It conﬁrms our approach could learn the discriminative representations.
The PASCAL VOC dataset experiments emphasize the above analysis.
The PASCAL VOC experiment shows a competitive performance.
Qualitative results .
Fig.5 reports some qualitative examples from PASCAL VOC, with SwinZS modeling the unseen classes more accurately than its competitors.
We notice that SwinZS3 can effectively reduce the false positive prediction.
reports some qualitative examples from PASCAL
VOC, with SwinZS3 modeling the unseen classes more accurately than its competitors.
C ONCLUSION
We have proposed a transformer-based framework that exploits the visual and language features on the joint embedding space for zero-shot semantic segmentation.
We have proposed to use the language-guided score map to better learn the discriminative space while innovatively modifying the decision boundary to reduce the seen bias problem.
Finally, we experiment with our approach on standard ZS3 benchmarks and achieve a new state-of-the-art performance.