Original: GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models
Revised: GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models

Original: Abstract
Revised: Abstract

Original: Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p ( class | pixel feature ) .
Revised: Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p ( class | pixel feature ) .

Original: Though straightforward, this de facto paradigm neglects the underlying data distribution p ( pixel feature | class ) , and struggles to identify out-of-distribution data.
Revised: Though straightforward, this de facto paradigm neglects the underlying data distribution p ( pixel feature | class ) , and struggles to identify out-of-distribution data.

Original: Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p ( pixel feature , class ) .
Revised: Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p ( pixel feature , class ) .

Original: For each class, GMMSeg builds Gaussian
Revised: For each class, GMMSeg builds Gaussian

Original: Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities.
Revised: Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities.

Original: Meanwhile, the deep dense representation is end-to-end trained in a discriminative manner, i.e ., maximizing p ( class | pixel feature ) .
Revised: Meanwhile, the deep dense representation is end-to-end trained in a discriminative manner, i.e ., maximizing p ( class | pixel feature ) .

Original: This endows GMMSeg with the strengths of both generative and discriminative models.
Revised: This endows GMMSeg with the strengths of both generative and discriminative models.

Original: With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets.
Revised: With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets.

Original: More impressively, without any modification, GMMSeg even performs well on open-world datasets.
Revised: More impressively, without any modification, GMMSeg even performs well on open-world datasets.

Original: We believe this work brings fundamental insights into the related fields.
Revised: We believe this work brings fundamental insights into the related fields.

Original:  Introduction
Revised:  Introduction

Original: Semantic segmentation aims to explain visual semantics at the pixel level.
Revised: Semantic segmentation aims to explain visual semantics at the pixel level.

Original: It is typically considered as a problem of pixel-wise classification, i e ., assigning a class label c ∈{ 1 , · · · , C } to each pixel data x .
Revised: It is typically considered as a problem of pixel-wise classification, i e ., assigning a class label c ∈{ 1 , · · · , C } to each pixel data x .

Original: Under this regime, deep-neural solutions are naturally built as a combination of two parts (Fig. 1(a)): an encoder-decoder, dense feature extractor that maps x to a high-dimensional feature representation x , and a dense classifier that conducts C -way classification given input pixel feature x .
Revised: Under this regime, deep-neural solutions are naturally built as a combination of two parts (Fig. 1(a)): an encoder-decoder, dense feature extractor that maps x to a high-dimensional feature representation x , and a dense classifier that conducts C -way classification given input pixel feature x .

Original: Starting from the first end-to-end segmentation solution – fully convolutional networks (FCN) [1], researchers leave the classifier as parametric softmax , and fully devote to improving the dense feature extractor for learning better representation.
Revised: Starting from the first end-to-end segmentation solution – fully convolutional networks (FCN) [1], researchers leave the classifier as parametric softmax , and fully devote to improving the dense feature extractor for learning better representation.

Original: As a result, a huge amount of FCN-based solutions [2–5] emerged and their state-of-the-art was further pushed forward by recent Transformer [6]-style algorithms [7–10].
Revised: As a result, a huge amount of FCN-based solutions [2–5] emerged and their state-of-the-art was further pushed forward by recent Transformer [6]-style algorithms [7–10].

Original: From a probabilistic perspective, the softmax classifier, supervised by the cross-entropy loss together with the feature extractor, directly models the class probability given an input, i.e ., posterior p ( c | x ) .
Revised: From a probabilistic perspective, the softmax classifier, supervised by the cross-entropy loss together with the feature extractor, directly models the class probability given an input, i.e ., posterior p ( c | x ) .

Original: This is known as a discriminative classifier, as the conditional probability distribution discriminates directly between the different values of c [11].
Revised: This is known as a discriminative classifier, as the conditional probability distribution discriminates directly between the different values of c [11].

Original: As discriminative classifiers directly find the classification rule with the smallest error rate, they often give excellent performance in downstream tasks, and hence become the de facto paradigm in segmentation.
Revised: As discriminative classifiers directly find the classification rule with the smallest error rate, they often give excellent performance in downstream tasks, and hence become the de facto paradigm in segmentation.

Original: Yet, due to the discriminative nature, softmaxbased segmentation models suffer from several limitations: First , they only learn the decision boundary between classes, without modeling the underlying data distribution [11].
Revised: Yet, due to the discriminative nature, softmaxbased segmentation models suffer from several limitations: First , they only learn the decision boundary between classes, without modeling the underlying data distribution [11].

Original: Second , as only one weight vector is learned per class, they assume unimodality for each class [12,13], bearing no within-class variation.
Revised: Second , as only one weight vector is learned per class, they assume unimodality for each class [12,13], bearing no within-class variation.

Original: Third , they learn a prediction space where the model accuracy deteriorates rapidly away
Revised: Third , they learn a prediction space where the model accuracy deteriorates rapidly away

Original: [Figure]
Revised: [Figure]

Original: from the decision boundaries [14] and thus yield poorly calibrated predictions [15], struggling to recognize out-of-distribution data [16].
Revised: from the decision boundaries [14] and thus yield poorly calibrated predictions [15], struggling to recognize out-of-distribution data [16].

Original: The first two limitations may hinder the expressive power of segmentation models, and the last one challenges the adoption of segmentation models in decisioncritical tasks ( e.g ., autonomous driving) and motivates the development of anomaly segmentation methods [17–19] (which, however, rely on pre-trained discriminative segmentation models).
Revised: The first two limitations may hinder the expressive power of segmentation models, and the last one challenges the adoption of segmentation models in decisioncritical tasks ( e.g ., autonomous driving) and motivates the development of anomaly segmentation methods [17–19] (which, however, rely on pre-trained discriminative segmentation models).

Original: As an alternative of discriminative classifiers, generative classifiers first find the joint probability p ( x , c ) , and use p ( x , c ) to evaluate the class-conditional densities p ( x | c ) .
Revised: As an alternative of discriminative classifiers, generative classifiers first find the joint probability p ( x , c ) , and use p ( x , c ) to evaluate the class-conditional densities p ( x | c ) .

Original: Then classification is conducted using Bayes rule.
Revised: Then classification is conducted using Bayes rule.

Original: Numerous theoretical and empirical comparisons [20,21] between these two approaches have been initiated even before the deep learning revolution.
Revised: Numerous theoretical and empirical comparisons [20,21] between these two approaches have been initiated even before the deep learning revolution.

Original: They reach the agreement that generative classifiers have potential to overcome shortcomings of their discriminative counterparts, as they are able to model the input data itself.
Revised: They reach the agreement that generative classifiers have potential to overcome shortcomings of their discriminative counterparts, as they are able to model the input data itself.

Original: This stimulates the recent investigation of generative (and discriminative-generative hybrid [22,23]) classifiers in trustworthy AI [24–27] and semi-supervised learning [22,23], while the discriminative classifiers are still dominant in most downstream tasks.
Revised: This stimulates the recent investigation of generative (and discriminative-generative hybrid [22,23]) classifiers in trustworthy AI [24–27] and semi-supervised learning [22,23], while the discriminative classifiers are still dominant in most downstream tasks.

Original: In light of this background, we propose a GMM based segmentation framework – GMMSeg – that addresses the limitations of current discriminative solutions from a generative perspective (Fig. 1(b)).
Revised: In light of this background, we propose a GMM based segmentation framework – GMMSeg – that addresses the limitations of current discriminative solutions from a generative perspective (Fig. 1(b)).

Original: Our work not only represents a novel effort to advocate generative classifiers for end-to-end segmentation, but also evidences the merits of generative approaches in a challenging, dense classification task setting.
Revised: Our work not only represents a novel effort to advocate generative classifiers for end-to-end segmentation, but also evidences the merits of generative approaches in a challenging, dense classification task setting.

Original: In particular, we adopt a separate mixture of Gaussians for modeling the data distribution of each class in the feature space, i.e ., class-conditional feature densities p ( x | c ) .
Revised: In particular, we adopt a separate mixture of Gaussians for modeling the data distribution of each class in the feature space, i.e ., class-conditional feature densities p ( x | c ) .

Original: During training,
Revised: During training,

Original: GMM classifier is online optimized by a momentum version of (Sinkhorn) EM [28] on large-scale, so as to ensure its generative nature and synchronization with the evolving feature space.
Revised: GMM classifier is online optimized by a momentum version of (Sinkhorn) EM [28] on large-scale, so as to ensure its generative nature and synchronization with the evolving feature space.

Original: Meanwhile, the feature extractor is end-to-end trained with the discriminative (cross-entropy) loss, i.e ., maximizing the conditional likelihood p
Revised: Meanwhile, the feature extractor is end-to-end trained with the discriminative (cross-entropy) loss, i.e ., maximizing the conditional likelihood p

Original: ( c | x ) derived with the generative GMM, so as to enable expressive representation learning.
Revised: ( c | x ) derived with the generative GMM, so as to enable expressive representation learning.

Original: In this way, GMMSeg smartly learns generative classification with end-to-end discriminative representation in a compact and collaborative manner, exploiting the benefit of both generative and discriminative approaches.
Revised: In this way, GMMSeg smartly learns generative classification with end-to-end discriminative representation in a compact and collaborative manner, exploiting the benefit of both generative and discriminative approaches.

Original: This also greatly distinguishes GMMSeg from most existing GMM based neural classifiers, which are either discriminatively trained [12,29–31] or trivially estimate a GMM in the feature space of a pre-trained discriminative classifier [19,32,33].
Revised: This also greatly distinguishes GMMSeg from most existing GMM based neural classifiers, which are either discriminatively trained [12,29–31] or trivially estimate a GMM in the feature space of a pre-trained discriminative classifier [19,32,33].

Original: GMMSeg has several appealing facets: First , with the hybrid training strategy – online EM based classifier optimization and end-to-end discriminative representation learning, GMMSeg can precisely approximate the data distribution over a robust feature space.
Revised: GMMSeg has several appealing facets: First , with the hybrid training strategy – online EM based classifier optimization and end-to-end discriminative representation learning, GMMSeg can precisely approximate the data distribution over a robust feature space.

Original: Second , the mixture components make
Revised: Second , the mixture components make

Original: GMMSeg a structured model that well adapts to multimodal data densities.
Revised: GMMSeg a structured model that well adapts to multimodal data densities.

Original: Third , the distributionpreserving property allows GMMSeg to naturally reject abnormal inputs, without neither architectural change (like [34–37]) nor re-training (like [38–40]) nor post-calibration (like [17,18,41–46]).
Revised: Third , the distributionpreserving property allows GMMSeg to naturally reject abnormal inputs, without neither architectural change (like [34–37]) nor re-training (like [38–40]) nor post-calibration (like [17,18,41–46]).

Original: Fourth ,
Revised: Fourth ,

Original: GMMSeg is a principled framework, fully compatible with modern segmentation network architectures.
Revised: GMMSeg is a principled framework, fully compatible with modern segmentation network architectures.

Original: For thorough examination, in §4.1, we approach GMMSeg on several representative segmentation architectures ( i.e ., DeepLab V3+ [47], OCRNet [48], UperNet [49], SegFormer [7]), with diverse backbones ( i.e ., ResNet[50], HRNet[51], Swin[52], MiT[7]).
Revised: For thorough examination, in §4.1, we approach GMMSeg on several representative segmentation architectures ( i.e ., DeepLab V3+ [47], OCRNet [48], UperNet [49], SegFormer [7]), with diverse backbones ( i.e ., ResNet[50], HRNet[51], Swin[52], MiT[7]).

Original: Experimental results demonstrate GMMSeg even outperforms the softmax-based discriminative counterparts, e.g ., 0.6% – 1.5% , 0.5% – 0.8% , and 0.7% – 1.7% mIoU gains over ADE 20K [53], Cityscapes [54], and COCO-Stuff [55], respectively.
Revised: Experimental results demonstrate GMMSeg even outperforms the softmax-based discriminative counterparts, e.g ., 0.6% – 1.5% , 0.5% – 0.8% , and 0.7% – 1.7% mIoU gains over ADE 20K [53], Cityscapes [54], and COCO-Stuff [55], respectively.

Original: Furthermore, in §4.2, we validate our approach on anomaly segmentation.
Revised: Furthermore, in §4.2, we validate our approach on anomaly segmentation.

Original: Without any modification, our Cityscapes-trained GMMSeg model is directly tested on Fishyscapes Lost&Found [56] and Road
Revised: Without any modification, our Cityscapes-trained GMMSeg model is directly tested on Fishyscapes Lost&Found [56] and Road

Original: Anomaly [36] datasets, and outperforms all hand-tailored discriminative competitors.
Revised: Anomaly [36] datasets, and outperforms all hand-tailored discriminative competitors.

Original: To our best knowledge, GMMSeg is the first semantic segmentation method that reports promising results on both closed-set and open-world scenarios by using a single model instance.
Revised: To our best knowledge, GMMSeg is the first semantic segmentation method that reports promising results on both closed-set and open-world scenarios by using a single model instance.

Original: More notably, our impressive results manifest the advantages of generative classifiers in a large-scale real-world setting.
Revised: More notably, our impressive results manifest the advantages of generative classifiers in a large-scale real-world setting.

Original: We feel this work opens a new avenue for research in this field.
Revised: We feel this work opens a new avenue for research in this field.

Original:  Related Work
Revised:  Related Work

Original: Semantic Segmentation.
Revised: Semantic Segmentation.

Original: Since the seminal work of FCN [1], deep-net segmentation solutions are typically built in a dense classification fashion, i.e ., learning dense representation and categorization end-to-end.
Revised: Since the seminal work of FCN [1], deep-net segmentation solutions are typically built in a dense classification fashion, i.e ., learning dense representation and categorization end-to-end.

Original: By directly adopting discriminative softmax for categorization, FCN-style solutions put focus on learning expressive dense representation; they modify the FCN architecture from various aspects, such as enlarging the receptive field [2,3,47,57–60], modeling multi-scale context [48,59, 61–77], investigating non-local operations [4,5,78–84], and exploring hierarchical information [85– 87].
Revised: By directly adopting discriminative softmax for categorization, FCN-style solutions put focus on learning expressive dense representation; they modify the FCN architecture from various aspects, such as enlarging the receptive field [2,3,47,57–60], modeling multi-scale context [48,59, 61–77], investigating non-local operations [4,5,78–84], and exploring hierarchical information [85– 87].

Original: With a similar goal of sharpening representation, later Transformer-style solutions [7–9, 88, 89] empower attentive networks with, for instance, local contiguity [7] and multi-level feature aggregation [9,10].
Revised: With a similar goal of sharpening representation, later Transformer-style solutions [7–9, 88, 89] empower attentive networks with, for instance, local contiguity [7] and multi-level feature aggregation [9,10].

Original: Two very recent attentive models [90,91] formulate the task in an alternative form of mask classification , however, still relying on discriminative softmax.
Revised: Two very recent attentive models [90,91] formulate the task in an alternative form of mask classification , however, still relying on discriminative softmax.

Original: From the discussion above, we can find that current prevalent segmentation solutions are in essence a pixel-wise, discriminative classifier , which only learns decision boundaries between classes in the pixel feature space [14, 92], without modeling the underlying data distribution.
Revised: From the discussion above, we can find that current prevalent segmentation solutions are in essence a pixel-wise, discriminative classifier , which only learns decision boundaries between classes in the pixel feature space [14,92], without modeling the underlying data distribution.

Original: In contrast, our GMMSeg tackles the task from a generative viewpoint.
Revised: In contrast, our GMMSeg tackles the task from a generative viewpoint.

Original: GMMSeg deeply embeds generative optimization of GMMs into end-to-end dense representation learning, so as to comprehensively describe the class-aware knowledge [93] in a discriminative feature space.
Revised: GMMSeg deeply embeds generative optimization of GMMs into end-to-end dense representation learning, so as to comprehensively describe the class-aware knowledge [93,94] in a discriminative feature space.

Original: GMMSeg is partly inspired by [13,94], that also probe data structures via intra-class clustering.
Revised: GMMSeg is partly inspired by [13,95], that also probe data structures via intra-class clustering.

Original: However, the dense classification in the two works are achieved via non-parametric, nearest centroid retrieving – still a discriminative model.
Revised: However, the dense classification in the two works are achieved via non-parametric, nearest centroid retrieving – still a discriminative model.

Original: In [95], though data density is estimated (as a mixture of vMF distributions [96]), it is only used as a supervisory signal for dense embedding learning, and the final prediction is still made by a discriminative classifier – k -NN.
Revised: In [96], though data density is estimated (as a mixture of vMF distributions [97]), it is only used as a supervisory signal for dense embedding learning, and the final prediction is still made by a discriminative classifier – k -NN.

Original: Our work represents the first step towards formulating (closed-set) semantic segmentation within a generative neural classification framework.
Revised: Our work represents the first step towards formulating (closed-set) semantic segmentation within a generative neural classification framework.

Original: Discriminative vs Generative Classifiers.
Revised: Discriminative vs Generative Classifiers.

Original: Generative classifiers and discriminative classifiers represent two contrasting ways of solving classification tasks [24].
Revised: Generative classifiers and discriminative classifiers represent two contrasting ways of solving classification tasks [24].

Original: Basically, the generative classifiers (such as Linear Discriminant Analysis and naive Bayes) learn the class densities p ( x | c ) , while the discriminative classifiers (such as softmax) learn the class boundaries p ( c | x ) without regard to the underlying class densities.
Revised: Basically, the generative classifiers (such as Linear Discriminant Analysis and naive Bayes) learn the class densities p ( x | c ) , while the discriminative classifiers (such as softmax) learn the class boundaries p ( c | x ) without regard to the underlying class densities.

Original: In practical classification tasks, softmax discriminative classifier is used exclusively [24], due to its simplicity and excellent discriminative performance.
Revised: In practical classification tasks, softmax discriminative classifier is used exclusively [24], due to its simplicity and excellent discriminative performance.

Original: Nonetheless, generative classifiers are widely agreed to have several advantages over their discriminative counterparts[21, 97], e.g ., accurately modeling the input distribution, and explicitly identifying unlikely inputs in a natural way.
Revised: Nonetheless, generative classifiers are widely agreed to have several advantages over their discriminative counterparts[21, 98], e.g ., accurately modeling the input distribution, and explicitly identifying unlikely inputs in a natural way.

Original: Driven by this common belief, a surge of deep learning literature [14,98–100] investigated the potential (and the limitation) of generative classifiers in adversarial defense[25,26,101–103], explainable AI [24], out-of-distribution detection [27,104], and semi-supervised learning [22,23,98].
Revised: Driven by this common belief, a surge of deep learning literature [14,99–101] investigated the potential (and the limitation) of generative classifiers in adversarial defense[25,26,102–104], explainable AI [24], out-of-distribution detection [27,105], and semi-supervised learning [22,23,99].

Original: As GMMs can express (almost) arbitrary continuous distributions, it has been adopted in many neural classifiers[23,105].
Revised: As GMMs can express (almost) arbitrary continuous distributions, it has been adopted in many neural classifiers[23,106].

Original: However, most of these GMM classifiers are discriminative models[12,14,29,30] that are trained ‘discriminatively’( i.e ., maximizing posteriors p ( c | x ) ).
Revised: However, most of these GMM classifiers are discriminative models[12,14,29,30] that are trained ‘discriminatively’( i.e ., maximizing posteriors p ( c | x ) ).

Original: In GMMSeg, the GMM is purely optimized via EM ( i.e ., estimating class densities p ( x | c ) ) while the deep representation is trained via gradient backpropagation of the discriminative loss.
Revised: In GMMSeg, the GMM is purely optimized via EM ( i.e ., estimating class densities p ( x | c ) ) while the deep representation is trained via gradient backpropagation of the discriminative loss.

Original: Thus the whole GMMSeg is a hybrid of generative GMM and discriminative representation, getting the best of two worlds.
Revised: Thus the whole GMMSeg is a hybrid of generative GMM and discriminative representation, getting the best of two worlds.

Original: Although bearing the general idea of trading-off between generative and discriminative classifiers [21–23,97,106,107], none of the previous hybrid algorithms demonstrate their utility in challenging segmentation tasks.
Revised: Although bearing the general idea of trading-off between generative and discriminative classifiers [21–23,98,107,108], none of the previous hybrid algorithms demonstrate their utility in challenging segmentation tasks.

Original: Anomaly Segmentation.
Revised: Anomaly Segmentation.

Original: Anomaly segmentation strives to identify unknown object regions, typically in road-driving scenarios [56].
Revised: Anomaly segmentation strives to identify unknown object regions, typically in road-driving scenarios [56].

Original: Existing solutions can be generally categorized into three classes: i)
Revised: Existing solutions can be generally categorized into three classes: i)

Original: Uncertainty estimation based algorithms [17,18,41–46] usually approximate the uncertainty from simple statistics of the classification probability or logits of pre-trained segmentation models [17,18, 41–43], or adopt Bayesian neural networks with Monte-Carlo dropout to capture pixel uncertainty [44– 46].
Revised: Uncertainty estimation based algorithms [17,18,41–46] usually approximate the uncertainty from simple statistics of the classification probability or logits of pre-trained segmentation models [17,18, 41–43], or adopt Bayesian neural networks with Monte-Carlo dropout to capture pixel uncertainty [44– 46].

Original: ii) Outlier exposure based algorithms make use of auxiliary datasets as training samples of unexpected objects [38–40].
Revised: ii) Outlier exposure based algorithms make use of auxiliary datasets as training samples of unexpected objects [38–40].

Original: Therefore, this type of algorithms requires re-training the segmentation network, resulting in performance degradation.
Revised: Therefore, this type of algorithms requires re-training the segmentation network, resulting in performance degradation.

Original: iii) Image resynthesis based algorithms reconstruct the input image and discriminate the anomaly instances according to the reconstruction error [34–37].
Revised: iii) Image resynthesis based algorithms reconstruct the input image and discriminate the anomaly instances according to the reconstruction error [34–37].

Original: With a generative classifier, our GMMSeg handles anomaly segmentation naturally, without neither external datasets of outliers, nor additional image resynthesis models.
Revised: With a generative classifier, our GMMSeg handles anomaly segmentation naturally, without neither external datasets of outliers, nor additional image resynthesis models.

Original: It also greatly differs from most uncertainty estimation-based methods that are post-processing techniques adjusting the prediction scoresofsoftmax-basedsegmentationnetworks[17,18,41–43].Themostrelevantonesaremaybeafew densityestimation-basedmodels[56,108,109],whichdirectlymeasurethelikelihoodofsamplesw.r.t.
Revised: It also greatly differs from most uncertainty estimation-based methods that are post-processing techniques adjusting the prediction scoresofsoftmax-basedsegmentationnetworks[17,18,41–43].Themostrelevantonesaremaybeafew densityestimation-basedmodels[56,109,110],whichdirectlymeasurethelikelihoodofsamplesw.r.t.

Original: the data distribution.
Revised: the data distribution.

Original: However, they are either limited to pre-trained representation [56] or specialized for anomaly detection with simple data [108,109].
Revised: However, they are either limited to pre-trained representation [56] or specialized for anomaly detection with simple data [109,110].

Original: To our best knowledge, this is the first time to report promising results on both closed-set and open-world large-scale settings, through a single model instance without any change of network architecture as well as training and inference protocols.
Revised: To our best knowledge, this is the first time to report promising results on both closed-set and open-world large-scale settings, through a single model instance without any change of network architecture as well as training and inference protocols.

Original:  Methodology
Revised:  Methodology

Original: In this section, we first formalize modern semantic segmentation models within a dense discriminative classification framework and discuss defects of such discriminative regime from a probabilistic viewpoint (§3.1).
Revised: In this section, we first formalize modern semantic segmentation models within a dense discriminative classification framework and discuss defects of such discriminative regime from a probabilistic viewpoint (§3.1).

Original: Then we describe our new segmentation framework – GMMSeg – that brings a paradigm shift from the discriminative to generative (§3.2).
Revised: Then we describe our new segmentation framework – GMMSeg – that brings a paradigm shift from the discriminative to generative (§3.2).

Original: Finally, in §3.3, we provide implementation details.
Revised: Finally, in §3.3, we provide implementation details.

Original: 3.1 Existing Segmentation Solutions: Dense Discriminative Classifier
Revised: 3.1 Existing Segmentation Solutions: Dense Discriminative Classifier

Original: In the standard semantic segmentation setting, we are given a training dataset D = { ( x n , c n ) } Nn =1 of
Revised: In the standard semantic segmentation setting, we are given a training dataset D = { ( x n , c n ) } Nn =1 of

Original: N pairs of pixel samples x n ∈ R 3 and corresponding semantic labels c n ∈{ 1 , · · · , C } .
Revised: N pairs of pixel samples x n ∈ R 3 and corresponding semantic labels c n ∈{ 1 , · · · , C } .

Original: The goal is to use D to learn a classification rule which can predict the label c ′ ∈{ 1 , · · · , C } of an unseen pixel x ′ .
Revised: The goal is to use D to learn a classification rule which can predict the label c ′ ∈{ 1 , · · · , C } of an unseen pixel x ′ .

Original: Recent mainstream solutions employ a deep neural network for pixel representation learning and softmax for semantic label prediction.
Revised: Recent mainstream solutions employ a deep neural network for pixel representation learning and softmax for semantic label prediction.

Original: Hence they are usually built as a composition of f ◦ g : •
Revised: Hence they are usually built as a composition of f ◦ g : •

Original: A dense feature extractor f θ : R 3 → R D , which is typically an encoder-decoder network that maps the input pixel x to a D -dimensional feature representation x , i.e ., x = f θ ( x ) ∈ R D ; 1 and • A dense classifier g ω : R D → R C , which is achieved by parametric softmax that maps each pixel representation x ∈ R D to C real-valued numbers {
Revised: A dense feature extractor f θ : R 3 → R D , which is typically an encoder-decoder network that maps the input pixel x to a D -dimensional feature representation x , i.e ., x = f θ ( x ) ∈ R D ; 1 and • A dense classifier g ω : R D → R C , which is achieved by parametric softmax that maps each pixel representation x ∈ R D to C real-valued numbers {

Original: y c ∈ R } Cc =1 termed as logits , i.e ., {
Revised: y c ∈ R } Cc =1 termed as logits , i.e ., {

Original: y c } Cc =1 = g ω ( x ) , and uses the logits to compute the posterior probability:
Revised: y c } Cc =1 = g ω ( x ) , and uses the logits to compute the posterior probability:

Original: [Equation]
Revised: [Equation]

Original: where w c ∈ R D and b c ∈ R are the weight and bias for class c , respectively; and ω = { w 1: C , b 1: C } .
Revised: where w c ∈ R D and b c ∈ R are the weight and bias for class c , respectively; and ω = { w 1: C , b 1: C } .

Original: The final prediction is the class with the highest predicted probability: arg max c p ( c | x ; ω , θ ) .
Revised: The final prediction is the class with the highest predicted probability: arg max c p ( c | x ; ω , θ ) .

Original: The feature extractor f and softmax-based classifier g are jointly trained end-to-end.
Revised: The feature extractor f and softmax-based classifier g are jointly trained end-to-end.

Original: Their corresponding parameters { θ , ω } are optimized by minimizing the so-called cross-entropy loss on D :
Revised: Their corresponding parameters { θ , ω } are optimized by minimizing the so-called cross-entropy loss on D :

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to maximizing conditional likelihood, i.e ., Π ( x,c )
Revised: which is equivalent to maximizing conditional likelihood, i.e ., Π ( x,c )

Original: ∈D p ( c | x ) .
Revised: ∈D p ( c | x ) .

Original: In some literature[11, 110], such learning strategy is called discriminative training .
Revised: In some literature[11, 111], such learning strategy is called discriminative training .

Original: As softmax directly models the conditional probability distribution p ( c | x ) with no concern for modeling the input distribution p ( x , c ) , existing softmax-based segmentation models are in essence a dense discriminative classifier.
Revised: As softmax directly models the conditional probability distribution p ( c | x ) with no concern for modeling the input distribution p ( x , c ) , existing softmax-based segmentation models are in essence a dense discriminative classifier.

Original: Discriminative softmax typically gives good predictive performance, as the pixel classification rule depends only on the conditional distribution p ( c | x ) in the sense of minimum error rate and softmax optimizes the quantity of interest in a concise manner, i.e ., learning a direct map from inputs x to the class labels c .
Revised: Discriminative softmax typically gives good predictive performance, as the pixel classification rule depends only on the conditional distribution p ( c | x ) in the sense of minimum error rate and softmax optimizes the quantity of interest in a concise manner, i.e ., learning a direct map from inputs x to the class labels c .

Original: In spite of its prevalence and effectiveness, this dense discriminative regime has some drawbacks that are still poorly understood: First , it attends only to learning the decision boundaries between the C classes on the pixel embedding space, i.e ., splitting the D -dimensional feature space using C different ( D − 1 )-dimensional hyperplanes.
Revised: In spite of its prevalence and effectiveness, this dense discriminative regime has some drawbacks that are still poorly understood: First , it attends only to learning the decision boundaries between the C classes on the pixel embedding space, i.e ., splitting the D -dimensional feature space using C different ( D − 1 )-dimensional hyperplanes.

Original: It achieves a simplified approach that eliminates extra parameters for modeling the data (representation) distribution [111].
Revised: It achieves a simplified approach that eliminates extra parameters for modeling the data (representation) distribution [112].

Original: However, from another perspective, it fails to capture the intrinsic class characteristics and is hard to achieve good generalization on unseen data.
Revised: However, from another perspective, it fails to capture the intrinsic class characteristics and is hard to achieve good generalization on unseen data.

Original: Second , in softmax, each class c corresponds to only a single weight ( w c , b c ).
Revised: Second , in softmax, each class c corresponds to only a single weight ( w c , b c ).

Original: That means existing segmentation models rely on an implicit assumption of unimodality of data of each class in the feature space [12,112,113].
Revised: That means existing segmentation models rely on an implicit assumption of unimodality of data of each class in the feature space [12,113,114].

Original: However, this unimodality assumption is rarely the case in real-world scenarios and makes the model less tolerant of intra-class variances [13], especially when the multimodality remains in the feature space [12].
Revised: However, this unimodality assumption is rarely the case in real-world scenarios and makes the model less tolerant of intra-class variances [13], especially when the multimodality remains in the feature space [12].

Original: Third , softmax is not capable of inferring the data distribution – it is notorious with inflating the probability of the predicted class as a result of the exponent employed on the network outputs [114].
Revised: Third , softmax is not capable of inferring the data distribution – it is notorious with inflating the probability of the predicted class as a result of the exponent employed on the network outputs [115].

Original: Thus the prediction score of a class is useless besides itscomparativevalue against other classes.
Revised: Thus the prediction score of a class is useless besides itscomparativevalue against other classes.

Original: This is the root cause of why existing segmentation models are hard to identify pixel samples x ′ of an unseen class (out-of-distribution data), i.e ., c ′
Revised: This is the root cause of why existing segmentation models are hard to identify pixel samples x ′ of an unseen class (out-of-distribution data), i.e ., c ′

Original: ̸∈{ 1 , · · · , C } .
Revised: ̸∈{ 1 , · · · , C } .

Original: Accordingly, we argue that the time might be right to rethink the current de facto , discriminative segmentation regime, where the softmax classifier may actually cause more harm than good.
Revised: Accordingly, we argue that the time might be right to rethink the current de facto , discriminative segmentation regime, where the softmax classifier may actually cause more harm than good.

Original: 3.2 GMMSeg: Dense GMM Generative Classification
Revised: 3.2 GMMSeg: Dense GMM Generative Classification

Original: Our GMMSeg reformulates the task from a dense generative classification point of view.
Revised: Our GMMSeg reformulates the task from a dense generative classification point of view.

Original: Instead of building posterior p ( c | x ) directly, generative classifiers predict labels using Bayes rule.
Revised: Instead of building posterior p ( c | x ) directly, generative classifiers predict labels using Bayes rule.

Original: Specifically, generative classifiers model the joint distribution p ( x , c ) , by estimating the class-conditional distribution p ( x | c ) along with the class prior p ( c ) .
Revised: Specifically, generative classifiers model the joint distribution p ( x , c ) , by estimating the class-conditional distribution p ( x | c ) along with the class prior p ( c ) .

Original: Then, following Bayes rule, the posterior is derived as:
Revised: Then, following Bayes rule, the posterior is derived as:

Original: [Equation]
Revised: [Equation]

Original: Since the class probabilities p ( c ) are typically set as a uniform prior (also in our case), estimating the class-conditional distributions ( i.e ., data densities) p ( x | c ) is the core and most difficult part of building a generative classifier.
Revised: Since the class probabilities p ( c ) are typically set as a uniform prior (also in our case), estimating the class-conditional distributions ( i.e ., data densities) p ( x | c ) is the core and most difficult part of building a generative classifier.

Original: It is also worth noting that generative classifiers are optimized by approximating the data distribution Π ( x,c ) ∈D p
Revised: It is also worth noting that generative classifiers are optimized by approximating the data distribution Π ( x,c ) ∈D p

Original: ( x | c ) , which is called generative training [11].
Revised: ( x | c ) , which is called generative training [11].

Original: Although discriminative classifiers demonstrate impressive performance in many application tasks, there are several crucial reasons for using generative rather than discriminative classifiers, which can be succinctly articulated by Feynman’s mantra “What I cannot create, I do not understand”.
Revised: Although discriminative classifiers demonstrate impressive performance in many application tasks, there are several crucial reasons for using generative rather than discriminative classifiers, which can be succinctly articulated by Feynman’s mantra “What I cannot create, I do not understand”.

Original: Surprisingly, generative classifiers have been rarely investigated in modern segmentation models.
Revised: Surprisingly, generative classifiers have been rarely investigated in modern segmentation models.

Original: Driven by the belief that generative classifiers are the right way to remove the shortcomings of discriminative approaches, we revisit GMM – one of the most classic generative probabilistic classifiers.
Revised: Driven by the belief that generative classifiers are the right way to remove the shortcomings of discriminative approaches, we revisit GMM – one of the most classic generative probabilistic classifiers.

Original: We couple the generative EM optimization of GMMs with the discriminative learning of the dense feature extractor f – the most successful part of modern segmentation models, leading to a powerful, principled, and dense generative classification based segmentation framework – GMMSeg (Fig. 2).
Revised: We couple the generative EM optimization of GMMs with the discriminative learning of the dense feature extractor f – the most successful part of modern segmentation models, leading to a powerful, principled, and dense generative classification based segmentation framework – GMMSeg (Fig. 2).

Original: Specifically, GMMSeg adopts a weighted mixture of M multivariate Gaussians for modeling the pixel data distribution of each class c in the D -dimensional embedding space:
Revised: Specifically, GMMSeg adopts a weighted mixture of M multivariate Gaussians for modeling the pixel data distribution of each class c in the D -dimensional embedding space:

Original: [Equation]
Revised: [Equation]

Original: Here m | c ∼ Multinomial( π c ) is the prior probability, i.e ., (cid:80) m π cm =1 ; µ cm ∈ R D and Σ cm ∈ R D
Revised: Here m | c ∼ Multinomial( π c ) is the prior probability, i.e ., (cid:80) m π cm =1 ; µ cm ∈ R D and Σ cm ∈ R D

Original: × D are the mean vector and covariance matrix for component m in class c ; and ϕ c = { π c , µ c , Σ c } .
Revised: × D are the mean vector and covariance matrix for component m in class c ; and ϕ c = { π c , µ c , Σ c } .

Original: The mixture nature allows GMMSeg to accurately approximate the data densities and to be superior over softmax assuming unimodality for each class.
Revised: The mixture nature allows GMMSeg to accurately approximate the data densities and to be superior over softmax assuming unimodality for each class.

Original: Each Gaussian component has an independent covariance structure, enabling a flexible local measure of importance along different feature dimensions.
Revised: Each Gaussian component has an independent covariance structure, enabling a flexible local measure of importance along different feature dimensions.

Original: To find the optimal parameters of the GMM classifier, i.e ., { ϕ ∗ c } Cc =1 , a standard approach is EM [115], i.e ., maximizing the log likelihood over the feature-label pairs { ( x n , c n ) }
Revised: To find the optimal parameters of the GMM classifier, i.e ., { ϕ ∗ c } Cc =1 , a standard approach is EM [116], i.e ., maximizing the log likelihood over the feature-label pairs { ( x n , c n ) }

Original: Nn =1 in the training dataset D :
Revised: Nn =1 in the training dataset D :

Original: [Equation]
Revised: [Equation]

Original: EM starts with some initial guess at the maximum likelihood parameters ϕ (0) c , and then proceeds to iteratively create successive estimates ϕ ( t ) c for t =1 , 2 , · · · , by repeatedly optimizing a F function[116]:
Revised: EM starts with some initial guess at the maximum likelihood parameters ϕ (0) c , and then proceeds to iteratively create successive estimates ϕ ( t ) c for t =1 , 2 , · · · , by repeatedly optimizing a F function[117]:

Original: [Equation]
Revised: [Equation]

Original: q c [ m ]= p ( m | x , c ; ϕ c ) gives the probability that data x is assigned to component m .
Revised: q c [ m ]= p ( m | x , c ; ϕ c ) gives the probability that data x is assigned to component m .

Original: F is defined as:
Revised: F is defined as:

Original: [Equation]
Revised: [Equation]

Original: where E q c [ · ] gives the expectation w
Revised: where E q c [ · ] gives the expectation w

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: t. the distribution over the M components given by q c
Revised: t. the distribution over the M components given by q c

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: H ( q c )= − E q c [log q c [ m ]] defines the entropy of q c .
Revised: H ( q c )= − E q c [log q c [ m ]] defines the entropy of q c .

Original: Based on Eqs.
Revised: Based on Eqs.

Original: 4-7, for ∀ x n : c n = c , we have:
Revised: 4-7, for ∀ x n : c n = c , we have:

Original: [Equation]
Revised: [Equation]

Original: trainingpixelsamples dense feature extractor data flow parameter optimization cross-entropy { x n } Nn = { p ( c n | x n ) } Nn = { c n } Nn = stochastic gradient descent of discriminative representation learning
Revised: trainingpixelsamples dense feature extractor data flow parameter optimization cross-entropy { x n } Nn = { p ( c n | x n ) } Nn = { c n } Nn = stochastic gradient descent of discriminative representation learning

Original: EM optimization of generative GMM ϕϕϕ{ ϕ 1 , ϕ 2 , ϕ 3 } { q 1 ,q 2 ,q 3 } f θ p ( x n | cn ; ϕ c ) p ( x n | 1; ϕ 1)+ p ( x n | 2; ϕ
Revised: EM optimization of generative GMM ϕϕϕ{ ϕ 1 , ϕ 2 , ϕ 3 } { q 1 ,q 2 ,q 3 } f θ p ( x n | cn ; ϕ c ) p ( x n | 1; ϕ 1)+ p ( x n | 2; ϕ

Original: 2)+ p ( x n | 3; ϕ 3)= p ( c n | x n )
Revised: 2)+ p ( x n | 3; ϕ 3)= p ( c n | x n )

Original: M-step (Eq.8)
Revised: M-step (Eq.8)

Original: E-step (Eq.9) (Eq.11) where N c is the number of training samples labeled as c and N cm = (cid:80) n : c n = c
Revised: E-step (Eq.9) (Eq.11) where N c is the number of training samples labeled as c and N cm = (cid:80) n : c n = c

Original: q cn [ m ] .
Revised: q cn [ m ] .

Original: In E-step, we recompute the posterior q ( t ) c over the M components given the old parameters ϕ
Revised: In E-step, we recompute the posterior q ( t ) c over the M components given the old parameters ϕ

Original: ( t − 1) .
Revised: ( t − 1) .

Original: In M-step, with the soft cluster assignment q ( t ) c , the parameters are updated as ϕ ( t ) c such that the F function is maximized.
Revised: In M-step, with the soft cluster assignment q ( t ) c , the parameters are updated as ϕ ( t ) c such that the F function is maximized.

Original: In practice, we find standard EM suffers from slow convergence and delivers unsatisfactory results ( cf . §4.3).
Revised: In practice, we find standard EM suffers from slow convergence and delivers unsatisfactory results ( cf . §4.3).

Original: A potential reason is the parameter sensitivity of EM – convergent parameters may change vastly even with slightly different initialization [117].
Revised: A potential reason is the parameter sensitivity of EM – convergent parameters may change vastly even with slightly different initialization [118].

Original: Drawing inspiration from recent optimal transport (OT) based clustering algorithms [118,119], we introduce a uniform prior on the mixture weights π c , i.e ., ∀ c, m : π cm = 1 M .
Revised: Drawing inspiration from recent optimal transport (OT) based clustering algorithms [119,120], we introduce a uniform prior on the mixture weights π c , i.e ., ∀ c, m : π cm = 1 M .

Original: Recalling q c [ m ]= p ( m | x , c ) , we can derive a constraint
Revised: Recalling q c [ m ]= p ( m | x , c ) , we can derive a constraint

Original: Q c = { q c : 1 N c (cid:80) x n : c n = c p ( m | x n , c )= 1 M } .
Revised: Q c = { q c : 1 N c (cid:80) x n : c n = c p ( m | x n , c )= 1 M } .

Original: Then E-step in Eq.
Revised: Then E-step in Eq.

Original: 6 is performed by restricting the optimization of q c over the set Q c :
Revised: 6 is performed by restricting the optimization of q c over the set Q c :

Original: [Equation]
Revised: [Equation]

Original: This can be intuitively viewed as an equipartition constraint guided clustering process: inside each class c , we expect the N c pixel samples to be evenly assigned to M components.
Revised: This can be intuitively viewed as an equipartition constraint guided clustering process: inside each class c , we expect the N c pixel samples to be evenly assigned to M components.

Original: As indicated by[28],
Revised: As indicated by[28],

Original: Eq. 9 is analogous to entropy-regularized OT:
Revised: Eq. 9 is analogous to entropy-regularized OT:

Original: [Equation]
Revised: [Equation]

Original: where the transport matrix Q c ( i.e ., target solution) can be viewed as the posterior distribution q c of
Revised: where the transport matrix Q c ( i.e ., target solution) can be viewed as the posterior distribution q c of

Original: N c samples over the M components ( i.e ., Q c ( n, m )=
Revised: N c samples over the M components ( i.e ., Q c ( n, m )=

Original: q cn [ m ] ), the cost matrix O c ∈ R N c × M is given as the negative log-likelihood, i.e ., O c ( n, m )
Revised: q cn [ m ] ), the cost matrix O c ∈ R N c × M is given as the negative log-likelihood, i.e ., O c ( n, m )

Original: = − log p ( x n | c, m ) , and the entropy H ( · ) is penalized by ϵ .
Revised: = − log p ( x n | c, m ) , and the entropy H ( · ) is penalized by ϵ .

Original: The set Q ′ c encapsulates all the desired constraints over Q c , where 1 M is a M -dimensional all-ones vector.
Revised: The set Q ′ c encapsulates all the desired constraints over Q c , where 1 M is a M -dimensional all-ones vector.

Original: Intuitively, the more plausible a pixel sample x n is with respect to component m , the less it costs to transport the underlying mass.
Revised: Intuitively, the more plausible a pixel sample x n is with respect to component m , the less it costs to transport the underlying mass.

Original: Eq.
Revised: Eq.

Original: 10 can be efficiently solved via Sinkhorn-Knopp
Revised: 10 can be efficiently solved via Sinkhorn-Knopp

Original: Iteration [119], where ϵ is set as the default ( i.e ., 0 . 05 ).
Revised: Iteration [120], where ϵ is set as the default ( i.e ., 0 . 05 ).

Original: This optimization scheme, called Sinkhorn
Revised: This optimization scheme, called Sinkhorn

Original: EM , is proved to have the same global optimum with the EM in Eq.
Revised: EM , is proved to have the same global optimum with the EM in Eq.

Original: 9 yet is less prone to getting stuck in local optima [28], which is in line with our empirical results ( cf .
Revised: 9 yet is less prone to getting stuck in local optima [28], which is in line with our empirical results ( cf .

Original: Our GMMSeg adopts a hybrid training strategy that is partly generative and partly discriminative:
Revised: Our GMMSeg adopts a hybrid training strategy that is partly generative and partly discriminative:

Original: Generative Optimization (Sinkhorn EM) of GMM Classifier : { ϕ ∗ c } Cc =1 = {
Revised: Generative Optimization (Sinkhorn EM) of GMM Classifier : { ϕ ∗ c } Cc =1 = {

Original: [Equation]
Revised: [Equation]

Original: Discriminative Learning (Cross-Entropy Loss) of Dense Representation : θ ∗ =
Revised: Discriminative Learning (Cross-Entropy Loss) of Dense Representation : θ ∗ =

Original: arg min θ −
Revised: arg min θ −

Original: [Equation]
Revised: [Equation]

Original: In GMMSeg, GMM classifier (has C × M components in total) is purely optimized in a generative fashion, i.e ., applying Sinkhorn EM to model the data densities p ( x
Revised: In GMMSeg, GMM classifier (has C × M components in total) is purely optimized in a generative fashion, i.e ., applying Sinkhorn EM to model the data densities p ( x

Original: | c ) within each class c in the feature space f θ .
Revised: | c ) within each class c in the feature space f θ .

Original: The feature extractor/space f θ , in contrast, is end-to-end trained in a discriminative manner, i.e ., minimizing the cross-entropy loss over the posteriors output by the GMM.
Revised: The feature extractor/space f θ , in contrast, is end-to-end trained in a discriminative manner, i.e ., minimizing the cross-entropy loss over the posteriors output by the GMM.

Original: During each training iteration, the extractor’s parameters θ are only updated by the gradient backpropagated from the discriminative loss, while the GMM’s parameters { ϕ c } c are only optimized by EM.
Revised: During each training iteration, the extractor’s parameters θ are only updated by the gradient backpropagated from the discriminative loss, while the GMM’s parameters { ϕ c } c are only optimized by EM.

Original: To accurately estimate the GMM distributions, an external memory is adopted to store a large set of pixel representations, sampled from several preceding training batches, enabling large-scale EM.
Revised: To accurately estimate the GMM distributions, an external memory is adopted to store a large set of pixel representations, sampled from several preceding training batches, enabling large-scale EM.

Original: Moreover, since the feature space f θ gradually evolves during training, we opt for a momentum EM: we directly use the GMM’s parameters { ˆ ϕ c } c estimated in the latest iteration as the initial guess in the current iteration { ϕ (0) c } c , and adopt momentum update in the M-Step, i.e ., { ϕ ( t ) c ← (1 − τ ) ϕ ( t ) c
Revised: Moreover, since the feature space f θ gradually evolves during training, we opt for a momentum EM: we directly use the GMM’s parameters { ˆ ϕ c } c estimated in the latest iteration as the initial guess in the current iteration { ϕ (0) c } c , and adopt momentum update in the M-Step, i.e ., { ϕ ( t ) c ← (1 − τ ) ϕ ( t ) c

Original: + τ ˆ ϕ c } c , where the momentum coefficient is set as τ =0 .
Revised: + τ ˆ ϕ c } c , where the momentum coefficient is set as τ =0 .

Original: This makes our training more stable and accelerates the convergence of EM – we empirically find even one EM loop per training iteration is good enough.
Revised: This makes our training more stable and accelerates the convergence of EM – we empirically find even one EM loop per training iteration is good enough.

Original: This hybrid training scheme brings several advantages: First , GMMSeg achieves the merits of both generative and discriminative learning.
Revised: This hybrid training scheme brings several advantages: First , GMMSeg achieves the merits of both generative and discriminative learning.

Original: The online EM based generative optimization enables the
Revised: The online EM based generative optimization enables the

Original: GMM to best fit the data distribution even on the evolving feature space.
Revised: GMM to best fit the data distribution even on the evolving feature space.

Original: On the other hand, the feature space is discriminatively end-to-end trained under the guidance of the GMM classifier, so as to maximize the pixel-wise predictive performance.
Revised: On the other hand, the feature space is discriminatively end-to-end trained under the guidance of the GMM classifier, so as to maximize the pixel-wise predictive performance.

Original: Second , as the generative EM optimization and discriminative stochastic training work in an independent yet closely collaborative manner, GMMSeg is fully compatible with modern segmentation network architectures and existing discriminative training objectives.
Revised: Second , as the generative EM optimization and discriminative stochastic training work in an independent yet closely collaborative manner, GMMSeg is fully compatible with modern segmentation network architectures and existing discriminative training objectives.

Original: It can be further advanced with the development of network architectures of the discriminative counterparts.
Revised: It can be further advanced with the development of network architectures of the discriminative counterparts.

Original: Third , as GMMSeg explicitly models class-conditional data distribution p ( x | c ) , it can naturally handle off-manifold examples, i.e ., directly giving meaningful likelihood of the example fitting each class GMM distribution (see §4.2 for experiments on anomaly segmentation).
Revised: Third , as GMMSeg explicitly models class-conditional data distribution p ( x | c ) , it can naturally handle off-manifold examples, i.e ., directly giving meaningful likelihood of the example fitting each class GMM distribution (see §4.2 for experiments on anomaly segmentation).

Original: 3.3 Implementation Details
Revised: 3.3 Implementation Details

Original: Network Architecture.
Revised: Network Architecture.

Original: GMMSeg is a general framework that can be built upon any modern segmentation network by replacing softmax with the GMM classifier.
Revised: GMMSeg is a general framework that can be built upon any modern segmentation network by replacing softmax with the GMM classifier.

Original: In our experiments ( cf . §4.1), we approach GMMSeg on a variety of segmentation models [7,47–49] and backbones [50,51].
Revised: In our experiments ( cf . §4.1), we approach GMMSeg on a variety of segmentation models [7,47–49] and backbones [50,51].

Original: In the
Revised: In the

Original: GMM classifier, a 1 × 1 conv is used to compress each pixel feature to a 64-dimensional vector, i.e .,
Revised: GMM classifier, a 1 × 1 conv is used to compress each pixel feature to a 64-dimensional vector, i.e .,

Original: D = 64 , and the covariance matrices Σ ∈ R D
Revised: D = 64 , and the covariance matrices Σ ∈ R D

Original: × D are constrained to be diagonal, for computational efficiency.
Revised: × D are constrained to be diagonal, for computational efficiency.

Original: In our implementation, each class c is represented by a mixture of M = 5 Gaussians (there are a total of 5 C Gaussian components for a segmentation task with C semantic classes).
Revised: In our implementation, each class c is represented by a mixture of M = 5 Gaussians (there are a total of 5 C Gaussian components for a segmentation task with C semantic classes).

Original: Furthermore, we adopt the winner-take-all assumption [120,121], i.e ., the class-wise responsibility (Eq. 4) is dominated by the largest term, for better performance.
Revised: Furthermore, we adopt the winner-take-all assumption [121,122], i.e ., the class-wise responsibility (Eq. 4) is dominated by the largest term, for better performance.

Original: Training In each training iteration, we conduct one loop of momentum (Sinkhorn) EM ( i.e ., t =1 ) on current training batch as well as the external memory for the generative optimization of GMM, and backpropagate the gradient of the cross-entropy loss on current batch for the discriminative training of the feature extractor.
Revised: Training In each training iteration, we conduct one loop of momentum (Sinkhorn) EM ( i.e ., t =1 ) on current training batch as well as the external memory for the generative optimization of GMM, and backpropagate the gradient of the cross-entropy loss on current batch for the discriminative training of the feature extractor.

Original: The external memory maintains a queue for each component in each class; each queue gathers 32K pixel features from previous training batches in a first in, first out manner.
Revised: The external memory maintains a queue for each component in each class; each queue gathers 32K pixel features from previous training batches in a first in, first out manner.

Original: To improve the diversity of the stored pixel features, we sample a sparse set of 100 pixels per class from each image, instead of directly storing the whole images into the memory.
Revised: To improve the diversity of the stored pixel features, we sample a sparse set of 100 pixels per class from each image, instead of directly storing the whole images into the memory.

Original: Note that the memory is discarded after training, and does not introduce extra overheads in inference.
Revised: Note that the memory is discarded after training, and does not introduce extra overheads in inference.

Original: Inference.
Revised: Inference.

Original: GMMSeg only brings negligible delay in the inference speed compared to the discriminative counterparts (see experiments in §4.3).
Revised: GMMSeg only brings negligible delay in the inference speed compared to the discriminative counterparts (see experiments in §4.3).

Original: For standard (closed-set) semantic segmentation, pixel prediction is made using Bayes rule ( cf .
Revised: For standard (closed-set) semantic segmentation, pixel prediction is made using Bayes rule ( cf .

Original: Eq. 3): arg max c p ( c | x ) , where p ( c | x ) ∝ p ( x | c ) with the uniform class distribution prior: p ( c )=1 /C .
Revised: Eq. 3): arg max c p ( c | x ) , where p ( c | x ) ∝ p ( x | c ) with the uniform class distribution prior: p ( c )=1 /C .

Original: For anomaly segmentation, the pixel-wise uncertainty/anomaly score can be naturally raised as: − max c p ( x | c ) , i.e ., the outlier input should reside in low-probability regions [122].
Revised: For anomaly segmentation, the pixel-wise uncertainty/anomaly score can be naturally raised as: − max c p ( x | c ) , i.e ., the outlier input should reside in low-probability regions [123].

Original:  Experiments
Revised:  Experiments

Original: We respectively examine the efficacy and robustness of GMMSeg on semantic segmentation (§4.1) and anomaly segmentation (§4.2).
Revised: We respectively examine the efficacy and robustness of GMMSeg on semantic segmentation (§4.1) and anomaly segmentation (§4.2).

Original: In §4.3, we provide diagnostic analysis on our core model design.
Revised: In §4.3, we provide diagnostic analysis on our core model design.

Original: 4.1 Experiments
Revised: 4.1 Experiments

Original: on Semantic Segmentation
Revised: on Semantic Segmentation

Original: Datasets.
Revised: Datasets.

Original: We conduct experiments on three widely used semantic segmentation datasets:
Revised: We conduct experiments on three widely used semantic segmentation datasets:

Original: • ADE 20K [53] has 20K/2K/3K images in train / val / test set, with 150 stuff/object categories in total.
Revised: • ADE 20K [53] has 20K/2K/3K images in train / val / test set, with 150 stuff/object categories in total.

Original: • Cityscapes [54] has 2 , 975 / 500 / 1 , 524 fine-labeled images for train / val / test set with 19 classes.
Revised: • Cityscapes [54] has 2 , 975 / 500 / 1 , 524 fine-labeled images for train / val / test set with 19 classes.

Original: • COCO-Stuff
Revised: • COCO-Stuff

Original: [55] has 10 K images ( 9 K/ 1 K for train / test ), pixel-wise labeled with 171 classes.
Revised: [55] has 10 K images ( 9 K/ 1 K for train / test ), pixel-wise labeled with 171 classes.

Original: Base Segmentation Architectures and Backbones.
Revised: Base Segmentation Architectures and Backbones.

Original: For thorough evaluation, we apply GMMSeg to four famous segmentation architectures ( i.e ., DeepLab V3+ [47], OCRNet [48], UPerNet [49], Segformer [7]), with various backbones ( i.e ., ResNet [50], HRNet [51], Swin [52], MiT [7]).
Revised: For thorough evaluation, we apply GMMSeg to four famous segmentation architectures ( i.e ., DeepLab V3+ [47], OCRNet [48], UPerNet [49], Segformer [7]), with various backbones ( i.e ., ResNet [50], HRNet [51], Swin [52], MiT [7]).

Original: For fairness, we re-implement these models using the standardized hyper-parameter setting in MMSegmentation [123].
Revised: For fairness, we re-implement these models using the standardized hyper-parameter setting in MMSegmentation [124].

Original: Training Details.
Revised: Training Details.

Original: GMMSeg is implemented on MMSegmentation [123] and follows the standard training setting for each dataset.
Revised: GMMSeg is implemented on MMSegmentation [124] and follows the standard training setting for each dataset.

Original: All models are initialized with ImageNet-1K [124] pretrained back-
Revised: All models are initialized with ImageNet-1K [125] pretrained back-

Original: [Figure]
Revised: [Figure]

Original: bones and trained with commonly used data augmentations including resizing, flipping, color jittering and cropping.
Revised: bones and trained with commonly used data augmentations including resizing, flipping, color jittering and cropping.

Original: For ADE 20K /COCO-Stuff/Cityscapes, images are cropped to 512 × 512 / 512 × 512 / 768 × 768 and models are trained for 160 K/ 80 K/ 80 K iterations with 16 / 16 / 8 batch size, using 8/16 NVIDIA
Revised: For ADE 20K /COCO-Stuff/Cityscapes, images are cropped to 512 × 512 / 512 × 512 / 768 × 768 and models are trained for 160 K/ 80 K/ 80 K iterations with 16 / 16 / 8 batch size, using 8/16 NVIDIA

Original: Tesla A100 GPUs.
Revised: Tesla A100 GPUs.

Original: Other training hyper-parameters ( i.e ., optimizers, learning rates, weight decays, schedulers) are set as the default in MMSegmentation and can be found in the supplementary.
Revised: Other training hyper-parameters ( i.e ., optimizers, learning rates, weight decays, schedulers) are set as the default in MMSegmentation and can be found in the supplementary.

Original: Inference Details.
Revised: Inference Details.

Original: For ADE 20K and COCO-Stuff, we keep the aspect ratio of test images and rescale the short side to 512.
Revised: For ADE 20K and COCO-Stuff, we keep the aspect ratio of test images and rescale the short side to 512.

Original: For Cityscapes, sliding window inference is used with 768 × 768 window size.
Revised: For Cityscapes, sliding window inference is used with 768 × 768 window size.

Original: Note that for fairness, all our results are reported without any test-time data augmentation.
Revised: Note that for fairness, all our results are reported without any test-time data augmentation.

Original: Quantitative Results.
Revised: Quantitative Results.

Original: Table 1 demonstrates our quantitative results.
Revised: Table 1 demonstrates our quantitative results.

Original: Although mainly focusing on the comparison with the four base segmentation models [7, 47–49], we further include five widely recognized methods [1,3,8,9,90] for completeness.
Revised: Although mainly focusing on the comparison with the four base segmentation models [7, 47–49], we further include five widely recognized methods [1,3,8,9,90] for completeness.

Original: As can be seen, our GMMSeg outperforms all its discriminative counterparts across various datasets, backbones, and network architectures (FCN-style
Revised: As can be seen, our GMMSeg outperforms all its discriminative counterparts across various datasets, backbones, and network architectures (FCN-style

Original: [Table]
Revised: [Table]

Original: and Transformer-like):
Revised: and Transformer-like):

Original: • ADE 20K [53] val .
Revised: • ADE 20K [53] val .

Original: With FCNstyle segmentation neural architectures, i
Revised: With FCNstyle segmentation neural architectures, i

Original: [Table]
Revised: [Table]

Original: e ., DeepLab V3+ and OCR, GMMSeg provides 1.2 % / 1.5 % mIoU gains over corresponding discriminative models.
Revised: e ., DeepLab V3+ and OCR, GMMSeg provides 1.2 % / 1.5 % mIoU gains over corresponding discriminative models.

Original: Similar performance improvements,
Revised: Similar performance improvements,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e ., 1.0 % and 0.6 % , are also obtained with attentive neural architectures, i.e ., Swin-UperNet and SegFormer, manifesting the universality and efficacy of GMMSeg.
Revised: e ., 1.0 % and 0.6 % , are also obtained with attentive neural architectures, i.e ., Swin-UperNet and SegFormer, manifesting the universality and efficacy of GMMSeg.

Original: • Cityscapes[54] val .Again
Revised: • Cityscapes[54] val .Again

Original: our GMMSeg surpasses all its discriminative counterparts by large margins, e.g ., 0.5 % over DeepLab V3+ , 0.8 % over OCRNet, 0.7 % over Swin-UperNet, and 0.6 % over SegFormer, suggesting its wide utility in this field.
Revised: our GMMSeg surpasses all its discriminative counterparts by large margins, e.g ., 0.5 % over DeepLab V3+ , 0.8 % over OCRNet, 0.7 % over Swin-UperNet, and 0.6 % over SegFormer, suggesting its wide utility in this field.

Original: • COCO-Stuff
Revised: • COCO-Stuff

Original: [55] test .
Revised: [55] test .

Original: Our GMMSeg also demonstrates promising results.
Revised: Our GMMSeg also demonstrates promising results.

Original: This is particularly impressive considering these results are achieved by a dense generative classifier, while the semantic segmentation task is commonly considered as a battlefield for discriminative approaches.
Revised: This is particularly impressive considering these results are achieved by a dense generative classifier, while the semantic segmentation task is commonly considered as a battlefield for discriminative approaches.

Original: Qualitative Results.
Revised: Qualitative Results.

Original: In Fig.
Revised: In Fig.

Original: 3, we illustrate the qualitative comparisons of our GMMSeg against
Revised: 3, we illustrate the qualitative comparisons of our GMMSeg against

Original: SegFormer [7].
Revised: SegFormer [7].

Original: It is evident that, among the representative samples in the three datasets, our method yields more accurate predictions when facing challenging scenarios, e.g ., unconspicuous objects.
Revised: It is evident that, among the representative samples in the three datasets, our method yields more accurate predictions when facing challenging scenarios, e.g ., unconspicuous objects.

Original: 4.2 Experiments on Anomaly Segmentation
Revised: 4.2 Experiments on Anomaly Segmentation

Original: Datasets.
Revised: Datasets.

Original: To fully reveal the merits of our generative method, we next test its robustness for abnormal data, i.e ., identifying test samples of unseen classes, using two popular anomaly segmentation datasets:
Revised: To fully reveal the merits of our generative method, we next test its robustness for abnormal data, i.e ., identifying test samples of unseen classes, using two popular anomaly segmentation datasets:

Original: • Fishyscapes Lost&Found [56], built upon [125], has 100 / 275 val / test images.
Revised: • Fishyscapes Lost&Found [56], built upon [126], has 100 / 275 val / test images.

Original: It is collected under the same setup as Cityscapes [54] but with real obstacles on the road.
Revised: It is collected under the same setup as Cityscapes [54] but with real obstacles on the road.

Original: Pixels are labeled as either background ( i.e ., pre-defined Cityscapes classes) or anomaly ( i.e ., other unexpected classes like crate).
Revised: Pixels are labeled as either background ( i.e ., pre-defined Cityscapes classes) or anomaly ( i.e ., other unexpected classes like crate).

Original: • Road Anomaly [36] has 60 images containing anomalous objects in unusual road conditions.
Revised: • Road Anomaly [36] has 60 images containing anomalous objects in unusual road conditions.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: The area under receiver operating characteristics (AUROC), average precision (AP), and false positive rate (FPR 95 ) at a true positive rate of 95%, are adopted following [18,35,56].
Revised: The area under receiver operating characteristics (AUROC), average precision (AP), and false positive rate (FPR 95 ) at a true positive rate of 95%, are adopted following [18,35,56].

Original: [Table]
Revised: [Table]

Original: val .
Revised: val .

Original: Experiment Protocol.
Revised: Experiment Protocol.

Original: As in [17,18,126], we adopt ResNet 101 -DeepLab V3+ architecture.
Revised: As in [17,18,127], we adopt ResNet 101 -DeepLab V3+ architecture.

Original: For completeness, we also report the results of our GMMSeg based on ResNet 101 -FCN and MiT B5 -SegFormer.
Revised: For completeness, we also report the results of our GMMSeg based on ResNet 101 -FCN and MiT B5 -SegFormer.

Original: All our models are the same ones in Table1, i.e ., trained on Cityscapes train only.
Revised: All our models are the same ones in Table1, i.e ., trained on Cityscapes train only.

Original: As GMMSeg estimates class densities p ( x | c ) , it can naturally reject unlikely inputs ( cf .§3.3), i.e ., directly thresholding − max c p ( x | c ) for computing the anomaly segmentation metrics, without any post-processing .
Revised: As GMMSeg estimates class densities p ( x | c ) , it can naturally reject unlikely inputs ( cf .§3.3), i.e ., directly thresholding − max c p ( x | c ) for computing the anomaly segmentation metrics, without any post-processing .

Original: Quantitative Results.
Revised: Quantitative Results.

Original: As shown in Table 2, based on DeepLab V3+ architecture, GMMSeg outperforms all the competitors under the same setting, i.e ., neither using external out-of-distribution data nor extra resynthesis module.
Revised: As shown in Table 2, based on DeepLab V3+ architecture, GMMSeg outperforms all the competitors under the same setting, i.e ., neither using external out-of-distribution data nor extra resynthesis module.

Original: Note that, [17–19] rely on pre-trained discriminative segmentation models and thus have to make post-calibration.
Revised: Note that, [17–19] rely on pre-trained discriminative segmentation models and thus have to make post-calibration.

Original: However, GMMSeg directly derives meaningful confidence scores from likelihood p ( x | c ) .
Revised: However, GMMSeg directly derives meaningful confidence scores from likelihood p ( x | c ) .

Original: Mahalanobis [19] also models data density, yet, merely on pre-trained feature space with a single Gaussian per class.
Revised: Mahalanobis [19] also models data density, yet, merely on pre-trained feature space with a single Gaussian per class.

Original: In contrast, GMMSeg performs much better, proving the superiority of mixture modeling and hybrid training.
Revised: In contrast, GMMSeg performs much better, proving the superiority of mixture modeling and hybrid training.

Original: Even with a weaker architecture, i.e ., FCN,
Revised: Even with a weaker architecture, i.e ., FCN,

Original: GMMSeg still performs robustly.
Revised: GMMSeg still performs robustly.

Original: When adopting SegFormer, better performance is achieved.
Revised: When adopting SegFormer, better performance is achieved.

Original: Qualitative Results.
Revised: Qualitative Results.

Original: In Fig.
Revised: In Fig.

Original: 4, we visualize the anomaly score heatmaps generated by MSP [17]-
Revised: 4, we visualize the anomaly score heatmaps generated by MSP [17]-

Original: DeepLab V3+ [47] and GMMSeg-DeepLab V

Revised: DeepLab V3+ [47] and GMMSeg-DeepLab V

Original: + .
Revised: + .

Original: The softmax based counterpart ignores the anomalies with overconfident predictions; in contrast, GMMSeg naturally rejects them (red colored regions).
Revised: The softmax based counterpart ignores the anomalies with overconfident predictions; in contrast, GMMSeg naturally rejects them (red colored regions).

Original: 4.3 Diagnostic Experiments
Revised: 4.3 Diagnostic Experiments

Original: For in-depth analysis, we conduct ablative studies using DeepLab V3+ [47]-ResNet 101 [50] segmentation architecture.
Revised: For in-depth analysis, we conduct ablative studies using DeepLab V3+ [47]-ResNet 101 [50] segmentation architecture.

Original: Due to limited space, we put some diagnostic experiments in our supplementary material.
Revised: Due to limited space, we put some diagnostic experiments in our supplementary material.

Original: (§4.3), evaluated on ADE 20K [
Revised: (§4.3), evaluated on ADE 20K [

Original: [Table]
Revised: [Table]

Original: Online Hybrid Training.
Revised: Online Hybrid Training.

Original: We first investigate our hybrid training strategy ( cf .
Revised: We first investigate our hybrid training strategy ( cf .

Original: Eq. 11), where the discriminative feature extractor and generative GMM classifier are online optimized iteratively.
Revised: Eq. 11), where the discriminative feature extractor and generative GMM classifier are online optimized iteratively.

Original: Owe to this ingenious design, both components are gradually updated, aligned with and adaptive to each other, making GMMSeg a compact model.
Revised: Owe to this ingenious design, both components are gradually updated, aligned with and adaptive to each other, making GMMSeg a compact model.

Original: To fully demonstrate the effectiveness, we study a variant, DeepLab V3+ + GMM, where a GMM classifier is directly fitted onto the feature space trained with the softmax classifier beforehand.
Revised: To fully demonstrate the effectiveness, we study a variant, DeepLab V3+ + GMM, where a GMM classifier is directly fitted onto the feature space trained with the softmax classifier beforehand.

Original: As shown in Table 3, a clear performance drop is observed, i.e ., mIoU: 46 . 0% → 31 .
Revised: As shown in Table 3, a clear performance drop is observed, i.e ., mIoU: 46 . 0% → 31 .

Original: 6% , revealing the appealing efficacy of our end-to-end hybrid training strategy.
Revised: 6% , revealing the appealing efficacy of our end-to-end hybrid training strategy.

Original: [Table]
Revised: [Table]

Original: Discriminative GMMSeg vs .
Revised: Discriminative GMMSeg vs .

Original: Generative GMMSeg.
Revised: Generative GMMSeg.

Original: Our GMMSeg learns generative GMM via EM, i.e ., max p ( x | c ; ϕ ) , with discriminative representation learning, i.e ., max p ( c | x ; θ ) .
Revised: Our GMMSeg learns generative GMM via EM, i.e ., max p ( x | c ; ϕ ) , with discriminative representation learning, i.e ., max p ( c | x ; θ ) .

Original: A discriminative counterpart can be achieved by end-to-end learning all the parameters, i.e ., { ϕ , θ } , with cross-entropy loss, i.e ., max p ( c | x ; ϕ , θ ) .
Revised: A discriminative counterpart can be achieved by end-to-end learning all the parameters, i.e ., { ϕ , θ } , with cross-entropy loss, i.e ., max p ( c | x ; ϕ , θ ) .

Original: Discriminative GMMSeg sacrifices data characterization for more flexiblility in discrimination, and yields poor performance in open-world setting.
Revised: Discriminative GMMSeg sacrifices data characterization for more flexiblility in discrimination, and yields poor performance in open-world setting.

Original: While inapparent effect on closed-set Cityscapes is observed, which
Revised: While inapparent effect on closed-set Cityscapes is observed, which

Original: [Table]
Revised: [Table]

Original: Standard vs .
Revised: Standard vs .

Original: Sinkhorn
Revised: Sinkhorn

Original: In our
Revised: In our

Original: GMMSeg, we leverage the entropic OT based Sinkhorn EM [28] ( cf .
Revised: GMMSeg, we leverage the entropic OT based Sinkhorn EM [28] ( cf .

Original: Eq. 10) instead of the classic one ( cf .
Revised: Eq. 10) instead of the classic one ( cf .

Original: Eq.
Revised: Eq.

Original: 8) for the generative optimization of the GMM.
Revised: 8) for the generative optimization of the GMM.

Original: In Table 5a, we investigate the impacts of these two different EM algorithms and show that Sinkhorn EM is more favored.
Revised: In Table 5a, we investigate the impacts of these two different EM algorithms and show that Sinkhorn EM is more favored.

Original: More specifically, during the Estep, rather than the vanilla EM assigning data samples to Gaussian components independently,
Revised: More specifically, during the Estep, rather than the vanilla EM assigning data samples to Gaussian components independently,

Original: Sinkhorn EM restricts the assignment with an equipartition constraint.
Revised: Sinkhorn EM restricts the assignment with an equipartition constraint.

Original: As pointed out in [28], incorporating such prior information about the mixing weights of GMM components leads to higher curvature around the global optimum.
Revised: As pointed out in [28], incorporating such prior information about the mixing weights of GMM components leads to higher curvature around the global optimum.

Original: Our empirical results confirm this theoretical finding.
Revised: Our empirical results confirm this theoretical finding.

Original: Number of EM Loop per Training Iteration.
Revised: Number of EM Loop per Training Iteration.

Original: EM algorithm alternates between E-step and M-step for maximum-likelihood inference ( cf .
Revised: EM algorithm alternates between E-step and M-step for maximum-likelihood inference ( cf .

Original: Eq. 6).
Revised: Eq. 6).

Original: In GMMSeg, in order to blend EM with stochastic gradient descent, we adopt an online version of (Sinkhorn) EM based on momentum update.
Revised: In GMMSeg, in order to blend EM with stochastic gradient descent, we adopt an online version of (Sinkhorn) EM based on momentum update.

Original: In
Revised: In

Original: Table 5a, we also study the influence of looping EM different times per training iteration.
Revised: Table 5a, we also study the influence of looping EM different times per training iteration.

Original: We can find that one loop per iteration is enough to catch the drift of the gradually updated feature space.
Revised: We can find that one loop per iteration is enough to catch the drift of the gradually updated feature space.

Original: Number of Gaussian Components per Class.
Revised: Number of Gaussian Components per Class.

Original: In GMMSeg, data distribution of each class is modeled by a mixture of M Gaussian components ( cf .
Revised: In GMMSeg, data distribution of each class is modeled by a mixture of M Gaussian components ( cf .

Original: Eq. 4).
Revised: Eq. 4).

Original: Table 5b shows the results with different values of M .
Revised: Table 5b shows the results with different values of M .

Original: When M = 1 , each class corresponds to a single Gaussian, which is directly estimated via
Revised: When M = 1 , each class corresponds to a single Gaussian, which is directly estimated via

Original: Gaussian Discriminant Analysis, without EM.
Revised: Gaussian Discriminant Analysis, without EM.

Original: This baseline achieves 44 .
Revised: This baseline achieves 44 .

Original: 2% mIoU. After adopting the mixture model, i.e ., M :

Revised: 2% mIoU. After adopting the mixture model, i.e ., M :

Original: → 3 → 5 , the performance is greatly improved, i.e ., mIoU: 44 .
Revised: → 3 → 5 , the performance is greatly improved, i.e ., mIoU: 44 .

Original: 2% → 45 .
Revised: 2% → 45 .

Original: 3% → 46 . 0% .
Revised: 3% → 46 . 0% .

Original: This verifies our hypothesis of class multimodality.
Revised: This verifies our hypothesis of class multimodality.

Original: Yet, further increasing component number ( i.e ., M :5 → 15 ) only brings marginal even negative gains, due to overparameterization.
Revised: Yet, further increasing component number ( i.e ., M :5 → 15 ) only brings marginal even negative gains, due to overparameterization.

Original: [Figure]
Revised: [Figure]

Original: Confidence Calibration.
Revised: Confidence Calibration.

Original: We further study the model calibration of GMMSeg and the discriminative counterpart, i.e ., DeepLab V3+ [49] with the softmax classifier.
Revised: We further study the model calibration of GMMSeg and the discriminative counterpart, i.e ., DeepLab V3+ [49] with the softmax classifier.

Original: In Fig.
Revised: In Fig.

Original: 5, we illustrate the
Revised: 5, we illustrate the

Original: Expected Calibration Error (ECE) [15] along with reliability diagrams, which plot the expected pixel accuracy as a function of confidence [15].
Revised: Expected Calibration Error (ECE) [15] along with reliability diagrams, which plot the expected pixel accuracy as a function of confidence [15].

Original: As seen,
Revised: As seen,

Original: GMMSeg yields better calibrated prefictions, i.e ., smaller gaps between the expected accuracy and confidence.
Revised: GMMSeg yields better calibrated prefictions, i.e ., smaller gaps between the expected accuracy and confidence.

Original: On the other hand, the discriminative softmax produces confidences that deviate more from the true probabilities, and suffers higher calibration error accordingly.
Revised: On the other hand, the discriminative softmax produces confidences that deviate more from the true probabilities, and suffers higher calibration error accordingly.

Original: This again verifies the better reliability and interpretability of GMMSeg compared to its discriminative counterparts.
Revised: This again verifies the better reliability and interpretability of GMMSeg compared to its discriminative counterparts.

Original: Runtime Analysis.
Revised: Runtime Analysis.

Original: The inference speed of GMMSeg is 13 .
Revised: The inference speed of GMMSeg is 13 .

Original: 37 fps, which only yields negligible overhead w.r.t.
Revised: 37 fps, which only yields negligible overhead w.r.t.

Original: its discriminative softmax counterpart, i.e ., 13 .
Revised: its discriminative softmax counterpart, i.e ., 13 .

Original: 37 vs .
Revised: 37 vs .

Original: 16 fps.
Revised: 16 fps.

Original: We measure the fps on a single NVIDIA GeForce RTX 3090 GPU with a batch size of one.
Revised: We measure the fps on a single NVIDIA GeForce RTX 3090 GPU with a batch size of one.

Original:  Conclusion
Revised:  Conclusion

Original: We presented GMMSeg, the first generative neural framework for semantic segmentation.
Revised: We presented GMMSeg, the first generative neural framework for semantic segmentation.

Original: By explicitly modeling data distribution as GMMs, GMMSeg shows promise to solve the intrinsic limitations of current softmax based discriminative regime.
Revised: By explicitly modeling data distribution as GMMs, GMMSeg shows promise to solve the intrinsic limitations of current softmax based discriminative regime.

Original: It successfully optimizes generative GMM with endto-end discriminative representation learning in a compact and collaborative manner.
Revised: It successfully optimizes generative GMM with endto-end discriminative representation learning in a compact and collaborative manner.

Original: This makes
Revised: This makes

Original: GMMSeg principled and well applicable in both closed-set and open-world settings.
Revised: GMMSeg principled and well applicable in both closed-set and open-world settings.

Original: We believe this work provides fundamental insights and can benefit a broad range of application tasks.
Revised: We believe this work provides fundamental insights and can benefit a broad range of application tasks.

Original: As a part of our future work, we will explore our algorithm in image classification and trustworthy AI related tasks.
Revised: As a part of our future work, we will explore our algorithm in image classification and trustworthy AI related tasks.

Original: Acknowledgement.
Revised: Acknowledgement.

Original: This work was partially supported by the Fundamental Research Funds for the Central Universities (No. 226-2022-00087), and by the National Key R&D Program of China (No. 2020AAA0108800).
Revised: This work was partially supported by the Fundamental Research Funds for the Central Universities (No. 226-2022-00087), and by the National Key R&D Program of China (No. 2020AAA0108800).

Original: Wenguan Wang acknowledges partial support from Australian Research
Revised: Wenguan Wang acknowledges partial support from Australian Research

Original: Council (ARC), DECRA DE220101390.
Revised: Council (ARC), DECRA DE220101390.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] Our main claims accurately reflect the paper’s contributions and scope.
Revised: [Yes] Our main claims accurately reflect the paper’s contributions and scope.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We provide detailed discussions on the limitations in the supplemental material.
Revised: [Yes] We provide detailed discussions on the limitations in the supplemental material.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We provide detailed discussions on potential negative societal impacts in the supplemental material.
Revised: [Yes] We provide detailed discussions on potential negative societal impacts in the supplemental material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] We have read the full ethics review guidelines and ensure that our paper conforms to them.
Revised: [Yes] We have read the full ethics review guidelines and ensure that our paper conforms to them.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: No theoretical contribution is claimed.
Revised: No theoretical contribution is claimed.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: No theoretical contribution is claimed.
Revised: No theoretical contribution is claimed.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] We promise code and instructions shall be made publicly available right after acceptance.
Revised: [No] We promise code and instructions shall be made publicly available right after acceptance.

Original: And we believe we have revealed sufficient details of our pipeline (§3.2), implementation details on network architecture/training/inference (§3.3, §4.1, §4.2), publicly available data and commonly adopted evaluation protocols (§4.1, §4.2).
Revised: And we believe we have revealed sufficient details of our pipeline (§3.2), implementation details on network architecture/training/inference (§3.3, §4.1, §4.2), publicly available data and commonly adopted evaluation protocols (§4.1, §4.2).

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] The training details are reported in §4.1.
Revised: [Yes] The training details are reported in §4.1.

Original: Also see §4.3 and the supplemental material for diagnostic experiments.
Revised: Also see §4.3 and the supplemental material for diagnostic experiments.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] We follow the standard evaluation protocol in this field [7,8,10,13] to report and compare the results.
Revised: [No] We follow the standard evaluation protocol in this field [7,8,10,13] to report and compare the results.

Original: It is unaffordable in terms of both time and money to run all experiments multiple times in our paper.
Revised: It is unaffordable in terms of both time and money to run all experiments multiple times in our paper.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Eight or sixteen NVIDIA Tesla A100 GPUs are used for all experiments.
Revised: [Yes] Eight or sixteen NVIDIA Tesla A100 GPUs are used for all experiments.

Original: Also see §4.1.
Revised: Also see §4.1.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite the used assets in §4.1 and §4.2.
Revised: [Yes] We cite the used assets in §4.1 and §4.2.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] We list the license of the used assets in the supplemental material.
Revised: [Yes] We list the license of the used assets in the supplemental material.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] We do not claim newly released assets.
Revised: [N/A] We do not claim newly released assets.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] We provide detailed discussions about data using consent in the supplemental material.
Revised: [Yes] We provide detailed discussions about data using consent in the supplemental material.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] The data used in our paper neither contain any personally identifiable information nor offensive content.
Revised: [Yes] The data used in our paper neither contain any personally identifiable information nor offensive content.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: Neither crowdsourcing nor human subjective research is conducted.
Revised: Neither crowdsourcing nor human subjective research is conducted.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: Neither crowdsourcing nor human subjective research is conducted.
Revised: Neither crowdsourcing nor human subjective research is conducted.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A] Not applicable.
Revised: [N/A] Not applicable.

Original: Neither crowdsourcing nor human subjective research is conducted.
Revised: Neither crowdsourcing nor human subjective research is conducted.