Original: MEDOE: A M ULTI -E XPERT D ECODER AND O UTPUT E NSEMBLE F RAMEWORK FOR L ONG - TAILED S EMANTIC S EGMENTATION
Revised: MEDOE: A M ULTI -E XPERT D ECODER AND O UTPUT E NSEMBLE F RAMEWORK FOR L ONG - TAILED S EMANTIC S EGMENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Long-tailed distribution of semantic categories, which has been often ignored in conventional methods, causes unsatisfactory performance in semantic segmentation on tail categories.
Revised: Long-tailed distribution of semantic categories, which has been often ignored in conventional methods, causes unsatisfactory performance in semantic segmentation on tail categories.

Original: In this paper, we focus on the problem of long-tailed semantic segmentation.
Revised: In this paper, we focus on the problem of long-tailed semantic segmentation.

Original: Although some long-tailed recognition methods ( e g. , resampling/re-weighting) have been proposed in other problems, they are likely to compromise crucial contextual information in semantic segmentation.
Revised: Although some long-tailed recognition methods ( e g. , resampling/re-weighting) have been proposed in other problems, they are likely to compromise crucial contextual information in semantic segmentation.

Original: Therefore, these methods are hardly adaptable to the problem of long-tailed semantic segmentation.
Revised: Therefore, these methods are hardly adaptable to the problem of long-tailed semantic segmentation.

Original: To address this problem, we propose a novel method, named MEDOE, by ensembling and grouping contextual information.
Revised: To address this problem, we propose a novel method, named MEDOE, by ensembling and grouping contextual information.

Original: Specifically, our MEDOE is a two-sage framework comprising a multi-expert decoder (MED) and a multiexpert output ensemble (MOE).
Revised: Specifically, our MEDOE is a two-sage framework comprising a multi-expert decoder (MED) and a multiexpert output ensemble (MOE).

Original: The MED includes several “experts”, each of which takes as input the dataset masked according to the specific categories based on frequency distribution and generates contextual information self-adaptively for classification.
Revised: The MED includes several “experts”, each of which takes as input the dataset masked according to the specific categories based on frequency distribution and generates contextual information self-adaptively for classification.

Original: The MOE then ensembles the experts’ outputs with learnable decision weights.
Revised: The MOE then ensembles the experts’ outputs with learnable decision weights.

Original: As a model-agnostic framework, MEDOE can be flexibly and efficiently coupled with various popular deep neural networks ( e g. , Deeplabv

Revised: As a model-agnostic framework, MEDOE can be flexibly and efficiently coupled with various popular deep neural networks ( e g. , Deeplabv

Original: +,
Revised: +,

Original: OCRNet, and PSPNet) to improve the performance in long-tailed semantic segmentation.
Revised: OCRNet and PSPNet) to improve the performance in long-tailed semantic segmentation.

Original: Experimental results show that the proposed framework outperforms the current methods on both Cityscapes and ADE20K datasets by up to 2% in mIoU and 6% in mAcc.
Revised: Experimental results show that the proposed framework outperforms the current methods on both Cityscapes and ADE20K datasets by up to 2% in mIoU and 6% in mAcc.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Semantic segmentation is defined as the task to predict the semantic category for each pixel in an image.
Revised: Semantic segmentation is defined as the task to predict the semantic category for each pixel in an image.

Original: As a fundamental computer vision task, semantic segmentation is of great importance in various real-world applications ( e.g. , clinical analysis and automatic driving).
Revised: As a fundamental computer vision task, semantic segmentation is of great importance in various real-world applications ( e.g. , clinical analysis and automatic driving).

Original: Conventional methods follow the backbone-context module architecture, where the context module contributes significantly and can be modified to enhance the extraction and aggregation of contextual information  (Chen et al., 2017; He et al., 2019; Huang et al., 2019; Yuan et al., 2019; Zhao et al., 2017).
Revised: Conventional methods follow the backbone-context module architecture, where the context module contributes significantly and can be modified to enhance the extraction and aggregation of surrounding pixels and global information through large-scale field convolution model or attention mechanisms (Chen et al., 2018; He et al., 2019; Huang et al., 2019; Yuan et al., 2019; Zhao et al., 2017).

Original: Such modifications have enabled the recent state-of-the-art methods to improve the performance in semantic segmentation on various benchmark datasets (Caesar et al., 2018;
Revised: Such modifications have enabled the recent state-of-the-art methods to improve the performance in semantic segmentation on various benchmark datasets (Caesar et al., 2018;

Original: Ding et al., 2019; Zhu et al., 2019).
Revised: Ding et al., 2019; Zhu et al., 2019).

Original: [Table]
Revised: [Table]

Original: Despite the overall impressive performance, the above-mentioned semantic segmentation methods still face challenges from the perspective of data distribution.
Revised: Despite the overall impressive performance, the above-mentioned semantic segmentation methods still face challenges from the perspective of data distribution.

Original: For example, Table 1 shows the results for head , body , and tail categories ( i.e. , categories ranked from top to bottom by pixel frequency) on benchmark datasets using
Revised: For example, Table 1 shows the results for head , body , and tail categories ( i.e. , categories ranked from top to bottom by pixel frequency) on benchmark datasets using

Original: DeepLabv3+ (He et al., 2016), where positive correlations can be found between results and data distribution.
Revised: DeepLabv3+ (He et al., 2016), where positive correlations can be found between results and data distribution.

Original: In other words, the performance declines for tail categories.
Revised: In other words, the performance declines for tail categories.

Original: This suggests the problem of long-tailed distribution in semantic segmentation: a few head categories dominate the majority of pixels, whereas many tail categories correspond to significantly less pixels.
Revised: This suggests the problem of long-tailed distribution in semantic segmentation: a few head categories dominate the majority of pixels, whereas many tail categories correspond to significantly less pixels.

Original: Specifically, if all categories are processed in the same pattern while a long-tailed distribution exists, head categories may excessively influence the training and negatively impact on learning the contextual information about tail ones, leading to unsatisfactory pixel-level classification.
Revised: Specifically, if all categories are processed in the same pattern while a long-tailed distribution exists, head categories may excessively influence the training and negatively impact on learning the contextual information about tail ones, leading to unsatisfactory pixel-level classification.

Original: Solving the problem of long-tailed semantic segmentation is critical to real-world scenarios.
Revised: Solving the problem of long-tailed semantic segmentation is critical to real-world scenarios.

Original: As shown in Figure 1, distinguishing poles from street scene images may prevent potential accidents.
Revised: As shown in Figure 1, distinguishing poles from street scene images may prevent potential accidents.

Original: To the best of our knowledge, this paper is the first to explicitly focus on the long-tailed semantic segmentation, and aims to provide a straightforward solution by extending the existent recognition methods, which adopt various strategies to solve the problem of long-tailed distribution in image classification (Buda et al., 2018; Huang et al., 2016; Gupta et al., 2019).
Revised: To the best of our knowledge, this paper is the first to explicitly focus on the long-tailed semantic segmentation, and aims to provide a straightforward solution by extending the existent recognition methods, which adopt various strategies to solve the problem of long-tailed distribution in image classification (Buda et al., 2018; Huang et al., 2016; Gupta et al., 2019).

Original: These methods can be generally categorized into re-weighting , re-sampling , and ensembling and grouping .
Revised: These methods can be generally categorized into re-weighting , re-sampling , and ensembling and grouping .

Original: The re-weighting methods increase the weights of tail categories but decrease those of head ones (Cao et al., 2019;
Revised: The re-weighting methods increase the weights of tail categories but decrease those of head ones (Cao et al., 2019b;

Original: Liu et al., 2019), following the assumption that images are nearly independent and identically ( i.e. , i.i.d.) distributed to address the imbalance of training set.
Revised: Liu et al., 2019), following the assumption that images are nearly independent and identically ( i.e. , i.i.d.) distributed to address the imbalance of training set.

Original: This assumption enables the classification accuracy for each category to depend on the frequency of the corresponding images (Cao et al., 2019; Cui et al., 2022).
Revised: This assumption enables the classification accuracy for each category to depend on the frequency of the corresponding images (Cao et al., 2019b; Cui et al., 2022).

Original: The re-sampling methods conduct under-sampling for the head categories and over-sampling or even data augmentation for the tail ones (He & Garcia, 2009;
Revised: The re-sampling methods conduct under-sampling for the head categories and over-sampling or even data augmentation for the tail ones (He & Garcia, 2009;

Original: Kim et al., 2020;
Revised: Kim et al., 2020;

Original: Chu et al., 2020), usually following a random sampling strategy to ensure fairness.
Revised: Chu et al., 2020), usually following a random sampling strategy to ensure fairness.

Original: The ensembling and grouping methods start with training a feature extractor on the whole of an imbalanced dataset as representation learning, then adjust the margins of classifiers using multi-expert frameworks (Xiang et al., 2020; Zhang et al., 2021;
Revised: The ensembling and grouping methods start with training a feature extractor on the whole of an imbalanced dataset as representation learning, then adjust the margins of classifiers using multi-expert frameworks (Xiang et al., 2020; Zhang et al., 2021;

Original: Zhou et al., 2020) for re-balancing (Kang et al., 2019).
Revised: Zhou et al., 2020) for re-balancing (Kang et al., 2019).

Original: Nevertheless, the afore-mentioned recognition methods for long-tailed image classification can hardly be adapted for long-tailed semantic segmentation.
Revised: Nevertheless, the afore-mentioned recognition methods for long-tailed image classification can hardly be adapted for long-tailed semantic segmentation.

Original: Specifically, the re-weighting methods are not able to serve pixel-level classification as a pixel is usually highly correlated to the surrounding ones ( i.e. , not i.i.d.)
Revised: Specifically, the re-weighting methods are not able to serve pixel-level classification as a pixel is usually highly correlated to the surrounding ones ( i.e. , not i.i.d.)

Original: given the contextual information in the image (Lin et al., 2017).
Revised: given the contextual information in the image (Lin et al., 2017).

Original: They may also cause a see-saw phenomenon: the accuracy for head categories is compromised whereas tail categories are on purpose emphasized.
Revised: They may also cause a see-saw phenomenon: the accuracy for head categories is compromised whereas tail categories are on purpose emphasized.

Original: Based on random sampling, the re-sampling methods lead to a large number of independent pixels that undermine the contextual information of an image and can be detrimental to semantic segmentation.
Revised: Based on random sampling, the re-sampling methods lead to a large number of independent pixels that undermine the contextual information of an image and can be detrimental to semantic segmentation.

Original: The ensembling and grouping methods rely heavily on re-balancing, where classifier re-adjusting is not well adapted for semantic segmentation due to ignoring the difference among head, body, and tail categories in contextual information.
Revised: The ensembling and grouping methods rely heavily on re-balancing, where classifier re-adjusting is not well adapted for semantic segmentation due to ignoring the difference among head, body, and tail categories in contextual information.

Original: Motivated by the observation about the ensembling and grouping methods, we propose MEDOE, a two-stage multi-expert decoder and output ensemble framework for long-tailed semantic segmentation.
Revised: Motivated by the observation about the ensembling and grouping methods, we propose MEDOE, a two-stage multi-expert decoder and output ensemble framework for long-tailed semantic segmentation.

Original: At Stage 1, the feature map extracted by a backbone trained on the whole of an imbalanced dataset, which represents the elementary knowledge, is first passed to a multi-expert decoder (MED).
Revised: At Stage 1, the feature map extracted by a backbone trained on the whole of an imbalanced dataset, which represents the elementary knowledge, is first passed to a multi-expert decoder (MED).

Original: Each expert ( i.e. , a pair of context module and classifier head) in the MED works on the dataset where pixels and their corresponding labels of dominant categories ( i.e. , body and tail categories) in each image are masked.
Revised: Each expert ( i.e. , a pair of context module and classifier head) in the MED works on the dataset where pixels and their corresponding labels of dominant categories ( i.e. , body and tail categories) in each image are masked.

Original: Together with other constraints, this expert-specific pixel-masking strategy enables the experts to reduce the impact of head categories and irrelevant pixels and acquire the contextual information of body or tail categories.
Revised: Together with other constraints, this expert-specific pixel-masking strategy enables the experts to reduce the impact of head categories and irrelevant pixels and acquire the contextual information of body or tail categories.

Original: The following Stage 2 deploys a multi-expert output ensemble (MOE) to ensemble the outputs of all experts from Stage 1 using decision weights.
Revised: The following Stage 2 deploys a multi-expert output ensemble (MOE) to ensemble the outputs of all experts from Stage 1 using decision weights.

Original: Instead of being user-specified, these weights are learned by a decision-maker to avoid the negative impact of the constraints.
Revised: Instead of being user-specified, these weights are learned by a decision-maker to avoid the negative impact of the constraints.

Original: The proposed framework is model-agnostic and can be integrated with any popular semanticsegmentation method, such as DeepLabv3+ (Chen et al., 2017),
Revised: The proposed framework is model-agnostic and can be integrated with any popular semantic

Original: To evaluate the MEDOE’s performance in long-tailed semantic segmentation, especially for tail categories, we conduct theoretical and experimental analyses using the metrics of mean Intersectionover-Union (mIoU) and mean pixel accuracy (mAcc).
Revised: To evaluate the MEDOE’s performance in long-tailed semantic segmentation, especially for tail categories, we conduct theoretical and experimental analyses using the metrics of mean Intersectionover-Union (mIoU) and mean pixel accuracy (mAcc).

Original: They demonstrate the effectiveness of the proposed framework in boosting mAcc while keeping mIoU from decrease.
Revised: They demonstrate the effectiveness of the proposed framework in boosting mAcc while keeping mIoU from decrease.

Original: In addition, we examine the potentials of MED and MOE despite that the information needed by the expert-specific pixelmasking strategy is unavailable in the inference phase of long-tailed semantic segmentation.
Revised: In addition, we examine the potentials of MED and MOE despite that the information needed by the expert-specific pixelmasking strategy is unavailable in the inference phase of long-tailed semantic segmentation.

Original: The key contributions of this paper can be summarized as follows:
Revised: The key contributions of this paper can be summarized as follows:

Original: • We conduct an empirical study to discover the problem of long-tailed distribution in semantic segmentation and reveal its significance.
Revised: • We conduct an empirical study to discover the problem of long-tailed distribution in semantic segmentation and reveal its significance.

Original: • We advocate mAcc as a more important metric to evaluate the performance for body and tail categories in long-tailed semantic segmentation.
Revised: • We advocate mAcc as a more important metric to evaluate the performance for body and tail categories in long-tailed semantic segmentation.

Original: • We propose a model-agnostic multi-expert decoder and output ensemble framework that outperforms the several popular methods up to 2% in mIoU and 6% in mAcc on Cityscapes (Cordts et al., 2016) and
Revised: • We propose a model-agnostic multi-expert decoder and output ensemble framework that outperforms the several popular methods up to 2% in mIoU and 6% in mAcc on Cityscapes (Cordts et al., 2016) and

Original: ADE20K
Revised: ADE20K

Original: (Zhou et al., 2017) datasets.
Revised: (Zhou et al., 2017) datasets.

Original: • We demonstrate the ideal information referred to Oracle improves substantially the results, with an average gain of 5% in mIoU and 6% in mAcc on Cityscapes and 12% in mIoU and 19% in mAcc on ADE20K datasets.
Revised: • We demonstrate the ideal information referred to Oracle improves substantially the results, with an average gain of 5% in mIoU and 6% in mAcc on Cityscapes and 12% in mIoU and 19% in mAcc on ADE20K datasets.

Original: The results open a very promising direction for future research.
Revised: The results open a very promising direction for future research.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Semantic Segmentation.
Revised: Semantic Segmentation.

Original: FCN (Long et al., 2015) is regarded as the pioneer in the field because of introducing the full convolution on the whole image and formulating the semantic segmentation task as per-pixel classification with the basic framework of backbone-context module .
Revised: FCN (Long et al., 2015) is regarded as the pioneer in the field because of introducing the full convolution on the whole image and formulating the semantic segmentation task as per-pixel classification with the basic framework of backbone-context module .

Original: Subsequently, various advanced methods have been introduced based on FCN, they can be roughly divided into two directions.
Revised: Subsequently, various advanced methods have been introduced based on FCN, they can be roughly divided into two directions.

Original: One is to design novel backbone for more robust feature representation (Wang et al., 2020a; Yu & Koltun, 2017).
Revised: One is to design novel backbone for more robust feature representation (Wang et al., 2020a; Yu & Koltun, 2017).

Original: HRNet introduced a parallel backbone network to generate highresolution representations.
Revised: HRNet introduced a parallel backbone network to generate highresolution representations.

Original: The other is to enrich contextual information for each pixel (Fu et al., 2019;
Revised: The other is to enrich contextual information for each pixel (Fu et al., 2019;

Original: He et al., 2019; Huang et al., 2019; Yuan et al., 2019).
Revised: He et al., 2019; Huang et al., 2019; Yuan et al., 2019).

Original: For instance, combining high-level feature and low-level feature to extract global information (Amirul Islam et al., 2017; Badrinarayanan et al., 2017), introducing large receptive field, such as dilated or atrous convolutions (Chen et al., 2017; Zhao et al., 2017) to gather multi-scale contextual cues and building the feature pyramids (Kirillov et al., 2019).
Revised: For instance, combining high-level feature and low-level feature to extract global information (Amirul Islam et al., 2017; Badrinarayanan et al., 2017), introducing large receptive field, such as dilated or atrous convolutions (Chen et al., 2018; Zhao et al., 2017) to gather multi-scale contextual cues and building the feature pyramids (Kirillov et al., 2019).

Original: However, these approaches ignore the impact of data distribution, and our work is the first to explicitly focus on the long-tailed data distribution in semantic segmentation.
Revised: However, these approaches ignore the impact of data distribution, and our work is the first to explicitly focus on the long-tailed data distribution in semantic segmentation.

Original: Long-tailed visual recognition.
Revised: Long-tailed visual recognition.

Original: Re-balancing/Re-weighting: The most widely-used and straightforward solution for long-tailed distribution is re-balancing the contribution of each category in the training phase.
Revised: Re-balancing/Re-weighting: The most widely-used and straightforward solution for long-tailed distribution is re-balancing the contribution of each category in the training phase.

Original: Re-weighting approaches (Cao et al., 2019; Liu et al., 2019; Lin et al., 2017) adjust the loss function or boost larger weights on tail categories.
Revised: Re-weighting approaches (Cao et al., 2019b; Liu et al., 2019; Lin et al., 2017) adjust the loss function or boost larger weights on tail categories.

Original: Re-balancing approaches (He & Garcia, 2009; Kim et al., 2020;
Revised: Re-balancing approaches (He & Garcia, 2009; Kim et al., 2020;

Original: Chu et al., 2020) achieve data balance based on over-sampling the lowfrequency categories, under-sampling the high-frequency categories or even data augmentation, by generating additional samples to complement tail categories.
Revised: Chu et al., 2020) achieve data balance based on over-sampling the lowfrequency categories, under-sampling the high-frequency categories or even data augmentation, by generating additional samples to complement tail categories.

Original: However, whether sample-wise or losswise re-balancing approaches increase the performance of minority categories while the accuracy of majority ones is compromised.
Revised: However, whether sample-wise or losswise re-balancing approaches increase the performance of minority categories while the accuracy of majority ones is compromised.

Original: Ensembling and Grouping:
Revised: Ensembling and Grouping:

Original: Recent studies show a new trend of overcoming the long-tailed problem by multi-expert or multi-branch strategy.
Revised: Recent studies show a new trend of overcoming the long-tailed problem by multi-expert or multi-branch strategy.

Original: BBN (Zhou et al., 2020) adopted two-branches to focus on normal and reversed sampling.
Revised: BBN (Zhou et al., 2020) adopted two-branches to focus on normal and reversed sampling.

Original: LFME (Xiang et al., 2020),
Revised: LFME (Xiang et al., 2020),

Original: RIDE (Wang et al., 2020b) and ACE (Cai et al., 2021) trained on relatively balanced sub-groups, which separated training data by category frequency, and ensembled together into a multi-expert architecture.
Revised: RIDE (Wang et al., 2020b) and ACE (Cai et al., 2021) trained on relatively balanced sub-groups, which separated training data by category frequency, and ensembled together into a multi-expert architecture.

Original: Those methods learn diverse classifiers parallel with knowledge distillation, distributionaware expert selection or complementary experts.
Revised: Those methods learn diverse classifiers parallel with knowledge distillation, distributionaware expert selection or complementary experts.

Original: Despite being suffered from re-balance tricks, the success gained on image recognition has shown the great potential of these methods.
Revised: Despite being suffered from re-balance tricks, the success gained on image recognition has shown the great potential of these methods.

Original:  M ETHODS
Revised:  M ETHODS

Original: Following the case described in Table 1, each category in semantic segmentation is tagged head , body , or tail , according to its rank based on the corresponding pixel frequency.
Revised: Following the case described in Table 1, each category in semantic segmentation is tagged head , body , or tail , according to its rank based on the corresponding pixel frequency.

Original: In particular, body categories refer to those ranked lower than the head but higher than the tail.
Revised: In particular, body categories refer to those ranked lower than the head but higher than the tail.

Original: Figure 2 provides an overview about the two-stage MEDOE framework.
Revised: Figure 2 provides an overview about the two-stage MEDOE framework.

Original: At Stage 1, given a low-level feature map ex- tracted by a backbone from the input image, a multi-expert decoder (MED) is employed to discover contextual information, which has considerable importance in semantic segmentation.
Revised: At Stage 1, given a low-level feature map extracted by a backbone from the input image, a multi-expert decoder (MED) is employed to discover contextual information, which has considerable importance in semantic segmentation.

Original: The MED includes several experts, each of which is composed of a context module and a classifier head, corresponding to head, body, and tail categories (respectively referred to as head expert , body expert , and tail expert ).
Revised: The MED includes several experts, each of which is composed of a context module and a classifier head, corresponding to head, body, and tail categories (respectively referred to as head expert , body expert , and tail expert ).

Original: Following an expert-specific pixel-masking strategy, each expert processes a unique dataset with the image pixels of certain categories masked, which ensures the expert to focus on its corresponding categories and predict with higher confidence.
Revised: Following an expert-specific pixel-masking strategy, each expert processes a unique dataset with the image pixels of certain categories masked, which ensures the expert to focus on its corresponding categories and predict with higher confidence.

Original: Minimizing both Cross-Entropy (CE) and auxiliary loss functions, the head expert updates the backbone’s parameters on the entire dataset, whereas the others refine their own context modules, in order to constrain the effect of interfering pixels and the distribution of masked data.
Revised: Minimizing both Cross-Entropy (CE) and auxiliary loss functions, the head expert updates the backbone’s parameters on the entire dataset, whereas the others refine their own context modules, in order to constrain the effect of interfering pixels and the distribution of masked data.

Original: At Stage 2, a multi-expert output ensemble (MOE) is designed to aggregate the outputs from different experts for predicting the label of each pixel labels.
Revised: At Stage 2, a multi-expert output ensemble (MOE) is designed to aggregate the outputs from different experts for predicting the label of each pixel labels.

Original: The weights used by MOE are from a learnable decision-maker, instead of those user-specified, so as to avoid the negative impact of the afore-mentioned constraints.
Revised: The weights used by MOE are from a learnable decision-maker, instead of those user-specified, so as to avoid the negative impact of the afore-mentioned constraints.

Original: 3. MED: M ULTI -E XPERT D ECODER
Revised: 3. MED: M ULTI -E XPERT D ECODER

Original: The MED is designed to draw out different contextual information from the feature map extracted by the backbone, in order to adjust the boundaries of classifiers, based on the observation that a model’s performance in semantic segmentation tend to be superior for head categories.
Revised: The MED is designed to draw out different contextual information from the feature map extracted by the backbone, in order to adjust the boundaries of classifiers, based on the observation that a model’s performance in semantic segmentation tend to be superior for head categories.

Original: Specifically, the parameters of an expert are determined by its corresponding pixel-masking dataset, while the local optima of head categories can be avoided.
Revised: Specifically, the parameters of an expert are determined by its corresponding pixel-masking dataset, while the local optima of head categories can be avoided.

Original: The conventional context modules with largescale receptive field convolutions, such as ASPP (Chen et al., 2017), PPM (Zhao et al., 2017) and
Revised: The conventional context modules with largescale receptive field convolutions, such as ASPP (Chen et al., 2018), PPM (Zhao et al., 2017) and

Original: OCR (Yuan et al., 2019), can self-adaptively generate different contextual information from dominant categories in the expert-specific pixel-masking datasets.
Revised: OCR (Yuan et al., 2019), can self-adaptively generate different contextual information from dominant categories in the expert-specific pixel-masking datasets.

Original: This process can be defined as follows,
Revised: This process can be defined as follows,

Original: Given a training set D=
Revised: Given a training set D

Original:  {  X ; Y  } , X denotes the data and Y denotes ground truth labels.
Revised: = { ( X , Y ); C } , X denotes the data and Y denotes ground truth labels, with C categories in total.

Original: Then we realign the labels Y according to the categories frequency in pixel level, Y = {y 1 ,y 2 , . . .
Revised: Then we realign C according to the categories frequency in pixel level, C = { 1 , 2 , . . .

Original: , y C } , where ( F (y i ) >
Revised: , c } , where ( F ( i ) >

Original: F (y j ) , if ( i > j )) , F (y i ) denotes the frequency of categoriesy i , C denotes the categories in total.
Revised: F ( j ) , if ( i > j )) , F ( i ) denotes the frequency of categories i .

Original: For K experts E = {E 1 , E 2 , . . .
Revised: For K experts E = {E 1 , E 2 , . . .

Original: , E K } , we assign K expert-specific masked datasets for differenct experts S
Revised: , E K } , we assign K expert-specific masked datasets for different experts S

Original: = {S 1 , S 2 , . . .
Revised: = {S 1 , S 2 , . . .

Original: , S K } .
Revised: , S K } .

Original: In this paper, we set K = 3 by default, based 
Revised: In this paper, we set K = 3 by default, based on the intuition of head, body and tail categories, c b and c t denotes the first body and tail categories.

Original: [Equation]
Revised: [Equation]

Original: In this case, the body and tail categories can be exposed to their dominant  experts.
Revised: In this case, the body and tail categories can be exposed to their dominant experts, and we also adopted some overlapped categories to expose fewer categories to more experts.

Revised: For hard pixel segmentation, the overlapping strategy allows the experts who dominate the category to integrate the outputs of others.

Original: Followed the shared backbone f θ , experts have chance to extract contextual information focused on “experts” own categories by MED.
Revised: Followed the shared backbone f θ , experts have chance to extract contextual information focused on “experts” own categories by MED.

Original: To reduce overfitting in minority categories, we only adopt a small number of convolutions in context module Ψ =
Revised: To reduce overfitting in minority categories, we only adopt a small number of convolutions in context module Ψ =

Original: { ψ θ 1 , ψ [Equation]
Revised: { ψ θ 1 , ψ θ 2 , . . .

Original: , ψ θ K } , and classifier head 
Revised: , ψ θ K } , and classifier head H

Original: [Equation] = [Equation]
Revised: = { h θ 1 , h θ 2 , . . .

Original: [Equation] K } .
Revised: , h θ K } .

Original: Finally, the expert E i outputs prediction:
Revised: Finally, the expert E i outputs prediction:

Original: [Equation]
Revised: [Equation]

Original: Diverse data distribution-aware loss function.
Revised: Diverse data distribution-aware loss function.

Original: We require each expert to perform well in their expert-specific masked dataset.
Revised: We require each expert to perform well in their expert-specific masked dataset.

Original: Therefore, instead of constraining the final prediction as prior approaches, we apply the diverse data distribution-aware loss function on each separately.
Revised: Therefore, instead of constraining the final prediction as prior approaches, we apply the diverse data distribution-aware loss function on each separately.

Original: For each expert E i , we use the CE loss function to guide the pixel classification on the dataset S i .
Revised: For each expert E i , we use the CE loss function to guide the pixel classification on the dataset S i .

Original: [Equation]
Revised: [Equation]

Original: However, semantic segmentation train each image end to end, and some pixels were defined as interfering pixels (IP).
Revised: However, semantic segmentation trains each image end to end, and some categories were defined as interfering categories (IC).

Original: IP set S IPi refers to the pixels belong to the image but not in S i , which means S IPi ∪S
Revised: IC set S ICi refers to the pixels’ categories not in S i , which means S ICi ∪ S

Original: i =
Revised: i =

Original: Y .
Revised: C .Since

Original: It is inevitable that we need to classify these pixels, even if they fall into majority categories, which can confuse E i .
Revised: we expect each expert to focus on the dominating categories, inevitably, S ICi is the main source of confusion to E i .

Original: We devise a novel suppression compensation measure by defining an auxiliary loss function to suppress the IP with a regularization term.
Revised: We devise a novel suppression compensation measure by defining an auxiliary loss function to suppress the IC with an L2 regularization term.

Original: In addition to avoid the over-confidence of minority categories, for each expert E i , we minimize KL-divergence between the expert classification probability p with the reality data distribution q  over S
Revised: In addition to avoiding the over-confidence of confusing categories, for each expert E i , we minimize KL-divergence between the expert classification probability p with the reality label distribution q in ground truth y over S

Original: i categories.
Revised: i categories.

Original: [Equation]

Original: aux

Original: [Equation]
Revised: [Equation]

Original: Overall, the diverse data distribution-aware loss function for expert E i is
Revised: Overall, the diverse data distribution-aware loss function for expert E i is

Original: [Equation]
Revised: [Equation]

Original: where α is the hyper-parameter to balance the loss of L ce and L aux .
Revised: where α is the hyper-parameter to balance the loss of L ce and L aux .

Original: We empirically set α = 0.2 by default.
Revised: We empirically set α = 0.2 by default.

Original: Since these dominant categories are complementary to each other, the K experts learned at
Revised: Since these dominant categories are complementary to each other, the K experts learned at

Original: Stage 1 are good and distinctive from each other.
Revised: Stage 1 are good and distinctive from each other.

Original: Oracle case with known decision information.
Revised: Oracle case with known decision information.

Original: As mentioned, the MED, loss terms and expertspecific pixel-masking strategy enable each expert to focus better on their dominant categories.
Revised: As mentioned, the MED, loss terms and expertspecific pixel-masking strategy enable each expert to focus better on their dominant categories.

Original: For the intuitive assumption of the Stage 1, our model will achieve the theoretically optimal result when each pixel gets the prediction through the expert who dominates its category.
Revised: For the intuitive assumption of the Stage 1, our model will achieve the theoretically optimal result when each pixel gets the prediction through the expert who dominates its category.

Original: We call this situation the “Oracle” case, and take such a case as the theoretical upper bound of our method.
Revised: We call this situation the “Oracle” case, and take such a case as the theoretical upper bound of our method.

Original: Specifically, for pixel instance x , when its ground truth label y  satisfied: y ∈ S n & y / ∈ S n +1 , we choose expert E n to output the final classification probability p O of instance x :
Revised: Specifically, due to the overlap strategy, when the ground truth label y of pixel x satisfied: y ∈ S n & y / ∈ S n +1 , we choose expert E n to output the final classification probability p O of instance x :

Original: [Equation]
Revised: [Equation]

Original: 3. MOE: M ULTI - EXPERT O UTPUT E NSEMBLE
Revised: 3. MOE: M ULTI - EXPERT O UTPUT E NSEMBLE

Original: The MOE with a learnable decision-maker is devised upon unavailable information on decisions, and aims to ensemble the probabilities for classification.
Revised: The MOE with a learnable decision-maker is devised upon unavailable information on decisions, and aims to ensemble the probabilities for classification.

Original: The MOE takes as input the outputs from all experts at Stage 1, and makes a decision on label selection.
Revised: The MOE takes as input the outputs from all experts at Stage 1, and makes a decision on label selection.

Original: For the n-th expert, if the ground truth label y of a pixel x belongs to the expert’s dominant categories ( i.e. , y ∈ S n ), ideally the expert should be selected, y se = n .
Revised: For the n-th expert, if the ground truth label y of a pixel x belongs to the expert’s dominant categories ( i.e. , y ∈ S n ), ideally the expert should be selected, y se = n .

Original: We construct a feature embedding layer and a classifier to learn this decision-maker.
Revised: We construct a feature embedding layer and a classifier to learn this decision-maker.

Original: Specifically, we concatenate the classification probabilities p for x through the K experts from first to k-th expert and project them to a scale by a 2D convolutional layer W 1 , followed by ReLU, and finally, apply a k-th classifier W k 2 with a Softmax function to get selection probabilities s between each expert in [0,1]:
Revised: Specifically, we concatenate the classification probabilities p for x through the K experts from first to k-th expert and project them to a scale by a 2D convolutional layer W 1 , followed by ReLU, and finally, apply a k-th classifier W k 2 with a Softmax function to get selection probabilities s between each expert in [0,1]:

Original: [Equation]
Revised: [Equation]

Original: In the training phase, the decision-maker is optimized with a CE loss:
Revised: In the training phase, the decision-maker is optimized with a CE loss:

Original: [Equation]
Revised: [Equation]

Original: In the inference phase, instead of a simply hard decision strategy, for instance x we adopt the select probability s as a soft decision weight W for all experts, the final classification probabilities:
Revised: In the inference phase, instead of a simply hard decision strategy, for instance x we adopt the select probability s as a soft decision weight W for all experts, the final classification probabilities:

Original: [Equation]
Revised: [Equation]

Original: 3. M ETRIC THEORETICAL ANALYSIS
Revised: 3. M ETRIC THEORETICAL ANALYSIS

Original: Theoretical analysis between mIoU and mAcc in Semantic Segmentation.
Revised: Theoretical analysis between mIoU and mAcc in Semantic Segmentation.

Original: Due to the experimental results on Cityscapes, in Sec 4, we were surprised to find that the mAcc made a great improvement while the mIoU remained stable.
Revised: Due to the experimental results on Cityscapes, in Sec 4, we were surprised to find that the mAcc made a great improvement while the mIoU remained stable.

Original: This misalignment inspires us to explore the correlation between mIoU and mAcc in long-tailed semantic segmentation because we use mIoU as the primary evaluation metric in segmentation, but traditional long-tailed recognition adopts mAcc as a standard metric.
Revised: This misalignment inspires us to explore the correlation between mIoU and mAcc in long-tailed semantic segmentation because we use mIoU as the primary evaluation metric in segmentation, but traditional long-tailed recognition adopts mAcc as a standard metric.

Original: We refer TP , FN and FP to true positives, false negatives and false positives.
Revised: We refer TP , FN and FP to true positives, false negatives and false positives.

Original: Acc i and IoU i refer to pixel accuracy and Intersection-over-Union of category i can be present as,
Revised: Acc i and IoU i refer to pixel accuracy and Intersection-over-Union of category i can be present as,

Original: [Equation]
Revised: [Equation]

Original: The correlation between FP and FN refers to : (cid:80) ci =

Revised: The correlation between FP and FN refers to : (cid:80) ci =

Original: FP i = (cid:80) ci
Revised: FP i = (cid:80) ci

Original: =1 FN i .
Revised: =1 FN i .

Original: Meanwhile, (cid:80) ci
Revised: Meanwhile, (cid:80) ci

Original: =1 FN i is more affected by the incorrect instances in dominant head categories than the tail categories (See
Revised: =1 FN i is more affected by the incorrect instances in dominant head categories than the tail categories (See

Original: Appendix for details).
Revised: Appendix for details).

Original: Therefore, when evaluating the IoU on tail categories, the FN -related FP will be suppressed by head categories, and make IoU remain stable.
Revised: Therefore, when evaluating the IoU on tail categories, the FN -related FP will be suppressed by head categories, and make IoU remain stable.

Original: We believe that mIoU is a metric to focus on the segmentation results of whole images, which will be affected by head categories.
Revised: We believe that mIoU is a metric to focus on the segmentation results of whole images, which will be affected by head categories.

Original: Different from mIoU, the terms in mAcc formula are correlated to their own categories, which is a more fair metric of the improvement of body and tail categories.
Revised: Different from mIoU, the terms in mAcc formula are correlated to their own categories, which is a more fair metric of the improvement of body and tail categories.

Original: Thus, we regarded mAcc as a relatively important metric in our experiments.
Revised: Thus, we regarded mAcc as a relatively important metric in our experiments.

Original: Meanwhile, consider a situation where, in pursuit of accuracy growth, we classify all the pixels nearby the tail category instances into this category, resulting in a substantial increase about FP i , which is harmful for segmentation.
Revised: Meanwhile, consider a situation where, in pursuit of accuracy growth, we classify all the pixels nearby the tail category instances into this category, resulting in a substantial increase about FP i , which is harmful for segmentation.

Original: According to
Revised: According to

Original: Remark 1 , only satisfies the precondition of no significant drop in mIoU, the mAcc increase does not cause the harm mentioned previously and tends to better performance in tail categories.
Revised: Remark 1 , only satisfies the precondition of no significant drop in mIoU, the mAcc increase does not cause the harm mentioned previously and tends to better performance in tail categories.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted the experiments on two challenging benchmark datasets, by integrating our MEDOE framework into DeepLabv3+, PSPNet, OCRNet and Segformer to evaluate the effectiveness.
Revised: We conducted the experiments on two challenging benchmark datasets, by integrating our MEDOE framework into DeepLabv3+, PSPNet, OCRNet and Segformer to evaluate the effectiveness.

Original: Due to the limited space, benchmarks & implementation details are left in the Appendix .
Revised: Due to the limited space, benchmarks & implementation details are left in the Appendix .

Original: 4. C OMPARISON R ESULTS
Revised: 4. C OMPARISON R ESULTS

Original: Cityscapes.
Revised: Cityscapes.

Original: To validate the flexibility and efficiency of our MEDOE framework, we experiment with our method on Cityscapes.
Revised: To validate the flexibility and efficiency of our MEDOE framework, we experiment with our method on Cityscapes.

Original: As shown in Table 2 and 4, compared with the current methods,our MEDOE significantly improved mAcc without reducing mIoU, which has been proved effective in
Revised: As shown in Table 2 and 4, compared with the current methods, MEDOE significantly improved mAcc without reducing mIoU and too many extra parameters.

Original: Sec 3. Specifically, for ResNet-50c, ResNet-101c, HRNet-W48 and MIT-B3 (Transformer-based), there are 3.36%, 3.46%, 1.03% and 2.88% gains in mAcc for DeepLabv3+, PSPNet, OCRNet and SegFormer.
Revised: Specifically, for ResNet-50c, ResNet-101c, HRNet-W48 and MIT-B3 (Transformer-based), there are 3.36%, 3.46%, 1.03% and 2.88% gains in mAcc for DeepLabv3+, PSPNet, OCRNet and SegFormer.

Original: Especially, our method achieved more impressive results than the ordinary re-weighting method (Focal Loss).
Revised: Especially, our method achieved more impressive results than the ordinary re-weighting method (Focal Loss).

Original: This is because Focal Loss through the simple re-weighting method slightly improves the accuracy of the tail categories with a significant decrease in the performance of the head categories, which will cause an unacceptable decline in mIoU.
Revised: This is because Focal Loss through the simple re-weighting method slightly improves the accuracy of the tail categories with a significant decrease in the performance of the head categories, which will cause an unacceptable decline in mIoU.

Original: ADE20K.
Revised: ADE20K.

Original: ADE20K is a challenging benchmark due to its various image scales and plenty of semantic categories in reality, which cause the long-tailed distribution.
Revised: ADE20K is a challenging benchmark due to its various image scales and plenty of semantic categories in reality, which cause the long-tailed distribution.

Original: In this scenario, our method 
Revised: In this scenario, our method shows better performance than the current methods. The experimental results are summarized in

Original: [Table] [Table]
Revised: [Table] [Table]

Original: SegFormer
Revised: SegFormer

Original: [Table]
Revised: [Table]

Original: Xie et al.
Revised: Xie et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: SegFormer
Revised: SegFormer

Original: [Table]
Revised: [Table]

Original: shows better performance than the current methods.

Original: The experimental results are summarized in

Original: Table 3 and 4, our method achieved 1.71%, 1.30%, 0.78% and 1.09% gains in mIoU and 5.89%, 2.85%, 4.05% and 3.22% gains in mAcc for DeepLabv3+, PSPNet, OCRNet and SegFormer.
Revised: Table 3 and 4, our method achieved 1.71%, 1.30%, 0.78% and 1.09% gains in mIoU and 5.89%, 2.85%, 4.05% and 3.22% gains in mAcc for DeepLabv3+, PSPNet, OCRNet and SegFormer.

Original: Furthermore, we illustrated the qualitative results in Figure 4 on the validation set of ADE20K.
Revised: Furthermore, we illustrated the qualitative results in Figure 4 on the validation set of ADE20K.

Original: We can see our method achieved better segmentation results than baseline on these tail categories in different scenarios, such as pillow and lamp, which shows the consistent effectiveness of our method.
Revised: We can see our method achieved better segmentation results than baseline on these tail categories in different scenarios, such as pillow and lamp, which shows the consistent effectiveness of our method.

Original: Results in diverse category data subsets.
Revised: Results in diverse category data subsets.

Original: As illustrated in Table 4, our method achieved impressive mAcc gains for body and tail data subsets on Cityscapes and ADE20K.
Revised: As illustrated in Table 4, our method achieved impressive mAcc gains for body and tail data subsets on Cityscapes and ADE20K.

Original: Specifically, we achieved overall improvement on ADE20K. For further, Category-wise performance improved comparisons between MEDOE and baseline is shown in Figure 3.
Revised: Specifically, we achieved overall improvement on ADE20K. For further, Category-wise performance improved comparisons between MEDOE and baseline is shown in Figure 3.

Original: MEDOE achieved both IoU and Acc improvements on Cityscapes and ADE20K benchmark datasets in almost all categories.
Revised: MEDOE achieved both IoU and Acc improvements on Cityscapes and ADE20K benchmark datasets in almost all categories.

Original: Especially, our method has significant advantages for body and tail categories data subsets, which verifies our motivation that MED can extract effective contextual information to help pixel classify.
Revised: Especially, our method has significant advantages for body and tail categories data subsets, which verifies our motivation that MED can extract effective contextual information to help pixel classify.

Original: Oracle results.
Revised: Oracle results.

Original: We investigated the ideal scenario to provide the exact experts’ selecting decisions in the inference phase substitute the MOE, such that W = W ∗ .
Revised: We investigated the ideal scenario to provide the exact experts’ selecting decisions in the inference phase substitute the MOE, such that W = W ∗ .

Original: Reported results in this ideal scenario, referred to as Oracle in Table 2 and 5, shown impressive improvements in mIoU and mAcc over both 
Revised: Reported results in this ideal scenario, referred to as Oracle in Table 2 and 5, shown impressive improvements in mIoU and mAcc over both current methods, with a consistent gain across all datasets. Specifically, our method gains 3%-4%

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: current methods, with a consistent gain across all datasets.

Original: Specifically, our method gains 3%-4% in mIoU and 5%-7% in mAcc on Cityscapes and particularly the improvements raise to 11%-13% in mIoU and 19%-22% in mAcc on ADE20K.
Revised: in mIoU and 5%-7% in mAcc on Cityscapes and particularly the improvements raise to 11%-13% in mIoU and 19%-22% in mAcc on ADE20K.

Original: We believe these impressive improvements convey several important messages.
Revised: We believe these impressive improvements convey several important messages.

Original: 1) It proves that our components in Stage 1 have trained a series of robust and effective experts which focused on their dominant categories.
Revised: 1) It proves that our components in Stage 1 have trained a series of robust and effective experts which focused on their dominant categories.

Original: Compared to the results of our method, they indicate that there exists a strategy building on top of MED that can largely outperform state-of-the-art methods.
Revised: Compared to the results of our method, they indicate that there exists a strategy building on top of MED that can largely outperform state-of-the-art methods.

Original: This suggests that MED has significant prospects, more efforts could be directed towards to constrain the expert decision and opening a door to promising avenues [Figure]
Revised: This suggests that MED has significant prospects, more efforts could be directed towards to constrain the expert decision and opening a door to promising avenues.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Ablation of MED.
Revised: Ablation of MED.

Original: To illustrate that our MED self-adaptively extracts different contextual information guided by diverse masked datasets, indeed, rather than attributing entirely to the modifying classifiers’ boundaries operation or simply model-ensemble method.
Revised: To illustrate that our MED self-adaptively extracts different contextual information guided by diverse masked datasets, indeed, rather than attributing entirely to the modifying classifiers’ boundaries operation or simply increasing extra parameters ( i

Original:  We designed M ulti- C lassifier network ( MC ) with a shared backbone and context module and model-ensemble method for a comparison experiment in the whole datasets on Cityscapes, the results are shown in Table 5.
Revised: [Figure] e. model-ensemble method). We designed M ulti- C lassifier network ( MC ) with a shared backbone and context module and modelensemble method for a comparison experiment in the whole datasets on Cityscapes, the results are shown in Table 5.

Original: Just modifying the classifier boundaries is not effective in the tail categories, and even reduces the overall performance.
Revised: Just modifying the classifier boundaries is not effective in the tail categories, and even reduces the overall performance.

Original: The model-ensemble method slightly improves the head categories segmentation by increasing a large number of parameters (multi-backbones).
Revised: Since the task is really difficult, the model-ensemble method simply enhancing extra parameters (multi-backbones) can not make overall improvements.

Original: Meanwhile, it improves overall performance but ignores the tail categories.
Revised: Without our training strategies, it improves head performance but ignores the tail categories.

Original: Thus MED helps the network better classifythe pixels by different contextual information.
Revised: Thus MED helps the network better classify pixels by different contextual information.

Original: Ablation of MOE.
Revised: Ablation of MOE.

Original: As we mentioned, we introduced various user-specified output ensemble methods.
Revised: As we mentioned, we introduced various user-specified output ensemble methods.

Original: For each pixel instance x , (2) Softmax method get the output
Revised: For each pixel instance x , (2) Softmax method get the output

Original: z i of expert E i from 1 to k-th[Table] and pixel category c i ==
Revised: z i of expert E i from 1 to k-th and pixel category c i ==

Original: j if (softmax( z i )) >
Revised: j if (softmax( z i )) >

Original: β , β is set as 0.3, (3) Argmax method determine  the outputs according to the confidence of the maximum softmax on each expert, (4) Group average method ensemble z i of E i by grouping average weight.
Revised: β , β is set as 0.3, (3) Argmax method determine [Table] the outputs according to the confidence of the maximum softmax on each expert, (4) Group average method ensemble z i of E i by grouping average weight.

Original: Comparisons between MOE with the above-mentioned methods in Table 6.
Revised: Comparisons between MOE with the above-mentioned methods in Table 6.

Original: Our MOE achieved improvements overall. 
Revised: Our MOE achieved improvements overall. Methods

Original: [Table]
Revised: [Table]

Original: +Ensemble
Revised: Ensemble

Original: [Table]
Revised: [Table]

Original: +MEDOE
Revised: +

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: Aggregation

Original: [Table]
Revised: [Table]

Original: Effectiveness of the auxiliary loss function.
Revised: Effectiveness of the auxiliary loss function.

Original: An ablation with or without an auxiliary loss function and the hyper-parameter α on ADE20K was conducted to prove the motivation that we introduce auxiliary loss function.
Revised: An ablation with or without an auxiliary loss function and the hyper-parameter α on ADE20K was conducted to prove the motivation that we introduce auxiliary loss function.

Original: The results in Figure 5 shown that in the case of using CE loss function alone (Ours with α =0.0), our method can achieve the improvement compared to baseline, but the improvement is limited.
Revised: The results in Figure 5 shown that in the case of using CE loss function alone (Ours with α =0.0), our method can achieve the improvement compared to baseline, but the improvement is limited.

Original: Meanwhile, when the hyper-parameter α =0.2 we gain the best performance.
Revised: Meanwhile, when the hyper-parameter α =0.2 we gain the best performance.

Original: Training without the auxiliary loss function constraint is more likely to be confused by the unavoidable IP and decrease the performance overall.
Revised: Training without the auxiliary loss function constraint is more likely to be confused by the unavoidable IP and decrease the performance overall.

Original: The introduction of auxiliary loss function obtains a more consistent and stable improvement.
Revised: The introduction of auxiliary loss function obtains a more consistent and stable improvement.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we investigate and identify the long-tailed distribution in semantic segmentation, and motivated by the fact that existing methods ignore this issue and the performance declines for tail categories.
Revised: In this paper, we investigate and identify the long-tailed distribution in semantic segmentation, and motivated by the fact that existing methods ignore this issue and the performance declines for tail categories.

Original: We proposed a MEDOE framework to overcome this challenging problem.
Revised: We proposed a MEDOE framework to overcome this challenging problem.

Original: The MED,
Revised: The MED,

Original: MOE module and a series of strategies ( e.g. , auxiliary loss function and priori distribution) are introduced to help each expert extract different contextual information and promote the performance in dominant categories.
Revised: MOE module and a series of strategies ( e.g. , auxiliary loss function and priori distribution) are introduced to help each expert extract different contextual information and promote the performance in dominant categories.

Original: Our method achieved great improvements across different current methods up to 2% gains in mIoU and 6% gains in mAcc on Cityscapes and ADE20K.
Revised: Our method achieved great improvements across different current methods up to 2% gains in mIoU and 6% gains in mAcc on Cityscapes and ADE20K.

Original: Furthermore, training a robust output ensemble decision-maker appears to be a very promising way of constraining the inference, as demonstrated with the significantly improved results obtained by the oracle.
Revised: Furthermore, training a robust output ensemble decision-maker appears to be a very promising way of constraining the inference, as demonstrated with the significantly improved results obtained by the oracle.

Original: A A PPENDIX A. B ENCHMARKS & I MPLEMENTATION D ETAILS
Revised: A A PPENDIX A. B ENCHMARKS & I MPLEMENTATION D ETAILS

Original: Benchmarks: Cityscapes (Cordts et al., 2016) is a dataset that focuses on semantic understanding of urban street scenes and contains 19 semantic classes.
Revised: Benchmarks: Cityscapes (Cordts et al., 2016) is a dataset that focuses on semantic understanding of urban street scenes and contains 19 semantic classes.

Original: It contains 5K annotated images with pixellevel fine annotations and 20K coarsely annotated images.
Revised: It contains 5K annotated images with pixellevel fine annotations and 20K coarsely annotated images.

Original: The finely annotated 5K images are split into sets with numbers 2975, 500 and 1525 for training, validation and testing.
Revised: The finely annotated 5K images are split into sets with numbers 2975, 500 and 1525 for training, validation and testing.

Original: ADE20K
Revised: ADE20K

Original: (Zhou et al., 2017) is a challenging scene parsing dataset.
Revised: (Zhou et al., 2017) is a challenging scene parsing dataset.

Original: It contains 150 categories and diverse scenes with 1038 image-level labels.
Revised: It contains 150 categories and diverse scenes with 1038 image-level labels.

Original: The training and validation sets consist of 20K and 2K images, respectively.
Revised: The training and validation sets consist of 20K and 2K images, respectively.

Original: Implementation Details.
Revised: Implementation Details.

Original: We initialize backbones with the weights pre-trained on ImageNet.
Revised: We initialize backbones with the weights pre-trained on ImageNet.

Original: For
Revised: For

Original: DeepLabv3+, we using CNN-based ResNet-50c and ResNet-101c as backbones, which switch the first 7 × 7 convolution layer to three 3 × 3 convolutions.
Revised: DeepLabv3+, we using CNN-based ResNet-50c and ResNet-101c as backbones, which switch the first 7 × 7 convolution layer to three 3 × 3 convolutions.

Original: HRNet-W48 (Wang et al., 2020a) is adopted for OCRNet.
Revised: HRNet-W48 (Wang et al., 2020a) is adopted for OCRNet.

Original: MIT-B3 as a Transformer-based backbone is adopted for SegFormer (Xie et al., 2021), both of which are popular in semantic segmentation.
Revised: MIT-B3 as a Transformer-based backbone is adopted for SegFormer (Xie et al., 2021), both of which are popular in semantic segmentation.

Original: For CNN-based network, we use SDG and poly learning rate schedule (Zhao et al., 2017) with factor (cid:16) 1 −
Revised: For CNN-based network, we use SDG and poly learning rate schedule (Zhao et al., 2017) with factor (cid:16) 1 −

Original: [Table]
Revised: [Table]

Original: The initial learning rate is set as 0.01 and weight decay is 0.0005.
Revised: The initial learning rate is set as 0.01 and weight decay is 0.0005.

Original: We adopt AdamW with 6 × 10 − 5 learning rate and 0.01 weight decay.
Revised: We adopt AdamW with 6 × 10 − 5 learning rate and 0.01 weight decay.

Original: We set the image crop size to 512 × 1024, batch size as 8 and training iterations as 80K on Cityscapes by default.
Revised: We set the image crop size to 512 × 1024, batch size as 8 and training iterations as 80K on Cityscapes by default.

Original: For ADE20K, the crop size of images is set as 512 × 512, the batch size is set as 16 and training iterations are set as 80K if not stated otherwise.
Revised: For ADE20K, the crop size of images is set as 512 × 512, the batch size is set as 16 and training iterations are set as 80K if not stated otherwise.

Original: In the training phase, we augment data samples with the standard random scale in the range of [0.5, 2.0], random horizontal flipping, random cropping, as well as random color jittering.
Revised: In the training phase, we augment data samples with the standard random scale in the range of [0.5, 2.0], random horizontal flipping, random cropping, as well as random color jittering.

Original: For inference, the input image size of ADE20K is the same as the size during training, but for Cityscapes the input image is scaled to 1024 × 2048, no tricks( e.g. multi-scale with flipping) will be adopted during testing.
Revised: For inference, the input image size of ADE20K is the same as the size during training, but for Cityscapes the input image is scaled to 1024 × 2048, no tricks( e.g. multi-scale with flipping) will be adopted during testing.

Original: All experiments are implemented on the Nvidia A6000.
Revised: All experiments are implemented on the Nvidia A6000.

Original: A. A DDITIONAL PROOF OF THEORETICAL ANALYSIS BETWEEN M I O U AND M A CC
Revised: A. A DDITIONAL PROOF OF THEORETICAL ANALYSIS BETWEEN M I O U AND M A CC

Original: Before we proceed with the additional proof, we introduce some formulas for mAcc and mIoU.
Revised: Before we proceed with the additional proof, we introduce some formulas for mAcc and mIoU.

Original: [Equation]
Revised: [Equation]

Original: A.2. A DDITIONAL PROOF OF M A CC IS THE BETTER TAIL - SENSITIVE METRIC
Revised: A.2. A DDITIONAL PROOF OF M A CC IS THE BETTER TAIL - SENSITIVE METRIC

Original: According to the prior information from experimental statistics, the proportions of pixel instances of the head, body, and tail categories in the semantic segmentation are: 80%, 15% and 5%.
Revised: According to the prior information from experimental statistics, the proportions of pixel instances of the head, body, and tail categories in the semantic segmentation are: 80%, 15% and 5%.

Original: We analyzed the items in the mIoU and mAcc formulas and found that the difference between them was the FP i .
Revised: We analyzed the items in the mIoU and mAcc formulas and found that the difference between them was the FP i .

Original: According to Eq.
Revised: According to Eq.

Original: 12 We derive
Revised: 12 We derive

Original: [Equation]
Revised: [Equation]

Original: where, h , b and t refer to head, body, tail categoris and NUM refers to the pixel instance number of overall datasets.
Revised: where, h , b and t refer to head, body, tail categoris and NUM refers to the pixel instance number of overall datasets.

Original: From Eq.
Revised: From Eq.

Original: 14, we obtain the conclusion that due to the large instance base of the head categories, FN i and FN -related
Revised: 14, we obtain the conclusion that due to the large instance base of the head categories, FN i and FN -related

Original: FP i items are dominated by head categories.
Revised: FP i items are dominated by head categories.

Original: Therefore, the IoU i of each category i and mIoU will be dominated by head categories because of the item FP i .
Revised: Therefore, the IoU i of each category i and mIoU will be dominated by head categories because of the item FP i .

Original: Instead of mIoU , for each category i of mAcc , the items of Acc i are only related to its own category i , and mAcc will not be dominated by the head categories.
Revised: Instead of mIoU , for each category i of mAcc , the items of Acc i are only related to its own category i , and mAcc will not be dominated by the head categories.

Original: Thus mAcc is a fair and tail-sensitive metric.
Revised: Thus mAcc is a fair and tail-sensitive metric.

Original: A.2. A DDITIONAL PROOF OF R EMARK
Revised: A.2. A DDITIONAL PROOF OF R EMARK

Original: To better understand the correlation between mean IoU and mean Acc in long-tailed semantic segmentation and Remark 1 .
Revised: To better understand the correlation between mean IoU and mean Acc in long-tailed semantic segmentation and Remark 1 .

Original: According to the equation Eq.
Revised: According to the equation Eq.

Original: 10 and the precondition from the experiments.
Revised: 10 and the precondition from the experiments.

Original: The category Acc and IoU in our method become :
Revised: The category Acc and IoU in our method become :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: And it is clear from Eq.13,
Revised: And it is clear from Eq.13,

Original: We obtain the result:
Revised: We obtain the result:

Original: [Equation]
Revised: [Equation]

Original: where ∆ FP i is the increased false positive from baseline to our method.
Revised: where ∆ FP i is the increased false positive from baseline to our method.

Original: To guarantee the classifier effective and segment more tail categories, it should satisfied:
Revised: To guarantee the classifier effective and segment more tail categories, it should satisfied:

Original: [Equation]
Revised: [Equation]

Original: We observe that the results in Cityscapes benchmark has achieved a high value, e.g. IoU i = 0.8 and
Revised: We observe that the results in Cityscapes benchmark has achieved a high value, e.g. IoU i = 0.8 and

Original: Acc i = 0.85.
Revised: Acc i = 0.85.

Original: According to this precondition and Eq. 18, FP i = 0.

Revised: According to this precondition and Eq. 18, FP i = 0.

Original: × num i and p are both small value, which means ∆ FP i is a minimum value and satisfied Eq.
Revised: × n u m i and p are both small value, which means ∆ FP i is a minimum value and satisfied Eq.

Original: This is just the reason that Acc improved while IoU not decrease is significant for tail categories segmentation.
Revised: This is just the reason that Acc improved while IoU not decrease is significant for tail categories segmentation.

Original: A. A DDITIONAL PROOF OF THE CORRELATION BETWEEN PERFORMANCE AND DATA DISTRIBUTION
Revised: A. A DDITIONAL PROOF OF THE CORRELATION BETWEEN PERFORMANCE AND DATA DISTRIBUTION

Original: As shown of Figure 6, the baseline method on Cityscapes and ADE20K perform not well on certain categories, and we can clearly see that these categories mainly fall in the tail and body data subsets.
Revised: As shown of Figure 6, the baseline method on Cityscapes and ADE20K perform not well on certain categories, and we can clearly see that these categories mainly fall in the tail and body data subsets.

Original: This further demonstrates that the long-tailed data distribution limits the overall performance of the baseline method by constraining the accuracy of certain categories.
Revised: This further demonstrates that the long-tailed data distribution limits the overall performance of the baseline method by constraining the accuracy of certain categories.

Original: [Figure]
Revised: [Figure]

Original: A. A BLATION WITH R E - BALANCE METHODS
Revised: A. A BLATION WITH L ONG - TAILED METHODS

Original: [Table]
Revised: [Table]

Original: In this section we first calculate the Pearson correlation coefficient to show the correlation between category accuracy and category frequency (image category frequency on CIFAR-100) in Table 7.
Revised: In this section we first calculate the Pearson correlation coefficient to show the correlation between category accuracy and category frequency (image category frequency on CIFAR-100) in Table 7.

Original: The weak correlation between category accuracy and pixel level frequency on Cityscapes and
Revised: The weak correlation between category accuracy and pixel level frequency on Cityscapes and

Original: ADE20K causes challenge to re-weighting in semantic segmentation, which can be demonstrated in Table 8.
Revised: ADE20K causes challenge to re-weighting in semantic segmentation, which can be demonstrated in Table 8.

Revised: We compared our work with traditional long-tailed classification methods to further explore the contribution of our work.

Original: We adopt  re-weighting method to modify  loss function and put larger weights on tail categories.
Revised: We adopt the re-weighting method to modify the loss function and put larger weights on tail categories.

Original: Despite improving mAcc, the re-weighting method caused a lot of decrease in mIoU. For re-sampling, the contextual information is corrupted resulting in segmentation metrics at a very low level.
Revised: Despite improving mAcc, the re-weighting method caused a lot of decrease in mIoU. For re-sampling, the contextual information is corrupted resulting in segmentation metrics at a very low level.

Original: In general, the current Pixel level re-balance approaches can not work well with the long tail distribution on semantic segmentationre-sampling, all networks adopt
Revised: In general, the current Pixel level re-balance approaches can not work well with the long tail distribution on semantic segmentation

Original: ResNet-50c backbone. - denotes that the metric is in a very low level. Re-weighting with † refers to Focal Loss method.
Revised: [Table]

Original: [Table]
Revised: [Figure]

Original: A. A NALYSIS OF THE PERFORMANCE GAP BETWEEN C ITYSCAPES AND ADE20K
Revised: A. A NALYSIS OF THE PERFORMANCE GAP BETWEEN C ITYSCAPES AND ADE20K

Original: According to the experiments’ results of Sec4, our method has achieved impressive performance in both mIoU and mAcc on ADE20K dataset.
Revised: According to the experiments’ results of Sec4, our method has achieved impressive performance in both mIoU and mAcc on ADE20K dataset.

Original: However, there seems to be a gap between the performance on Cityscapes and ADE20K.
Revised: However, there seems to be a gap between the performance on Cityscapes and ADE20K.

Original: We analyze the causes: 1) Compared to ADE20K, there are fewer categories in Cityscapes and a more pronounced long-tail distribution (higher proportion of[Figure] head category instances), so it is easier to fall into local optimum when training middle and tail experts, resulting in the overfitting of these categories.
Revised: We analyze the causes: 1) Compared to ADE20K, there are fewer categories in Cityscapes and a more pronounced long-tail distribution (higher proportion of head category instances), so it is easier to fall into local optimum when training body and tail experts, resulting in the overfitting of these categories.

Original: Finally, it caused that the increase of FN ht
Revised: Finally, it caused that the increase of FN ht

Original: and FP ht on the overall datasets.
Revised: and FP ht on the overall datasets.

Original: FN ij and FP ij denote to the pixel instance which the ground truth belongs to category i
Revised: FN ij and FP ij denote to the pixel instance which the ground truth belongs to category i

Original: but the prediction is j . 2)
Revised: but the prediction is j . 2)

Original: The performance of baseline methods on Cityscapes is at a high level.
Revised: The performance of baseline methods on Cityscapes is at a high level.

Original: According to the above two reasons, our method improves the overall TP on Cityscapes, but it will increase FN ht of the head categories and FP ht of the tail categories, respectively, resulting in mIoU being at a relatively stable value.
Revised: According to the above two reasons, our method improves the overall TP on Cityscapes, but it will increase FN ht of the head categories and FP ht of the tail categories, respectively, resulting in mIoU being at a relatively stable value.

Original: From the macro level of image visualization, as shown in Figure 7, our method segments the surrounding part of the head categories into tail categories while segmenting the tail categories.
Revised: From the macro level of image visualization, as shown in Figure 7, our method segments the surrounding part of the head categories into tail categories while segmenting the tail categories.

Original: We believe that such head accuracy decreases are acceptable and meaningful in real-world scenarios.
Revised: We believe that such head accuracy decreases are acceptable and meaningful in real-world scenarios.

Original: A. Q UANTITATIVE V ISUALIZATION C OMPARISONS ON C ITYSCAPES AND ADE20K
Revised: A. Q UANTITATIVE V ISUALIZATION C OMPARISONS ON C ITYSCAPES AND ADE20K

Original: In this section, we demonstrate the better performance of MEDOE framework with quantitative visualization on Cityscapes and ADE20K shown in Figure 7 and 8.
Revised: In this section, we demonstrate the better performance of MEDOE framework with quantitative visualization on Cityscapes and ADE20K shown in Figure 7 and 8.

Original: We adopt ResNet-50c as backbone and all models trained under the same setting.
Revised: We adopt ResNet-50c as backbone and all models trained under the same setting.

Original: In most semantic scenarios, our MEDOE method can achieve better performance in segmenting tail categories.
Revised: In most semantic scenarios, our MEDOE method can achieve better performance in segmenting tail categories.

Revised: A. A DDITIONAL EXPLANATIONS A.7. C ONTEXTUAL MODULE

Revised: Contextual module , as an important module in semantic segmentation, refers to the extraction and aggregation of contextual information for pixels through a series of operations ( i.e. , feature pyramids, atrous convolution, large-scale convolutional, attention mechanisms, and global pooling).

Revised: Existing contextual modules include: ASPP (Chen et al., 2018), PSP (Zhao et al., 2017), or nonlocal (Wang et al., 2017).

Revised: Contextual information means the relationship between this pixel and the surrounding pixels and global information is regarded as contextual information.

Original:  [Figure] 
Revised: The reasons why segmentation needs contextual module. When solving semantic segmentation tasks if each pixel considers only its deep features, such as texture and color, it will be difficult to classify into the correct category ( i.e. the [Figure] deep features of leaves and green grass will be very similar).

Revised: At the same time, as stated in the work, semantic segmentation believes that each pixel is not I.I.D., but is related to the surrounding pixels.

Revised: The correlation information can better help the segmentation task ( i.e. the leaves are surrounded by branches, and the grass is likely to be surrounded by roads).

Revised: A. A BLATION OF DIFFERENT EXPERTS

Revised: To measure the goodness and distinctiveness, We adopt the mAcc and per-category bias in Table 9 as the metrics to describe each expert.

Revised: It seems to demonstrate that the K experts learned at Stage 1 are good and distinctive from each other, rather than simply boosting confidence through multi-expert training.

Revised: Furthermore, according to the well-known bias-variance decomposition (Wang et al., 2020b), percategory bias denotes:

Revised: [Equation]

Revised: [Table]

Revised: A. L ONG - TAILED SEMANTIC SEGMENTATION

Revised: To the best of our knowledge, there is a contemporaneous (Cui et al., 2022) work with our paper.

Revised: Although, we almost simultaneously focus on the long-tailed distribution as an important reason for constraining semantic segmentation performance, there are some differences between our work and Region rebalance (Cui et al., 2022):

Revised: Perspective difference: Region rebalance was concerned about solving the problem of category rebalance, while our work was more focused on improving the recognition of tail and body categories and placed special emphasis on the significance of segmenting the body, and tail categories.

Revised: Methods difference: Region rebalance relieved the categories imbalance with an auxiliary region classification branch by adjusting segmentation boundaries however motivated by the ensembling and grouping methods, we proposed MEDOE framework to encourage different experts to learn more balanced distribution in the feature space, and finally adjust classification boundaries.

Revised: Compared to Region Rebalance, the motivation of our work was completely different and provided different research directions.

Revised: Interpretations of mIoU and mAcc difference: Region rebalance only explained the cause of mIoU with previous long-tailed methods.

Revised: Different from Region rebalance, we took a further step and explored the significance of mAcc in segmenting the body and tail categories.

Revised: A. S IGNIFICANCE OF BODY AND TAIL CATEGORIES IN REAL - WORLD SCENARIOS

Revised: We believe in a large number of real-world scenarios, it is more important to be able to identify body and tail categories than accurately segment the edge pixels of head categories.

Revised: ( i.e. In the automatic driving scenario, we need to segment some tail categories objects that appear on the driving path, such as “poles” or “fire hydrants”, to avoid traffic accidents.

Revised: Segmenting small lesions on medical images can help doctors detect underlying diseases).

Revised: Generally, the benefits of this segmentation are far greater than the decrease of certain head categories edges.

Revised: A. C OMPARISON WITH TRADITIONAL MULTI - EXPERTS METHODS

Revised: We compared the differences from existing multi-expert methods, there is three main difference:

Revised: Model architecture:

Revised: The pipeline of traditional multi-expert methods contain a backbone and multi-classifiers to adjust classifier boundaries and finally ensemble the outputs.

Revised: 1)We pioneered the combination of contextual modules and classifiers to become experts and learn more balanced distribution in the feature space, and finally adjust classification boundaries.

Revised: 2)Then we provided each expert with soft weight based on the final contextual information and classification results through a learning mechanism to ensemble the outputs.

Revised: Training strategies: Traditional methods often focus on constraining dominating categories and ignoring the confusing categories.

Revised: Differing from them, we proposed the expert-specific pixel-masking strategy and diverse data distributionaware loss function to ensure our model architecture focuses on the confusing categories and has better performance.

Revised: Training step: Advanced multi-expert methods in long-tailed classification, such as BBN (Zhou et al., 2020), RIDE (Wang et al., 2020b), LFME (Xiang et al., 2020).

Revised: They all take multiple steps to train, mainly including 1. training backbone, 2.

Revised: training classifiers, and 3.

Revised: Distillation learning (optional).

Revised: However, our method can update the backbone parameters while training the head expert, so it is one-step training.