Original: E FFICIENTLY T ROUBLESHOOTING I MAGE S EGMENTATION M ODELS WITH H UMAN -I N -T HE -L OOP
Revised: E FFICIENTLY T ROUBLESHOOTING I MAGE S EGMENTATION M ODELS WITH H UMAN -I N -T HE -L OOP

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Image segmentation lays the foundation for many high-stakes vision applications such as autonomous driving and medical image analysis.
Revised: Image segmentation lays the foundation for many high-stakes vision applications such as autonomous driving and medical image analysis.

Original: It is, therefore, of great importance to not only improve the accuracy of segmentation models on wellestablished benchmarks, but also enhance their robustness in the real world so as to avoid sparse but fatal failures.
Revised: It is, therefore, of great importance to not only improve the accuracy of segmentation models on wellestablished benchmarks, but also enhance their robustness in the real world so as to avoid sparse but fatal failures.

Original: In this paper, instead of chasing state-of-the-art performance on existing benchmarks, we turn our attention to a new challenging problem: how to efﬁciently expose failures of “top-performing” segmentation models in the real world and how to leverage such counterexamples to rectify the models.
Revised: In this paper, instead of chasing state-of-the-art performance on existing benchmarks, we turn our attention to a new challenging problem: how to efﬁciently expose failures of “top-performing” segmentation models in the real world and how to leverage such counterexamples to rectify the models.

Original: To achieve this with minimal human labelling effort, we ﬁrst automatically sample a small set of images that are likely to falsify the target model from a large corpus of web images via the maximum discrepancy competition principle.
Revised: To achieve this with minimal human labelling effort, we ﬁrst automatically sample a small set of images that are likely to falsify the target model from a large corpus of web images via the maximum discrepancy competition principle.

Original: We then propose a weakly labelling strategy to further reduce the number of false positives, before time-consuming pixel-level labelling by humans.
Revised: We then propose a weakly labelling strategy to further reduce the number of false positives, before time-consuming pixel-level labelling by humans.

Original: Finally, we ﬁne-tune the model to harness the identiﬁed failures, and repeat the whole process, resulting in an efﬁcient and progressive framework for troubleshooting segmentation models.
Revised: Finally, we ﬁne-tune the model to harness the identiﬁed failures, and repeat the whole process, resulting in an efﬁcient and progressive framework for troubleshooting segmentation models.

Original: We demonstrate the feasibility of our framework using the semantic segmentation task in PASCAL VOC, and ﬁnd that the ﬁne-tuned model exhibits signiﬁcantly improved generalization when applied to real-world images with greater content diversity.
Revised: We demonstrate the feasibility of our framework using the semantic segmentation task in PASCAL VOC, and ﬁnd that the ﬁne-tuned model exhibits signiﬁcantly improved generalization when applied to real-world images with greater content diversity.

Original: All experimental codes will be publicly released upon acceptance.
Revised: All experimental codes will be publicly released upon acceptance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image segmentation (i.e., pixel-level image labelling) has recently risen to explosive popularity, due in part to its profound impact on many high-stakes vision applications, such as autonomous driving and medical image analysis.
Revised: Image segmentation (i.e., pixel-level image labelling) has recently risen to explosive popularity, due in part to its profound impact on many high-stakes vision applications, such as autonomous driving and medical image analysis.

Original: While the performance of segmentation models, as measured by excessively reused test sets (Everingham et al., 2010; Lin et al., 2014), keeps boosting (Chen et al., 2018a; Badrinarayanan et al., 2017; Yu et al., 2018), two scientiﬁc questions have arisen to capture the community’s curiosity, and motivate the current work:
Revised: While the performance of segmentation models, as measured by excessively reused test sets (Everingham et al., 2010; Lin et al., 2014), keeps boosting (Chen et al., 2018a; Badrinarayanan et al., 2017; Yu et al., 2018), two scientiﬁc questions have arisen to capture the community’s curiosity, and motivate the current work:

Original: Q1:
Revised: Q1:

Original: Do “top-performing” segmentation models on existing benchmarks generalize to the real world with much richer variations?
Revised: Do “top-performing” segmentation models on existing benchmarks generalize to the real world with much richer variations?

Original: Q2:
Revised: Q2:

Original: Can we identify and rectify the trained models’ sparse but fatal mistakes, without incurring signiﬁcant workload of human labelling?
Revised: Can we identify and rectify the trained models’ sparse but fatal mistakes, without incurring signiﬁcant workload of human labelling?

Original: The answer to the ﬁrst question is conceptually clearer, by taking reference to a series of recent work on image classiﬁcation (Recht et al., 2019;
Revised: The answer to the ﬁrst question is conceptually clearer, by taking reference to a series of recent work on image classiﬁcation (Recht et al., 2019;

Original: Hendrycks et al., 2019).
Revised: Hendrycks et al., 2019).

Original: A typical test set for image classiﬁcation can only include a maximum of ten thousands of images because human labelling (or veriﬁcation of predicted labels) is expensive and time-consuming.
Revised: A typical test set for image classiﬁcation can only include a maximum of ten thousands of images because human labelling (or veriﬁcation of predicted labels) is expensive and time-consuming.

Original: Considering the high dimensionality of image space and the “human-level” performance of existing methods, such test sets may only spot an extremely small subset of possible mistakes that the model will make, suggesting their insufﬁciency to cover hard examples that may be encountered in the real world (Wang et al., 2020).
Revised: Considering the high dimensionality of image space and the “human-level” performance of existing methods, such test sets may only spot an extremely small subset of possible mistakes that the model will make, suggesting their insufﬁciency to cover hard examples that may be encountered in the real world (Wang et al., 2020).

Original: The existence of natural adversarial examples (Hendrycks et al., 2019) also echos such hidden fragility of the classiﬁers to unseen examples, despite the impressive accuracy on existing benchmarks.
Revised: The existence of natural adversarial examples (Hendrycks et al., 2019) also echos such hidden fragility of the classiﬁers to unseen examples, despite the impressive accuracy on existing benchmarks.

Original: While the above problem has not been studied in the context of image segmentation, we argue that it would only be much ampliﬁed for two main reasons.
Revised: While the above problem has not been studied in the context of image segmentation, we argue that it would only be much ampliﬁed for two main reasons.

Original: First, segmentation benchmarks require pixel-level dense annotation.
Revised: First, segmentation benchmarks require pixel-level dense annotation.

Original: Compared to classiﬁcation databases, they are much more expensive, laborious, and error-prone to label 1 , making existing segmentation datasets even more restricted in scale.
Revised: Compared to classiﬁcation databases, they are much more expensive, laborious, and error-prone to label 1 , making existing segmentation datasets even more restricted in scale.

Original: Second, it is much harder for segmentation data to be class-balanced in the pixel level, making highly skewed class distributions notoriously common for this particular task (Kervadec et al., 2019; Bischke et al., 2018).
Revised: Second, it is much harder for segmentation data to be class-balanced in the pixel level, making highly skewed class distributions notoriously common for this particular task (Kervadec et al., 2019; Bischke et al., 2018).

Original: Besides, the “universal” background class (often set to cover the distracting or uninteresting classes (Everingham et al., 2010)) adds additional complicacy to image segmentation
Revised: Besides, the “universal” background class (often set to cover the distracting or uninteresting classes (Everingham et al., 2010)) adds additional complicacy to image segmentation

Original: (Mostajabi et al., 2015).
Revised: (Mostajabi et al., 2015).

Original: Thus, it remains questionable to what extent the impressive performance on existing benchmarks can be interpreted as (or translated into) real-world robustness.
Revised: Thus, it remains questionable to what extent the impressive performance on existing benchmarks can be interpreted as (or translated into) real-world robustness.

Original: If “top-performing” segmentation models make sparse yet catastrophic mistakes that have not be spotted beforehand, they will fall short of the need by high-stakes applications.
Revised: If “top-performing” segmentation models make sparse yet catastrophic mistakes that have not be spotted beforehand, they will fall short of the need by high-stakes applications.

Original: The answer to the second question constitutes the main body of our technical work.
Revised: The answer to the second question constitutes the main body of our technical work.

Original: In order to identify sparse failures of existing segmentation models, it is necessary to expose them to a much larger corpus of real-world labelled images (on the order millions or even billions).
Revised: In order to identify sparse failures of existing segmentation models, it is necessary to expose them to a much larger corpus of real-world labelled images (on the order millions or even billions).

Original: This is, however, implausible due to the expensiveness of dense labelling in image segmentation.
Revised: This is, however, implausible due to the expensiveness of dense labelling in image segmentation.

Original: The core question essentially boils down to: how to efﬁciently decide what to label from the massive unlabelled images, such that a small number of annotated images maximally expose corner-case defects, and can be leveraged to improve the models.
Revised: The core question essentially boils down to: how to efﬁciently decide what to label from the massive unlabelled images, such that a small number of annotated images maximally expose corner-case defects, and can be leveraged to improve the models.

Original: [Figure]
Revised: [Figure]

Original: In this paper, we introduce a two-stage framework with human-in-the-loop for efﬁciently troubleshooting image segmentation models (see Figure 1).
Revised: In this paper, we introduce a two-stage framework with human-in-the-loop for efﬁciently troubleshooting image segmentation models (see Figure 1).

Original: The ﬁrst stage automatically mines, from a large pool D of unlabelled images, a small image set M , which are the most informative in exposing weaknesses of the target model.
Revised: The ﬁrst stage automatically mines, from a large pool D of unlabelled images, a small image set M , which are the most informative in exposing weaknesses of the target model.

Original: Speciﬁcally, inspired by previous studies on model falsiﬁcation as model comparison (Wang &
Revised: Speciﬁcally, inspired by previous studies on model falsiﬁcation as model comparison (Wang &

Original: Simoncelli, 2008; Ma et al., 2018; Wang et al., 2020), we let the target model compete with a set of state-of-the-art methods with different design methodologies, and sample images by MAximizing the
Revised: Simoncelli, 2008; Ma et al., 2018; Wang et al., 2020), we let the target model compete with a set of state-of-the-art methods with different design methodologies, and sample images by MAximizing the

Original: Discrepancy (MAD) between the methods.
Revised: Discrepancy (MAD) between the methods.

Original: To reduce the number of false positives, we propose a weakly labelling method of ﬁltering M to obtain a smaller reﬁned set S , subject to segmentation by human subjects.
Revised: To reduce the number of false positives, we propose a weakly labelling method of ﬁltering M to obtain a smaller reﬁned set S , subject to segmentation by human subjects.

Original: In the second stage, we ﬁne-tune the target model to learn from the counterexamples in S without forgetting previously seen data.
Revised: In the second stage, we ﬁne-tune the target model to learn from the counterexamples in S without forgetting previously seen data.

Original: The two stages may be iterated, enabling progressive troubleshooting of image segmentation models.
Revised: The two stages may be iterated, enabling progressive troubleshooting of image segmentation models.

Original: Experiments on PASCAL VOC (Everingham et al., 2010) demonstrate the feasibility of the proposed method to address this new challenging problem, where we successfully discover corner-case errors of a “top-performing” segmentation model (Chen et al., 2017), and ﬁx it for improved generalization in the wild.
Revised: Experiments on PASCAL VOC (Everingham et al., 2010) demonstrate the feasibility of the proposed method to address this new challenging problem, where we successfully discover corner-case errors of a “top-performing” segmentation model (Chen et al., 2017), and ﬁx it for improved generalization in the wild.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: MAD competition The proposed method takes inspiration from the MAD competition (Wang &
Revised: MAD competition The proposed method takes inspiration from the MAD competition (Wang &

Original: Simoncelli, 2008; Wang et al., 2020) to efﬁciently spot model failures.
Revised: Simoncelli, 2008; Wang et al., 2020) to efﬁciently spot model failures.

Original: Previous works focused on performance evaluation.
Revised: Previous works focused on performance evaluation.

Original: We take one step further to also ﬁx the model errors detected in the MAD competition.
Revised: We take one step further to also ﬁx the model errors detected in the MAD competition.

Original: To the best of our knowledge, our work is the ﬁrst to extend the MAD idea to image segmentation, where labeling efﬁciency is more desired since pixel-wise human annotation for image segmentation is much more time-consuming than image quality assessment (Wang & Simoncelli, 2008) and image classiﬁcation (Wang et al., 2020)
Revised: To the best of our knowledge, our work is the ﬁrst to extend the MAD idea to image segmentation, where labeling efﬁciency is more desired since pixel-wise human annotation for image segmentation is much more time-consuming than image quality assessment (Wang & Simoncelli, 2008) and image classiﬁcation (Wang et al., 2020)

Original: tasks previously explored.
Revised: tasks previously explored.

Original: Differential testing Our method is also loosely related to the cross-disciplinary ﬁeld of software/system testing, especially the differential testing technique (McKeeman, 1998).
Revised: Differential testing Our method is also loosely related to the cross-disciplinary ﬁeld of software/system testing, especially the differential testing technique (McKeeman, 1998).

Original: By providing the same tests to multiple software implementations, differential testing aims to ﬁnd bug-exposing test samples that lead to different results.
Revised: By providing the same tests to multiple software implementations, differential testing aims to ﬁnd bug-exposing test samples that lead to different results.

Original: Programmers can then dig into these test cases for potential bug ﬁxing.
Revised: Programmers can then dig into these test cases for potential bug ﬁxing.

Original: While debugging software is very different from troubleshooting machine learning algorithms, a handful of recent work explored this idea to ﬁnd pitfalls of deep learning-based autonomous driving systems (Pei et al., 2017; Tian et al., 2018; Zhang et al., 2018).
Revised: While debugging software is very different from troubleshooting machine learning algorithms, a handful of recent work explored this idea to ﬁnd pitfalls of deep learning-based autonomous driving systems (Pei et al., 2017; Tian et al., 2018; Zhang et al., 2018).

Original: Aiming to be fully automated without human intervention, these methods have to make strong assumptions such as the ground truth labels can be determined by majority vote 2 or are unchanged under some synthetic image transformations(e.g., brightness and contrast change, or style transfer).
Revised: Aiming to be fully automated without human intervention, these methods have to make strong assumptions such as the ground truth labels can be determined by majority vote 2 or are unchanged under some synthetic image transformations(e.g., brightness and contrast change, or style transfer).

Original: Therefore, it is unclear how to generalize the results obtained in such noisy and often unrealistic settings, to the real world with both great content ﬁdelity and diversity.
Revised: Therefore, it is unclear how to generalize the results obtained in such noisy and often unrealistic settings, to the real world with both great content ﬁdelity and diversity.

Original: Adversarial examples Introduced by Dalvi et al.
Revised: Adversarial examples Introduced by Dalvi et al.

Original: (Dalvi et al., 2004) and reignited by Szegedy et al.
Revised: (Dalvi et al., 2004) and reignited by Szegedy et al.

Original: (Szegedy et al., 2013), most adversarial attacks add small synthetic perturbations to inputs of computational models that cause them to make incorrect decisions.
Revised: (Szegedy et al., 2013), most adversarial attacks add small synthetic perturbations to inputs of computational models that cause them to make incorrect decisions.

Original: In image classiﬁcation, Hendrycks et al.
Revised: In image classiﬁcation, Hendrycks et al.

Original: (Hendrycks et al., 2019) identiﬁed a set of natural images that behave like synthetic adversarial examples, which possess inherent transferability to falsify different image classiﬁers with the same type of errors.
Revised: (Hendrycks et al., 2019) identiﬁed a set of natural images that behave like synthetic adversarial examples, which possess inherent transferability to falsify different image classiﬁers with the same type of errors.

Original: The selected counterexamples by the proposed framework might be treated as a new type of natural adversarial examples, that force the two models to make distinct predictions, therefore capable of fooling at least one model.
Revised: The selected counterexamples by the proposed framework might be treated as a new type of natural adversarial examples, that force the two models to make distinct predictions, therefore capable of fooling at least one model.

Original: Similar as natural counterexamples focused in this work, synthetic adversarial examples pose security risks of deploying machine learning algorithms in real-world applications.
Revised: Similar as natural counterexamples focused in this work, synthetic adversarial examples pose security risks of deploying machine learning algorithms in real-world applications.

Original: A large body of research (Madry et al., 2018; Zhang et al., 2019) delves deep into adversarial training, trying to defend against adversarial perturbations at the expensive cost of sacriﬁcing the generalization on original test sets without perturbations (Tsipras et al., 2019;
Revised: A large body of research (Madry et al., 2018; Zhang et al., 2019) delves deep into adversarial training, trying to defend against adversarial perturbations at the expensive cost of sacriﬁcing the generalization on original test sets without perturbations (Tsipras et al., 2019;

Original: Zhang et al., 2019; Schmidt et al., 2018).
Revised: Zhang et al., 2019; Schmidt et al., 2018).

Original: This seems to suggest a trade-off between generalization to real-world benign examples and robustness to adversarial attacks.
Revised: This seems to suggest a trade-off between generalization to real-world benign examples and robustness to adversarial attacks.

Original: Semantic segmentation with deep learning Fully convolutional network (Long et al., 2015) was among the ﬁrst deep architectures adopted for high-quality segmentation.
Revised: Semantic segmentation with deep learning Fully convolutional network (Long et al., 2015) was among the ﬁrst deep architectures adopted for high-quality segmentation.

Original: Skip connection (Ronneberger et al., 2015), recurrent module (Zheng et al., 2015), max index pooling (Noh et al., 2015;
Revised: Skip connection (Ronneberger et al., 2015), recurrent module (Zheng et al., 2015), max index pooling (Noh et al., 2015;

Original: Badrinarayanan et al., 2017), dilated convolution (Chen et al., 2014; Yu & Koltun, 2016;
Revised: Badrinarayanan et al., 2017), dilated convolution (Chen et al., 2014; Yu & Koltun, 2016;

Original: Chen et al., 2018a), and multi-scale training (Chen et al., 2016; 2018a) are typical strategies to boost the segmentation performance.
Revised: Chen et al., 2018a), and multi-scale training (Chen et al., 2016; 2018a) are typical strategies to boost the segmentation performance.

Original: Conditional random ﬁelds (Ladick ` y et al., 2010) used to dominate image segmentation before the advent of deep learning were also combined with convolutional networks to model spatial relationships (Zheng et al., 2015).
Revised: Conditional random ﬁelds (Ladick ` y et al., 2010) used to dominate image segmentation before the advent of deep learning were also combined with convolutional networks to model spatial relationships (Zheng et al., 2015).

Original: We refer interested readers to (Minaee et al., 2020) for a comprehensive survey of this ﬁeld.
Revised: We refer interested readers to (Minaee et al., 2020) for a comprehensive survey of this ﬁeld.

Original:  P ROPOSED METHOD
Revised:  P ROPOSED METHOD

Original: Suppose we have a target segmentation model f t : R h × w (cid:55)→ { 1 , · · · , c } h × w , where h and w are the height and width of the input image, and c denotes the number of categories.
Revised: Suppose we have a target segmentation model f t : R h × w (cid:55)→ { 1 , · · · , c } h × w , where h and w are the height and width of the input image, and c denotes the number of categories.

Original: Our goal is to efﬁciently identify and ﬁx the failure cases of f t encountered in the real world, while minimizing human labelling effort in this process.
Revised: Our goal is to efﬁciently identify and ﬁx the failure cases of f t encountered in the real world, while minimizing human labelling effort in this process.

Original: We start by constructing a large image database D , whose collection may be guided by the keywords that represent the c categories.
Revised: We start by constructing a large image database D , whose collection may be guided by the keywords that represent the c categories.

Original: Rather than conducting large-scale subjective testing to obtain the ground truth segmentation map for each x ∈ D , we choose to create a small subset of images M ⊂ D , which are strong candidates for revealing corner-case behaviors of f t .
Revised: Rather than conducting large-scale subjective testing to obtain the ground truth segmentation map for each x ∈ D , we choose to create a small subset of images M ⊂ D , which are strong candidates for revealing corner-case behaviors of f t .

Original: To further reduce false positive examples in M , we describe a method to gather a weak label for each x ∈ M as an overall indicator of segmentation quality.
Revised: To further reduce false positive examples in M , we describe a method to gather a weak label for each x ∈ M as an overall indicator of segmentation quality.

Original: Based on the labels, an even smaller set S ⊂ M can be obtained for dense annotation by humans.
Revised: Based on the labels, an even smaller set S ⊂ M can be obtained for dense annotation by humans.

Original: Last, we ﬁne-tune f t on the combination of S and previously trained data, in an attempt to learn from the found failures without forgetting (Li & Hoiem, 2017).
Revised: Last, we ﬁne-tune f t on the combination of S and previously trained data, in an attempt to learn from the found failures without forgetting (Li & Hoiem, 2017).

Original: 3. F AILURE IDENTIFICATION
Revised: 3. F AILURE IDENTIFICATION

Original: Constructing M Inspired by model falsiﬁcation methodologies from computational vision (Wang & Simoncelli, 2008) and software engineering (McKeeman, 1998), we construct the set M =
Revised: Constructing M Inspired by model falsiﬁcation methodologies from computational vision (Wang & Simoncelli, 2008) and software engineering (McKeeman, 1998), we construct the set M =

Original: { x i } n 2 i = by sampling the most “controversial” images from the large-scale unlabelled database D = { x i } n 1 i =1 , where n 2 (cid:28) n 1 .
Revised: { x i } n 2 i = by sampling the most “controversial” images from the large-scale unlabelled database D = { x i } n 1 i =1 , where n 2 (cid:28) n 1 .

Original: Speciﬁcally, given the target model f t , we let it compete with a group of state-ofthe-art segmentation models { g j } mj =1 by maximizing the discrepancy (Wang et al., 2020) between f t and g j on D :
Revised: Speciﬁcally, given the target model f t , we let it compete with a group of state-ofthe-art segmentation models { g j } mj =1 by maximizing the discrepancy (Wang et al., 2020) between f t and g j on D :

Original: [Equation]
Revised: [Equation]

Original: where d ( · ) is a distance metric to gauge the dissimilarity between two segmentation maps (e.g., negative pixel accuracy, or mean region intersection over union (mIoU)).
Revised: where d ( · ) is a distance metric to gauge the dissimilarity between two segmentation maps (e.g., negative pixel accuracy, or mean region intersection over union (mIoU)).

Original: ˆ x ( j ) represents the most controversial image according to f t and g j , and therefore is the most informative in distinguishing between them.
Revised: ˆ x ( j ) represents the most controversial image according to f t and g j , and therefore is the most informative in distinguishing between them.

Original: If the competing model g j performs at a high level, and differs from the target model f t in design, ˆ x ( j ) is likely to be a failure of f t .
Revised: If the competing model g j performs at a high level, and differs from the target model f t in design, ˆ x ( j ) is likely to be a failure of f t .

Original: [Figure]
Revised: [Figure]

Original: To avoid identifying different instantiations of the same underlying root cause (Pei et al., 2017) and to encourage content diversity of the candidate images, we describe a “contentaware” method for constructing M .
Revised: To avoid identifying different instantiations of the same underlying root cause (Pei et al., 2017) and to encourage content diversity of the candidate images, we describe a “contentaware” method for constructing M .

Original: We ﬁrst partition D into c overlapped subgroups {D t,k } ck =1 based on f t ’s predicted maps, where x ∈ D t,k if at least one pixel in f t ( x ) belongs to the k -th category.
Revised: We ﬁrst partition D into c overlapped subgroups {D t,k } ck =1 based on f t ’s predicted maps, where x ∈ D t,k if at least one pixel in f t ( x ) belongs to the k -th category.

Original: After that, we add a content constraint by restricting the size of predicted pixels in the k -th category, i.e., (cid:80) 

Revised: After that, we add a content constraint by restricting the size of predicted pixels in the k -th category, i.e., (cid:80)

Original: [ f t ( x ) =
Revised: [ f t ( x ) =

Original: = k ] / ( h × w ) , within the range of [ p k , q k ] .
Revised: = k ] / ( h × w ) , within the range of [ p k , q k ] .

Original: This allows excluding images of exceedingly large (or small) object sizes, which may be of less practical relevance.
Revised: This allows excluding images of exceedingly large (or small) object sizes, which may be of less practical relevance.

Original: Moreover, instead of focusing on the most controversial example deﬁned in Eq.
Revised: Moreover, instead of focusing on the most controversial example deﬁned in Eq.

Original: (1), we look at top- nimages in D t,k with n 3 largest distances computed by
Revised: (1), we look at top- nimages in D t,k with n 3 largest distances computed by

Original: [Equation]
Revised: [Equation]

Original: We then repeat this procedure, but with the roles of f t and g j reversed.
Revised: We then repeat this procedure, but with the roles of f t and g j reversed.

Original: That is, we partition D into c subgroups {D j,k } ck =1 according to g j , and solve the maximization problem over D j,k .
Revised: That is, we partition D into c subgroups {D j,k } ck =1 according to g j , and solve the maximization problem over D j,k .

Original: Finally, we gather all candidate images to arrive at the set M = { x i } n 2 i
Revised: Finally, we gather all candidate images to arrive at the set M = { x i } n 2 i

Original: =1 , where n 2 ≤ 2 mcn 3 (cid:28) n 1 4 .
Revised: =1 , where n 2 ≤ 2 mcn 3 (cid:28) n 1 4 .

Original: Constructing S
Revised: Constructing S

Original: Although images in the candidate set M have great potentials of being counterexamples of f t , some false positives may be included (see Figure 2), especially when g j is inferior to f t .
Revised: Although images in the candidate set M have great potentials of being counterexamples of f t , some false positives may be included (see Figure 2), especially when g j is inferior to f t .

Original: In addition, no data screening is involved in the construction of D , increasing chances of including images that are out-of-domain (e.g., falling out of the c categories and/or containing inappropriate content).
Revised: In addition, no data screening is involved in the construction of D , increasing chances of including images that are out-of-domain (e.g., falling out of the c categories and/or containing inappropriate content).

Original: In view of these, we reduce false positives in M via a weakly labelling strategy.
Revised: In view of these, we reduce false positives in M via a weakly labelling strategy.

Original: For each x ∈ M , we ask human subjects to give a discrete score on an absolute category rating (ARC) scale to indicate the segmentation quality of f t ( x ) .
Revised: For each x ∈ M , we ask human subjects to give a discrete score on an absolute category rating (ARC) scale to indicate the segmentation quality of f t ( x ) .

Original: The labels on the scale are “bad”, “poor”, “fair”, “good”, and “excellent” (see Figure 7).
Revised: The labels on the scale are “bad”, “poor”, “fair”, “good”, and “excellent” (see Figure 8).

Original: We then rank all images in M by the mean opinion scores, and choose top- n 4 images with the smallest scores to form the counterexample set S
Revised: We then rank all images in M by the mean opinion scores, and choose top- n 4 images with the smallest scores to form the counterexample set S

Original: = { x i } n 4 i =1 .
Revised: = { x i } n 4 i =1 .

Original: Finally, we seek pixel-level segmentation results from human annotators for each image in S (see Figure 8).
Revised: Finally, we seek pixel-level segmentation results from human annotators for each image in S (see Figure 9).

Original: 3. M ODEL RECTIFICATION
Revised: 3. M ODEL RECTIFICATION

Original: Algorithm 1: The proposed framework efﬁciently troubleshooting segmentation models
Revised: Algorithm 1: The proposed framework efﬁciently troubleshooting segmentation models

Original: Input: An unlabelled image set D , a target model f (0) t and the dataset S (0) on which it is pre-trained, a group of competing models { g j } mj =1 , the maximum number r of ﬁne-tuning rounds, hyper-parameters n 3 , n
Revised: Input: An unlabelled image set D , a target model f (0) t and the dataset S (0) on which it is pre-trained, a group of competing models { g j } mj =1 , the maximum number r of ﬁne-tuning rounds, hyper-parameters n 3 , n

Original: Output: Improved f ( r ) t 1 S ← ∅ 2 for j ← 1 to m do
Revised: Output: Improved f ( r ) t 1 S ← ∅ 2 for j ← 1 to m do

Original: Compute segmentation predictions { g j ( x ) , x ∈ D} 4 end 5 for i ← 0 to r − 1 do
Revised: Compute segmentation predictions { g j ( x ) , x ∈ D} 4 end 5 for i ← 0 to r − 1 do

Original: M ( i +1) ← ∅
Revised: M ( i +1) ← ∅

Original: Compute segmentation predictions { f ( i ) t ( x ) , x ∈ D} for j ← 1 to m do
Revised: Compute segmentation predictions { f ( i ) t ( x ) , x ∈ D} for j ← 1 to m do

Original: Compute the distances { d ( f ( i ) t ( x ) , g j ( x )) , x ∈ D}
Revised: Compute the distances { d ( f ( i ) t ( x ) , g j ( x )) , x ∈ D}

Original: Divide D into c subgroups according to f ( i ) t
Revised: Divide D into c subgroups according to f ( i ) t

Original: Filter images by the content constraint
Revised: Filter images by the content constraint

Original: Select top- n 3 images by solving
Revised: Select top- n 3 images by solving

Original: Eq.
Revised: Eq.

Original: (2) to form M j
Revised: (2) to form M j

Original: M ( i +1) ← M ( i +1) (cid:83) M j
Revised: M ( i +1) ← M ( i +1) (cid:83) M j

Original: Reverse the roles of f ( i ) t and g j , and repeat Steps 10 to end
Revised: Reverse the roles of f ( i ) t and g j , and repeat Steps 10 to end

Original: Source weak human scores for M ( i +1)
Revised: Source weak human scores for M ( i +1)

Original: Select top- n 4 images with the lowest quality scores and collect pixel-level labels from humans to form S ( i +1)
Revised: Select top- n 4 images with the lowest quality scores and collect pixel-level labels from humans to form S ( i +1)

Original: S ← S (cid:83) S ( i +1) ;
Revised: S ← S (cid:83) S ( i +1) ;

Original: Fine-tune f ( i ) t on the combination of S and S (0)
Revised: Fine-tune f ( i ) t on the combination of S and S (0)

Original: 20 end
Revised: 20 end

Original: The labelled images in S give us a great opportunity to improve f t by learning from these failures.
Revised: The labelled images in S give us a great opportunity to improve f t by learning from these failures.

Original: In order to be resistant to catastrophic forgetting (McCloskey & Cohen, 1989), the ﬁne-tuning can also include labelled data previously used to train f t .
Revised: In order to be resistant to catastrophic forgetting (McCloskey & Cohen, 1989), the ﬁne-tuning can also include labelled data previously used to train f t .

Original: We may iterate through the whole procedure of Failure identiﬁcation and model rectiﬁcation several rounds, leading to a progressive two-stage framework for efﬁciently troubleshooting segmentation models with human-in-the-loop.
Revised: We may iterate through the whole procedure of Failure identiﬁcation and model rectiﬁcation several rounds, leading to a progressive two-stage framework for efﬁciently troubleshooting segmentation models with human-in-the-loop.

Original: When the iterative setting is enabled, the size of S is growing:
Revised: When the iterative setting is enabled, the size of S is growing:

Original: S =
Revised: S =

Original: (cid:83) ri =1 S ( i ) , where S ( i ) is the counterexample set created in the i -th ﬁne-tuning round.
Revised: (cid:83) ri =1 S ( i ) , where S ( i ) is the counterexample set created in the i -th ﬁne-tuning round.

Original: Denoting the initial target model by f (0) t ,
Revised: Denoting the initial target model by f (0) t ,

Original: we ﬁne-tune f
Revised: we ﬁne-tune f

Original: ( i − 1) t on the combination of S accumulated in the previous i − 1 rounds and S (0) used for pre-training.
Revised: ( i − 1) t on the combination of S accumulated in the previous i − 1 rounds and S (0) used for pre-training.

Original: We summarize the proposed framework in Algorithm 1.
Revised: We summarize the proposed framework in Algorithm 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we use the semantic segmentation task deﬁned in PASCAL VOC (Everingham et al., 2010) as a speciﬁc application to demonstrate the feasibility of our method.
Revised: In this section, we use the semantic segmentation task deﬁned in PASCAL VOC (Everingham et al., 2010) as a speciﬁc application to demonstrate the feasibility of our method.

Original: It is worth noting that the proposed framework can be applied to other segmentation tasks, such as those required in autonomous driving (Ess et al., 2009; Cordts et al., 2016) and medical image analysis (Ronneberger et al., 2015).
Revised: It is worth noting that the proposed framework can be applied to other segmentation tasks, such as those required in autonomous driving (Ess et al., 2009; Cordts et al., 2016) and medical image analysis (Ronneberger et al., 2015).

Original: 4. E XPERIMENTAL SETUPS
Revised: 4. E XPERIMENTAL SETUPS

Original: Segmentation models In our experiments, we choose the target model f t to be the state-of-the-art DeepLabV3Plus (Chen et al., 2018b) with DRN (Yu et al., 2017) as the backbone (termed DeepLabV3P-DRN).
Revised: Segmentation models In our experiments, we choose the target model f t to be the state-of-the-art DeepLabV3Plus (Chen et al., 2018b) with DRN (Yu et al., 2017) as the backbone (termed DeepLabV3P-DRN).

Original: We include ﬁve competing models:
Revised: We include ﬁve competing models:

Original: DeepLabV3Plus with
Revised: DeepLabV3Plus with

Original: ResNet101 (He et al., 2016) (termed DeepLabV3P-RN101), DeepLabV3 (Chen et al., 2017) with
Revised: ResNet101 (He et al., 2016) (termed DeepLabV3P-RN101), DeepLabV3 (Chen et al., 2017) with

Original: ResNet101 (termed DeepLabV3-RN101), DFN (Yu et al., 2018), Light-Weight ReﬁneNet (Nekrasov et al., 2018) with ResNet50 and MobileNetV2 (Sandler et al., 2018) (termed LReﬁneNet-RN50 and
Revised: ResNet101 (termed DeepLabV3-RN101), DFN (Yu et al., 2018), Light-Weight ReﬁneNet (Nekrasov et al., 2018) with ResNet50 and MobileNetV2 (Sandler et al., 2018) (termed LReﬁneNet-RN50 and

Original: LReﬁneNet-MNV2, respectively).
Revised: LReﬁneNet-MNV2, respectively).

Original: Publicly available pre-trained weights on PASCAL VOC(Everingham et al., 2010) are used for all models.
Revised: Publicly available pre-trained weights on PASCAL VOC(Everingham et al., 2010) are used for all models.

Original: Following (Chen et al., 2017; 2018b), images are cropped and resized to 513 × 513 before inference.
Revised: Following (Chen et al., 2017; 2018b), images are cropped and resized to 513 × 513 before inference.

Original: Constructing D
Revised: Constructing D

Original: We ﬁrst brieﬂy introduce the semantic segmentation database in PASCAL
Revised: We ﬁrst brieﬂy introduce the semantic segmentation database in PASCAL

Original: VOC (Everingham et al., 2010).
Revised: VOC (Everingham et al., 2010).

Original: It contains 1 , 464 images for training (denoted by S (0) ) and 1 , 149 for validation (denoted by T (0) ) with 20 scene categories (e.g., aeroplane, bicycle and person).
Revised: It contains 1 , 464 images for training (denoted by S (0) ) and 1 , 149 for validation (denoted by T (0) ) with 20 scene categories (e.g., aeroplane, bicycle and person).

Original: We use the 20 class labels and combinations of them as keywords to crawl images from the Internet.
Revised: We use the 20 class labels and combinations of them as keywords to crawl images from the Internet.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: No other constraints are imposed during data collection.
Revised: No other constraints are imposed during data collection.

Original: As a result
Revised: As a result

Original: [Table]
Revised: [Table]

Original: the database D includes a total of more than 40 , 000 images, which is much larger than PASCAL VOC training and validation sets.
Revised: the database D includes a total of more than 40 , 000 images, which is much larger than PASCAL VOC training and validation sets.

Original: Constructing M
Revised: Constructing M

Original: We empirically set the two parameters p k and q k in the content constraint to the ﬁrst and the third quartiles of the k -th category sizes of the training images in PASCAL VOC, respectively.
Revised: We empirically set the two parameters p k and q k in the content constraint to the ﬁrst and the third quartiles of the k -th category sizes of the training images in PASCAL VOC, respectively.

Original: To strike a good balance between the human effort of weakly labelling and the content diversity of M , we choose n 3 = 25 in Eq.
Revised: To strike a good balance between the human effort of weakly labelling and the content diversity of M , we choose n 3 = 25 in Eq.

Original: Due to the existence of duplicated images, the actual sizes of
Revised: Due to the existence of duplicated images, the actual sizes of

Original: {M ( i ) } range from 1 , 500 to 2 , 000 for different rounds
Revised: {M ( i ) } range from 1 , 500 to 2 , 000 for different rounds

Original: Constructing S
Revised: Constructing S

Original: We collect weak labels for images in M from three volunteer graduate students, who have adequate knowledge of computer vision, and are told the detailed purpose of the study.
Revised: We collect weak labels for images in M from three volunteer graduate students, who have adequate knowledge of computer vision, and are told the detailed purpose of the study.

Original: Each subject is asked to give an integer score between one and ﬁve for each image to represent one of the ﬁve categories, with a higher value indicating better segmentation quality.
Revised: Each subject is asked to give an integer score between one and ﬁve for each image to represent one of the ﬁve categories, with a higher value indicating better segmentation quality.

Original: All out-of-domain images are given a score of positive inﬁnity, meaning that any subject is granted to eliminate an image without agreement from the other two.
Revised: All out-of-domain images are given a score of positive inﬁnity, meaning that any subject is granted to eliminate an image without agreement from the other two.

Original: The mean opinion scores averaged across all subjects are used to rank images in M .
Revised: The mean opinion scores averaged across all subjects are used to rank images in M .

Original: The hyper-parameter n 4 in Algorithm 1 is set to 100 .
Revised: The hyper-parameter n 4 in Algorithm 1 is set to 100 .

Original: Representative examples in S are shown in Figure 3 and Figure 6 along with images in T (0) .
Revised: Representative examples in S are shown in Figure 3 and Figure 6 along with images in T (0) .

Original: It is clear that images in S are visually much harder.
Revised: It is clear that images in S are visually much harder.

Original: We invite the same three students to provide ground truth segmentation results for images in S , following the annotation guidance in PASCAL VOC, with the help of the online annotation tool Labelbox 5 .
Revised: We invite the same three students to provide ground truth segmentation results for images in S , following the annotation guidance in PASCAL VOC, with the help of the online annotation tool Labelbox 5 .

Original: Our annotation process includes two stages with the goal of obtaining consistent segmentation maps.
Revised: Our annotation process includes two stages with the goal of obtaining consistent segmentation maps.

Original: In the ﬁrst stage, each subject is assigned one-third of the images to provide pixel-level labels.
Revised: In the ﬁrst stage, each subject is assigned one-third of the images to provide pixel-level labels.

Original: In the second stage, we carry out cross validation to improve the annotation consistency.
Revised: In the second stage, we carry out cross validation to improve the annotation consistency.

Original: Each student takes turn to check the segmentation maps completed by others, marking the positions and types of possible annotation errors.
Revised: Each student takes turn to check the segmentation maps completed by others, marking the positions and types of possible annotation errors.

Original: During cross checking, the subjects can discuss with each other to reach an agreement on a small part of ambiguous annotation.
Revised: During cross checking, the subjects can discuss with each other to reach an agreement on a small part of ambiguous annotation.

Original: Iterative model correction We perform a total of r = 2 ﬁne-tuning rounds.
Revised: Iterative model correction We perform a total of r = 2 ﬁne-tuning rounds.

Original: As suggested in (Chen et al., 2017), ﬁne-tuning for each round is carried out by stochastic gradient descent with momentum
Revised: As suggested in (Chen et al., 2017), ﬁne-tuning for each round is carried out by stochastic gradient descent with momentum

Original: [Figure]
Revised: [Figure]

Original: 9 and weight decay 5 × 10 − 4 .
Revised: 9 and weight decay 5 × 10 − 4 .

Original: We grid search the initial learning rate from { 1 , 5 , 10 } × 10 − 5 , and choose the one with the highest mIoU on T (0) .
Revised: We grid search the initial learning rate from { 1 , 5 , 10 } × 10 − 5 , and choose the one with the highest mIoU on T (0) .

Original: The learning rate decay is guided by the polynomial policy.
Revised: The learning rate decay is guided by the polynomial policy.

Original: We set the mini-batch size and the maximum epoch number to 2 and 80 , respectively.
Revised: We set the mini-batch size and the maximum epoch number to 2 and 80 , respectively.

Original: Model evaluation How to reliably probe the generalization of computer vision models when deployed in the real world is by itself a challenging problem being actively investigated (Hendrycks et al., 2019; Arnab et al., 2018; Recht et al., 2019).
Revised: Model evaluation How to reliably probe the generalization of computer vision models when deployed in the real world is by itself a challenging problem being actively investigated (Hendrycks et al., 2019; Arnab et al., 2018; Recht et al., 2019).

Original: Observing progress on T (0) is not a wise choice because this set may only contain few catastrophic failures of the target model
Revised: Observing progress on T (0) is not a wise choice because this set may only contain few catastrophic failures of the target model

Original: f ( i ) t .
Revised: f ( i ) t .

Original: It may also be unfair to employ the counterexample set S ( i +1) to evaluate the relative progress of f ( i ) t against the set of competing models { g j } mj =1 due to the inclusion of the weakly labelling and ﬁltering procedure.
Revised: It may also be unfair to employ the counterexample set S ( i +1) to evaluate the relative progress of f ( i ) t against the set of competing models { g j } mj =1 due to the inclusion of the weakly labelling and ﬁltering procedure.

Original: Inspired by the maximum discrepancy competition methodology for image classiﬁcation (Wang et al., 2020), here we construct a new unbiased test set T ( i +1) ⊂
Revised: Inspired by the maximum discrepancy competition methodology for image classiﬁcation (Wang et al., 2020), here we construct a new unbiased test set T ( i +1) ⊂

Original: M ( i +1) to compare f ( i ) t with { g j } mj =1 by adding the following constraints.
Revised: M ( i +1) to compare f ( i ) t with { g j } mj =1 by adding the following constraints.

Original: First, similar as the construction of S , all out-of-domain images are ﬁltered out.
Revised: First, similar as the construction of S , all out-of-domain images are ﬁltered out.

Original: Second, to encourage class diversity, we retain a maximum of four images that contain the same main foreground object (i.e., T ( i +1) has at most four “car” images).
Revised: Second, to encourage class diversity, we retain a maximum of four images that contain the same main foreground object (i.e., T ( i +1) has at most four “car” images).

Original: Third, to keep evaluation fair for the competing models, images used to ﬁne-tune the target model
Revised: Third, to keep evaluation fair for the competing models, images used to ﬁne-tune the target model

Original: f
Revised: f

Original: ( i ) t are not included in T ( i +1) .
Revised: ( i ) t are not included in T ( i +1) .

Original: In our experiments, the size of T ( i +1) is set to 30 .
Revised: In our experiments, the size of T ( i +1) is set to 30 .

Original: 4. M AIN RESULTS
Revised: 4. M AIN RESULTS

Original: Quantitative results We use a standard evaluation metric - mIoU to quantify the semantic segmentation performance.
Revised: Quantitative results We use a standard evaluation metric - mIoU to quantify the semantic segmentation performance.

Original: All results are listed in Table 1, where we ﬁnd that, before the ﬁrst round of ﬁne-tuning, all models achieve competitive results on T (0) , implying close-to-saturation performance on PASCAL VOC.
Revised: All results are listed in Table 1, where we ﬁnd that, before the ﬁrst round of ﬁne-tuning, all models achieve competitive results on T (0) , implying close-to-saturation performance on PASCAL VOC.

Original: However, when tested on T (1) , the performance of all models drops signiﬁcantly, indicating that many images in T (1) are able to falsify both the target model f (0) t and the associated competing model g j .
Revised: However, when tested on T (1) , the performance of all models drops signiﬁcantly, indicating that many images in T (1) are able to falsify both the target model f (0) t and the associated competing model g j .

Original: This also provides direct evidence that existing segmentation models could be particularly weak at certain real-world generalization, which is not surprising because the 1 ,training images are deemed to be extremely sparsely distributed in the space of natural images.
Revised: This also provides direct evidence that existing segmentation models could be particularly weak at certain real-world generalization, which is not surprising because the 1 ,training images are deemed to be extremely sparsely distributed in the space of natural images.

Original: Moreover, the top- 1 model on T (0) does not necessarily perform the best on T (1) , conforming to the results in (Wang et al., 2020).
Revised: Moreover, the top- 1 model on T (0) does not necessarily perform the best on T (1) , conforming to the results in (Wang et al., 2020).

Original: After the ﬁrst round of ﬁne-tuning, f (1) t achieves noticeable improvements on T (2) , whereas all competing models experience different degrees of performance drops.
Revised: After the ﬁrst round of ﬁne-tuning, f (1) t achieves noticeable improvements on T (2) , whereas all competing models experience different degrees of performance drops.

Original: This suggests that the target model begins to introspect and learn from its counterexamples in S (1) .
Revised: This suggests that the target model begins to introspect and learn from its counterexamples in S (1) .

Original: After the second round of ﬁne-tuning, the mIoU of f (2) t on T (3) is boosted by around 18% , surpassing all competing models by a larger margin than the previous round.
Revised: After the second round of ﬁne-tuning, the mIoU of f (2) t on T (3) is boosted by around 18% , surpassing all competing models by a larger margin than the previous round.

Original: This shows that our method successfully learns from and combines the best aspects of the competing models to ﬁx its own defects, with approximately the same performance on T (0) .
Revised: This shows that our method successfully learns from and combines the best aspects of the competing models to ﬁx its own defects, with approximately the same performance on T (0) .

Original: In our experiments, we only perform two rounds of ﬁne-tuning due to limited computation and human resources, while we expect further performance gains under the proposed framework if tuned for more rounds.
Revised: In our experiments, we only perform two rounds of ﬁne-tuning due to limited computation and human resources, while we expect further performance gains under the proposed framework if tuned for more rounds.

Original: Qualitative results We show representative test images before and after the ﬁrst round of ﬁnetuning in Figure 4.
Revised: Qualitative results We show representative test images before and after the ﬁrst round of ﬁnetuning in Figure 4.

Original: Before ﬁne-tuning, the competing models can effectively ﬁnd defects of the target model with incorrect semantics and/or boundaries.
Revised: Before ﬁne-tuning, the competing models can effectively ﬁnd defects of the target model with incorrect semantics and/or boundaries.

Original: After ﬁne-tuning, the target model does a clearly better job on the corner-case samples.
Revised: After ﬁne-tuning, the target model does a clearly better job on the corner-case samples.

Original: We also visually compare f (0) t and f (1) t on T (2) , where we observe the improved robustness to unseen corner-case images.
Revised: We also visually compare f (0) t and f (1) t on T (2) , where we observe the improved robustness to unseen corner-case images.

Original: Another interesting ﬁnding is that our method can ﬁx a general class of model mistakes by learning from a small set of valuable failures.
Revised: Another interesting ﬁnding is that our method can ﬁx a general class of model mistakes by learning from a small set of valuable failures.

Original: Figure 5 shows such an example, where the target model only sees 10 “car” images in S (1) , and is able to generalize to images with similar unusual viewpoints.
Revised: Figure 5 shows such an example, where the target model only sees 10 “car” images in S (1) , and is able to generalize to images with similar unusual viewpoints.

Original: The generalization of the target model on “car” images with unusual viewpoints is largely improved after the ﬁrst
Revised: The generalization of the target model on “car” images with unusual viewpoints is largely improved after the ﬁrst

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have formulated a new computer vision problem that aims to efﬁciently expose the failures of top-performing segmentation models and learn from such failures for improved generalization in the real world.
Revised: We have formulated a new computer vision problem that aims to efﬁciently expose the failures of top-performing segmentation models and learn from such failures for improved generalization in the real world.

Original: We proposed an efﬁcient human-in-the-loop framework for troubleshooting segmentation models, and demonstrated its promise under realistic settings.
Revised: We proposed an efﬁcient human-in-the-loop framework for troubleshooting segmentation models, and demonstrated its promise under realistic settings.

Original: We hope this work sheds light on a new line of research that requires both machine vision and human interaction.
Revised: We hope this work sheds light on a new line of research that requires both machine vision and human interaction.

Original: In the future, we plan to explore the idea of leveraging natural failures for improving model robustness in broader vision problems, such as video understanding (Tran et al., 2015), computer-aided diagnosis (Mansoor et al., 2015), and computational neuroscience (Golan et al., 2019).
Revised: In the future, we plan to explore the idea of leveraging natural failures for improving model robustness in broader vision problems, such as video understanding (Tran et al., 2015), computer-aided diagnosis (Mansoor et al., 2015), and computational neuroscience (Golan et al., 2019).

Original: Moreover, while the current work remains to focus on improving the model’s standard generalization (with an emphasis on natural corner-case samples), our future study will investigate how this could be jointly optimized with adversarial robustness.
Revised: Moreover, while the current work remains to focus on improving the model’s standard generalization (with an emphasis on natural corner-case samples), our future study will investigate how this could be jointly optimized with adversarial robustness.

Original: A C OMPARISON WITH ENTROPY - BASED METHOD
Revised: A C OMPARISON WITH ENTROPY - BASED METHOD

Original: Entropy has been used as an uncertainty metric to select hard samples (Joshi et al., 2009).
Revised: Entropy has been used as an uncertainty metric to select hard samples (Joshi et al., 2009).

Original: The selected hard samples can be further used as training data to improve target model performance.
Revised: The selected hard samples can be further used as training data to improve target model performance.

Original: In this section, we compare our method with the entropy-based method in (Joshi et al., 2009) to show the advantage of our method than traditional entropy-based methods.
Revised: In this section, we compare our method with the entropy-based method in (Joshi et al., 2009) to show the advantage of our method than traditional entropy-based methods.

Original: Since our goal is to improve model performance on open-world images instead of ﬁxed benchmark datasets, we need to tell which method can achieve models with better performance on open-world images, which can be efﬁciently done by the MAD competition in (Wang et al., 2020).
Revised: Since our goal is to improve model performance on open-world images instead of ﬁxed benchmark datasets, we need to tell which method can achieve models with better performance on open-world images, which can be efﬁciently done by the MAD competition in (Wang et al., 2020).

Original: [Table]
Revised: [Table]

Original: For both methods, we conduct ﬁne-tuning for one round and all experimental settings (including target model structure, human labelling budget, etc.) are kept identical.
Revised: For both methods, we conduct ﬁne-tuning for one round and all experimental settings (including target model structure, human labelling budget, etc.) are kept identical.

Original: The only difference comes from the sampling strategy: our method samples according to Algorithm 1, while entropy-based method selects images with largest entropy.
Revised: The only difference comes from the sampling strategy: our method samples according to Algorithm 1, while entropy-based method selects images with largest entropy.

Original: We then conduct MAD competition Wang et al.
Revised: We then conduct MAD competition Wang et al.

Original: (2020) on the two models obtained from each method to compare their performance on open-world images.
Revised: (2020) on the two models obtained from each method to compare their performance on open-world images.

Original: Numerical results are shown in Table 2.
Revised: Numerical results are shown in Table 2.

Original: As we can see, the model achieved by our method is winning the MAD competition with a noticeable mIoU advantage, showing that the model achieved by our method has better performance on open-world images than the model ﬁne-tuned by entropy-based method.
Revised: As we can see, the model achieved by our method is winning the MAD competition with a noticeable mIoU advantage, showing that the model achieved by our method has better performance on open-world images than the model ﬁne-tuned by entropy-based method.

Original: B M ORE VISUALIZATION RESULTS
Revised: B M ORE VISUALIZATION RESULTS

Original: [Figure]
Revised: [Figure]

Revised: C A BLATION S TUDY

Revised: In order to show the testing results hold as we increase the size of T ( i ) (denoted as k ), we ﬁrst evaluate the performance of the six models with different number of testing images, and conduct a global ranking among the six models.

Revised: We use top- k ranking to denote the global ranking evaluated with k testing images.

Revised: Then we calculate the SRCC values between the top- 40 ranking (as reference) and other top-

Revised: k rankings with k ∈ { 20 , 21 , · · · , 39 } .

Revised: As shown in Figure 7, the ranking results remain stable when k > 25 .

Revised: And our active ﬁnetuned target model always achieves the best performance on

Revised: T (1) among the six models, for any k between 20 and 40 .

Original: C S UBJECTIVE TESTING GUI S
Revised: D S UBJECTIVE TESTING GUI S

Original: Graphical user interfaces (GUIs) for our weakly and pixel-wise labelling experiments are shown in
Revised: Graphical user interfaces (GUIs) for our weakly and pixel-wise labelling experiments are shown in

Original: Figure 7 and Figure 8, respectively.
Revised: Figure 8 and Figure 9, respectively.

Original: For weakly labelling experiments, we build our own GUI: an image is rendered alongside the prediction made by the target model; a scale-and-slider applet is utilized to collect the absolute category rating score of that image as described in Section 3.1.
Revised: For weakly labelling experiments, we build our own GUI: an image is rendered alongside the prediction made by the target model; a scale-and-slider applet is utilized to collect the absolute category rating score of that image as described in Section 3.1.

Original: For pixel-wise segmentation labelling experiments, we use LabelBox segmentation template 6 to build our
Revised: For pixel-wise segmentation labelling experiments, we use LabelBox segmentation template 6 to build our

Original: GUI
Revised: GUI.