Original: C ONNECTIVITY - CONSTRAINED INTERACTIVE ANNOTATIONS FOR PANOPTIC SEGMENTATION
Revised: C ONNECTIVITY - CONSTRAINED INTERACTIVE ANNOTATIONS FOR PANOPTIC SEGMENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large-scale ground truth data sets are of crucial importance for deep learning based segmentation models, but annotating per-pixel masks is prohibitively time consuming.
Revised: Large-scale ground truth data sets are of crucial importance for deep learning based segmentation models, but annotating per-pixel masks is prohibitively time consuming.

Original: In this paper, we investigate interactive graph-based segmentation algorithms that enforce connectivity
Revised: In this paper, we investigate interactive graph-based segmentation algorithms that enforce connectivity

Original: To be more precise, we introduce an instanceaware heuristic of a discrete Potts model, and a class-aware Integer Linear Programming
Revised: To be more precise, we introduce an instanceaware heuristic of a discrete Potts model, and a class-aware Integer Linear Programming

Original: ILP formulation that ensures global optimum.
Revised: ILP formulation that ensures global optimum.

Original: Both algorithms can take RGB or utilize the feature maps from any DCNN, whether trained on the target dataset or not as input.
Revised: Both algorithms can take RGB or utilize the feature maps from any DCNN, whether trained on the target dataset or not as input.

Original: We present competitive semantic (and panoptic) segmentation results on the PASCAL VOC 2012 and Cityscapes dataset given initial scribbles
Revised: We present competitive semantic (and panoptic) segmentation results on the PASCAL VOC 2012 and Cityscapes dataset given initial scribbles

Original: We also demonstrate that our interactive approach can reach
Revised: We also demonstrate that our interactive approach can reach

Original: 6% mIoU on
Revised: 6% mIoU on

Original: VOC validation set with an overhead of just 3 correction scribbles
Revised: VOC validation set with an overhead of just 3 correction scribbles

Original: They are thus suitable for interactive annotation on new or existing datasets or can be used inside any weakly supervised learning framework on new datasets.
Revised: They are thus suitable for interactive annotation on new or existing datasets or can be used inside any weakly supervised learning framework on new datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Convolutional Neural Networks (DCNNs) excel at a wide range of image recognition tasks (He et al., 2016;
Revised: Deep Convolutional Neural Networks (DCNNs) excel at a wide range of image recognition tasks (He et al., 2016;

Original: Ren et al., 2017; Shelhamer et al., 2017), such as semantic segmentation (Chen et al., 2018b; Shelhamer et al., 2017; Zhao et al., 2017) and panoptic segmentation (Kirillov et al., 2019b;
Revised: Ren et al., 2017; Shelhamer et al., 2017), such as semantic segmentation (Chen et al., 2018b; Shelhamer et al., 2017; Zhao et al., 2017) and panoptic segmentation (Kirillov et al., 2019b;

Original: Yang et al., 2019; Xiong et al., 2019; Kirillov et al., 2019a).
Revised: Yang et al., 2019; Xiong et al., 2019; Kirillov et al., 2019a).

Original: Semantic segmentation studies the tasks of assigning a class label to each pixel of an image, while instance segmentation (Chen et al., 2018a) detects and segment each object instance.
Revised: Semantic segmentation studies the tasks of assigning a class label to each pixel of an image, while instance segmentation (Chen et al., 2018a) detects and segment each object instance.

Original: Panoptic segmentation uniﬁes both tasks that investigate to segment both things (such as person, cars) and stuff (such as road, sky) classes.
Revised: Panoptic segmentation uniﬁes both tasks that investigate to segment both things (such as person, cars) and stuff (such as road, sky) classes.

Original: While DCNNs show outstanding results for semantic and panoptic segmentation, they have two conceptional problems.
Revised: While DCNNs show outstanding results for semantic and panoptic segmentation, they have two conceptional problems.

Original: First, they require huge amounts of annotated data.
Revised: First, they require huge amounts of annotated data.

Original: Annotating image segmentation masks is a very time consuming and labor extensive task.
Revised: Annotating image segmentation masks is a very time consuming and labor extensive task.

Original: For example, annotating a semantic image mask took “more than 1.5h on average” on the Cityscapes dataset (Cordts et al., 2016).
Revised: For example, annotating a semantic image mask took “more than 1.5h on average” on the Cityscapes dataset (Cordts et al., 2016).

Original: In addition, DCNNs rely on their implicitly learned generalization probability and most of the stateof-the-art architectures do not make use of any domain speciﬁc knowledge, such as neighborhood relations and connectivity priors for segments.
Revised: In addition, DCNNs rely on their implicitly learned generalization probability and most of the stateof-the-art architectures do not make use of any domain speciﬁc knowledge, such as neighborhood relations and connectivity priors for segments.

Original: On the contrary, classical graph based segmentation models (Boykov et al., 2001; Yedidia et al., 2003) do not require any learning data and can incorporate speciﬁc domain knowledge.
Revised: On the contrary, classical graph based segmentation models (Boykov et al., 2001; Yedidia et al., 2003) do not require any learning data and can incorporate speciﬁc domain knowledge.

Original: Their major drawback is that they rely on human-designed similarity features and require complex optimization algorithms or solvers, which are mainly CPU based and non-suitable for real time applications.
Revised: Their major drawback is that they rely on human-designed similarity features and require complex optimization algorithms or solvers, which are mainly CPU based and non-suitable for real time applications.

Original: In this work, we explore the combination of DCNNs and graph-based algorithms for annotating ground truth segmentation.
Revised: In this work, we explore the combination of DCNNs and graph-based algorithms for annotating ground truth segmentation.

Original: We propose two interactive algorithms that both enforce the connectivity prior (to be more precise in Sec. 2.1).
Revised: We propose two interactive algorithms that both enforce the connectivity prior (to be more precise in Sec. 2.1).

Original: Speciﬁcally, we design a heuristic region growing method based on the Potts model (Potts, 1952), as well as an integer linear programming (ILP) formulation of the markov random ﬁeld (MRF) (Boykov et al., 1998).
Revised: Speciﬁcally, we design a heuristic region growing method based on the Potts model (Potts, 1952), as well as an integer linear programming (ILP) formulation of the markov random ﬁeld (MRF) (Boykov et al., 1998).

Original: We utilize scribble based annotations from human annotators as initialized or iterative hard constraints for our algorithms, which is typical in a human-in-the-loop (HITL) annotation process.
Revised: We utilize scribble based annotations from human annotators as initialized or iterative hard constraints for our algorithms, which is typical in a human-in-the-loop (HITL) annotation process.

Original: We explore two different scenarios.
Revised: We explore two different scenarios.

Original: In the ﬁrst scenario, we assume that for the segmentation task, there is already a pre-trained DCNN with the same class mapping.
Revised: In the ﬁrst scenario, we assume that for the segmentation task, there is already a pre-trained DCNN with the same class mapping.

Original: Thus, we could make full use of DCNN’s probability map and scribbles as input to our algorithms.
Revised: Thus, we could make full use of DCNN’s probability map and scribbles as input to our algorithms.

Original: In the second scenario, we assume that no pre-trained DCNN for the same objective is available.
Revised: In the second scenario, we assume that no pre-trained DCNN for the same objective is available.

Original: This is true for a lot of existing datasets, e.g., Cityscapes does not contain any class labels for lane marking, which is crucial information for an autonomous vehicle.
Revised: This is true for a lot of existing datasets, e.g., Cityscapes does not contain any class labels for lane marking, which is crucial information for an autonomous vehicle.

Original: In this case, we cannot use the class speciﬁc probability map, but the more generic low level features of a DCNN can be utilized as feature description for the algorithm.
Revised: In this case, we cannot use the class speciﬁc probability map, but the more generic low level features of a DCNN can be utilized as feature description for the algorithm.

Original: To investigate the general purpose of low level DCNN features, we compare DCNN trained on
Revised: To investigate the general purpose of low level DCNN features, we compare DCNN trained on

Original: ImageNet (Russakovsky et al., 2015) and COCO (Lin et al., 2014) with and without ﬁne tuning on the target dataset.
Revised: ImageNet (Russakovsky et al., 2015) and COCO (Lin et al., 2014) with and without ﬁne tuning on the target dataset.

Original: Our experiments show that incorporating the connectivity prior as well as the
Revised: Our experiments show that incorporating the connectivity prior as well as the

Original: DCNN features greatly improves the algorithm performance.
Revised: DCNN features greatly improves the algorithm performance.

Original: We present competitive semantic (and panoptic) segmentation results on the PASCAL VOC 2012 (Everingham et al., 2015) and Cityscapes dataset.
Revised: We present competitive semantic (and panoptic) segmentation results on the PASCAL VOC 2012 (Everingham et al., 2015) and Cityscapes dataset.

Original: To the best of our knowledge, our method is the ﬁrst “non-DCNN” panoptic segmentation algorithm on Cityscapes with competing results, which shows the potential improvement gained by a combined approach.
Revised: To the best of our knowledge, our method is the ﬁrst “non-DCNN” panoptic segmentation algorithm on Cityscapes with competing results, which shows the potential improvement gained by a combined approach.

Original: See Fig.
Revised: See Fig.

Original: 1 for a visualization that uses initialized scribbles as input.
Revised: 1 for a visualization that uses initialized scribbles as input.

Original: We also prototype our algorithms for the task of interactive annotation.
Revised: We also prototype our algorithms for the task of interactive annotation.

Original: Based on initial scribbles and a baseline DCNN’s probability map, our algorithms acheive 90 .
Revised: Based on initial scribbles and a baseline DCNN’s probability map, our algorithms acheive 90 .

Original: 9% mIoU on VOC validation set, with just 3 rounds ( 1 scribbles per image each round) of correction scribbles.
Revised: 9% mIoU on VOC validation set, with just 3 rounds ( 1 scribbles per image each round) of correction scribbles.

Original: Summarized, our key contributions are
Revised: Summarized, our key contributions are

Original: • a novel heuristic algorithm and a class-aware ILP formulation with connectivity prior, • an in depth analysis of a combination of DCNN and graph-based segmentation algorithms, • extensive evaluation of the scribble based interactive algorithms for semantic and panoptic segmentations on two challenging datasets.
Revised: • a novel heuristic algorithm and a class-aware ILP formulation with connectivity prior, • an in depth analysis of a combination of DCNN and graph-based segmentation algorithms, • extensive evaluation of the scribble based interactive algorithms for semantic and panoptic segmentations on two challenging datasets.

Original: Our proposed algorithms have multiple use cases in annotating datasets for segmentation.
Revised: Our proposed algorithms have multiple use cases in annotating datasets for segmentation.

Original: First, they can be used inside any HITL annotation tool, as the annotator interacts with the image in forms of scribbles until satisfaction.
Revised: First, they can be used inside any HITL annotation tool, as the annotator interacts with the image in forms of scribbles until satisfaction.

Original: Second, given initial scribbles, they can be used inside any weakly or semi-supervised learning framework for semantic (and panoptic) segmentation.
Revised: Second, given initial scribbles, they can be used inside any weakly or semi-supervised learning framework for semantic (and panoptic) segmentation.

Original: Related Work.
Revised: Related Work.

Original: The procedure of annotating per pixel segmentation masks is similar to interactive image segmentation, which is widely studied in the past decade.
Revised: The procedure of annotating per pixel segmentation masks is similar to interactive image segmentation, which is widely studied in the past decade.

Original: The method using bounding boxes (Rother et al., 2004) is suitable for instance segmentation, which requires the user to draw the box as tight as possible.
Revised: The method using bounding boxes (Rother et al., 2004) is suitable for instance segmentation, which requires the user to draw the box as tight as possible.

Original: Recently, 4 extreme points (Maninis et al., 2018; Papadopoulos et al., 2017) clicking is used as an alternative which shows superior result.
Revised: Recently, 4 extreme points (Maninis et al., 2018; Papadopoulos et al., 2017) clicking is used as an alternative which shows superior result.

Original: On the other hand, polygon based methods (Russell et al., 2008) require users to carefully click the extreme points of things and stuff, and the accuracy heavily depends on the number of clicks.
Revised: On the other hand, polygon based methods (Russell et al., 2008) require users to carefully click the extreme points of things and stuff, and the accuracy heavily depends on the number of clicks.

Original: Among others, scribbles are recognized as a more user-friendly way (Boykov & Jolly, 2001;
Revised: Among others, scribbles are recognized as a more user-friendly way (Boykov & Jolly, 2001;

Original: Levin et al., 2008).
Revised: Levin et al., 2008).

Original: Moreover, it is also natural to annotate stuff classes using scribbles.
Revised: Moreover, it is also natural to annotate stuff classes using scribbles.

Original: Modern annotation tools often adopt interactive deep learning based methods, including PolygonRNN++ (Acuna et al., 2018) and Curve-GCN (Ling et al., 2019), which allows the annotator to click on the boundaries.
Revised: Modern annotation tools often adopt interactive deep learning based methods, including PolygonRNN++ (Acuna et al., 2018) and Curve-GCN (Ling et al., 2019), which allows the annotator to click on the boundaries.

Original: Notably, a recent approach (Agustsson et al., 2019) enables both 4 extreme points clicking and scribbles correction.
Revised: Notably, a recent approach (Agustsson et al., 2019) enables both 4 extreme points clicking and scribbles correction.

Original: Besides, they can also take advantage of ensemble learning (Zhou, 2009), to combine several inference results to produce a better segmentation.
Revised: Besides, they can also take advantage of ensemble learning (Zhou, 2009), to combine several inference results to produce a better segmentation.

Original: However, these methods requires an existing ground truth dataset for DCNNs to learn on the ﬁrst hand, which may not be available when unknown domains or new classes are introduced.
Revised: However, these methods requires an existing ground truth dataset for DCNNs to learn on the ﬁrst hand, which may not be available when unknown domains or new classes are introduced.

Original: As a cheaper alternative, weakly supervised learning has drawn a lot of attention recently.
Revised: As a cheaper alternative, weakly supervised learning has drawn a lot of attention recently.

Original: (Dai et al., 2015; Khoreva et al., 2017; Li et al., 2018) claim that weakly iteratively trained by just bounding boxes and image tags, the DCNN can achieve 95% segmentation score compared to fully supervised on VOC.
Revised: (Dai et al., 2015; Khoreva et al., 2017; Li et al., 2018) claim that weakly iteratively trained by just bounding boxes and image tags, the DCNN can achieve 95% segmentation score compared to fully supervised on VOC.

Original: Since this method emphasizes on thing classes, it has worse score (or even none) on stuff classes.
Revised: Since this method emphasizes on thing classes, it has worse score (or even none) on stuff classes.

Original: Instead, (Lin et al., 2016; Can et al., 2018) claim that iteratively training a DCNN by scribble annotations suffers only a small degradation in performance on both thing and stuff classes.
Revised: Instead, (Lin et al., 2016; Can et al., 2018) claim that iteratively training a DCNN by scribble annotations suffers only a small degradation in performance on both thing and stuff classes.

Original: For graph-based methods, the (discrete) Potts model (Potts, 1952) is widely used for denoising and segmentation.
Revised: For graph-based methods, the (discrete) Potts model (Potts, 1952) is widely used for denoising and segmentation.

Original: The authors of (Shen et al., 2017) formulate the problem as an ILP and try to solve the global optimum, but only to a reduced image size, while (Nguyen & Brown, 2015) proposed an efﬁcient region-fusion-based heuristic algorithm.
Revised: The authors of (Shen et al., 2017) formulate the problem as an ILP and try to solve the global optimum, but only to a reduced image size, while (Nguyen & Brown, 2015) proposed an efﬁcient region-fusion-based heuristic algorithm.

Original: The MAP-MRF (maximizing a posterior in an MRF) has been well studied for image segmentation.
Revised: The MAP-MRF (maximizing a posterior in an MRF) has been well studied for image segmentation.

Original: Previous methods focus on local priors (Boykov &
Revised: Previous methods focus on local priors (Boykov &

Original: Veksler, 2006; Kr¨ahenb¨uhl & Koltun, 2011), and efﬁcient approximate algorithms exist, e.g., graph cut (Boykov et al., 2001) and belief propagation (Yedidia et al., 2003).
Revised: Veksler, 2006; Kr¨ahenb¨uhl & Koltun, 2011), and efﬁcient approximate algorithms exist, e.g., graph cut (Boykov et al., 2001) and belief propagation (Yedidia et al., 2003).

Original: MRF with connectivity prior has been studied in (Rempﬂer et al., 2016; Nowozin & Lampert, 2009) and formulated as an ILP, which can be solved by any ILP solver (Anand et al., 2017).
Revised: MRF with connectivity prior has been studied in (Rempﬂer et al., 2016; Nowozin & Lampert, 2009) and formulated as an ILP, which can be solved by any ILP solver (Anand et al., 2017).

Original: Since solving an ILP exactly is in general NP -hard
Revised: Since solving an ILP exactly is in general NP -hard

Original: (Land & Doig, 1960), they either solve a relaxation of the ILP, or focus on the binary MRF case.
Revised: (Land & Doig, 1960), they either solve a relaxation of the ILP, or focus on the binary MRF case.

Original: Our proposed interactive (in forms of scribble) multi-label annotation algorithms are graph based, with global connectivity prior, i.e., they enforce pixels of the same label to be connected, which allows the annotator to better control over the ﬁnal segmentation.
Revised: Our proposed interactive (in forms of scribble) multi-label annotation algorithms are graph based, with global connectivity prior, i.e., they enforce pixels of the same label to be connected, which allows the annotator to better control over the ﬁnal segmentation.

Original:  P ROPOSED APPROACH 2. P REREQUISITE
Revised:  P ROPOSED APPROACH 2. P REREQUISITE

Original: Given an image, we build an undirected graph G =
Revised: Given an image, we build an undirected graph G =

Original: ( V, E ) where the node set V represents a set of pixels (or superpixels) and E a set of edges consisting of un-ordered pairs of nodes.
Revised: ( V, E ) where the node set V represents a set of pixels (or superpixels) and E a set of edges consisting of un-ordered pairs of nodes.

Original: Image segmentation can be transformed into a graph labeling problem, where the label set C is pre-deﬁned.
Revised: Image segmentation can be transformed into a graph labeling problem, where the label set C is pre-deﬁned.

Original: When talking about segmentation, we need to ﬁrst distinguish between class, instance, and region
Revised: When talking about segmentation, we need to ﬁrst distinguish between class, instance, and region

Original: ID of a node.
Revised: ID of a node.

Original: In semantic segmentation, the task is to assign a class label to each node in a graph.
Revised: In semantic segmentation, the task is to assign a class label to each node in a graph.

Original: In panoptic segmentation, one has to further assign an instance ID to the node that belong to the “thing” class.
Revised: In panoptic segmentation, one has to further assign an instance ID to the node that belong to the “thing” class.

Original: In this paper, our algorithms require an additional region ID, which is linked to a scribble and we assume nodes with the same region ID must be connected (to be explained in Sec. 3.2).
Revised: In this paper, our algorithms require an additional region ID, which is linked to a scribble and we assume nodes with the same region ID must be connected (to be explained in Sec. 3.2).

Original: This is to deal with the case where an object of the ‘thing” or “stuff” class is separated into several connected regions, e.g., the car in Fig.
Revised: This is to deal with the case where an object of the ‘thing” or “stuff” class is separated into several connected regions, e.g., the car in Fig.

Original: 2 is separated into two regions by a tree.
Revised: 2 is separated into two regions by a tree.

Original: Afterwards, the class and region labels can be used to generate a panoptic segmentation.
Revised: Afterwards, the class and region labels can be used to generate a panoptic segmentation.

Original: 2. C ONNECTIVITY - CONSTRAINED OPTIMIZATION ALGORITHMS
Revised: 2. C ONNECTIVITY - CONSTRAINED OPTIMIZATION ALGORITHMS

Original: We ﬁrst discuss the formal deﬁnition of connectivity, and the two proposed algorithms in details, i.e., the class-agnostic heuristic algorithm of the discrete Potts model and the class-aware ILP of
Revised: We ﬁrst discuss the formal deﬁnition of connectivity, and the two proposed algorithms in details, i.e., the class-agnostic heuristic algorithm of the discrete Potts model and the class-aware ILP of

Original: MRF with connectivity constraints
Revised: MRF with connectivity constraints

Original: . 2.2. T HE CONNECTIVITY PRIOR
Revised: . 2.2. T HE CONNECTIVITY PRIOR

Original: Two nodes u, v in a graph G are connected if there is a ( u, v ) -path in G .
Revised: Two nodes u, v in a graph G are connected if there is a ( u, v ) -path in G .

Original: G is called connected if every pair of nodes are connected in G , otherwise it is disconnected .
Revised: G is called connected if every pair of nodes are connected in G , otherwise it is disconnected .

Original: Let ¯ G (cid:96) ⊆
Revised: Let ¯ G (cid:96) ⊆

Original: G be a connected subgraph where every node is labeled (cid:96) ∈ C .
Revised: G be a connected subgraph where every node is labeled (cid:96) ∈ C .

Original: Then, the image segmentation with connectivity constrains corresponds to ﬁnd a partition of G into k ( k = | C | ) connected (and disjoint) subgraphs.
Revised: Then, the image segmentation with connectivity constrains corresponds to ﬁnd a partition of G into k ( k = | C | ) connected (and disjoint) subgraphs.

Original: Enforcing connectivity constraints itself is proven to be NP -hard in (Nowozin & Lampert, 2009).
Revised: Enforcing connectivity constraints itself is proven to be NP -hard in (Nowozin & Lampert, 2009).

Original: 2.2. T HE (cid:96) 0 REGION FUSION BASED HEURISTIC
Revised: 2.2. T HE (cid:96) 0 REGION FUSION BASED HEURISTIC

Original: Given a graph G ( V, E ) , let y i be the information (either RGB or features from any DCNN) of node i , and w i be its estimated value, the discrete Potts model (Potts, 1952) has the following form:
Revised: Given a graph G ( V, E ) , let y i be the information (either RGB or features from any DCNN) of node i , and w i be its estimated value, the discrete Potts model (Potts, 1952) has the following form:

Original: [Equation]
Revised: [Equation]

Original: where λ is the regularization parameter.
Revised: where λ is the regularization parameter.

Original: Here, the ﬁrst term is the data ﬁtting and the second is the regularization term.
Revised: Here, the ﬁrst term is the data ﬁtting and the second is the regularization term.

Original: We recall that the (cid:96) 0 norm of a vector gives its number of nonzero entries.
Revised: We recall that the (cid:96) 0 norm of a vector gives its number of nonzero entries.

Original: In this paper, we introduce an iterative scribble based region fusion heuristic algorithm (which we call (cid:96) 0 H ) with the “class” and “region” ID for each node.
Revised: In this paper, we introduce an iterative scribble based region fusion heuristic algorithm (which we call (cid:96) 0 H ) with the “class” and “region” ID for each node.

Original: In the beginning, the nodes covered by the same scribble are grouped together and labeled with the same IDs, while all other nodes are unlabeled and in their individual group.
Revised: In the beginning, the nodes covered by the same scribble are grouped together and labeled with the same IDs, while all other nodes are unlabeled and in their individual group.

Original: Note that different regions can share the same class ID.
Revised: Note that different regions can share the same class ID.

Original: Then, the algorithm iterates over each group (outer loop) and its neighbors (inner loop) and decide whether to merge the neighbors or not, by checking their region IDs.
Revised: Then, the algorithm iterates over each group (outer loop) and its neighbors (inner loop) and decide whether to merge the neighbors or not, by checking their region IDs.

Original: If both groups have region IDs and are different, they cannot be merged.
Revised: If both groups have region IDs and are different, they cannot be merged.

Original: In all other cases, i.e., if both groups have no region ID or only one has it, the following merging criteria (Nguyen & Brown, 2015) are checked:
Revised: In all other cases, i.e., if both groups have no region ID or only one has it, the following merging criteria (Nguyen & Brown, 2015) are checked:

Original: [Equation]
Revised: [Equation]

Original: where σ i denotes the number of pixels in group i , Y i the mean of image information (e.g., RGB color) of group i , and γ ij denotes the number of neighboring pixels between groups i and j .
Revised: where σ i denotes the number of pixels in group i , Y i the mean of image information (e.g., RGB color) of group i , and γ ij denotes the number of neighboring pixels between groups i and j .

Original: Here, β is the regularization parameter, and it increases over the outer loop number.
Revised: Here, β is the regularization parameter, and it increases over the outer loop number.

Original: If (2) is satisﬁed, two groups are merged, and their labels are updated according to the following rule.
Revised: If (2) is satisﬁed, two groups are merged, and their labels are updated according to the following rule.

Original: If both groups have no “region” ID, the merged group still have none, hence unlabeled.
Revised: If both groups have no “region” ID, the merged group still have none, hence unlabeled.

Original: If only one group has region ID, the merged group inherits the label, hence labeled.
Revised: If only one group has region ID, the merged group inherits the label, hence labeled.

Original: After one outer loop over all groups, β will increase and follows the exponential growing strategy of (Nguyen & Brown, 2015), i.e., β = ( iter 50 ) 2 . 2 ∗ η , where “iter” is the current number of outer loop and η is the growing parameter.
Revised: After one outer loop over all groups, β will increase and follows the exponential growing strategy of (Nguyen & Brown, 2015), i.e., β = ( iter 50 ) 2 . 2 ∗ η , where “iter” is the current number of outer loop and η is the growing parameter.

Original: The procedure goes on until all groups are labeled, and the complexity of (cid:96) 

Revised: The procedure goes on until all groups are labeled, and the complexity of (cid:96)

Original: H is O ( n ) , where n is the number of nodes.
Revised: H is O ( n ) , where n is the number of nodes.

Original: Note that the above algorithm is approximate to problem (1), and connectivity of each region is enforced at every step.
Revised: Note that the above algorithm is approximate to problem (1), and connectivity of each region is enforced at every step.

Original: Given desired scribbles, (cid:96) 

Revised: Given desired scribbles, (cid:96)

Original: H is able to generate panoptic segmentations (and also semantic segmentations).
Revised: H is able to generate panoptic segmentations (and also semantic segmentations).

Original: Also note that the class ID does not play any role in the algorithm, it inherits from the scribble and propagates with the region ID.
Revised: Also note that the class ID does not play any role in the algorithm, it inherits from the scribble and propagates with the region ID.

Original: Hence, this algorithm is class-agnostic.
Revised: Hence, this algorithm is class-agnostic.

Original: 2.2. T HE ILP FORMULATION WITH CONNECTIVITY CONSTRAINTS
Revised: 2.2. T HE ILP FORMULATION WITH CONNECTIVITY CONSTRAINTS

Original: The MRF with pairwise data term can be formulated as the following ILP:
Revised: The MRF with pairwise data term can be formulated as the following ILP:

Original: [Equation]
Revised: [Equation]

Original: where c (cid:96)i denotes the unary data term for assigning class label (cid:96) to node i (hence class-aware), d ij the simpliﬁed pairwise term for assigning i , j different labels, and λ is the regularization parameter.
Revised: where c (cid:96)i denotes the unary data term for assigning class label (cid:96) to node i (hence class-aware), d ij the simpliﬁed pairwise term for assigning i , j different labels, and λ is the regularization parameter.

Original: Constraint (3a) enforces that to each node is assigned exactly one label, i.e., x (cid:96) i
Revised: Constraint (3a) enforces that to each node is assigned exactly one label, i.e., x (cid:96) i

Original: = 

Revised: =

Original: if and only if node i is labeled (cid:96) .
Revised: if and only if node i is labeled (cid:96) .

Original: Note that the absolute term can be easily transformed into linear terms by introducing additional continuous variables.
Revised: Note that the absolute term can be easily transformed into linear terms by introducing additional continuous variables.

Original: The model above does, however, not guarantee connectivity, which is instead enforced as follows.
Revised: The model above does, however, not guarantee connectivity, which is instead enforced as follows.

Original: Connectivity constraints with root node Let r (the ﬁrst node of scribble (cid:96) ) denote the root node of subgraph ¯ G (cid:96) .
Revised: Connectivity constraints with root node Let r (the ﬁrst node of scribble (cid:96) ) denote the root node of subgraph ¯ G (cid:96) .

Original: Then, the following constraints of (Rempﬂer et al., 2016) sufﬁce to characterize the set of all connected subgraphs of label (cid:96) that contain r
Revised: Then, the following constraints of (Rempﬂer et al., 2016) sufﬁce to characterize the set of all connected subgraphs of label (cid:96) that contain r

Original: [Equation]
Revised: [Equation]

Original: where S is the vertex-separator set of { i, r } , i.e., if the removal of S from G disconnects i and r .
Revised: where S is the vertex-separator set of { i, r } , i.e., if the removal of S from G disconnects i and r .

Original: And S ( i, r ) is the collection of all vertex-separator sets of { i, r } .
Revised: And S ( i, r ) is the collection of all vertex-separator sets of { i, r } .

Original: The number of constraints (4) is exponential with respect to the number of nodes in G , hence it is not possible to include into problem (3) all of them at in once.
Revised: The number of constraints (4) is exponential with respect to the number of nodes in G , hence it is not possible to include into problem (3) all of them at in once.

Original: In practice, they are added iteratively when needed (called the cutting planes method (Kelley, 1960)).
Revised: In practice, they are added iteratively when needed (called the cutting planes method (Kelley, 1960)).

Original: In this paper, we adopt the Knearest cut generation strategy (Rempﬂer et al., 2016) to add a few of them at each iteration.
Revised: In this paper, we adopt the Knearest cut generation strategy (Rempﬂer et al., 2016) to add a few of them at each iteration.

Original: See appendix A.1 for more details.
Revised: See appendix A.1 for more details.

Original: For the simple case where the region ID coincides with the class ID, i.e., the number of regions equals that of classes, problem (3) with connectivity prior is solved as follows.
Revised: For the simple case where the region ID coincides with the class ID, i.e., the number of regions equals that of classes, problem (3) with connectivity prior is solved as follows.

Original: We ﬁrst solve (3) and check if all subgraphs G i are connected.
Revised: We ﬁrst solve (3) and check if all subgraphs G i are connected.

Original: If not, we identify disconnected components, adds constraints (4), and solve the resulting ILP again.
Revised: If not, we identify disconnected components, adds constraints (4), and solve the resulting ILP again.

Original: This procedure continues until all subgraphs are connected.
Revised: This procedure continues until all subgraphs are connected.

Original: This method ensures global optimumality if no time limit is restricted.
Revised: This method ensures global optimumality if no time limit is restricted.

Original: For the case where k regions share the same class ID, we introduce an improved formulation (we call it ILP-P) where we add k − 1 pseudo edges that “connect” all separated regions of the same class (illustrated in Fig. 2).
Revised: For the case where k regions share the same class ID, we introduce an improved formulation (we call it ILP-P) where we add k − 1 pseudo edges that “connect” all separated regions of the same class (illustrated in Fig. 2).

Original: In particular, this does not increase the number of variables and is class-aware.
Revised: In particular, this does not increase the number of variables and is class-aware.

Original: Then, the solving process follows the aforementioned framework.
Revised: Then, the solving process follows the aforementioned framework.

Original: But ILP-P is only designed for semantic segmentations, i.e., it is not region-aware.
Revised: But ILP-P is only designed for semantic segmentations, i.e., it is not region-aware.

Original: Post-processing methods are needed to separate instances within the same class for panoptic segmentation.
Revised: Post-processing methods are needed to separate instances within the same class for panoptic segmentation.

Original:  O VERALL WORKFLOW
Revised:  O VERALL WORKFLOW

Original: In this section, we describe the pre-processing steps as well as the scribble policy for our panoptic and interactive segmentation.
Revised: In this section, we describe the pre-processing steps as well as the scribble policy for our panoptic and interactive segmentation.

Original: 3. S UPERPIXELS AS DIMENSION REDUCTION
Revised: 3. S UPERPIXELS AS DIMENSION REDUCTION

Original: Superpixels have long been used for image segmentation (Stutz et al., 2017; Achanta et al., 2012), as they can greatly reduce the problem size while not sacriﬁcing much of the accuracy.
Revised: Superpixels have long been used for image segmentation (Stutz et al., 2017; Achanta et al., 2012), as they can greatly reduce the problem size while not sacriﬁcing much of the accuracy.

Original: In this paper, we adopt SEEDS (den Bergh et al., 2012) to generate superpixels on the PASCAL VOC 2012 dataset, while using a deep learning based method (Tu et al., 2018) on the more challenging Cityscapes dataset.
Revised: In this paper, we adopt SEEDS (den Bergh et al., 2012) to generate superpixels on the PASCAL VOC 2012 dataset, while using a deep learning based method (Tu et al., 2018) on the more challenging Cityscapes dataset.

Original: We then build a region adjacency graph (RAG) G ( V, E ) of the superpixels, where each superpixel forms a node (vertex) and edges connect two adjacent superpixels.
Revised: We then build a region adjacency graph (RAG) G ( V, E ) of the superpixels, where each superpixel forms a node (vertex) and edges connect two adjacent superpixels.

Original: 3. E XTRACTING IMAGE FEATURES USING SCRIBBLES
Revised: 3. E XTRACTING IMAGE FEATURES USING SCRIBBLES

Original: Our segmentation algorithms are scribble supervised, which are two folded.
Revised: Our segmentation algorithms are scribble supervised, which are two folded.

Original: On the one hand, the node labels, such as class, instance and region IDs are ﬁxed if the nodes are covered by the corresponding scribbles.
Revised: On the one hand, the node labels, such as class, instance and region IDs are ﬁxed if the nodes are covered by the corresponding scribbles.

Original: On the other hand, for the ILP algorithm, if no high level image information (i.e., probability map) exists, the scribbled superpixels will be used to extract information for the class, i.e., one can use the average of the scribbled superpixels information to represent each class.
Revised: On the other hand, for the ILP algorithm, if no high level image information (i.e., probability map) exists, the scribbled superpixels will be used to extract information for the class, i.e., one can use the average of the scribbled superpixels information to represent each class.

Original: Although the input to the algorithms can be as simple as image RGB information, one can also take advantage of the modern DCNN to extract deeper features.
Revised: Although the input to the algorithms can be as simple as image RGB information, one can also take advantage of the modern DCNN to extract deeper features.

Original: We distinguish two scenarios:
Revised: We distinguish two scenarios:

Original: • No previous training data is available – one starts annotating images in a new dataset.
Revised: • No previous training data is available – one starts annotating images in a new dataset.

Original: • Training data available – one continues annotating more images of an existing dataset
Revised: • Training data available – one continues annotating more images of an existing dataset

Original: In the former case, other than RGB, one can also adopt any base network (i.e., ResNet 101 (He et al., 2016)) pre-trained on other datasets (i.e., ImageNet) and use the output of the low level features that extracts image edges, textures, etc.
Revised: In the former case, other than RGB, one can also adopt any base network (i.e., ResNet 101 (He et al., 2016)) pre-trained on other datasets (i.e., ImageNet) and use the output of the low level features that extracts image edges, textures, etc.

Original: In the later case, one can fully utilize any modern DCNN trained on the existing dataset, and use the output of the ﬁnal layer (i.e., probability map).
Revised: In the later case, one can fully utilize any modern DCNN trained on the existing dataset, and use the output of the ﬁnal layer (i.e., probability map).

Original: Scribble generation policy For panoptic segmentation, ﬁrst of all, the scribble itself must be connected.
Revised: Scribble generation policy For panoptic segmentation, ﬁrst of all, the scribble itself must be connected.

Original: Second, one has to draw as many scribbles as there are connected regions (both “thing” and “stuff” class) presented in the image.
Revised: Second, one has to draw as many scribbles as there are connected regions (both “thing” and “stuff” class) presented in the image.

Original: For example, if an object is cut into separated regions, one has to draw a scribble on each region.
Revised: For example, if an object is cut into separated regions, one has to draw a scribble on each region.

Original: One sample image with scribbles is shown in Fig.
Revised: One sample image with scribbles is shown in Fig.

Original: 3. I NTERACTIVE SEGMENTATION USING SCRIBBLES
Revised: 3. I NTERACTIVE SEGMENTATION USING SCRIBBLES

Original: In a HITL annotation framework, it is typical that the annotator makes corrections to the current segmentation until satisfaction.
Revised: In a HITL annotation framework, it is typical that the annotator makes corrections to the current segmentation until satisfaction.

Original: In this paper, we simulate the human correction scribbles (described in Appendix A.3).
Revised: In this paper, we simulate the human correction scribbles (described in Appendix A.3).

Original: We then report analysis of the segmentation results up to 3 rounds of scribbles correction in Sec. 4.4.
Revised: We then report analysis of the segmentation results up to 3 rounds of scribbles correction in Sec. 4.4.

Original:  S EMANTIC AND P ANOPTIC S EGMENTATION E XPERIMENTS 4. E XPERIMENTAL SETUP
Revised:  S EMANTIC AND P ANOPTIC S EGMENTATION E XPERIMENTS 4. E XPERIMENTAL SETUP

Original: In this session, we conduct extensive experiments on the Pascal VOC 2012 and Cityscapes validation set.
Revised: In this session, we conduct extensive experiments on the Pascal VOC 2012 and Cityscapes validation set.

Original: In all our experiments, when we mention base network, we refer to the publicly released ResNet 101 that is pre-trained on ImageNet and COCO dataset.
Revised: In all our experiments, when we mention base network, we refer to the publicly released ResNet 101 that is pre-trained on ImageNet and COCO dataset.

Original: We adopt DeepLab V2 (Chen et al., 2018b) (without CRF as post-processing) and DRN (Yu et al., 2017) as our baseline DCNN to get the probability maps, trained on their corresponding training sets.
Revised: We adopt DeepLab V2 (Chen et al., 2018b) (without CRF as post-processing) and DRN (Yu et al., 2017) as our baseline DCNN to get the probability maps, trained on their corresponding training sets.

Original: We adopt IBM Cplex (Bliek et al., 2014) version 12 .
Revised: We adopt IBM Cplex (Bliek et al., 2014) version 12 .

Original: 8 to solve the ILP.
Revised: 8 to solve the ILP.

Original: All computational experiments are performed on a Intel(R)
Revised: All computational experiments are performed on a Intel(R)

Original: Xeon(R) CPU E 5 - 2620 v 4 machine, with 64 GB memory.
Revised: Xeon(R) CPU E 5 - 2620 v 4 machine, with 64 GB memory.

Original: We report the semantic and panoptic segmentation scores, where the pixel mean intersection over union (mIoU) is commonly used for semantic segmentation, and the panoptic quality (PQ) metric is newly introduce in (Kirillov et al., 2019b) and is a combination of segmentation quality (SQ) and recognition quality (RQ).
Revised: We report the semantic and panoptic segmentation scores, where the pixel mean intersection over union (mIoU) is commonly used for semantic segmentation, and the panoptic quality (PQ) metric is newly introduce in (Kirillov et al., 2019b) and is a combination of segmentation quality (SQ) and recognition quality (RQ).

Original: The parameters for both heuristic and ILP is described in Appendix A.4.
Revised: The parameters for both heuristic and ILP is described in Appendix A.4.

Original: 4. R ESULTS ON P ASCAL VOC
Revised: 4. R ESULTS ON P ASCAL VOC

Original: Pascal VOC 2012 has 20 “thing” classes and a single “background” class for all other classes.
Revised: Pascal VOC 2012 has 20 “thing” classes and a single “background” class for all other classes.

Original: We evaluate our algorithms on the 1449 validation images.
Revised: We evaluate our algorithms on the 1449 validation images.

Original: We ﬁrst apply (den Bergh et al., 2012) to produce around 700 superpixels, and use the public scribbles set provided by (Lin et al., 2016) as initial scribbles.
Revised: We ﬁrst apply (den Bergh et al., 2012) to produce around 700 superpixels, and use the public scribbles set provided by (Lin et al., 2016) as initial scribbles.

Original: Since some of these scribbles do not meet our policy (described in Sec. 3.2), only the semantic segmentation is reported, in terms of mIoU averaged across the 21 classes.
Revised: Since some of these scribbles do not meet our policy (described in Sec. 3.2), only the semantic segmentation is reported, in terms of mIoU averaged across the 21 classes.

Original: Visualization of 2 experiments are presented in Appendix A.5.
Revised: Visualization of 2 experiments are presented in Appendix A.5.

Original: No training data
Revised: No training data

Original: In this case, one can either use RGB or the output of lower level features of a base network as input ( y i ) to our algorithms.
Revised: In this case, one can either use RGB or the output of lower level features of a base network as input ( y i ) to our algorithms.

Original: We compare our class-agnostic heuristic ( (cid:96) 0 H ) using
Revised: We compare our class-agnostic heuristic ( (cid:96) 0 H ) using

Original: RGB or different low level features from ResNet 101 as input, against ILP-P.
Revised: RGB or different low level features from ResNet 101 as input, against ILP-P.

Original: We do not set any time or step limit for the heuristic, but a time limit of 10 seconds for ILP-P (denoted ILP-P-10).
Revised: We do not set any time or step limit for the heuristic, but a time limit of 10 seconds for ILP-P (denoted ILP-P-10).

Original: We report in Table 1 the detailed comparison, where we use the RGB, ﬁrst and third layer of ResNet 101 as input to (cid:96) 0 H , and “Dim” is the dimension of the input feature map.
Revised: We report in Table 1 the detailed comparison, where we use the RGB, ﬁrst and third layer of ResNet 101 as input to (cid:96) 0 H , and “Dim” is the dimension of the input feature map.

Original: We can see in Table 1 the advantage by incorporating lower level features maps of ResNet 101 , that improves (cid:96) 

Revised: We can see in Table 1 the advantage by incorporating lower level features maps of ResNet 101 , that improves (cid:96)

Original: H of RGB by 1 . 8% , even though it is pre-trained on completely different dataset.
Revised: H of RGB by 1 . 8% , even though it is pre-trained on completely different dataset.

Original: ILP-P adopts (cid:96) 

Revised: ILP-P adopts (cid:96)

Original: H -layer 3 as initial solution, and further increase the mIoU by 0 .
Revised: H -layer 3 as initial solution, and further increase the mIoU by 0 .

Original: With training data If training data is available, we use the probability map of DeepLab V 2 (with baseline mIoU 70 .
Revised: With training data If training data is available, we use the probability map of DeepLab V 2 (with baseline mIoU 70 .

Original: 5% ) trained on VOC tarining set as input to our algorithms, (cid:96) 

Revised: 5% ) trained on VOC training set as input to our algorithms, (cid:96)

Original: H get another huge boost of 10 . 1% to 81 .
Revised: H get another huge boost of 10 . 1% to 81 .

Original: 7% compared to using layer 3 .
Revised: 7% compared to using layer 3 .

Original: We compare ILP (3) without connectivity
Revised: We compare ILP (3) without connectivity

Original: [Table] [Table]
Revised: [Table] [Table]

Original: that is solved to optimum (MRF-prob) with (cid:96) 

Revised: that is solved to optimum (MRF-prob) with (cid:96)

Original: H and found out it is 0 .
Revised: H and found out it is 0 .

Original: 9% worse, which shows the importance of the connectivity prior.
Revised: 9% worse, which shows the importance of the connectivity prior.

Original: We then compare ILP-P with (cid:96) 

Revised: We then compare ILP-P with (cid:96)

Original: H as initial solutions ( 81 . 7% mIoU) and a time limit of 10 seconds, Table 2 suggests that ILP-P- 10 improves the baseline of
Revised: H as initial solutions ( 81 . 7% mIoU) and a time limit of 10 seconds, Table 2 suggests that ILP-P- 10 improves the baseline of

Original: DeepLab V 2 by 14 . 1% and (cid:96) 

Revised: DeepLab V 2 by 14 . 1% and (cid:96)

Original: H -prob by 2 . 9% .
Revised: H -prob by 2 . 9% .

Original: An example of visual comparison is illustrated in
Revised: An example of visual comparison is illustrated in

Original: Fig.
Revised: Fig.

Original: Since ILP (3) is class-aware and encodes pairwise term d ij , further boost on the performance can be expected, given better baseline DCNN and an edge detector.
Revised: Since ILP (3) is class-aware and encodes pairwise term d ij , further boost on the performance can be expected, given better baseline DCNN and an edge detector.

Original: 4. R ESULTS ON C ITYSCAPES
Revised: 4. R ESULTS ON C ITYSCAPES

Original: Cityscapes has 8 “thing” classes, and 11 “stuff” classes.
Revised: Cityscapes has 8 “thing” classes, and 11 “stuff” classes.

Original: While there is no public scribbles set, we simulate our own that meet the policy in Sec. 3.2.
Revised: While there is no public scribbles set, we simulate our own that meet the policy in Sec. 3.2.

Original: The procedure is described in Appendix A.2, and one can see Fig.
Revised: The procedure is described in Appendix A.2, and one can see Fig.

Original: 3 for an example.
Revised: 3 for an example.

Original: We evaluate our algorithms on the 500 validation images with the scribbles, where we ﬁrst apply (Tu et al., 2018) to produce around 2000 superpixels.
Revised: We evaluate our algorithms on the 500 validation images with the scribbles, where we ﬁrst apply (Tu et al., 2018) to produce around 2000 superpixels.

Original: No training data
Revised: No training data

Original: We use RGB and output of layer 3 of Resnet 101 as input to (cid:96) 0 H , and report in Table 3 both mIoU and PQ.
Revised: We use RGB and output of layer 3 of Resnet 101 as input to (cid:96) 0 H , and report in Table 3 both mIoU and PQ.

Original: We also compare (cid:96) 

Revised: We also compare (cid:96)

Original: H to one recent weekly supervised learning method (Li et al., 2018).
Revised: H to one recent weakly supervised learning method (Li et al., 2018).

Original: It uses a more powerful PSPNet (Zhao et al., 2017) supervised by bounding boxes and image tags, but requires end to end training.
Revised: It uses a more powerful PSPNet (Zhao et al., 2017) supervised by bounding boxes and image tags, but requires end to end training.

Original: (cid:96) 

Revised: (cid:96)

Original: H shows superior results on both semantic and panoptic segmentation, both around 10% boost compared to (Li et al., 2018).
Revised: H shows superior results on both semantic and panoptic segmentation, both around 10% boost compared to (Li et al., 2018).

Original: Note that it is not a fair comparison since ours requires scribbles while (Li et al., 2018) does not at inference time.
Revised: Note that it is not a fair comparison since ours requires scribbles while (Li et al., 2018) does not at inference time.

Original: With training data We use the public full-supervised (trained on Cityscapes) DRN (Yu et al., 2017) as our baseline ( 71 . 4% mIoU), and run (cid:96) 

Revised: With training data We use the public full-supervised (trained on Cityscapes) DRN (Yu et al., 2017) as our baseline ( 71 . 4% mIoU), and run (cid:96)

Original: H and ILP-P using its probability map.
Revised: H and ILP-P using its probability map.

Original: Tableshows (cid:96) 

Revised: Tableshows (cid:96)

Original: H -prob improves the baseline by 4 . 4% , and by 1 . 5% compared to (cid:96) 

Revised: H -prob improves the baseline by 4 . 4% , and by 1 . 5% compared to (cid:96)

Original: H -layer 3 .
Revised: H -layer 3 .

Original: Besides,
Revised: Besides,

Original: PQ also increases from 49 .
Revised: PQ also increases from 49 .

Original: 6% to 51 .
Revised: 6% to 51 .

Original: Because of the rich class and instance information (more than 15 classes per image) of Cityscapes and also that we use around 2000 superpixels (around 30 k
Revised: Because of the rich class and instance information (more than 15 classes per image) of Cityscapes and also that we use around 2000 superpixels (around 30 k

Original: [Table]
Revised: [Table]

Original: binary variables), ILP struggles to ﬁnd any better solution within the time limit of 10 seconds.
Revised: binary variables), ILP struggles to ﬁnd any better solution within the time limit of 10 seconds.

Original: The score remains unchanged compared to (cid:96) 0 H , and hence is not shown in the table.
Revised: The score remains unchanged compared to (cid:96) 0 H , and hence is not shown in the table.

Original: More visualization examples can be seen in Appendix A.6.
Revised: More visualization examples can be seen in Appendix A.6.

Original: 4. R ESULTS ON INTERACTIVE SEGMENTATION
Revised: 4. R ESULTS ON INTERACTIVE SEGMENTATION

Original: As described in Appendix A.3, we simulate 3 rounds of scribbles ( 1 scribble per image per round) to correct the largest error of previous segmentation.
Revised: As described in Appendix A.3, we simulate 3 rounds of scribbles ( 1 scribble per image per round) to correct the largest error of previous segmentation.

Original: We report their mIoU, compared to the baseline DCNN (DeepLab V 2 ) and  state-of-the-art (SOTA) deep learning approach (DeepLab V 3+ with 84 .
Revised: We report their mIoU, compared to the baseline DCNN (DeepLab V 2 ) and current reported state-of-the-art (SOTA) deep learning approach (DeepLab V 3+ with 84 .

Original: (Chen et al., 2018c) on VOC validation set.
Revised: (Chen et al., 2018c) on VOC validation set.

Original: Results show that our algorithms beneﬁt from addition scribbles and the performance increases signiﬁcantly with few iterations.
Revised: Results show that our algorithms beneﬁt from addition scribbles and the performance increases signiﬁcantly.

Original: The best result is reported by ILP-P- 10 at iteration 3 ( 90 . 6% ), a 6 . 0% mIoU gain with just 3 correction scribbles.
Revised: The best result is reported by ILP-P- 10 at iteration 3 ( 90 . 6% ), a 6 . 0% mIoU gain with just 3 correction scribbles.

Original:  C ONCLUSIONS AND FUTURE WORK
Revised:  C ONCLUSIONS AND FUTURE WORK

Original: In this paper, we proposed two interactive multi-label graph based algorithms with connectivity prior.
Revised: In this paper, we proposed two interactive multi-label graph based algorithms with connectivity prior.

Original: With DCNN’s features and initial scribbles as input, both achieve competitive semantic (and panoptic) segmentation results on VOC and Cityscapes validation set.
Revised: With DCNN’s features and initial scribbles as input, both achieve competitive semantic (and panoptic) segmentation results on VOC and Cityscapes validation set.

Original: The interactive approach further boost the performance with little overhead of correction scribbles.
Revised: The interactive approach further boost the performance with little overhead of correction scribbles.

Original: It would be interesting to investigate our approach in a weakly supervised setting, where connectivity constraints are imposed on the outputs of a deep network to leverage unlabeled data.
Revised: It would be interesting to investigate our approach in a weakly supervised setting, where connectivity constraints are imposed on the outputs of a deep network to leverage unlabeled data.

Original: Training with such discrete high-order constraints can be explored via ADM (Alternating Direction Method) schemes, for instance, in way similar to Marin et al.
Revised: Training with such discrete high-order constraints can be explored via ADM (Alternating Direction Method) schemes, for instance, in way similar to Marin et al.

Original: (2019), which showed promising performances of imposing discrete MRF on the outputs of convolutional networks.
Revised: (2019), which showed promising performances of imposing discrete MRF on the outputs of convolutional networks.

Original: Finally, note that all the reported performance are upper bounded by the accuracy of superpixels.
Revised: Finally, note that all the reported performance are upper bounded by the accuracy of superpixels.

Original: Hence, using better superpixel algorithms or increasing its number may further inﬂuence the performance.
Revised: Hence, using better superpixel algorithms or increasing its number may further inﬂuence the performance.

Original: A A PPENDIX A. K - NEAREST CUT GENERATION ALGORITHM
Revised: A A PPENDIX A. K - NEAREST CUT GENERATION ALGORITHM

Original: We restrict ourselves to label (cid:96) only, and the approach described below will be repeated for the other labels.
Revised: We restrict ourselves to label (cid:96) only, and the approach described below will be repeated for the other labels.

Original: Upon solving problem (3), we denote those with x (cid:96) = 1 as active nodes, and we look for all connected components in G (cid:96) .
Revised: Upon solving problem (3), we denote those with x (cid:96) = 1 as active nodes, and we look for all connected components in G (cid:96) .

Original: We then iterate over all active components pair ( H, r ) , and use the following strategy to select K constraints (4) each time.
Revised: We then iterate over all active components pair ( H, r ) , and use the following strategy to select K constraints (4) each time.

Original: Precisely, we run a breath-ﬁrst search algorithm starting from H to collect K disjoint sets S m ( m = 1 , . . .
Revised: Precisely, we run a breath-ﬁrst search algorithm starting from H to collect K disjoint sets S m ( m = 1 , . . .

Original: , K ) where S m contains all the inactive nodes with distance m to H .
Revised: , K ) where S m contains all the inactive nodes with distance m to H .

Original: The search terminates if K equals the number of nodes in H or if another active node is reached.
Revised: The search terminates if K equals the number of nodes in H or if another active node is reached.

Original: The idea is illustrated in Fig.
Revised: The idea is illustrated in Fig.

Original: 5, where active nodes are shown in black and r denotes the root node.
Revised: 5, where active nodes are shown in black and r denotes the root node.

Original: The two separator sets are marked in red and blue.
Revised: The two separator sets are marked in red and blue.

Original: Here K = 2 , because it reaches the number of nodes in H .
Revised: Here K = 2 , because it reaches the number of nodes in H .

Original: The K -Nearest strategy is reported in (Rempﬂer et al., 2016) to be one of the most successful (among ﬁve) in terms of solved instances and computational efﬁciency.
Revised: The K -Nearest strategy is reported in (Rempﬂer et al., 2016) to be one of the most successful (among ﬁve) in terms of solved instances and computational efﬁciency.

Original: A. S IMULATING SCRIBBLE GENERATIONS
Revised: A. S IMULATING SCRIBBLE GENERATIONS

Original: Since there is no public scribble set for Cityscapes, we hack the ground truth instance segment and apply erosion and skeleton algorithms to simulate scribbles that meet our policy in Sec. 3.2.
Revised: Since there is no public scribble set for Cityscapes, we hack the ground truth instance segment and apply erosion and skeleton algorithms to simulate scribbles that meet our policy in Sec. 3.2.

Original: We generated artiﬁcial scribbles for both training and validation sets of Cityscapes.
Revised: We generated artiﬁcial scribbles for both training and validation sets of Cityscapes.

Original: We adopt kernel size adapted erosion and skeleton method and apply them on the instance-level ground truth masks of Cityscapes.
Revised: We adopt kernel size adapted erosion and skeleton method and apply them on the instance-level ground truth masks of Cityscapes.

Original: Therefore, the format of the scribbles are also the same as Cityscapes, which isbit grayscale PNG images with both class and instance labels.
Revised: Therefore, the format of the scribbles are also the same as Cityscapes, which isbit grayscale PNG images with both class and instance labels.

Original: The code that generates the scribbles will be open-source.
Revised: The code that generates the scribbles will be open-source.

Original: All scribbles can be downloaded online 1 .
Revised: All scribbles can be downloaded online 1 .

Original: A. S IMULATING CORRECTION SCRIBBLES
Revised: A. S IMULATING CORRECTION SCRIBBLES

Original: For VOC 2012, we simulate only one scribble per image per iteration.
Revised: For VOC 2012, we simulate only one scribble per image per iteration.

Original: Given the VOC ground truth and current segmentation mask, we ﬁrst identify the largest error region which is connected and mislabelled.
Revised: Given the VOC ground truth and current segmentation mask, we ﬁrst identify the largest error region which is connected and mislabelled.

Original: To simulate scribble from the error region, we ﬁrst apply opening morphological operation to smooth the region.
Revised: To simulate scribble from the error region, we ﬁrst apply opening morphological operation to smooth the region.

Original: Then, we use skeletonization to shrink the region to 1 pixel wide without breaking the connectivity.
Revised: Then, we use skeletonization to shrink the region to 1 pixel wide without breaking the connectivity.

Original: We can then get the scribble by cutting branches in the skeleton tree.
Revised: We can then get the scribble by cutting branches in the skeleton tree.

Original: One can see Fig.
Revised: One can see Fig.

Original: 6 for an illustration of generating correction scribbles.
Revised: 6 for an illustration of generating correction scribbles.

Original: The code will also be open-source soon.
Revised: The code will also be open-source soon.

Original: A. P ARAMETERS SETTING FOR THE EXPERIMENTS
Revised: A. P ARAMETERS SETTING FOR THE EXPERIMENTS

Original: Parameter y
Revised: Parameter y

Original: i is the information contained in pixel i , which could come from a vector of RGB channels, layer 1 or layer 3 feature of ResNet 101 , or from the probability map of a trained DCNN.
Revised: i is the information contained in pixel i , which could come from a vector of RGB channels, layer 1 or layer 3 feature of ResNet 101 , or from the probability map of a trained DCNN.

Original: For VOC 2012, the probability map comes from a trained DeepLab V 2 .
Revised: For VOC 2012, the probability map comes from a trained DeepLab V 2 .

Original: For Cityscapes, we use
Revised: For Cityscapes, we use

Original: DRN to get the probability map inference.
Revised: DRN to get the probability map inference.

Original: For (cid:96) 

Revised: For (cid:96)

Original: H , after several trials, the growing parameter η is set to 0 .
Revised: H , after several trials, the growing parameter η is set to 0 .

Original: 1 , 20 , 100 and 0 .
Revised: 1 , 20 , 100 and 0 .

Original: 3 for RGB, layer 1 , layer 3 and probability map of both VOC 2012 and cityscapes.
Revised: 3 for RGB, layer 1 , layer 3 and probability map of both VOC 2012 and cityscapes.

Original: For ILP, when training data is available, we can use the probability map p i and c (cid:96)i =
Revised: For ILP, when training data is available, we can use the probability map p i and c (cid:96)i =

Original: (cid:13)(cid:13) 1 (cid:96) − p
Revised: (cid:13)(cid:13) 1 (cid:96) − p

Original: i (cid:13)(cid:13) 2 , where 1 (cid:96) is an k ( k being the number of classes) dimensional vector with (cid:13)(cid:13) 1 (cid:96) (cid:13)(cid:13) 1 = 1 and the (cid:96) ’s position equals 1 .
Revised: i (cid:13)(cid:13) 2 , where 1 (cid:96) is an k ( k being the number of classes) dimensional vector with (cid:13)(cid:13) 1 (cid:96) (cid:13)(cid:13) 1 = 1 and the (cid:96) ’s position equals 1 .

Original: When there is no training data, we compute the average of the nodes information ( y i ) covered by scribbles of the same class (i.e., class (cid:96) ), and use this to represent class (cid:96) (denote as
Revised: When there is no training data, we compute the average of the nodes information ( y i ) covered by scribbles of the same class (i.e., class (cid:96) ), and use this to represent class (cid:96) (denote as

Original: Y (cid:96) ).
Revised: Y (cid:96) ).

Original: Then c (cid:96)i = (cid:107) y
Revised: Then c (cid:96)i = (cid:107) y

Original: i − Y (cid:96) (cid:107) 2 .
Revised: i − Y (cid:96) (cid:107) 2 .

Original: The regularization parameter λ for all the ILP is set to 100 , and the pairwise term d ij = e −(cid:107)
Revised: The regularization parameter λ for all the ILP is set to 100 , and the pairwise term d ij = e −(cid:107)

Original: y i − y j (cid:107) 2 .
Revised: y i − y j (cid:107) 2 .

Original: A. V ISUALIZATION OF EXPERIMENTS ON P ASCAL VOC
Revised: A. V ISUALIZATION OF EXPERIMENTS ON P ASCAL VOC

Original: In this section, we present 2 experiments on Pascal VOC using our optimization algorithms, shown in Fig.
Revised: In this section, we present 2 experiments on Pascal VOC using our optimization algorithms, shown in Fig.

Original: The ﬁgures are displayed in order of Table 4, where ILP-P-prob adopts the result of (cid:96) 

Revised: The ﬁgures are displayed in order of Table 4, where ILP-P-prob adopts the result of (cid:96)

Original: H -prob as its initial solution.
Revised: H -prob as its initial solution.

Original: [Table]
Revised: [Table]

Original: In both experiments, we could see that using probability map is better than using low level feature maps.
Revised: In both experiments, we could see that using probability map is better than using low level feature maps.

Original: ILP-P-prob improves the result in the ﬁrst one, while veriﬁes the optimally in the second experiment, i.e., the initial solution of (cid:96) 

Revised: ILP-P-prob improves the result in the ﬁrst one, while veriﬁes the optimally in the second experiment, i.e., the initial solution of (cid:96)

Original: H -prob is already the global optimization for ILP-P-prob.
Revised: H -prob is already the global optimization for ILP-P-prob.

Original: We observe that ILP-P-prob-5 has the same solution as MRF-prob in the ﬁrst experiment, while improves MRF-prob in the second one.
Revised: We observe that ILP-P-prob-5 has the same solution as MRF-prob in the ﬁrst experiment, while improves MRF-prob in the second one.

Original: A. V ISUALIZATION OF EXPERIMENTS ON C ITYSCAPES
Revised: A. V ISUALIZATION OF EXPERIMENTS ON C ITYSCAPES

Original: In this section, we present 3 experiments on Cityscapes using our optimization algorithms.
Revised: In this section, we present 3 experiments on Cityscapes using our optimization algorithms.

Original: The ﬁgures are displayed in order of Table 5, where “SEEG” and “Panoptic” denotes semantic and panoptic segmentation, respectively.
Revised: The ﬁgures are displayed in order of Table 5, where “SEEG” and “Panoptic” denotes semantic and panoptic segmentation, respectively.

Original: Supervised by scribbles, both semantic and panoptic segmentation have already good shape and sharp boundaries, even using just low level features of DCNN (pre-trained on other completely different datasets).
Revised: Supervised by scribbles, both semantic and panoptic segmentation have already good shape and sharp boundaries, even using just low level features of DCNN (pre-trained on other completely different datasets).

Original: The probability map from Cityscapes trained DCNN further improves the performance.
Revised: The probability map from Cityscapes trained DCNN further improves the performance.

Original: But they are still limited by the quality of superpixels, especially for small and thin objects
Revised: But they are still limited by the quality of superpixels, especially for small and thin objects

Original: [Table] [Table]
Revised: [Table] [Table]