Original: A MULTI - TASK U- NET FOR SEGMENTATION WITH LAZY LABELS
Revised: A MULTI - TASK U- NET FOR SEGMENTATION WITH LAZY LABELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The need for labour intensive pixel-wise annotation is a major limitation of many fully supervised learning methods for image segmentation.
Revised: The need for labour intensive pixel-wise annotation is a major limitation of many fully supervised learning methods for image segmentation.

Original: In this paper, we propose a deep convolutional neural network for multi-class segmentation that circumvents this problem by being trainable on coarse data labels combined with only a very small number of images with pixel-wise annotations.
Revised: In this paper, we propose a deep convolutional neural network for multi-class segmentation that circumvents this problem by being trainable on coarse data labels combined with only a very small number of images with pixel-wise annotations.

Original: We call this new labelling strategy ‘lazy’ labels.
Revised: We call this new labelling strategy ‘lazy’ labels.

Original: Image segmentation is then stratiﬁed into three connected tasks: rough detection of class instances, separation of wrongly connected objects without a clear boundary, and pixel-wise segmentation to ﬁnd the accurate boundaries of each object.
Revised: Image segmentation is then stratiﬁed into three connected tasks: rough detection of class instances, separation of wrongly connected objects without a clear boundary, and pixel-wise segmentation to ﬁnd the accurate boundaries of each object.

Original: These problems are integrated into a multitask learning framework and the model is trained end-to-end in a semi-supervised fashion.
Revised: These problems are integrated into a multitask learning framework and the model is trained end-to-end in a semi-supervised fashion.

Original: The method is demonstrated on two segmentation datasets, including food microscopy images and histology images of tissues respectively.
Revised: The method is demonstrated on two segmentation datasets, including food microscopy images and histology images of tissues respectively.

Original: We show that the model gives accurate segmentation results even if exact boundary labels are missing for a majority of the annotated data.
Revised: We show that the model gives accurate segmentation results even if exact boundary labels are missing for a majority of the annotated data.

Original: This allows more ﬂexibility and efﬁciency for training deep neural networks that are data hungry in a practical setting where manual annotation is expensive, by collecting more lazy (rough) annotations than precisely segmented images.
Revised: This allows more ﬂexibility and efﬁciency for training deep neural networks that are data hungry in a practical setting where manual annotation is expensive, by collecting more lazy (rough) annotations than precisely segmented images.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image segmentation has been an active research ﬁeld in the past decades.
Revised: Image segmentation has been an active research ﬁeld in the past decades.

Original: Deep learning approaches play an increasingly important role and have become state-of-the-art in various segmentation tasks (Huang et al., 2018; Khoreva et al., 2017; Tsutsui et al., 2018; Ghosh et al., 2018; Litjens et al., 2017).
Revised: Deep learning approaches play an increasingly important role and have become state-of-the-art in various segmentation tasks (Huang et al., 2018; Khoreva et al., 2017; Tsutsui et al., 2018; Ghosh et al., 2018; Litjens et al., 2017).

Original: Though fully supervised segmentation neural networks have a shown great success, one of their most challenging issues is the need for pixel-level annotations to train them.
Revised: Though fully supervised segmentation neural networks have a shown great success, one of their most challenging issues is the need for pixel-level annotations to train them.

Original: Obtaining such annotations usually requires a great amount of manual work and is therefore expensive.
Revised: Obtaining such annotations usually requires a great amount of manual work and is therefore expensive.

Original: In this paper, we propose a multi-class and multi-instance segmentation approach that we split into three relevant tasks: detection, separation and segmentation (cf.
Revised: In this paper, we propose a multi-class and multi-instance segmentation approach that we split into three relevant tasks: detection, separation and segmentation (cf.

Original: Figure 1).
Revised: Figure 1).

Original: Doing so, we obtain a semi-supervised learning approach that is trained with so-called "lazy" labels, that is a lot of coarse annotations of class instances together with only a few pixel-wise annotated images that can be obtained from the coarse labels in a semi-automated way.
Revised: Doing so, we obtain a semi-supervised learning approach that is trained with so-called "lazy" labels, that is a lot of coarse annotations of class instances together with only a few pixel-wise annotated images that can be obtained from the coarse labels in a semi-automated way.

Original: In the following, we will refer to weak (resp.
Revised: In the following, we will refer to weak (resp.

Original: strong) annotations for coarse (resp.
Revised: strong) annotations for coarse (resp.

Original: accurate) labels and denote them as WL and SL.
Revised: accurate) labels and denote them as WL and SL.

Original: Task 1 detects and classiﬁes each object and roughly determines its region through an undersegmentation mask.
Revised: Task 1 detects and classiﬁes each object and roughly determines its region through an undersegmentation mask.

Original: Instance counting can be obtained as a by-product of this task.
Revised: Instance counting can be obtained as a by-product of this task.

Original: As the main objective is instance detection, exact labels for the whole object or its boundary are not necessary at this stage.
Revised: As the main objective is instance detection, exact labels for the whole object or its boundary are not necessary at this stage.

Original: We use instead weakly annotated images in which a rough region inside each object is marked, cf.
Revised: We use instead weakly annotated images in which a rough region inside each object is marked, cf.

Original: the most top left part of Figure 1.
Revised: the most top left part of Figure 1.

Original: For segmentation problems with a dense population of instances, such as the food components (see e.g., Figure 1), cells (Guerrero-Pena et al., 2018;
Revised: For segmentation problems with a dense population of instances, such as the food components (see e.g., Figure 1), cells (Guerrero-Pena et al., 2018;

Original: Ronneberger et al., 2015), glandular tissue, or people in a crowd (Wang et al., 2018b), separating objects sharing a common boundary is a well known challenge.
Revised: Ronneberger et al., 2015), glandular tissue, or people in a crowd (Wang et al., 2018b), separating objects sharing a common boundary is a well known challenge.

Original: We can optionally perform a second task ( Task 2 ) that focuses on the separation of instances that are connected without a clear boundary dividing them.
Revised: We can optionally perform a second task ( Task 2 ) that focuses on the separation of instances that are connected without a clear boundary dividing them.

Original: Also for this task we rely on WL to reduce the burden of manual annotations: touching interfaces are speciﬁed with rough scribbles, cf.
Revised: Also for this task we rely on WL to reduce the burden of manual annotations: touching interfaces are speciﬁed with rough scribbles, cf.

Original: top left part of Figure 1.
Revised: top left part of Figure 1.

Original: Task 3 ﬁnally tackles the pixel-wise classiﬁcation of the instances.
Revised: Task 3 ﬁnally tackles the pixel-wise classiﬁcation of the instances.

Original: It requires strong annotations that are accurate up to the boundaries of the objects.
Revised: It requires strong annotations that are accurate up to the boundaries of the objects.

Original: Thanks to the information brought by the weak annotations, we here just
Revised: Thanks to the information brought by the weak annotations, we here just

Original: [Figure]
Revised: [Figure]

Original: need a very small set of accurate segmentation masks,
Revised: need a very small set of accurate segmentation masks,

Original: cf. bottom left part of Figure 1.
Revised: cf. bottom left part of Figure 1.

Original: To that end, we propose to reﬁne some of the coarse labels resulting from task 1 using a semi-automatic segmentation method which requires additional manual intervention.
Revised: To that end, we propose to reﬁne some of the coarse labels resulting from task 1 using a semi-automatic segmentation method which requires additional manual intervention.

Original: The three tasks are handled by a single deep neural network and are jointly optimized.
Revised: The three tasks are handled by a single deep neural network and are jointly optimized.

Original: The network architecture is inspired by the widely used segmentation network named U-net (Ronneberger et al., 2015).
Revised: The network architecture is inspired by the widely used segmentation network named U-net (Ronneberger et al., 2015).

Original: With all three tasks sharing the same contracting path, we introduce a new multi-task block for the expansive path.
Revised: With all three tasks sharing the same contracting path, we introduce a new multi-task block for the expansive path.

Original: The network has three outputs and is fed with a combination of WL and
Revised: The network has three outputs and is fed with a combination of WL and

Original: SL described above.
Revised: SL described above.

Original: Since weakly and strongly annotated training data is shared between the tasks, part of the annotations are missing, especially for task 3.
Revised: Since weakly and strongly annotated training data is shared between the tasks, part of the annotations are missing, especially for task 3.

Original: To accommodate for this we introduce a weighted loss function over the samples.
Revised: To accommodate for this we introduce a weighted loss function over the samples.

Original: Accurate segmentation labels for training are usually not easy to obtain, however, with our approach we demonstrate that exact labels for the whole training set are not needed for good segmentation learning performance.
Revised: Accurate segmentation labels for training are usually not easy to obtain, however, with our approach we demonstrate that exact labels for the whole training set are not needed for good segmentation learning performance.

Original: We evaluate the performance of the proposed approach on two applications, namely for the segmentation of SEM images of food microstructure and stained histology images of glandular tissues.
Revised: We evaluate the performance of the proposed approach on two applications, namely for the segmentation of SEM images of food microstructure and stained histology images of glandular tissues.

Original: In summary, our contributions are as follows.
Revised: In summary, our contributions are as follows.

Original: We propose a decomposition of the segmentation problems into three tasks and a corresponding user friendly labeling strategy.
Revised: We propose a decomposition of the segmentation problems into three tasks and a corresponding user friendly labeling strategy.

Original: We develop a multi-task learning framework that learns directly from the manual labels and is trained end-to-end.
Revised: We develop a multi-task learning framework that learns directly from the manual labels and is trained end-to-end.

Original: Our approach outperforms state-of-the-art weakly supervised methods such as Khoreva et al.
Revised: Our approach outperforms state-of-the-art weakly supervised methods such as Khoreva et al.

Original: The network predicts segmentation mask as well as the object inner regions and touching object interfaces.
Revised: The network predicts segmentation mask as well as the object inner regions and touching object interfaces.

Original: Touching objects are effectively disconnected as a side product.
Revised: Touching objects are effectively disconnected as a side product.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In image segmentation problems, one needs to classify an image at pixel level.
Revised: In image segmentation problems, one needs to classify an image at pixel level.

Original: It is a vast topic with a diversity of algorithms being developed, including traditional unsupervised methods like k -means that splits the image into homogeneous regions according to image low level features, curve evolution based methods like snakes (Caselles et al., 1997), graph-cut based methods like Grabcut (Rother et al., 2004), just to name a few.
Revised: It is a vast topic with a diversity of algorithms being developed, including traditional unsupervised methods like k -means that splits the image into homogeneous regions according to image low level features, curve evolution based methods like snakes (Caselles et al., 1997), graph-cut based methods like Grabcut (Rother et al., 2004), just to name a few.

Original: Interactive approaches such as snakes or Grabcut enable getting involved users’ knowledge by means of initializing regions or putting constraints on the segmentation results.
Revised: Interactive approaches such as snakes or Grabcut enable getting involved users’ knowledge by means of initializing regions or putting constraints on the segmentation results.

Original: Deep convolutional neural network (DCNN) approaches have been developed for segmenting complex images, especially in the semantic setting.
Revised: Deep convolutional neural network (DCNN) approaches have been developed for segmenting complex images, especially in the semantic setting.

Original: In particular, fully convolutional networks (FCN) (Long et al., 2015) replace the last few fully connected layers of a conventional classiﬁcation network by up-sampling layers and convolutional layers, to preserve spatial information.
Revised: In particular, fully convolutional networks (FCN) (Long et al., 2015) replace the last few fully connected layers of a conventional classiﬁcation network by up-sampling layers and convolutional layers, to preserve spatial information.

Original: FCNs have many variants for semantic segmentation.
Revised: FCNs have many variants for semantic segmentation.

Original: The DeepLab (Chen et al., 2018) uses a technique called atrous convolution to handle spatial information together with a fully connected conditional random ﬁeld (CRF)
Revised: The DeepLab (Chen et al., 2018) uses a technique called atrous convolution to handle spatial information together with a fully connected conditional random ﬁeld (CRF)

Original: (Chen et al., 2014) for reﬁning the segmentation results.
Revised: (Chen et al., 2014) for reﬁning the segmentation results.

Original: Fully connected CRF can be used as post-processing or can be integrated into the network architecture, allowing for end-to-end training (Zheng et al., 2015).
Revised: Fully connected CRF can be used as post-processing or can be integrated into the network architecture, allowing for end-to-end training (Zheng et al., 2015).

Original: Other types of FCNs are deep encoder-decoder networks (Badrinarayanan et al., 2017; Ronneberger et al., 2015).
Revised: Other types of FCNs are deep encoder-decoder networks (Badrinarayanan et al., 2017; Ronneberger et al., 2015).

Original: They have multiple up-sampling layers for better localizing boundary details.
Revised: They have multiple up-sampling layers for better localizing boundary details.

Original: One of the most well-known models is the U-net (Ronneberger et al., 2015).
Revised: One of the most well-known models is the U-net (Ronneberger et al., 2015).

Original: It is a fully convolutional network made of a contracting path, which brings the input images into very low resolution features with a sequence of down-sampling, and an expansive path that has an equal amount of up-sampling layers.
Revised: It is a fully convolutional network made of a contracting path, which brings the input images into very low resolution features with a sequence of down-sampling, and an expansive path that has an equal amount of up-sampling layers.

Original: The higher resolution feature maps on the contracting path are merged with up-sampled layers via long skip connections to recover boundary information after down-sampling.
Revised: The higher resolution feature maps on the contracting path are merged with up-sampled layers via long skip connections to recover boundary information after down-sampling.

Original: Weakly/semi-supervised segmentation learning.
Revised: Weakly/semi-supervised segmentation learning.

Original: Motivated by the heavy cost of pixel-level annotation needed for fully supervised learning, there has been a growing interest in weakly supervised learning with coarse annotations for semantic segmentation in the last years.
Revised: Motivated by the heavy cost of pixel-level annotation needed for fully supervised learning, there has been a growing interest in weakly supervised learning with coarse annotations for semantic segmentation in the last years.

Original: Common weak annotations include image-level labels (Huang et al., 2018;
Revised: Common weak annotations include image-level labels (Huang et al., 2018;

Original: Papandreou et al., 2015; Lee et al., 2019), bounding boxes(Khoreva et al., 2017), scribbles (Lin et al., 2016) and points (Bearman et al., 2016).
Revised: Papandreou et al., 2015; Lee et al., 2019), bounding boxes(Khoreva et al., 2017), scribbles (Lin et al., 2016) and points (Bearman et al., 2016).

Original: Most weakly supervised deep learning methods for segmentation are built on top of a classiﬁcation network.
Revised: Most weakly supervised deep learning methods for segmentation are built on top of a classiﬁcation network.

Original: The training of such networks may be realized using segmentation masks explicitly generated from weak annotations (Wei et al., 2017; Khoreva et al., 2017; Lee et al., 2019; Wei et al., 2018; Tsutsui et al., 2018).
Revised: The training of such networks may be realized using segmentation masks explicitly generated from weak annotations (Wei et al., 2017; Khoreva et al., 2017; Lee et al., 2019; Wei et al., 2018; Tsutsui et al., 2018).

Original: The segmentation masks can be improved recursively, which involves several rounds of training of the segmentation network (Wei et al., 2017; Jing et al., 2018; Ezhov et al., 2018).
Revised: The segmentation masks can be improved recursively, which involves several rounds of training of the segmentation network (Wei et al., 2017; Jing et al., 2018; Ezhov et al., 2018).

Original: Other techniques do not use any explicit pseudo segmentation masks as training data, but instead compute a composite loss from other guiding principles.
Revised: Other techniques do not use any explicit pseudo segmentation masks as training data, but instead compute a composite loss from other guiding principles.

Original: For instance, the SEC method (Kolesnikov &
Revised: For instance, the SEC method (Kolesnikov &

Original: Lampert, 2016) combines localization cues (Seed), classiﬁcation coherence (Expand) and Constrainto-boundary with a fully-connected CRF.
Revised: Lampert, 2016) combines localization cues (Seed), classiﬁcation coherence (Expand) and Constrainto-boundary with a fully-connected CRF.

Original: Special modules could be designed to further exploit the information of sources, such as the Deep Seeded Region Growing (Huang et al., 2018) and saliency seeded region growing (Sun & Li, 2019).
Revised: Special modules could be designed to further exploit the information of sources, such as the Deep Seeded Region Growing (Huang et al., 2018) and saliency seeded region growing (Sun & Li, 2019).

Original: Semi-supervised deep learning provides an alternative way to reduce the pixel-level annotation burden (Baur et al., 2017; Perone & Cohen-Adad, 2018).
Revised: Semi-supervised deep learning provides an alternative way to reduce the pixel-level annotation burden (Baur et al., 2017; Perone & Cohen-Adad, 2018).

Original: These methods require only a small amount of strongly annotated data and a large set of unlabelled data.
Revised: These methods require only a small amount of strongly annotated data and a large set of unlabelled data.

Original: Several training methods that combine weak annotations with a limited set of strong annotations have already been explored (Papandreou et al., 2015; Wei et al., 2018; Lee et al., 2019).
Revised: Several training methods that combine weak annotations with a limited set of strong annotations have already been explored (Papandreou et al., 2015; Wei et al., 2018; Lee et al., 2019).

Original: The pixel-wise labels can be integrated through an additional loss function, and used along with their semi-supervised counterparts.
Revised: The pixel-wise labels can be integrated through an additional loss function, and used along with their semi-supervised counterparts.

Original: Better weak supervision performance can be obtained by feeding the network with learned object localization maps (Wei et al., 2018; Lee et al., 2019).
Revised: Better weak supervision performance can be obtained by feeding the network with learned object localization maps (Wei et al., 2018; Lee et al., 2019).

Original: However, the networks are trained with image level labels from large scale datasets like ImageNet (Deng et al., 2009) or PASCAL VOC 2012 (Everingham et al., 2015).
Revised: However, the networks are trained with image level labels from large scale datasets like ImageNet (Deng et al., 2009) or PASCAL VOC 2012 (Everingham et al., 2015).

Original: The weakly and semi-supervised deep learning methods are also explored for object localization or segmentation on relatively smaller datasets, especially in medical imaging.
Revised: The weakly and semi-supervised deep learning methods are also explored for object localization or segmentation on relatively smaller datasets, especially in medical imaging.

Original: For these methods, the weak annotations can be of image level (Mlynarski et al., 2018; Playout et al., 2019;
Revised: For these methods, the weak annotations can be of image level (Mlynarski et al., 2018; Playout et al., 2019;

Original: Zhou et al., 2019; Shin et al., 2019) or in forms of bounding boxes (Shah et al., 2018).
Revised: Zhou et al., 2019; Shin et al., 2019) or in forms of bounding boxes (Shah et al., 2018).

Original: Multi-task learning.
Revised: Multi-task learning.

Original: Multi-task learning algorithms consider several related tasks to improve the overall performance, taking beneﬁts from the underlying common information that may be ignored by a single task learning.
Revised: Multi-task learning algorithms consider several related tasks to improve the overall performance, taking beneﬁts from the underlying common information that may be ignored by a single task learning.

Original: In deep neural based multi-task models, the common information is conveyed by soft or hard parameter sharing (Ruder, 2017).
Revised: In deep neural based multi-task models, the common information is conveyed by soft or hard parameter sharing (Ruder, 2017).

Original: Various multi-task deep learning methods have been developed for segmentation, for example, the stacked U-net for extracting roads from satellite imagery (Sun et al., 2018), the two stage 3D U-net framework for 3 D CT or MR data segmentation (Wang et al., 2018a), encoder decoder networks for depth regression, semantic and instance segmentation (Kendall et al., 2018), and for building footprint segmentation (Bischke et al., 2017).
Revised: Various multi-task deep learning methods have been developed for segmentation, for example, the stacked U-net for extracting roads from satellite imagery (Sun et al., 2018), the two stage 3D U-net framework for 3 D CT or MR data segmentation (Wang et al., 2018a), encoder decoder networks for depth regression, semantic and instance segmentation (Kendall et al., 2018), and for building footprint segmentation (Bischke et al., 2017).

Original: Compared to these works, our method handles the tasks with a multi-task block at different feature resolutions and is designed upon both strong and weak notations.
Revised: Compared to these works, our method handles the tasks with a multi-task block at different feature resolutions and is designed upon both strong and weak notations.

Original: Our work is more closely related to the multi-task learning methods in (Playout et al., 2018; 2019) for retinal lesions segmentation and (Mlynarski et al., 2018) for brain tumour segmentation.
Revised: Our work is more closely related to the multi-task learning methods in (Playout et al., 2018; 2019) for retinal lesions segmentation and (Mlynarski et al., 2018) for brain tumour segmentation.

Original: Three tasks are considered in (Playout et al., 2019): red lesion segmentation, bright lesions segmentation and image level lesion detection.
Revised: Three tasks are considered in (Playout et al., 2019): red lesion segmentation, bright lesions segmentation and image level lesion detection.

Original: The proposed encoder-decoder architecture has one branch in the encoder part and two branches into decoder part speciﬁed for the red/bright lesion segmentation and the image level classiﬁcation respectively.
Revised: The proposed encoder-decoder architecture has one branch in the encoder part and two branches into decoder part speciﬁed for the red/bright lesion segmentation and the image level classiﬁcation respectively.

Original: The network is weakly supervised as only image level labels are used in one of the training phases.
Revised: The network is weakly supervised as only image level labels are used in one of the training phases.

Original: In (Mlynarski et al., 2018), a U-net architecture is used for jointly segmenting and classifying brain tumours.
Revised: In (Mlynarski et al., 2018), a U-net architecture is used for jointly segmenting and classifying brain tumours.

Original: An additional branch takes the last but one layer of a standard segmentation U-net as input, followed by a mean pooling layer, and outputs a score for the classiﬁcation task.
Revised: An additional branch takes the last but one layer of a standard segmentation U-net as input, followed by a mean pooling layer, and outputs a score for the classiﬁcation task.

Original: Our network structure, similar to these architectures, shares an encoder part for all tasks.
Revised: Our network structure, similar to these architectures, shares an encoder part for all tasks.

Original: The learning is also supervised by a mixture of strong annotations and weak annotations.
Revised: The learning is also supervised by a mixture of strong annotations and weak annotations.

Original: However, the proposed method does not include supervision from image level annotations and is more specialized in distinguishing the different object instances and clarifying their boundaries in every single image.
Revised: However, the proposed method does not include supervision from image level annotations and is more specialized in distinguishing the different object instances and clarifying their boundaries in every single image.

Original:  M ULTI - TASK LEARNING FRAMEWORK
Revised:  M ULTI - TASK LEARNING FRAMEWORK

Original: Fully supervised learning for segmentation approximates the conditional probability distribution of the segmentation mask given the image.
Revised: Fully supervised learning for segmentation approximates the conditional probability distribution of the segmentation mask given the image.

Original: Let s (3) be the segmentation mask
Revised: Let s (3) be the segmentation mask

Original: and I be the image, then the segmentation task aims to estimate p (cid:16) s (3)
Revised: and I be the image, then the segmentation task aims to estimate p (cid:16) s (3)

Original: | I (cid:17) based on a set of sample images I =
Revised: | I (cid:17) based on a set of sample images I =

Original: { I 1 , I 2 , · · · , I n } and the corresponding labels { s (3)1 , s (3)2 , · · · , s (3) n } .
Revised: { I 1 , I 2 , · · · , I n } and the corresponding labels { s (3)1 , s (3)2 , · · · , s (3) n } .

Original: The set I is randomly drawn from an unknown distribution.
Revised: The set I is randomly drawn from an unknown distribution.

Original: In our setting, having the whole set of segmentation labels { s (3) i } 1 , ··· ,n is impractical, and we introduce two auxiliary tasks for which the labels can be more easily generated to achieve an overall small cost on labeling.
Revised: In our setting, having the whole set of segmentation labels { s (3) i } 1 , ··· ,n is impractical, and we introduce two auxiliary tasks for which the labels can be more easily generated to achieve an overall small cost on labeling.

Original: For a given image I ∈
Revised: For a given image I ∈

Original: I , we denote as s (1) the rough instance detection mask, s (2) a map containing some interfaces shared by touching objects.
Revised: I , we denote as s (1) the rough instance detection mask, s (2) a map containing some interfaces shared by touching objects.

Original: To create s (1) , the contours of the objects are not treated carefully, resulting in a coarse mask that misses most of the boundary pixels, cf, the left of Figure 1.
Revised: To create s (1) , the contours of the objects are not treated carefully, resulting in a coarse mask that misses most of the boundary pixels, cf, the left of Figure 1.

Original: Let I k
Revised: Let I k

Original: ⊂ I denote the subset of images labelled for task k
Revised: ⊂ I denote the subset of images labelled for task k

Original: ( k = 1 , 2 , 3) .
Revised: ( k = 1 , 2 , 3) .

Original: As we collect a different amount of annotations for each task, the number of annotated images |I
Revised: As we collect a different amount of annotations for each task, the number of annotated images |I

Original: k | may not be the same for different k .
Revised: k | may not be the same for different k .

Original: Typically the number of images with strong annotations |I 3 | (cid:28) n .
Revised: Typically the number of images with strong annotations |I 3 | (cid:28) n .

Original: The set of samples in I 3 for segmentation being small, the computation of an accurate approximation of the true probability distribution p (cid:16) s (3) (cid:12)(cid:12)(cid:12)
Revised: The set of samples in I 3 for segmentation being small, the computation of an accurate approximation of the true probability distribution p (cid:16) s (3) (cid:12)(cid:12)(cid:12)

Original: I (cid:17) is a challenging issue.
Revised: I (cid:17) is a challenging issue.

Original: Given that much more samples of s (1) and s (2) are observed, it is relatively easier to learn the statistics of the weak labels.
Revised: Given that much more samples of s (1) and s (2) are observed, it is relatively easier to learn the statistics of the weak labels.

Original: Therefore, in a multi-task learning setting, one aims at approximating also the conditional probability p (cid:16) s (1) (cid:12)(cid:12)(cid:12) I (cid:17) and p (cid:16) s (2) (cid:12)(cid:12)(cid:12)
Revised: Therefore, in a multi-task learning setting, one aims at approximating also the conditional probability p (cid:16) s (1) (cid:12)(cid:12)(cid:12) I (cid:17) and p (cid:16) s (2) (cid:12)(cid:12)(cid:12)

Original: I (cid:17) for the other two tasks, or the joint probability p (cid:16)(cid:16) s (1) , s (2) , s (3) (cid:17)
Revised: I (cid:17) for the other two tasks, or the joint probability p (cid:16)(cid:16) s (1) , s (2) , s (3) (cid:17)

Original: | I (cid:17) .
Revised: | I (cid:17) .

Original: The three tasks are connected to each other.
Revised: The three tasks are connected to each other.

Original: By the deﬁnition of the detection task, it is not difﬁcult to see that p (cid:0) s (3) =
Revised: By the deﬁnition of the detection task, it is not difﬁcult to see that p (cid:0) s (3) =

Original: z (cid:12)(cid:12) s (1) =
Revised: z (cid:12)(cid:12) s (1) =

Original: x (cid:1) = 0 for x and z satisfying x i,c = 1 and z i,c = 0 for some pixel i and class c other than the background.
Revised: x (cid:1) = 0 for x and z satisfying x i,c = 1 and z i,c = 0 for some pixel i and class c other than the background.

Original: The map of interfaces s (2) indicates small gaps between two connected instances, and is therefore a subset of boundary pixels of the mask s (3) .
Revised: The map of interfaces s (2) indicates small gaps between two connected instances, and is therefore a subset of boundary pixels of the mask s (3) .

Original: Let us consider the probabilities given by the models p (cid:16) s ( k ) (cid:12)(cid:12)(cid:12) I ; θ (cid:17) ( k = 1 , 2 , 3) parametrized by θ which is determined such that the models match the desired probability distributions.
Revised: Let us consider the probabilities given by the models p (cid:16) s ( k ) (cid:12)(cid:12)(cid:12) I ; θ (cid:17) ( k = 1 , 2 , 3) parametrized by θ which is determined such that the models match the desired probability distributions.

Original: The parameter θ is shared among all the tasks.
Revised: The parameter θ is shared among all the tasks.

Original: We do not optimize θ for each individual task, but instead consider a joint probability p (cid:16)(cid:16) s (1) , s (2) , s (3) (cid:17) | I ; θ (cid:17) .
Revised: We do not optimize θ for each individual task, but instead consider a joint probability p (cid:16)(cid:16) s (1) , s (2) , s (3) (cid:17) | I ; θ (cid:17) .

Original: Assuming that s (1) (rough under-segmented instance detection) and s (2) (subset of shared boundaries) are conditionally independent given image I , and if the samples are i.i.d, we deﬁne the maximum likelihood (ML) estimator for θ as
Revised: Assuming that s (1) (rough under-segmented instance detection) and s (2) (subset of shared boundaries) are conditionally independent given image I , and if the samples are i.i.d, we deﬁne the maximum likelihood (ML) estimator for θ as

Original: [Equation]
Revised: [Equation]

Original: The set I 3 may not be evenly distributed across I , but we assume that it is generated by a ﬁxed distribution as well.
Revised: The set I 3 may not be evenly distributed across I , but we assume that it is generated by a ﬁxed distribution as well.

Original: Provided that the term (cid:110) p (cid:16) s (3) | s (1) , s (2) , I (cid:17)(cid:111) I ∈I can be approximated correctly by p (cid:16) s (3) | s (1) , s (2) , I ; θ (cid:17) even if θ is computed without s (3) speciﬁed for I\I 3 , then
Revised: Provided that the term (cid:110) p (cid:16) s (3) | s (1) , s (2) , I (cid:17)(cid:111) I ∈I can be approximated correctly by p (cid:16) s (3) | s (1) , s (2) , I ; θ (cid:17) even if θ is computed without s (3) speciﬁed for I\I 3 , then

Original: [Equation]
Revised: [Equation]

Original: If furthermore, the segmentation mask does not depend on s (1) or s (2) given I ∈
Revised: If furthermore, the segmentation mask does not depend on s (1) or s (2) given I ∈

Original: I 3 , and if |I 1 | , |I 2 | are large enough, then from Equations (1), and (2), we approximate the ML estimator by
Revised: I 3 , and if |I 1 | , |I 2 | are large enough, then from Equations (1), and (2), we approximate the ML estimator by

Original: [Equation]
Revised: [Equation]

Original: in which α 1 , α 2 , α 3 are non negative constants.
Revised: in which α 1 , α 2 , α 3 are non negative constants.

Original: 3. L OSS FUNCTION
Revised: 3. L OSS FUNCTION

Original: Let the output of the approximation models be denoted respectively by h (1) θ ( I ) , h (2)
Revised: Let the output of the approximation models be denoted respectively by h (1) θ ( I ) , h (2)

Original: θ ( I ) , and h (3)
Revised: θ ( I ) , and h (3)

Original: θ ( I ) , with (cid:104) h ( k
Revised: θ ( I ) , with (cid:104) h ( k

Original: ) θ ( I ) (cid:105) i,c
Revised: ) θ ( I ) (cid:105) i,c

Original: the estimated probability of pixel i being in class c of task k .
Revised: the estimated probability of pixel i being in class c of task k .

Original: For the label s ( k ) of I , the log likelihood function for each task is decomposed into
Revised: For the label s ( k ) of I , the log likelihood function for each task is decomposed into

Original: [Equation]
Revised: [Equation]

Original: in which s ( k ) i,c denotes the element of the label s ( k ) at pixel i for class c and C k is the set of classes for task k .
Revised: in which s ( k ) i,c denotes the element of the label s ( k ) at pixel i for class c and C k is the set of classes for task k .

Original: For example, for ice cream images, we have three classes including air bubbles, ice crystals and the rest (background or parts of the objects ignored by the weak labels), so C 1 , C 3 = { 1 , 2 , 3 } .
Revised: For example, for ice cream images, we have three classes including air bubbles, ice crystals and the rest (background or parts of the objects ignored by the weak labels), so C 1 , C 3 = { 1 , 2 , 3 } .

Original: For the separation task, there are only two classes for pixels (belonging or not to a touching interface):
Revised: For the separation task, there are only two classes for pixels (belonging or not to a touching interface):

Original: C 2 = { 1 , 2 } .
Revised: C 2 = { 1 , 2 } .

Original: According to Equation (3), the network is trained by minimizing the weighted cross entropy loss:
Revised: According to Equation (3), the network is trained by minimizing the weighted cross entropy loss:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here 1 I k ( · ) is an indicator function which is 1 if I ∈ I k and 0 otherwise.
Revised: Here 1 I k ( · ) is an indicator function which is 1 if I ∈ I k and 0 otherwise.

Original: 3. M ULTI - TASK N ETWORK
Revised: 3. M ULTI - TASK N ETWORK

Original: We follow a convolutional encoder-decoder network structure for the multi-task learning.
Revised: We follow a convolutional encoder-decoder network structure for the multi-task learning.

Original: The network architecture is illustrated in Figure 2.
Revised: The network architecture is illustrated in Figure 2.

Original: As an extension of the U-net structure for multiple tasks, we only have one contracting path that encodes shared features representation for all the tasks.
Revised: As an extension of the U-net structure for multiple tasks, we only have one contracting path that encodes shared features representation for all the tasks.

Original: On the expansive branch, we introduce a multi-task block at each resolution to support different learning purposes (blue blocks in Figure 2).
Revised: On the expansive branch, we introduce a multi-task block at each resolution to support different learning purposes (blue blocks in Figure 2).

Original: Every multi-task block runs three paths, with three inputs and three corresponding outputs, and it consists of several sub-blocks.
Revised: Every multi-task block runs three paths, with three inputs and three corresponding outputs, and it consists of several sub-blocks.

Original: In each multi-task block, the detection task (task 1) and the segmentation task (task 3) have a common path similar to the decoder part of the standard U-net.
Revised: In each multi-task block, the detection task (task 1) and the segmentation task (task 3) have a common path similar to the decoder part of the standard U-net.

Original: They share the same weights and use the same concatenation with feature maps from contracting path via the skip connections.
Revised: They share the same weights and use the same concatenation with feature maps from contracting path via the skip connections.

Original: However, we insert an additional residual sub-block for the segmentation task.
Revised: However, we insert an additional residual sub-block for the segmentation task.

Original: The residual sub-block provides extra network parameters to learn information not known from the detection task, e.g. object boundary localization.
Revised: The residual sub-block provides extra network parameters to learn information not known from the detection task, e.g. object boundary localization.

Original: The path for the separation task (task 2) is built on the top of detection/segmentation ones.
Revised: The path for the separation task (task 2) is built on the top of detection/segmentation ones.

Original: It is also a U-net decoder block structure, but the long skip connections start from the sub-blocks of the detection/segmentation paths instead of the contracting path.
Revised: It is also a U-net decoder block structure, but the long skip connections start from the sub-blocks of the detection/segmentation paths instead of the contracting path.

Original: The connections extract higher resolution features from the segmentation task and use them in the separation task.
Revised: The connections extract higher resolution features from the segmentation task and use them in the separation task.

Original: To formulate the multi-task blocks, let x l and z l denote respectively the output of the detection path and segmentation path at the multi-task block l , and let c l be the feature maps received from the contracting path with the skip connections.
Revised: To formulate the multi-task blocks, let x l and z l denote respectively the output of the detection path and segmentation path at the multi-task block l , and let c l be the feature maps received from the contracting path with the skip connections.

Original: Then for task 1 and task 3 we have
Revised: Then for task 1 and task 3 we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: in which W l , W l +1 / 2 ∈ θ are parameters of the network and F W l , F W l +12 are determined respectively by a sequence of layers of the network (Cf.
Revised: in which W l , W l +1 / 2 ∈ θ are parameters of the network and F W l , F W l +12 are determined respectively by a sequence of layers of the network (Cf.

Original: the small grey blocks on the right of Figure 2).
Revised: the small grey blocks on the right of Figure 2).

Original: For taskthe output at l th block y l +1 is computed by y l +

Revised: For taskthe output at l th block y l +1 is computed by y l +

Original: =
Revised: =

Original: G ˜ W l ( z l +1 , y l ) with additional network parameters ˜ W l ∈ θ .
Revised: G ˜ W l ( z l +1 , y l ) with additional network parameters ˜ W l ∈ θ .

Original: Finally, after the last multi-task block, softmax layers are added, outputting a probability map for each task.
Revised: Finally, after the last multi-task block, softmax layers are added, outputting a probability map for each task.

Original: Implementation details.
Revised: Implementation details.

Original: We implement a multi-task U-net with 6 levels of spatial resolution and input images of size 256 × 256 .
Revised: We implement a multi-task U-net with 6 levels of spatial resolution and input images of size 256 × 256 .

Original: A sequence of down-sampling via max-pooling with pooling
Revised: A sequence of down-sampling via max-pooling with pooling

Original: [Figure]
Revised: [Figure]

Original: Task Block ( ! "
Revised: Task Block ( ! "

Original: #$ ,& "#$ , ' "#$ ) ( ! " ,& " , ' " )
Revised: #$ ,& "#$ , ' "#$ ) ( ! " ,& " , ' " )

Original: ( ! " ,' " ) & " & "#$ ! "
Revised: ( ! " ,' " ) & " & "#$ ! "

Original: #$ ' "#$ ' "#$/) size 2 × 2 is used for the contracting path of the network.
Revised: #$ ' "#$ ' "#$/) size 2 × 2 is used for the contracting path of the network.

Original: Different from the conventional U-net (Ronneberger et al., 2015), each small gray block (see Figure 2) consists of a convolution layer and a batch normalization (Ioffe & Szegedy, 2015), followed by a leaky ReLU activation with a leakiness parameter 0 .
Revised: Different from the conventional U-net (Ronneberger et al., 2015), each small gray block (see Figure 2) consists of a convolution layer and a batch normalization (Ioffe & Szegedy, 2015), followed by a leaky ReLU activation with a leakiness parameter 0 .

Original: The same setting is also applied to gray sub-blocks of the 4 multi-task blocks.
Revised: The same setting is also applied to gray sub-blocks of the 4 multi-task blocks.

Original: On the expansive path of the network, feature maps are up-sampled (with factor 2 × 2 ) by bilinear interpolation from a low resolution multi-task block to the next one.
Revised: On the expansive path of the network, feature maps are up-sampled (with factor 2 × 2 ) by bilinear interpolation from a low resolution multi-task block to the next one.

Original: 3. M ETHODS FOR L AZY LABELS GENERATION
Revised: 3. M ETHODS FOR L AZY LABELS GENERATION

Original: We now explain our strategy for generating all the lazy annotations that are used for training the multi-task U-net.
Revised: We now explain our strategy for generating all the lazy annotations that are used for training the multi-task U-net.

Original: We introduce our method with a data set of ice cream SEM images but any other similar dataset could be used.
Revised: We introduce our method with a data set of ice cream SEM images but any other similar dataset could be used.

Original: Typical images of ice cream samples are shown in the top row of the left part of Figure 3.
Revised: Typical images of ice cream samples are shown in the top row of the left part of Figure 3.

Original: The segmentation problem is challenging since the images contain densely distributed small object instances ( i.e. , air bubble and ice crystals), and poor contrast between the foreground and the background.
Revised: The segmentation problem is challenging since the images contain densely distributed small object instances ( i.e. , air bubble and ice crystals), and poor contrast between the foreground and the background.

Original: The sizes of the objects can vary signiﬁcantly in a single sample.
Revised: The sizes of the objects can vary signiﬁcantly in a single sample.

Original: Textures on the surfaces of objects also appear.
Revised: Textures on the surfaces of objects also appear.

Original: As a ﬁrst step, scribble-based labelling is applied to obtain detection regions of air bubbles and ice crystals for task 1.
Revised: As a ﬁrst step, scribble-based labelling is applied to obtain detection regions of air bubbles and ice crystals for task 1.

Original: This can be done in a very fast way as no effort is put on the exact object boundaries.
Revised: This can be done in a very fast way as no effort is put on the exact object boundaries.

Original: We adopt a lazy strategy by picking out an inner region for each object in the images (see e.g. , the second row of the left part of Figure 3).
Revised: We adopt a lazy strategy by picking out an inner region for each object in the images (see e.g. , the second row of the left part of Figure 3).

Original: Though one could get these rough regions as accurate as possible, we delay such reﬁnement to task 3, for better efﬁciency of the global annotation process.
Revised: Though one could get these rough regions as accurate as possible, we delay such reﬁnement to task 3, for better efﬁciency of the global annotation process.

Original: Compared to the commonly used bounding box annotations in computer vision tasks, these labels give more conﬁdence for a particular part of the region of interest.
Revised: Compared to the commonly used bounding box annotations in computer vision tasks, these labels give more conﬁdence for a particular part of the region of interest.

Original: In the second step, we focus on tailored labels for those instances that are close one to each other (task 2), without a clear boundary separating them.
Revised: In the second step, we focus on tailored labels for those instances that are close one to each other (task 2), without a clear boundary separating them.

Original: Again, we use scribbles to mark their interface.
Revised: Again, we use scribbles to mark their interface.

Original: Examples for such annotations are given in Figure 3 (top line, right part)
Revised: Examples for such annotations are given in Figure 3 (top line, right part)

Original: The work can be carried out efﬁciently especially when the target scribbles have a sparse distribution.
Revised: The work can be carried out efﬁciently especially when the target scribbles have a sparse distribution.

Original: On the other hand, as no labelling is needed for the objects that are well separated, we can collect sufﬁcient labelled images in a limited amount of time and cover the complex ice cream sample conditions.
Revised: On the other hand, as no labelling is needed for the objects that are well separated, we can collect sufﬁcient labelled images in a limited amount of time and cover the complex ice cream sample conditions.

Original: Lazy manual labeling of tasks 1 and 2 are done independently.
Revised: Lazy manual labeling of tasks 1 and 2 are done independently.

Original: It follows the assumption made in Section 3 that s (1) and s (2) are conditionally independent given image I .
Revised: It follows the assumption made in Section 3 that s (1) and s (2) are conditionally independent given image I .

Original: The precise labels for task 3 are created using interactive segmentation tools.
Revised: The precise labels for task 3 are created using interactive segmentation tools.

Original: Starting from the rough (inner) regions of task 1, a natural idea is to let these regions grow and stop when the boundaries are reached.
Revised: Starting from the rough (inner) regions of task 1, a natural idea is to let these regions grow and stop when the boundaries are reached.

Original: This can be done with geodesic active contours (Caselles et al., 1997).
Revised: This can be done with geodesic active contours (Caselles et al., 1997).

Original: Unfortunately, such a method fails to capture sharp corners and the contour evolution tends to ignore boundaries with low contrast.
Revised: Unfortunately, such a method fails to capture sharp corners and the contour evolution tends to ignore boundaries with low contrast.

Original: The annotation then requires frequent and time consuming user interaction.
Revised: The annotation then requires frequent and time consuming user interaction.

Original: Instead, we use
Revised: Instead, we use

Original: Grabcut (Rother et al., 2004; Hong et al., 2015) a graph-cut based method.
Revised: Grabcut (Rother et al., 2004; Hong et al., 2015) a graph-cut based method.

Original: The initial labels obtained from the ﬁrst step give a good guess of the whole object regions.
Revised: The initial labels obtained from the ﬁrst step give a good guess of the whole object regions.

Original: The Grabcut works well on isolated objects.
Revised: The Grabcut works well on isolated objects.

Original: However, it gives poor results when the objects are close to each other and have boundaries with inhomogeneous colors.
Revised: However, it gives poor results when the objects are close to each other and have boundaries with inhomogeneous colors.

Original: As corrections may be needed for each image, only a few images of the whole dataset are processed.
Revised: As corrections may be needed for each image, only a few images of the whole dataset are processed.

Original: A fully segmented example is shown in the last row of Figure 3.
Revised: A fully segmented example is shown in the last row of Figure 3.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we demonstrate the performance of our approach using two different datasets.
Revised: In this section, we demonstrate the performance of our approach using two different datasets.

Original: For both datasets we use strong labels (SL) as well as weak labels (WL).
Revised: For both datasets we use strong labels (SL) as well as weak labels (WL).

Original: We prepare the labels and design the network as described in Section 3.
Revised: We prepare the labels and design the network as described in Section 3.

Original: The overall method is summarized with the 2 procedures presented in Algorithm 1 (see the appendix).
Revised: The overall method is summarized with the 2 procedures presented in Algorithm 1 (see the appendix).

Original: 4. SEM IMAGES OF ICE CREAM
Revised: 4. SEM IMAGES OF ICE CREAM

Original: Scanning Electron Microscopy (SEM) constitutes the state-of-the-art for analysing food microstructures as it enables the efﬁcient acquisition of high quality images for food materials, resulting into a huge amount of image data available for analysis.
Revised: Scanning Electron Microscopy (SEM) constitutes the state-of-the-art for analysing food microstructures as it enables the efﬁcient acquisition of high quality images for food materials, resulting into a huge amount of image data available for analysis.

Original: However, to better delineate the microstructures and provide exact statistical information, the partition of the images into different food components and instances is needed.
Revised: However, to better delineate the microstructures and provide exact statistical information, the partition of the images into different food components and instances is needed.

Original: The structures of food, especially soft solid materials, are usually complex which makes automated segmentation a difﬁcult task.
Revised: The structures of food, especially soft solid materials, are usually complex which makes automated segmentation a difﬁcult task.

Original: Some SEM images of ice cream in our dataset are shown on the top right of Figure 1.
Revised: Some SEM images of ice cream in our dataset are shown on the top right of Figure 1.

Original: A typical ice cream sample consists of air bubbles, ice crystals and a concentrated unfrozen solution.
Revised: A typical ice cream sample consists of air bubbles, ice crystals and a concentrated unfrozen solution.

Original: In most situations, the air bubbles and ice crystals appear as foam in the images, while the solution ﬁlls the gaps between them.
Revised: In most situations, the air bubbles and ice crystals appear as foam in the images, while the solution ﬁlls the gaps between them.

Original: We treat the solution as the background and aim at detecting and computing a pixel-wise classiﬁcation for each air bubbles and ice crystals instances.
Revised: We treat the solution as the background and aim at detecting and computing a pixel-wise classiﬁcation for each air bubbles and ice crystals instances.

Original: The set of ice-cream SEM dataset consists of 38 images that are split into three sets (53% for training, 16% for validation and 31% testing respectievly).
Revised: The set of ice-cream SEM dataset consists of 38 images that are split into three sets (53% for training, 16% for validation and 31% testing respectievly).

Original: Each image contains a rich set of instances with an overall number of instances around 13300 for 2 classes (ice crystals and air bubbles).
Revised: Each image contains a rich set of instances with an overall number of instances around 13300 for 2 classes (ice crystals and air bubbles).

Original: For comparison, the PASCAL VOC 2012 dataset has 27450 objects in total for 20 classes.
Revised: For comparison, the PASCAL VOC 2012 dataset has 27450 objects in total for 20 classes.

Original: For training the network, data augmentation is applied to prevent over-ﬁtting.
Revised: For training the network, data augmentation is applied to prevent over-ﬁtting.

Original: The size of the raw images is 960 × 1280 .
Revised: The size of the raw images is 960 × 1280 .

Original: They are rescaled and rotated randomly, and then cropped into an input size of 256 × 256 for feeding the network.
Revised: They are rescaled and rotated randomly, and then cropped into an input size of 256 × 256 for feeding the network.

Original: Random ﬂipping is also performed during training.
Revised: Random ﬂipping is also performed during training.

Original: The network is trained using Adam optimizer (Kingma & Ba, 2014) with a learning rate r = 2 × 10 − 4 and a batch size of 16 .
Revised: The network is trained using Adam optimizer (Kingma & Ba, 2014) with a learning rate r = 2 × 10 − 4 and a batch size of 16 .

Original: In the inference phase, the network outputs for each patch a probability map of size 256 × 256 .
Revised: In the inference phase, the network outputs for each patch a probability map of size 256 × 256 .

Original: The patches are then aggregated to obtain a probability map for the whole image.
Revised: The patches are then aggregated to obtain a probability map for the whole image.

Original: In general, the pixels near the boundaries of each patch are harder to classify.
Revised: In general, the pixels near the boundaries of each patch are harder to classify.

Original: We thus weight the spatial inﬂuence of the patches with a Gaussian kernel to emphasize the network prediction at patch center.
Revised: We thus weight the spatial inﬂuence of the patches with a Gaussian kernel to emphasize the network prediction at patch center.

Original: [Figure]
Revised: [Figure]

Original: We now evaluate the multi-task U-net and compare it to the traditional single task U-net.
Revised: We now evaluate the multi-task U-net and compare it to the traditional single task U-net.

Original: The performance of each model is tested on 12 images, and average results are shown in Table 1.
Revised: The performance of each model is tested on 12 images, and average results are shown in Table 1.

Original: In the table, the dice score for a class c is deﬁned as d c = 2 (cid:80) i x
Revised: In the table, the dice score for a class c is deﬁned as d c = 2 (cid:80) i x

Original: i,c y i,c / (cid:0)(cid:80) i x i,c + (cid:80) i
Revised: i,c y i,c / (cid:0)(cid:80) i x i,c + (cid:80) i

Original: y i,c (cid:1) where x is the computed segmentation mask and y the ground truth. 
Revised: y i,c (cid:1) where x is the computed segmentation mask and y the ground truth. The models

Original: [Table]
Revised: [Table]

Original:  We train a single task U-net ( i.e. , without the multi-task block) on the weakly labelled set (task 1), with the 15 annotated images.
Revised: [Figure] We train a single task U-net ( i.e. , without the multi-task block) on the weakly labelled set (task 1), with the 15 annotated images.

Original: The single task U-net on weak annotations gives an overall dice score at 0 .
Revised: The single task U-net on weak annotations gives an overall dice score at 0 .

Original: 72 , the lowest one among the three other methods tested.
Revised: 72 , the lowest one among the three other methods tested.

Original: One reason for the low accuracy of the single task U-net on weak (inaccurate) annotations is that in the training labels, the object boundaries are mostly ignored.
Revised: One reason for the low accuracy of the single task U-net on weak (inaccurate) annotations is that in the training labels, the object boundaries are mostly ignored.

Original: Hence the U-net is not trained to recover them, leaving large parts of the object not recognized.
Revised: Hence the U-net is not trained to recover them, leaving large parts of the object not recognized.

Original: Second, we consider strong annotations as training data, without the data of the other tasks, i.e. only 2 images with accurate segmentation masks are used.
Revised: Second, we consider strong annotations as training data, without the data of the other tasks, i.e. only 2 images with accurate segmentation masks are used.

Original: The score of the U-net trained on SL is only 0 .
Revised: The score of the U-net trained on SL is only 0 .

Original: 82 , which is signiﬁcantly lower than the 0 .
Revised: 82 , which is signiﬁcantly lower than the 0 .

Original: 94 obtained by the multi-task U-net.
Revised: 94 obtained by the multi-task U-net.

Original: We also compare our multi-task U-net results with one of the major weakly supervised approaches that makes use of pseudo labels (PL) (see e.g., Khoreva et al.
Revised: We also compare our multi-task U-net results with one of the major weakly supervised approaches that makes use of pseudo labels (PL) (see e.g., Khoreva et al.

Original: ; Jing et al.
Revised: ; Jing et al.

Original: In these approaches, the pseudo segmentation masks are created from WLs and are used to feed a segmentation network.
Revised: In these approaches, the pseudo segmentation masks are created from WLs and are used to feed a segmentation network.

Original: Following the work of Khoreva et al.
Revised: Following the work of Khoreva et al.

Original: (2017), we use the Grabcut method to create the
Revised: (2017), we use the Grabcut method to create the

Original: PLs, and in our setting the partial masks are used to initialize the Grabcut.
Revised: PLs, and in our setting the partial masks are used to initialize the Grabcut.

Original: For the small subset of images that are strongly annotated, the full segmentation masks are used instead of PLs.
Revised: For the small subset of images that are strongly annotated, the full segmentation masks are used instead of PLs.

Original: The PL are created without human correction, and then used for feeding the segmentation network.
Revised: The PL are created without human correction, and then used for feeding the segmentation network.

Original: Here we use the baseline single task U-net as the segmentation network in order to make comparison with the multi-task U-net.
Revised: Here we use the baseline single task U-net as the segmentation network in order to make comparison with the multi-task U-net.

Original: Our multi-task U-net outperforms the PL approach as shown in Table 1.
Revised: Our multi-task U-net outperforms the PL approach as shown in Table 1.

Revised: Figure 4 displays the error bars for the two methods with dice score collected from 8 different runs.

Original: A signiﬁcant limitation of the PL method here is that its performance relies on the tools used for pseudo segmentation mask generations.
Revised: A signiﬁcant limitation of the PL method here is that its performance relies on the tools used for pseudo segmentation mask generations.

Original: Having common errors, instead of random ones, on the object boundaries in the training data, the segmentation network of PL also learns to have those patterns in the prediction.
Revised: Having common errors, instead of random ones, on the object boundaries in the training data, the segmentation network of PL also learns to have those patterns in the prediction.

Original: The images in the left part of Figure 4 show that the predicted label of an object tends to merge with some background pixels when there are edges of another object nearby.
Revised: The images in the left part of Figure 5 show that the predicted label of an object tends to merge with some background pixels when there are edges of another object nearby.

Original: Similar errors from the GrabCut are illustrated in the appendix.
Revised: Similar errors from the GrabCut are illustrated in the appendix.

Original: [Figure]

Original: Besides the number of pixels that are correctly classiﬁed, the separation of touching instances is also of interest.
Revised: Besides the number of pixels that are correctly classiﬁed, the separation of touching instances is also of interest.

Original: In additional to the dice scores in Table 1, we study the learning performance of multi-task
Revised: In additional to the dice scores in Table 1, we study the learning performance of multi-task

Original: U-net on task 2, which specializes in the separation aspect.
Revised: U-net on task 2, which specializes in the separation aspect.

Original: The test results on the 12 images give an[Table] overall precision of 0 .
Revised: The test results on the 12 images give an overall precision of 0 .

Original: 70 of the detected interfaces, while 0 .
Revised: 70 of the detected interfaces, while 0 .

Original: 82 of the touching objects are recognized.
Revised: 82 of the touching objects are recognized.

Original: We show some examples of computed separations and ground truth in the right part of Figure 4.
Revised: We show some examples of computed separations and ground truth in the right part of Figure 5.

Original: For the detection task, the network predicts a probability map for the inner regions of the object instances.
Revised: For the detection task, the network predicts a probability map for the inner regions of the object instances.

Original: An output of the network is shown in Figure 5.
Revised: An output of the network is shown in Figure 6.

Original: With partial masks as coarse labels for this task, the network learns to identify the object instance as shown in the ﬁgure.
Revised: With partial masks as coarse labels for this task, the network learns to identify the object instance as shown in the ﬁgure.

Original: We ﬁnally consider the work of Bearman et al.
Revised: We ﬁnally consider the work of Bearman et al.

Original: (2016) that investigate the cost related to different types of annotations.
Revised: (2016) that investigate the cost related to different types of annotations.

Original: Based on the data reported (Bearman et al., 2016) and our estimated annotation time, the collecting of WL for detection is considerably (more than 6x) faster than obtaining SL.
Revised: Based on the data reported (Bearman et al., 2016) and our estimated annotation time, the collecting of WL for detection is considerably (more than 6x) faster than obtaining SL.

Original: For a fair comparison with the baseline U-net we use a larger ratio of SL for the single task learning accordingly (since no WL is used here), and the results are reported in Table 2.
Revised: For a fair comparison with the baseline U-net we use a larger ratio of SL for the single task learning accordingly (since no WL is used here), and the results are reported in Table 2.

Original: The proposed method still outperforms the U-net by a large margin on similar annotation time budgets, and additional SL after the ﬁrst 10% do not help signiﬁcantly. 
Revised: The proposed method still outperforms the U-net by a large margin on similar annotation time budgets, and additional SL after the ﬁrst 10% do not help signiﬁcantly. [Figure]

Original: 4. H&E- STAINED IMAGE DATESET FOR GLAND SEGMENTATION
Revised: 4. H&E- STAINED IMAGE DATESET FOR GLAND SEGMENTATION

Original: We also apply the approach to the segmentation of tissue in histology images.
Revised: We also apply the approach to the segmentation of tissue in histology images.

Original: In this experiment, we use the GlaS challenge dataset (Sirinukunwattana et al., 2017) that consists of 165 H&E stained images.
Revised: In this experiment, we use the GlaS challenge dataset (Sirinukunwattana et al., 2017) that consists of 165 H&E stained images.

Original: The dataset is split into three parts, with 85 images for training, and 60 for offsite test andimages for onsite test (we will call the latter two sets Test part A and Test part B respectively in the following).
Revised: The dataset is split into three parts, with 85 images for training, and 60 for offsite test andimages for onsite test (we will call the latter two sets Test part A and Test part B respectively in the following).

Original: Apart from the SL available from the dataset, we create a set of a weak labels for the detection task and separation task (as illustrated in Appendix, Figure 10).
Revised: Apart from the SL available from the dataset, we create a set of a weak labels for the detection task and separation task (as illustrated in Appendix, Figure 11).

Original: These weak labels together with a part of the strong labels are used for training the multi-task U-net as illustrated in Algorithm 1 (in the appendix).
Revised: These weak labels together with a part methods labels dice score Annotation budgetmulti-task 10% SL + WL 0. single task 20% SL 0. Annotation budgetmulti-task 20% SL + WL 0.

Revised: single task 30% SL 0.

Revised: Annotation budgetmulti-task 50% SL + WL 0.

Revised: single task 75% SL 0.

Revised: From budgetto budget 3 , we increase the amount of labels (that means more annotation time is needed) in the training data, and the dice score is reported for each case.

Revised: [Table]

Revised: [Figure]

Revised: of the strong labels

Revised: are used for training the multi-task U-net as illustrated in Algorithm 1 (in the appendix).

Original: In this experiment, we test the algorithm on different ratios of SL, and compare it with the baseline
Revised: In this experiment, we test the algorithm on different ratios of SL, and compare it with the baseline

Original: U-net, and the Multi-scale Densely Connected U-Net (DMUnet) (Zhang et al., 2018).
Revised: U-net, and the Multi-scale Densely Connected U-Net (DMUnet) (Zhang et al., 2018).

Original: The results on two sets of test data are reported in Table 3.
Revised: The results on two sets of test data are reported in Table 3.

Original: As the SL ratios increase from 2 . 4% to 9 .
Revised: As the SL ratios increase from 2 . 4% to 9 .

Original: 4% , an improvement of performance of the multi-task U-net is gained, and when it reaches 9 . 4% SL, the multi-task framework achieves comparable score with the fully supervised version.
Revised: 4% , an improvement of performance of the multi-task U-net is gained, and when it reaches 9 . 4% SL, the multi-task framework achieves comparable score with the fully supervised version.

Original: We emphasize that the 9 . 4% SL and WL are at a much lower annotation cost than that of the 100% SL used for fully supervised learning.
Revised: We emphasize that the 9 . 4% SL and WL are at a much lower annotation cost than that of the 100% SL used for fully supervised learning.

Original: Example of segmentation results are displayed in Figure 6. [Table] [Figure]
Revised: Example of segmentation results are displayed in Figure 7.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we develop a multi-task learning framework for image segmentation problems, which relaxes the requirement for numerous and accurate annotations to train the network.
Revised: In this paper, we develop a multi-task learning framework for image segmentation problems, which relaxes the requirement for numerous and accurate annotations to train the network.

Original: It is suitable for segmentation problem with a dense population of object instances.
Revised: It is suitable for segmentation problem with a dense population of object instances.

Original: The model separates the segmentation problem into three smaller tasks.
Revised: The model separates the segmentation problem into three smaller tasks.

Original: One of them is dedicated to the instance detection and therefore do not need exact boundary information.
Revised: One of them is dedicated to the instance detection and therefore do not need exact boundary information.

Original: This gives potential ﬂexibility as one could concentrate on the classiﬁcation and rough location of the instances during data collection.
Revised: This gives potential ﬂexibility as one could concentrate on the classiﬁcation and rough location of the instances during data collection.

Original: The second one focuses on the separation of objects sharing a common boundary.
Revised: The second one focuses on the separation of objects sharing a common boundary.

Original: The ﬁnal task aims at extracting pixel-wise boundary information.
Revised: The ﬁnal task aims at extracting pixel-wise boundary information.

Original: Thanks to the information shared within the multi-task learning, this accurate segmentation can be obtained using very few annotated data.
Revised: Thanks to the information shared within the multi-task learning, this accurate segmentation can be obtained using very few annotated data.

Original: Our model learns directly the statistics of WL as auxiliary tasks, so no further processing steps are needed before training the network.
Revised: Our model learns directly the statistics of WL as auxiliary tasks, so no further processing steps are needed before training the network.

Original: For the partial masks that ignore boundary pixels, the annotation can also be done when the boundaries of object are hard to detect.
Revised: For the partial masks that ignore boundary pixels, the annotation can also be done when the boundaries of object are hard to detect.

Original: As a small amount of SL is needed and the collection of WL can be fast and cheap, the proposed framework is potentially effective for applications with growing datasets.
Revised: As a small amount of SL is needed and the collection of WL can be fast and cheap, the proposed framework is potentially effective for applications with growing datasets.

Original: The weakly annotated set for detection purpose could be augmented if necessary and the new images could easily be incorporated into our end-to-end framework
Revised: The weakly annotated set for detection purpose could be augmented if necessary and the new images could easily be incorporated into our end-to-end framework.

Original: A A PPENDIX A. T HE A LGORITHM
Revised: A A PPENDIX A. T HE A LGORITHM

Original: We summarize the overall procedures, including the steps of creating the labels and the multi-task training into Algorithm .[Table]
Revised: We summarize the overall procedures, including the steps of creating the labels and the multi-task training into Algorithm .

Original: iteration 8: return θ

Original: A. F URTHER RESULTS FOR THE ICE CREAM IMAGE DATASET
Revised: A. F URTHER RESULTS FOR THE ICE CREAM IMAGE DATASET

Original: Figure 7 shows the Dice scores on the 12 test images from Figure 8.
Revised: Figure 8 shows the Dice scores on the 12 test images from Figure 9.

Original: One reason for the low accuracy of the single task U-net on weak (inaccurate) annotations is that the U-net is not trained to recover object boundaries.
Revised: One reason for the low accuracy of the single task U-net on weak (inaccurate) annotations is that the U-net is not trained to recover object boundaries.

Original: Therefore, the corresponding dice score on every test image is low, as shown by the blue curve.
Revised: Therefore, the corresponding dice score on every test image is low, as shown by the blue curve.

Original: On the other hand, the network trained on strong annotations has a very good performance on only a few test images, as shown by the red line.
Revised: On the other hand, the network trained on strong annotations has a very good performance on only a few test images, as shown by the red line.

Original: Learning from both strong and weak annotations, the multi-task U-net improves from the other two, and the dice scores are between 0 .
Revised: Learning from both strong and weak annotations, the multi-task U-net improves from the other two, and the dice scores are between 0 .

Original: 9 to 0 .
Revised: 9 to 0 .

Original: 95 for all of the images.
Revised: 95 for all of the images.

Original: [Figure]
Revised: [Figure]

Original: We show the scores for the U-net baseline with weak annotations, strong annotations and pseudo labels (PL) respectively, compared along with the multi- [Figure]
Revised: We show the scores for the U-net baseline with weak annotations, strong annotations and pseudo labels (PL) respectively, compared along with the multi-task U-net

Original: A.2. A DDITIONAL RESULTS
Revised: A.2. A DDITIONAL RESULTS

Original: We compare the segmentation results given by the multi-task U-net with the ones obtained using
Revised: We compare the segmentation results given by the multi-task U-net with the ones obtained using

Original: GAC Caselles et al.
Revised: GAC Caselles et al.

Original: and Grabcut Rother et al.
Revised: and Grabcut Rother et al.

Original: (Figure 9).
Revised: (Figure 10).

Original: To apply GAC and Grabcut, we convert the problem into a single class and single object segmentation problem.
Revised: To apply GAC and Grabcut, we convert the problem into a single class and single object segmentation problem.

Original: We use the rough inner regions from the ﬁrst step of the labelling to estimate a bounding box around each object.
Revised: We use the rough inner regions from the ﬁrst step of the labelling to estimate a bounding box around each object.

Original: We then apply GAC and Grabcut inside each box.
Revised: We then apply GAC and Grabcut inside each box.

Original: The box-wise results are aggregated to get the segmentation mask of the whole image.
Revised: The box-wise results are aggregated to get the segmentation mask of the whole image.

Original: Both GAC and Grabcut fail to capture the boundaries in various situations, such as intensity variations within the objects and background fragments connected to the objects.
Revised: Both GAC and Grabcut fail to capture the boundaries in various situations, such as intensity variations within the objects and background fragments connected to the objects.

Original: This demonstrates that a precise labelling requires many additional user interventions and thus the interest of our lazy labelling.
Revised: This demonstrates that a precise labelling requires many additional user interventions and thus the interest of our lazy labelling.

Original:  As shown in the third column of Figure 9), the Grabcut repeatedly misclassiﬁed small pieces of background as parts of the object especially when two objects are close to each other with some weak boundaries.
Revised: [Figure] As shown in the third column of Figure 10), the Grabcut repeatedly misclassiﬁed small pieces of background as parts of the object especially when two objects are close to each other with some weak boundaries.

Original: The PL learning methods also give this kind of error in the inference phase.
Revised: The PL learning methods also give this kind of error in the inference phase.

Original: This means that the error has been learned by the segmentation network.
Revised: This means that the error has been learned by the segmentation network.

Original: The manual detection and correction of such small misclassiﬁed regions is a tedious task as well, and therefore is a limitation of the PL approach.
Revised: The manual detection and correction of such small misclassiﬁed regions is a tedious task as well, and therefore is a limitation of the PL approach.

Original: [Figure] [Figure]
Revised: Figure 7 shows examples of WL and SL that are used for the segmentation of gland

Original: [Figure]
Revised: [Figure]

Original: Figure 6 shows examples of WL and SL that are used for the segmentation of gland. can be sparse with some  as  are not 
Revised: The labels for separation can be sparse with some images (see the third row) having zero annotation as the instances are not touching each other.