Original: C UT OUT THE ANNOTATOR , KEEP THE CUTOUT : BETTER SEGMENTATION WITH WEAK SUPERVISION
Revised: C UT OUT THE ANNOTATOR , KEEP THE CUTOUT : BETTER SEGMENTATION WITH WEAK SUPERVISION

Original: A BSTRACT
Revised: A BSTRACT

Original: Constructing large, labeled training datasets for segmentation models is an expensive and labor-intensive process.
Revised: Constructing large, labeled training datasets for segmentation models is an expensive and labor-intensive process.

Original: This is a common challenge in machine learning, addressed by methods that require few or no labeled data points such as few-shot learning (FSL) and weakly-supervised learning (WS).
Revised: This is a common challenge in machine learning, addressed by methods that require few or no labeled data points such as few-shot learning (FSL) and weakly-supervised learning (WS).

Original: Such techniques, however, have limitations when applied to image segmentation—FSL methods often produce noisy results and are strongly dependent on which few datapoints are labeled, while WS models struggle to fully exploit rich image information.
Revised: Such techniques, however, have limitations when applied to image segmentation—FSL methods often produce noisy results and are strongly dependent on which few datapoints are labeled, while WS models struggle to fully exploit rich image information.

Original: We propose a framework that fuses FSL and WS for segmentation tasks, enabling users to train high-performing segmentation networks with very few hand-labeled training points.
Revised: We propose a framework that fuses FSL and WS for segmentation tasks, enabling users to train high-performing segmentation networks with very few hand-labeled training points.

Original: We use FSL models as weak sources in a WS framework, requiring a very small set of reference labeled images, and introduce a new WS model that focuses on key areas—areas with contention among noisy labels—of the image to fuse these weak sources.
Revised: We use FSL models as weak sources in a WS framework, requiring a very small set of reference labeled images, and introduce a new WS model that focuses on key areas—areas with contention among noisy labels—of the image to fuse these weak sources.

Original: Empirically, we evaluate our proposed approach over seven well-motivated segmentation tasks.
Revised: Empirically, we evaluate our proposed approach over seven well-motivated segmentation tasks.

Original: We show that our methods can achieve within 1.4 Dice points compared to fully supervised networks while only requiring ﬁve hand-labeled training points.
Revised: We show that our methods can achieve within 1.4 Dice points compared to fully supervised networks while only requiring ﬁve hand-labeled training points.

Original: Compared to existing FSL methods, our approach improves performance by a mean 3.6 Dice points over the next-best method.
Revised: Compared to existing FSL methods, our approach improves performance by a mean 3.6 Dice points over the next-best method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Automated image segmentation has seen rapid improvements with recent developments in deep learning (Li et al., 2018;
Revised: Automated image segmentation has seen rapid improvements with recent developments in deep learning (Li et al., 2018;

Original: Chen et al., 2017; Milletari et al., 2016).
Revised: Chen et al., 2017; Milletari et al., 2016).

Original: Convolutional neural networks (CNNs) achieve high segmentation performance—but  require large training datasets.
Revised: Convolutional neural networks (CNNs) achieve high segmentation performance—but can require large, labeled training datasets.

Original: Acquiring these labels is laborious and slow. Hand-segmenting images is particularlydifﬁcult for medical images, as segmentation is often  3- or 4-dimensional and requires expertise.
Revised: Acquiring training labels is laborious and slow, particularly for medical images where expert segmentation is often required in 3- or 4-dimensions.

Original: While large datasets and pre-trained networks exist for natural images, medical image segmentation is a more targeted task, typically requiring new training sets for every imaging modality, scanner type, anatomical structure, and patient population.
Revised: While large datasets and pre-trained networks exist for natural images, medical image segmentation is a more targeted task, typically requiring new training sets for every imaging modality, scanner type, anatomical structure, and patient population.

Original: Such difﬁculties abound, but the signiﬁcant impact of improved medical image segmentation motivates tackling these challenges (Hesamian et al., 2019).
Revised: Such difﬁculties abound, but the signiﬁcant impact of improved medical image segmentation motivates tackling these challenges (Hesamian et al., 2019).

Original: Many few-shot learning (FSL) approaches have been proposed to mitigate these difﬁculties by training networks using only a few labeled examples.
Revised: Many few-shot learning (FSL) approaches have been proposed to mitigate these difﬁculties by training networks using only a few labeled examples.

Original: For example, data augmentation can reduce the needed amount of labeled data by introducing additional variation into small, labeled training sets through operations such as afﬁne transforms or learned transformations (e.g. deformations learned by GANs)
Revised: For example, data augmentation can reduce the needed amount of labeled data by introducing additional variation into small, labeled training sets through operations such as afﬁne transforms or learned transformations (e.g. deformations learned by GANs)

Original: (Zhao et al., 2019; Eaton-Rosen et al., 2018; C¸ic¸ek et al., 2016).
Revised: (Zhao et al., 2019; Eaton-Rosen et al., 2018; C¸ic¸ek et al., 2016).

Original: Many semi-supervised approaches aim to learn a useful representation from unlabeled data then ﬁne-tune on a fewnumber of manually-annotated images (Chen et al., 2020; Bai et al., 2019; Chen et al., 2019; Chaitanya et al., 2020a).
Revised: Many semisupervised approaches aim to learn a useful representation from unlabeled data then ﬁne-tune on a few manually-annotated images (Chen et al., 2020; Bai et al., 2019; Chen et al., 2019; Chaitanya et al., 2020a).

Original: Finally, other approaches aim to transfer knowledge learned from segmenting one class in order to segment a previously unseen class (Shaban et al., 2017;
Revised: Finally, other approaches aim to transfer knowledge learned from segmenting one class in order to segment a previously unseen class (Shaban et al., 2017;

Original: Rakelly et al., 2018; Roy et al., 2020; Ouyang et al., 2020).
Revised: Rakelly et al., 2018; Roy et al., 2020; Ouyang et al., 2020).

Original: However, these FSL approaches have limitations.
Revised: However, these FSL approaches have limitations.

Original: For example, effective data augmentation often requires model- and task-speciﬁc transformations that are difﬁcult to generalize to all tasks, and transferring knowledge from one segmentation target to another relies on the segmentations of other classes being present in training datasets.
Revised: For example, effective data augmentation often requires model- and task-speciﬁc transformations that are difﬁcult to generalize to all tasks, and transferring knowledge from one segmentation target to another relies on the segmentations of other classes being present in training datasets.

Original: More generally, bringing all of the known domain expertise or structural information to bear is prohibitive.
Revised: More generally, bringing all of the known domain expertise or structural information to bear is prohibitive.

Original: Weak supervision (WS) is a different approach to training without manual labels that takes advantage of multiple weak labeling sources (Mintz et al., 2009; Craven & Kumlien, 1999; Takamatsu et al., 2012; Gupta & Manning, 2014; Ratner et al., 2019).
Revised: Weak supervision (WS) is a different approach to training without manual labels that takes advantage of multiple weak labeling sources (Mintz et al., 2009; Craven & Kumlien, 1999; Takamatsu et al., 2012; Gupta & Manning, 2014; Ratner et al., 2019).

Original: In many WS workﬂows, labeling functions (LFs) are crafted to encode the decision-making processes of experts into small functions that programatically label training data.
Revised: In many WS workﬂows, labeling functions (LFs) are crafted to encode the decision-making processes of experts into small functions that programatically label training data.

Original: LFs can drastically reduce labeling time, but the resulting labels are often noisy and conﬂicting.
Revised: LFs can drastically reduce labeling time, but the resulting labels are often noisy and conﬂicting.

Original: The driving idea behind WS is that the accuracies of and correlations between LFs can be learned without access to ground truth labels, then used to aggregate the LF outputs.
Revised: The driving idea behind WS is that the accuracies of and correlations between LFs can be learned without access to ground truth labels, then used to aggregate the LF outputs.

Original: This WS workﬂow enables rapid construction of training sets without manual annotation.
Revised: This WS workﬂow enables rapid construction of training sets without manual annotation.

Original: Due to the widespread success of weak supervision methods (Dunnmon et al., 2020; Sheng et al., 2020; Bach et al., 2019), weak supervision is an appealing approach for generating  segmentation training sets without relying on hand-labeled data.
Revised: Due to the widespread success of weak supervision methods (Dunnmon et al., 2020; Sheng et al., 2020; Bach et al., 2019), weak supervision is an appealing approach for generating labeled segmentation training sets without relying on hand-labeled data.

Original: However, existing WS frameworks do not readily extend to the segmentation setting for the following two reasons:
Revised: However, existing WS frameworks do not readily extend to the segmentation setting for the following two reasons:

Original: • Injecting knowledge.
Revised: • Injecting knowledge.

Original: It is unclear how users can construct multiple segmentation LFs that inject knowledge for a segmentation task at hand—and are suitable for any image.
Revised: It is unclear how users can construct multiple segmentation LFs that inject knowledge for a segmentation task at hand—and are suitable for any image.

Original: • Modeling LF accuracies.
Revised: • Modeling LF accuracies.

Original: We need to fuse the LFs, but the PGMs of existing WS frameworks rely on estimating the accuracies for each labeling function, which is a problematic metric for segmentation: it is not sensitive to small differences at segmentation borders nor robust to arbitrary image crops (Figure 1, right).
Revised: We need to fuse the LFs, but the probabilistic graphical models (PGMs) of existing WS frameworks rely on estimating the accuracies for each labeling function, which is a problematic metric for segmentation: it is not sensitive to small differences at segmentation borders nor robust to arbitrary image crops (Figure 1, right).

Original: We propose a best-of-both-worlds approach that fuses FSL and WS for segmentation tasks.
Revised: We propose a best-of-both-worlds approach that fuses FSL and WS for segmentation tasks.

Original: Specifically, we use FSL models as LFs, obviating the need to craft expertise-injecting LFs by hand.
Revised: Specifically, we use FSL models as LFs, obviating the need to craft expertise-injecting LFs by hand.

Original: To resolve the difﬁculty in tracking accuracies per LF, which are too coarse of a metric for segmentation, we proposing a new WS PGM.
Revised: To resolve the difﬁculty in tracking accuracies per LF, which are too coarse of a metric for segmentation, we proposing a new WS PGM.

Original: Our PGM introduces conditional terms that “focus” on areas of contention, where LFs disagree, more ﬁnely tracking the LF performance in such areas.
Revised: Our PGM introduces conditional terms that “focus” on areas of contention, where LFs disagree, more ﬁnely tracking the LF performance in such areas.

Original: We validate our approach on seven medical image segmentation tasks.
Revised: We validate our approach on seven medical image segmentation tasks.

Original: We show that our method can achieve within 1.4 Dice points of a fully supervised network using only ﬁve labeled images.
Revised: We show that our method can achieve within 1.4 Dice points of a fully supervised network using only ﬁve labeled images.

Original: We compare our proposed method to ﬁve other few-shot segmentation methods, and ﬁnd that our method exceeds the next-best by a mean 3.6 Dice points.
Revised: We compare our proposed method to ﬁve other few-shot segmentation methods, and ﬁnd that our method exceeds the next-best by a mean 3.6 Dice points.

Original: Finally, we analyze how different FSL methods scale with additional unlabeled training data and compare our PGM to existing WS models.
Revised: Finally, we analyze how different FSL methods scale with additional unlabeled training data and compare our PGM to existing WS models.

Original: There are two primary contributions in this work.
Revised: There are two primary contributions in this work.

Original: First, we present a weak supervision pipeline tailored to the segmentation use case which utilizes a small number of labeled images and many unlabeled images to train a high-performing segmentation network.
Revised: First, we present a weak supervision pipeline tailored to the segmentation use case which utilizes a small number of labeled images and many unlabeled images to train a high-performing segmentation network.

Original: Second, we develop a novel
Revised: Second, we develop a novel

Original: PGM to aggregate noisy segmentation masks and show that the parameters of this model can be learned without access to ground truth labels.
Revised: PGM to aggregate noisy segmentation masks and show that the parameters of this model can be learned without access to ground truth labels.

Original: We show that our new approach to few-shot image segmentation outperforms many competitive baselines.
Revised: We show that our new approach to few-shot image segmentation outperforms many competitive baselines.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Data augmentation  has been shown to enable training on few labeled examples, particularly in medical imaging.
Revised: Data augmentation Data augmentation has been shown to enable training on few labeled examples, particularly in medical imaging.

Original: These augmentations range from simple operations such as afﬁne transforms, elastic transforms, and contrast jitter (C¸ic¸ek et al., 2016; Cires¸an et al., 2011) to more complex learned operations, such as the use of GANs to learn intensity and spatial deformation transforms to generate synthetic medical imagesfrom a few labeled images (Chaitanya et al., 2020b).
Revised: These augmentations range from simple operations such as afﬁne transforms, elastic transforms, and contrast jitter (C¸ic¸ek et al., 2016; Cires¸an et al., 2011) to more complex learned operations, such as the use of GANs to learn intensity and spatial deformation transforms to generate synthetic medical images (Chaitanya et al., 2020b).

Original: Transferring knowledge A popular FSL approach enables prediction on previously unseen classes by leveraging representations learned from prior experience (Snell et al., 2017; Sung et al., 2018).
Revised: Transferring knowledge A popular FSL approach enables prediction on previously unseen classes by leveraging representations learned from prior experience (Snell et al., 2017; Sung et al., 2018).

Original: Applications to segmentation include Rakelly et al.
Revised: Applications to segmentation include Rakelly et al.

Original: ; Shaban et al.
Revised: ; Shaban et al.

Original: ; Dong & Xing (2018).
Revised: ; Dong & Xing (2018).

Original: For example, PANet is a proposed approach for few-shot semantic segmentation that utilizes representations for each semantic class from the support set, then labels query images by propagating labels from the support set in embedding space (Wang et al., 2019).
Revised: For example, PANet is a proposed approach for few-shot semantic segmentation that utilizes representations for each semantic class from the support set, then labels query images by propagating labels from the support set in embedding space (Wang et al., 2019).

Original: Few-shot segmentation was extended speciﬁcally to the medical imaging case by Roy et al.
Revised: Few-shot segmentation was extended speciﬁcally to the medical imaging case by Roy et al.

Original: (2020), where ”squeeze and excite” blocks are used to learn class representations without pretrained networks.
Revised: (2020), where “squeeze and excite” blocks are used to learn class representations without pretrained networks.

Original: However, these approaches rely on the availability of labels of other classes during training.
Revised: However, these approaches rely on the availability of labels of other classes during training.

Original: Self- and semi-supervised learning Self- and semi-supervised methods have been proposed for image segmentation without relying on the availability of other classes during training (Chen et al., 2020; Bai et al., 2019;
Revised: Self- and semi-supervised learning Self- and semi-supervised methods have been proposed for image segmentation without relying on the availability of other classes during training (Chen et al., 2020; Bai et al., 2019;

Original: Chen et al., 2019).
Revised: Chen et al., 2019).

Original: Instead, semi-supervised segmentation methods often pretrain on large unlabeled datasets to learn useful representations without any annotations.
Revised: Instead, semi-supervised segmentation methods often pretrain on large unlabeled datasets to learn useful representations without any annotations.

Original: For example, contrastive learning has been proposed to pretrain CNNs, which are then ﬁne-tuned using a small number of manually segmented images (Chaitanya et al., 2020a).
Revised: For example, contrastive learning has been proposed to pretrain CNNs, which are then ﬁne-tuned using a small number of manually segmented images (Chaitanya et al., 2020a).

Original: Additionally, superpixel segmentations have been proposed to self-supervise networks, which can be used to segment test images without ﬁne tuning (Ouyang et al., 2020).
Revised: Additionally, superpixel segmentations have been proposed to self-supervise networks, which can be used to segment test images without ﬁne tuning (Ouyang et al., 2020).

Original: In the Appendix, we also discuss conditional random ﬁelds and using coarse labels (e.g., bounding boxes, scribbles, and image-level labels) to train segmentation models.
Revised: In the Appendix, we also discuss conditional random ﬁelds and using coarse labels (e.g., bounding boxes, scribbles, and image-level labels) to train segmentation models.

Original:  S EGMENTATION WITH FSL AND WS F USION
Revised:  S EGMENTATION WITH FSL AND WS F USION

Original: We ﬁrst give background on WS and state the problem setting.
Revised: We ﬁrst give background on WS and state the problem setting.

Original: Then, we follow with details of the primary challenges and solutions in each stage of our method.
Revised: Then, we follow with details of the primary challenges and solutions in each stage of our method.

Original: Speciﬁcally, we discuss LFs and a new PGM that fuses FSL with WS.
Revised: Speciﬁcally, we discuss LFs and a new PGM that fuses FSL with WS.

Original: We end with a summary of our proposed workﬂow.
Revised: We end with a summary of our proposed workﬂow.

Original: 3. W EAK S UPERVISION B ACKGROUND
Revised: 3. W EAK S UPERVISION B ACKGROUND

Original: We build our approach off of principles from WS.
Revised: We build our approach off of principles from WS.

Original: We note that WS is an overloaded term in literature.
Revised: We note that WS is an overloaded term in literature.

Original: In this work, we refer to WS as the class of approaches in which multiple noisy labeling sources are aggregated into probabilistic training labels without access to ground truth labels (Ratner et al., 2019).
Revised: In this work, we refer to WS as the class of approaches in which multiple noisy labeling sources are aggregated into probabilistic training labels without access to ground truth labels (Ratner et al., 2019).

Original: The goal of WS is to best utilize multiple sources of noisy signal (i.e. LFs) to label training data without relying on manual annotation.
Revised: The goal of WS is to best utilize multiple sources of noisy signal (i.e. LFs) to label training data without relying on manual annotation.

Original: Each LF takes as input an unlabeled training point and outputs a vote for the label of that training point.
Revised: Each LF takes as input an unlabeled training point and outputs a vote for the label of that training point.

Original: These LFs can replace manual training data annotation, however LF votes can be imperfect and conﬂict with one another.
Revised: These LFs can replace manual training data annotation, however LF votes can be imperfect and conﬂict with one another.

Original: A latent variable model (referred to as a label model ) can be used to model these observed LFs and their relationship to the latent ground truth label.
Revised: A latent variable model (referred to as a label model ) can be used to model these observed LFs and their relationship to the latent ground truth label.

Original: In Figure 2a, we show a simple model where ﬁve LFs each vote on the label for a single unobserved ground truth label.
Revised: In Figure 2a, we show a simple model where ﬁve LFs each vote on the label for a single unobserved ground truth label.

Original: Importantly, the parameters of this model need to be estimated without access to ground truth annotation.
Revised: Importantly, the parameters of this model need to be estimated without access to ground truth annotation.

Original: The learned label model can be then used to produce probabilistic training labels to train a downstream network, referred to as an end model .
Revised: The learned label model can be then used to produce probabilistic training labels to train a downstream network, referred to as an end model .

Original: Building off of this past work, our weak supervision pipeline proceeds in three steps: (1) noisy labels are generated by multiple LFs, (2) the noisy labels are aggregated into weak labels by a probabilistic graphical model (PGM), and (3) the weak labels are used to train a segmentation model (Figure 1).
Revised: Building off of this past work, our weak supervision pipeline proceeds in three steps: (1) noisy labels are generated by multiple LFs, (2) the noisy labels are aggregated into weak labels by a probabilistic graphical model (PGM), and (3) the weak labels are used to train a segmentation model (Figure 1).

Original: Y λ 1 λ 2 λ 3 ... λ(a)
Revised: Y λ 1 λ 2 λ 3 ... λ(a)

Original: Y 0 , Y 0 ,
Revised: Y 0 , Y 0 ,

Original: Y 0 , Y 0 ,
Revised: Y 0 , Y 0 ,

Original: Y 1 , Y 1 ,
Revised: Y 1 , Y 1 ,

Original: Y 1 , Y 1 ,
Revised: Y 1 , Y 1 ,

Original: Y 2 , Y 2 ,
Revised: Y 2 , Y 2 ,

Original: Y 2 , Y 2 ,
Revised: Y 2 , Y 2 ,

Original: Y 3 , Y 3 ,
Revised: Y 3 , Y 3 ,

Original: Y 3 ,
Revised: Y 3 ,

Original: Y 3 ,(b)
Revised: Y 3 ,(b)

Original: Y 0 , 0 λ 10 , 0 λ 20 , 0 λ 30 ,
Revised: Y 0 , 0 λ 10 , 0 λ 20 , 0 λ 30 ,

Original: Y 0 , 1 λ 10 , 1 λ 20 , 1 λ 30 ,
Revised: Y 0 , 1 λ 10 , 1 λ 20 , 1 λ 30 ,

Original: Y 0 , 2 λ 10 , 2 λ 20 , 2 λ 30 ,
Revised: Y 0 , 2 λ 10 , 2 λ 20 , 2 λ 30 ,

Original: Y 0 , 3 λ 10 , 3 λ 20 , 3 λ 30 ,
Revised: Y 0 , 3 λ 10 , 3 λ 20 , 3 λ 30 ,

Original: Y 1 , 0 λ 11 , 0 λ 21 , 0 λ 31 , Y 1 , 1 λ 11 , 1 λ 21 , 1 λ 31 ,
Revised: Y 1 , 0 λ 11 , 0 λ 21 , 0 λ 31 , Y 1 , 1 λ 11 , 1 λ 21 , 1 λ 31 ,

Original: Y 1 , 2 λ 11 , 2 λ 21 , 2 λ 31 ,
Revised: Y 1 , 2 λ 11 , 2 λ 21 , 2 λ 31 ,

Original: Y 1 , 3 λ 11 , 3 λ 21 , 3 λ 31 ,
Revised: Y 1 , 3 λ 11 , 3 λ 21 , 3 λ 31 ,

Original: Y 2 , 0 λ 12 , 0 λ 22 , 0 λ 32 ,
Revised: Y 2 , 0 λ 12 , 0 λ 22 , 0 λ 32 ,

Original: Y 2 , 1 λ 12 , 1 λ 22 , 1 λ 32 ,
Revised: Y 2 , 1 λ 12 , 1 λ 22 , 1 λ 32 ,

Original: Y 2 , 2 λ 12 , 2 λ 22 , 2 λ 32 , Y 2 , 3 λ 12 , 3 λ 22 , 3 λ 32 ,
Revised: Y 2 , 2 λ 12 , 2 λ 22 , 2 λ 32 , Y 2 , 3 λ 12 , 3 λ 22 , 3 λ 32 ,

Original: Y 3 , 0 λ 13 , 0 λ 23 , 0 λ 33 ,
Revised: Y 3 , 0 λ 13 , 0 λ 23 , 0 λ 33 ,

Original: Y 3 , 1 λ 13 , 1 λ 23 , 1 λ 33 ,
Revised: Y 3 , 1 λ 13 , 1 λ 23 , 1 λ 33 ,

Original: Y 3 , 2 λ 13 , 2 λ 23 , 2 λ 33 ,
Revised: Y 3 , 2 λ 13 , 2 λ 23 , 2 λ 33 ,

Original: Y 3 , 3 λ 13 , 3 λ 23 , 3 λ 33 ,(c)
Revised: Y 3 , 3 λ 13 , 3 λ 23 , 3 λ 33 ,(c)

Original: 3. P ROBLEM S ETTING
Revised: 3. P ROBLEM S ETTING

Original: We are given a training set of n images, X =
Revised: We are given a training set of n images, X =

Original: [ X 1 , X 2 , ..., X n ] , where X i ∈
Revised: [ X 1 , X 2 , ..., X n ] , where X i ∈

Original: R I × J × K × T is a tensor of P pixels.
Revised: R I × J × K × T is a tensor of P pixels.

Original: I, J, K, and T represent the dimensions of image
Revised: I, J, K, and T represent the dimensions of image

Original: X i , where K > 1 for volumetric images and T > 1 for time series images.
Revised: X i , where K > 1 for volumetric images and T > 1 for time series images.

Original: For each image X i there is a segmentation mask
Revised: For each image X i there is a segmentation mask

Original: Y i ∈ { c } I × J × K × T , where { c } is the set of class labels, yielding the set Y =
Revised: Y i ∈ { c } I × J × K × T , where { c } is the set of class labels, yielding the set Y =

Original: [ Y 1 , Y 2 , ..., Y n ] of all unobserved ground truth segmentation masks.
Revised: [ Y 1 , Y 2 , ..., Y n ] of all unobserved ground truth segmentation masks.

Original: We have access to m labeling functions that provide noisy segmentation masks λ li ∈ { c } I × J × K × T for each X i , where l ∈ { 1 , 2 , ..., m } .
Revised: We have access to m labeling functions that provide noisy segmentation masks λ li ∈ { c } I × J × K × T for each X i , where l ∈ { 1 , 2 , ..., m } .

Original: From these observed sources, we seek to generate the set of weak labels ˆY to estimate Y .
Revised: From these observed sources, we seek to generate the set of weak labels ˆY to estimate Y .

Original: ˆY provides labels for a large dataset, and we can use these to train any end model (e.g., a segmentation CNN).
Revised: ˆY provides labels for a large dataset, and we can use these to train any end model (e.g., a segmentation CNN).

Original: We tackle the binary segmentation case, { c } = {− 1 , 1 } ; simple multiclass extensions are possible.
Revised: We tackle the binary segmentation case, { c } = {− 1 , 1 } ; simple multiclass extensions are possible.

Original: Model Structure The (unobserved) ground truth mask Y and the (observed) sources λ are jointly modeled by an undirected graphical model (Koller & Friedman, 2009; Wainwright & Jordan, 2008) with graph G =
Revised: Model Structure The (unobserved) ground truth mask Y and the (observed) sources λ are jointly modeled by an undirected graphical model (Koller & Friedman, 2009; Wainwright & Jordan, 2008) with graph G =

Original: ( V, E ) .
Revised: ( V, E ) .

Original: Since Y is unobserved, this is a latent-variable model.
Revised: Since Y is unobserved, this is a latent-variable model.

Original: We use conventional structures for each component and combine them in a natural way:
Revised: We use conventional structures for each component and combine them in a natural way:

Original: • In a standard weak supervision approach, the m weak sources are connected to a groundtruth label Y (Fig. 2a), • The segmentation mask Y is a multi-dimensional grid over the pixels (Fig. 2b), • Our full model has one copy of the WS graph for each pixel in the grid (Fig. 2c).
Revised: • In a standard weak supervision approach, the m weak sources are connected to a groundtruth label Y (Fig. 2a), • The segmentation mask Y is a multi-dimensional grid over the pixels (Fig. 2b), • Our full model has one copy of the WS graph for each pixel in the grid (Fig. 2c).

Original: Our goal is to construct effective LFs and to complete the model speciﬁcation above by deﬁning the potentials associated with the edges in E and the corresponding parameters.
Revised: Our goal is to construct effective LFs and to complete the model speciﬁcation above by deﬁning the potentials associated with the edges in E and the corresponding parameters.

Original: 3. I NJECTING KNOWLEDGE WITH FSL- BASED LABELING FUNCTIONS
Revised: 3. I NJECTING KNOWLEDGE WITH FSL- BASED LABELING FUNCTIONS

Original: In Step 1 of our workﬂow (Figure 1), we use LFs to generate multiple noisy segmentation masks for each unlabeled image.
Revised: In Step 1 of our workﬂow (Figure 1), we use LFs to generate multiple noisy segmentation masks for each unlabeled image.

Original: These LFs transfer the decision-making processes about labeling data into functions that can be run programmatically over unlabeled training data.
Revised: These LFs transfer the decision-making processes about labeling data into functions that can be run programmatically over unlabeled training data.

Original: Challenge LF development typically involves writing short programs expressing heuristics (Ratner et al., 2018) or external knowledge base lookups (Mintz et al., 2009; Takamatsu et al., 2012).
Revised: Challenge LF development typically involves writing short programs expressing heuristics (Ratner et al., 2018) or external knowledge base lookups (Mintz et al., 2009; Takamatsu et al., 2012).

Original: However, even with computer vision expertise, it is difﬁcult for a new user to write segmentation heuristics as LFs for each new task at hand, as is required of users from existing WS tools.
Revised: However, even with computer vision expertise, it is difﬁcult for a new user to write segmentation heuristics as LFs for each new task at hand, as is required of users from existing WS tools.

Original: Solution We propose FSL models as a uniﬁed way to inject labeling knowledge into LFs.
Revised: Solution We propose FSL models as a uniﬁed way to inject labeling knowledge into LFs.

Original: In this work, we use one-shot learning: we train a CNN from a single labeled data point for each LF.
Revised: In this work, we use one-shot learning: we train a CNN from a single labeled data point for each LF.

Original: However, we note that any few-shot model could be used as a labeling function.
Revised: However, we note that any few-shot model could be used as a labeling function.

Original: Fusing FSL and
Revised: Fusing FSL and

Original: WS allows us to build upon the strengths of each:
Revised: WS allows us to build upon the strengths of each:

Original: • From WS, we build from the idea that we can track the noise in multiple labeling sources to aggregate labels.
Revised: • From WS, we build from the idea that we can track the noise in multiple labeling sources to aggregate labels.

Original: We improve upon existing WS tools by showing that few-shot networks—
Revised: We improve upon existing WS tools by showing that few-shot networks—

Original: which obviate the need for users to write their own LFs, as is required by existing WS frameworks—are ﬂexible and strong LFs in a WS segmentation pipeline.
Revised: which obviate the need for users to write their own LFs, as is required by existing WS frameworks—are ﬂexible and strong LFs in a WS segmentation pipeline.

Original: • From FSL, we take ideas for how to train networks with very few labeled images.
Revised: • From FSL, we take ideas for how to train networks with very few labeled images.

Original: We improve upon these methods by tracking the noise in each FSL model with WS.
Revised: We improve upon these methods by tracking the noise in each FSL model with WS.

Original: Our FSL-based LFs depart from the domain-expert LF development approach, which is often repeated for each task.
Revised: Our FSL-based LFs depart from the domain-expert LF development approach, which is often repeated for each task.

Original: Instead, the same process is followed for all tasks explored here: ﬁve images are annotated by hand, each of which are used to train a one-shot model.
Revised: Instead, the same process is followed for all tasks explored here: ﬁve images are annotated by hand, each of which are used to train a one-shot model.

Original: The one-shot models function as LFs that weakly label the rest of the training set.
Revised: The one-shot models function as LFs that weakly label the rest of the training set.

Original: In Section A.2 of the Appendix, we discuss other segmentation LFs we considered.
Revised: In Section A.2 of the Appendix, we discuss other segmentation LFs we considered.

Original: 3. M ODELING ACCURACIES WITH CONTENTION - BASED PROBABILISTIC MODEL
Revised: 3. M ODELING ACCURACIES WITH CONTENTION - BASED PROBABILISTIC MODEL

Original: In Step 2 of our workﬂow (Figure 1), we aggregate the noisy segmentation masks produced by the
Revised: In Step 2 of our workﬂow (Figure 1), we aggregate the noisy segmentation masks produced by the

Original: LFs into probabilistic training labels.
Revised: LFs into probabilistic training labels.

Original: Challenge To aggregate the outputs of the LFs, we need to deﬁne the graph structure G (given in
Revised: Challenge To aggregate the outputs of the LFs, we need to deﬁne the graph structure G (given in

Original: Figure 2c) and the potentials/parameters associated with the edges in G .
Revised: Figure 2c) and the potentials/parameters associated with the edges in G .

Original: However, it is not obvious how to do so.
Revised: However, it is not obvious how to do so.

Original: On one extreme, separate parameters for each edge in the model in Figure 2c easily results in millions of parameters to be learned.
Revised: On one extreme, separate parameters for each edge in the model in Figure 2c easily results in millions of parameters to be learned.

Original: At the other extreme, treating each pixel as independent of all other pixels (as in separate copies of Figure 2a) results in the problems summarized in
Revised: At the other extreme, treating each pixel as independent of all other pixels (as in separate copies of Figure 2a) results in the problems summarized in

Original: Figure 1 (right): the parameters tend to be equivalent, since all LF accuracies are high and similar, and can be changed by arbitrary crops.
Revised: Figure 1 (right): the parameters tend to be equivalent, since all LF accuracies are high and similar, and can be changed by arbitrary crops.

Original: Solution Our approach hits a sweet spot.
Revised: Solution Our approach hits a sweet spot.

Original: It exploits a notion of LF contention , trading off between the naive model and the highly-complex model that has no parameter-tying.
Revised: It exploits a notion of LF contention , trading off between the naive model and the highly-complex model that has no parameter-tying.

Original: Model Potentials, Parameters, and Contention
Revised: Model Potentials, Parameters, and Contention

Original: The latent variable model in Figure 2a expresses the relationship between the sources and unobserved ground truth label.
Revised: The latent variable model in Figure 2a expresses the relationship between the sources and unobserved ground truth label.

Original: The potentials are typically those from a binary Ising model (Bach et al., 2017; Ratner et al., 2019; Varma et al., 2019; Fu et al., 2020).
Revised: The potentials are typically those from a binary Ising model (Bach et al., 2017; Ratner et al., 2019; Varma et al., 2019; Fu et al., 2020).

Original: The density function is deﬁned as,
Revised: The density function is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: where θ Y ∈ R and θ i ∈ R .
Revised: where θ Y ∈ R and θ i ∈ R .

Original: Intuitively, the θ i parameters control the accuracies of the sources.
Revised: Intuitively, the θ i parameters control the accuracies of the sources.

Original: If θ i = 0 , then λ i is independent of Y —and thus can only output a random guess of Y .
Revised: If θ i = 0 , then λ i is independent of Y —and thus can only output a random guess of Y .

Original: As θ i increases, λ i and Y vote together more often, and so λ i is more accurate.
Revised: As θ i increases, λ i and Y vote together more often, and so λ i is more accurate.

Original: For this reason, it is crucial to estimate the parameters of this model; naively combining the noisy labels will over-weight the inaccurate sources.
Revised: For this reason, it is crucial to estimate the parameters of this model; naively combining the noisy labels will over-weight the inaccurate sources.

Original: In our segmentation structure (Fig. 2c), there are many copies of this model.
Revised: In our segmentation structure (Fig. 2c), there are many copies of this model.

Original: Using unique parameters for each copy yields an intractably complex model.
Revised: Using unique parameters for each copy yields an intractably complex model.

Original: Tying parameters naively (i.e., each LF has the same accuracy across all pixels) is too coarse—even a poor LF will typically be very accurate across the entire image.
Revised: Tying parameters naively (i.e., each LF has the same accuracy across all pixels) is too coarse—even a poor LF will typically be very accurate across the entire image.

Original: To tackle this, we introduce a notion of LF contention into our density function.
Revised: To tackle this, we introduce a notion of LF contention into our density function.

Original: Speciﬁcally, we use indicator variables that depend on LFs agreeing or disagreeing on a given pixel.
Revised: Speciﬁcally, we use indicator variables that depend on LFs agreeing or disagreeing on a given pixel.

Original: Intuitively, this will allow the model to learn a certain set of parameters when the
Revised: Intuitively, this will allow the model to learn a certain set of parameters when the

Original: LFs are all agreeing (e.g. image background) vs. when the LFs have some disagreement (e.g. segmentation border).
Revised: LFs are all agreeing (e.g. image background) vs. when the LFs have some disagreement (e.g. segmentation border).

Original: By tying parameters based on LF agreements, this distribution allots additional model parameters to areas of contention without introducing too many free parameters to model the grid-structured PGM.
Revised: By tying parameters based on LF agreements, this distribution allots additional model parameters to areas of contention without introducing too many free parameters to model the grid-structured PGM.

Original: Moreover, this approach enables differential model parameters to be learned even when global LF accuracies are highly similar, and is robust to arbitrary image crops.
Revised: Moreover, this approach enables differential model parameters to be learned even when global LF accuracies are highly similar, and is robust to arbitrary image crops.

Original: We refer to a set of LF agreement conditions over some LF subset D as C D .
Revised: We refer to a set of LF agreement conditions over some LF subset D as C D .

Original: C D exhaustively enumerates each possible combination of LF agreements and disagreements, C i .
Revised: C D exhaustively enumerates each possible combination of LF agreements and disagreements, C i .

Original: For example, the
Revised: For example, the

Original: C D for D = { λ 1 , λ 2 , λ 3 } is given by
Revised: C D for D = { λ 1 , λ 2 , λ 3 } is given by

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: C 1 =
Revised: C 1 =

Original: [ λ 1 (cid:54) =
Revised: [ λ 1 (cid:54) =

Original: λ 2 , λ 1 (cid:54) =
Revised: λ 2 , λ 1 (cid:54) =

Original: λ 3 , λ
Revised: λ 3 , λ

Original: =
Revised: =

Original: λ 3 ] , C 2 =
Revised: λ 3 ] , C 2 =

Original: [ λ 1 (cid:54) =
Revised: [ λ 1 (cid:54) =

Original: λ 2 , λ 1 = λ 3 , λ 2 (cid:54) = λ 3 ]
Revised: λ 2 , λ 1 = λ 3 , λ 2 (cid:54) = λ 3 ]

Original: [Equation]
Revised: [Equation]

Original: A technical detail we exploit for learning: we always leave two LFs out of the contention set D to ensure sufﬁcient conditional independence for parameter recovery, as we describe next.
Revised: A technical detail we exploit for learning: we always leave two LFs out of the contention set D to ensure sufﬁcient conditional independence for parameter recovery, as we describe next.

Original: The set of these left-out LFs is I ; if there are m = 5 LFs, then I = { λ 4 , λ 5 } .
Revised: The set of these left-out LFs is I ; if there are m = 5 LFs, then I = { λ 4 , λ 5 } .

Original: Formally, we deﬁne the density of the contention-based model as
Revised: Formally, we deﬁne the density of the contention-based model as

Original: [Equation]
Revised: [Equation]

Original: where each canonical parameter θ i ∈ R .
Revised: where each canonical parameter θ i ∈ R .

Original: In practice we only sum over a single element of D , which is an equivalent but minimal distribution.
Revised: In practice we only sum over a single element of D , which is an equivalent but minimal distribution.

Original: enabling backwards mapping as described below.
Revised: enabling backwards mapping as described below.

Original: Learning & Inference
Revised: Learning & Inference

Original: Given observed values of LFs over a set of images, we wish to learn the parameters θ from (2).
Revised: Given observed values of LFs over a set of images, we wish to learn the parameters θ from (2).

Original: Critically, this must be done without relying on ground truth segmentation masks—so that only the λ variables, and not Y , are observed.
Revised: Critically, this must be done without relying on ground truth segmentation masks—so that only the λ variables, and not Y , are observed.

Original: We use an extension of Fu et al.
Revised: We use an extension of Fu et al.

Original: (2020)’s method-of-moments -based approach.
Revised: (2020)’s method-of-moments -based approach.

Original: For the model (1), E [ λ i λ j ] =
Revised: For the model (1), E [ λ i λ j ] =

Original: E [ λ i Y ] E [ λ j Y ] .
Revised: E [ λ i Y ] E [ λ j Y ] .

Original: The left-hand side can be estimated (as we observe the LFs); a system with three such equations, for the pairs ( i, j ) , ( i, k ) , ( j, k ) , determines E
Revised: The left-hand side can be estimated (as we observe the LFs); a system with three such equations, for the pairs ( i, j ) , ( i, k ) , ( j, k ) , determines E

Original: [ λ i Y ] up to sign.
Revised: [ λ i Y ] up to sign.

Original: These terms are the mean parameters of the model, and under mild technical conditions, they determine the canonical parameters θ (Wainwright & Jordan, 2008).
Revised: These terms are the mean parameters of the model, and under mild technical conditions, they determine the canonical parameters θ (Wainwright & Jordan, 2008).

Original: Our extension incorporates the contention term into the decomposition.
Revised: Our extension incorporates the contention term into the decomposition.

Original: The full details are in the Appendix.
Revised: The full details are in the Appendix A.3.

Original: After solving for the label model parameters, our goal is to compute the marginal probability for each pixel Y pi given the observed values of the labeling functions for all pixels in our graph.
Revised: After solving for the label model parameters, our goal is to compute the marginal probability for each pixel Y pi given the observed values of the labeling functions for all pixels in our graph.

Original: Various methods have been proposed for performing inference over a large and cyclical graphical model.
Revised: Various methods have been proposed for performing inference over a large and cyclical graphical model.

Original: Here, we approximate the exact conditional using a common graph-cut algorithm.
Revised: Here, we approximate the exact conditional using a common graph-cut algorithm.

Original: We learn the parameters and compute the unary potentials at each pixel, treating each pixel as independent at this stage.
Revised: We learn the parameters and compute the unary potentials at each pixel, treating each pixel as independent at this stage.

Original: The max-ﬂow/min-cut algorithm proposed by Boykov & Kolmogorov (2004) is used to efﬁciently compute the ﬁnal segmentation mask considering pairwise interaction between all neighboring pixels.
Revised: The max-ﬂow/min-cut algorithm proposed by Boykov & Kolmogorov (2004) is used to efﬁciently compute the ﬁnal segmentation mask considering pairwise interaction between all neighboring pixels.

Original: However, we note that many approximate inference algorithms could be used as well, including approximate belief propagation algorithms that would return probabilistic labels.
Revised: However, we note that many approximate inference algorithms could be used as well, including approximate belief propagation algorithms that would return probabilistic labels.

Original: 3. S UMMARY OF WORKFLOW
Revised: 3. S UMMARY OF WORKFLOW

Original: Taking the solutions developed in Sections 3.3 and 3.4, we put the pieces together and describe the full workﬂow of our proposed approach (Figure 3).
Revised: Taking the solutions developed in Sections 3.3 and 3.4, we put the pieces together and describe the full workﬂow of our proposed approach (Figure 3).

Original: • Step 1.
Revised: • Step 1.

Original: Each one of the ﬁve manually annotated images is used to train a one-shot CNN.
Revised: Each one of the ﬁve manually annotated images is used to train a one-shot CNN.

Original: With ﬁve manually annotated images, there are ﬁve one-shot CNNs.
Revised: With ﬁve manually annotated images, there are ﬁve one-shot CNNs.

Original: Each of these one-shot CNNs is a labeling function whichnoisily labels all unlabeled images in the training set.
Revised: Each of these one-shot CNNs is a labeling function which labels all unlabeled images in the training set.

Original: • Step 2.
Revised: • Step 2.

Original: A PGM is used to aggregate the ﬁve noisy segmentation masks for each image in the unlabeled training set.
Revised: A PGM is used to aggregate the ﬁve noisy segmentation masks for each image in the unlabeled training set.

Original: The parameters of this PGM are learned for each dataset without using ground truth segmentation masks.
Revised: The parameters of this PGM are learned for each dataset without using ground truth segmentation masks.

Original: • Step 3.
Revised: • Step 3.

Original: A CNN is trained using the estimated segmentation masks output from the PGM, then ﬁne-tuned with the few hand-labeled images.
Revised: A CNN is trained using the estimated segmentation masks output from the PGM, then ﬁne-tuned with the few hand-labeled images.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We describe our experimental setup in Section 4.1.
Revised: We describe our experimental setup in Section 4.1.

Original: The primary goal of our evaluation is to show that our proposed weak supervision method can achieve high performance across a range of medical image segmentation tasks (Section 4.2).
Revised: The primary goal of our evaluation is to show that our proposed weak supervision method can achieve high performance across a range of medical image segmentation tasks (Section 4.2).

Original: Secondly, we aim to more deeply understand the beneﬁts of the proposed method compared to other methods (Section 4.3).
Revised: Secondly, we aim to more deeply understand the beneﬁts of the proposed method compared to other methods (Section 4.3).

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: Datasets We select public segmentation datasets to evaluate our proposed method.
Revised: Datasets We select public segmentation datasets to evaluate our proposed method.

Original: We chose datasets to include varied imaging modalities, body regions, and structures.
Revised: We chose datasets to include varied imaging modalities, body regions, and structures.

Original: • ACDC.
Revised: • ACDC.

Original: 100 volumetric short-axis cardiac magnetic resonance (CMR) cine images available for download, totaling 951 2D axial slices.
Revised: 100 volumetric short-axis cardiac magnetic resonance (CMR) cine images available for download, totaling 951 2D axial slices.

Original: The right ventricle, the left ventricle endocardium, and the myocardium segmentations were provided (Bernard et al., 2018).
Revised: The right ventricle, the left ventricle endocardium, and the myocardium segmentations were provided (Bernard et al., 2018).

Original: • Abdominal CT.
Revised: • Abdominal CT.

Original: 43 contrast-enhanced abdominal computed tomography (ACT) scans, totalling 10,235 2D axial slices.
Revised: 43 contrast-enhanced abdominal computed tomography (ACT) scans, totaling 10,235 2D axial slices.

Original: We use the segmentations provided for the spleen, esophagus, and liver (Gibson et al., 2018; Roth et al., 2016; 2015; Clark et al., 2013).
Revised: We use the segmentations provided for the spleen, esophagus, and liver (Gibson et al., 2018; Roth et al., 2016; 2015; Clark et al., 2013).

Original: • BRATS.
Revised: • BRATS.

Original: 210 head magnetic resonance (HMR) scans, totalling 31,775 axial slices.
Revised: 210 head magnetic resonance (HMR) scans, totaling 31,775 axial slices.

Original: We use the ﬂair HMR sequence and segment any region positive for tumor (Menze et al., 2014; Bakas et al., 2017; 2018).
Revised: We use the ﬂair HMR sequence and segment any region positive for tumor (Menze et al., 2014; Bakas et al., 2017; 2018).

Original: Baselines We compare our proposed approach to ﬁve other few-shot segmentation methods.
Revised: Baselines We compare our proposed approach to ﬁve other few-shot segmentation methods.

Original: We select the baselines that have similar inputs (a few labeled images and many unlabeled images) and outputs (a trained segmentation neural network) to our proposed approach, but use a distinct method, and have published implementations by the authors.
Revised: We select the baselines that have similar inputs (a few labeled images and many unlabeled images) and outputs (a trained segmentation neural network) to our proposed approach, but use a distinct method, and have published implementations by the authors.

Original: In each case, we use the same unlabeled and ﬁve labeled images we use in our proposed approach to ensure the training datasets are comparable.
Revised: In each case, we use the same unlabeled and ﬁve labeled images we use in our proposed approach to ensure the training datasets are comparable.

Original: • Data augmentation.
Revised: • Data augmentation.

Original: We compare against a data augmentation method that trains generative adversarial networks (GANs) using unlabeled images, then transforms the labeled images using the trained GANs to generate a larger labeled training set (”GAN DA”)
Revised: We compare against a data augmentation method that trains generative adversarial networks (GANs) using unlabeled images, then transforms the labeled images using the trained GANs to generate a larger labeled training set (“GAN DA”)

Original: (Chaitanya et al., 2020b).
Revised: (Chaitanya et al., 2020b).

Original: • Knowledge transfer.
Revised: • Knowledge transfer.

Original: We compare against PANet (Wang et al., 2019), a FSL approach to semantic segmentation developed for natural images.
Revised: We compare against PANet (Wang et al., 2019), a FSL approach to semantic segmentation developed for natural images.

Original: We also compare against Squeeze and Excite (“S&E”), a few-shot approach developed for medical images (Roy et al., 2020).
Revised: We also compare against Squeeze and Excite (“S&E”), a few-shot approach developed for medical images (Roy et al., 2020).

Original: • Semi- and self-supervised learning (SSL).
Revised: • Semi- and self-supervised learning (SSL).

Original: We compare against a semi-supervised method that pretrains using a contrastive loss over many unlabeled images, then ﬁne-tunes on a few manually-annotated images (”CL”)
Revised: We compare against a semi-supervised method that pretrains using a contrastive loss over many unlabeled images, then ﬁne-tunes on a few manually-annotated images (“CL”)

Original: (Chaitanya et al., 2020a).
Revised: (Chaitanya et al., 2020a).

Original: We also compare against SSL-ALPNet, which utilizes a self-supervised superpixel segmentation task then uses the learned representations to segment new classes without ﬁne-tuning (Ouyang et al., 2020).
Revised: We also compare against SSL-ALPNet, which utilizes a self-supervised superpixel segmentation task then uses the learned representations to segment new classes without ﬁne-tuning (Ouyang et al., 2020).

Original: • Traditional supervision.
Revised: • Traditional supervision.

Original: We train a model using only the ﬁve labeled reference images as the lower bound of expected performance (”FS-5”).
Revised: We train a model using only the ﬁve labeled reference images as the lower bound of expected performance (“FS-5”).

Original: We also train a fully supervised network using all ground truth segmentations of all training data as the upper bound of expected performance (”FS-all”).
Revised: We also train a fully supervised network using all ground truth segmentations of all training data as the upper bound of expected performance (“FS-all”).

Original: [Table]
Revised: [Table]

Original: Training and Evaluation Each dataset is split into 60% training, 20% validation, and 20% testing.
Revised: Training and Evaluation Each dataset is split into 60% training, 20% validation, and 20% testing.

Original: Image preprocessing is described in the Appendix.
Revised: Image preprocessing is described in the Appendix A.4.

Original: For the LFs, FS-5, FS-all, and our proposed approach, we train a 2D U-Net (Ronneberger et al., 2015) to minimize the binary cross entropy loss using the Adam optimizer with learning rate set to 1e-3.
Revised: For the LFs, FS-5, FS-all, and our proposed approach, we train a 2D U-Net (Ronneberger et al., 2015) to minimize the binary cross entropy loss using the Adam optimizer with learning rate set to 1e-3.

Original: Batch size was set to 16.
Revised: Batch size was set to 16.

Original: Random afﬁne transforms, color jitter, and elastic transforms were used during training for all models except FS-5.
Revised: Random afﬁne transforms, color jitter, and elastic transforms were used during training for all models except FS-5.

Original: We note that the performance of FS-all will be lower than that reported in public leaderboards due to our simple end model.
Revised: We note that the performance of FS-all will be lower than that reported in public leaderboards due to our simple end model.

Original: Because we aim to show how well our proposed method utilizes unlabeled data and a small set of labeled data, we keep the end model simple to be comparable to baselines.
Revised: Because we aim to show how well our proposed method utilizes unlabeled data and a small set of labeled data, we keep the end model simple to be comparable to baselines.

Original: More complex networks could easily be substituted into Step 3.
Revised: More complex networks could easily be substituted into Step 3.

Original: The Dice coefﬁcient, Dice ( Y i , Y j ) = 

Revised: The Dice coefﬁcient, Dice ( Y i , Y j ) =

Original: | Y i ∩ Y
Revised: | Y i ∩ Y

Original: j | | Y i | + | Y j | , is used to evaluate networks.
Revised: j | | Y i | + | Y j | , is used to evaluate networks.

Original: Each network was trained with three different random seeds.
Revised: Each network was trained with three different random seeds.

Original: The mean (standard deviation) Dice scores are reported.
Revised: The mean (standard deviation) Dice scores are reported.

Original: 4. P ERFORMANCE OF PROPOSED METHOD 4.2. C OMPARISON TO BASELINES
Revised: 4. P ERFORMANCE OF PROPOSED METHOD 4.2. C OMPARISON TO BASELINES

Original: The Dice scores of our proposed approach and baselines are shown for the seven segmentation tasks in Table 1; visualizations of the segmentations are in Figure 4 and in the Appendix Figure A.1.
Revised: The Dice scores of our proposed approach and baselines are shown for the seven segmentation tasks in Table 1; visualizations of the segmentations are in Figure 4 and in the Appendix Figure A.1.

Original: Despite using only ﬁve labeled images, we obtain near fully supervised performance on the endocardium, liver, and spleen, achieving on average within 1.9 points of the fully supervised network.
Revised: Despite using only ﬁve labeled images, we obtain near fully supervised performance on the endocardium, liver, and spleen, achieving on average within 1.9 points of the fully supervised network.

Original: On the more difﬁcult segmentation tasks, we still see boost compared to existing approaches, though there is larger gap between FS and WS performance.
Revised: On the more difﬁcult segmentation tasks, we still see boost compared to existing approaches, though there is larger gap between FS and WS performance.

Original: This is to be expected, since the segmentation of smaller structures is more difﬁcult.
Revised: This is to be expected, since the segmentation of smaller structures is more difﬁcult.

Original: Still, we see strong agreement in the shape and location of the
Revised: Still, we see strong agreement in the shape and location of the

Original: WS segmentations compared to the ground truth segmentations.
Revised: WS segmentations compared to the ground truth segmentations.

Original: 4.2. I NFLUENCE OF SELECTED LABELED IMAGES
Revised: 4.2. I NFLUENCE OF SELECTED LABELED IMAGES

Original: We evaluate how our results change depending on the ﬁve manually-annotated images.
Revised: We evaluate how our results change depending on the ﬁve manually-annotated images.

Original: The original set of manually-annotated images were selected randomly.
Revised: The original set of manually-annotated images were selected randomly.

Original: We select three additional sets of reference images and re-run our proposed method; all sets are mutually exclusive.
Revised: We select three additional sets of reference images and re-run our proposed method; all sets are mutually exclusive.

Original: The results from each new set (run with the same random seed) are shown in Table 2.
Revised: The results from each new set (run with the same random seed) are shown in Table 2.

Original: xIn general, we see good agreement across different randomly-selected labeled images.
Revised: In general, we see good agreement across different randomly-selected labeled images.

Original: For all datasets except the right ventricle and the esophagus, we see standard deviations  less than 1.5 Dice points.
Revised: For all datasets except the right ventricle and the esophagus, we see standard deviations of less than 1.5 Dice points.

Original: Surfacing images to label from an unlabeled dataset to improve end model results would be an interesting direction for future work.
Revised: Selecting images to label from an unlabeled dataset to improve end model results would be an interesting direction for future work.

Original: [Table]
Revised: [Table]

Original: 4. T RADEOFFS WITH OTHER APPROACHES
Revised: 4. T RADEOFFS WITH OTHER APPROACHES

Original: Performance scaling with amount of unlabeled data Since unlabeled data is much easier to collect than manually-annotated data, FSL approaches that are able to effectively utilize unlabeled data may have an advantage when deployed to real-world applications.
Revised: Performance scaling with amount of unlabeled data Since unlabeled data is much easier to collect than manually-annotated data, FSL approaches that are able to effectively utilize unlabeled data may have an advantage when deployed to real-world applications.

Original: In Figure 5 (left), we show the performance of our proposed approach, the highest performing knowledge transfer approach (PANet), the data augmentation network (GAN-DA), and the highest performing self- or semisupervised network (CL) as more unlabeled data is provided during training for the CMR right ventricle task.
Revised: In Figure 5 (left), we show the performance of our proposed approach, the highest performing knowledge transfer approach (PANet), the data augmentation network (GAN-DA), and the highest performing self- or semisupervised network (CL) as more unlabeled data is provided during training for the CMR right ventricle task.

Original: We observe that the performance separation between WS and other approaches is larger with more unlabeled data.
Revised: We observe that the performance separation between WS and other approaches is larger with more unlabeled data.

Original: In the low data regime, WS, CL, and GAN-DA perform similarly.
Revised: In the low data regime, WS, CL, and GAN-DA perform similarly.

Original: Label model performance To show the advantages of the proposed segmentation LM versus existing WS models, we conduct an experiment with synthetic datasets.
Revised: Label model performance To show the advantages of the proposed segmentation LM versus existing WS models, we conduct an experiment with synthetic datasets.

Original: We generate a dataset according to Equation 2, and pad this dataset with negatively labeled examples.
Revised: We generate a dataset according to Equation 2, and pad this dataset with negatively labeled examples.

Original: This process mimics cropping an image around the ROI.
Revised: This process mimics cropping an image around the ROI.

Original: We show the performance of our proposed LM, majority vote (MV), and a leading existing WS framework (Flying Squid, Fu et al.
Revised: We show the performance of our proposed LM, majority vote (MV), and a leading existing WS framework (Flying Squid, Fu et al.

Original: in Figure 5 (right).
Revised: in Figure 5 (right).

Original: We see our proposed model is unaffected by background cropping (or equivalently, LF disagreement rate) while the existing WS tool has varying performance as a function of arbitrary crops.
Revised: We see our proposed model is unaffected by background cropping (or equivalently, LF disagreement rate) while the existing WS tool has varying performance as a function of arbitrary crops.

Original: We report the Dice score of MV, the proposed LM, and Flying Squid in the Appendix.
Revised: We report the Dice score of MV, the proposed LM, and Flying Squid in the Appendix A.4.

Original: In the Appendix, we also include an experiment on these FSL models’ generalization performance.
Revised: Finally, in Appendix A.4, we also include an experiment on the generalization performance of the FSL models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Automated medical image segmentation is a difﬁcult problem.
Revised: Automated medical image segmentation is a difﬁcult problem.

Original: There are numerous imaging modalities, views, and structures to segment and each applications demands high performance, leading to extensive labeling costs from experts.
Revised: There are numerous imaging modalities, views, and structures to segment and each application demands high performance, leading to extensive labeling costs from experts.

Original: We present a general approach to training segmentation networks with limited hand labels, fusing FSL and WS to build upon the beneﬁts of each.
Revised: We present a general approach to training segmentation networks with limited hand labels, fusing FSL and WS to build upon the beneﬁts of each.

Original: This approach is ﬂexible, capable of utilizing various forms of noisy signal to weakly label training data, which can in turn be used to train any downstream model.
Revised: This approach is ﬂexible, capable of utilizing various forms of noisy signal to weakly label training data, which can in turn be used to train any downstream model.

Original: Our method is compatible with future improvements in segmentation methods: new few-shot approaches can be used as LFs in Step 1, and more complex segmentation networks or loss functions can be used in Step 3.
Revised: Our method is compatible with future improvements in segmentation methods: new few-shot approaches can be used as LFs in Step 1, and more complex segmentation networks or loss functions can be used in Step 3.

Original: For example, unsupervised pre-training could be used to initialize the one-shot networks used as LFs, such as the methods proposed in Chaitanya et al.
Revised: For example, unsupervised pre-training could be used to initialize the one-shot networks used as LFs, such as the methods proposed in Chaitanya et al.

Original: (2020a).
Revised: (2020a).

Original: In fact, any of the baselines we compare against in this work could be used  LFs.
Revised: In fact, any of the baselines we compare against in this work could be used as LFs.

Original: Finally, we have shown that the proposed approach to training segmentation networks with few labels outperforms many competitive baselines on a range of different imaging modalities and anatomical structures.
Revised: Finally, we have shown that the proposed approach to training segmentation networks with few labels outperforms many competitive baselines on a range of different imaging modalities and anatomical structures.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Sarah Hooper is supported by the Fannie and John Hertz Foundation, the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518, and as a Texas Instruments
Revised: Sarah Hooper is supported by the Fannie and John Hertz Foundation, the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518, and as a Texas Instruments

Original: Fellow under the Stanford Graduate Fellowship in Science and Engineering.
Revised: Fellow under the Stanford Graduate Fellowship in Science and Engineering.

Original: We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos.
Revised: We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos.

Original: CCF (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ONR
Revised: CCF (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ONR

Original: under No.
Revised: under No.

Original: N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA,
Revised: N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA,

Original: Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss
Revised: Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss

Original: Re, Total, the HAI-AWS Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare.
Revised: Re, Total, the HAI-AWS Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare.

Original: The Mobilize Center is a Biomedical Technology Resource Center, funded by the NIH National Institute of Biomedical Imaging and Bioengineering through Grant P41EB027060.
Revised: The Mobilize Center is a Biomedical Technology Resource Center, funded by the NIH National Institute of Biomedical Imaging and Bioengineering through Grant P41EB027060.

Original: The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.
Revised: The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.

Original: Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
Revised: Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.

Original: A A PPENDIX A. E XTENDED R ELATED W ORK
Revised: A A PPENDIX A. E XTENDED R ELATED W ORK

Original: Coarse labels Coarser forms of annotation per image can be used to train segmentation models with less manual labor.
Revised: Coarse labels Coarser forms of annotation per image can be used to train segmentation models with less manual labor.

Original: For example, bounding boxes, scribbles, and class labels can be used to provide supervision for segmentation models (Sakinis et al., 2019; Xu et al., 2015; Khoreva et al., 2017).
Revised: For example, bounding boxes, scribbles, and class labels can be used to provide supervision for segmentation models (Sakinis et al., 2019; Xu et al., 2015; Khoreva et al., 2017).

Original: This approach is sometimes referred to as ”weak supervision” in the segmentation literature.
Revised: This approach is sometimes referred to as “weak supervision” in the segmentation literature.

Original: We note that this technique is different than the WS approach presented here, as it still requires each training sample to be annotated and there is not a uniﬁed model to aggregate multiple noisy labeling sources.
Revised: We note that this technique is different than the WS approach presented here, as it still requires each training sample to be annotated and there is not a uniﬁed model to aggregate multiple noisy labeling sources.

Original: We refer to weak supervision as the process by which multiple noisy segmentation masks are generated and aggregated automatically.
Revised: We refer to weak supervision as the process by which multiple noisy segmentation masks are generated and aggregated automatically.

Original: Conditional random ﬁelds Traditional approaches to modeling segmentation masks for images use conditional random ﬁelds (CRFs).
Revised: Conditional random ﬁelds Traditional approaches to modeling segmentation masks for images use conditional random ﬁelds (CRFs).

Original: These distributions often utilize unary potentials over each pixel and pairwise potentials between pixels.
Revised: These distributions often utilize unary potentials over each pixel and pairwise potentials between pixels.

Original: Many methods were proposed for deﬁning the unary potentials—including engineering features over the color, location, and texture of regions in the image (Shotton et al., 2009)—as well as pairwise interactions between pixels, such as how smooth segmentations were expected to be (Kohli et al., 2009; Kr ¨ ahenb ¨ uhl & Koltun, 2011).
Revised: Many methods were proposed for deﬁning the unary potentials—including engineering features over the color, location, and texture of regions in the image (Shotton et al., 2009)—as well as pairwise interactions between pixels, such as how smooth segmentations were expected to be (Kohli et al., 2009; Kr ¨ ahenb ¨ uhl & Koltun, 2011).

Original: Additional work was devoted to performing inference over the large CRFs, developing new methods for approximate inference via belief propagation and graph cuts (Kohli et al., 2007; Boykov et al., 2001;
Revised: Additional work was devoted to performing inference over the large CRFs, developing new methods for approximate inference via belief propagation and graph cuts (Kohli et al., 2007; Boykov et al., 2001;

Original: Boykov & Kolmogorov, 2004).
Revised: Boykov & Kolmogorov, 2004).

Original: We note that the graph structure we use (Figure 2c) bears resemblance to those used in past work in CRFs.
Revised: We note that the graph structure we use (Figure 2c) bears resemblance to those used in past work in CRFs.

Original: However, in most traditional CRF frameworks, the parameters of the PGM are learned or tuned from labeled training data.
Revised: However, in most traditional CRF frameworks, the parameters of the PGM are learned or tuned from labeled training data.

Original: Here, we work with a latent ground truth variable Y and generate probabilistic labels for each pixel by learning how to aggregate LF votes without access to ground truth data
Revised: Here, we work with a latent ground truth variable Y and generate probabilistic labels for each pixel by learning how to aggregate LF votes without access to ground truth data.

Original: . A. A LTERNATIVE L ABELING F UNCTIONS
Revised: A. A LTERNATIVE L ABELING F UNCTIONS

Original: We experimented with many different types of LFs in the course of this work.
Revised: We experimented with many different types of LFs in the course of this work.

Original: We focus on the oneshot models used in this work because they were general (i.e., can be applied to any task without requiring user-modiﬁcation) and produced strong results.
Revised: We focus on the oneshot models used in this work because they were general (i.e., can be applied to any task without requiring user-modiﬁcation) and produced strong results.

Original: However, for certain tasks, additional LFs may be included in Step 1 to provide additional signal to the PGM.
Revised: However, for certain tasks, additional LFs may be included in Step 1 to provide additional signal to the PGM.

Original: For example,
Revised: For example,

Original: • Physical models.
Revised: • Physical models.

Original: For segmentation problems where structures in all images are arranged similarly, physical models (e g., atlases) can be registered to the unlabeled images and provide weak labels via label propagation.
Revised: For segmentation problems where structures in all images are arranged similarly, physical models (e.g., atlases) can be registered to the unlabeled images and provide weak labels via label propagation.

Original: • Pre-trained networks.
Revised: • Pre-trained networks.

Original: If the new segmentation task is similar to commonly-studied tasks (e
Revised: If the new segmentation task is similar to commonly-studied tasks (e

Original: g. in natural image segmentation), off-the-shelf networks may be deployed.
Revised: g. in natural image segmentation), off-the-shelf networks may be deployed.

Original: • Image features.
Revised: • Image features.

Original: For simpler tasks, latent image features (e.g. color, intensity) can provide weak segmentation signal.
Revised: For simpler tasks, latent image features (e.g. color, intensity) can provide weak segmentation signal.

Original: These features can also be utilized via traditional CV segmentation approaches such as active contour, random walker, and watershed segementation.
Revised: These features can also be utilized via traditional CV segmentation approaches such as active contour, random walker, and watershed segementation.

Original: • Clustering.
Revised: • Clustering.

Original: Unsupervised segmentation methods such as SLIC, which clusters images in XYZ-color space, can be easily implemented for tasks that have distinct boundaries.
Revised: Unsupervised segmentation methods such as SLIC, which clusters images in XYZ-color space, can be easily implemented for tasks that have distinct boundaries.

Original: A. L EARNING M ODEL P ARAMETERS
Revised: A. L EARNING M ODEL P ARAMETERS

Original: We rely on a technical fact: our proposed distribution is minimal, so that ﬁnding the canonical parameters θ can be accomplished by ﬁnding the mean parameters of the model: E [ 1 { C d } Y pi ] ,
Revised: We rely on a technical fact: our proposed distribution is minimal, so that ﬁnding the canonical parameters θ can be accomplished by ﬁnding the mean parameters of the model: E [ 1 { C d } Y pi ] ,

Original: E [ 1 { C d } Y pi Y qi ] , and E [ 1 { C d } λ p,ki Y pi ] and performing the mean-to-canonical parameter mapping.
Revised: E [ 1 { C d } Y pi Y qi ] , and E [ 1 { C d } λ p,ki Y pi ] and performing the mean-to-canonical parameter mapping.

Original: See Wainwright & Jordan (2008) for more details.
Revised: See Wainwright & Jordan (2008) for more details.

Original: We take E [ 1 { C d } Y pi ] and E [ 1 { C d } Y pi Y qi ] as inputs, representing the conditional class balance of all pixels and the conditional rate of agreement of neighboring pixels, respectively.
Revised: We take E [ 1 { C d } Y pi ] and E [ 1 { C d } Y pi Y qi ] as inputs, representing the conditional class balance of all pixels and the conditional rate of agreement of neighboring pixels, respectively.

Original: We note that these values can be estimated from a validation set, tuned, or estimated from the observed values of the labeling functions.
Revised: We note that these values can be estimated from a validation set, tuned, or estimated from the observed values of the labeling functions.

Original: The remaining mean parameters to be estimated are then E [ 1 { C d } λ p,ki Y pi ] .
Revised: The remaining mean parameters to be estimated are then E [ 1 { C d } λ p,ki Y pi ] .

Original: For conciseness, we will refer to Y pi as Y and λ p,k
Revised: For conciseness, we will refer to Y pi as Y and λ p,k

Original: The mean parameter E [ 1 { C d } λ k Y ] can be interpreted as the scaled accuracy of labeling function
Revised: The mean parameter E [ 1 { C d } λ k Y ] can be interpreted as the scaled accuracy of labeling function

Original: S k under condition C d , a kd .
Revised: S k under condition C d , a kd .

Original: That is, rather than being between 0 and 1, it is scaled to be between −and 1 :
Revised: That is, rather than being between 0 and 1, it is scaled to be between −and 1 :

Original: [Equation]
Revised: [Equation]

Original: This mean parameter cannot be directly estimated (i.e., from samples of λ and Y ) as we do not observe the ground truth label Y .
Revised: This mean parameter cannot be directly estimated (i.e., from samples of λ and Y ) as we do not observe the ground truth label Y .

Original: Instead, building off of the triplet method for parameter recovery presented in Fu et al.
Revised: Instead, building off of the triplet method for parameter recovery presented in Fu et al.

Original: (2020), the conditional accuracy of each labeling function can be efﬁciently computed in closed-form using a system of equations.
Revised: (2020), the conditional accuracy of each labeling function can be efﬁciently computed in closed-form using a system of equations.

Original: We use the following decomposition to factor an observable quantity into the product of two conditional accuracies a jd and a kd .
Revised: We use the following decomposition to factor an observable quantity into the product of two conditional accuracies a jd and a kd .

Original: We take j to be in the contention set C and k to be in the non-contention part I .
Revised: We take j to be in the contention set C and k to be in the non-contention part I .

Original: The decomposition holds due to the fact that these terms are uncorrelated, which follows from the conditional independence of the underlying λ variables.
Revised: The decomposition holds due to the fact that these terms are uncorrelated, which follows from the conditional independence of the underlying λ variables.

Original: [Equation]
Revised: [Equation]

Original: Having access to conditionally independent labeling functions is precisely why we selected two LFs and kept them in the set I , which does not overlap with the contention set D .
Revised: Having access to conditionally independent labeling functions is precisely why we selected two LFs and kept them in the set I , which does not overlap with the contention set D .

Original: With the equation above, we can observe all of the terms such as 1 { C d } λ p,ji λ p,ki , so that we can empirically estimate the ﬁnal term.
Revised: With the equation above, we can observe all of the terms such as 1 { C d } λ p,ji λ p,ki , so that we can empirically estimate the ﬁnal term.

Original: If we form triplets of such equations, for the pairs ( j, k ) , ( j, (cid:96) ) , ( k, (cid:96) ) , where j ∈ D and k, (cid:96) ∈ I , then we can use the following system of equations to solve for the a jd , a kd , a (cid:96)d , up to sign.
Revised: If we form triplets of such equations, for the pairs ( j, k ) , ( j, (cid:96) ) , ( k, (cid:96) ) , where j ∈ D and k, (cid:96) ∈ I , then we can use the following system of equations to solve for the a jd , a kd , a (cid:96)d , up to sign.

Original: Speciﬁcally, the system is
Revised: Speciﬁcally, the system is

Original: [Equation]
Revised: [Equation]

Original: The magnitude of the solution is for j
Revised: The magnitude of the solution is for j

Original: [Equation]
Revised: [Equation]

Original: In practice, we use the empirical versions of each of the expectation terms.
Revised: In practice, we use the empirical versions of each of the expectation terms.

Original: In general, as long as the accuracies are better than random, the sign is positive, so that we fully recover a jd .
Revised: In general, as long as the accuracies are better than random, the sign is positive, so that we fully recover a jd .

Original: Even in cases where the signs of some accuracies are negative, it is still possible to recover in a range of conditions, using the technique detailed in Ratner et al.
Revised: Even in cases where the signs of some accuracies are negative, it is still possible to recover in a range of conditions, using the technique detailed in Ratner et al.

Original: This procedure can be done for any of the LFs and choices of the contention.
Revised: This procedure can be done for any of the LFs and choices of the contention.

Original: All that is required is to select valid triplets.
Revised: All that is required is to select valid triplets.

Original: By design, there is only one possible valid grouping of triplets: each labeling function in D upon which the conditions rely is grouped with the two labeling functions in I upon which no conditions rely.
Revised: By design, there is only one possible valid grouping of triplets: each labeling function in D upon which the conditions rely is grouped with the two labeling functions in I upon which no conditions rely.

Original: We take the median conditional accuracy estimate for each labeling function in I .
Revised: We take the median conditional accuracy estimate for each labeling function in I .

Original: The division of labeling functions into I and D is chosen to maximize the number of LF disagreements caught by the conditionals in the dependent set.
Revised: The division of labeling functions into I and D is chosen to maximize the number of LF disagreements caught by the conditionals in the dependent set.

Original: With this we have recovered all of the mean parameters.
Revised: With this we have recovered all of the mean parameters.

Original: To obtain the canonical parameters—the θ ’s in (2)—we use the mean-to-canonical mapping, as detailed in Wainwright & Jordan (2008).
Revised: To obtain the canonical parameters—the θ ’s in (2)—we use the mean-to-canonical mapping, as detailed in Wainwright & Jordan (2008).

Original: A. A DDITIONAL E XPERIMENTAL R ESULTS
Revised: A. A DDITIONAL E XPERIMENTAL R ESULTS

Original: Image preprocessing The CMR images were corrected with an N4 bias ﬁeld correction and normalized with histogram equalization.
Revised: Image preprocessing The CMR images were corrected with an N4 bias ﬁeld correction and normalized with histogram equalization.

Original: The ACT images were preprocessed with a CT window (window level = 50, window width = 400) and each axial slice was resized to 224x224, then normalized to the range [0,1].
Revised: The ACT images were preprocessed with a CT window (window level = 50, window width = 400) and each axial slice was resized to 224x224, then normalized to the range [0,1].

Original: The BRATS dataset was resized such that each axial slice was 224x224, then normalized to the range [0,1].
Revised: The BRATS dataset was resized such that each axial slice was 224x224, then normalized to the range [0,1].

Original: Ablations We explore the effect of each step of our pipeline in Table 
Revised: Ablations We explore the effect of each step of our pipeline in Table 1 below.

Original: We note that these are the Dice scores reported over the validation sets because the test set is never processed by the LFs.
Revised: We note that these are the Dice scores reported over the validation sets because the test set is never processed by the LFs.

Original: • Effect of LF aggregation.
Revised: • Effect of LF aggregation.

Original: We report the score of each of our one-shot LFs and compare it to the Dice score of the aggregated LFs.
Revised: We report the score of each of our one-shot LFs and compare it to the Dice score of the aggregated LFs.

Original: • Effect of aggregation rule.
Revised: • Effect of aggregation rule.

Original: We compare different aggregation rules on our datasets: the majority vote of the labeling functions, our proposed label model, and Flying Squid (Fu et al.
Revised: We compare different aggregation rules on our datasets: the majority vote of the labeling functions, our proposed label model, and Flying Squid (Fu et al.

Original: We note that the current implementation of Flying Squid was unable to perform inference over the large number of pixels in the ACT and BRATS datasets.
Revised: We note that the current implementation of Flying Squid was unable to perform inference over the large number of pixels in the ACT and BRATS datasets.

Original: • Effect of training end model.
Revised: • Effect of training end model.

Original: We show the impact of training an end model with the weak labels compared to the weak labels alone.
Revised: We show the impact of training an end model with the weak labels compared to the weak labels alone.

Original: • Effect of ﬁne-tuning.
Revised: • Effect of ﬁne-tuning.

Original: We show the impact of ﬁne-tuning the end model with the small number of hand-labeled examples.
Revised: We show the impact of ﬁne-tuning the end model with the small number of hand-labeled examples.

Original: [Table]
Revised: [Table]

Original: Additional qualitative results In Figure A.1, we show qualitative visualizations of the segmentations described in Section 4.2.
Revised: Additional qualitative results In Figure A.1, we show qualitative visualizations of the segmentations described in Section 4.2.1.

Original: [Table]

Original: Generalization performance For methods that use fewer training labels, we hypothesized that the ﬁnal CNNs may be less robust to natural image variations than fully supervised networks.
Revised: Generalization performance For methods that use fewer training labels, we hypothesized that the ﬁnal CNNs may be less robust to natural image variations than fully supervised networks.

Original: We assessed the models’ abilities to generalize to new test sets for two tasks.
Revised: We assessed the models’ abilities to generalize to new test sets for two tasks.

Original: While the new datasets are of the same structure, view
Revised: While the new datasets are of the same structure, view

Original: [Table]
Revised: [Table]

Original: and modality as the originals, they represent different patient populations and scanners.
Revised: and modality as the originals, they represent different patient populations and scanners.

Original: As an additional CMR test set, we collected 100 CMR images with endocardium segmentations through collaboration with the NIH.
Revised: As an additional CMR test set, we collected 100 CMR images with endocardium segmentations through collaboration with the NIH.

Original: As an additional ACT test set, we use images and liver segmentations from the CHAOS challenge (Kavur et al., 2020a;b; 2019) [Table]
Revised: As an additional ACT test set [Table] we use images and liver segmentations from the CHAOS challenge (Kavur et al., 2020a;b; 2019).

Original: [Table]
Revised: [Table]

Original: We compare the performance of the trained models on these new test sets and report the change in performance compared to the original test set in Table 2.
Revised: We compare the performance of the trained models on these new test sets and report the change in performance compared to the original test set in Table 2 of the Appendix.

Original: We compared the best performing method from the knowledge transfer approaches (PANet), the best performing method from the self- and semi- supervised approaches (CL), along with our proposed approach, the data augmentation approach, and the fully supervised network.
Revised: We compared the best performing method from the knowledge transfer approaches (PANet), the best performing method from the self- and semi- supervised approaches (CL), along with our proposed approach, the data augmentation approach, and the fully supervised network.

Original: We see that the WS approach behaves most similarly to the fully supervised networks.
Revised: We see that the WS approach behaves most similarly to the fully supervised networks.

Original: However, PANet actually is able to best generalize to the new test sets (relative to its original performance), possibly because of its inference procedure, which utilizes labeled images from the new test set.
Revised: However, PANet actually is able to best generalize to the new test sets (relative to its original performance), possibly because of its inference procedure, which utilizes labeled images from the new test set.