Original: D EEP L EARNING F ROM C ROWDSOURCED L ABELS : C OUPLED C ROSS - ENTROPY M INIMIZATION , I DENTIFIABILITY , AND R EGULARIZATION
Revised: D EEP L EARNING F ROM C ROWDSOURCED L ABELS : C OUPLED C ROSS - ENTROPY M INIMIZATION , I DENTIFIABILITY , AND R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Using noisy crowdsourced labels from multiple annotators, a deep learning-based end-to-end (E2E) system aims to learn the label correction mechanism and the neural classifier simultaneously.
Revised: Using noisy crowdsourced labels from multiple annotators, a deep learning-based end-to-end (E2E) system aims to learn the label correction mechanism and the neural classifier simultaneously.

Original: To this end, many E2E systems concatenate the neural classifier with multiple annotator-specific “label confusion” layers and co-train the two parts in a parameter-coupled manner.
Revised: To this end, many E2E systems concatenate the neural classifier with multiple annotator-specific “label confusion” layers and co-train the two parts in a parameter-coupled manner.

Original: The formulated coupled cross-entropy minimization (CCEM)-type criteria are intuitive and work well in practice.
Revised: The formulated coupled cross-entropy minimization (CCEM)-type criteria are intuitive and work well in practice.

Original: Nonetheless, theoretical understanding of the CCEM criterion has been limited.
Revised: Nonetheless, theoretical understanding of the CCEM criterion has been limited.

Original: The contribution of this work is twofold: First, performance guarantees of the CCEM criterion are presented.
Revised: The contribution of this work is twofold: First, performance guarantees of the CCEM criterion are presented.

Original: Our analysis reveals for the first time that
Revised: Our analysis reveals for the first time that

Original: the
Revised: the

Original: CCEM can indeed correctly identify the annotators’ confusion characteristics and the desired “ground-truth” neural classifier under realistic conditions, e.g., when only incomplete annotator labeling and finite samples are available.
Revised: CCEM can indeed correctly identify the annotators’ confusion characteristics and the desired “ground-truth” neural classifier under realistic conditions, e.g., when only incomplete annotator labeling and finite samples are available.

Original: Second, based on the insights learned from our analysis, two regularized variants of the CCEM are proposed.
Revised: Second, based on the insights learned from our analysis, two regularized variants of the CCEM are proposed.

Original: The regularization terms provably enhance the identifiability of the target model parameters in various more challenging cases.
Revised: The regularization terms provably enhance the identifiability of the target model parameters in various more challenging cases.

Original: A series of synthetic and real data experiments are presented to showcase the effectiveness of our approach.
Revised: A series of synthetic and real data experiments are presented to showcase the effectiveness of our approach.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The success of deep learning has escalated the demand for labeled data to an unprecedented level.
Revised: The success of deep learning has escalated the demand for labeled data to an unprecedented level.

Original: Some learning tasks can easily consume millions of labeled data (Najafabadi et al., 2015; Goodfellow et al., 2016).
Revised: Some learning tasks can easily consume millions of labeled data (Najafabadi et al., 2015; Goodfellow et al., 2016).

Original: However, acquiring data labels is a nontrivial task—it often requires a pool of annotators with sufficient domain expertise to manually label the data items.
Revised: However, acquiring data labels is a nontrivial task—it often requires a pool of annotators with sufficient domain expertise to manually label the data items.

Original: For example, the popular Microsoft
Revised: For example, the popular Microsoft

Original: COCO dataset contains 2.5 million images and around 20,000 work hours aggregated from multiple annotators were used for its category labeling (Lin et al., 2014).
Revised: COCO dataset contains 2.5 million images and around 20,000 work hours aggregated from multiple annotators were used for its category labeling (Lin et al., 2014).

Original: Crowdsourcing is considered an important working paradigm for data labeling.
Revised: Crowdsourcing is considered an important working paradigm for data labeling.

Original: In crowdsourcing platforms, e.g., Amazon Mechanical Turk (Buhrmester et al., 2011), Crowdflower (Wazny, 2017), and
Revised: In crowdsourcing platforms, e.g., Amazon Mechanical Turk (Buhrmester et al., 2011), Crowdflower (Wazny, 2017), and

Original: ClickWork (Vakharia & Lease, 2013), data items are dispatched and labeled by many annotators; the annotations are then integrated to produce reliable labels.
Revised: ClickWork (Vakharia & Lease, 2013), data items are dispatched and labeled by many annotators; the annotations are then integrated to produce reliable labels.

Original: A notable challenge is that annotator-output labels are sometimes considerably noisy.
Revised: A notable challenge is that annotator-output labels are sometimes considerably noisy.

Original: Training machine learning models using noisy labels could seriously degrade the system performance (Arpit et al., 2017; Zhang et al., 2016a).
Revised: Training machine learning models using noisy labels could seriously degrade the system performance (Arpit et al., 2017; Zhang et al., 2016a).

Original: In addition, the labels provided by individual annotators are often largely incomplete, as a dataset is often divided and dispatched to different annotators.
Revised: In addition, the labels provided by individual annotators are often largely incomplete, as a dataset is often divided and dispatched to different annotators.

Original: Early crowdsourcing methods often treat annotation integration and downstream operations, e.g., classification, as separate tasks; see, (Dawid & Skene, 1979;
Revised: Early crowdsourcing methods often treat annotation integration and downstream operations, e.g., classification, as separate tasks; see, (Dawid & Skene, 1979;

Original: Karger et al., 2011a; Whitehill et al., 2009; Snow et al., 2008; Welinder et al., 2010; Liu et al., 2012; Zhang et al., 2016b).
Revised: Karger et al., 2011a; Whitehill et al., 2009; Snow et al., 2008; Welinder et al., 2010; Liu et al., 2012; Zhang et al., 2016b).

Original: This pipeline estimates the annotators’ confusion parameters (e.g., the confusion matrices under the Dawid &
Revised: This pipeline estimates the annotators’ confusion parameters (e.g., the confusion matrices under the Dawid &

Original: Skene (DS) model (Dawid & Skene, 1979)) in the first stage.
Revised: Skene (DS) model (Dawid & Skene, 1979)) in the first stage.

Original: Then, the corrected and integrated labels along with the data are used for training the downstream tasks’ classifiers.
Revised: Then, the corrected and integrated labels along with the data are used for training the downstream tasks’ classifiers.

Original: However, simultaneously learning the annotators’ confusions and the classifier in an end-to-end (E2E) manner has shown substantially improved performance in practice.
Revised: However, simultaneously learning the annotators’ confusions and the classifier in an end-to-end (E2E) manner has shown substantially improved performance in practice.

Original: (Raykar et al., 2010; Khetan et al., 2018; Tanno et al., 2019; Rodrigues & Pereira, 2018; Chu et al., 2021;
Revised: (Raykar et al., 2010; Khetan et al., 2018; Tanno et al., 2019; Rodrigues & Pereira, 2018; Chu et al., 2021;

Original: Guan et al., 2018; Cao et al., 2019; Li et al., 2020; Wei et al., 2022; Chen et al., 2020).
Revised: Guan et al., 2018; Cao et al., 2019; Li et al., 2020; Wei et al., 2022; Chen et al., 2020).

Original: To achieve the goal of E2E-based learning under crowdsourced labels, a class of methods concatenate a “confusion layer” for each annotator to the output of a neural classifier, and jointly learn these parts via a parameter-coupled manner.
Revised: To achieve the goal of E2E-based learning under crowdsourced labels, a class of methods concatenate a “confusion layer” for each annotator to the output of a neural classifier, and jointly learn these parts via a parameter-coupled manner.

Original: This gives rise to a coupled cross entropy minimization (CCEM) criterion (Rodrigues & Pereira, 2018; Tanno et al., 2019; Chen et al., 2020;
Revised: This gives rise to a coupled cross entropy minimization (CCEM) criterion (Rodrigues & Pereira, 2018; Tanno et al., 2019; Chen et al., 2020;

Original: Chu et al., 2021;
Revised: Chu et al., 2021;

Original: Wei et al., 2022).
Revised: Wei et al., 2022).

Original: In essence, the CCEM criterion models the observed labels as annotators’ confused outputs from the “ground-truth” predictor (GTP)—i.e., the classifier as if being trained using the noiseless labels and with perfect generalization—which is natural and intuitive.
Revised: In essence, the CCEM criterion models the observed labels as annotators’ confused outputs from the “ground-truth” predictor (GTP)—i.e., the classifier as if being trained using the noiseless labels and with perfect generalization—which is natural and intuitive.

Original: A notable advantage of the CCEM criteria is that they often lead to computationally convenient optimization problems, as the confusion characteristics are modeled as just additional structured layers added to the neural networks.
Revised: A notable advantage of the CCEM criteria is that they often lead to computationally convenient optimization problems, as the confusion characteristics are modeled as just additional structured layers added to the neural networks.

Original: Nonetheless, the seemingly simple CCEM criterion and its variants have shown promising performance.
Revised: Nonetheless, the seemingly simple CCEM criterion and its variants have shown promising performance.

Original: For example, the crowdlayer method is a typical CCEM approach, which has served as a widely used benchmark for E2E crowdsourcing since its proposal(Rodrigues & Pereira, 2018).
Revised: For example, the crowdlayer method is a typical CCEM approach, which has served as a widely used benchmark for E2E crowdsourcing since its proposal(Rodrigues & Pereira, 2018).

Original: In (Tanno et al., 2019), a similar CCEM-type criterion is employed, with a trace regularization added.
Revised: In (Tanno et al., 2019), a similar CCEM-type criterion is employed, with a trace regularization added.

Original: More CCEM-like learning criteria are seen in (Chen et al., 2020;
Revised: More CCEM-like learning criteria are seen in (Chen et al., 2020;

Original: Chu et al., 2021;
Revised: Chu et al., 2021;

Original: Wei et al., 2022), with some additional constraints and considerations.
Revised: Wei et al., 2022), with some additional constraints and considerations.

Original: Challenges.
Revised: Challenges.

Original: Apart from its empirical success, understanding of the CCEM criterion has been limited.
Revised: Apart from its empirical success, understanding of the CCEM criterion has been limited.

Original: Particularly, it is often unclear if CCEM can correctly identify the annotators’ confusion characteristics (that are often modeled as “confusion matrices”) and the GTP under reasonable settings (Rodrigues & Pereira, 2018;
Revised: Particularly, it is often unclear if CCEM can correctly identify the annotators’ confusion characteristics (that are often modeled as “confusion matrices”) and the GTP under reasonable settings (Rodrigues & Pereira, 2018;

Original: Chu et al., 2021; Chen et al., 2020; Wei et al., 2022), while identifiability stands as the key for ensured performance.
Revised: Chu et al., 2021; Chen et al., 2020; Wei et al., 2022), but identifiability stands as the key for ensured performance.

Original: The only existing identifiability result on CCEM was derived under restricted conditions, e.g., the availability of infinite samples and the assumption that there exists annotators with diagonally-dominant confusion matrices (Tanno et al., 2019).
Revised: The only existing identifiability result on CCEM was derived under restricted conditions, e.g., the availability of infinite samples and the assumption that there exists annotators with diagonally-dominant confusion matrices (Tanno et al., 2019).

Original: To our best knowledge, model identifiability of CCEM has not been established under realistic conditions, e.g., in the presence of incomplete annotator labeling and non-experts under finite samples.
Revised: To our best knowledge, model identifiability of CCEM has not been established under realistic conditions, e.g., in the presence of incomplete annotator labeling and non-experts under finite samples.

Original: We should note that a couple of non-CCEM approaches proposed identifiability-guaranteed solutions for E2E crowdsourcing.
Revised: We should note that a couple of non-CCEM approaches proposed identifiability-guaranteed solutions for E2E crowdsourcing.

Original: The work in (Khetan et al., 2018) showed identifiability under their expectation maximization (EM)-based learning approach, but the result is only applicable to binary classification.
Revised: The work in (Khetan et al., 2018) showed identifiability under their expectation maximization (EM)-based learning approach, but the result is only applicable to binary classification.

Original: An information-theoretic loss-based learning approach proposed in (Cao et al., 2019) presents some identifiability results, but under the presence of a group of independent expert annotators and infinite samples.
Revised: An information-theoretic loss-based learning approach proposed in (Cao et al., 2019) presents some identifiability results, but under the presence of a group of independent expert annotators and infinite samples.

Original: These conditions are hard to meet or verify.
Revised: These conditions are hard to meet or verify.

Original: In addition, the computation of these methods are often more complex relative to the CCEM-based methods.
Revised: In addition, the computation of these methods are often more complex relative to the CCEM-based methods.

Original: Contributions.
Revised: Contributions.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • Identifiability Characterizations of CCEM-based E2E Crowdsourcing.
Revised: • Identifiability Characterizations of CCEM-based E2E Crowdsourcing.

Original: In this work, we show that the CCEM criterion can indeed provably identify the annotators’ confusion matrices and the GTP up to inconsequential ambiguities under mild conditions.
Revised: In this work, we show that the CCEM criterion can indeed provably identify the annotators’ confusion matrices and the GTP up to inconsequential ambiguities under mild conditions.

Original: Specifically, we show that, if the number of annotator-labeled items is sufficiently large and some other reasonable assumptions hold, the two parts can be recovered with bounded errors.
Revised: Specifically, we show that, if the number of annotator-labeled items is sufficiently large and some other reasonable assumptions hold, the two parts can be recovered with bounded errors.

Original: Our result is the first finite-sample identifiability result for CCEM-based crowdsourcing.
Revised: Our result is the first finite-sample identifiability result for CCEM-based crowdsourcing.

Original: Moreover, our analysis reveals that the success of CCEM does not rely on conditional independence among the annotators.
Revised: Moreover, our analysis reveals that the success of CCEM does not rely on conditional independence among the annotators.

Original: This is favorable (as annotator independence is a stringent requirement) and surprising, since conditional independence is often used to derive the CCEM criterion; see, e.g., (Tanno et al., 2019; Rodrigues & Pereira, 2018; Chu et al., 2021).
Revised: This is favorable (as annotator independence is a stringent requirement) and surprising, since conditional independence is often used to derive the CCEM criterion; see, e.g., (Tanno et al., 2019; Rodrigues & Pereira, 2018; Chu et al., 2021).

Original: • Regularization Design for CCEM With Provably Enhanced Identfiability.
Revised: • Regularization Design for CCEM With Provably Enhanced Identfiability.

Original: Based on the key insights revealed in our analysis, we propose two types of regularizations that can provide enhanced identifiability guarantees under challenging scenarios.
Revised: Based on the key insights revealed in our analysis, we propose two types of regularizations that can provide enhanced identifiability guarantees under challenging scenarios.

Original: To be specific, the first regularization term ensures that the confusion matrices
Revised: To be specific, the first regularization term ensures that the confusion matrices

Original: and the GTP can be identified without having any expert annotators if one has sufficiently large amount of data.
Revised: and the GTP can be identified without having any expert annotators if one has sufficiently large amount of data.

Original: The second regularization term ensures identifiability when class specialists are present among annotators.
Revised: The second regularization term ensures identifiability when class specialists are present among annotators.

Original: These identifiability-enhanced approaches demonstrate promising label integration performance in our experiments.
Revised: These identifiability-enhanced approaches demonstrate promising label integration performance in our experiments.

Original: Notation.
Revised: Notation.

Original: The notations are summarized in the supplementary material.
Revised: The notations are summarized in the supplementary material.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Crowdsourcing is a core working paradigm for labeling data, as individual annotators are often not reliable enough to produce quality labels.
Revised: Crowdsourcing is a core working paradigm for labeling data, as individual annotators are often not reliable enough to produce quality labels.

Original: Consider a set of N items { x n } N n =1 .
Revised: Consider a set of N items { x n } Nn =1 .

Original: Each item belongs to one of the K classes.
Revised: Each item belongs to one of the K classes.

Original: Here, x n ∈ R D denote the feature vector of n th data item.
Revised: Here, x n ∈ R D denote the feature vector of n th data item.

Original: Let { y n } Nn =  denote the set of ground-truth labels, where y n ∈ [ K ] , ∀ n .
Revised: Let { y n } N n =  denote the set of ground-truth labels, where y n ∈ [ K ] , ∀ n .

Original: The ground-truth labels { y n } Nn =1 are unknown .
Revised: The ground-truth labels { y n } Nn =1 are unknown .

Original: We ask M annotators to provide their labels for their assigned items.
Revised: We ask M annotators to provide their labels for their assigned items.

Original: Consequently, each item x n is labeled by a subset of annotators indexed by S n ⊆
Revised: Consequently, each item x n is labeled by a subset of annotators indexed by S n ⊆

Original: [ M ] .
Revised: [ M ] .

Original: Let { (cid:98) y
Revised: Let { (cid:98) y

Original: ( m ) n } ( m,n ) ∈S denote the set of labels provided by all annotators, where S = { ( m, n ) | m ∈ S n , n ∈ [ N ] } and (cid:98)
Revised: ( m ) n } ( m,n ) ∈S denote the set of labels provided by all annotators, where S = { ( m, n ) | m ∈ S n , n ∈ [ N ] } and (cid:98)

Original: y ( m ) n ∈
Revised: y ( m ) n ∈

Original: [ K ] .
Revised: [ K ] .

Original: Note that |S| ≪ NM usually holds and that the annotators may often incorrectly label the items, leading to an incomplete and noisy { (cid:98) y ( m ) n } ( m,n ) ∈S .
Revised: Note that |S| ≪ NM usually holds and that the annotators may often incorrectly label the items, leading to an incomplete and noisy { (cid:98) y ( m ) n } ( m,n ) ∈S .

Original: Here, “incomplete” means that S does not cover all possible combinations of ( m, n ) —i.e., not all annotators label all items.
Revised: Here, “incomplete” means that S does not cover all possible combinations of ( m, n ) —i.e., not all annotators label all items.

Original: The goal of an
Revised: The goal of an

Original: E2E crowdsourcing system is to train a reliable machine learning model (often a classifier) using such noisy and incomplete annotations.
Revised: E2E crowdsourcing system is to train a reliable machine learning model (often a classifier) using such noisy and incomplete annotations.

Original: This is normally done via blindly estimating the confusion characteristics of the annotators from the noisy labels.
Revised: This is normally done via blindly estimating the confusion characteristics of the annotators from the noisy labels.

Original: Classic Approaches.
Revised: Classic Approaches.

Original: Besides the naive majority voting method, one of the most influential crowdsourcing model is by Dawid & Skene (Dawid & Skene, 1979).
Revised: Besides the naive majority voting method, one of the most influential crowdsourcing model is by Dawid & Skene (Dawid & Skene, 1979).

Original: Dawid & Skene models the confusions of the annotators using the conditional probabilities
Revised: Dawid & Skene models the confusions of the annotators using the conditional probabilities

Original: Pr ( (cid:98) y ( m ) n | y n ) .
Revised: Pr ( (cid:98) y ( m ) n | y n ) .

Original: Consequently, annotator m ’s confusion is fully characterized by a matrix defined as follows:
Revised: Consequently, annotator m ’s confusion is fully characterized by a matrix defined as follows:

Original: A m ( k, k ′ ) :
Revised: A m ( k, k ′ ) :

Original: = Pr ( (cid:98) y ( m ) n = k | y n = k ′ ) , ∀ k, k ′ ∈
Revised: = Pr ( (cid:98) y ( m ) n = k | y n = k ′ ) , ∀ k, k ′ ∈

Original: [ K ] .
Revised: [ K ] .

Original: Dawid & Skene also proposed an EM algorithm that estimate
Revised: Dawid & Skene also proposed an EM algorithm that estimate

Original: A m ’s under a naive
Revised: A m ’s under a naive

Original: Bayes generative model.
Revised: Bayes generative model.

Original: Many crowdsourcing methods follow Dawid & Skene’s modeling ideas (Ghosh et al., 2011;
Revised: Many crowdsourcing methods follow Dawid & Skene’s modeling ideas (Ghosh et al., 2011;

Original: Dalvi et al., 2013; Karger et al., 2013; Zhang et al., 2016b; Traganitis et al., 2018).
Revised: Dalvi et al., 2013; Karger et al., 2013; Zhang et al., 2016b; Traganitis et al., 2018).

Original: Like (Dawid & Skene, 1979), many of these methods do not exploit the data features x n while learning the confusion matrices of the annotators.
Revised: Like (Dawid & Skene, 1979), many of these methods do not exploit the data features x n while learning the confusion matrices of the annotators.

Original: They often treat confusion estimation and classifier training as separate and sequential tasks.
Revised: They often treat confusion estimation and classifier training as two sequential tasks.

Original: Such two-stage approaches may suffer from error propagation.
Revised: Such two-stage approaches may suffer from error propagation.

Original: Problem Setting of E2E Crowdsourcing.
Revised: E2E Crowdsourcing and The Identifiability Challenge.

Original: The E2E approaches in (Rodrigues &Pereira, 2018;
Revised: The E2E approaches in (Rodrigues &

Original:  Tanno et al., 2019; Khetan et al., 2018) model the probability of m th annotator’s response to the data item x n as follows:
Revised: Pereira, 2018; Tanno et al., 2019; Khetan et al., 2018) model the probability of m th annotator’s response to the data item x n as follows:

Original: [Equation]
Revised: [Equation]

Original: where the assumption that the annotator confusion is data-independent has been used to derive the right-hand side.
Revised: where the assumption that the annotator confusion is data-independent has been used to derive the right-hand side.

Original: In the above, the distribution Pr ( (cid:98) y ( m ) n | y n ) models the confusions in annotator responses given the true label y n and Pr ( y n | x n ) denotes the true label distribution of the data item x n .
Revised: In the above, the distribution Pr ( (cid:98) y ( m ) n | y n ) models the confusions in annotator responses given the true label y n and Pr ( y n | x n ) denotes the true label distribution of the data item x n .

Original: The distribution Pr ( y n | x n ) can be represented using a mapping f : R D → [0 , 1] K , [ f ( x n )]
Revised: The distribution Pr ( y n | x n ) can be represented using a mapping f : R D → [0 , 1] K , [ f ( x n )]

Original: k =
Revised: k =

Original: Pr ( y n = k | x n ) .
Revised: Pr ( y n = k | x n ) .

Original: Define a probability vector p ( m ) n ∈ [0 , 1] K for every m and n , such that [ p ( m ) n ] k =
Revised: Define a probability vector p ( m ) n ∈ [0 , 1] K for every m and n , such that [ p ( m ) n ] k =

Original: Pr ( (cid:98) y ( m ) n = k | x n ) .
Revised: Pr ( (cid:98) y ( m ) n = k | x n ) .

Original: With these notations, the following model holds:
Revised: With these notations, the following model holds:

Original: [Equation]
Revised: [Equation]

Original: In practice, one does not observe p ( m ) n .
Revised: In practice, one does not observe p ( m ) n .

Original: Instead, the annotations 
Revised: Instead, the annotations (cid:98) y ( m ) n are categorical realizations of the distribution p ( m ) n ; i.e.,

Original: [Equation]
Revised: [Equation]

Original:  y ( m ) n are categorical realizations of
Revised: The goal of E2E crowdsourcing is to identify A m and f in (2) from noisy labels { (cid:98) y ( m ) n } ( m,n ) ∈S and data { x n } Nn =1 .

Original: the distribution p ( m ) n ; i.e.,
Revised: Note that even if p ( m ) n in (2) is observed, A m and f ( · ) are in general not identifiable .

Original: [Equation]
Revised: The reason is that one can easily find nonsingular matrices Q ∈ R K × K such that p ( m ) n = (cid:101) A m (cid:101) f ( x n ) , where (cid:101) A m =

Original: The goal of E2E crowdsourcing is to identify A m  and  fin (2) from noisy labels { (cid:98) y ( m ) n} ( m,n ) ∈S and data { x n } Nn =1 . Learning f not only identifies the true labels for the training data (data for which the annotations have been collected), but is useful to predict the true labels for unseen test data items.
Revised: A m Q and (cid:101) f ( x n ) = Q − 1 f ( x n ) . If an E2E approach does not ensure the identifiability of A m and f , but outputs (cid:101) f ( x n ) =

Revised: Q − 1 f ( x n ) , then the learned predictor is not likely to work reasonably.

Revised: Identifying the ground-truth f not only identifies the true labels for the training data (data for which the annotations have been collected), but is useful to predict the true labels for unseen test data items.

Original: In addition, learning A m ’s helps quantify how reliable each annotator is.
Revised: In addition, correctly identifying A m ’s helps quantify how reliable each annotator is.

Original: CCEM-based Approaches and Challenges.
Revised: CCEM-based Approaches and Challenges.

Original: A number of existing E2E methods are inspired by the model in (2) 
Revised: A number of existing E2E methods are inspired by the model in (2) to formulate CCEM-based learning losses.

Original: to formulate

Original: CCEM

Original: -based

Original: learning

Original: losses.

Original: • Crowdlayer (Rodrigues & Pereira, 2018):
Revised: • Crowdlayer (Rodrigues & Pereira, 2018):

Original: For each ( m, n ) ∈ S , the generative model in (2) naturally leads to a cross-entropy based learning loss,i.
Revised: For each ( m, n ) ∈ S , the generative model in (2) naturally leads to a cross-entropy based learning loss,

Original: e
Revised: i

Original:  CE ( A m f ( x n ) , (cid:98) y ( m ) n ) .
Revised: e., CE ( A m f ( x n ) , (cid:98) y ( m ) n ) .

Original: As some annotators co-label items, the components f  x n 
Revised: As some annotators co-label items, the components f ( x n ) associated with the co-labeled items are shared by these annotators.

Original: associated with the co

Original: labeled items are shared by these annotators With such latent component coupling and putting the cross-entropy losses for all ( m, n ) ∈ S together, the work in (Rodrigues & Pereira, 2018) formulate the following CCEM loss:
Revised: With such latent component coupling and putting the cross-entropy losses for all ( m, n ) ∈ S together, the work in (Rodrigues & Pereira, 2018) formulate the following CCEM loss:

Original: [Equation]
Revised: [Equation]

Original: where F ⊆ { f ( x ) ∈ R K | f ( x ) ∈ ∆ K , ∀ x } is a function class and the softmax operator is applied as the output of A m f ( x n ) is supposed to be a probability mass function (PMF).
Revised: where F ⊆ { f ( x ) ∈ R K | f ( x ) ∈ ∆ K , ∀ x } is a function class and the softmax operator is applied as the output of A m f ( x n ) is supposed to be a probability mass function (PMF).

Original: The method based on the formulation in (4) was called crowdlayer (Rodrigues & Pereira, 2018).
Revised: The method based on the formulation in (4) was called crowdlayer (Rodrigues & Pereira, 2018)

Original: The name comes from the fact that the confusion matrices of the “crowd” A m ’s can be understood as additional layers if f is represented by a neural network.
Revised: The name comes from the fact that the confusion matrices of the “crowd” A m ’s can be understood as additional layers if f is represented by a neural network

Original: The loss function in (4) is relatively easy to optimize, as any off-the-shelf neural network training algorithms can be directly applied.
Revised: The loss function in (4) is relatively easy to optimize, as any off-the-shelf neural network training algorithms can be directly applied This seemingly

Original: This seemingly natural and simple CCEM approach demonstrated substantial performanceimprovement upon classic methods.
Revised: natural and simple CCEM approach demonstrated substantial performance

Revised: improvement

Revised: upon

Revised: classic

Revised: methods.

Original: However, there is no performance characterization for (4).
Revised: However, there is no performance characterization for (

Original: • TraceReg (Tanno et al., 2019):
Revised: • TraceReg (Tanno et [Equation]

Original: The work in (Tanno et al., 2019) proposed a similar loss function— with a regularization term in order to establish identifiability of the A m ’s and f 
Revised: The work in (Tanno et al., 2019) proposed a similar loss function— with a regularization term in order to establish identifiability of the A m ’s and f :

Original: [Equation]
Revised: [Equation]

Original: where A is the constrained set { A ∈ R K × K | A ≥ 0 , 1 ⊤ A = 1 ⊤ } .
Revised: where A is the constrained set { A ∈ R K × K | A ≥ 0 , 1 ⊤ A = 1 ⊤ } .

Original: It was shown that (i) if p ( m ) n instead of (cid:98) y ( m ) n is observed for every m and n and (ii) if the mean of the A m ’s are diagonally dominant, then solving (5) with CE (
Revised: It was shown that (i) if p ( m ) n instead of (cid:98) y ( m ) n is observed for every m and n and (ii) if the mean of the A m ’s are diagonally dominant, then solving (5) with CE (

Original: A m f ( x n ) , (cid:98) y ( m ) n ) replaced by enforcing p ( m ) n
Revised: A m f ( x n ) , (cid:98) y ( m ) n ) replaced by enforcing p ( m ) n

Original: [Equation]
Revised: [Equation]

Original: ensures identifying A m for all m , up to column permutations.
Revised: ensures identifying A m for all m , up to column permutations.

Original: Even though, the result provides some justification for using CCEM criterion under trace regularization, the result is clearly unsatisfactory, as it requires many stringent assumptions to establish identifiability.
Revised: Even though, the result provides some justification for using CCEM criterion under trace regularization, the result is clearly unsatisfactory, as it requires many stringent assumptions to establish identifiability.

Original: • SpeeLFC (Chen et al., 2020), Union-Net (Wei et al., 2022), CoNAL (Chu et al., 2021):
Revised: • SpeeLFC (Chen et al., 2020), Union-Net (Wei et al., 2022), CoNAL (Chu et al., 2021):

Original: A number of other variants of the CCEM criterion are also proposed for E2E learning with crowdsourced labels.
Revised: A number of other variants of the CCEM criterion are also proposed for E2E learning with crowdsourced labels.

Original: The work in (Chen et al., 2020) uses a similar criterion as in (4), but with the softmax operator applied on the columns of A m ’s, instead on the output of A m f ( x n ) .
Revised: The work in (Chen et al., 2020) uses a similar criterion as in (4), but with the softmax operator applied on the columns of A m ’s, instead on the output of A m f ( x n ) .

Original: The work in (Wei et al., 2022) concatenates A m ’s vertically and applies the softmax operator on the columns of such concatenated matrix, while formulating the CCEM criterion.
Revised: The work in (Wei et al., 2022) concatenates A m ’s vertically and applies the softmax operator on the columns of such concatenated matrix, while formulating the CCEM criterion.

Original: The method in (Chu et al., 2021) models a common confusion matrix in addition to the annotator-specific confusion matrices A m ’s and employs a CCEM-type criterion to learn both in a coupled fashion.
Revised: The method in (Chu et al., 2021) models a common confusion matrix in addition to the annotator-specific confusion matrices A m ’s and employs a CCEM-type criterion to learn both in a coupled fashion.

Original: Nonetheless, none of these approaches have successfully tackled the identifiability aspect.
Revised: Nonetheless, none of these approaches have successfully tackled the identifiability aspect.

Original: The work (Wei et al., 2022) claims theoretical support for their approach. However, their proofstrategy is flawed, based on our analysis.
Revised: The work (Wei et al., 2022) claims theoretical support for their approach, but the proof is flawed.

Original: In fact, understanding to the success of CCEM-based methods under practical E2E learning settings is still unclear.
Revised: In fact, understanding to the success of CCEM-based methods under practical E2E learning settings is still unclear.

Original: Other Related Works.
Revised: Other Related Works.

Original: A couple of non-CCEM approaches have also considered the identifiability aspects (Khetan et al., 2018; Cao et al., 2019).
Revised: A couple of non-CCEM approaches have also considered the identifiability aspects (Khetan et al., 2018; Cao et al., 2019).

Original: The work in (Khetan et al., 2018) presents an EMinspired alternating optimization strategy.
Revised: The work in (Khetan et al., 2018) presents an EMinspired alternating optimization strategy.

Original: Such procedure involves training the neural classifier multiple times, making it a computationally intensive approach.
Revised: Such procedure involves training the neural classifier multiple times, making it a computationally intensive approach.

Original: In addition, its identifiability guarantees are under certain strict assumptions, e.g., binary classification, identical confusions and conditional independence for annotators—which are hard to satisfy in practice.
Revised: In addition, its identifiability guarantees are under certain strict assumptions, e.g., binary classification, identical confusions and conditional independence for annotators—which are hard to satisfy in practice.

Original: The approach in (Cao et al., 2019) employs an information-theoretic loss function to jointly train a predictor network and annotation aggregation network.
Revised: The approach in (Cao et al., 2019) employs an information-theoretic loss function to jointly train a predictor network and annotation aggregation network.

Original: The identifiability guarantees are again under restrictive assumptions, e.g., infinite data, no missing annotations and existence of mutually independent expert annotators.
Revised: The identifiability guarantees are again under restrictive assumptions, e.g., infinite data, no missing annotations and existence of mutually independent expert annotators.

Original: We should also note that there exist works considering data-dependent annotator confusions as well, e.g., (Zhang et al., 2020) (multiple annotator case) and (Cheng et al., 2021; Xia et al., 2020; Zhu et al., 2022) (single annotator case)—leveraging more complex models and learning criteria.
Revised: We should also note that there exist works considering data-dependent annotator confusions as well, e.g., (Zhang et al., 2020) (multiple annotator case) and (Cheng et al., 2021; Xia et al., 2020; Zhu et al., 2022) (single annotator case)—leveraging more complex models and learning criteria.

Original: In our work, we focus on the criterion designed using a model in (1) as it is shown to be effective in practice.
Revised: In our work, we focus on the criterion designed using a model in (1) as it is shown to be effective in practice.

Original:  I DENTIFIABILITY OF CCEM-B ASED E2E C ROWDSOURCING
Revised:  I DENTIFIABILITY OF CCEM-B ASED E2E C ROWDSOURCING

Original: In this section, we offer identifiability analysis of the CCEM learning loss under realistic settings.
Revised: In this section, we offer identifiability analysis of the CCEM learning loss under realistic settings.

Original: We consider the following re-expressed objective function of CCEM:[Equation] 6a
Revised: We consider the following re-expressed objective function of CCEM:

Original: [Equation]
Revised: [Equation]

Original: subject to f ∈ F , A m ∈
Revised: subject to f ∈ F , A m ∈

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ∀ m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: 6b) where F ⊆ { f ( x ) ∈ R K | f ( x ) ∈ ∆ K , ∀ x } is a function class and A is the constrained set { A ∈
Revised: 6b [Equation] where F ⊆ { f ( x ) ∈ R K | f ( x ) ∈ ∆ K , ∀ x } is a function class and A is the constrained set { A ∈

Original: R K × K | A ≥ 0 , 1 ⊤ A = 1 ⊤ } , since each column of A m ’s are conditional probability distributions.
Revised: R K × K | A ≥ 0 , 1 ⊤ A = 1 ⊤ } , since each column of A m ’s are conditional probability distributions.

Original: The rationale is that the confusion matrices A m ’s act as correction terms for annotator’s labeling noise to output a true label classifier f .
Revised: The rationale is that the confusion matrices A m ’s act as correction terms for annotator’s labeling noise to output a true label classifier f .

Original: Intuitively, the objective in (6a) encourages the output estimates (cid:98) f , { (cid:98) A m } to satisfy the relation p ( m ) n =
Revised: Intuitively, the objective in (6a) encourages the output estimates (cid:98) f , { (cid:98) A m } to satisfy the relation p ( m ) n =

Original: [Equation]
Revised: [Equation]

Original: A m (cid:98) f ( x n ) , ∀ m, n .
Revised: A m (cid:98) f ( x n ) , ∀ m, n .

Original: One can easily note that the identifiability guarantees for A m and f ( i.e., when does the ground-truth A m and the ground-truth f can be identified) are not so straightforward since there exists an infinite number of nonsingular matrices Q ∈ R K × K such that p ( m ) n
Revised: One can easily note that the identifiability guarantees for A m and f ( i.e., when does the ground-truth A m and the ground-truth f can be identified) are not so straightforward since there exists an infinite number of nonsingular matrices Q ∈ R K × K such that p ( m ) n

Original: [Equation]
Revised: [Equation]

Original: To proceed, we make the following assumptions:
Revised: To proceed, we make the following assumptions:

Original: Assumption 1 Each data item x n , n ∈ [ N ] is drawn from a distribution D independently at random and has a bounded ℓ 2 norm.
Revised: Assumption 1 Each data item x n , n ∈ [ N ] is drawn from a distribution D independently at random and has a bounded ℓ 2 norm.

Original: The observed index pairs ( m, n ) are included in S uniformly at random.
Revised: The observed index pairs ( m, n ) are included in S uniformly at random.

Original: Assumption 2 The neural network function class F has a complexity measure denoted as R F .
Revised: Assumption 2 The neural network function class F has a complexity measure denoted as R F .

Original: Here, we use the sensitive complexity parameter introduced in (Lin & Zhang, 2019) as the complexity measure R F .
Revised: Here, we use the sensitive complexity parameter introduced in (Lin & Zhang, 2019) as the complexity measure R F .

Original: In essence, R F gets larger as the neural network function class F gets deeper and wider—also see supplementary material Sec.
Revised: In essence, R F gets larger as the neural network function class F gets deeper and wider—also see supplementary material Sec.

Original: L for more discussion.
Revised: L for more discussion.

Original: Assumption 3 There exists a GTP f ♮ :
Revised: Assumption 3 There exists a GTP f ♮ :

Original: R D
Revised: R D

Original: →
Revised: →

Original: [(1 /β ) , 1] K , for a certain β > 1 , such that
Revised: [(1 /β ) , 1] K , for a certain β > 1 , such that

Original: [ f ♮ ( x n )] k =
Revised: [ f ♮ ( x n )] k =

Original: Pr ( y n = k | x n ) , where y n denotes the true label of the data item x n .
Revised: Pr ( y n = k | x n ) , where y n denotes the true label of the data item x n .

Original: In addition, there exists (cid:101) f ∈ F such that ∥ (cid:101) f ( x n ) − f ♮ ( x n ) ∥ 2 ≤ ν for all x n ∼ D , where 0 ≤ ν < ∞ .
Revised: In addition, there exists (cid:101) f ∈ F such that ∥ (cid:101) f ( x n ) − f ♮ ( x n ) ∥ 2 ≤ ν for all x n ∼ D , where 0 ≤ ν < ∞ .

Original: Assumption 4 (Near-Class Specialist Assumption) Let A ♮m denote the ground-truth confusion matrix for annotator m .
Revised: Assumption 4 (Near-Class Specialist Assumption) Let A ♮m denote the ground-truth confusion matrix for annotator m .

Original: For each class k , there exists a near-class specialist, indexed by m k , such that ∥ A ♮m k ( k, :) − e ⊤ k ∥ 2 ≤ ξ 1 , where 0 ≤ ξ 1 < ∞ .
Revised: For each class k , there exists a near-class specialist, indexed by m k , such that ∥ A ♮m k ( k, :) − e ⊤ k ∥ 2 ≤ ξ 1 , where 0 ≤ ξ 1 < ∞ .

Original: Assumption 5 (Near-Anchor Point Assumption)
Revised: Assumption 5 (Near-Anchor Point Assumption)

Original: For each class k , there exists a near-anchor point, x n k , such that ∥ f ♮ ( x n k ) − e k ∥ 2 ≤ ξ 2 ) , where 0 ≤ ξ 2 < ∞ .
Revised: For each class k , there exists a near-anchor point, x n k , such that ∥ f ♮ ( x n k ) − e k ∥ 2 ≤ ξ 2 ) , where 0 ≤ ξ 2 < ∞ .

Revised: Assumptions 1-3 are standard conditions for analyzing performance of neural models under finite sample settings.

Original: The Near-Class Specialist Assumption (NCSA) implies that there exist annotators for each class k ∈
Revised: The Near-Class Specialist Assumption (NCSA) implies that there exist annotators for each class k ∈

Original: [ K ] who can correctly distinguish items belonging to class k from those belonging to other classes.
Revised: [ K ] who can correctly distinguish items belonging to class k from those belonging to other classes.

Original: It is important to note that NCSA assumption is different from the commonly used assumptions such as the existence of  expert annotators (Cao et al., 2019) or diagonal dominance of A ♮m ’s
Revised: It is important to note that NCSA assumption is much milder compared to the commonly used assumptions such as the existence of all-class expert annotators (Cao et al., 2019) or diagonally dominant A ♮m ’s

Original: (Tanno et al., 2019).
Revised: (Tanno et al., 2019).

Original: Such assumptions require the same annotators to be specialists with respect to all classes—also see Fig.
Revised: Instead, the NCSA only requires that for each class, there exists one annotator who is specialized for it (i.e., class-specialist)—but the annotator needs not be an all-class expert; see Fig.

Original: 1 in the supplementary material in Sec. M.
Revised: 1 in the supplementary material for illustration.

Original: Indeed, NCSA assumption is more relaxed, by making use of the collective “wisdom of the crowd”.

Original: The Near-Anchor Point Assumption (NAPA) is the relaxed version of the exact anchor point assumption commonly used in provable label noise learning methods (Xia et al., 2019;
Revised: The Near-Anchor Point Assumption (NAPA) is the relaxed version of the exact anchor point assumption commonly used in provable label noise learning methods (Xia et al., 2019;

Original: Li et al., 2021).
Revised: Li et al., 2021).

Original: Under Assumptions 1-5, we have the following result:
Revised: Under Assumptions 1-5, we have the following result:

Original: Theorem 1 Assume that each [ A m f ( x n )] k , ∀
Revised: Theorem 1 Assume that each [ A m f ( x n )] k , ∀

Original: A m ∈ A , ∀ f ∈ F is at least (1 /β ) .
Revised: A m ∈ A , ∀ f ∈ F is at least (1 /β ) .

Original: Also assume that σ max ( A ♮ m ) ≤ σ, ∀ m , for a certain σ > 0 .
Revised: Also assume that σ max ( A ♮m ) ≤ σ, ∀ m , for a certain σ > 0 .

Original: Then, with probability greater than 1 − K / N 18 , any optimal solution (cid:98)
Revised: Then, with probability greater than 1 − K / N 1 8 , any optimal solution (cid:98)

Original: A m and (cid:98) f of the problem (6) satisfies the following relations:
Revised: A m and (cid:98) f of the problem (6) satisfies the following relations:

Original: [Equation]
Revised: [Equation]

Original: where Π ∈ { 0 , 1 } K denotes a column permutation matrix,
Revised: 7b

Original: [Equation]
Revised: [Equation]

Revised: where η 2 = O (cid:16) βN 18 / √ S (cid:16) √ M log N + ( ∥ X ∥ F R F ) 14 (cid:17) + β √ KN 18 ν (cid:17) , Π ∈ { 0 , 1 } K is a permutation matrix, and X =

Original: and X = [ x 1 , . . 
Revised: [ x 1 , . . .

Original: , x N ] , if conditions ξ 1 , ξ 2 ≤ 1 / K , ν ≤ 1 / βK 2 N 18 , and S = |S| = Ω (cid:16) β 2 N 14 K 2 max (cid:16) M log N, (cid:112)
Revised: , x N ] , if conditions ξ 1 , ξ 2 ≤ 1 / K , ν ≤ 1 / βK 2 N 18 , and S = |S| = Ω (cid:16) β 2 N 14 K 2 max (cid:16) M log N, (cid:112)

Original: ∥
Revised: ∥

Original: X ∥ F R F (cid:17)(cid:17) hold.
Revised: X ∥ F R F (cid:17)(cid:17) hold.

Original: The proof of Theorem 1 is relegated to the supplementary material in Sec.
Revised: The proof of Theorem 1 is relegated to the supplementary material in Sec.

Original: B. 
Revised: B. The takeaways are as follows: First, the CCEM can provably identify the A ♮m ’s and the GTP f ♮ under finite samples—and such finite-sample identifiability of CCEM was not established before. Second, some restrictive conditions (e.g., the existence of all-class experts (Tanno et al., 2019; Cao et al., 2019)) used in the literature for establishing CCEM’s identifiability are actually not needed. Third, many works derived the CCEM criterion under the maximum likelihood principle via assuming that the annotators are conditionally independent; see (Rodrigues & Pereira, 2018; Chen et al., 2020; Tanno et al., 2019;

Revised: Chu et al., 2021).

Revised: Interestingly, as we revealed in our analysis, the identifiability of A ♮m and f ♮ under the CCEM criterion does not rely on the annotators’ independence.

Revised: These new findings support and explain the effectiveness of CCEM in a wide range of scenarios observed in the literature.

Original:  E NHANCING I DENTIFIABILITY VIA R EGULARIZATION
Revised:  E NHANCING I DENTIFIABILITY VIA R EGULARIZATION

Original: Theorem 1 confirms thatthe CCEM is a provably effective criterion for identifying A ♮m ’s and f ♮ .
Revised: Theorem 1 confirms that CCEM is a provably effective criterion for identifying A ♮m ’s and f ♮ .

Original: The caveats lie in Assumptions 4 and 5.
Revised: The caveats lie in Assumptions 4 and 5.

Original: Essentially, these assumptions require that some rows of the collection { A ♮m } Mm =1 and some vectors of the collection { f ♮ ( x n ) }
Revised: Essentially, these assumptions require that some rows of the collection { A ♮m } Mm =1 and some vectors of the collection { f ♮ ( x n ) }

Original: Nn =1 are close to the canonical vectors, respectively.
Revised: Nn =1 are close to the canonical vectors.

Original: These are reasonable assumptions, but satisfying them simultaneously may not always be possible.
Revised: These are reasonable assumptions, but satisfying them simultaneously may not always be possible.

Original: A natural question is if these assumptions can be relaxed, at least partially.
Revised: A natural question is if these assumptions can be relaxed, at least partially.

Original: In this section, we propose variants ofthe CCEM that admits enhanced identifiability.
Revised: In this section, we propose variants of CCEM that admits enhanced identifiability.

Original: To this end, we introduce the following geometric condition that is often used in theliterature of nonnegative matrix factorization (NMF):
Revised: To this end, we use a condition that is often adopted in the nonnegative matrix factorization (NMF) literature:

Original: Definition 1 (SSC)(Gillis, 2020) Consider the second-order cone C = { x ∈ R K |√ K − 1 ∥ x ∥ 2 ≤ 1 ⊤ x }.
Revised: Definition 1 (SSC)(Gillis, 2020) Consider the second-order cone C = { x ∈ R K |√ K − 1 ∥ x ∥ 2 ≤ 1 ⊤ x }.

Original: A nonnegative matrix Z ∈ R L × K + satisfies the sufficiently scattered condition (SSC) if (i)
Revised: A nonnegative matrix Z ∈ R L × K + satisfies the sufficiently scattered condition (SSC) if (i)

Original: C ⊆ cone( Z ⊤ ) and (ii) cone { Z ⊤ } ⊆ cone { Q } does not hold for any orthogonal Q ∈ R K × K except for the permutation matrices.
Revised: C ⊆ cone( Z ⊤ ) and (ii) cone { Z ⊤ } ⊆ cone { Q } does not hold for any orthogonal Q ∈ R K × K except for the permutation matrices.

Original: Geometrically, the matrix Z satisfying the SSC means that its rows span a large “area” within the nonnegative orthant, so that the conic hull spanned by the rows contains the second order cone C as a subset.
Revised: Geometrically, the matrix Z satisfying the SSC means that its rows span a large “area” within the nonnegative orthant, so that the conic hull spanned by the rows contains the second order cone C as a subset.

Original: Note that the SSC condition subsumes the the NCSA (Assumption (4) where ξ 1 = 0 ) and the
Revised: Note that the SSC condition subsumes the the NCSA (Assumption (4) where ξ 1 = 0 ) and the

Original: NAPA (Assumption (5) where ξ 1 = 0 ) as special cases.
Revised: NAPA (Assumption (5) where ξ 1 = 0 ) as special cases.

Original: Using SSC, we first show that the CCEM criterion in (6) attains identifiability of A ♮m ’s and f ♮ without using the NCSA or the NAPA, when the problem size grows sufficiently large.
Revised: Using SSC, we first show that the CCEM criterion in (6) attains identifiability of A ♮m ’s and f ♮ without using the NCSA or the NAPA, when the problem size grows sufficiently large.

Original: Theorem 2 Suppose that the incomplete labeling paradigm in Assumption 1 holds with S = Ω( t ) and N =
Revised: Theorem 2 Suppose that the incomplete labeling paradigm in Assumption 1 holds with S = Ω( t ) and N =

Original: O ( t 3 ) , for a certain t > 0 .
Revised: O ( t 3 ) , for a certain t > 0 .

Original: Also, assume that there exists Z = { n 1 , . . .
Revised: Also, assume that there exists Z = { n 1 , . . .

Original: , n T } such that
Revised: , n T } such that

Original: F ♮ Z =
Revised: F ♮ Z =

Original: [ f ♮ ( x n 1 ) , . . . , f ♮ ( x n T )] satisfies the SSC and W ♮ =
Revised: [ f ♮ ( x n 1 ) , . . . , f ♮ ( x n T )] satisfies the SSC and W ♮ =

Original: [ A ♮ ⊤ 1 , . . .
Revised: [ A ♮ ⊤ 1 , . . .

Original: , A ♮ ⊤ M ] satisfies the SSC as well.
Revised: , A ♮ ⊤ M ] satisfies the SSC as well.

Original: Then, at the limit of t → ∞ , if f ♮ ∈ F , the optimal solutions ( { (cid:98) A m } , (cid:98) f ) of (6) satisfies(cid:98)
Revised: Then, at the limit of t → ∞ , if f ♮ ∈ F , the optimal solutions ( { (cid:98) A m } , (cid:98) f ) of (6) satisfies

Original:  A m =
Revised: (cid:98) A m =

Original: A ♮m Π for all m and (cid:98) f ( x ) =
Revised: A ♮m Π for all m and (cid:98) f ( x ) =

Original: Π ⊤ f ♮ ( x ) for all x ∼ D , where Π∈ { 0 , 1 } K × K is acolumn permutation matrix.
Revised: Π ⊤ f ♮ ( x ) for all x ∼ D , where Π is a permutation matrix.

Original: The proof is via connecting the CCEM problem to the classic nonnegative matrix factorization (NMF) problem at the limit of t → ∞ ; see Sec.
Revised: The proof is via connecting the CCEM problem to the classic nonnegative matrix factorization (NMF) problem at the limit of t → ∞ ; see Sec.

Original: D. It is also worth to note that Theorem 2 does not require complete observation of the annotations to guarantee identifiability.
Revised: D. It is also worth to note that Theorem 2 does not require complete observation of the annotations to guarantee identifiability.

Original: Another remark is that W ♮ and F ♮ satisfying the SSC is more relaxed compared to that they satisfying the NCSA and NAPA, respectively.
Revised: Another remark is that W ♮ and F ♮ satisfying the SSC is more relaxed compared to that they satisfying the NCSA and NAPA, respectively.

Original: For example, one may need class specialists with much higher expertise to satisfy
Revised: For example, one may need class specialists with much higher expertise to satisfy

Original: NCSA than to satisfy SSC on W ♮ —also see geometric illustration in Sec.
Revised: NCSA than to satisfy SSC on W ♮ —also see geometric illustration in Sec.

Original: M.
Revised: M.

Original: Compared to the result in Theorem 1, Theorem 2 implies that when N is very large, the CCEM should work under less stringent conditions relative to the NCSA and NAPA.
Revised: Compared to the result in Theorem 1, Theorem 2 implies that when N is very large, the CCEM should work under less stringent conditions relative to the NCSA and NAPA.

Original: Nonetheless, the assumption that both the GTP and the annotators’ confusion matrices should meet certain requirements may not always hold.
Revised: Nonetheless, the assumption that both the GTP and the annotators’ confusion matrices should meet certain requirements may not always hold.

Original: In the following, we further show that the SSC on either W ♮ or F ♮ can be relaxed:
Revised: In the following, we further show that the SSC on either W ♮ or F ♮ can be relaxed:

Original: Theorem 3 Suppose that the incomplete labeling paradigm in Assumption 1 holds with S = Ω( t ) and N =
Revised: Theorem 3 Suppose that the incomplete labeling paradigm in Assumption 1 holds with S = Ω( t ) and N =

Original: O ( t 3 ) , for a certain t > 0 .
Revised: O ( t 3 ) , for a certain t > 0 .

Original: Then, at the limit of t → ∞ , if f ♮ ∈ F , we have A ∗ m =
Revised: Then, at the limit of t → ∞ , if f ♮ ∈ F , we have A ∗ m =

Original: A ♮m Π for all m and f ∗ ( x ) =
Revised: A ♮m Π for all m and f ∗ ( x ) =

Original: Π ⊤ f ♮ ( x ) for all x ∼ D , when any of the following conditions hold:
Revised: Π ⊤ f ♮ ( x ) for all x ∼ D , when any of the following conditions hold

Original:  (a) If there exists Z = { n 1 , . . .
Revised: : (a) If there exists Z = { n 1 , . . .

Original: , n T } such that  F ♮ Z  =
Revised: , n T } such that ( F ♮ Z ) ⊤ =

Original: [ f ♮ ( x n 1 ) , . . . , f ♮ ( x n T )]  satisfies the SSC, rank( W ♮ )
Revised: [ f ♮ ( x n 1 ) , . . . , f ♮ ( x n T )] ⊤ satisfies the SSC, rank( W ♮ )

Original: =
Revised: =

Original: K , and ( { A ∗ m } , f ∗ ) is the optimal solution of (6) with the maximal log det( F ∗ F ∗⊤ ) .
Revised: K , and ( { A ∗ m } , f ∗ ) is the optimal solution of (6) with the maximal log det( F ∗ F ∗⊤ ) .

Original: (b) If W ♮ =
Revised: (b) If W ♮ =

Original: [ A ♮ 1 ⊤ , . . .
Revised: [ A ♮ 1 ⊤ , . . .

Original: , A ♮ ⊤ M ]  satisfies the SSC, rank( F ♮ ) =
Revised: , A ♮ ⊤ M ] ⊤ satisfies the SSC, rank( F ♮ ) =

Original: K , and ( A ∗ m , f ∗ ) is the optimal solution of (6) with the maximal log det(( W ∗ ) ⊤ W ∗ ) .
Revised: K , and ( A ∗ m , f ∗ ) is the optimal solution of (6) with the maximal log det(( W ∗ ) ⊤ W ∗ ) .

Original: Theorem 3(a) shows that if the number of annotated data items is large and the GTP-outputs associated with x n are diverse enough to satisfy the SSC, then f ♮ and A ♮m are identifiable even if there are no class specialists available.
Revised: Theorem 3(a) shows that if the number of annotated data items is large and the GTP-outputs associated with x n are diverse enough to satisfy the SSC, then f ♮ and A ♮m are identifiable even if there are no class specialists available.

Original: Compared to Theorem 2, the identifiability is clearly enhanced, as the conditions on the annotators have been relaxed.
Revised: Compared to Theorem 2, the identifiability is clearly enhanced, as the conditions on the annotators have been relaxed.

Original: Theorem 3(b) implies that if W ♮ satisfies SSC, identifiability can be established irrespective of the geometry of F ♮ .
Revised: Theorem 3(b) implies that if W ♮ satisfies SSC, identifiability can be established irrespective of the geometry of F ♮ .

Original: Intuitively, when there are more annotators available, W ♮ is more likely to satisfy SSC.
Revised: Intuitively, when there are more annotators available, W ♮ is more likely to satisfy SSC (the relation between the size of W ♮ and SSC was shown in (Ibrahim et al., 2019, Theorem 4) under reasonable conditions).

Original: Hence, in practice, Theorem 3(b) may offer ensured performance in cases where more annotators are available.
Revised: Hence, in practice, Theorem 3(b) may offer ensured performance in cases where more annotators are available.

Original: Similar to the proof in Theorem 2, the proof here connects the CCEM-based E2E crowdsourcing problem to the simplex-structured matrix factorization (SSMF) problem (see (Gillis, 2020)) at the limit of t → ∞ .
Revised: Similar to the proof in Theorem 2, the proof here connects the CCEM-based E2E crowdsourcing problem to the simplex-structured matrix factorization (SSMF) problem (see (Gillis, 2020)) at the limit of t → ∞ .

Original: The detailed proof is provided in Sec.
Revised: The detailed proof is provided in Sec.

Original: E. We should mention that various connections between matrix factorization models and data labeling problems were also observed in related prior works, e.g, the classic approaches based on Dawid & Skene model (Dawid & Skene, 1979;
Revised: E. We should mention that various connections between matrix factorization models and data labeling problems were also observed in related prior works, e.g, the classic approaches based on Dawid & Skene model (Dawid & Skene, 1979;

Original: Zhang et al., 2016b; Traganitis et al., 2018) and E2E learning under noisy labels (Li et al., 2021).
Revised: Zhang et al., 2016b; Traganitis et al., 2018) and E2E learning under noisy labels (Li et al., 2021).

Original: Nonetheless, the former does not consider cross-entropy losses or involve any feature extractor, whereas, the latter does not consider multiple annotators or incomplete labeling.
Revised: Nonetheless, the former does not consider cross-entropy losses or involve any feature extractor, whereas, the latter does not consider multiple annotators or incomplete labeling.

Original: Implementation via Regularization Theorem 3 naturally leads to regularized versions of CCEM.
Revised: Implementation via Regularization Theorem 3 naturally leads to regularized versions of CCEM.

Original: Correspondingly, we have the following two cases:
Revised: Correspondingly, we have the following two cases:

Original: Following Theorem 3(a), when the GTP is believed to be diverse over different x n ’s, we propose to employ the following regularized CCEM:
Revised: Following Theorem 3(a), when the GTP is believed to be diverse over different x n ’s, we propose to employ the following regularized CCEM:

Original: [Equation]
Revised: [Equation]

Original: In similar spirit, following Theorem 3(b), we propose the following regularized CCEM for enhanced identifiability when M is large (so that W ♮ likely satisfies SSC):
Revised: In similar spirit, following Theorem 3(b), we propose the following regularized CCEM for enhanced identifiability when M is large (so that W ♮ likely satisfies SSC):

Original: [Equation]
Revised: [Equation]

Original: The implementations for (8) and (9) are called as geometry-regularized crowdsourcing network, abbreviated as GeoCrowdNet(F) and GeoCrowdNet(W) , respectively.
Revised: The implementations for (8) and (9) are called as geometry-regularized crowdsourcing network, abbreviated as GeoCrowdNet(F) and GeoCrowdNet(W) , respectively.

Revised: Remark 1 Based on our analyses, our suggested rule of thumb for choosing regularization is as follows: When M is large but N is relatively small, using GeoCrowdNet(W) is recommended, as W is more likely to satisfy the SSC compared to F ⊤ .

Revised: However, when N is large, using

Revised: GeoCrowdNet(F) is expected to have a better performance as F ⊤ is likely to satisfy the SSC in this case.

Original: Remark 1 Under big data settings, N is often large and the GTP outputs of x n ’s are often reasonably diverse.
Revised: Under big data settings, N is often large and the GTP outputs of x n ’s are often reasonably diverse.

Original: Hence, (8) oftentimes works well, as will be seen in our experiments.
Revised: Hence, (8) oftentimes works well, as will be seen in our experiments.

Original: Nonetheless, for cases where the geometry of F does not satisfy the SSC, the employment of more annotators can help via using (9)—which is also an intuitive advantage of crowdsourcing.[-Remark 
-]
Revised: Nonetheless, for cases where the geometry of F ⊤ violates the SSC, the employment of more annotators can help via using (9)—which is also an intuitive advantage of crowdsourcing.[-Remark 
-]

Original: Many works derive the CCEM criterion using the assumption that the annotators are conditionally independent; see (Rodrigues & Pereira, 2018;

Original: Chen et al., 2020; Tanno et al., 2019).

Original: The CCEM is a maximum likelihood criterion if such independence holds.

Original: Interestingly, as we revealed in our analysis, the identifiability of A ♮m and f ♮ under the CCEM criterion does not rely on the annotators’ independence with each other.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Baselines.
Revised: Baselines.

Original: The proposed methods are compared with a number of existing E2E crowdsourcing methods, namely, TraceReg (Tanno et al., 2019), MBEM (Khetan et al., 2018), CrowdLayer(Rodrigues & Pereira, 2018), CoNAL (Chu et al., 2021), and Max-MIG (Cao et al., 2019). In addition
Revised: The proposed methods are compared with a number of existing E2E crowdsourcing methods, namely, TraceReg (Tanno et al., 2019), MBEM (Khetan et al., 2018), CrowdLayer

Original: [Table]
Revised: [Table]

Original: to these baselines
Revised: (Rodrigues & Pereira, 2018), CoNAL (Chu et al., 2021), and Max-MIG (Cao et al., 2019).

Original: [Table] we also learn neural network classifiers using the labels aggregated from majority voting and the classic EM algorithm proposed by Dawid &S kene (Dawid & Skene, 1979), denoted as NN-MV and NN-DSEM , respectively.Semi-Real Data Experiments.
Revised: In addition to these baselines, we also learn neural network classifiers using the labels aggregated from majority voting and the classic EM algorithm proposed by Dawid &S kene (Dawid & Skene, 1979), denoted as NN-MV and NN-DSEM , respectively.

Original: We consider CIFAR-10 (Krizhevsky, 2009) dataset for experiments with labels generated following the model in (3).

Original: Due to page limitations, we present the results in Sec. O. The results are used to validate our identifiability claims under ideal scenarios where no model mismatches exist.
Revised: Real

Original: Real-Data Experiments with Machine Annotations.
Revised: Data Experiments with Machine Annotations.

Original: The settings are as follows:
Revised: The settings are as follows

Original:  Dataset.
Revised: : Dataset.

Original: We use the MNIST dataset (Deng, 2012) which consists of 70 , 000 handwritten digit images of 0 to 9 (i.e., K = 10 ) see more details in Sec.
Revised: We use the MNIST dataset (Deng, 2012) and Fashion-MNIST dataset (Xiao et al., 2017); see more details in Sec.

Original: O.
Revised: O.

Original: Noisy Label Generation.
Revised: Noisy Label Generation.

Original: We train a number of machine classifiers to produce noisy labels.
Revised: We train a number of machine classifiers to produce noisy labels.

Original: Five types of classifiers, including support vector machines (SVM), k -Nearest Neighbour (kNN), logistic regression, convolutional neural network (CNN), and fully connected neural network are considered to act as annotators.
Revised: Five types of classifiers, including support vector machines (SVM), k -Nearest Neighbour (kNN), logistic regression, convolutional neural network (CNN), and fully connected neural network are considered to act as annotators.

Original: To create more annotators, we change some of their parameters (e.g., the number of nearest neighbour parameter k of kNN and the number of epochs for training the CNN) while training them.
Revised: To create more annotators, we change some of their parameters (e.g., the number of nearest neighbour parameter k of kNN and the number of epochs for training the CNN).

Original: We consider two different cases and use the following strategies for training the machine annotators:

Original:  (i) Case 1: NCSA (Assumption 4) holding: Each annotator is chosen to be a specialist uniformly at random with probability 0 .1 (Here, we choose the same annotator to be a specialist for all classes.)
Revised: We consider two different cases: (i) Case 1: NCSA (Assumption 4) holding: Each annotator is chosen to be an all-class expert with probability 0 .

Original: If a particular annotator is chosen to be a specialist, it is trained carefully so that its classification accuracy exceeds a certain threshold; see details in Sec. 
Revised: If an annotator is chosen to be an all-class expert, it is trained carefully so that its classification accuracy exceeds a certain threshold; see details in Sec. O.

Revised: Note that the existence of an expert implies that the NCSA and the SSC are likely satisfied by W ♮ , but the reverse is not necessarily true.

Revised: We use this strategy to enforce NCSA only for validation purpose.

Revised: It does not mean that the CCEM criterion needs the existence of experts to work.

Original: O. (ii) Case 2: NCSA not holding: Each machine annotator is trained by randomly choosing avery small subset of theMNIST training data (size ranging between 100 to 500 ) so that the accuracy of these machine annotators areexpected to be low. 
Revised: (ii) Case 2: NCSA not holding: Each machine annotator is trained by randomly choosing a small subset of the training data (with 100 to 500 samples) so that the accuracy of these machine annotators are low. This way, any all-class expert or class-specialist is unlikely to exist.

Revised: We use the two cases to validate our theorems.

Revised: Under our analyses, GeoCrowdNet(W) is expected to work better under Case 1, as NCSA approximately implies SSC.

Revised: In addition, GeoCrowdNet(F) does not rely on the geometry of W ♮ , and thus can work under both cases, if N is reasonably large (which implies that ( F ♮ )

Revised: ⊤ is more likely to satisfy the SSC).

Original: Once the machine annotators are trained, we let them label unseen data items of size N .
Revised: Once the machine annotators are trained, we let them label unseen data items of size N .

Original: To evaluate the methods under incomplete labeling,we observe each label from an annotator uniformly at random witha probability of 0 . 1 .
Revised: To evaluate the methods under incomplete labeling, an annotator labels any data item with probability p = 0 . Under this labeling strategy, every data item is only labeled by a small subset of the annotators.

Original: Settings.
Revised: Settings.

Original: The neural network classifier architecture for MNIST dataset is chosen to be [-Lenet-
-]
Revised: The neural network classifier architecture for MNIST dataset is chosen to be [-Lenet-
-]Lenet-

Original: (Lecun et al., 1998), which consists of two sets of convolutional and max pooling layers, followed by a flattening convolutional layer, two fully-connected layers and finally a softmax layer. 
Revised: (Lecun et al., 1998), which consists of two sets of convolutional and max pooling layers, followed by a flattening convolutional layer, two fully-connected layers and finally a softmax layer. For

Revised: Fashion-MNIST dataset, we use the ResNet-18 architecture (He et al., 2016).

Original: Adam (Kingma & Ba, 2015) is used an optimizer with weight decay of 10 − 4 and batch size of 128.
Revised: Adam (Kingma & Ba, 2015) is used an optimizer with weight decay of 10 − 4 and batch size of 128.

Original: The regularization parameter λ and the initial learning rate of the Adam optimizer are chosen via grid search method using the validation set from { 0 . 01 , 0 . 001 , 0 . 0001 } and { 0 . 01 , 0 . 001 } , respectively.
Revised: The regularization parameter λ and the initial learning rate of the Adam optimizer are chosen via grid search method using the validation set from { 0 . 01 , 0 . 001 , 0 . 0001 } and { 0 . 01 , 0 . 001 } , respectively.

Original: We choose the same neural network structures for all the baselines.
Revised: We choose the same neural network structures for all the baselines.

Original: The confusion matrices are initialized with identity matrices of size K for proposed methods and the baselines TraceReg and CrowdLayer .[Table]
Revised: The confusion matrices are initialized with identity matrices of size K for proposed methods and the baselines TraceReg and CrowdLayer .

Original: Results.
Revised: Results.

Original: Table 1 presents the average label prediction accuracies on the testing data of the MNIST  over 5 random trials, for various cases.
Revised: Table 1 presents the average label prediction accuracy on the testing data of the MNIST and the Fashion-MNIST over 5 random trials, for various cases.

Original: One can observe that GeoCrowdNet(F) performs the best when there are more number of annotated items, even when there are no class specialist annotators present (i.e., case 2 with NCSA not holding).
Revised: One can observe that GeoCrowdNet(F) performs the best when there are more number of annotated items, even when there are no class specialist annotators present (i.e., case 2).

Original: On the other hand, GeoCrowdNet(W) starts showing  its advantage over GeoCrowdNet(F) when there are more number of annotators and class specialists among them.
Revised: On the other hand, GeoCrowdNet(W) starts showing [Table] its advantage over GeoCrowdNet(F) when there are more number of annotators and class specialists among them.

Original: Both are consistent with our theorems.
Revised: Both are consistent with our theorems.

Revised: In addition, the baseline MaxMIG performs competitively when there are all-class experts—which is consistent with their identifiability analysis.

Revised: However, when there are no such experts (case 2), its performance drops compared to the proposed methods, especially GeoCrowdNet(F) whose identifiability does not rely on the existence of all-class experts or class specialists.

Original: Overall, GeoCrowdNet(F) exhibits consistently good performance in all the cases.
Revised: Overall, GeoCrowdNet(F) exhibits consistently good performance.

Original: Real-Data Experiments with Human Annotators.
Revised: Real-Data Experiments with Human Annotators.

Original: The settings are as follows:
Revised: The settings are as follows:

Original: Datasets.
Revised: Datasets.

Original: We consider two different datasets, namely, LabelMe ( M = 59 ) (Rodrigues et al., 2017;
Revised: We consider two different datasets, namely, LabelMe ( M = 59 ) (Rodrigues et al., 2017;

Original: Russell et al., 2007) and Music ( M = 44 ) (Rodrigues et al., 2014), both having noisyand incomplete labels provided by human workers from AMT.
Revised: Russell et al., 2007) and Music ( M = 44 ) Rodrigues et al., 2014), both having noisy

Original: Settings.
Revised: and incomplete labels provided by human workers from AMT.

Original:  For LabelMe, we employ the pretrained VGG-16 embeddings followed by a fully connected layer as used in (Rodrigues & Pereira, 2018).
Revised: Settings. For LabelMe , we employ the pretrained VGG-16 embeddings followed by a fully connected layer as used in (Rodrigues & Pereira, 2018).

Original: For Music, we choose a similar architecture, but with batch normalization layers.
Revised: For Music, we choose a similar architecture, but with batch normalization layers.

Original: We use a batch size of 128 for LabelMe and a batch size of 100 for Music dataset.
Revised: We use a batch size of 128 for LabelMe and a batch size of 100 for Music dataset.

Original: All other settings are the same as the machine annotator case.
Revised: All other settings are the same as the machine annotator case.

Original: More details of the datasets and architecture can be seen in Sec.
Revised: More details of the datasets and architecture can be seen in Sec.

Original: O.
Revised: O.

Original: Results.
Revised: Results.

Original: Table 2 shows the average label prediction accuracies on the test data for 5 random trials.
Revised: Table 2 shows the average label prediction accuracies on the test data for 5 random trials.

Original: One can see that GeoCrowdNet ( λ =0) works reasonably well relative to the machine annotation case in the previous experiment.
Revised: One can see that GeoCrowdNet ( λ =0) works reasonably well relative to the machine annotation case in the previous experiment.

Original: This may be because the number of annotators for both LabelMe and Music are reasonably large ( M = 59 and M = 44 , respectively), which makes W ♮ satisfying
Revised: This may be because the number of annotators for both LabelMe and Music are reasonably large ( M = 59 and M = 44 , respectively), which makes W ♮ satisfying

Original: Assumption 4 become easier than before.
Revised: Assumption 4 become easier than before.

Original: This validates our claims in Theorem 1, but also shows the limitations of the plain-vanilla CCEM—that is, it may require a fairly large number of annotators to start showing promising results.
Revised: This validates our claims in Theorem 1, but also shows the limitations of the plain-vanilla CCEM—that is, it may require a fairly large number of annotators to start showing promising results.

Original: Similar to the previous experiment, both GeoCrowdNet(F) and GeoCrowdNet(W) outperform the unregularized version GeoCrowdNet ( λ =0), showing the advantages of the identifiability-enhancing regularization terms. 
Revised: Similar to the previous experiment, both GeoCrowdNet(F) and GeoCrowdNet(W) outperform the unregularized version GeoCrowdNet ( λ =0), showing the advantages of the identifiability-enhancing regularization terms. More Experiments. More experiments and details are presented in Sec. O, due to page limitations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we revisited the CCEM criterion—one of the most popular E2E learning criteria for crowdsourced label integration.
Revised: In this work, we revisited the CCEM criterion—one of the most popular E2E learning criteria for crowdsourced label integration.

Original: We provided the first finite-sample identifiability characterization of the confusion matrices and the neural classifier which are learned using the CCEM criterion.
Revised: We provided the first finite-sample identifiability characterization of the confusion matrices and the neural classifier which are learned using the CCEM criterion.

Original: Compared to many exiting identifiability results, our guarantees are under more relaxed and more realistic settings.
Revised: Compared to many exiting identifiability results, our guarantees are under more relaxed and more realistic settings.

Original: In particular, a take-home point revealed in our analysis is that CCEM can provably identify the desired model parameters even if the annotators are dependent, which is a surprising but favorable result.
Revised: In particular, a take-home point revealed in our analysis is that CCEM can provably identify the desired model parameters even if the annotators are dependent, which is a surprising but favorable result.

Original: We also proposed two regularized variants of the CCEM, based on the insights learned from our identifiability analysis for the plain-vanilla CCEM.
Revised: We also proposed two regularized variants of the CCEM, based on the insights learned from our identifiability analysis for the plain-vanilla CCEM.

Original: The regularized CCEM criteria provably enhance the identifiability of the confusion matrices and the neural classifier under more challenging scenarios.
Revised: The regularized CCEM criteria provably enhance the identifiability of the confusion matrices and the neural classifier under more challenging scenarios.

Original: We evaluated the proposed approaches on various synthetic and real datasets.
Revised: We evaluated the proposed approaches on various synthetic and real datasets.

Original: The results corroborate our theoretical claims.
Revised: The results corroborate our theoretical claims.

Original: Supplementary Material of “ Deep Learning From Crowdsourced Labels: Coupled Cross-entropy Minimization, Identifiability, and Regularization” A N OTATION
Revised: Supplementary Material of “ Deep Learning From Crowdsourced Labels: Coupled Cross-entropy Minimization, Identifiability, and Regularization” A N OTATION

Original: x , x , and X represent a scalar, a vector, and a matrix, respectively.
Revised: x , x , and X represent a scalar, a vector, and a matrix, respectively.

Original: [ x ] i and x ( i ) both denote the i th entry of the vector x .
Revised: [ x ] i and x ( i ) both denote the i th entry of the vector x .

Original: X (: , j ) denotes the j th column vector of X and X ( i, :) denotes the i th row vector of X .
Revised: X (: , j ) denotes the j th column vector of X and X ( i, :) denotes the i th row vector of X .

Original: [ X ] i,j and X ( i, j ) both mean the ( i, j ) th entry of X .
Revised: [ X ] i,j and X ( i, j ) both mean the ( i, j ) th entry of X .

Original: ∥ x ∥ 2 and ∥ X ∥ F mean the
Revised: ∥ x ∥ 2 and ∥ X ∥ F mean the

Original: Euclidean (Frobenius) norm of the augment.
Revised: Euclidean (Frobenius) norm of the augment.

Original: [ I ] means an integer set { 1 , 2 , . .
Revised: [ I ] means an integer set { 1 , 2 , . .

Original: , I } .
Revised: , I } .

Original: ⊤ denote transpose.
Revised: ⊤ denote transpose.

Original: ∆ K = { x ∈ R K : (cid:80) i x ( i ) = 1 , x ≥ 0 } denotes the probability simplex.
Revised: ∆ K = { x ∈ R K : (cid:80) i x ( i ) = 1 , x ≥ 0 } denotes the probability simplex.

Original: X ≥ 0 implies that all the entries of the matrix X are nonnegative.
Revised: X ≥ 0 implies that all the entries of the matrix X are nonnegative.

Original: I [ A ] denotes an indicator function for the event
Revised: I [ A ] denotes an indicator function for the event

Original: A such that I
Revised: A such that I

Original: [ A ] = 1 if the event A happens, otherwise I [ A ] = 0 .
Revised: [ A ] = 1 if the event A happens, otherwise I [ A ] = 0 .

Original: CE ( x , y )
Revised: CE ( x , y )

Original: = − (cid:80)
Revised: = − (cid:80)

Original: Kk =

Revised: Kk =

Original: I [ y = k ] log( x ( k )) denotes the cross entropy function.
Revised: I [ y = k ] log( x ( k )) denotes the cross entropy function.

Original: I K denotes an identity matrix of size K × K .
Revised: I K denotes an identity matrix of size K × K .

Original: σ max ( X ) and σ min ( X ) denote the largest and the smallest singular values of the matrix X , respectively.
Revised: σ max ( X ) and σ min ( X ) denote the largest and the smallest singular values of the matrix X , respectively.

Original: cone ( X ) denotes the conic hull formed by the columns of the matrix X .
Revised: cone ( X ) denotes the conic hull formed by the columns of the matrix X .

Original: det( X ) represents the determinant of X . vec ( X ) denotes the vectorization operation that concatenates the columns of X
Revised: det( X ) represents the determinant of X . vec ( X ) denotes the vectorization operation that concatenates the columns of X

Original: . trace ( X ) outputs the trace (the sum of the diagonal elements) of X .
Revised: . trace ( X ) outputs the trace (the sum of the diagonal elements) of X .

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: If the vectors p ( m ) n for all m, n are available, then one can represent the model in (2) as a nonnegative matrix factorization (NMF) model as follows:
Revised: If the vectors p ( m ) n for all m, n are available, then one can represent the model in (2) as a nonnegative matrix factorization (NMF) model as follows:

Original: [Equation]
Revised: [Equation]

Original: where the factors W and F are both nonnegative per their physical meaning.
Revised: where the factors W and F are both nonnegative per their physical meaning.

Original: Let P ♮ denote the ground-truth and p ♮ ( m ) n denote the ( m, n ) th block in P ♮ , following the representation in (10).
Revised: Let P ♮ denote the ground-truth and p ♮ ( m ) n denote the ( m, n ) th block in P ♮ , following the representation in (10).

Original: Note that we do not observe the entire P ♮ but only the (cid:98) y ( m ) n sampled from p ♮ ( m ) n for ( m, n ) ∈ S .
Revised: Note that we do not observe the entire P ♮ but only the (cid:98) y ( m ) n sampled from p ♮ ( m ) n for ( m, n ) ∈ S .

Original: We first show that the CCEM criterion in (6) implicitly estimate P ♮ from incomplete labeling represented by S .
Revised: We first show that the CCEM criterion in (6) implicitly estimate P ♮ from incomplete labeling represented by S .

Original: To this end, we consider the objective function in (6a) by choosing
Revised: To this end, we consider the objective function in (6a) by choosing

Original: CE
Revised: CE

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) Y denotes the collection of all (cid:98) y ( m ) n , ( m, n ) ∈ S .
Revised: where (cid:98) Y denotes the collection of all (cid:98) y ( m ) n , ( m, n ) ∈ S .

Original: Let { (cid:98)
Revised: Let { (cid:98)

Original: A m } M m =1 and (cid:98) f denote the estimates of the learning criterion in (6).
Revised: A m } M m =1 and (cid:98) f denote the estimates of the learning criterion in (6).

Original: Then, the criterion (6) can be re-written as:
Revised: Then, the criterion (6) can be re-written as:

Original: [Equation]
Revised: [Equation]

Original: Y ) , (12) where (cid:98) P =
Revised: Y ) , (12) where (cid:98) P =

Original: (cid:99) W (cid:2) (cid:98) f ( x 1 ) . . .
Revised: (cid:99) W (cid:2) (cid:98) f ( x 1 ) . . .

Original: (cid:98) f ( x N ) (cid:3) , (cid:99) W =
Revised: (cid:98) f ( x N ) (cid:3) , (cid:99) W =

Original: [ (cid:98)
Revised: [ (cid:98)

Original: A ⊤ 1 , . . .
Revised: A ⊤ 1 , . . .

Original: , (cid:98) A ⊤ M ] ⊤ and
Revised: , (cid:98) A ⊤ M ] ⊤ and

Original: P ≜ { P
Revised: P ≜ { P

Original: [Equation]
Revised: [Equation]

Original: MK × N | P = W
Revised: MK × N | P = W

Original: [ f
Revised: [ f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Our main goal is to characterize the estimation errors of { (cid:98) A m } Mm =1 and (cid:98) f .
Revised: Our main goal is to characterize the estimation errors of { (cid:98) A m } Mm =1 and (cid:98) f .

Original: This can be done via first bounding the estimation errors ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 , where p ♮ ( m ) n is given by:
Revised: This can be done via first bounding the estimation errors ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 , where p ♮ ( m ) n is given by:

Original: [Equation]
Revised: [Equation]

Original: in which A ♮m and f ♮ denote the m th ground-truth confusion matrix and the GTP, respectively and (cid:98) p ( m ) n denote the ( m, n ) th block in (cid:98) P .
Revised: in which A ♮m and f ♮ denote the m th ground-truth confusion matrix and the GTP, respectively and (cid:98) p ( m ) n denote the ( m, n ) th block in (cid:98) P .

Original: B. E STIMATING
Revised: B. E STIMATING

Original: [Equation]
Revised: [Equation]

Original: We have the following proposition:
Revised: We have the following proposition:

Original: Proposition 1 Under the assumptions given in Theorem 1, the following results hold:
Revised: Proposition 1 Under the assumptions given in Theorem 1, the following results hold:

Original: With probability greater than 1 − δ :
Revised: With probability greater than 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: where R = 16 √ S
Revised: where R = 16 √ S

Original: [Equation]
Revised: [Equation]

Original: With probability greater than 1 − δ − 1 N 18 , for any n ∈ [ N ] :
Revised: With probability greater than 1 − δ − 1 N 18 , for any n ∈ [ N ] :

Original: [Equation]
Revised: [Equation]

Original: The proof is provided in Sec.
Revised: The proof is provided in Sec.

Original: C.
Revised: C.

Original: B. E STIMATION ERRORS FOR (cid:98) A m AND (cid:98) f
Revised: B. E STIMATION ERRORS FOR (cid:98) A m AND (cid:98) f

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we characterize the estimation errors for the confusion matrices and the GTP.
Revised: we characterize the estimation errors for the confusion matrices and the GTP.

Original: Before proceeding
Revised: Before proceeding

Original: [Equation]
Revised: [Equation]

Original: we re-express the results in Proposition 1:
Revised: we re-express the results in Proposition 1:

Original: [Equation]
Revised: [Equation]

Original: with probability greater than 1 − δ and 1 − δ − 1 N 18 , respectively, for certain ζ, φ > 0 , where p ♮ ( m ) n denotes ( m
Revised: with probability greater than 1 − δ and 1 − δ − 1 N 18 , respectively, for certain ζ, φ > 0 , where p ♮ ( m ) n denotes ( m

Original: n
Revised: n

Original: ) th block in the matrix P ♮ and (cid:98) p ( m ) n denotes ( m, n ) th block in the matrix (cid:98) P —see (10)
Revised: ) th block in the matrix P ♮ and (cid:98) p ( m ) n denotes ( m, n ) th block in the matrix (cid:98) P —see (10)

Original: Assumptions 4-5 imply that there exists index sets Λ = { ˜ m 1 , . . . ,
Revised: Assumptions 4-5 imply that there exists index sets Λ = { ˜ m 1 , . . . ,

Original: ˜ m K } and Ψ = { ˜ n 1 , . .
Revised: ˜ m K } and Ψ = { ˜ n 1 , . .

Original: ˜ n K } such that:
Revised: ˜ n K } such that:

Original: [Equation]
Revised: [Equation]

Original: where ∥ N W ∥ F ≤ √ Kξ 1 , and ∥ N F ∥ F ≤ √ Kξ 2 .
Revised: where ∥ N W ∥ F ≤ √ Kξ 1 , and ∥ N F ∥ F ≤ √ Kξ 2 .

Original: Note that the conditions do not require any confusion matrices to be near identity, i.e., we do not require any annotators to be all-class specialists.
Revised: Note that the conditions do not require any confusion matrices to be near identity, i.e., we do not require any annotators to be all-class specialists.

Original: Nonetheless, for notation simplicity, we let Λ = { 1 , . . .
Revised: Nonetheless, for notation simplicity, we let Λ = { 1 , . . .

Original: , K } and Ψ = { 1 , . .
Revised: , K } and Ψ = { 1 , . .

Original: , K } , without loss of generality.
Revised: , K } , without loss of generality.

Original: Hence, we have the following:
Revised: Hence, we have the following:

Original: [Equation]
Revised: [Equation]

Original: where A ♮ 1 =
Revised: where A ♮ 1 =

Original: I K + N W , F ♮ 

Revised: I K + N W , F ♮

Original: =
Revised: =

Original: I K + N F , and F ♮ 2 ∈ R K × ( N − K ) , ∥ N W ∥ F ≤ √ Kξ 1 , and ∥ N F ∥ F ≤ √ Kξ 2 .
Revised: I K + N F , and F ♮ 2 ∈ R K × ( N − K ) , ∥ N W ∥ F ≤ √ Kξ 1 , and ∥ N F ∥ F ≤ √ Kξ 2 .

Original: Note that letting Λ = { 1 , . .
Revised: Note that letting Λ = { 1 , . .

Original: , K } and Ψ = { 1 , . .
Revised: , K } and Ψ = { 1 , . .

Original: , K } for our analysis is without loss of any generality since we will be analyzing the error term ∥
Revised: , K } for our analysis is without loss of any generality since we will be analyzing the error term ∥

Original: A ♮ 1 F ♮ 1 − (cid:98)
Revised: A ♮ 1 F ♮ 1 − (cid:98)

Original: A 1 (cid:98) F 1 ∥ 2F and it satisfies
Revised: A 1 (cid:98) F 1 ∥ 2F and it satisfies

Original: [Equation]
Revised: [Equation]

Original: Let (cid:98)
Revised: Let (cid:98)

Original: A m and (cid:98) f denote the estimated values of A ♮m and f ♮ using the learning criterion (6) such that
Revised: A m and (cid:98) f denote the estimated values of A ♮m and f ♮ using the learning criterion (6) such that

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) A m ∈ R K × K , (cid:98) F 1 ∈ R K
Revised: where (cid:98) A m ∈ R K × K , (cid:98) F 1 ∈ R K

Original: × K
Revised: × K

Original: [Equation]
Revised: [Equation]

Original: and (cid:98) F 2 ∈ R K × ( N − K ) .
Revised: and (cid:98) F 2 ∈ R K × ( N − K ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Let us define the error matrices as below:
Revised: Let us define the error matrices as below:

Original: [Equation]
Revised: [Equation]

Original: where Π ∈ [0 , 1] K × K is a column permutation matrix and is the same across all the error blocks.
Revised: where Π ∈ [0 , 1] K × K is a column permutation matrix and is the same across all the error blocks.

Original: We hope to characterize the norm of these error matrices.
Revised: We hope to characterize the norm of these error matrices.

Original: Upper bound for ∥ E (1) A ∥ F and ∥ E (1) F ∥ F .
Revised: Upper bound for ∥ E (1) A ∥ F and ∥ E (1) F ∥ F .

Original: We start by considering the first term on the R.H.S. of (15), i.e., ∥ I K + N W
Revised: We start by considering the first term on the R.H.S. of (15), i.e., ∥ I K + N W

Original: + N F + N W N F − (cid:98)
Revised: + N F + N W N F − (cid:98)

Original: A 1 (cid:98) F 1 ∥ 2F .
Revised: A 1 (cid:98) F 1 ∥ 2F .

Original: From (14), we have the following with probability greater than 1 −
Revised: From (14), we have the following with probability greater than 1 −

Original: Kδ − K N 18 :
Revised: Kδ − K N 18 :

Original: [Equation]
Revised: [Equation]

Original: where (cid:102) N = N W + N F + N W N F .
Revised: where (cid:102) N = N W + N F + N W N F .

Original: The absolute value of the largest entry in (cid:102) N can be bounded by ξ 1 + ξ 2 + √
Revised: The absolute value of the largest entry in (cid:102) N can be bounded by ξ 1 + ξ 2 + √

Original: Kξ
Revised: Kξ

Original: 1 ξ 2 .
Revised: 1 ξ 2 .

Original: Let us denote κ = φ + ξ 1 + ξ 2 + √
Revised: Let us denote κ = φ + ξ 1 + ξ 2 + √

Original: Kξ
Revised: Kξ

Original: ξ 2 for conciseness.
Revised: ξ 2 for conciseness.

Original: Then, from (18), we get the following conditions:
Revised: Then, from (18), we get the following conditions:

Original: [Equation]
Revised: [Equation]

Original: From the conditions (19a) and (19b), we have the following result:
Revised: From the conditions (19a) and (19b), we have the following result:

Original: Lemma 1 Assume that the conditions in (19a) and (19b) hold and that κ ≤ 1 K +1 .
Revised: Lemma 1 Assume that the conditions in (19a) and (19b) hold and that κ ≤ 1 K +1 .

Original: Then, we have the following relations satisfied:
Revised: Then, we have the following relations satisfied:

Original: [Equation]
Revised: [Equation]

Original: The proof of the lemma is given in Sec.
Revised: The proof of the lemma is given in Sec.

Original: F.
Revised: F.

Original: From the lower bound condition in (19a), we have the following result: 
Revised: From the lower bound condition in (19a) [Equation] we have the following result: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last equality is obtained due to the probability simplex constraints on the columns of (cid:98) F 1 .
Revised: where the last equality is obtained due to the probability simplex constraints on the columns of (cid:98) F 1 .

Original: From the lower bound condition in (19a), we have the below result as well:
Revised: From the lower bound condition in (19a), we have the below result as well:

Original: [Equation]
Revised: [Equation]

Original: where the last equality is obtained due to the probability simplex constraints on all the columns of (cid:98)
Revised: where the last equality is obtained due to the probability simplex constraints on all the columns of (cid:98)

Original: A 1 .
Revised: A 1 .

Original: Next, we characterize the term ∥ E (1) A ∥ 2F .
Revised: Next, we characterize the term ∥ E (1) A ∥ 2F .

Original: To achieve this, by employing the result (20a) in Lemma 1, we fix the column permutation Π as follows:
Revised: To achieve this, by employing the result (20a) in Lemma 1, we fix the column permutation Π as follows:

Original: [Equation]
Revised: [Equation]

Original: We get the following set of relations:
Revised: We get the following set of relations:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality ( b ) is by (21) and the relation ( a ) is obtained by choosing Π as defined in (23).
Revised: where the last inequality ( b ) is by (21) and the relation ( a ) is obtained by choosing Π as defined in (23).

Original: Hence, we have the following with probability greater than 1 −
Revised: Hence, we have the following with probability greater than 1 −

Original: Kδ − K N 18 :
Revised: Kδ − K N 18 :

Original: [Equation]
Revised: [Equation]

Original: We proceed to consider the error term ∥ E (1) F ∥ 2F .
Revised: We proceed to consider the error term ∥ E (1) F ∥ 2F .

Original: To achieve this, consider the following:
Revised: To achieve this, consider the following:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality ( b ) is by (22) and the relation ( a ) is by combining the definition of Π in (23) and (20c) in Lemma 1.
Revised: where the last inequality ( b ) is by (22) and the relation ( a ) is by combining the definition of Π in (23) and (20c) in Lemma 1.

Original: Hence, we get the following with probability greater than 1 −
Revised: Hence, we get the following with probability greater than 1 −

Original: Kδ − K N 18 :
Revised: Kδ − K N 18 :

Original: [Equation]
Revised: [Equation]

Original: Upper bound for ∥ E (2) F ∥ F .
Revised: Upper bound for ∥ E (2) F ∥ F .

Original: Next, we consider the second term in (15), i.e., ∥ F ♮ 2 + N W F ♮ 2 − (cid:98)
Revised: Next, we consider the second term in (15), i.e., ∥ F ♮ 2 + N W F ♮ [Equation] − [Equation]

Original: A 1 (cid:98) F 2 ∥ F .
Revised: A 1 (cid:98) F 2 ∥ F .

Original: From (13), we have the following with probability greater than 1 − δ :[Equation] ≥
Revised: From (13), we have the following with probability greater than 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: where the inequality ( a ) is by using the triangle inequality
Revised: where the inequality ( a ) is by using the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: ( b ) is obtained by applying the following two relations for any two matrices A ∈ R K × K , B ∈ R K × L , L ≥ K :
Revised: ( b ) is obtained by applying the following two relations for any two matrices A ∈ R K × K , B ∈ R K × L , L ≥ K :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The inequality ( c ) is obtained by applying (24).
Revised: The inequality ( c ) is obtained by applying (24).

Original: Hence, (26) combined with the fact that ∥ F 2 ∥ F ≤ ∥ F ♮ ∥
Revised: Hence, (26) combined with the fact that ∥ F 2 ∥ F ≤ ∥ F ♮ ∥

Original: F gives the below:
Revised: F gives the below:

Original: [Equation]
Revised: [Equation]

Original: Next, we use the following lemma to characterize σ min ( (cid:98) A 1 ) :
Revised: Next, we use the following lemma to characterize σ min ( (cid:98) A 1 ) :

Original: Lemma 2 Suppose that the matrix X ∈ R K × K takes the form X
Revised: Lemma 2 Suppose that the matrix X ∈ R K × K takes the form X

Original: =
Revised: =

Original: I K
Revised: I K

Original: + E 1 + E 2 .
Revised: + E 1 + E 2 .

Original: Assume that ∥ E 1 ∥ F ≤ υ 1 and ∥ E 2 ∥ F ≤ υ 2 , for a certain υ 1 , υ 2 > 0 .
Revised: Assume that ∥ E 1 ∥ F ≤ υ 1 and ∥ E 2 ∥ F ≤ υ 2 , for a certain υ 1 , υ 2 > 0 .

Original: Then, we have σ min ( X ) ≥ | 1 − υ 1 − υ
Revised: Then, we have σ min ( X ) ≥ | 1 − υ 1 − υ

Original: | .
Revised: | .

Original: The proof is relegated to Sec.
Revised: The proof is relegated to Sec.

Original: G. Applying Lemma 2, we get the following with probability greater than 1 − ( K + 1) δ − K N 18 :
Revised: G. Applying Lemma 2, we get the following with probability greater than 1 − ( K + 1) δ − K N 18 :

Original: [Equation]
Revised: [Equation]

Original: for certain constant c 1 > 0 .
Revised: for certain constant c 1 > 0 .

Original: where we used the fact that ξ 1 ≤ √ κ since κ = φ + ξ 1 + ξ 2 + √ Kξ 1 ξand and κ ≤ 1 .
Revised: where we used the fact that ξ 1 ≤ √ κ since κ = φ + ξ 1 + ξ 2 + √ Kξ 1 ξand and κ ≤ 1 .

Original: Upper bound for ∥ E ( m )
Revised: Upper bound for ∥ E ( m )

Original: A ∥ F , m > 1 .
Revised: A ∥ F , m > 1 .

Original: We consider the third term on the R.H.S. of (15).
Revised: We consider the third term on the R.H.S. of (15).

Original: From (14), for each m , we have the following with probability greater than 1 −
Revised: From (14), for each m , we have the following with probability greater than 1 −

Original: Kδ − K N 18 :
Revised: Kδ − K N 18 :

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the relation ( a ) by the triangle inequality, ( b ) is by applying (27) and (28), and ( c ) is via (25).
Revised: where the relation ( a ) by the triangle inequality, ( b ) is by applying (27) and (28), and ( c ) is via (25).

Original: Hence, ∀ m > 1 , we get the following with probability greater than 1 − 2 Kδ − 2 K N 18 :
Revised: Hence, ∀ m > 1 , we get the following with probability greater than 1 − 2 Kδ − 2 K N 18 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for certain constant c 2 > 0 , where we used the fact that φ ≤ √ κ and ξ 2 ≤ √ κ since κ = φ + ξ 1 + ξ 2 + √ Kξ 1 ξ 2 and κ ≤ 1 .
Revised: for certain constant c 2 > 0 , where we used the fact that φ ≤ √ κ and ξ 2 ≤ √ κ since κ = φ + ξ 1 + ξ 2 + √ Kξ 1 ξ 2 and κ ≤ 1 .

Original: In the above, we have also applied σ min ( (cid:98) F 1 ) ≥ | 1 − √ 2 Kκ − 2 √
Revised: In the above, we have also applied σ min ( (cid:98) F 1 ) ≥ | 1 − √ 2 Kκ − 2 √

Original: Kξ 2 | following Lemma 2 and (25).
Revised: Kξ 2 | following Lemma 2 and (25).

Original: Putting Together.
Revised: Putting Together.

Original: By applying the upper bounds obtained in (24) and (31), we have the following with probability greater than 1 − 2 Kδ − 2 K N 18 :
Revised: By applying the upper bounds obtained in (24) and (31), we have the following with probability greater than 1 − 2 Kδ − 2 K N 18 :

Original: [Equation]
Revised: [Equation]

Original: where we have used the fact that σ max ( A ♮m ) ≤ ∥
Revised: where we have used the fact that σ max ( A ♮m ) ≤ ∥

Original: A ♮m ∥ F ≤ √ K.
Revised: A ♮m ∥ F ≤ √ K.

Original: Similarly, by combining (25) and (30), we have the following with probability greater than 1 − (2 K + 1) δ − 2 K N
Revised: Similarly, by combining (25) and (30), we have the following with probability greater than 1 − (2 K + 1) δ − 2 K N

Original: [Equation]
Revised: [Equation]

Original: for certain constant c 3 > 0 , where we have used the fact that ∥ F ♮ ∥
Revised: for certain constant c 3 > 0 , where we have used the fact that ∥ F ♮ ∥

Original: F ≤ √ N.
Revised: F ≤ √ N.

Original: We hope to characterize the generalization performance of the predicted function (cid:98) f from (32).
Revised: We hope to characterize the generalization performance of the predicted function (cid:98) f from (32).

Original: Towards this, we have the following result:
Revised: Towards this, we have the following result:

Original: Lemma 3 Under Assumptions 1 and 2, the following holds with probability greater than 1 − δ :
Revised: Lemma 3 Under Assumptions 1 and 2, the following holds with probability greater than 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: The proof is given in Sec.
Revised: The proof is given in Sec.

Original: H. Hence, combining Lemma 3 with (32), we have the following with probability greater than 1 − 2( K + 1) δ − 2 K N
Revised: H. Hence, combining Lemma 3 with (32), we have the following with probability greater than 1 − 2( K + 1) δ − 2 K N

Original: [Equation]
Revised: [Equation]

Original: C P ROOF OF P ROPOSITION
Revised: C P ROOF OF P ROPOSITION

Original: Let us consider the following notation:
Revised: Let us consider the following notation:

Original: [Equation]
Revised: [Equation]

Original: where ω =
Revised: where ω =

Original: ( m, n ) ∈
Revised: ( m, n ) ∈

Original: [ M ] × [ N ] ; i.e., P ( ω ) “reads out” the ( m, n ) th block in (10).
Revised: [ M ] × [ N ] ; i.e., P ( ω ) “reads out” the ( m, n ) th block in (10).

Original: Similarly, we define (cid:98) Y ( ω ) =
Revised: Similarly, we define (cid:98) Y ( ω ) =

Original: (cid:98) y
Revised: (cid:98) y

Original: ( m ) n .
Revised: ( m ) n .

Original: With this notation, we have
Revised: With this notation, we have

Original: [Equation]
Revised: [Equation]

Original: where CE ( x , y )
Revised: where CE ( x , y )

Original: = − (cid:80)
Revised: = − (cid:80)

Original: Kk =

Revised: Kk =

Original: I [ y = k ] log( x ( k )) , S = |S| , and ω s = ( m s , n s ) ∈ S .
Revised: I [ y = k ] log( x ( k )) , S = |S| , and ω s = ( m s , n s ) ∈ S .

Original: Under Assumption 1, we can define:
Revised: Under Assumption 1, we can define:

Original: [Equation]
Revised: [Equation]

Original: where Π denotes the uniform distribution and π ( m ) n = 1 NM denotes the probability of observing the annotation for the index pair ( m, n ) .
Revised: where Π denotes the uniform distribution and π ( m ) n = 1 NM denotes the probability of observing the annotation for the index pair ( m, n ) .

Original: Eq.
Revised: Eq.

Original: (12) implies
Revised: (12) implies

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) P is given by:
Revised: where (cid:101) P is given by:

Original: [Equation]
Revised: [Equation]

Original: and f is function constructed under Assumption 3.
Revised: and f is function constructed under Assumption 3.

Original: Hence, by
Revised: Hence, by

Original: (cid:98) Y we have
Revised: (cid:98) Y we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is obtained from (35).
Revised: where the first inequality is obtained from (35).

Original: Let us consider the L.H.S. of (36):
Revised: Let us consider the L.H.S. of (36):

Original: [Equation]
Revised: [Equation]

Original: where expectation is taken w.r.t.
Revised: where expectation is taken w.r.t.

Original: (cid:98) Y (while taking expectation, we used the uniform probability π ( m ) n = 1 NM , ∀ , n, m for observing each annotation (cid:98) y ( m ) n and used the ground-truth probability
Revised: (cid:98) Y (while taking expectation, we used the uniform probability π ( m ) n = 1 NM , ∀ , n, m for observing each annotation (cid:98) y ( m ) n and used the ground-truth probability

Original: P ♮ (( m − 1) K + k, n ) for each event
Revised: P ♮ (( m − 1) K + k, n ) for each event

Original: I [ (cid:98) y
Revised: I [ (cid:98) y

Original: ( m ) n = k ] ) and D KL ( P ♮ , (cid:98) P ) is the average Kullback–Leibler (KL) divergence between the entries of the matrices P ♮ and (cid:98) P ∈ R MK × N , which is given by
Revised: ( m ) n = k ] ) and D KL ( P ♮ , (cid:98) P ) is the average Kullback–Leibler (KL) divergence between the entries of the matrices P ♮ and (cid:98) P ∈ R MK × N , which is given by

Original: [Equation]
Revised: [Equation]

Original: Upper-bounding the first term on the R.H.S of (36) .
Revised: Upper-bounding the first term on the R.H.S of (36) .

Original: Next, we characterize the first term on the
Revised: Next, we characterize the first term on the

Original: R.H.S. of (36).
Revised: R.H.S. of (36).

Original: To achieve this, we invoke the following theorem (Theorem 26.5 in (Shalev-Shwartz & Ben-David, 2014)) :
Revised: To achieve this, we invoke the following theorem (Theorem 26.5 in (Shalev-Shwartz & Ben-David, 2014)) :

Original: Theorem 4 (Shalev-Shwartz & Ben-David, 2014, Theorem 26.5) Assume that for all y and for all x , we have | CE ( x ; y ) | ≤ z max .
Revised: Theorem 4 (Shalev-Shwartz & Ben-David, 2014, Theorem 26.5) Assume that for all y and for all x , we have | CE ( x ; y ) | ≤ z max .

Original: Then for any P ∈ P , the following holds with probability greater than
Revised: Then for any P ∈ P , the following holds with probability greater than

Original: [Equation]
Revised: [Equation]

Original: where ℓ ◦ P ◦ S denotes the set
Revised: where ℓ ◦ P ◦ S denotes the set

Original: [Equation]
Revised: [Equation]

Original: and R S ( X ) denotes the empirical Rademacher complexity of the set X .
Revised: and R S ( X ) denotes the empirical Rademacher complexity of the set X .

Original: To apply Theorem 4, we will characterize R S ( ℓ ◦ P ◦ S ) which is defined as follows (Shalev-Shwartz & Ben-David, 2014):
Revised: To apply Theorem 4, we will characterize R S ( ℓ ◦ P ◦ S ) which is defined as follows (Shalev-Shwartz & Ben-David, 2014):

Original: [Equation]
Revised: [Equation]

Original: where expectation is w.r.t.
Revised: where expectation is w.r.t.

Original: the independent Rademacher random variables σ s ∈ {− 1 , 1 } .
Revised: the independent Rademacher random variables σ s ∈ {− 1 , 1 } .

Original: Note that
Revised: Note that

Original: P ( ω ) is a vector-valued [see Eq.
Revised: P ( ω ) is a vector-valued [see Eq.

Original: Hence, we invoke the following contraction result to upper bound R S ( ℓ ◦ P ◦ S ) :
Revised: Hence, we invoke the following contraction result to upper bound R S ( ℓ ◦ P ◦ S ) :

Original: Lemma 4 (Maurer, 2016) Let P be a class of mappings { P : X → R K } , where X be any set and ( ω 1 , . . .
Revised: Lemma 4 (Maurer, 2016) Let P be a class of mappings { P : X → R K } , where X be any set and ( ω 1 , . . .

Original: , ω S ) ∈ X S .
Revised: , ω S ) ∈ X S .

Original: Also assume that ℓ : R K → R has the Lipschitz constant L .
Revised: Also assume that ℓ : R K → R has the Lipschitz constant L .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where P k ( w ) denotes the k th component of P ( ω ) and σ sk is an independent (doubly indexed)
Revised: where P k ( w ) denotes the k th component of P ( ω ) and σ sk is an independent (doubly indexed)

Original: Rademacher random variable.
Revised: Rademacher random variable.

Original: Let us define a vector z ≜ ( P 1 ( ω 1 ) , P 2 ( ω 1 ) , . .
Revised: Let us define a vector z ≜ ( P 1 ( ω 1 ) , P 2 ( ω 1 ) , . .

Original: , P K − 1 ( ω s ) , P K ( ω s ))
Revised: , P K − 1 ( ω s ) , P K ( ω s ))

Original: ∈ R SK and the set Z ≜ { z = ( P 1 ( ω 1 ) , . .
Revised: ∈ R SK and the set Z ≜ { z = ( P 1 ( ω 1 ) , . .

Original: , P K ( ω s ))
Revised: , P K ( ω s ))

Original: | P ∈ P} .
Revised: | P ∈ P} .

Original: With these definitions, we apply Lemma 4 in (39) and obtain
Revised: With these definitions, we apply Lemma 4 in (39) and obtain

Original: [Equation]
Revised: [Equation]

Original: where β is an upper bound of the Lipschitz constant of the cross entropy loss function
Revised: where β is an upper bound of the Lipschitz constant of the cross entropy loss function

Original: CE ( x ; y ) =
Revised: CE ( x ; y ) =

Original: − (cid:80)
Revised: − (cid:80)

Original: Kk
Revised: Kk

Original: =

Revised: =

Original: I
Revised: I

Original: [ y = k ] log x ( k ) when x ∈ ∆ K with x ( k ) > (1 /β ) .
Revised: [ y = k ] log x ( k ) when x ∈ ∆ K with x ( k ) > (1 /β ) .

Original: ∀ k .
Revised: ∀ k .

Original: Next, we will characterize R S
Revised: Next, we will characterize R S

Original: ( Z ) using the covering number of the set Z .
Revised: ( Z ) using the covering number of the set Z .

Original: Definition 2 (Vershynin, 2012)
Revised: Definition 2 (Vershynin, 2012)

Original: The ϵ -net covering of the set Z (denoted as Z ) is a finite subset of Z (i.e., Z ⊆ Z ) such that for any z ∈ Z , there exists an z ∈ Z satisfying ∥ z − z ∥ 22 ≤ ϵ .
Revised: The ϵ -net covering of the set Z (denoted as Z ) is a finite subset of Z (i.e., Z ⊆ Z ) such that for any z ∈ Z , there exists an z ∈ Z satisfying ∥ z − z ∥ 22 ≤ ϵ .

Original: The smallest cardinality of the ϵ -nets of Z is known as the covering number of Z , which is denoted as N ( ϵ, Z ) .
Revised: The smallest cardinality of the ϵ -nets of Z is known as the covering number of Z , which is denoted as N ( ϵ, Z ) .

Original: Let us consider a pair of vectors z , z ∈ Z as below:
Revised: Let us consider a pair of vectors z , z ∈ Z as below:

Original: [Equation]
Revised: [Equation]

Original: where all A m , A m , F , and F satisfy the nonnegativity constraints and have unit ℓ 1 -norm on the columns.
Revised: where all A m , A m , F , and F satisfy the nonnegativity constraints and have unit ℓ 1 -norm on the columns.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses the fact the Frobenius norm of W is bounded by √ MK and the
Revised: where the last inequality uses the fact the Frobenius norm of W is bounded by √ MK and the

Original: Frobenius norm of F is bounded by √ N .
Revised: Frobenius norm of F is bounded by √ N .

Original: Hence, to obtain an ε -net covering for the set Z
Revised: Hence, to obtain an ε -net covering for the set Z

Original: (i.e., ∥ z − z ∥ ≤ ε ), we only need to show that there exists a ε 2 √ MK -net covering for F and a ε 2 √ N -net covering for each W since
Revised: (i.e., ∥ z − z ∥ ≤ ε ), we only need to show that there exists a ε 2 √ MK -net covering for F and a ε 2 √ N -net covering for each W since

Original: [Equation]
Revised: [Equation]

Original: Note that the full rank matrix W ∈ R MK ×
Revised: Note that the full rank matrix W ∈ R MK ×

Original: K can be represented as an MK 2 -dimensional vector whose Euclidean norm is bounded by √ MK .
Revised: K can be represented as an MK 2 -dimensional vector whose Euclidean norm is bounded by √ MK .

Original: Hence, the cardinality of the ε 2 √ N -net covering for W ∈ R
Revised: Hence, the cardinality of the ε 2 √ N -net covering for W ∈ R

Original: MK × K is at most (cid:16) 4 MK √ NK
Revised: MK × K is at most (cid:16) 4 MK √ NK

Original: ε (cid:17) MK 2 (Shalev-Shwartz & Ben-David, 2014).
Revised: ε (cid:17) MK 2 (Shalev-Shwartz & Ben-David, 2014).

Original: We have
Revised: We have

Original: F (: , n ) = f
Revised: F (: , n ) = f

Original: ( x n ) , where f belongs to a function class F .
Revised: ( x n ) , where f belongs to a function class F .

Original: Hence, we use the covering number corresponding to the neural network function class F .
Revised: Hence, we use the covering number corresponding to the neural network function class F .

Original: Using Lemma 14 of (Lin & Zhang, 2019), we get the cardinality of the ε 2 √ MK -net covering for F =
Revised: Using Lemma 14 of (Lin & Zhang, 2019), we get the cardinality of the ε 2 √ MK -net covering for F =

Original: [ f ( x 1 ) , . .
Revised: [ f ( x 1 ) , . .

Original: f ( x N )] , f ∈ F as below:
Revised: f ( x N )] , f ∈ F as below:

Original: [Equation]
Revised: [Equation]

Original: where X =
Revised: where X =

Original: [ x 1 , . .
Revised: [ x 1 , . .

Original: , x N ] ∈ R d × N and the parameter R F is from Assumption 2.
Revised: , x N ] ∈ R d × N and the parameter R F is from Assumption 2.

Original: Using the covering number results, the cardinality of the ε -net covering of set Z is bounded by the following:
Revised: Using the covering number results, the cardinality of the ε -net covering of set Z is bounded by the following:

Original: [Equation]
Revised: [Equation]

Original: Now that we have characterized N ( ϵ, Z ) , we invoke the below lemma to obtain the Rademacher complexity R S ( Z ) :
Revised: Now that we have characterized N ( ϵ, Z ) , we invoke the below lemma to obtain the Rademacher complexity R S ( Z ) :

Original: Lemma 5 (Bartlett et al., 2017, Lemma A.5)
Revised: Lemma 5 (Bartlett et al., 2017, Lemma A.5)

Original: The empirical Rademacher complexity of the set Z with respect to the observed set S having size SK is upper bounded as follows:
Revised: The empirical Rademacher complexity of the set Z with respect to the observed set S having size SK is upper bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: We apply (41) in Lemma 5. and obtain
Revised: We apply (41) in Lemma 5. and obtain

Original: [Equation]
Revised: [Equation]

Original: In the above
Revised: In the above

Original: [Equation]
Revised: [Equation]

Original: the first inequality ( a ) is obtained by using the relation
Revised: the first inequality ( a ) is obtained by using the relation

Original: [Equation]
Revised: [Equation]

Original: which holds because (cid:113) log N ( µ, Z ) decreases monotonically as µ increases.
Revised: which holds because (cid:113) log N ( µ, Z ) decreases monotonically as µ increases.

Original: The inequality ( b ) is obtained by applying (41), and ( d ) is obtained by fixing a = √ MK which is smaller than √ SK .
Revised: The inequality ( b ) is obtained by applying (41), and ( d ) is obtained by fixing a = √ MK which is smaller than √ SK .

Original: Combining the upperbound of R S ( Z ) given by (43) with the upper bound of R S ( ℓ ◦ P ◦ S ) as given by (40) with the result in (38)
Revised: Combining the upperbound of R S ( Z ) given by (43) with the upper bound of R S ( ℓ ◦ P ◦ S ) as given by (40) with the result in (38)

Original: [Equation]
Revised: [Equation]

Original: we get that with probability greater than 1 − δ ,
Revised: we get that with probability greater than 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: where R S ( Z ) is upper bounded by (43) and z max is the upperbound of the value of the function
Revised: where R S ( Z ) is upper bounded by (43) and z max is the upperbound of the value of the function

Original: CE ( x , y ) which can be characterized as below:
Revised: CE ( x , y ) which can be characterized as below:

Original: [Equation]
Revised: [Equation]

Original: Upper-bounding the second term on the R.H.S of (36) .
Revised: Upper-bounding the second term on the R.H.S of (36) .

Original: Next, we proceed to upper bound the second term on the R.H.S of (36).
Revised: Next, we proceed to upper bound the second term on the R.H.S of (36).

Original: Let us consider the Hoeffding’s inequality
Revised: Let us consider the Hoeffding’s inequality

Original: Lemma 6 Let Z 1 , . . .
Revised: Lemma 6 Let Z 1 , . . .

Original: , Z S be independent bounded random variables with Z s ∈ [ z min , z max ] for all s where −∞ < z min ≤ z max < ∞ .
Revised: , Z S be independent bounded random variables with Z s ∈ [ z min , z max ] for all s where −∞ < z min ≤ z max < ∞ .

Original: Then for all t ≥ 0 ,
Revised: Then for all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: To use Lemma 6, let us define the random variable Z ( m ) n as follows:
Revised: To use Lemma 6, let us define the random variable Z ( m ) n as follows:

Original: [Equation]
Revised: [Equation]

Original: where p ♮ ( m ) n =
Revised: where p ♮ ( m ) n =

Original: A ♮m f ♮ ( x n ) .
Revised: A ♮m f ♮ ( x n ) .

Original: The maximum and minimum values of Z ( m ) n are z max = log( β ) (see (45)) and z min = 0 , respectively.
Revised: The maximum and minimum values of Z ( m ) n are z max = log( β ) (see (45)) and z min = 0 , respectively.

Original: Then, invoking Lemma 6, one can obtain
Revised: Then, invoking Lemma 6, one can obtain

Original: [Equation]
Revised: [Equation]

Original: Hence, by substituting t = log ( β ) (cid:113) log ( 1 δ ) 2 S , where δ ∈ (0 , 1) in (46), we get that with probability greater than 1 − δ
Revised: Hence, by substituting t = log ( β ) (cid:113) log ( 1 δ ) 2 S , where δ ∈ (0 , 1) in (46), we get that with probability greater than 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Upper-bounding the third term on
Revised: Upper-bounding the third term on

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: where the first inequality uses the triangle inequality, the second inequality uses the concavity of the log function, the third inequality is via Cauchy Schwartz inequality, and the last inequality employs
Revised: where the first inequality uses the triangle inequality, the second inequality uses the concavity of the log function, the third inequality is via Cauchy Schwartz inequality, and the last inequality employs

Original: Assumption 3.
Revised: Assumption 3.

Original: Putting Together.
Revised: Putting Together.

Original: Hence, by combining the result in (44) with (37), (36), (47), and (48), we get that with probability greater than 1 − 2 δ ,
Revised: Hence, by combining the result in (44) with (37), (36), (47), and (48), we get that with probability greater than 1 − 2 δ ,

Original: [Equation]
Revised: [Equation]

Original: Using Pinsker’s inequality (S, 1960; Fedotov et al., 2003), we get
Revised: Using Pinsker’s inequality (S, 1960; Fedotov et al., 2003), we get

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses the fact that ∥ x ∥ 1 ≥ ∥ x ∥ 2 .
Revised: where the last inequality uses the fact that ∥ x ∥ 1 ≥ ∥ x ∥ 2 .

Original: The above relation combined with (49), implies that with probability greater than 1 − 2 δ :
Revised: The above relation combined with (49), implies that with probability greater than 1 − 2 δ :

Original: [Equation]
Revised: [Equation]

Original: where R S ( Z ) is upper-bounded in (43)
Revised: where R S ( Z ) is upper-bounded in (43)

Original: Next, we aim to bound each term ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 2 2 .
Revised: Next, we aim to bound each term ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 2 2 .

Original: Towards this, we have the below result:
Revised: Towards this, we have the below result:

Original: Lemma 7 Suppose that 1 NM (cid:80) Nn =1 (cid:80)
Revised: Lemma 7 Suppose that 1 NM (cid:80) Nn =1 (cid:80)

Original: Mm =1 ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 is upper-bounded.
Revised: Mm =1 ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 is upper-bounded.

Original: Then, with probability greater than 1 − δ − 1 N 18 , we have for any m, n , ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 ≤ N
Revised: Then, with probability greater than 1 − δ − 1 N 18 , we have for any m, n , ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 ≤ N

Original: [Equation]
Revised: [Equation]

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: The proof of Theorem 2 utilizes some key results from the proof of Theorem 1.
Revised: The proof of Theorem 2 utilizes some key results from the proof of Theorem 1.

Original: We start by employing the result in Proposition 1.
Revised: We start by employing the result in Proposition 1.

Original: It implies that when f ♮ ∈ F , i.e., ν = 0 , S ≥ C 1 t , N ≤ C 2 t 3 , for certain constants C 1 , C 2 > 0 and at the limit of t → ∞ , we get the following:
Revised: It implies that when f ♮ ∈ F , i.e., ν = 0 , S ≥ C 1 t , N ≤ C 2 t 3 , for certain constants C 1 , C 2 > 0 and at the limit of t → ∞ , we get the following:

Original: [Equation]
Revised: [Equation]

Original: with probability 1.
Revised: with probability 1.

Original: Since all the entries of the matrices P ♮ and (cid:98) P are nonnegative, it implies that (cid:98) P = P
Revised: Since all the entries of the matrices P ♮ and (cid:98) P are nonnegative, it implies that (cid:98) P = P

Original: ♮ , i.e.,
Revised: ♮ , i.e.,

Original: [Equation]
Revised: [Equation]

Original: On the other hand, the matrix (cid:98) P can be constructed using the estimates of the CCEM criterion (6) via
Revised: On the other hand, the matrix (cid:98) P can be constructed using the estimates of the CCEM criterion (6) via

Original: [Equation]
Revised: [Equation]

Original: Hence, in order to identify W ♮ and F ♮ from the NMF model in (52), we invoke the following result:
Revised: Hence, in order to identify W ♮ and F ♮ from the NMF model in (52), we invoke the following result:

Original: Lemma 8 (Gillis, 2020) Consider the matrix factorization model Z = XY , where X ∈ R
Revised: Lemma 8 (Gillis, 2020) Consider the matrix factorization model Z = XY , where X ∈ R

Original: I × K ,
Revised: I × K ,

Original: Y ∈ R K × J and rank ( X ) =
Revised: Y ∈ R K × J and rank ( X ) =

Original: rank ( Y ) =
Revised: rank ( Y ) =

Original: K .
Revised: K .

Original: If X , Y ≥ 0 and both X and Y satisfy SSC, then any (cid:99) X and (cid:98) Y that satisfy Z = (cid:99)
Revised: If X , Y ≥ 0 and both X and Y satisfy SSC, then any (cid:99) X and (cid:98) Y that satisfy Z = (cid:99)

Original: X (cid:98) Y must have the following form:
Revised: X (cid:98) Y must have the following form:

Original: [Equation]
Revised: [Equation]

Original: where Π is a column permutation matrix and Σ is a diagonal nonnegative and scaling matrix.
Revised: where Π is a column permutation matrix and Σ is a diagonal nonnegative and scaling matrix.

Original: Next, we show that Lemma 8 can be applied given the conditions in Theorem 2.
Revised: Next, we show that Lemma 8 can be applied given the conditions in Theorem 2.

Original: The matrix
Revised: The matrix

Original: F ♮ Z =
Revised: F ♮ Z =

Original: [ f ♮ ( x n 1 ) , . . . , f ♮ ( x n T )] includes a subset of columns of F ♮ , i.e., cone
Revised: [ f ♮ ( x n 1 ) , . . . , f ♮ ( x n T )] includes a subset of columns of F ♮ , i.e., cone

Original: ( F ♮ Z ) ⊆ cone( F ♮ ) .
Revised: ( F ♮ Z ) ⊆ cone( F ♮ ) .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where C = { x ∈ R K |√ K − 

Revised: where C = { x ∈ R K |√ K −

Original: ∥ x ∥ 2 ≤ 1 ⊤ x } is the second-order cone.
Revised: ∥ x ∥ 2 ≤ 1 ⊤ x } is the second-order cone.

Original: Also for any orthogonal matrix Q ∈ R K × K except for the permutation matrices, the below holds:
Revised: Also for any orthogonal matrix Q ∈ R K × K except for the permutation matrices, the below holds:

Original: [Equation]
Revised: [Equation]

Original: Eqs (54) and (55) imply that F ♮ satisfies SSC, given the assumption that F ♮ Z satisfies SSC.
Revised: Eqs (54) and (55) imply that F ♮ satisfies SSC, given the assumption that F ♮ Z satisfies SSC.

Original: In addition, the column scaling does not affect the conic hull, i.e,
Revised: In addition, the column scaling does not affect the conic hull, i.e,

Original: [Equation]
Revised: [Equation]

Original: Since both W ♮ and F ♮ satisfy SSC, rank ( F ♮ )
Revised: Since both W ♮ and F ♮ satisfy SSC, rank ( F ♮ )

Original: =
Revised: =

Original: rank ( W ♮ )
Revised: rank ( W ♮ )

Original: =
Revised: =

Original: K hold (see (Gillis, 2020)).
Revised: K hold (see (Gillis, 2020)).

Original: Hence, the conditions in Lemma 8 holds for (52).
Revised: Hence, the conditions in Lemma 8 holds for (52).

Original: Comparing (52) and (53) and invoking Lemma 8, we get
Revised: Comparing (52) and (53) and invoking Lemma 8, we get

Original: [Equation]
Revised: [Equation]

Original: where the scaling Σ is automatically removed due to the sum-to-one constraints on the columns of (cid:98) A m ’s and (cid:98) F .
Revised: where the scaling Σ is automatically removed due to the sum-to-one constraints on the columns of (cid:98) A m ’s and (cid:98) F .

Original: The first result (56a) implies that
Revised: The first result (56a) implies that

Original: [Equation]
Revised: [Equation]

Original: Applying the second result (56b) in Lemma 3 along with the assumption that N → ∞ , we get
Revised: Applying the second result (56b) in Lemma 3 along with the assumption that N → ∞ , we get

Original: [Equation]
Revised: [Equation]

Original: Fact 1 Let X be a nonnegative random variable with E [ X ] = 0 .
Revised: Fact 1 Let X be a nonnegative random variable with E [ X ] = 0 .

Original: Then, X is zero almost surely, i.e.,
Revised: Then, X is zero almost surely, i.e.,

Original: Pr ( X = 0) = 1 .
Revised: Pr ( X = 0) = 1 .

Original: Employing Fact 1 in (57), we get that
Revised: Employing Fact 1 in (57), we get that

Original: [Equation]
Revised: [Equation]

Original: f ♮ ( x ) , ∀ x ∼ D , due to the nonnegativity of f ♮ and (cid:98) f .
Revised: f ♮ ( x ) , ∀ x ∼ D , due to the nonnegativity of f ♮ and (cid:98) f .

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: To prove Theorem 3, let us first consider the CCEM criterion in (6).
Revised: To prove Theorem 3, let us first consider the CCEM criterion in (6).

Original: Let (cid:99) W and (cid:98) F be any optimal solution of (6) and
Revised: Let (cid:99) W and (cid:98) F be any optimal solution of (6) and

Original: [Equation]
Revised: [Equation]

Original: From Proposition 1, using the conditions in the statement of Theorem 3, i.e., f ♮ ∈ F implying ν = 0 ,
Revised: From Proposition 1, using the conditions in the statement of Theorem 3, i.e., f ♮ ∈ F implying ν = 0 ,

Original: S ≥ C 1 t , N ≤ C 2 t 3 , for certain constants C 1 , C 2 > 0 , we get the following at the limit of t → ∞ :
Revised: S ≥ C 1 t , N ≤ C 2 t 3 , for certain constants C 1 , C 2 > 0 , we get the following at the limit of t → ∞ :

Original: [Equation]
Revised: [Equation]

Original: with probability 1.
Revised: with probability 1.

Original: Since all the entries of the matrices P ♮ and (cid:98) P are nonnegative, it implies that (cid:98) P = P
Revised: Since all the entries of the matrices P ♮ and (cid:98) P are nonnegative, it implies that (cid:98) P = P

Original: ♮ , i.e.,
Revised: ♮ , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 3(a):
Revised: Proof of Theorem 3(a):

Original: We consider the following result which is distilled and summarized from the proof of Theorem 1 in (Fu et al., 2015):
Revised: We consider the following result which is distilled and summarized from the proof of Theorem 1 in (Fu et al., 2015):

Original: Lemma 9 Suppose a matrix Y ∈ R K
Revised: Lemma 9 Suppose a matrix Y ∈ R K

Original: × J satisfies Y ≥ 0 , 1 ⊤ Y = 1 ⊤ , rank ( Y ) =
Revised: × J satisfies Y ≥ 0 , 1 ⊤ Y = 1 ⊤ , rank ( Y ) =

Original: K , and SSC.
Revised: K , and SSC.

Original: Then, for any (cid:98) Y
Revised: Then, for any (cid:98) Y

Original: =
Revised: =

Original: QY satisfying (cid:98) Y ≥ 0 , 1 ⊤ (cid:98) Y = 1 ⊤ , the following holds:
Revised: QY satisfying (cid:98) Y ≥ 0 , 1 ⊤ (cid:98) Y = 1 ⊤ , the following holds:

Original: [Equation]
Revised: [Equation]

Original: The equality holds only if Q is a permutation matrix.
Revised: The equality holds only if Q is a permutation matrix.

Original: Let us start by considering the following criterion:
Revised: Let us start by considering the following criterion:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) P is the optimal solution from the CCEM criterion (6) and hence, as shown before, (cid:98) P satisfies (59).
Revised: where (cid:98) P is the optimal solution from the CCEM criterion (6) and hence, as shown before, (cid:98) P satisfies (59).

Original: Let W ∗ and F ∗ be optimal solutions of (60), then the following holds:
Revised: Let W ∗ and F ∗ be optimal solutions of (60), then the following holds:

Original: [Equation]
Revised: [Equation]

Original: From (59), we observe that W ∗ and F ∗ satisfies W ∗ = W ♮ Q − 1 , F ∗ =
Revised: From (59), we observe that W ∗ and F ∗ satisfies W ∗ = W ♮ Q − 1 , F ∗ =

Original: QF ♮ , for a certain invertible matrix Q .
Revised: QF ♮ , for a certain invertible matrix Q .

Original: Let us assume that Q is not a permutation matrix, Then, we have
Revised: Let us assume that Q is not a permutation matrix, Then, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is from Lemma 9 using the SSC condition on F ♮ .
Revised: where the last inequality is from Lemma 9 using the SSC condition on F ♮ .

Original: Note that the result is a contradiction from (61).
Revised: Note that the result is a contradiction from (61).

Original: Hence, Q must be a permutation matrix.
Revised: Hence, Q must be a permutation matrix.

Original: This implies that the optimal solution W ∗ and F ∗ of Problem 60 satisfies the following:
Revised: This implies that the optimal solution W ∗ and F ∗ of Problem 60 satisfies the following:

Original: [Equation]
Revised: [Equation]

Original: Following the last part of Theorem 2, by using Lemma 3 and Fact 1, we have
Revised: Following the last part of Theorem 2, by using Lemma 3 and Fact 1, we have

Original: [Equation]
Revised: [Equation]

Original: Note that since log is a monotonically increasing function, by using log det( F F ⊤ ) in (60) in place of det( F F ) ⊤ does not change the optimal solution, yet it keeps the objective function in differentiable domain (this is because, det( X ) becomes zero if X is singular).
Revised: Note that since log is a monotonically increasing function, by using log det( F F ⊤ ) in (60) in place of det( F F ) ⊤ does not change the optimal solution, yet it keeps the objective function in differentiable domain (this is because, det( X ) becomes zero if X is singular).

Original: Hence, we can conclude that the following criterion
Revised: Hence, we can conclude that the following criterion

Original: [Equation]
Revised: [Equation]

Original: also results in the same optimal solutions W ∗ and F ∗ satisfying (62).
Revised: also results in the same optimal solutions W ∗ and F ∗ satisfying (62).

Original: Proof of Theorem 3(b):
Revised: Proof of Theorem 3(b):

Original: We consider the following result:
Revised: We consider the following result:

Original: Lemma 10 (Huang et al., 2015) Suppose a matrix X ∈ R
Revised: Lemma 10 (Huang et al., 2015) Suppose a matrix X ∈ R

Original: I × K satisfies X ≥ 0 , 1 ⊤ X = ρ 1 ⊤ , ρ > 0 , rank ( X ) =
Revised: I × K satisfies X ≥ 0 , 1 ⊤ X = ρ 1 ⊤ , ρ > 0 , rank ( X ) =

Original: K , and SSC.
Revised: K , and SSC.

Original: Then, for any (cid:99) X = XQ satisfying (cid:99) X ≥ 0 , 1 ⊤ (cid:99) X
Revised: Then, for any (cid:99) X = XQ satisfying (cid:99) X ≥ 0 , 1 ⊤ (cid:99) X

Original: = ρ 1 ⊤ , the following holds:
Revised: = ρ 1 ⊤ , the following holds:

Original: [Equation]
Revised: [Equation]

Original: The equality holds only if Q is a permutation matrix.
Revised: The equality holds only if Q is a permutation matrix.

Original: Note that, in Lemma 10, we have 1 ⊤ X = ρ 1 ⊤ .
Revised: Note that, in Lemma 10, we have 1 ⊤ X = ρ 1 ⊤ .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Since 1 ⊤ XQ = ρ 1 ⊤ holds by the assumption, we get
Revised: Since 1 ⊤ XQ = ρ 1 ⊤ holds by the assumption, we get

Original: [Equation]
Revised: [Equation]

Original: The result in (64) is used in order to conclude Lemma 10 from the results in (Huang et al., 2015).
Revised: The result in (64) is used in order to conclude Lemma 10 from the results in (Huang et al., 2015).

Original: We consider the following criterion:
Revised: We consider the following criterion:

Original: [Equation]
Revised: [Equation]

Original: where we used the constraint 1 ⊤ W = M 1 ⊤ since 1 ⊤ A m = 1 ⊤ for all m .
Revised: where we used the constraint 1 ⊤ W = M 1 ⊤ since 1 ⊤ A m = 1 ⊤ for all m .

Original: Let W ∗ and F ∗ be optimal solutions of (65), then we have:
Revised: Let W ∗ and F ∗ be optimal solutions of (65), then we have:

Original: [Equation]
Revised: [Equation]

Original: Applying (59), W ∗ and F ∗ satisfies W ∗ = W ♮ Q , F ∗
Revised: Applying (59), W ∗ and F ∗ satisfies W ∗ = W ♮ Q , F ∗

Original: [Equation]
Revised: [Equation]

Original: − 1 F ♮ , for a certain invertible matrix Q .
Revised: − 1 F ♮ , for a certain invertible matrix Q .

Original: Let us assume that Q is not a permutation matrix, Then, we have
Revised: Let us assume that Q is not a permutation matrix, Then, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is from Lemma 10 using the SSC condition on W ♮ which is a contradiction from (66).
Revised: where the last inequality is from Lemma 10 using the SSC condition on W ♮ which is a contradiction from (66).

Original: Hence, Q must be a permutation matrix.
Revised: Hence, Q must be a permutation matrix.

Original: This implies that the optimal solution W ∗ and
Revised: This implies that the optimal solution W ∗ and

Original: F ∗ of Problem 65 satisfies the following:
Revised: F ∗ of Problem 65 satisfies the following:

Original: [Equation]
Revised: [Equation]

Original: Similar to the last part of Theorem 2, by using Lemma 3 and Fact 1, we have
Revised: Similar to the last part of Theorem 2, by using Lemma 3 and Fact 1, we have

Original: [Equation]
Revised: [Equation]

Original: In this case as well, the optimal solutions do not change if we employ log det( W ⊤ W ) in (65) in place of det( W ⊤ W ) .
Revised: In this case as well, the optimal solutions do not change if we employ log det( W ⊤ W ) in (65) in place of det( W ⊤ W ) .

Original: Remark 3 Geometrically, Theorem 3 seeks maximum volume solutions w.r.t.
Revised: Remark 2 Geometrically, Theorem 3 seeks maximum volume solutions w.r.t.

Original: the conic hulls of F and W ⊤ , respectively.
Revised: the conic hulls of F and W ⊤ , respectively.

Original: One can also note that in both cases of Theorem 3, the corresponding optimal solutions do not change if we minimize log det( W ⊤ W ) in (60) and minimize log det( F F ⊤ ) in (65).
Revised: One can also note that in both cases of Theorem 3, the corresponding optimal solutions do not change if we minimize log det( W ⊤ W ) in (60) and minimize log det( F F ⊤ ) in (65).

Original: However, relying on the SSC of F ♮ ( W ♮ ) and minimizing the volume of conic hull of W ⊤ ( F ) may be inefficient since P ♮ =
Revised: However, relying on the SSC of F ♮ ( W ♮ ) and minimizing the volume of conic hull of W ⊤ ( F ) may be inefficient since P ♮ =

Original: W ♮ F ♮ does not hold exactly in practice.
Revised: W ♮ F ♮ does not hold exactly in practice.

Original: F P ROOF OF L EMMA
Revised: F P ROOF OF L EMMA

Original: Let us define some notations:
Revised: Let us define some notations:

Original: [Equation]
Revised: [Equation]

Original: Consider the following conditions given in Lemma
Revised: Consider the following conditions given in Lemma

Original: [Equation]
Revised: [Equation]

Original: 68b
Revised: 68b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We begin by noticing the below relation from (68a):
Revised: We begin by noticing the below relation from (68a):

Original: [Equation]
Revised: [Equation]

Original: where the first equality employs the probability simplex constraints on the columns of (cid:98) F .
Revised: where the first equality employs the probability simplex constraints on the columns of (cid:98) F .

Original: We further proceed to prove by using contradiction.
Revised: We further proceed to prove by using contradiction.

Original: First, assume the below for certain k ̸
Revised: First, assume the below for certain k ̸

Original: = j ,
Revised: = j ,

Original: [Equation]
Revised: [Equation]

Original: Under the assumption (70)
Revised: Under the assumption (70)

Original: [Equation]
Revised: [Equation]

Original: consider (68b).
Revised: consider (68b).

Original: For (68b) to hold
Revised: For (68b) to hold

Original: [Equation]
Revised: [Equation]

Original: we need to satisfy the below for a certain k ̸
Revised: we need to satisfy the below for a certain k ̸

Original: =
Revised: =

Original: j :
Revised: j :

Original: [Equation]
Revised: [Equation]

Original: where the last relation is obtained from (69).
Revised: where the last relation is obtained from (69).

Original: This also implies that
Revised: This also implies that

Original: [Equation]
Revised: [Equation]

Original: However, Lemma 1 assumes that κ ≤ 1 K +1 , Hence, the assumption (70) does not hold and we get
Revised: However, Lemma 1 assumes that κ ≤ 1 K +1 , Hence, the assumption (70) does not hold and we get

Original: [Equation]
Revised: [Equation]

Original: F. P ROVING ℓ ∗ k = q ∗ k , ∀ k
Revised: F. P ROVING ℓ ∗ k = q ∗ k , ∀ k

Original: From the result in (71), we consider (68b) for a certain k ̸
Revised: From the result in (71), we consider (68b) for a certain k ̸

Original: = j .
Revised: = j .

Original: [Equation]
Revised: [Equation]

Original: where the last relation is obtained from (69).
Revised: where the last relation is obtained from (69).

Original: Since (cid:80) q (cid:98) F 1 ( q, j ) = 1 , we get
Revised: Since (cid:80) q (cid:98) F 1 ( q, j ) = 1 , we get

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From the result in (71) and the condition in (68b), we also have the following for a certain k ̸ =
Revised: From the result in (71) and the condition in (68b), we also have the following for a certain k ̸ =

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: where the last relation by (73).
Revised: where the last relation by (73).

Original: Next, we proceed to prove by contradiction.
Revised: Next, we proceed to prove by contradiction.

Original: First, assume the below for certain k
Revised: First, assume the below for certain k

Original: [Equation]
Revised: [Equation]

Original: Hence, for each k , we have
Revised: Hence, for each k , we have

Original: [Equation]
Revised: [Equation]

Original: where we have used the fact that all the entries of (cid:98) A 1 and (cid:98) F 1 are smaller than 1, in the first inequality and have applied (72) and (74) in the last inequality.
Revised: where we have used the fact that all the entries of (cid:98) A 1 and (cid:98) F 1 are smaller than 1, in the first inequality and have applied (72) and (74) in the last inequality.

Original: The lower-bound condition in (68a) gives that for each k ,
Revised: The lower-bound condition in (68a) gives that for each k ,

Original: [Equation]
Revised: [Equation]

Original: Then, comparing both (76) and (77), we hope to have
Revised: Then, comparing both (76) and (77), we hope to have

Original: [Equation]
Revised: [Equation]

Original: However, K +1 − √ ( K +1) 2 − 4≥ 1 K +1 , which leads to a contradiction to our assumption that κ ≤ 1 K
Revised: However, K +1 − √ ( K +1) 2 − 4≥ 1 K +1 , which leads to a contradiction to our assumption that κ ≤ 1 K

Original: +1 .
Revised: +1 .

Original: Hence, the assumption (75) does not hold and we get
Revised: Hence, the assumption (75) does not hold and we get

Original: [Equation]
Revised: [Equation]

Original: Combining both (78) and (78), we get
Revised: Combining both (78) and (78), we get

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: G P ROOF OF L EMMA
Revised: G P ROOF OF L EMMA

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the inequality ( a ) employs the triangle inequality, ( b ) is obtained via the matrix norm equivalence relation ∥ E 1 + E 2 ∥ 2 ≤ ∥ E 1 + E 2 ∥ F ≤ ∥ E 1 ∥ F + ∥ E 2 ∥ F , ( c ) is obtained by the assumption
Revised: where the inequality ( a ) employs the triangle inequality, ( b ) is obtained via the matrix norm equivalence relation ∥ E 1 + E 2 ∥ 2 ≤ ∥ E 1 + E 2 ∥ F ≤ ∥ E 1 ∥ F + ∥ E 2 ∥ F , ( c ) is obtained by the assumption

Original: that ∥ E 1 ∥ F ≤ υ 1 and ∥ E 2 ∥ F ≤ υ 2 .
Revised: that ∥ E 1 ∥ F ≤ υ 1 and ∥ E 2 ∥ F ≤ υ 2 .

Original: H P ROOF OF L EMMA
Revised: H P ROOF OF L EMMA

Original: First, let us define the following w.r.t the loss function g ( f , y ) =
Revised: First, let us define the following w.r.t the loss function g ( f , y ) =

Original: ∥
Revised: ∥

Original: f − y ∥ 22 , ∀ f ∈ F , y ∈ R K and the input data X
Revised: f − y ∥ 22 , ∀ f ∈ F , y ∈ R K and the input data X

Original: = ( x 1 , . . .
Revised: = ( x 1 , . . .

Original: , x N ) where each x n is sampled i.i.d.
Revised: , x N ) where each x n is sampled i.i.d.

Original: from the distribution D under
Revised: from the distribution D under

Original: Assumption 1:
Revised: Assumption 1:

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Revised: f

Revised: [Equation]

Original:  We invoke Theorem 26.5 in (Shalev-Shwartz & Ben-David, 2014) and get that with probability greater than 1 − δ :
Revised: f [Equation] [Equation] We invoke Theorem 26.5 in (Shalev-Shwartz & Ben-David, 2014) and get that with probability greater than 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: where the last inequality utilizes the definitions in (80), the contraction lemma (Lemma 26.9 from (Shalev-Shwartz & Ben-David, 2014)) and also applied ¯ c = 4 since | g ( f , y ) | ≤ 4 in our case.
Revised: where the last inequality utilizes the definitions in (80), the contraction lemma (Lemma 26.9 from (Shalev-Shwartz & Ben-David, 2014)) and also applied ¯ c = 4 since | g ( f , y ) | ≤ 4 in our case.

Original: The term R N ( F ) denotes the empirical Rademacher complexity of the neural network function class F which is upperbounded via the sensitive complexity parameter R F as follows (Lin & Zhang, 2019):
Revised: The term R N ( F ) denotes the empirical Rademacher complexity of the neural network function class F which is upperbounded via the sensitive complexity parameter R F as follows (Lin & Zhang, 2019):

Original: [Equation]
Revised: [Equation]

Original: I P ROOF OF L EMMA
Revised: I P ROOF OF L EMMA

Original: Our goal is to bound ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 for each m, n .
Revised: Our goal is to bound ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 for each m, n .

Original: To achieve this, we consider the below lemma:
Revised: To achieve this, we consider the below lemma:

Original: Lemma 11 (McDiarmid’s inequality) Let U be some set and let v : U N → R be a function of N variables such that for some c > 0 , for all i ∈ [ N ] and for all u 1 , . . .
Revised: Lemma 11 (McDiarmid’s inequality) Let U be some set and let v : U N → R be a function of N variables such that for some c > 0 , for all i ∈ [ N ] and for all u 1 , . . .

Original: , u N , u ′ i ∈
Revised: , u N , u ′ i ∈

Original: U we have
Revised: U we have

Original: [Equation]
Revised: [Equation]

Original: Let U 1 , . .
Revised: Let U 1 , . .

Original: , U N be N independent random variables taking values in U .
Revised: , U N be N independent random variables taking values in U .

Original: Then, with probability greater than 1 −
Revised: Then, with probability greater than 1 −

Original: [Equation]
Revised: [Equation]

Original: where expectation is taken w.r.t.
Revised: where expectation is taken w.r.t.

Original: U 1 , . .
Revised: U 1 , . .

Original: , U N .
Revised: , U N .

Original: In our settings, let us define u n = { x n ,
Revised: In our settings, let us define u n = { x n ,

Original: { (cid:98) y
Revised: { (cid:98) y

Original: ( m ) n } m ∈S n , S n } where S n denotes the indices of the annotators who labeled the data item x n and let U n be the random variable associated with u n , One can note that U 1 , . . .
Revised: ( m ) n } m ∈S n , S n } where S n denotes the indices of the annotators who labeled the data item x n and let U n be the random variable associated with u n , One can note that U 1 , . . .

Original: , U N are independent in this case.
Revised: , U N are independent in this case.

Original: Now, let us define the following:
Revised: Now, let us define the following:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) P is determined by u 1 , . .
Revised: where (cid:98) P is determined by u 1 , . .

Original: u N .
Revised: u N .

Original: Next, we aim to characterize the constant c in Lemma 11 under the definition in (82).
Revised: Next, we aim to characterize the constant c in Lemma 11 under the definition in (82).

Original: Towards this, we first characterize ∥ (cid:98) P ′ − (cid:98) P ∥
Revised: Towards this, we first characterize ∥ (cid:98) P ′ − (cid:98) P ∥

Original: 2F , where (cid:98) P ′ is the optimal solution obtained by using data u 1 , . . .
Revised: 2F , where (cid:98) P ′ is the optimal solution obtained by using data u 1 , . . .

Original: u i − 1 , u ′ i , u i +1 , . .
Revised: u i − 1 , u ′ i , u i +1 , . .

Original: u N .
Revised: u N .

Original: Lemma 12 Let (cid:98) P is the optimal solution of the CCM criterion 6 using the data u 1 , . . .
Revised: Lemma 12 Let (cid:98) P is the optimal solution of the CCM criterion 6 using the data u 1 , . . .

Original: , u N and let (cid:98) P ′ is the optimal solution of the CCM criterion 6 using the data u 1 , . . .
Revised: , u N and let (cid:98) P ′ is the optimal solution of the CCM criterion 6 using the data u 1 , . . .

Original: u i − 1 , u ′ i , u i +1 , . .
Revised: u i − 1 , u ′ i , u i +1 , . .

Original: u N .
Revised: u N .

Original: Assume that each (cid:98) P and (cid:98) P ′ are at least (1 /β ) , for a certain β > 1 .
Revised: Assume that each (cid:98) P and (cid:98) P ′ are at least (1 /β ) , for a certain β > 1 .

Original: Then, the following holds
Revised: Then, the following holds

Original: [Equation]
Revised: [Equation]

Original: where ¯ c > 0 is a constant
Revised: where ¯ c > 0 is a constant

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where ¯ c 1 > 0 is a constant.
Revised: where ¯ c 1 > 0 is a constant.

Original: where the first inequality is by triangle inequality, the second inequality is by using Lemma 12 and by the relation ∥ P 1 − P 

Revised: where the first inequality is by triangle inequality, the second inequality is by using Lemma 12 and by the relation ∥ P 1 − P

Original: ∥ F ≤ √ 2 NM, for any P 1 , P 2 ∈ P .
Revised: ∥ F ≤ √ 2 NM, for any P 1 , P 2 ∈ P .

Original: Hence, using the result in (83) and Lemma
Revised: Hence, using the result in (83) and Lemma

Original: [Equation]
Revised: [Equation]

Original: we get the following with probability greater than
Revised: we get the following with probability greater than

Original: [Equation]
Revised: [Equation]

Original: To proceed
Revised: To proceed

Original: [Equation]
Revised: [Equation]

Original: we have the following lemma:
Revised: we have the following lemma:

Original: Lemma 13 Let v n ( u 1 , . . .
Revised: Lemma 13 Let v n ( u 1 , . . .

Original: , u N ) ≜ (cid:80)
Revised: , u N ) ≜ (cid:80)

Original: Mm =1 ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 .
Revised: Mm =1 ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 .

Original: Then
Revised: Then

Original: E [ v n ( u 1 , . .
Revised: E [ v n ( u 1 , . .

Original: . , u N )]
Revised: . , u N )]

Original: =
Revised: =

Original: E [ v n ′ ( u 1 , . .
Revised: E [ v n ′ ( u 1 , . .

Original: . , u N )] , ∀ , n, n ′ .
Revised: . , u N )] , ∀ , n, n ′ .

Original: The proof is provided in Sec.
Revised: The proof is provided in Sec.

Original: K.
Revised: K.

Original: From Lemma 13, we get
Revised: From Lemma 13, we get

Original: E [ v ( u 1 , . . .
Revised: E [ v ( u 1 , . . .

Original: , u N )]
Revised: , u N )]

Original: =
Revised: =

Original: E [ v n ( u 1 , . .
Revised: E [ v n ( u 1 , . .

Original: . , u N )] , ∀ n.
Revised: . , u N )] , ∀ n.

Original: Hence, combining (84) with Lemma 13, we get the following with probability 1 − δ ,
Revised: Hence, combining (84) with Lemma 13, we get the following with probability 1 − δ ,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Applying Markov’s inequality,
Revised: Applying Markov’s inequality,

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Let t = N 18 E (cid:104)(cid:80) Mm =
Revised: Let t = N 18 E (cid:104)(cid:80) Mm =

Original: 1 ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 (cid:105) , then with probability greater than 1 − 1 N 18 , we have
Revised: 1 ∥ p ♮ ( m ) n − (cid:98) p ( m ) n ∥ 22 (cid:105) , then with probability greater than 1 − 1 N 18 , we have

Original: M (cid:88) m =
Revised: M (cid:88) m =

Original: [Equation]
Revised: [Equation]

Original: Combining (85) and (86), we get that with probability greater than 1 − 1 N 18 − δ , for any n ,
Revised: Combining (85) and (86), we get that with probability greater than 1 − 1 N 18 − δ , for any n ,

Original: M (cid:88)
Revised: M (cid:88)

Original: [Equation]
Revised: [Equation]

Original: J P ROOF OF L EMMA
Revised: J P ROOF OF L EMMA

Original: To prove Lemma 12, we follow the stability analysis presented in Chapter 13 of (Shalev-Shwartz & Ben-David, 2014).
Revised: To prove Lemma 12, we follow the stability analysis presented in Chapter 13 of (Shalev-Shwartz & Ben-David, 2014).

Original: Let u n = { x n , (cid:98) y
Revised: Let u n = { x n , (cid:98) y

Original: ( m ) n , m ∈ S n } and let T collects all u n ’s which is used for estimating (cid:98) P , i.e.,
Revised: ( m ) n , m ∈ S n } and let T collects all u n ’s which is used for estimating (cid:98) P , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Let us also define the set T ′ as the following:
Revised: Let us also define the set T ′ as the following:

Original: [Equation]
Revised: [Equation]

Original: where u ′ ̸
Revised: where u ′ ̸

Original: = u i hold.
Revised: = u i hold.

Original: The goal is to characterize the change in the optimal solutions of the CCEM criterion in (6) when T ′ is given as input in place of T .
Revised: The goal is to characterize the change in the optimal solutions of the CCEM criterion in (6) when T ′ is given as input in place of T .

Original: Let us concisely denote the objective function in (6) as below:
Revised: Let us concisely denote the objective function in (6) as below:

Original: [Equation]
Revised: [Equation]

Original: where ℓ ( u n ; P ) =
Revised: where ℓ ( u n ; P ) =

Original: (cid:80) m ∈S n CE ( p ( m ) n , (cid:98)
Revised: (cid:80) m ∈S n CE ( p ( m ) n , (cid:98)

Original: y ( m ) n ) .
Revised: y ( m ) n ) .

Original: We have the corresponding optimal solution defined as follows:
Revised: We have the corresponding optimal solution defined as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, for any P 1 , P 2 ∈ P ,
Revised: Then, for any P 1 , P 2 ∈ P ,

Original: [Equation]
Revised: [Equation]

Original: The above relation applies for (cid:98) P and (cid:98) P ′ as well and hence, we have
Revised: The above relation applies for (cid:98) P and (cid:98) P ′ as well and hence, we have

Original: [Equation]
Revised: [Equation]

Original: since L T ′ ( (cid:98) P ′ )
Revised: since L T ′ ( (cid:98) P ′ )

Original: − L T ( (cid:98) P ′ ) ≤ 0 as (cid:98) P ′ is the minimizer of L T ′ ( P ) .
Revised: − L T ( (cid:98) P ′ ) ≤ 0 as (cid:98) P ′ is the minimizer of L T ′ ( P ) .

Original: We use the fact that the function L T ( P ) is strongly convex w.r.t.
Revised: We use the fact that the function L T ( P ) is strongly convex w.r.t.

Original: P , with a parameter µ > 0 since each entry of the P matrix is at least 1 /β , where β > 1 .
Revised: P , with a parameter µ > 0 since each entry of the P matrix is at least 1 /β , where β > 1 .

Original: Since (cid:98) P is the minimizer of L T ( P ) and using the strong convexity of the function L T ( P ) , we have
Revised: Since (cid:98) P is the minimizer of L T ( P ) and using the strong convexity of the function L T ( P ) , we have

Original: [Equation]
Revised: [Equation]

Original: Hence, we get
Revised: Hence, we get

Original: [Equation]
Revised: [Equation]

Original: , where µ > 0 .
Revised: , where µ > 0 .

Original: K P ROOF OF L EMMA
Revised: K P ROOF OF L EMMA

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where u n = { x n , { (cid:98) y ( m ) n } m ∈S n , S n } and (cid:98) f is the estimate obtained using the data u 1 , . . .
Revised: where u n = { x n , { (cid:98) y ( m ) n } m ∈S n , S n } and (cid:98) f is the estimate obtained using the data u 1 , . . .

Original: , u n , . .
Revised: , u n , . .

Original: . , u n ′ , . .
Revised: . , u n ′ , . .

Original: u N .
Revised: u N .

Original: Note that f does not depend on the data u 1 , . .
Revised: Note that f does not depend on the data u 1 , . .

Original: . , u n , . . . , u n ′ , . .
Revised: . , u n , . . . , u n ′ , . .

Original: u N .
Revised: u N .

Original: Consider n ′ ̸ = n , then
Revised: Consider n ′ ̸ = n , then

Original: [Equation]
Revised: [Equation]

Original: where u n ′ = { x n ′ , { (cid:98) y ( m ) n ′ } m ∈S n ′ , S n ′ } where (cid:98) f
Revised: where u n ′ = { x n ′ , { (cid:98) y ( m ) n ′ } m ∈S n ′ , S n ′ } where (cid:98) f

Original: ′ is the estimate obtained using u 1 , . . .
Revised: ′ is the estimate obtained using u 1 , . . .

Original: , u n ′ , . .
Revised: , u n ′ , . .

Original: . , u n , . .
Revised: . , u n , . .

Original: , u N and is the same as (cid:98) f since optimal solutions do not depend on the order of the input.
Revised: , u N and is the same as (cid:98) f since optimal solutions do not depend on the order of the input.

Original: Hence, we can use the following notation
Revised: Hence, we can use the following notation

Original: [Equation]
Revised: [Equation]

Original: where g is a certain real valued function and U captures the dependency of the function g w.r.t all u n ’s.
Revised: where g is a certain real valued function and U captures the dependency of the function g w.r.t all u n ’s.

Original: Next, we consider taking expectation w.r.t.
Revised: Next, we consider taking expectation w.r.t.

Original: the random variables U 1 , . .
Revised: the random variables U 1 , . .

Original: , U N :
Revised: , U N :

Original: [Equation]
Revised: [Equation]

Original: where q ( U ) denotes the joint probability of the random variables U 1 , . .
Revised: where q ( U ) denotes the joint probability of the random variables U 1 , . .

Original: , U N .
Revised: , U N .

Original: Since x n and x n ′ are identically distributed, we get
Revised: Since x n and x n ′ are identically distributed, we get

Original: [Equation]
Revised: [Equation]

Original: L M ORE DETAILS ON THE COMPLEXITY MEASURE R F
Revised: L M ORE DETAILS ON THE COMPLEXITY MEASURE R F

Original: The work in (Lin & Zhang, 2019) introduces a complexity measure for measuring the expressiveness of different family of neural network classes.
Revised: The work in (Lin & Zhang, 2019) introduces a complexity measure for measuring the expressiveness of different family of neural network classes.

Original: For example, consider a general convolution neural network (CNN) architecture with L layers such that
Revised: For example, consider a general convolution neural network (CNN) architecture with L layers such that

Original: [Equation]
Revised: [Equation]

Original: where σ i ( · ) =
Revised: where σ i ( · ) =

Original: [ σ i ( · ) , . . .
Revised: [ σ i ( · ) , . . .

Original: , σ i ( · )] ∈ R
Revised: , σ i ( · )] ∈ R

Original: d i represents an element-wise activation function at i th layer satisfying σ i (0)
Revised: d i represents an element-wise activation function at i th layer satisfying σ i (0)

Original: = 

Revised: =

Original: and σ i being ρ -Lipschitz continuous for all i .
Revised: and σ i being ρ -Lipschitz continuous for all i .

Original: Let G F denote the set of layer indices with fully connected layers and G C denote the layer indices with convolutional layers.
Revised: Let G F denote the set of layer indices with fully connected layers and G C denote the layer indices with convolutional layers.

Original: If i ∈ G F , we have γ
Revised: If i ∈ G F , we have γ

Original: i
Revised: i

Original: ( H i )
Revised: ( H i )

Original: =
Revised: =

Original: H i ∈
Revised: H i ∈

Original: R d i × d i − 1 .
Revised: R d i × d i − 1 .

Original: Suppose i ∈ G C , then we have γ i
Revised: Suppose i ∈ G C , then we have γ i

Original: ( H i )
Revised: ( H i )

Original: z
Revised: z

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where the matrix H i ∈
Revised: where the matrix H i ∈

Original: R
Revised: R

Original: c
Revised: c

Original: i × r
Revised: i × r

Original: i contains c i convolutional filters each of which having dimension r i ,
Revised: i contains c i convolutional filters each of which having dimension r i ,

Original: z i − 1 denotes the output of the ( i − 1) th layer, and ⊛ denotes the convolution operation.
Revised: z i − 1 denotes the output of the ( i − 1) th layer, and ⊛ denotes the convolution operation.

Original: Here d 0 =
Revised: Here d 0 =

Original: d is the dimension of the of the input data items x n ’s. Then, its sensitive complexity R F is defined as follows:
Revised: d is the dimension of the of the input data items x n ’s. Then, its sensitive complexity R F is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that such as a general CNN architecture covers many popular neural networks, e.g., fully connected neural networks, CNNs such as Lenet-5 (Lecun et al., 1998), VGG-16 (Liu & Deng, 2015), and so on.
Revised: Note that such as a general CNN architecture covers many popular neural networks, e.g., fully connected neural networks, CNNs such as Lenet-5 (Lecun et al., 1998), VGG-16 (Liu & Deng, 2015), and so on.

Original: M N EAR -C LASS S PECIALIST A SSUMPTION AND SSC
Revised: M N EAR -C LASS S PECIALIST A SSUMPTION AND SSC

Original: NCSA is a relaxed condition compared to having  expert annotators or having diagonally dominant annotators–see Fig.
Revised: NCSA is a relaxed condition compared to having all-class expert annotators or having diagonally dominant annotators–see Fig.

Original: It can be understood that NCSA assumption does not require any single annotator to be specialists with respect to all classes.
Revised: It can be understood that NCSA assumption does not require any single annotator to be specialists with respect to all classes.

Original: Nonetheless, having an  expert annotator m ∗ satisfies NCSA, but not vice versa.
Revised: Nonetheless, having an all-class expert annotator m ∗ satisfies NCSA, but not vice versa.

Original: The SSC (Definition 1) is a relaxed condition compared to the NCSA (Assumption 4).
Revised: The SSC (Definition 1) is a relaxed condition compared to the NCSA (Assumption 4).

Original: To illustrate this, we consider the geometry of the matrix W ♮ satisfying the NCSA and the SSC as shown in Fig.
Revised: To illustrate this, we consider the geometry of the matrix W ♮ satisfying the NCSA and the SSC as shown in Fig.

Original: N A LGORITHM D ESCRIPTION
Revised: N A LGORITHM D ESCRIPTION

Original: In this section, we detail the implementation of the regularized criteria in (8) and (9).
Revised: In this section, we detail the implementation of the regularized criteria in (8) and (9).

Original: The neural network predictor function f ∈ F is parameterized using θ and can be denoted as f θ : R d → ∆ .
Revised: The neural network predictor function f ∈ F is parameterized using θ and can be denoted as f θ : R d → ∆ .

Original: Let ψ = [ vec ( A 1 ) ⊤ , . .
Revised: Let ψ = [ vec ( A 1 ) ⊤ , . .

Original: , vec ( A M ) ⊤ , vec ( θ ) ⊤ ] ⊤ .
Revised: , vec ( A M ) ⊤ , vec ( θ ) ⊤ ] ⊤ .

Original: Let us also define the following for certain batch
Revised: Let us also define the following for certain batch

Original: B ⊂ [ N ] :
Revised: B ⊂ [ N ] :

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) F =
Revised: where (cid:101) F =

Original: [ f ( x n 1 ) , . . . ,
Revised: [ f ( x n 1 ) , . . . ,

Original: f ( x n |B| )] , n 1 , . .
Revised: f ( x n |B| )] , n 1 , . .

Original: . , n |B| ∈ B and W =
Revised: . , n |B| ∈ B and W =

Original: [ A ⊤ 1 , . . .
Revised: [ A ⊤ 1 , . . .

Original: , A ⊤ M ] ⊤ .
Revised: , A ⊤ M ] ⊤ .

Original: Let ∇ ψ L F ( ψ , B ) denote the stochastic gradient of L F ( ψ , B ) w.r.t.
Revised: Let ∇ ψ L F ( ψ , B ) denote the stochastic gradient of L F ( ψ , B ) w.r.t.

Original: ψ and ∇ ψ L W ( ψ , B ) denote the stochastic gradient of L W ( ψ , B ) w.r.t.
Revised: ψ and ∇ ψ L W ( ψ , B ) denote the stochastic gradient of L W ( ψ , B ) w.r.t.

Original: ψ .
Revised: ψ .

Original: Under these notations, the proposed methods
Revised: Under these notations, the proposed methods

Original: GeoCrowdNet(F) and GeoCrowdNet(W) are described in Algorithm 1 and 2, respectively.
Revised: GeoCrowdNet(F) and GeoCrowdNet(W) are described in Algorithm 1 and 2, respectively.

Original: while
Revised: while

Original:  Draw a random batch B ; Compute ∇ ψ L F ( ψ , B ) ; ψ ← AdamOptimizer ( ψ
Revised:  Draw a random batch B ; Compute ∇ ψ L F ( ψ , B ) ; ψ ← AdamOptimizer ( ψ

Original: end end 13 end
Revised: end end 13 end

Original: O A DDITIONAL E XPERIMENTS AND I MPLEMENTATION D ETAILS O. S YNTHETIC A NNOTATORS
Revised: O A DDITIONAL E XPERIMENTS AND I MPLEMENTATION D ETAILS O. S YNTHETIC A NNOTATORS

Original: Dataset.
Revised: Dataset.

Original: We consider CIFAR-10 (Krizhevsky, 2009) dataset for synthetic noisy label experiments.
Revised: We consider CIFAR-10 (Krizhevsky, 2009) dataset for synthetic noisy label experiments.

Original: CIFAR-10 dataset comprises of 60 , 000 labeled color images of animals, vehicles and so on, each having a size of 32 × 32 and belonging K = 10 different classes.
Revised: CIFAR-10 dataset comprises of 60 , 000 labeled color images of animals, vehicles and so on, each having a size of 32 × 32 and belonging K = 10 different classes.

Original: We use 45 , 000 images for training, 5 , 000 images for validation, and 10 , 000 images for testing.
Revised: We use 45 , 000 images for training, 5 , 000 images for validation, and 10 , 000 images for testing.

Original: Noisy Label Generation.
Revised: Noisy Label Generation.

Original: In order to produce noisy annotations for the images of the training data, we simulate M = 5 synthetic annotators.
Revised: In order to produce noisy annotations for the images of the training data, we simulate M = 5 synthetic annotators.

Original: We randomly choose an annotator m ∗ among M annotators
Revised: We randomly choose an annotator m ∗ among M annotators

Original: [Table]
Revised: [Table]

Original: and its confusion matrix is generated by A ♮m ∗
Revised: and its confusion matrix is generated by A ♮m ∗

Original: =
Revised: =

Original: I K + γ rand
Revised: I K + γ rand

Original: ( K, K ) , followed by normalization w.r.t.
Revised: ( K, K ) , followed by normalization w.r.t.

Original: ℓ 1 norm on the columns.
Revised: ℓ 1 norm on the columns.

Original: Here I K denotes the identity matrix of size K and rand ( K, K ) denotes the K × K matrix with its entries randomly chosen from a uniform distribution between 0 and 1.
Revised: Here I K denotes the identity matrix of size K and rand ( K, K ) denotes the K × K matrix with its entries randomly chosen from a uniform distribution between 0 and 1.

Original: The parameter γ controls how well the annotator m ∗ correctly identifies the ground-truth labels.
Revised: The parameter γ controls how well the annotator m ∗ correctly identifies the ground-truth labels.

Original: The confusion matrices of the remaining M − 1 annotators are generated such that they provide labels uniformly at random.
Revised: The confusion matrices of the remaining M − 1 annotators are generated such that they provide labels uniformly at random.

Original: This type of modeling for the confusion matrices resembles the hammer-spammer model as employed in the works (Rodrigues & Pereira, 2018; Tanno et al., 2019).
Revised: This type of modeling for the confusion matrices resembles the hammer-spammer model as employed in the works (Rodrigues & Pereira, 2018; Tanno et al., 2019).

Original: Using the ground-truth label y n ’s provided by the dataset and the generated confusion matrices, the noisy annotations (cid:98) y ( m ) n ’s are produced.
Revised: Using the ground-truth label y n ’s provided by the dataset and the generated confusion matrices, the noisy annotations (cid:98) y ( m ) n ’s are produced.

Original: Each annotation (cid:98) y ( m ) n is observed independently at random such that only 20% of the total annotations are available for training.
Revised: Each annotation (cid:98) y ( m ) n is observed independently at random such that only 20% of the total annotations are available for training.

Original: Note that, the true labels are not accessible by any of the methods.
Revised: Note that, the true labels are not accessible by any of the methods.

Original: Neural Network Architectures and Settings.
Revised: Neural Network Architectures and Settings.

Original: For CIFAR-10 dataset, we choose ResNet-9 architecture (He et al., 2016).
Revised: For CIFAR-10 dataset, we choose ResNet-9 architecture (He et al., 2016).

Original: Adam (Kingma & Ba, 2015) is used as the optimizer with weight decay of 10 − 4 and batch size of 128.
Revised: Adam (Kingma & Ba, 2015) is used as the optimizer with weight decay of 10 − 4 and batch size of 128.

Original: The regularization parameter λ and the initial learning rate of the Adam optimizer are chosen via grid search method using the validation set from { 0 . 01 , 0 . 001 , 0 . 0001 } and { 0 . 01 , 0 . 001 } , respectively.
Revised: The regularization parameter λ and the initial learning rate of the Adam optimizer are chosen via grid search method using the validation set from { 0 . 01 , 0 . 001 , 0 . 0001 } and { 0 . 01 , 0 . 001 } , respectively.

Original: We choose the same neural network structures for all the baselines.
Revised: We choose the same neural network structures for all the baselines.

Original: The confusion matrices are initialized with identity matrices of size K for the proposed method and the baselines TraceReg and CrowdLayer .
Revised: The confusion matrices are initialized with identity matrices of size K for the proposed method and the baselines TraceReg and CrowdLayer .

Original: Results.
Revised: Results.

Original: Table 3 presents the results for various values of γ —when γ is smaller, the expertise of annotator m ∗ in correctly labeling the data items is better.
Revised: Table 3 presents the results for various values of γ —when γ is smaller, the expertise of annotator m ∗ in correctly labeling the data items is better.

Original: One can note that the proposed approach, notably GeoCrowdNet(F) , outperforms the baselines in all these challenging scenarios.
Revised: One can note that the proposed approach, notably GeoCrowdNet(F) , outperforms the baselines in all these challenging scenarios.

Original: When annotator m ∗ ’s labeling accuracy drops drastically as well, i.e., γ becomes larger, GeoCrowdNet(F) performs the best, This implies that even if there are no class specialists, GeoCrowdNet(F) thrives better for this large N scenario.
Revised: When annotator m ∗ ’s labeling accuracy drops drastically as well, i.e., γ becomes larger, GeoCrowdNet(F) performs the best, This implies that even if there are no class specialists, GeoCrowdNet(F) thrives better for this large N scenario.

Original: Figs.
Revised: Figs.

Original: 3, 4 and 5 show the ground-truth confusion matrices and the corresponding estimates when γ = 0 .
Revised: 3, 4 and 5 show the ground-truth confusion matrices and the corresponding estimates when γ = 0 .

Original: 01 for GeoCrowdNet methods.
Revised: 01 for GeoCrowdNet methods.

Original: One can see all methods estimate the confusion matrices reasonably well, corroborating our identifiability claims.
Revised: One can see all methods estimate the confusion matrices reasonably well, corroborating our identifiability claims.

Original: The
Revised: The

Original: GeoCrowdNet(F) exhibits lower average mean squared error (MSE) for the estimated confusion matrices.
Revised: GeoCrowdNet(F) exhibits lower average mean squared error (MSE) for the estimated confusion matrices.

Original: O.
Revised: O.

Original: R EAL
Revised: R EAL

Original: D ATA E XPERIMENTS - M ACHINE A NNOTATIONS
Revised: D ATA E XPERIMENTS - M ACHINE A NNOTATIONS

Original: Here, we provide additional details of the real data experiments.
Revised: Here, we provide additional details of the real data experiments.

Original: For experiments with machine annotations, we consider the MNIST  dataset.
Revised: For experiments with machine annotations, we consider the MNIST and the Fashion-MNIST dataset.

Original: Each image in the dataset is of size 28 × 28 in grey-scale format.
Revised: Each image in these datasets is of size 28 × 28 in grey-scale format.

Original: We consider 57 , 000 images as training data, 3000 images for validation, and 10 , 000 images for testing.
Revised: We consider 57 , 000 images as training data, 3000 images for validation, and 10 , 000 images for testing.

Original: For machine annotations, we train a number of machine classifiers to act as annotators.
Revised: For machine annotations, we train a number of machine classifiers to act as annotators.

Original: In Table 1, under case 2, we choose M = 5 annotators.
Revised: In Table 1, under case 2, we choose M = 5 annotators.

Original: Specifically, Linear SVM, logistic regression, k -NN with k = 5 , CNN with two convolution layers followed by a max pooling layer, a fully connected layer and a softmax layer, fully connected neural network (FCNN) with 1 hidden layer andhidden units are trained.
Revised: Specifically, Linear SVM, logistic regression, k -NN with k = 5 , CNN with two convolution layers followed by a max pooling layer, a fully connected layer and a softmax layer, fully connected neural network (FCNN) with 1 hidden layer andhidden units are trained.

Original: The CNN is trained for 5 epochs and the FCNN is trained for 10 epochs.
Revised: The CNN is trained for 5 epochs and the FCNN is trained for 10 epochs.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Figure]

Original: [Figure]
Revised: [Figure]

Original: The individual label prediction accuracy of these annotators on unseen data items of size N ranges from 15.88% to 82.23% (averaged over 5 random trials) and 2 annotators are having less than 50%  on average.
Revised: For example, for the MNIST dataset, the individual label prediction accuracy of these annotators on unseen data items of size N ranges from 15.88% to 82.23% (averaged over 5 random trials) andannotators are having less than 50% accuracy on average.

Original: For case 2, we trained more annotators by changing some parameters of the above mentioned machine classifiers.
Revised: For case 2, we trained more annotators by changing some parameters of the above mentioned machine classifiers.

Original: Specifically, we use linear SVM, polynomial kernel-based SVM, and Gaussian
Revised: Specifically, we use linear SVM, polynomial kernel-based SVM, and Gaussian

Original: SVM as different variants of SVM.
Revised: SVM as different variants of SVM.

Original: For k -NN, we choose k from { 3 , 5 , 7 , 10 } .
Revised: For k -NN, we choose k from { 3 , 5 , 7 , 10 } .

Original: FCNN, CNN, and logistic regression are trained with different number of epochs { 10 , 15 , 20 , 25 } and with random initializations for each case.
Revised: FCNN, CNN, and logistic regression are trained with different number of epochs { 10 , 15 , 20 , 25 } and with random initializations for each case.

Original: Under case 1, if an annotator is chosen to be a specialist, it is trained with more number of samples ( 10 , 000 samples) with more number of epochs such that its label prediction accuracy is higher than 90%.
Revised: Under case 1, if an annotator is chosen to be a specialist, it is trained with more number of samples ( 10 , 000 samples) with more number of epochs such that its label prediction accuracy is higher than 90%.

Original: Under this strategy, in Table 1 for the case M = 15 , the individual label prediction accuracy of the annotators on unseen data items of size N ranges from 3.15% to 95.27% (2 annotators with more than 90% accuracy) .
Revised: Under this strategy, in Table 1 for the case M = 15 , the individual label prediction accuracy of the annotators on unseen data items of size N ranges from 3.15% to 95.27% (2 annotators with more than 90% accuracy) .

Original: For the case with M = 20 , the individual label prediction accuracy of the annotators on unseen data items of size N ranges from 3.24% to 95.73% (3 annotators with more than 90% accuracy).
Revised: For the case with M = 20 , the individual label prediction accuracy of the annotators on unseen data items of size N ranges from 3.24% to 95.73% (3 annotators with more than 90% accuracy).

Original: For experiments with annotations collected from AMT, we use LabelMe dataset (Rodrigues et al., 2017; Russell et al., 2007) and Music dataset (Rodrigues et al., 2014).
Revised: Additional experiment results using the MNIST and Fashion-MNIST datasets can be found in Tables 4-6.

Original: LabelMe dataset (Rodrigues et al., 2017; Russell et al., 2007) is an image classification dataset consisting of 2688 images from K = 8 different classes, namely, highway, inside city, tall building,
Revised: As expected, GeoCrowdNet(W) and GeoCrowdNet(F) offer the best performance in Case 1 and Case 2, respectively. .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: For experiments with annotations collected from AMT, we use LabelMe dataset (Rodrigues et al., 2017; Russell et al., 2007) and Music dataset (Rodrigues et al., 2014). LabelMe dataset (Rodrigues et al., 2017; Russell et al., 2007) is an image classification dataset consisting of 2688 images from K = 8 different classes, namely, highway, inside city, tall building, street, forest, coast, mountain, and open country. From the available images, 1000 images are

Original:  [Table]
Revised: [Figure] [Figure] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Figure]
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Figure]

Revised: the

Original: [Figure]
Revised: [Figure]

Revised: Methods

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Figure]
Revised: TraceReg [Table] [Table] annotated by M = 59 AMT workers.

Original: [Table]
Revised: In total, about 2547 image annotations are obtained by the AMT workers whose labeling accuracy ranges from 0% to 100% with a mean accuracy of 69 . 2% . In order to enrich the training dataset, standard augmentation techniques such as rescaling, cropping, horizontal flips etc. are employed and accordingly, the training dataset consists of 10 , 000 images annotated by 59 workers. The validation set consists of 500 images and the remaining 1188 images are used for testing.

Original: [Figure]
Revised: [Table]

Original:  Music dataset (Rodrigues et al., 2014) consists of audio samples of 10 different genres of music such as classical, country, disco, hiphop, jazz, rock, blues, reggae, pop and metal.
Revised: [Table] [Table] Music dataset (Rodrigues et al., 2014) consists of audio samples of 10 different genres of music such as classical, country, disco, hiphop, jazz, rock, blues, reggae, pop, and metal.

Original: The dataset has about 1000 samples of songs each having a duration of 30 seconds.
Revised: The dataset has about 1000 samples of songs each having a duration of 30 seconds.

Original: About 700 samples are annotated by
Revised: About 700 samples are annotated by

Original: AMT workers (overall, about 2946 annotations are observed) and the remaining 300 are allocated for testing.
Revised: AMT workers (overall, about 2946 annotations are observed) and the remaining 300 are allocated for testing.

Original: Out of the annotated 700 samples, we consider 595 samples for training, and the remaining 105 samples are used for validation.
Revised: Out of the annotated 700 samples, we consider 595 samples for training, and the remaining 105 samples are used for validation.

Original: For LabelMe dataset, we employ the settings used in (Rodrigues & Pereira, 2018).
Revised: For LabelMe dataset, we employ the settings used in (Rodrigues & Pereira, 2018).

Original: The pretrained
Revised: The pretrained

Original: VGG-16 embeddings for the images are given as inputs to a fully connected (FC) neural network with one hidden layer having 128 hidden units and ReLU activation functions.
Revised: VGG-16 embeddings for the images are given as inputs to a fully connected (FC) neural network with one hidden layer having 128 hidden units and ReLU activation functions.

Original: A dropout layer with rate 50% is also used, followed by a softmax layer.
Revised: A dropout layer with rate 50% is also used, followed by a softmax layer.

Original: For Music dataset, we choose the same FC layer and the softmax layer as employed for LabelMe, but with batch normalization layers before each of these layers.[Figure] [Figure] [Table]
Revised: For Music dataset, we choose the same FC layer and the softmax layer as employed for LabelMe, but with batch normalization layers before each of these layers.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]