Original: U NIVERSALIZING W EAK S UPERVISION
Revised: U NIVERSALIZING W EAK S UPERVISION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Weak supervision (WS) frameworks are a popular way to bypass hand-labeling large datasets for training data-hungry models.
Revised: Weak supervision (WS) frameworks are a popular way to bypass hand-labeling large datasets for training data-hungry models.

Original: These approaches synthesize multiple noisy but cheaply-acquired estimates of labels into a set of high-quality pseudolabels for downstream training.
Revised: These approaches synthesize multiple noisy but cheaply-acquired estimates of labels into a set of high-quality pseudolabels for downstream training.

Original: However, the synthesis technique is speciﬁc to a particular kind of label, such as binary labels or sequences, and each new label type requires manually designing a new synthesis algorithm.
Revised: However, the synthesis technique is speciﬁc to a particular kind of label, such as binary labels or sequences, and each new label type requires manually designing a new synthesis algorithm.

Original: Instead, we propose a universal technique that enables weak supervision over any label type while still offering desirable properties, including practical ﬂexibility, computational efﬁciency, and theoretical guarantees.
Revised: Instead, we propose a universal technique that enables weak supervision over any label type while still offering desirable properties, including practical ﬂexibility, computational efﬁciency, and theoretical guarantees.

Original: We apply this technique to important problems previously not tackled by WS frameworks including learning to rank, regression, and learning in hyperbolic space.
Revised: We apply this technique to important problems previously not tackled by WS frameworks including learning to rank, regression, and learning in hyperbolic space.

Original: Theoretically, our synthesis approach produces a consistent estimator for learning a challenging but important generalization of the exponential family model.
Revised: Theoretically, our synthesis approach produces a consistent estimators for learning some challenging but important generalizations of the exponential family model.

Original: Experimentally, we validate our framework and show improvement over baselines in diverse settings including real-world learning-to-rank and regression problems along with learning on hyperbolic manifolds.
Revised: Experimentally, we validate our framework and show improvement over baselines in diverse settings including real-world learning-to-rank and regression problems along with learning on hyperbolic manifolds.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Weak supervision (WS) frameworks help overcome the labeling bottleneck: the challenge of building a large dataset for use in training data-hungry deep models.
Revised: Weak supervision (WS) frameworks help overcome the labeling bottleneck: the challenge of building a large dataset for use in training data-hungry deep models.

Original: WS approaches replace hand-labeling with synthesizing multiple noisy but cheap sources, called labeling functions, applied to unlabeled data.
Revised: WS approaches replace hand-labeling with synthesizing multiple noisy but cheap sources, called labeling functions, applied to unlabeled data.

Original: As these sources may vary in quality and be dependent, a crucial step is to model their accuracies and correlations.
Revised: As these sources may vary in quality and be dependent, a crucial step is to model their accuracies and correlations.

Original: Informed by this model, high-quality pseudolabels are produced and used to train a downstream model.
Revised: Informed by this model, high-quality pseudolabels are produced and used to train a downstream model.

Original: This simple yet ﬂexible approach is highly successful in research and industry settings (Bach et al., 2019; R ´ e et al., 2020) .
Revised: This simple yet ﬂexible approach is highly successful in research and industry settings (Bach et al., 2019; R ´ e et al., 2020) .

Original: WS frameworks offer three advantages: they are (i) ﬂexible and subsume many existing ways to integrate side information, (ii) computationally efﬁcient, and (iii) they offer theoretical guarantees, including estimator consistency.
Revised: WS frameworks offer three advantages: they are (i) ﬂexible and subsume many existing ways to integrate side information, (ii) computationally efﬁcient, and (iii) they offer theoretical guarantees, including estimator consistency.

Original: Unfortunately, these beneﬁts come at the cost of being particular to very speciﬁc problem settings: categorical, usually binary, labels.
Revised: Unfortunately, these beneﬁts come at the cost of being particular to very speciﬁc problem settings: categorical, usually binary, labels.

Original: Extensions, e.g., to time-series data (Safranchik et al., 2020), or to segmentation masks (Hooper et al., 2021), require a new model for source synthesis, a new algorithm, and more.
Revised: Extensions, e.g., to time-series data (Safranchik et al., 2020), or to segmentation masks (Hooper et al., 2021), require a new model for source synthesis, a new algorithm, and more.

Original: We seek to side-step this expensive one-at-a time process via a universal approach , enabling WS to work in any problem setting while still providing the three advantageous properties above.
Revised: We seek to side-step this expensive one-at-a time process via a universal approach , enabling WS to work in any problem setting while still providing the three advantageous properties above.

Original: The main technical challenge for universal WS is the diversity of label settings: classiﬁcation, structured prediction, regression, rankings, and more.
Revised: The main technical challenge for universal WS is the diversity of label settings: classiﬁcation, structured prediction, regression, rankings, and more.

Original: Each of these settings seemingly demands a different approach for learning the source synthesis model, which we refer to as the label model .
Revised: Each of these settings seemingly demands a different approach for learning the source synthesis model, which we refer to as the label model .

Original: For example, Ratner et al.
Revised: For example, Ratner et al.

Original: (2019) assumed that the distribution of the sources and latent true label is an Ising model and relied on a property of such distributions: that the inverse covariance matrix is graph-structured.
Revised: (2019) assumed that the distribution of the sources and latent true label is an Ising model and relied on a property of such distributions: that the inverse covariance matrix is graph-structured.

Original: It is not clear how to lift such a property to spaces of permutations or to Riemannian manifolds.
Revised: It is not clear how to lift such a property to spaces of permutations or to Riemannian manifolds.

Original: We propose a general recipe to handle any type of label.

Original: The data generation process for the weak sources is modeled with an exponential family distribution that can represent a label from any metric space ( Y , d Y ) .

Original: We [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 𝓧
Revised: 𝓧

Original: [Figure]
Revised: [Figure]

Original: 𝒌
Revised: 𝒌

Original: [Figure]
Revised: [Figure]

Original: {𝐂
Revised: 𝓨∈{𝐂

Original: [Figure]
Revised: [Figure]

Original: 𝐃
Revised: 𝐃

Original: [Figure]
Revised: [Figure]

Original: 𝐓
Revised: 𝐓

Original: [Figure]
Revised: [Figure]

Original: 𝛌
Revised: Weak

Original: [Figure]
Revised: [Figure]

Original: 𝛌
Revised: Supervision

Original: [Figure]
Revised: [Figure]

Original: ෡𝒀 ෡𝒀 ෡𝒀 ෡𝒀 ෡𝒀 𝛌 1 𝛌m … 𝛌 1 𝛌 m … 𝛌 1 𝛌 m
Revised: 𝛌 1 𝛌

Original: [Figure]
Revised: [Figure]

Original:  𝛌 1 𝛌 m … ෝ𝒚 𝒊 ∈𝑺ෝ𝒚 𝒊 ∈ℝ ෝ𝒚 𝒊 ∈{𝐂,𝐃,𝐓} ෝ𝒚 𝒊 ∈𝑴 ෝ𝒚 𝒊
Revised: ෡𝒀 ෡𝒀 ෡𝒀 ෡𝒀 ෡𝒀 𝛌 1 𝛌 m … 𝛌 1 𝛌 m … 𝛌 1 𝛌 m … 𝛌 1 𝛌 m … ෝ𝒚 𝒊 ∈𝑺ෝ𝒚 𝒊 ∈ℝ ෝ𝒚 𝒊 ∈{𝐂,𝐃,𝐓} ෝ𝒚 𝒊 ∈𝑴 ෝ𝒚 𝒊

Original: [Figure]
Revised: [Figure]

Original: 𝛌 𝒊
Revised: 𝛌 𝒊

Original: [Figure]
Revised: [Figure]

Original: 𝓧↦𝑺
Revised: 𝓧↦𝑺

Original: [Figure]
Revised: [Figure]

Original: 𝛌 𝒊
Revised: 𝛌 𝒊

Original: [Figure]
Revised: [Figure]

Revised: 𝓨

Revised: [Figure]

Revised: We propose a general recipe to handle any type of label.

Revised: The data generation process for the weak sources is modeled with an exponential family distribution that can represent a label from any metric space ( Y , d Y ) .

Original:  embed labels from Y into two tractable choices of space: the Boolean hypercube {± 1 } d and Euclidean space R d .
Revised: We embed labels from Y into two tractable choices of space: the Boolean hypercube {± 1 } d and Euclidean space R d .

Original: The label model (used for source synthesis) is learned with an efﬁcient method-of-moments approach in the embedding space.
Revised: The label model (used for source synthesis) is learned with an efﬁcient method-of-moments approach in the embedding space.

Original: It only requires solving asmall number of scalar linear or quadratic equations.
Revised: It only requires solving a number of scalar linear or quadratic equations.

Original: Better yet, for isometric embeddings, we show this estimator is consistent via ﬁnite sample bounds.
Revised: Better yet, for certain cases, we show this estimator is consistent via ﬁnite sample bounds.

Original: Experimentally, we demonstrate our approach on ﬁve choices of problems never before tackled in WS:
Revised: Experimentally, we demonstrate our approach on ﬁve choices of problems never before tackled in WS

Original:  • Learning rankings: on two real-world rankings tasks, our approach with as few as ﬁve sources performs better than supervised learning with a smaller number of true labels.
Revised: : • Learning rankings: on two real-world rankings tasks, our approach with as few as ﬁve sources performs better than supervised learning with a smaller number of true labels.

Original: In contrast, an adaptation of the Snorkel (Ratner et al., 2018) framework cannot reach this performance with as many as 18 sources.
Revised: In contrast, an adaptation of the Snorkel (Ratner et al., 2018) framework cannot reach this performance with as many as 18 sources.

Original: • Regression: on two real-world regression datasets, when using 6 or more labeling function, the performance of our approach is comparable to fully-supervised models.
Revised: • Regression: on two real-world regression datasets, when using 6 or more labeling function, the performance of our approach is comparable to fully-supervised models.

Original: • Learning in hyperbolic spaces: on a geodesic regression task in hyperbolic space, we consistently outperform fully-supervised learning, even when using only 3 labeling functions (LFs).
Revised: • Learning in hyperbolic spaces: on a geodesic regression task in hyperbolic space, we consistently outperform fully-supervised learning, even when using only 3 labeling functions (LFs).

Original: • Estimation in generic metric spaces: in a synthetic setting of metric spaces induced by random graphs, we demonstrate that our method handles LF heterogeneity better than the majority vote baseline.
Revised: • Estimation in generic metric spaces: in a synthetic setting of metric spaces induced by random graphs, we demonstrate that our method handles LF heterogeneity better than the majority vote baseline.

Original: • Learning parse trees: in semantic dependency parsing, we outperform strong baseline models.
Revised: • Learning parse trees: in semantic dependency parsing, we outperform strong baseline models.

Original: In each experiment, we conﬁrm the following desirable behaviors of weak supervision: (i) more high-quality and independent sources yield better pseudolabels, (ii) more pseudolabels can yield better performance compared to training on fewer clean labels, (iii) when source accuracy varies, our approach outperforms generalizations of majority vote.
Revised: In each experiment, we conﬁrm the following desirable behaviors of weak supervision: (i) more high-quality and independent sources yield better pseudolabels, (ii) more pseudolabels can yield better performance compared to training on fewer clean labels, (iii) when source accuracy varies, our approach outperforms generalizations of majority vote.

Original:  P ROBLEM F ORMULATION AND L ABEL M ODEL
Revised:  P ROBLEM F ORMULATION AND L ABEL M ODEL

Original: We give background on weak supervision frameworks, provide the problem formulation, describe our choice of universal label model, special cases, the embedding technique we use to reduce our problem to two easilyhandled cases, and discuss end model training.
Revised: We give background on weak supervision frameworks, provide the problem formulation, describe our choice of universal label model, special cases, the embedding technique we use to reduce our problem to two easilyhandled cases, and discuss end model training.

Original: Background WS frameworks are a principled way to integrate weak or noisy information to produce label estimates.
Revised: Background: WS frameworks are a principled way to integrate weak or noisy information to produce label estimates.

Original: These weak sources include small pieces of code expressing heuristic principles, crowdworkers, lookups in external knowledge bases, pretrained models, and many more (Karger et al., 2011; Mintz et al., 2009; Gupta & Manning, 2014; Dehghani et al., 2017; Ratner et al., 2018).
Revised: These weak sources include small pieces of code expressing heuristic principles, crowdworkers, lookups in external knowledge bases, pretrained models, and many more (Karger et al., 2011; Mintz et al., 2009; Gupta & Manning, 2014; Dehghani et al., 2017; Ratner et al., 2018).

Original: Given an unlabeled dataset, users construct a set of labeling functions (LFs) based on weak sources and apply them to the data.
Revised: Given an unlabeled dataset, users construct a set of labeling functions (LFs) based on weak sources and apply them to the data.

Original: The estimates produced by each LF are synthesized to produce pseudolabels that can be used to train a downstream model.
Revised: The estimates produced by each LF are synthesized to produce pseudolabels that can be used to train a downstream model.

Original: Problem Formulation Let x 1 , x 2 , . . .
Revised: Problem Formulation: Let x 1 , x 2 , . . .

Original: , x n be a dataset of unlabeled datapoints from X .
Revised: , x n be a dataset of unlabeled datapoints from X .

Original: Associated with these are labels from an arbitrary metric space Y with metric d Y  .
Revised: Associated with these are labels from an arbitrary metric space Y with metric d Y 1 .

Original: In conventional supervised learning, we would have pairs ( x 1 , y 1 ) , . . .
Revised: In conventional supervised learning, we would have pairs ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) ; however, in WS, we do not have access to the labels.
Revised: , ( x n , y n ) ; however, in WS, we do not have access to the labels.

Original: Instead, we have estimates of y from m labeling functions (LFs).
Revised: Instead, we have estimates of y from m labeling functions (LFs).

Original: Each such LF s : X → Y produces an estimate of the true label y from a datapoint x .
Revised: Each such LF s : X → Y produces an estimate of the true label y from a datapoint x .

Original: We write λ a ( i ) ∈ Y for the output of the a th labeling function s a applied to the i th sample.
Revised: We write λ a ( i ) ∈ Y for the output of the a th labeling function s a applied to the i th sample.

Original: Our goal is to obtain an estimate of the true label
Revised: Our goal is to obtain an estimate of the true label

Original: y i using the LF outputs
Revised: y i using the LF outputs

Original: λ 1 ( i ) , . .
Revised: λ 1 ( i ) , . .

Original: λ m ( i ) .
Revised: λ m ( i ) .

Original: This estimate ˆ y , is used to train a downstream model.
Revised: This estimate ˆ y , is used to train a downstream model.

Original: To produce it, we learn the label model P ( λ 1 , λ 2 , . . .
Revised: To produce it, we learn the label model P ( λ 1 , λ 2 , . . .

Original: , λ m | y ) .
Revised: , λ m | y ) .

Original: The main challenge is that we never observe samples of y ; it is latent .
Revised: The main challenge is that we never observe samples of y ; it is latent .

Original: We can summarize the weak supervision procedure in two steps:
Revised: We can summarize the weak supervision procedure in two steps:

Original: • Learning the label model : use the samples λ a ( i ) to learn the label model P ( λ 1 , λ 2 , . . .
Revised: • Learning the label model : use the samples λ a ( i ) to learn the label model P ( λ 1 , λ 2 , . . .

Original: , λ m | y ) , • Perform inference : compute ˆ y i , or P ( y i | λ 1 ( i ) , . . .
Revised: , λ m | y ) , • Perform inference : compute ˆ y i , or P ( y i | λ 1 ( i ) , . . .

Original: , λ m ( i )) , or a related quantity.
Revised: , λ m ( i )) , or a related quantity.

Original: Modeling the Sources Previous approaches, e.g., Ratner et al.
Revised: Modeling the sources Previous approaches, e.g., Ratner et al.

Original: (2019), select a particular parametric choice to model p ( λ 1 , . . .
Revised: (2019), select a particular parametric choice to model p ( λ 1 , . . .

Original: , λ m | y ) that balances two goals: (i) model richness that captures differing LF accuracies and correlations, and (ii) properties that permit efﬁcient learning.
Revised: , λ m | y ) that balances two goals: (i) model richness that captures differing LF accuracies and correlations, and (ii) properties that permit efﬁcient learning.

Original: Our setting demands greater generality.
Revised: Our setting demands greater generality.

Original: However, we still wish to exploit the properties of exponential family models.
Revised: However, we still wish to exploit the properties of exponential family models.

Original: The natural choice is
Revised: The natural choice is

Original: [Equation]
Revised: [Equation]

Original: Here, the set E is a set of correlations corresponding to the graphical representation of the model (Figure 1, right).
Revised: Here, the set E is a set of correlations corresponding to the graphical representation of the model (Figure 1, right).

Original: Observe how source quality is modeled in (1).
Revised: Observe how source quality is modeled in (1).

Original: If the value of θ a is very large, any disagreement between the estimate λ a and y is penalized through the distance d Y ( λ a , y )c and so has low probability.
Revised: If the value of θ a is very large, any disagreement between the estimate λ a and y is penalized through the distance d Y ( λ a , y ) and so has low probability.

Original: If θ a is very small, such disagreements will be common; the source is inaccurate .
Revised: If θ a is very small, such disagreements will be common; the source is inaccurate .

Original: We also consider a more general version of (1).
Revised: We also consider a more general version of (1).

Original: We replace − θ a d Y ( λ a , y ) with a per-source distance d θ a .
Revised: We replace − θ a d Y ( λ a , y ) with a per-source distance d θ a .

Original: For example, for Y = {± 1 } d , d θ a ( λ a , y ) =
Revised: For example, for Y = {± 1 } d , d θ a ( λ a , y ) =

Original: − θ Ta ( λ a − y ) abs , with θ a ∈ R d , generalizes the Hamming distance.
Revised: − θ Ta | λ a − y | , with θ a ∈ R d , generalizes the Hamming distance.

Original: Similarly, for Y = R d , we can generalize − θ a (cid:107) λ a − y (cid:107) 2 with d θ a ( λ a , y )
Revised: Similarly, for Y = R d , we can generalize − θ a (cid:107) λ a − y (cid:107) 2 with d θ a ( λ a , y )

Original: = − ( λ a − y )
Revised: = − ( λ a − y )

Original: T θ a ( λ a − y ) , with θ a ∈ R d × d p.d., so that LF errors are not necessarily isotropic.
Revised: T θ a ( λ a − y ) , with θ a ∈ R d × d p.d., so that LF errors are not necessarily isotropic.

Original: In the Appendix B, we detail variations and special cases of such models along with relationships to existing weak supervision work.
Revised: In the Appendix B, we detail variations and special cases of such models along with relationships to existing weak supervision work.

Original: Below, we give a selection of examples, noting that the last threeof which cannot be tackled with existing methods.
Revised: Below, we give a selection of examples, noting that the last three cannot be tackled with existing methods.

Original: • Binary classiﬁcation : Y = {± 1 } , d Y is the Hamming distance, c = 1 : this yields a shifted Ising model for standard binary classiﬁcation, as in Fu et al.
Revised: • Binary classiﬁcation : Y = {± 1 } , d Y is the Hamming distance: this yields a shifted Ising model for standard binary classiﬁcation, as in Fu et al.

Original: • Sequence learning : Y = {± 1 } d , d Y is the Hamming distance, c = 1 : this yields an Ising model for sequences, as in Sala et al.
Revised: • Sequence learning : Y = {± 1 } d , d Y is the Hamming distance: this yields an Ising model for sequences, as in Sala et al.

Original: (2019) and Safranchik et al.
Revised: (2019) and Safranchik et al.

Original: Ranking : Y = S ρ , the permutation group on { 1 , . . .
Revised: Ranking : Y = S ρ , the permutation group on { 1 , . . .

Original: , ρ } , d Y is the Kendall tau distance, c = 1 .
Revised: , ρ } , d Y is the Kendall tau distance.

Original: This is a heterogenous Mallows model , where rankings are produced from varying-quality sources.
Revised: This is a heterogenous Mallows model , where rankings are produced from varying-quality sources.

Original: If m = 1 , we obtain a variant of the conventional Mallows model (Mallows, 1957).
Revised: If m = 1 , we obtain a variant of the conventional Mallows model (Mallows, 1957).

Original: • Regression : Y =
Revised: • Regression : Y =

Original: R , d Y is the  (cid:96) 2 distance, c = 2 produces sources in R d with Gaussian errors.
Revised: R , d Y is the squared (cid:96) 2 distance: it produces sources in R d with Gaussian errors.

Original: • Learning on Riemannian manifolds :
Revised: • Learning on Riemannian manifolds :

Original: Y = M , a Riemannian manifold (e.g., hyperbolic space), d Y is the Riemannian distance d M , c = 1 .
Revised: Y = M , a Riemannian manifold (e g., hyperbolic space), d Y is the Riemannian distance d M induced by the space’s Riemannian metric.

Original: Majority Vote  and its Relatives A simplifying approach often used as a baseline in weak supervision is the majority vote .
Revised: Majority Vote (MV) and its Relatives A simplifying approach often used as a baseline in weak supervision is the majority vote .

Original: Assume that the sources are conditionally independent (i.e. E is empty in (1)) and all accuracies are identical.
Revised: Assume that the sources are conditionally independent (i.e. E is empty in (1)) and all accuracies are identical.

Original: In this case, there is no need to learn the model (1); instead, the most “popular”label is used.
Revised: In this case, there is no need to learn the model (1); instead, the most “popular”

Original: For binary labels, this is the majority label.

Original: In the universal setting, a natural generalization is

Original: [Equation]

Original: Special cases of (2) have their own name; for permutations, it is the Kemeny rule (Kemeny, 1959) used to solve the rank aggregation problem.

Original: For Riemannian manifolds, ˆ y MV is called the Fr ´ echet or Karcher mean .

Original: Problem
Revised: Problem

Original: Set
Revised: Set

Original: Distancec
Revised: Distance

Revised: [Table]

Original: MV Equivalent
Revised: Equivalent

Original: Im ( g )
Revised: Im [Table]

Original: End Model
Revised: End Model

Original: Binary Classiﬁcation { [Table] 1 } (cid:96)
Revised: Binary Classiﬁcation {± 1 } (cid:96)

Original: Majority Vote {± 1 }
Revised: Majority Vote { [Table]

Original: Binary Classiﬁer
Revised: Binary Classiﬁer

Original: Ranking
Revised: Ranking

Original: S ρ
Revised: S [Table]

Original: Kendall tau
Revised: Kendall tau

Original: Kemeny Rule {± 1 } ( ρ 2 )
Revised: Kemeny Rule {± 1 } ( ρ 2 )

Original: Learning to Rank
Revised: Learning to Rank

Original: Regression
Revised: Regression

Original: R  (cid:96)
Revised: R squared- (cid:96)

Original: Arithmetic Mean
Revised: Arithmetic Mean

Original: R
Revised: R

Original: Linear Regression
Revised: Linear Regression

Original: Riemannian Manifold
Revised: Riemannian Manifold

Original: M
Revised: M

Original: Riemannian
Revised: Riemannian

Original: Fr´echet Mean
Revised: Fr´echet Mean

Original: R d
Revised: R d

Original: Geodesic Regression
Revised: Geodesic Regression

Original: Dependency Parsing
Revised: Dependency Parsing

Original: T(cid:96) [Table]
Revised: T

Original: [Table]
Revised: [Table]

Original: Embedding: If g is not given, use Multidimensional Scaling (MDS) to obtain embeddings g ( λ a ) ∈ R d ∀ a for a ∈ { 1 , 2 , . . . , m } do
Revised: Parsing Model label is used.

Original: For b : ( a, b ) ∈ E Estimate Correlations: ∀ i, j , ˆ E (cid:2)
Revised: For binary labels, this is the majority label. In the universal setting, a natural generalization is

Original: [Table]
Revised: [Equation]

Original: ( λ a ) i g ( λ b ) j (cid:3) = 1 n (cid:80) nt = [Table] ( λ
Revised: Special cases of (2) have their own name; for S ρ , it is the Kemeny rule (Kemeny, 1959).

Original: [Table] ( t )) i g ( λ b ( t )) j
Revised: For d M , the squared

Original: Estimate Accuracy: Pick b, c : ( a, b ) (cid:54)∈ E, ( a, c ) (cid:54)∈ E, ( b, c ) (cid:54)∈ E . if Im ( g ) = {± 1 } d ∀ i , Estimate ˆ O a,b = ˆ P ( g ( λ a ) i = 1 , g ( λ b ) i = 1) , ˆ O a,c , ˆ O b,c , Estimate (cid:96) a = ˆ P ( g ( λ a ) i = 1) , (cid:96) b , (cid:96) c [Table]
Revised: Riemannian manifold distance, ˆ y MV is called the Fr´echet or Karcher mean .

Original: Majority vote, however, is insufﬁcient in cases where there is variation in the source qualities.
Revised: Majority vote, however, is insufﬁcient in cases where there is variation in the source qualities.

Original: We must learn the label model.
Revised: We must learn the label model.

Original: However, generically learning (1) is an ambitious goal.
Revised: However, generically learning (1) is an ambitious goal.

Original: Even cases that specialize (1) in multiple ways have only recently been fully solved, e.g., the permutation case for identical θ a ’s) was fully characterized by Mukherjee (2016).
Revised: Even cases that specialize (1) in multiple ways have only recently been fully solved, e.g., the permutation case for identical θ a ’s was fully characterized by Mukherjee (2016).

Original: To overcome the challenge of generality, we opt for an embedding approach that reduces our problem to two tractable cases.
Revised: To overcome the challenge of generality, we opt for an embedding approach that reduces our problem to two tractable cases.

Original: Universality via Embeddings To deal with the very high level of generality, we reduce the problem to just two metric spaces: the boolean hypercube {− 1 , +1 } d
Revised: Universality via Embeddings To deal with the very high level of generality, we reduce the problem to just two metric spaces: the boolean hypercube {− 1 , +1 } d

Original: and Euclidean space R d .
Revised: and Euclidean space R d .

Original: To accomplish this, let g :
Revised: To do so, let g :

Original: Y → {± 1 } d (or g :
Revised: Y → {± 1 } d (or g :

Original: Y → R d ) be an injective embedding function .
Revised: Y → R d ) be an injective embedding function .

Original: The advantage of this approach is that if g is isometric —that is, distance preserving—then probabilities are preserved under g .
Revised: The advantage of this approach is that if g is isometric —distance preserving—then probabilities are preserved under g .

Original: This is because the sufﬁcient statistics are preserved: for example, for g : Y → R d , − θ a d Y ( λ a , y )c
Revised: This is because the sufﬁcient statistics are preserved: for example, for g : Y → R d , − θ a d Y ( λ a , y )

Original: = − θ a d ( g ( λ a ) , g ( y ))
Revised: = − θ a d ( g ( λ a ) , g ( y ))

Original: c
Revised: =

Original: = − θ a (cid:107) g
Revised: − θ a (cid:107) g

Original: ( λ a ) − g ( y ) (cid:107) c , yielding is a  multivariate normal for c = 2 .
Revised: ( λ a ) − g ( y ) (cid:107) c , so that if g is a bijection, we obtain a multivariate normal for c = 2 .

Original: Even if g is not isometric, there is a rich literature on low-distortion embeddings, with Bourgain’s theorem as a cornerstone result (Bourgain, 1985).
Revised: If g is not isometric, there is a rich literature on low-distortion embeddings, with Bourgain’s theorem as a cornerstone result (Bourgain, 1985).

Original: This makes it possible to bound the error in recovering the parameters for any label type.
Revised: This can be used to bound the error in recovering the parameters for any label type.

Original: End Model
Revised: End Model

Original: Once we have produced pseudolabels—either by applying generalized majority vote or by learning the label model and performing inference—we can use the labels to train an end model.
Revised: Once we have produced pseudolabels—either by applying generalized majority vote or by learning the label model and performing inference—we can use the labels to train an end model.

Original: Tablesummarizes examples of problem types, explaining the underlying label set, the metric, theexponent c , the generalization of majority vote, the embedding space Im ( g ) , and an example of an end model.
Revised: Tablesummarizes examples of problem types, explaining the underlying label set, the metric, the generalization of majority vote, the embedding space Im ( g ) , and an example of an end model.

Original:  U NIVERSAL L ABEL M ODEL L EARNING
Revised:  U NIVERSAL L ABEL M ODEL L EARNING

Original: Now that we have a speciﬁcation of the label model distribution (1) (or its generalized form with per-source distances), we must learn the distribution from the observed LFs λ 1 , . . .
Revised: Now that we have a speciﬁcation of the label model distribution (1) (or its generalized form with per-source distances), we must learn the distribution from the observed LFs λ 1 , . . .

Original: , λ m .
Revised: , λ m .

Original: Afterwards, we can perform inference to compute ˆ y or p ( y | λ 1 , . . .
Revised: Afterwards, we can perform inference to compute ˆ y or p ( y | λ 1 , . . .

Original: , λ m ) or a related quantity, and use these to train a downstream model.
Revised: , λ m ) or a related quantity, and use these to train a downstream model.

Revised: A simpliﬁed model with an intuitive explanation for the isotropic Gaussian case is given in Appendix D.

Original: Learning the Universal Label Model Our general approach is described in Algorithm 1; its steps can be seen in the pipeline of Figure 1.
Revised: Learning the Universal Label Model Our general approach is described in Algorithm 1; its steps can be seen in the pipeline of Figure 1.

Original: It involves ﬁrst computing an embedding g ( λ a ) into {± 1 } d or R d ; we use multidimensional scaling into R d as our standard.
Revised: It involves ﬁrst computing an embedding g ( λ a ) into {± 1 } d or R d ; we use multidimensional scaling into R d as our standard.

Original: Next, we learn the per-source mean parameters , then ﬁnally compute the canonical parameters θ a .
Revised: Next, we learn the per-source mean parameters , then ﬁnally compute the canonical parameters θ a .

Original: The mean parameters are E [ d G ( g ( λ a ) g
Revised: The mean parameters are E [ d G ( g ( λ a ) g

Original: ( y )) c ] ; which reduce to moments like E [ g ( λ a ) i g ( y ) j ] .
Revised: ( y ))] ; which reduce to moments like E [ g ( λ a ) i g ( y ) i ] .

Revised: Here d G is some distance function associated with the embedding space.

Original: To estimate the mean parameters without observing y (and thus not knowing g ( y ) ), we exploit observable quantities and conditional independence.
Revised: To estimate the mean parameters without observing y (and thus not knowing g ( y ) ), we exploit observable quantities and conditional independence.

Original: As long as we can ﬁnd, for each LF, two others that are mutually conditionally independent, we can produce a simple non-linear system over the three sources (in each component of the moment).
Revised: As long as we can ﬁnd, for each LF, two others that are mutually conditionally independent, we can produce a simple non-linear system over the three sources (in each component of the moment).

Original: Solving this system recovers the mean parameters up to sign.
Revised: Solving this system recovers the mean parameters up to sign.

Original: We recover these as long as the LFs are better than random on average (see Ratner et al.
Revised: We recover these as long as the LFs are better than random on average (see Ratner et al.

Revised: Input: Output of labeling functions λ a ( i ) , correlation set E , prior p for Y , optionally embedding function g .

Revised: Embedding: If g is not given, use Multidimensional Scaling (MDS) to obtain embeddings g ( λ a ) ∈ R d ∀ a for a ∈ { 1 , 2 , . .

Revised: . , m } do

Revised: For b : ( a, b ) ∈ E Estimate Correlations: ∀ i, j , ˆ E (cid:2) g ( λ a ) i g ( λ b ) j (cid:3) =

Revised: 1 n (cid:80) nt =1 g ( λ a ( t ))

Original: The systems formed by the conditional independence relations differ based on whether g maps to the Boolean hypercube {± 1 } d or Euclidean space R d .
Revised: i g ( λ b ( t )) j Estimate Accuracy: Pick b, c : ( a, b ) (cid:54)∈ E, ( a, c ) (cid:54)∈ E, ( b, c ) (cid:54)∈ E . if Im ( g ) = {± 1 } d ∀ i , Estimate ˆ O a,b

Revised: = ˆ P ( g ( λ a ) i = 1 , g ( λ b ) i = 1) , ˆ O a,c , ˆ O b,c , Estimate (cid:96) a = ˆ P ( g ( λ a ) i = 1) , (cid:96) b , (cid:96) c

Revised: Accuracies ← Q UADRATIC T RIPLETS ( O a,b , O a,c , O b,c , (cid:96) a , (cid:96) b , (cid:96) c , p, i ) else ∀ i , Estimate ˆ

Revised: e a,b := ˆ E (cid:2) g ( λ a ) i g ( λ b ) i (cid:3) =

Revised: 1 n (cid:80) nt =1 g ( λ a ( t ))

Revised: i g ( λ b ( t ))

Revised: i , ˆ e a,c , ˆ e b,c

Revised: Accuracies ← C ONTINUOUS T RIPLETS (ˆ e a,b , ˆ e a,c , ˆ e b,c , p, i )

Revised: Recover accuracy signs (Fu et al., 2020) end for return ˆ θ a , ˆ θ a,b by running the backward mapping on accuracies

Revised: The systems formed by the conditional independence relations differ based on whether g maps to the Boolean hypercube {± 1 } d or Euclidean space R d .

Original: In the latter case we obtain a system that has a simple closed form solution, detailed in Algorithm 2.
Revised: In the latter case we obtain a system that has a simple closed form solution, detailed in Algorithm 2.

Original: In the discrete case (Algorithm 3, Appendix C), we need to use the quadratic formula to solve the system.
Revised: In the discrete case (Algorithm 4, Appendix E), we need to use the quadratic formula to solve the system.

Original: We require an estimate of the prior p on the label y ; there are techniques do so (Anandkumar et al., 2014; Ratner et al., 2019); we tacitly assume we have access to it.
Revised: We require an estimate of the prior p on the label y ; there are techniques do so (Anandkumar et al., 2014; Ratner et al., 2019); we tacitly assume we have access to it.

Original: The ﬁnal step uses the backward mapping from mean to canonical parameters (Wainwright & Jordan, 2008).
Revised: The ﬁnal step uses the backward mapping from mean to canonical parameters (Wainwright & Jordan, 2008).

Original: This approach is general, but it is easy in the R 2 embedding case: here, the canonical θ parameters are the inverse of the mean parameters.
Revised: This approach is general, but it is easy in special cases: for Gaussians the canonical θ parameters are the inverse of the mean parameters.

Original: Performing Inference: Maximum Likelihood Estimator Having estimated the label model canonical parameters ˆ θ a and ˆ θ a,b for all the sources, we use the maximum-likelihood estimator
Revised: Performing Inference: Maximum Likelihood Estimator Having estimated the label model canonical parameters ˆ θ a and ˆ θ a,b for all the sources, we use the maximum-likelihood estimator

Original: [Equation]
Revised: [Equation]

Original: Compare this approach to majority vote (2), observing that it can produce arbitrarily bad outcomes.
Revised: Compare this approach to majority vote (2), observing that MV can produce arbitrarily bad outcomes.

Original: For example, suppose that we have m LFs, one of which has a very high accuracy (e.g., large θ 1 ) and the others very low accuracy (e.g., θ 2 , θ 3 , . . .
Revised: For example, suppose that we have m LFs, one of which has a very high accuracy (e.g., large θ 1 ) and the others very low accuracy (e.g., θ 2 , θ 3 , . . .

Original: , θ m = 0) .
Revised: , θ m = 0) .

Original: Then, λ 1 is nearly always correct, while the other LFs are nearly random.
Revised: Then, λ 1 is nearly always correct, while the other LFs are nearly random.

Original: However, (2) weights them all equally, which will wash out the sole source of signal λ 1 .
Revised: However, (2) weights them all equally, which will wash out the sole source of signal λ 1 .

Original: On the other hand, (3) resolves the issue of equal weights on bad LFs by directly downweighting them.
Revised: On the other hand, (3) resolves the issue of equal weights on bad LFs by directly downweighting them.

Original: Simpliﬁcations While the above models can handle very general scenarios, special cases are dramatically simpler.
Revised: Simpliﬁcations While the above models can handle very general scenarios, special cases are dramatically simpler.

Original: In particular, in the case of isotropic Gaussian errors, where d θ a ( λ a , y ) = − θ a (cid:107) λ a − y (cid:107) 2 , there is no need to perform an embedding, since we can directly rely on empirical averages liken (cid:80) ni
Revised: In particular, in the case of isotropic Gaussian errors, where d θ a ( λ a , y ) = − θ a (cid:107) λ a − y (cid:107) 2 , there is no need to perform an embedding, since we can directly rely on empirical averages liken (cid:80) ni

Original: =

Revised: =

Original: d θ a ( λ a ( i ) , λ b ( i )) .
Revised: d θ a ( λ a ( i ) , λ b ( i )) .

Original: The continuous triplet step simpliﬁes to directly estimating the covariance entries ˆ θ − 1 a ; the backward map is simply inverting this.
Revised: The continuous triplet step simpliﬁes to directly estimating the covariance entries ˆ θ − 1 a ; the backward map is simply inverting this.

Original: Similarly, the inference procedure can be done entirely in the embedding step, even for non-isotropic Gaussians.
Revised: More details can be found in Appendix D.

Original:  T HEORETICAL A NALYSIS : E STIMATION E RROR & C ONSISTENCY
Revised:  T HEORETICAL A NALYSIS : E STIMATION E RROR & C ONSISTENCY

Original: We show that Algorithm 1 contains consistent estimators of the mean parameters under the condition that the embeddings are isometric.
Revised: We show that, under certain conditions, Algorithm 1 produces consistent estimators of the mean parameters.

Original: We provide convergence rates for the estimators.
Revised: We provide convergence rates for the estimators.

Original: As corollaries, we apply theseestimators to  rankings and regression, where isometric embeddings are available.
Revised: As corollaries, we apply these to the settings of rankings and regression, where isometric embeddings are available.

Original: Finally, we give a bound on the inconsistency due to embedding distortion in the non-isometric case.
Revised: Finally, we give a bound on the inconsistency due to embedding distortion in the non-isometric case.

Original: Boolean Hypercube Case We introduce the following ﬁnite-sample estimation error bound.
Revised: Boolean Hypercube Case We introduce the following ﬁnite-sample estimation error bound.

Original: It demonstrates consistency for mean parameter estimation for a triplet of LFs when using Algorithm 1.
Revised: It demonstrates consistency for mean parameter estimation for a triplet of LFs when using Algorithm 1.

Original: To keep our presentation simple, we assume a prior y over two values y ∈ { y 1 , y 2 } (i.e., there are two possible values ofy , but we do not know, for any sample, which of the two y i is). The result below treats the (assumed to be known) class balance P ( Y = y 1) as a constant and assumes that we can correctly recover signs.
Revised: To keep our presentation simple, we assume: (i) the class balance P ( Y = y ) are known, (ii) there are two possible values of Y , y 1 and y 2 , (iii) we can correctly recover signs (see Ratner et al. , (iv) we can ﬁnd at least three conditionally independent labeling functions that can form a triplet, (v) the embeddings are isometric.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For any δ > 0 , for some y 1 and y 2 with known class probabilities p = P ( Y = y 1 ) , the quadratic triplet method recovers α i
Revised: For any δ > 0 , for some y 1 and y 2 with known class probabilities p = P ( Y = y 1 ) , the quadratic triplet method recovers α i

Original: =
Revised: =

Original: P ( g ( λ a ) i = 1 | Y = y ) , β i = P ( g ( λ b ) i = 

Revised: P ( g ( λ a ) i = 1 | Y = y ) , β i = P ( g ( λ b ) i =

Original: | Y =
Revised: | Y =

Original: y ) , γ i =
Revised: y ) , γ i =

Original: P ( g ( λ c ) i = 1 | Y =
Revised: P ( g ( λ c ) i = 1 | Y =

Original: y )
Revised: y )

Original: up to error O
Revised: up to error O

Original: (( ln(2 m 2 /δ ) 2 n ) 1 / 4 ) with probability at least 1 − δ for all conditionally independent label functions λ a , λ b , λ c and for all i ∈ [ m ] .
Revised: (( ln(2 d 2 /δ ) 2 n ) 1 / 4 ) with probability at least 1 − δ for all conditionally independent label functions λ a , λ b , λ c and for all i ∈ [ d ] .

Original: Algorithm
Revised: Algorithm

Original: Input: Estimates ˆ e a,b , ˆ e a,c , ˆ e b,c , prior p , index
Revised: Input: Estimates ˆ e a,b , ˆ e a,c , ˆ e b,c , prior p , index

Original: i
Revised: i

Original: Obtain variance E (cid:2) g ( Y ) 2 i (cid:3) from prior p ˆ E [ g ( λ a ) g ( y )]
Revised: Obtain variance E (cid:2) g ( Y ) 2 i (cid:3) from prior p ˆ E [ g ( λ a ) g ( y )]

Original: ← (cid:112)
Revised: ← (cid:112)

Original: | ˆ e a,b | · | ˆ e a,c | · E [ Y 2 ] / | ˆ e b,c
Revised: | ˆ e a,b | · | ˆ e a,c | · E [ Y 2 ] / | ˆ e b,c

Original: | ˆ E [ g ( λ b ) g ( y )]
Revised: | ˆ E [ g ( λ b ) g ( y )]

Original: ← (cid:112)
Revised: ← (cid:112)

Original: | ˆ e a,b | · | ˆ e b,c | · E [ Y 2 ] / | ˆ e a,c | ˆ E [ g ( λ c ) g ( y )]
Revised: | ˆ e a,b | · | ˆ e b,c | · E [ Y 2 ] / | ˆ e a,c | ˆ E [ g ( λ c ) g ( y )]

Original: ← (cid:112)
Revised: ← (cid:112)

Original: | ˆ e a,c | · | ˆ e b,c | · E [ Y 2 ] / | ˆ e a,b | return Accuracies ˆ E [ g ( λ a ) g ( y )] , ˆ E [ g ( λ b ) g ( y )] , ˆ E [ g ( λ c ) g ( y )]
Revised: | ˆ e a,c | · | ˆ e b,c | · E [ Y 2 ] / | ˆ e a,b | return Accuracies ˆ E [ g ( λ a ) g ( y )] , ˆ E [ g ( λ b ) g ( y )] , ˆ E [ g ( λ c ) g ( y )]

Revised: We state our result via terms like α i =

Revised: P ( g ( λ a ) i | Y = y ) ; these can be used to obtain ˆ E [ g ( λ a ) i | y ] and so recover ˆ E [ d Y ( λ a , y )] .

Original: To showcase the power of this result, we apply it tothe case of ( S ρ , d τ ) , that is, rankings from the symmetric group S ρ equipped with the Kendall tau distance d τ (Kendall, 1938).
Revised: To showcase the power of this result, we apply it to rankings from the symmetric group S ρ equipped with the Kendall tau distance d τ (Kendall, 1938).

Original: The resulting estimation problem is more challenging than learning the conventional Mallowspermutation model (Mallows, 1957)—and the standard Kemeny rule (Kemeny, 1959) used for rank aggregation will fail if appied to it.
Revised: This estimation problem is more challenging than learning the conventional Mallows model (Mallows, 1957)—and the standard Kemeny rule (Kemeny, 1959) used for rank aggregation will fail if applied to it.

Original: Our result yields consistent estimation when coupled with the aggregation step (3).
Revised: Our result yields consistent estimation when coupled with the aggregation step (3).

Original: We use the isometric embedding g into {± 1 } ( ρ 2 ) :
Revised: We use the isometric embedding g into {± 1 } ( ρ 2 ) :

Original: Forsome π ∈ S ρ , each entry in g ( π ) corresponds to a pair ( i, j ) with i < j , and this entry is 1 if in π we have i < j and − 1 otherwise.
Revised: For π ∈ S ρ , each entry in g ( π ) corresponds to a pair ( i, j ) with i < j , and this entry is 1 if in π we have i < j and − 1 otherwise.

Original: We can showthat for π, γ ∈ S ρ, it holds that (cid:80) i
Revised: We can show for π, γ ∈ S ρ that (cid:80) i

Original: =1 g ( π ) i g ( γ ) i = (cid:0) ρ 2 (cid:1) − 2 d τ ( π, γ )  . 
Revised: =1 g ( π ) i g ( γ ) i = (cid:0) ρ 2 (cid:1) − 2 d τ ( π, γ ) , and so recover ˆ E [ g ( λ a ) i g ( y ) i ] , and thus ˆ E [ d τ ( λ a , y )] . Then,

Original: Corollary 4.1.1.
Revised: Corollary 4.1.1.

Original: For any δ > 0 , U > 0 ,  prior over y 1 and y 2 with known class probability p , and using Algorithm 1 and Algorithm 3, for all conditionally independent  λ a , λ b , λ c , with parameters
Revised: For any δ > 0 , U > 0 , a prior over y 1 and y 2 with known class probability p , and using Algorithm 1 and Algorithm 4, for any conditionally independent triplet λ a , λ b , λ c , with parameters

Original: U > θ a , θ b , θ c > 4 ln(2) , we can recover θ a , θ b , θ c up to error O ( ρ 2 (ln(2 m 2 ) / (2 δn )) 1 / 4/ | g 2 ( U ) | ) with probability at least 1 − δ , where g 2 ( U ) =
Revised: U > θ a , θ b , θ c > 4 ln(2) , we can recover θ a , θ b , θ c up to error O ( g − 1 2 ( θ + (log(2 ρ 2 ) / (2 δn )) 1 / 4 ) − θ ) with probability at least 1 − δ , where g 2 ( U ) =

Original: ( − ke − U ) /
Revised: ( − ρe − U ) /

Original: ((1 − e − U ) 2 )
Revised: ((1 − e − U ) 2 )

Original: + (cid:80) 2 j =1{+ρj =
+} ( j 2 e − Uj ) /
Revised: + (cid:80) {+ρj =
+} ( j 2 e − Uj ) /

Original: ((1 − e − Uj ) 2 ) .
Revised: ((1 − e − Uj ) 2 ) .

Original: Note that the error goes to zero as O ( n − 1 / 4 ) .
Revised: Note that the error goes to zero as O ( n − 1 / 4 ) .

Original: This is an outcome of using the quadratic system, which is needed for generality.
Revised: This is an outcome of using the quadratic system, which is needed for generality.

Original: In the easier cases, where the quadratic approach is not needed (including the regression case), the rate is O ( n − 1 / 2 ) .
Revised: In the easier cases, where the quadratic approach is not needed (including the regression case), the rate is O ( n − 1 / 2 ) .

Original: Next, note that the scaling in terms of the number of LFs m is
Revised: Next, note that the scaling in terms of the embedding dimension d is

Original: O ((ln m ) 1 / 4 ) —this is efﬁcient.
Revised: O ((log( d )) 1 / 4 ) —this is efﬁcient.

Original: The g 2 function captures the cost of the backward mapping.
Revised: The g 2 function captures the cost of the backward mapping.

Original: Euclidean Space R d Case The following is an estimation error bound for the continuous triplets method.
Revised: Euclidean Space R d Case The following is an estimation error bound for the continuous triplets method in regression, where we use the squared Euclidean distance.

Revised: The result is for d = 1 but can be easily extended.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Let ˆ E [ g ( λ a ) g ( y )] be an estimate of the accuracies
Revised: Let ˆ E [ g ( λ a ) g ( y )] be an estimate of the accuracies

Original: E
Revised: E

Original: [ g ( λ a ) g ( y )] usingˆ e computed from n samples, where all LFs are conditionally independent given Y .
Revised: [ g ( λ a ) g ( y )] using n samples, where all LFs are conditionally independent given Y .

Original: Ifwe assume the signs of a are recoverable, then with high probability
Revised: If the signs of a are recoverable, then with high probability

Original: [Equation]
Revised: [Equation]

Original: Here, a | min | = min i | E [ g ( λ i ) g ( y )] | and e max = max j,k e j,k .
Revised: Here, a | min | = min i | E (cid:2) g ( λ i ) g ( y ) (cid:3) | and e max = max j,k e j,k .

Original: The error tends to zero with rate O ( n − 1 / 2 ) as expected.
Revised: The error tends to zero with rate O ( n − 1 / 2 ) as expected.

Original: Distortion & Inconsistency Bound Next, we show how to control the inconsistency in parameter estimation as a function of the distortion.
Revised: Distortion & Inconsistency Bound Next, we show how to control the inconsistency in parameter estimation as a function of the distortion.

Original: We write θ to be the vector of the canonical parameters, θ (cid:48) for their distorted counterparts (obtained with a consistent estimator on the embedded distances), and µ, µ (cid:48) be the corresponding mean parameters.
Revised: We write θ to be the vector of the canonical parameters, θ (cid:48) for their distorted counterparts (obtained with a consistent estimator on the embedded distances), and µ, µ (cid:48) be the corresponding mean parameters.

Original: Let 1 − ε ≤ d g ( g ( y ) , g ( y (cid:48) ))
Revised: Let 1 − ε ≤ d g ( g ( y ) , g ( y (cid:48) ))

Original: /d
Revised: /d

Original: Y ( y, y (cid:48) ) ≤ 1 for all y, y (cid:48)
Revised: Y ( y, y (cid:48) ) ≤ 1 for all y, y (cid:48)

Original: ∈
Revised: ∈

Original: Y × Y .
Revised: Y × Y .

Original: Then, for a constant e min (the value of which we characterize in the Appendix).
Revised: Then, for a constant e min (the value of which we characterize in the Appendix).

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: The inconsistency in estimating θ is bounded as (cid:107) θ − θ (cid:48)
Revised: The inconsistency in estimating θ is bounded as (cid:107) θ − θ (cid:48)

Original: (cid:107) ≤ ε (cid:107) µ (cid:107) /e
Revised: (cid:107) ≤ ε (cid:107) µ (cid:107) /e

Original: min .
Revised: min .

Original: We note that we can further control ε in the worst case; operating over triplets, we need only embed threepoint metric spaces at a time, yielding ε ≈ 1 − 1 / (log 3) ≤ 0 .

Original: 1 (Bourgain, 1985).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluated our universal approach with four sample applications, all new to WS: learning to rank, regression, learning in hyperbolic space, and estimation in generic metric spaces given by random graphs.[Figure] [Figure]
Revised: We evaluated our universal approach with four sample applications, all new to WS: learning to rank, regression, learning in hyperbolic space, and estimation in generic metric spaces given by random graphs.

Original: Our hypothesis is that the universal approach is capable of learning a label model and producing high-quality pseudolabels with the following properties:
Revised: Our hypothesis is that the universal approach is capable of learning a label model and producing high-quality pseudolabels with the following properties:

Original: • The quality of the pseudolabels is a function of the number and quality of the available sources, with more high-quality, independent sources yielding greater pseudolabel quality, • Despite pseudolabels being noisy, an end model trained on more pseudolabels can perform as well or better than a fully supervised model trained on fewer true labels, • The label model and inference procedure (3) improves on the majority vote equivalent—but only when LFs are of varying quality; for LFs of similar quality, the two will have similar performance.
Revised: • The quality of the pseudolabels is a function of the number and quality of the available sources, with more high-quality, independent sources yielding greater pseudolabel quality, • Despite pseudolabels being noisy, an end model trained on more pseudolabels can perform as well or better than a fully supervised model trained on fewer true labels, • The label model and inference procedure (3) improves on the majority vote equivalent—but only when LFs are of varying quality; for LFs of similar quality, the two will have similar performance.

Original: Additionally, we expect to improve on naive applications of existing approaches that do not take structure into account (such as using Snorkel (Ratner et al., 2018) by mapping permutations to integer classes).
Revised: Additionally, we expect to improve on naive applications of existing approaches that do not take structure into account (such as using Snorkel (Ratner et al., 2018) by mapping permutations to integer classes).

Original: Application I: Rankings We applied our approach to obtain pseudorankings to train a downstream ranking model.
Revised: Application I: Rankings We applied our approach to obtain pseudorankings to train a downstream ranking model.

Original: We hypothesize that given enough signal, the produced pseudorankings can train a higher-quality model than using a smaller proportion of true labels.
Revised: We hypothesize that given enough signal, the produced pseudorankings can train a higher-quality model than using a smaller proportion of true labels.

Original: We expect our method produces better performance than the Snorkel baseline where permutations are converted into multi-class classiﬁcation.
Revised: We expect our method produces better performance than the Snorkel baseline where permutations are converted into multi-class classiﬁcation.

Original: We also anticipate that our inference procedure (3) improves on the MV baseline (2) when LFs have differing accuracies.
Revised: We also anticipate that our inference procedure (3) improves on the MV baseline (2) when LFs have differing accuracies.

Original: Approach, Datasets, Labeling Functions, and End Model We used the isotropic simpliﬁcation of Algorithm 1.
Revised: Approach, Datasets, Labeling Functions, and End Model We used the isotropic simpliﬁcation of Algorithm 1.

Original: For inference, we applied (3).
Revised: For inference, we applied (3).

Original: We compared against baseline fully-supervised models with only a proportion of the true labels (e.g., 20%, 50%, ...), the Snorkel framework (Ratner et al., 2018) converting rankings into classes, and majority vote (2).
Revised: We compared against baseline fully-supervised models with only a proportion of the true labels (e.g., 20%, 50%, ...), the Snorkel framework (Ratner et al., 2018) converting rankings into classes, and majority vote (2).

Original: We used real-world datasets compatible with multiple label types, including a movies dataset and the BoardGameGeek dataset (2020) (BGG), along with synthetic data.
Revised: We used real-world datasets compatible with multiple label types, including a movies dataset and the BoardGameGeek dataset (2020) (BGG), along with synthetic data.

Original: For our movies dataset, we combined IMDb, TMDb, Rotten Tomatoes, and MovieLens movie review data to obtain features and weak labels.
Revised: For our movies dataset, we combined IMDb, TMDb, Rotten Tomatoes, and MovieLens movie review data to obtain features and weak labels.

Original: In Movies dataset, rankings were generated by picking d = 5 ﬁlm items and producing a ranking based on their tMDb average rating.
Revised: In Movies dataset, rankings were generated by picking d = 5 ﬁlm items and producing a ranking based on their tMDb average rating.

Original: In BGG, we used the available rankings.
Revised: In BGG, we used the available rankings.

Original: We created both real and simulated LFs.
Revised: We created both real and simulated LFs.

Original: For simulated LFs, we sampled 1/3 of LFs from less noisy Mallows model, 2/3 of LFs from very noisy Mallows model.
Revised: For simulated LFs, we sampled 1/3 of LFs from less noisy Mallows model, 2/3 of LFs from very noisy Mallows model.

Original: Details are in the Appendix F. For real LFs, we built labeling functions using external KBs as WS sources based on alternative movie ratings along with popularity, revenue, and vote count-based LFs.
Revised: Details are in the Appendix H. For real LFs, we built labeling functions using external KBs as WS sources based on alternative movie ratings along with popularity, revenue, and vote count-based LFs.

Original: For the end model, we used PTRanking (Yu, 2020), with ListMLE (Xia et al., 2008) loss.
Revised: For the end model, we used PTRanking (Yu, 2020), with ListMLE (Xia et al., 2008) loss.

Original: We report the Kendall tau distance ( d τ ) .
Revised: We report the Kendall tau distance ( d τ ) .

Original:  Figure 2 reports end model performance for the two datasets with varying numbers of simulated LFs.
Revised: Results Figure 2 reports end model performance for the two datasets with varying numbers of simulated LFs.

Original: We observe that (i) as few as 12 LFs are sufﬁcient to improve on a fully-supervised model trained on less data (as much as 20% of the dataset) and that (ii) as more LFs are added, and signal improves, performance also improves—as expected.
Revised: We observe that (i) as few as 12 LFs are sufﬁcient to improve on a fully-supervised model trained on less data (as much as 20% of the dataset) and that (ii) as more LFs are added, and signal improves, performance also improves—as expected.

Original: Crucially, the Snorkel baseline, where rankings are mapped into classes, cannot[Table] perform as well as the universal approach; it is not meant to be effective general label settings.
Revised: Crucially, the Snorkel baseline, where rankings are mapped into classes, cannot perform as well as the universal approach; it is not meant to be effective general label settings.

Original: Tableshows the results when using real LFs, some good, some bad, constructed from alternative ratings and simple heuristics.
Revised: Tableshows the results when using real LFs, some good, some bad, constructed from alternative ratings and simple heuristics.

Original: Alternative ratings are quite accurate:
Revised: Alternative ratings are quite accurate:

Original: MV and our method perform similarly.
Revised: MV and our method perform similarly.

Original: However, when poorer-quality LFs are added, MV rule tends to degrade more than our proposed model, as we anticipated [Table]
Revised: However, when poorer-quality LFs are added, MV rule tends to degrade more than our proposed model, as we anticipated.

Original: Application II: Regression We used universal WS in the regression setting.
Revised: Application II: Regression We used universal WS in the regression setting.

Original: We expect that with more
Revised: We expect that with more

Original: LFs, we can obtain increasingly high-quality pseudolabels, eventually matching fully-supervised baselines.
Revised: LFs, we can obtain increasingly high-quality pseudolabels, eventually matching fully-supervised baselines.

Original: Approach, Datasets, Labeling Functions, End Model, Results We used Algorithm 1, which uses the continuous triplets approach 2.
Revised: Approach, Datasets, Labeling Functions, End Model, Results We used Algorithm 1, which uses the continuous triplets approach 2.

Original: For inference, we used the Gaussians simpliﬁcation.
Revised: For inference, we used the Gaussians simpliﬁcation.

Original: As before, we compared against baseline fully-supervised models with a fraction of the true labels (e.g., 20%, 50%, ...) and MV (2).
Revised: As before, we compared [Figure] [Figure] [Figure]

Revised: d τ

Revised: MSE

Revised: Setting d τ

Revised: MSE

Revised: Fully supervised (10%) 0.

Revised: WS (One LF

Revised: [Figure]

Revised: 0.4272 Fully supervised (25%) 0.

Revised: WS (One LF

Revised: [Figure]

Revised: IMDb score

Revised: [Figure]

Revised: 0.2990 Fully supervised (50%) 0.

Revised: WS (One LF

Revised: [Figure]

Revised: MovieLens score) 0.

Revised: 0.2690 Fully supervised (100%) 0.

Revised: WS (3 LFs

Revised: [Table]

Revised: against baseline fully-supervised models with a fraction of the true labels (e.g., 20%, 50%, ...) and MV (2).

Original: We used the Movies rating datasets with the label being the average rating of TMDb review scores across users and the BGG dataset with the label being the average rating of board games.
Revised: We used the Movies rating datasets with the label being the average rating of TMDb review scores across users and the BGG dataset with the label being the average rating of board games.

Original: We split data into 75% for training set, and 25% for the test set.
Revised: We split data into 75% for training set, and 25% for the test set.

Original: For real-world LFs, we used other movie ratings in the movies dataset.
Revised: For real-world LFs, we used other movie ratings in the movies dataset.

Original: Details are in the Appendix F for our synthetic LF generation procedure.
Revised: Details are in the Appendix H for our synthetic LF generation procedure.

Original: For the end model, we used gradient boosting (Friedman, 2001).
Revised: For the end model, we used gradient boosting (Friedman, 2001).

Original: The performance metric is MSE.
Revised: The performance metric is MSE.

Original: Figure 9 (Appendix) shows the end model performance with WS compared to fully supervised on the movie reviews and board game reviews datasets.
Revised: Figure ?? (Appendix) shows the end model performance with WS compared to fully supervised on the movie reviews and board game reviews datasets.

Original: We also show parameter estimation error in Figure 3.
Revised: We also show parameter estimation error in Figure 3.

Original: As expected, our parameter estimation error goes down in the amount of available data.
Revised: As expected, our parameter estimation error goes down in the amount of available data.

Original: Similarly, our label estimator is consistent, while majority vote is not.
Revised: Similarly, our label estimator is consistent, while majority vote is not.

Original: Table 2 also reports the result of using real LFs for movies.
Revised: Table 2 also reports the result of using real LFs for movies.

Original: Here, MV shows even worse performance than the best individual LF - Movie Lens Score.
Revised: Here, MV shows even worse performance than the best individual LF - Movie Lens Score.

Original: On the other side, our label model lower MSE than the best individual LF, giving similar performance with fully supervised learning with 50% training data.
Revised: On the other side, our label model lower MSE than the best individual LF, giving similar performance with fully supervised learning with 50% training data.

Original: Application III: Geodesic Regression in Hyperbolic Space
Revised: Application III: Geodesic Regression in Hyperbolic Space

Original: Next, we evaluate our approach on the problem of geodesic regression on a Riemannian manifold M , speciﬁcally, a hyperbolic space with curvature
Revised: Next, we evaluate our approach on the problem of geodesic regression on a Riemannian manifold M , speciﬁcally, a hyperbolic space with curvature

Original: K = − 50 .
Revised: K = − 50 .

Original: The goal of this task is analogous to Euclidean linear regression, except the dependent variable lies on a geodesic in hyperbolic space.
Revised: The goal of this task is analogous to Euclidean linear regression, except the dependent variable lies on a geodesic in hyperbolic space.

Original: Further background is in the Appendix F.
Revised: Further background is in the Appendix H.

Original: Approach, Datasets, LFs, Results We generate y ∗ i by taking points along a geodesic (a generalization of a line) parametrized by a tangent vector β starting a p , further affecting them by noise.
Revised: Approach, Datasets, LFs, Results We generate y ∗ i by taking points along a geodesic (a generalization of a line) parametrized by a tangent vector β starting a p , further affecting them by noise.

Original: The objective of the end-model is to recover parameters p and β , which is done using Riemannian gradient descent (RGD) to minimize the least-squares objective: ˆ p, ˆ β = arg min q,α (cid:80) ni =1 d (exp q ( x i α ) , y i ) 2 where d ( · , · ) is the hyperbolic distance.
Revised: The objective of the end-model is to recover parameters p and β , which is done using Riemannian gradient descent (RGD) to minimize the least-squares objective: ˆ p, ˆ β = arg min q,α (cid:80) ni =1 d (exp q ( x i α ) , y i ) 2 where d ( · , · ) is the hyperbolic distance.

Original: To generate each LF λ j , we use noisier estimates of y ∗ i , where the distribution is
Revised: To generate each LF λ j , we use noisier estimates of y ∗ i , where the distribution is

Original: N (0 , σ 2 j ) and σ 2 j ∼ U [1 . 5+(

Revised: N (0 , σ 2 j ) and σ 2 j ∼ U [1 . 5+(

Original: z j ) , 4 .
Revised: z j ) , 4 .

Original: 5+(

Revised: 5+(

Original: z j )] , z j ∼ Bernoulli (0 . 5) to simulate heterogeneity across LFs;  the noise vectors are parallel transported to the appropriate space.
Revised: z j )] , z j ∼ Bernoulli (0 . 5) to simulate heterogeneity across LFs; [Figure] [Figure] the noise vectors are parallel transported to the appropriate space.

Original: For label model learning, we used the isotropic Gaussian simpliﬁcation.
Revised: For label model learning, we used the isotropic Gaussian simpliﬁcation.

Original: Finally, inference used RGD to compute (3).
Revised: Finally, inference used RGD to compute (3).

Original: We include MV (2) as a[Figure] [Figure] baseline.
Revised: We include MV (2) as a baseline.

Original: We compare fully supervised end models each with n ∈ { 80 , 90 , 100 } labeled examples to weak supervision using only weak labels.
Revised: We compare fully supervised end models each with n ∈ { 80 , 90 , 100 } labeled examples to weak supervision using only weak labels.

Original: In Figure 4 we see that the Fr´echet mean baseline and our approach both outperform fully supervised geodesic regression despite the total lack of labeled examples.
Revised: In Figure 4 we see that the Fr´echet mean baseline and our approach both outperform fully supervised geodesic regression despite the total lack of labeled examples.

Original: Intuitively, this is because with multiple noisier labels, we can produce a better pseudolabel than a single (but less noisy) label.
Revised: Intuitively, this is because with multiple noisier labels, we can produce a better pseudolabel than a single (but less noisy) label.

Original: As expected, our approach yields consistently lower test loss than the Fr´echet mean baseline.
Revised: As expected, our approach yields consistently lower test loss than the Fr ´ echet mean baseline.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Application IV:
Revised: Application IV:

Original: Generic Metric Spaces We also evaluated our approach on a structureless problem— a generic metric space generated by selecting random graphs G with a ﬁxed number of nodes and edges.
Revised: Generic Metric Spaces We also evaluated our approach on a structureless problem— a generic metric space generated by selecting random graphs G with a ﬁxed number of nodes and edges.

Original: The metric is the shortest-hop distance between a pair of nodes.
Revised: The metric is the shortest-hop distance between a pair of nodes.

Original: We sampled nodes uniformly and obtain LF values with (1).
Revised: We sampled nodes uniformly and obtain LF values with (1).

Original: Despite the lack of structure, we still anticipate that our approach will succeed in recovering the latent nodes y when LFs have sufﬁciently high quality.
Revised: Despite the lack of structure, we still anticipate that our approach will succeed in recovering the latent nodes y when LFs have sufﬁciently high quality.

Original: We expect that the LM improves on MV (2) when LFs have heterogeneous quality, while the two will have similar performance on homogeneous quality LFs.
Revised: We expect that the LM improves on MV (2) when LFs have heterogeneous quality, while the two will have similar performance on homogeneous quality LFs.

Original: Approach, Datasets, LFs, Results We generated random graphs and computed the distance matrix yielding the metric.
Revised: Approach, Datasets, LFs, Results We generated random graphs and computed the distance matrix yielding the metric.

Original: We used Algorithm (1) with isotropic Gaussian embedding and continuous triplets.
Revised: We used Algorithm (1) with isotropic Gaussian embedding and continuous triplets.

Original: For label model inference, we used (3).
Revised: For label model inference, we used (3).

Original: Figure 5 shows results on our generic metric space experiment.
Revised: Figure 5 shows results on our generic metric space experiment.

Original: As expected, when LFs have a heterogeneous quality, LM yield better accuracy than MV.
Revised: As expected, when LFs have a heterogeneous quality, LM yield better accuracy than MV.

Original: However, when labeling functions are of similar quality, LM and MV gives a similar accuracy.
Revised: However, when labeling functions are of similar quality, LM and MV gives a similar accuracy.

Original: Application V: Semantic Dependency Parsing
Revised: Application V: Semantic Dependency Parsing

Original: We ran our technique on semantic dependency parsing tasks, using datasets in English and Czech from the Universal Dependencies collection (Nivre et al., 2020).
Revised: We ran our technique on semantic dependency parsing tasks, using datasets in English and Czech from the Universal Dependencies collection (Nivre et al., 2020).

Original: The LFs are off-the-shelf parsers from Stanza (Qi et al., 2020) trained on different datasets in the same language.
Revised: The LFs are off-the-shelf parsers from Stanza (Qi et al., 2020) trained on different datasets in the same language.

Original: We model a space of trees with a metric given by the (cid:96) 2 norm on the difference between the adjacency matrices.
Revised: We model a space of trees with a metric given by the (cid:96) 2 norm on the difference between the adjacency matrices.

Original: We measure the quality of the synthesized tree ˆ Y with the unlabeled attachment scores (UAS).
Revised: We measure the quality of the synthesized tree ˆ Y with the unlabeled attachment scores (UAS).

Original: Our results are shown in Table 3.
Revised: Our results are shown in Table 3.

Original: As expected, when the parsers are of different quality, we can obtain a better result.
Revised: As expected, when the parsers are of different quality, we can obtain a better result.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Weak supervision approaches allow users to overcome the manual labeling bottleneck for dataset construction.
Revised: Weak supervision approaches allow users to overcome the manual labeling bottleneck for dataset construction.

Original: While successful, such methods do not apply to each potential label type.
Revised: While successful, such methods do not apply to each potential label type.

Original: We proposed an approach to universally apply WS, demonstrating it for three applications new to WS: rankings, regression, and learning in hyperbolic space.
Revised: We proposed an approach to universally apply WS, demonstrating it for three applications new to WS: rankings, regression, and learning in hyperbolic space.

Original: We hope our proposed technique encourages applying WS to many more applications.
Revised: We hope our proposed technique encourages applying WS to many more applications.

Original: E THICS S TATEMENT

Original: While our proposed approaches to universal weak supervision are algorithmic and foundational in nature we note that the resulting tool could be used to inject bias into datasets.

Original: Indeed, the weak supervision concept replaces hand-labeling (which comes with its own biases) with sources that can be automatically and broadly applied.

Original: This could amplify the biases in dataset labels.

Original: We believe that special care must be taken to prevent such ampliﬁcation.

Original: This will require further study and effort and we believe that all users of our proposed system should invest into checking sources and resulting datasets.

Original: One important mitigation strategy that is easily built into the proposed weak supervision system is that once a source is identiﬁed as containing unacceptable biases, its accuracy can be easily downgraded to zero ensuring that it is not relied on in the resulting dataset

Original: A G LOSSARY
Revised: A G LOSSARY

Original: The glossary is given in Table 4 below.
Revised: The glossary is given in Table 4 below.

Original: Symbol
Revised: Symbol

Original: Deﬁnition
Revised: Deﬁnition

Original: X feature space
Revised: X feature space

Original: Y label metric space
Revised: Y label metric space

Original: d Y label metric x 1 , x 2 , . .
Revised: d Y label metric x 1 , x 2 , . .

Original: , x n unlabeled datapoints from X y 1 , y 2 , . . .
Revised: , x n unlabeled datapoints from X y 1 , y 2 , . . .

Original: , y n latent (unobserved) labels from Y s 1 , s 2 , . . .
Revised: , y n latent (unobserved) labels from Y s 1 , s 2 , . . .

Original: , s m labeling functions / sources λ 1 , λ 2 , . . .
Revised: , s m labeling functions / sources λ 1 , λ 2 , . . .

Original: , λ m output of labeling functions n number of data points m number of LFs λ a ( i ) output of a th labeling function applied to i
Revised: , λ m output of labeling functions n number of data points m number of LFs λ a ( i ) output of a th labeling function applied to i

Original: th sample x i θ a , θ a,b canonical parameters in model (1) E
Revised: th sample x i θ a , θ a,b canonical parameters in model (1) E

Original: [ d a ( λ a , y )] , E [ d a ( λ a , λ b )] mean parameters in (1) g injective mapping g : Y → R d or {± 1 } d ρ number of items in ranking setting e a,b
Revised: [ d a ( λ a , y )] , E [ d a ( λ a , λ b )] mean parameters in (1) g injective mapping g : Y → R d or {± 1 } d ρ number of items in ranking setting e a,b

Original: E [ g ( λ a ) i g ( λ b ) i ]
Revised: E [ g ( λ a ) i g ( λ b ) i ]

Original: O a,b
Revised: O a,b

Original: P ( g ( λ a ) i = 1 , g ( λ b ) i = 1) l a
Revised: P ( g ( λ a ) i = 1 , g ( λ b ) i = 1) l a

Original: P ( g ( λ a ) i = 1)
Revised: P ( g ( λ a ) i = 1)

Original: S ρ symmetric group on { 1 , ..., ρ } π permutation ∈ S ρ d τ ( · , · )
Revised: S ρ symmetric group on { 1 , ..., ρ } π permutation ∈ S ρ d τ ( · , · )

Original: Kendall tau distance on permutations (Kendall, 1938)
Revised: Kendall tau distance on permutations (Kendall, 1938)

Original: B R ELATED W ORK
Revised: B R ELATED W ORK

Original: Weak Supervision
Revised: Weak Supervision

Original: Existing weak supervision frameworks, starting with Ratner et al.
Revised: Existing weak supervision frameworks, starting with Ratner et al.

Original: (2016), select a particular model for the joint distribution among the sources and the latent true label, and then use the properties of the distribution to select an algorithm to learn the parameters.
Revised: (2016), select a particular model for the joint distribution among the sources and the latent true label, and then use the properties of the distribution to select an algorithm to learn the parameters.

Original: In Ratner et al.
Revised: In Ratner et al.

Original: (2016), a factor model is used and a Gibbs-sampling based optimizer is the algorithm of choice.
Revised: (2016), a factor model is used and a Gibbs-sampling based optimizer is the algorithm of choice.

Original: In Ratner et al.
Revised: In Ratner et al.

Original: , the model is a discrete Markov random ﬁeld (MRF), and in particular, an Ising model.
Revised: , the model is a discrete Markov random ﬁeld (MRF), and in particular, an Ising model.

Original: The algorithm used to learn the parameters solves a linear system obtained from a component of the inverse covariance matrix.
Revised: The algorithm used to learn the parameters solves a linear system obtained from a component of the inverse covariance matrix.

Original: In
Revised: In

Original: Sala et al.
Revised: Sala et al.

Original: and Fu et al.
Revised: and Fu et al.

Original: (2020), the requirement to use the inverse covariance matrix is removed, and a set of systems among as few as three sources are used instead.
Revised: (2020), the requirement to use the inverse covariance matrix is removed, and a set of systems among as few as three sources are used instead.

Original: These systems have closed-form solutions.
Revised: These systems have closed-form solutions.

Original: All of these models could be represented within our framework.
Revised: All of these models could be represented within our framework.

Original: The quadratic triplets idea is described, but not analyzed, in the appendix of Fu et al.
Revised: The quadratic triplets idea is described, but not analyzed, in the appendix of Fu et al.

Original: (2020), applied to just the particular case of Ising models with singleton potentials.
Revised: (2020), applied to just the particular case of Ising models with singleton potentials.

Original: Our work uses these ideas and their extensions to the general setting, and provides theoretical analyses for the two sample applications we are interested in.
Revised: Our work uses these ideas and their extensions to the general setting, and provides theoretical analyses for the two sample applications we are interested in.

Original: All of these papers refer to weak supervision frameworks; all of them permit the use of various types of information as labeling functions.
Revised: All of these papers refer to weak supervision frameworks; all of them permit the use of various types of information as labeling functions.

Original: For example, labeling functions might include crowdworkers Karger et al.
Revised: For example, labeling functions might include crowdworkers Karger et al.

Original: (2011), distant supervision Mintz et al.
Revised: (2011), distant supervision Mintz et al.

Original: (2009), co-training Blum & Mitchell (1998), and many others.
Revised: (2009), co-training Blum & Mitchell (1998), and many others.

Original: Note that works like Dehghani et al.
Revised: Note that works like Dehghani et al.

Original: (2017) provide one type of weak supervision signal for rankings and can be used as a high-quality labeling function in our framework for the rankings case.
Revised: (2017) provide one type of weak supervision signal for rankings and can be used as a high-quality labeling function in our framework for the rankings case.

Original: That is, our work can integrate such sources, but does not directly compete with them.
Revised: That is, our work can integrate such sources, but does not directly compete with them.

Original: The idea of learning notions like source accuracy and correlations, despite the presence of the latent label, is central to weak supervision.
Revised: The idea of learning notions like source accuracy and correlations, despite the presence of the latent label, is central to weak supervision.

Original: It has been shown up in other problems as well, such as crowdsourcing Dawid & Skene (1979); Karger et al.
Revised: It has been shown up in other problems as well, such as crowdsourcing Dawid & Skene (1979); Karger et al.

Original: (2011) or topic modeling Anandkumar et al.
Revised: (2011) or topic modeling Anandkumar et al.

Original: Early approaches use expectation maximization (EM), but in the last decade, a number of exciting approaches have been introduced based on the method of moments.
Revised: Early approaches use expectation maximization (EM), but in the last decade, a number of exciting approaches have been introduced based on the method of moments.

Original: These include the tensor power method approach of Anandkumar et al.
Revised: These include the tensor power method approach of Anandkumar et al.

Original: (2014) and its follow-on work, the explicitly crowdsourcing setting of Joglekar et al.
Revised: (2014) and its follow-on work, the explicitly crowdsourcing setting of Joglekar et al.

Original: (2013; 2015), and the graphical model learning procedures of Chaganty & Liang (2014); Raghunathan et al.
Revised: (2013; 2015), and the graphical model learning procedures of Chaganty & Liang (2014); Raghunathan et al.

Original: Our approach can be thought of as an extension of some of these approaches to the more general setting we consider.
Revised: Our approach can be thought of as an extension of some of these approaches to the more general setting we consider.

Original: Comparison With Existing Label Models There are several existing label models; these closely resemble (1) under particular specializations.
Revised: Comparison With Existing Label Models There are several existing label models; these closely resemble (1) under particular specializations.

Original: For example, one of the models used for binary labels in Ratner et al.
Revised: For example, one of the models used for binary labels in Ratner et al.

Original: ; Varma et al.
Revised: ; Varma et al.

Original: (2019) is the Ising model for λ 1 , . . .
Revised: (2019) is the Ising model for λ 1 , . . .

Original: , y ∈ {± 1 }
Revised: , y ∈ {± 1 }

Original: [Equation]
Revised: [Equation]

Original: A difference is that this is a joint model; it assumes y is part of the model and adds a singleton potential prior term to it.
Revised: A difference is that this is a joint model; it assumes y is part of the model and adds a singleton potential prior term to it.

Original: Note that this model promotes agreement λ a y rather than penalizes disagreement − λ a d Y
Revised: Note that this model promotes agreement λ a y rather than penalizes disagreement − λ a d Y

Original: ( λ a , y )c , but this is nearly equivalent.
Revised: ( λ a , y ) , but this is nearly equivalent.

Original: Rankings and Permutations One of our chief applications is to rankings.
Revised: Rankings and Permutations One of our chief applications is to rankings.

Original: There are several classes of distributions over permutations, including the Mallows model (Mallows, 1957) and the Plackett-Luce model (Plackett, 1975).
Revised: There are several classes of distributions over permutations, including the Mallows model (Mallows, 1957) and the Plackett-Luce model (Plackett, 1975).

Original: We are particularly concerned with the Mallows model in this work, as it can be extended naturally to include labeling functions and the latent true label.
Revised: We are particularly concerned with the Mallows model in this work, as it can be extended naturally to include labeling functions and the latent true label.

Original: Other generalizations include the generalized Mallows model (Marden, 2014) and the Mallows block model (Busa-Fekete et al., 2019).
Revised: Other generalizations include the generalized Mallows model (Marden, 2014) and the Mallows block model (Busa-Fekete et al., 2019).

Original: These generalizations, however, do not cover the heterogenous accuracy setting we are interested in.
Revised: These generalizations, however, do not cover the heterogenous accuracy setting we are interested in.

Original: A common goal is to learn the parameters of the model (the single parameter θ in the conventional Mallows model) and then to learn the central permutation that the samples are drawn from.
Revised: A common goal is to learn the parameters of the model (the single parameter θ in the conventional Mallows model) and then to learn the central permutation that the samples are drawn from.

Original: A number of works in this direction include Caragiannis et al.
Revised: A number of works in this direction include Caragiannis et al.

Original: ; Mukherjee (2016); Busa-Fekete et al.
Revised: ; Mukherjee (2016); Busa-Fekete et al.

Original: Our work extends results of this ﬂavor to the generalization on the Mallows case that we consider.
Revised: Our work extends results of this ﬂavor to the generalization on the Mallows case that we consider.

Original: In order to learn the center permutation, estimators like the Kemeny rule (the procedure we generalize in this work) are used.
Revised: In order to learn the center permutation, estimators like the Kemeny rule (the procedure we generalize in this work) are used.

Original: Studies of the Kemeny rule include Kenyon-Mathieu & Schudy (2007); Caragiannis et al.
Revised: Studies of the Kemeny rule include Kenyon-Mathieu & Schudy (2007); Caragiannis et al.

Original: C E XTENDED A LGORITHMIC D ETAILS
Revised: C E XTENSION TO MIXED LABEL TYPES

Revised: In the body of the paper, we discussed models for tasks where only one label type is considered.

Revised: As a simple extension, we can operate with multiple label types.

Revised: For example, this might include a classiﬁcation task where weak label sources give their output as class labels and also might provide conﬁdence values; that is, a pair of label types that include a discrete value and a continuous value.

Revised: The main idea is to construct a ﬁnite product space with individual label types.

Revised: Suppose there are k possible label types, i.e. Y 1 , Y 2 , · · · , Y k .

Revised: We construct the label space by the Cartesian product

Revised: [Equation]

Revised: All that is left is to deﬁne the distance:

Revised: [Equation]

Revised: where proj i is projection onto the i -th factor space.

Revised: Then, using this combination, we extend the exponential family model (1), yielding

Revised: [Equation]

Revised: We can learn the parameters θ ( i ) a , θ ( i ) a,b by Algorithm 1 using the same approach.

Revised: Similarly, the inference procedure (3) can be extended to

Revised: [Equation]

Revised: There are two additional aspects worth mentioning.

Revised: First, the user may wish to consider the scale of distances in each label space, since the scale of one of the factor spaces’ distance might be dominant.

Revised: To weight each label space properly, we can normalize each label space’s distance.

Revised: Second, we may wish to consider the abstention symbol as an additional element in each space.

Revised: This permits LFs to output one or another type of label without necessarily emitting a full vector.

Revised: This can also be handled; the underlying algorithm is simply modiﬁed to permit abstains as in Fu et al.

Revised: D U NIVERSAL L ABEL M ODEL F OR I SOTROPIC G AUSSIAN E MBEDDING

Revised: We illustrate the isotropic Gaussian version of the label model.

Revised: While simple, it captures all of the challenges involved in label model learning, and it performs well in practice.

Revised: The steps are shown in Algorithm 3.

Revised: Why does Algorithm 3 obtain the estimates of θ without observing the true label y ?

Revised: To see this, ﬁrst, note that post-embedding, the model we are working with is given by

Revised: [Equation]

Revised: If the embedding is a bijection, the resulting model is indeed is a multivariate Gaussian.

Revised: Note that the term “isotropic” here refers to the fact that for d > 1 , the covariance term for the random vector λ a is a multiple of I .

Revised: Now, observe that if ( a, b ) (cid:54)∈ E , we have that E (cid:2) (cid:107) λ a − λ b (cid:107) 2 (cid:3) =

Revised: E (cid:2) (cid:107) ( λ a − y ) − ( λ b − y ) (cid:107) 2 (cid:3) =

Revised: E (cid:2) (cid:107) λ a − y (cid:107) 2 (cid:3) + E (cid:2) (cid:107) λ

Revised: b − y (cid:107) 2 (cid:3) .

Revised: Note that we can estimate the left-hand side E (cid:2) (cid:107) λ a − λ b (cid:107) 2 (cid:3) from samples,

Revised: 1: Input: Output of labeling functions λ a ( i ) , correlation set E 2: for a ∈ { 1 , 2 , . . .

Revised: , m } do 3: for b ∈ { 1 , 2 , . . .

Revised: , m } \ a do 4:

Revised: Estimate Correlations: ∀ i, j , ˆ E (cid:2) d ( λ a , λ b ) (cid:3) ← 1 n (cid:80) nt =1 d ( λ a ( i ) , λ b ( i )) 5: end for 6:

Revised: Estimate Accuracy: Pick b, c : ( a, b ) (cid:54)∈ E, ( a, c ) (cid:54)∈ E, ( b, c ) (cid:54)∈ E ˆ E

Revised: [ d ( λ a , y )] ← 1 / 2( E (cid:2) d ( λ a , λ b ) (cid:3) + E [ d ( λ a , λ c )] − E (cid:2) d ( λ b , λ c ) (cid:3) ) 7:

Revised: Form estimated covariance matrix ˆΣ from accuracies and correlations; Compute ˆ θ ← ˆΣ −8: end for 9: return ˆ θ a , ˆ θ a,b while the right-hand side contains two of our accuracy terms.

Revised: We can then form two more equations of this type (involving the pairs a, c and b, c ) and solve the resulting linear system.

Revised: Concretely, we have that

Revised: [Equation]

Revised: To obtain the estimate of E (cid:2) (cid:107) g

Revised: ( λ a ) − y (cid:107) 2 (cid:3)

Revised: , we add the ﬁrst two equations, subtract the third, and divide by two.

Revised: This produces the accuracy estimation step in Algorithm 3.

Revised: To obtain the estimate of the canonical parameters θ , it is sufﬁcient to invert our estimate of the covariance matrix.

Revised: In practice, note that we need not actually compute an embedding; we can directly work with the original metric space distances by implicitly assuming that there exists an isometric embedding.

Revised: E A DDITIONAL A LGORITHMIC D ETAILS

Original: Quadratic Triplet
Revised: Quadratic Triplets

Original: We give the full statement of Algorithm 3, which is the accuracy estimation algorithms for the case Im ( g ) = {± 1 } d .
Revised: We give the full statement of Algorithm 4, which is the accuracy estimation algorithms for the case Im ( g ) = {± 1 } d .

Revised: Inference simpliﬁcation in R For the simpliﬁcation of inference in the isotropic Gaussian embedding, we do not even need to recover the canonical parameters; it is enough to use the mean parameters estimated in the label model learning step.

Original: Input: Estimates O a,b , O a,c , O b,c , (cid:96) a , (cid:96) b , (cid:96) c , prior p , index i for y in Y do
Revised: For example, if d = 1 , we can write these accuracy mean parameters as ˆ E (cid:2) g ( λ i ) g ( y ) (cid:3) and put them into a vector ˆ E [Σ] Λ y . The correlations can be placed into the sample covariance matrix ˆΣ g ( λ 1 ) ,...,g (( λ m ) . Then,

Revised: [Equation]

Revised: i ) , . . .

Revised: , λ m ( i )] .

Revised: This is simply the mean of a conditional (scalar) Gaussian.

Revised: F A DDITIONAL T HEORY D ETAILS

Revised: We discuss further details for several theoretical notions.

Revised: The ﬁrst provides more details on how to obtain the canonical accuracy parameters θ a from the mean parameters E

Revised: [ d τ ( λ a , y )] in the rankings case.

Revised: The second involves a discussion of learning to rank problems, with additional details on the weighted estimator (3).

Revised: Input: Estimates O a,b , O a,c , O b,c , (cid:96) a , (cid:96) b , (cid:96) c , prior p , index i for y in Y do

Original: Obtain probability p (cid:48)
Revised: Obtain probability p (cid:48)

Original: =
Revised: =

Original: P ( Y = y ) from prior p
Revised: P ( Y = y ) from prior p

Original: Set β ← ( O a,b (1 − p (cid:48) )
Revised: Set β ← ( O a,b (1 − p (cid:48) )

Original: + ( (cid:96) a − pz ) (cid:96) b ) / ( p (cid:48)
Revised: + ( (cid:96) a − pz ) (cid:96) b ) / ( p (cid:48)

Original: z − p (cid:48) (cid:96) a )
Revised: z − p (cid:48) (cid:96) a )

Original: Set γ ← ( O a,c (1 − p (cid:48) )
Revised: Set γ ← ( O a,c (1 − p (cid:48) )

Original: + ( (cid:96) a − pz ) (cid:96) c ) / ( p (cid:48)
Revised: + ( (cid:96) a − pz ) (cid:96) c ) / ( p (cid:48)

Original: z − p (cid:48) (cid:96) a )
Revised: z − p (cid:48) (cid:96) a )

Original: Solve quadratic ( pβγ + (cid:96) b (cid:96) c − pβ(cid:96) c − pγ(cid:96) b − O a,b (1 − p ))
Revised: Solve quadratic ( pβγ + (cid:96) b (cid:96) c − pβ(cid:96) c − pγ(cid:96) b − O a,b (1 − p ))

Original: ( p (cid:48)
Revised: ( p (cid:48)

Original: α − p (cid:48) (cid:96) a ) 2 = 0 in z ˆ P ( g ( λ a ) i | Y =
Revised: α − p (cid:48) (cid:96) a ) 2 = 0 in z ˆ P ( g ( λ a ) i | Y =

Original: y ) ← z ˆ P ( g ( λ b ) i | Y =
Revised: y ) ← z ˆ P ( g ( λ b ) i | Y =

Original: y ) ← ( O a,b (1 − p (cid:48) )
Revised: y ) ← ( O a,b (1 − p (cid:48) )

Original: + ( (cid:96) a − p
Revised: + ( (cid:96) a − p

Original: ˆ P ( g ( λ a ) i
Revised: ˆ P ( g ( λ a ) i

Original: | Y = y ))
Revised: | Y = y ))

Original: (cid:96) b ) / ( p (cid:48)
Revised: (cid:96) b ) / ( p (cid:48)

Original: ˆ P ( g ( λ a ) i | Y = y ) − p (cid:48) (cid:96) a ) ˆ P ( g ( λ c ) i | Y =
Revised: ˆ P ( g ( λ a ) i | Y = y ) − p (cid:48) (cid:96) a ) ˆ P ( g ( λ c ) i | Y =

Original: y ) ← ( O a,c (1 − p (cid:48) )
Revised: y ) ← ( O a,c (1 − p (cid:48) )

Original: + ( (cid:96) a − p
Revised: + ( (cid:96) a − p

Original: ˆ P ( g ( λ a ) i
Revised: ˆ P ( g ( λ a ) i

Original: | Y = y ))
Revised: | Y = y ))

Original: (cid:96) c ) / ( p (cid:48)
Revised: (cid:96) c ) / ( p (cid:48)

Original: ˆ P ( g ( λ a ) i | Y = y ) − p (cid:48) (cid:96) a ) end for return Accuracies
Revised: ˆ P ( g ( λ a ) i | Y = y ) − p (cid:48) (cid:96) a ) end for return Accuracies

Original: [Equation]

Original: Inference simpliﬁcation in R For the simpliﬁcation of inference in the isotropic Gaussian embedding, we do not even need to recover the canonical parameters; it is enough to use the mean parameters estimated in the label model learning step.

Original: For example, if d = 1 , we can write these accuracy mean parameters as ˆ

Original: E (cid:2) g ( λ i ) g ( y ) (cid:3) and put them into a vector ˆ E [Σ] Λ y .

Original: The correlations can be placed into the sample covariance matrix ˆΣ g ( λ 1 ) ,...,g (( λ m ) .

Original: Then,

Original: [Equation]

Original: This is simply the mean of a conditional (scalar) Gaussian.

Original: Extension to mixed label types In the body of the paper, we discussed models for tasks where only one label type is considered.

Original: As a simple extension, we can operate with multiple label types.

Original: For example, this might include a classiﬁcation task where weak label sources give their output as class labels and also might provide conﬁdence values; that is, a pair of label types that include a discrete value and a continuous value.

Original: The main idea is to construct a ﬁnite product space with individual label types.

Original: Suppose there are k possible label types, i.e. Y 1 , Y 2 , · · · , Y k .

Original: We construct the label space by the Cartesian product

Original: [Equation]

Original: All that is left is to deﬁne the distance:

Original: [Equation]
Revised: [Equation]

Original: where proj i is projection onto the i -th factor space.

Original: Then, using this combination, we extend the exponential family model (1), yielding

Original: [Equation]

Original: We can learn the parameters θ ( i ) a , θ ( i ) a,b by Algorithm 1 using the same approach.

Original: Similarly, the inference procedure (3) can be extended to

Original: [Equation]

Original: There are two additional aspects worth mentioning.

Original: First, the user may wish to consider the scale of distances in each label space, since the scale of one of the factor spaces’ distance might be dominant.

Original: To weight each label space properly, we can normalize each label space’s distance.

Original: Second, we may wish to consider the abstention symbol as an additional element in each space.

Original: This permits LFs to output one or another type of label without necessarily emitting a full vector.

Original: This can also be handled; the underlying algorithm is simply modiﬁed to permit abstains as in Fu et al.

Original: D A DDITIONAL T HEORY D ETAILS

Original: We discuss further details for several theoretical notions.

Original: The ﬁrst provides more details on how to obtain the canonical accuracy parameters θ a from the mean parameters E

Original: [ d τ ( λ a , y )] in the rankings case.

Original: The second involves a discussion of learning to rank problems, with additional details on the weighted estimator (3).

Original: More on the backward mapping for rankings We note, ﬁrst, that the backward mapping is quite simple for the Gaussian cases in R d : it just involves inverting the mean parameters.
Revised: More on the backward mapping for rankings We note, ﬁrst, that the backward mapping is quite simple for the Gaussian cases in R d : it just involves inverting the mean parameters.

Original: As an intuitive example, note that the canonical parameter in the interior of the multivariate Gaussian density is Σ − 1 , the inverse covariance matrix.
Revised: As an intuitive example, note that the canonical parameter in the interior of the multivariate Gaussian density is Σ − 1 , the inverse covariance matrix.

Original: The more challenging aspect is to deal with the discrete setting, and, in particular, its application to rankings.
Revised: The more challenging aspect is to deal with the discrete setting, and, in particular, its application to rankings.

Original: We do so below.
Revised: We do so below.

Original: We describe how we can recover the canonical accuracy parameter θ a for a label function λ a given accuracy estimates P ( g ( λ a ) i = 1 | Y =
Revised: We describe how we can recover the canonical accuracy parameter θ a for a label function λ a given accuracy estimates P ( g ( λ a ) i = 1 | Y =

Original: y )
Revised: y )

Original: for all y ∈ Y .
Revised: for all y ∈ Y .

Original: By equation (1), the marginal distribution of λ a is speciﬁed by
Revised: By equation (1), the marginal distribution of λ a is speciﬁed by

Original: [Equation]
Revised: [Equation]

Original: Since this is an exponential model, it follows from Fligner & Verducci (1986)
Revised: Since this is an exponential model, it follows from Fligner & Verducci (1986)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: where D = (cid:80) i 1 { g ( λ a ) i
Revised: where D = (cid:80) i 1 { g ( λ a ) i

Original: (cid:54) =
Revised: (cid:54) =

Original: g ( y ) i } and M ( t ) is the moment generating function of D under (5).
Revised: g ( y ) i } and M ( t ) is the moment generating function of D under (5).

Original: E [ D ] can be easily estimated from the accuracy parameters obtained from the triplet algorithm, and the inverse of (6) can then be solved for.
Revised: E [ D ] can be easily estimated from the accuracy parameters obtained from the triplet algorithm, and the inverse of (6) can then be solved for.

Original: For instance, in the rankings case, it can be shown (Fligner & Verducci, 1986)
Revised: For instance, in the rankings case, it can be shown (Fligner & Verducci, 1986)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: Additionally, we have that the partition function satisﬁes (Chierichetti et al., 2014)
Revised: Additionally, we have that the partition function satisﬁes (Chierichetti et al., 2014)

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Using this, we can then solve for (6) numerically.
Revised: Using this, we can then solve for (6) numerically.

Original: Rank aggregation and the weighted Kemeny estimator Next, we provide some additional details on the learning to rank problem.
Revised: Rank aggregation and the weighted Kemeny estimator Next, we provide some additional details on the learning to rank problem.

Original: The model (1) without correlations can be written
Revised: The model (1) without correlations can be written

Original: [Equation]
Revised: [Equation]

Original: Thus, if we only have one labeling function, we obtain the Mallows model (Mallows, 1957) for permutations, whose standard form is p ( λ 1 | y )
Revised: Thus, if we only have one labeling function, we obtain the Mallows model (Mallows, 1957) for permutations, whose standard form is p ( λ 1 | y )

Original: = 1 /Z exp( − θ 1 d
Revised: = 1 /Z exp( − θ 1 d

Original: τ ( λ 1 , y )) .
Revised: τ ( λ 1 , y )) .

Original: Permutation learning problems often use the
Revised: Permutation learning problems often use the

Original: Mallows model and its relatives.
Revised: Mallows model and its relatives.

Original: The permutation y (called the center ) is ﬁxed and n samples of λ 1 are drawn from the model.
Revised: The permutation y (called the center ) is ﬁxed and n samples of λ 1 are drawn from the model.

Original: The goal is to (1) estimate the parameter θ 1 and (2) estimate the center y .
Revised: The goal is to (1) estimate the parameter θ 1 and (2) estimate the center y .

Original: This neatly mirrors our setting, where (1) is label model learning and (2) is the inference procedure.
Revised: This neatly mirrors our setting, where (1) is label model learning and (2) is the inference procedure.

Original: However, our problem is harder in two ways.
Revised: However, our problem is harder in two ways.

Original: While we do have n samples from each marginal model with parameter θ i , these are for different centers y 1 , . .
Revised: While we do have n samples from each marginal model with parameter θ i , these are for different centers y 1 , . .

Original: , y n —so we cannot aggregate them or directly estimate θ i .
Revised: , y n —so we cannot aggregate them or directly estimate θ i .

Original: That is, for LF a , we get to observe λ a (1) , λ a (2) , . . .
Revised: That is, for LF a , we get to observe λ a (1) , λ a (2) , . . .

Original: , λ a ( n ) .
Revised: , λ a ( n ) .

Original: However, these all come from different conditional models P ( ·| y i ) , with a different y i for each draw.
Revised: However, these all come from different conditional models P ( ·| y i ) , with a different y i for each draw.

Original: In the uninteresting case where the y i ’s are identical, we obtain the standard setting.
Revised: In the uninteresting case where the y i ’s are identical, we obtain the standard setting.

Original: On the other hand, we do get to observe m views (from the LFs) of the same permutation y i .
Revised: On the other hand, we do get to observe m views (from the LFs) of the same permutation y i .

Original: But, unlike in standard rank aggregation, these do not come from the same model—the θ a accuracy parameters differ in (9).
Revised: But, unlike in standard rank aggregation, these do not come from the same model—the θ a accuracy parameters differ in (9).

Original: However, if θ a is identical (same accuracy) for all LFs, we get back to the standard case.
Revised: However, if θ a is identical (same accuracy) for all LFs, we get back to the standard case.

Original: Thus we can recover the standard permutation learning problem in the special cases of identical labels or identical accuracies—but such assumptions are unlikely to hold in practice.
Revised: Thus we can recover the standard permutation learning problem in the special cases of identical labels or identical accuracies—but such assumptions are unlikely to hold in practice.

Original: Note that our inference procedure (3) is the weighted version of the standard Kemeny procedure used for rank aggregation.
Revised: Note that our inference procedure (3) is the weighted version of the standard Kemeny procedure used for rank aggregation.

Original: Observe that this is the maximum likelihood estimator on the model (1), specialized to permutations.
Revised: Observe that this is the maximum likelihood estimator on the model (1), specialized to permutations.

Original: This is nearly immediate: maximizing the linear combination (where the parameters are weights) produces the smallest negative term on the inside of the exponent above.
Revised: This is nearly immediate: maximizing the linear combination (where the parameters are weights) produces the smallest negative term on the inside of the exponent above.

Original: Next, we note that it is possible to show that the sample complexity of learning the permutation y using this estimator is still Θ(log( m/ε )) by using the pairwise majority result in Caragiannis et al.
Revised: Next, we note that it is possible to show that the sample complexity of learning the permutation y using this estimator is still Θ(log( m/ε )) by using the pairwise majority result in Caragiannis et al.

Original: Finally, a brief comment on computational complexity: it is known that ﬁnding the minimizer of the Kemeny rule (or our weighted variant) is NP-hard (Davenport & Kalagnanam, 2004).
Revised: Finally, a brief comment on computational complexity: it is known that ﬁnding the minimizer of the Kemeny rule (or our weighted variant) is NP-hard (Davenport & Kalagnanam, 2004).

Original: However, there are PTAS available for it (Kenyon-Mathieu & Schudy, 2007).
Revised: However, there are PTAS available for it (Kenyon-Mathieu & Schudy, 2007).

Original: In practice, our permutations are likely to be reasonably short (for example, in our experiments, we typically use length 5) so that we can directly perform the minimization.
Revised: In practice, our permutations are likely to be reasonably short (for example, in our experiments, we typically use length 5) so that we can directly perform the minimization.

Original: In cases with longer permutations, we can rely on the PTAS.
Revised: In cases with longer permutations, we can rely on the PTAS.

Original: E P ROOF OF T HEOREMS
Revised: G P ROOF OF T HEOREMS

Original: Next we give the proofs of the proposition and our two theorems, restated for convenience.
Revised: Next we give the proofs of the proposition and our two theorems, restated for convenience.

Original: E. D ISTORTION B OUND
Revised: G. D ISTORTION B OUND

Original: Our result that captures the impact of distortion on parameter error is
Revised: Our result that captures the impact of distortion on parameter error is

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: The inconsistency in estimating θ is bounded as (cid:107) θ − θ (cid:48)
Revised: The inconsistency in estimating θ is bounded as (cid:107) θ − [Equation]

Original: (cid:107) ≤ ε (cid:107) µ (cid:107) /e min .
Revised: ≤ [Equation]

Original: Before we begin the proof, we restate, in greater detail, some of our notation.
Revised: Before we begin the proof, we restate, in greater detail, some of our notation.

Original: We write p θ ; d Y for the true model
Revised: We write p θ ; d Y for the true model

Original: [Equation]
Revised: [Equation]

Original: and p θ (cid:48)
Revised: and p θ (cid:48)

Original: ; d g for models that rely on distances in the embedding space
Revised: ; d g for models that rely on distances in the embedding space

Original: [Equation]
Revised: [Equation]

Original: d g ( g ( λ a [Equation] , g 
Revised: ( λ a ) , g ( λ b )) (cid:1) .

Original: [Equation]

Original: λ b [Equation] ) c (cid:1) . We also write θ to be the vector of the canonical parameters θ a and θ a,b in p θ ; d Y and θ (cid:48) to be its counterpart for p θ (cid:48) ; d g .
Revised: We also write θ to be the vector of the canonical parameters θ a and θ a,b in p θ ; d Y and θ (cid:48) to be its counterpart for p θ (cid:48) ; d g .

Original: Let µ be the vector of mean parameters whose terms are E [ d Y ( λ a , y ) c ] and E [ d Y ( λ a , λ b ) c ] .
Revised: Let µ be the vector of mean parameters whose terms are E [ d Y ( λ a , y )] and E [ d Y ( λ a , λ b )] .

Original: Similarly, we write µ (cid:48) for the version given by p θ (cid:48) ; d g .
Revised: Similarly, we write µ (cid:48) for the version given by p θ (cid:48) ; d g .

Original: Let Θ be a subspace so that θ, θ (cid:48) ∈ Θ .
Revised: Let Θ be a subspace so that θ, θ (cid:48) ∈ Θ .

Original: Since our exponential family is minimal, we have that the log partition function A (˜ θ ) has the property that ∇ 2 A (˜ θ ) is the covariance matrix and is positive deﬁnite.
Revised: Since our exponential family is minimal, we have that the log partition function A (˜ θ ) has the property that ∇ 2 A (˜ θ ) is the covariance matrix and is positive deﬁnite.

Original: Suppose that the smallest eigenvalue of ∇ 2 A (˜ θ ) for ˜ θ ∈ Θ is e min .
Revised: Suppose that the smallest eigenvalue of ∇ 2 A (˜ θ ) for ˜ θ ∈ Θ is e min .

Original: For our embedding, suppose we ﬁrst normalize (ie, divide by a constant) our embedding function so that
Revised: For our embedding, suppose we ﬁrst normalize (ie, divide by a constant) our embedding function so that

Original: [Equation]
Revised: [Equation]

Original: for all y, y (cid:48) ∈ Y × Y .
Revised: for all y, y (cid:48) ∈ Y × Y .

Original: In other words, our embedding function g never expands the distances in the original metric space, but, of course, it may compress them.
Revised: In other words, our embedding function g never expands the distances in the original metric space, but, of course, it may compress them.

Original: The distortion measures how bad the compression can be, Let ε be the smallest value so that
Revised: The distortion measures how bad the compression can be, Let ε be the smallest value so that

Original: [Equation]
Revised: [Equation]

Original: for all y, y (cid:48) ∈ Y × Y .
Revised: for all y, y (cid:48) ∈ Y × Y .

Original: If g is isometric, then we obtain ε = 0 .
Revised: If g is isometric, then we obtain ε = 0 .

Original: On the other hand, as ε approaches 1, the amount of compression can be arbitrarily bad.
Revised: On the other hand, as ε approaches 1, the amount of compression can be arbitrarily bad.

Original: Proof.
Revised: Proof.

Original: We use Lemma 8 from Fu et al.
Revised: We use Lemma 8 from Fu et al.

Original: It states that
Revised: It states that

Original: [Equation]
Revised: [Equation]

Original: To see the latter, we note that (cid:107) 
Revised: To see the latter, we note that (cid:107) µ

Original: [Equation]
Revised: [Equation]

Original: (cid:107) ε , where µ (cid:48) µ is the element-wise ratios, and where we applied our distortion bound.
Revised: ε , where µ (cid:48) µ is the element-wise ratios, and where we applied our distortion bound.

Original: E. B INARY H YPERCUBE C ASE
Revised: G. B INARY H YPERCUBE C ASE

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For any δ > 0 , for some y 1 and y 2 with known class probabilities p = P ( Y = y 1 ) , the quadratic triplet method recovers α i
Revised: For any δ > 0 , for some y 1 and y 2 with known class probabilities p = P ( Y = y 1 ) , the quadratic triplet method recovers α i

Original: =
Revised: =

Original: P ( g ( λ a ) i = 1 | Y = y ) , β i = P ( g ( λ b ) i = 

Revised: P ( g ( λ a ) i = 1 | Y = y ) , β i = P ( g ( λ b ) i =

Original: | Y =
Revised: | Y =

Original: y ) , γ i =
Revised: y ) , γ i =

Original: P ( g ( λ c ) i = 1 | Y =
Revised: P ( g ( λ c ) i = 1 | Y =

Original: y )
Revised: y )

Original: up to error O
Revised: up to error O

Original: (( ln(2 m 2 /δ ) 2 n ) 1 / 4 ) with probability at least 1 − δ for all conditionally independent label functions λ a , λ b , λ c and for all i ∈ [ m ] .
Revised: (( ln(2 d 2 /δ ) 2 n ) 1 / 4 ) with probability at least 1 − δ for all conditionally independent label functions λ a , λ b , λ c and for all i ∈ [ d ] .

Original: Proof.
Revised: Proof.

Original: Deﬁne µ
Revised: Deﬁne µ

Original: [Equation]
Revised: [Equation]

Original: = [Equation] P (
Revised: = (cid:20)

Original: [Equation]
Revised: [Equation]

Original: P ( g ( λ
Revised: P ( g ( λ

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  =−
Revised: i =

Original: [Equation]
Revised: [Equation]

Original:  | Y = y ) P ( g ( λ a ) i = − 1 | Y (cid:54) =
Revised: (cid:54) = y ) P ( g ( λ a ) i = − 1 | Y = y ) P ( g ( λ a ) i = − 1 | Y (cid:54) =

Original: y ) (cid:21) , P = (cid:20) P ( Y = y )
Revised: y ) (cid:21) , P = (cid:20) P ( Y = y )

Original: P ( Y (cid:54) =
Revised: P ( Y (cid:54) =

Original: y ) (cid:21) , and
Revised: y ) (cid:21) , and

Original: O abi =
Revised: O abi =

Original: [Equation]
Revised: [Equation]

Original: P (
Revised: P (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: g (
Revised: g (

Original: [Equation]
Revised: [Equation]

Original: = − 1)

Original: P ( g ( λ
Revised: P ( g ( λ

Original: [Equation][Equation] = [Equation]
Revised: [Equation]

Original: [Equation]

Original: g ( λ

Original: [Equation]
Revised: [Equation]

Original: ( λ
Revised: ( λ

Original: [Equation][Equation] [Equation] =
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1) (cid:21) .
Revised: ) (cid:21) .

Original: By conditional independence, we have that
Revised: By conditional independence, we have that

Original: [Equation]
Revised: [Equation]

Original: Note that we can express
Revised: Note that we can express

Original: [Equation]
Revised: [Equation]

Original: ) P
Revised: ) P

Original: [Equation]
Revised: [Equation]

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: = y )
Revised: = y )

Original: [Equation]
Revised: [Equation]

Original: We can therefore rewrite the top row of µ ai as [ α i , c ai − dα i ] where c ai = P ( g ( λ a ) i =1) P ( Y (cid:54) =
Revised: We can therefore rewrite the top row of µ ai as [ α i , q ai − rα i ] where q ai = P ( g ( λ a ) i =1) P ( Y (cid:54) =

Original: y ) and d = P ( Y = y ) P ( Y (cid:54) =
Revised: y ) and r = P ( Y = y ) P ( Y (cid:54) =

Original: y ) .
Revised: y ) .

Original: After estimating the entries of O i ’s and c i ’s, we consider the top-left entry of (10) for every pair of a, b and c, to get the following system of equations
Revised: After estimating the entries of O i ’s and q i ’s, we consider the top-left entry of (10) for every pair of a, b and c, to get the following system of equations

Original: [Equation]
Revised: [Equation]

Original: where β i and γ i are the top-left entries of µ bi and µ ci respectively.
Revised: where β i and γ i are the top-left entries of µ bi and µ ci , respectively, p = P ( Y = y ) , and t = p (1 − p ) 2 .

Original: Dropping the i subscript for ease of notation and rearranging the ﬁrst and third equations gives us expressions for α and γ in terms of β.
Revised: For ease of notation, we write ˆ q ai as ˆ q a and so on. Rearranging the ﬁrst and third equations gives us expressions for α and γ in terms of β.

Original: [Equation]
Revised: [Equation]

Original:  Substituting  into (7) gives
Revised: = [Equation] Substituting these expressions into the second equation of the system gives

Original: [Equation]
Revised: [Equation]

Original: We can multiply the equation by ((1 + d 2 ) β − ˆ c b d ) 2 to get
Revised: We can multiply the equation by ( tβ − ˆ q b r ) 2 to get

Original: [Equation]
Revised: [Equation]

Revised: ⇒ t

Original: [Equation]
Revised: [Equation]

Original:  The only sources of error are from estimating c ’s and O ’s. 
Revised: [Equation] The only sources of error are from estimating c ’s and O ’s. Let ε c (cid:48) denote the error for the c ’s. Let ε O denote the error from O ’s. Applying the quadratic formula we obtain β = ( − b (cid:48) ± (cid:112) ( b (cid:48) ) 2 − 4 a

Original: Let (cid:15) c  (cid:48) denote the error for the c ’s. Let (cid:15)
Revised: [Equation] c [Equation] ) / (2 a (cid:48) ) where the coefﬁcients are

Original: O denote the error from O ’s. The quadratic formula is β = − b ±√ b 2 − 4 ac 2 a where

Original: [Equation]
Revised: [Equation]

Original: Let (cid:15) a  , (cid:15) b  ,  ( 
Revised: Let ε a (cid:48) , ε b (cid:48) , ε c (cid:48) denote the error for each coefﬁcient. That is, ε a (cid:48) = | ( a (cid:48) ) ∗ − a (cid:48) | , where ( a (cid:48) ) ∗ is the populationlevel coefﬁcient, and similarly for b (cid:48) , c (cid:48) . Let ε c

Original: cid:15) c denote the  error from each term.
Revised: and ε O indicate the estimation error for the c and O terms.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original:  O 
Revised: rt 1 − p ( ε c + ε O ) (cid:19) , ε c (cid:48) = O (cid:18)(cid:18) t (1 − p ) 2 + r1 − p + r(cid:19) ε c + (cid:18) t(1 − p ) 2 + r1 − p (cid:19)

Original: [Equation]
Revised: ε O (cid:19) .

Original: Because d is deﬁned with respect to p, if we ignore the dependence on p, we get that
Revised: Because d and e are functions of p , if we ignore the dependence on p , we get that

Original: [Equation]
Revised: [Equation]

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Since O and c are Bernoulli random variables, by Hoeffding’s inequality and union bound, with probability at least 1 − δ 2 , (cid:15)
Revised: Next, note that O and c are both the averages of indicator variables, where the c

Original: [Equation] i ≤{+’s involve P ( g ( λ a ) i = 
+}
Revised: i {+’s involve P ( g ( λ a ) i = 
+}

Original: [Equation]
Revised: and the O abi ’s upper-left corners compute P ( g ( λ a ) i = 1 , g ( λ b ) i = 1) . Thu we can apply Hoeffding’s inequality and combine this with the union bound to bound the above terms.

Original: 2 n for all i ∈ [ m ] , and with probability at least 1 − δ 2 , (cid:15) O ij ≤ {+(cid:113) log(
+}
Revised: We have with probability at least 1 − δ 2 , ε c i ≤ {+(cid:113) log(
+}

Original: [Equation] ln{+d/δ ) 2 n and similarly with probability at least 1 − δ 2 , ε O i ≤ (cid:113) log(
+}
Revised: {+d/δ ) 2 n and similarly with probability at least 1 − δ 2 , ε O i ≤ (cid:113) log(
+}

Original: [Equation] for all i, j ∈ [ m ] .
Revised: d/δ ) 2 n for all i, j ∈ [ m ] .

Original: It follows that with probability at least 1 − δ, (cid:15) α = 
Revised: It follows that with probability at least 1 − δ, ε α = ε β = ε γ =

Original: [Equation] [Equation]
Revised: O (( log(2 d/δ ) 2 n ) 1 / 4 ) .

Original: We can now prove the main theorem in the rankings case.
Revised: We can now prove the main theorem in the rankings case.

Original: Corollary 4.1.1.
Revised: Corollary 4.1.1.

Original: For any δ > 0 , U > 0 ,  prior over y 1 and y 2 with known class probability p , and using Algorithm 1 and Algorithm 3, for all conditionally independent  λ a , λ b , λ c , with parameters
Revised: For any δ > 0 , U > 0 , a prior over y 1 and y 2 with known class probability p , and using Algorithm 1 and Algorithm 4, for any conditionally independent triplet λ a , λ b , λ c , with parameters

Original: U > θ a , θ b , θ c > 4 ln(2) , we can recover θ a , θ b , θ c up to error O ( ρ 2 (ln(2 m 2 ) / (2 δn )) 1 / 4/ | g 2 ( U ) | ) with probability at least 1 − δ , where g 2 ( U 
Revised: U > θ a , θ b , θ c > 4 ln(2) , we can recover θ a , θ b , θ c up to error O ( g − 1 2 ( θ + (log(2 ρ 2 ) / (2 δn )) 1 / 4 ) − θ ) with probability at least 1 − δ , where g 2 ( U ) =

Original: [Equation]
Revised: ( − ρe − U ) / ((1 − e − U ) 2 ) + (cid:80) ρj

Original: ={+=
+}
Revised: {+=
+}

Original: [Equation]
Revised: ( j 2 e − Uj ) /

Original:  − [Equation] [Equation]
Revised: ((1 − e − Uj ) 2 ) .

Original: Proof.
Revised: Proof.

Original: Consider a pair of items ( a, b ) .
Revised: Consider a pair of items ( a, b ) .

Original: Without loss of generality, suppose a ≺ y 1 b and a (cid:31)
Revised: Without loss of generality, suppose a ≺ y 1 b and a (cid:31)

Original: y 2 b. Deﬁne α a,b = P ( a ≺ λ a b | y 1 ) , α (cid:48) a,b = P ( a (cid:31)
Revised: y 2 b. Deﬁne α a,b = P ( a ≺ λ a b | y 1 ) , α (cid:48) a,b = P ( a (cid:31)

Original: λ a b | y 2 )
Revised: λ a b | y 2 )

Original: then our estimate for P ( λ a ( a,b ) ∼ Y ( a,b ) )
Revised: then our estimate for P ( λ a ( a,b ) ∼ Y ( a,b ) )

Original: where λ a ( a,b ) ∼
Revised: where λ a ( a,b ) ∼

Original: Y ( a,b ) denotes the event that label function i ranks ( a, b ) correctly would be
Revised: Y ( a,b ) denotes the event that label function i ranks ( a, b ) correctly would be

Original: [Equation]
Revised: [Equation]

Original: which has error O ( (cid:15) α )
Revised: which has error O ( (cid:15) α )

Original: Then, note that E [ d ( λ a , Y )]
Revised: Then, note that E [ d ( λ a , Y )]

Original: = (cid:80) a,b P ( λ a ( a,b ) ∼ Y ( a,b ) ) .
Revised: = (cid:80) a,b P ( λ a ( a,b ) ∼ Y ( a,b ) ) .

Original: Therefore, we can compute the estimate ˆ E [ d ( λ a , Y )]
Revised: Therefore, we can compute the estimate ˆ E [ d ( λ a , Y )]

Original: = (cid:80) a,b ˆ E
Revised: = (cid:80) a,b ˆ E

Original: [ λ i ∼ Y ] a,b which has error O ( (cid:0) ρ 2 (cid:1) ( ln(6) /δ 2 n ) 1 / 4 ) .
Revised: [ λ i ∼ Y ] a,b which has error O ( (cid:0) ρ 2 (cid:1) ( ln( [Equation] /δ 2 n ) 1 / 4 ) .

Original: Recall from (7) we have that
Revised: Recall from (7) we have that

Original: [Equation]
Revised: [Equation]

Original: where M ( t ) is the moment generating function, and recall from (8) that
Revised: where M ( t ) is the moment generating function, and recall from (8) that

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Let g k ( θ )
Revised: Let g k ( θ )

Original: = − ke − θ (1 − e − θ ) 

Revised: = − ke − θ (1 − e − θ )

Original: + (cid:80) j ≤ k j 2 e − θj (1 − e − θj ) 2 .
Revised: + (cid:80) j ≤ k j 2 e − θj (1 − e − θj ) 2 .

Original: By the lemma below, g k is non positive and increasing in θ for θ > 0 .
Revised: By the lemma below, g k is non positive and increasing in θ for θ > 0 .

Original: It follows that by numerically computing the inverse function of (7), our estimate of θ has error O ( ρ 2 | g 2 ( U ) | ( ln(2 m 2 ) /δ 2 n ) 1 / 4 ) .
Revised: This means we can numerically compute the inverse function of (7), with the stated error.

Original: Lemma E.1.
Revised: Lemma G.1.

Original: g k is non-positive and increasing in θ for θ > 4 ln(2) .
Revised: g k is non-positive and increasing in θ for θ > 4 ln(2) .

Original: Additionally, g k is decreasing in k.
Revised: Additionally, g k is decreasing in k.

Original: Proof.
Revised: Proof.

Original: We ﬁrst show non-positivity.
Revised: We ﬁrst show non-positivity.

Original: For this, it is sufﬁcient to show
Revised: For this, it is sufﬁcient to show

Original: [Equation]
Revised: [Equation]

Original: holds for all 1 ≤ j and θ > 0 .
Revised: holds for all 1 ≤ j and θ > 0 .

Original: This clearly holds for j = 1 .
Revised: This clearly holds for j = 1 .

Original: Rearranging gives
Revised: Rearranging gives

Original: [Equation]
Revised: [Equation]

Original: The right-hand term is greater than or equal to 1, so it sufﬁces to choose θ such that
Revised: The right-hand term is greater than or equal to 1, so it sufﬁces to choose θ such that

Original: [Equation]
Revised: [Equation]

Original: It can be shown that the right-hand term decreases with j for j > 1 , so it sufﬁces to take j = 2 , which implies θ ≥ 2 ln(2) .
Revised: It can be shown that the right-hand term decreases with j for j > 1 , so it sufﬁces to take j = 2 , which implies θ ≥ 2 ln(2) .

Original: By choice of θ this is clearly true.
Revised: By choice of θ this is clearly true.

Original: To show that g k is increasing in θ, we consider
Revised: To show that g k is increasing in θ, we consider

Original: [Equation]
Revised: [Equation]

Original: Similarly, it sufﬁces to show
Revised: Similarly, it sufﬁces to show

Original: [Equation]
Revised: [Equation]

Original: Rearranging, we have
Revised: Rearranging, we have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: The term on the right is greater than 12 for θ > 0 and j ≥ 1 .
Revised: The term on the right is greater than 12 for θ > 0 and j ≥ 1 .

Original: Therefore, it sufﬁces to choose θ such that
Revised: Therefore, it sufﬁces to choose θ such that

Original: [Equation]
Revised: [Equation]

Original: Once again, the term on the right is decreasing in j, so we can take j = 2 , giving θ ≥ 4 ln(2) , which is satisﬁed by our choice of θ.
Revised: Once again, the term on the right is decreasing in j, so we can take j = 2 , giving θ ≥ 4 ln(2) , which is satisﬁed by our choice of θ.

Original: Finally, the fact that g k is decreasing in k follows from (12).
Revised: Finally, the fact that g k is decreasing in k follows from (12).

Original: E. E UCLIDEAN E MBEDDING C ASE
Revised: G. E UCLIDEAN E MBEDDING C ASE

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Let ˆ E [ g ( λ a ) g ( y )] be an estimate of the accuracies
Revised: Let ˆ E [ g ( λ a ) g ( y )] be an estimate of the accuracies

Original: E
Revised: E

Original: [ g ( λ a ) g ( y )] usingˆ e computed from n samples, where all LFs are conditionally independent given Y .
Revised: [ g ( λ a ) g ( y )] using n samples, where all LFs are conditionally independent given Y .

Original: Ifwe assume the signs of a are recoverable, then with high probability
Revised: If the signs of a are recoverable, then with high probability

Original: [Equation]
Revised: [Equation]

Original: Here, a | min | = min i | E [ g ( λ i ) g ( y )] | and e max = max j,k e j,k .
Revised: Here, a | min | = min i | E (cid:2) g ( λ i ) g ( y ) (cid:3) | and e max = max j,k e j,k .

Original: Proof.
Revised: Proof.

Original: For three conditionally independent label functions λ a , λ b , λ c , our estimate of E [ g ( λ a ) g ( y )] is
Revised: For three conditionally independent label functions λ a , λ b , λ c , our estimate of E [ g ( λ a ) g ( y )] is

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if we deﬁne x a,b = | ˆ e a,b | | e a,b | , we can write the ratio of elements of ˆ a to a as
Revised: Furthermore, if we deﬁne x a,b = | ˆ e a,b | | e a,b | , we can write the ratio of elements of ˆ a to a as

Original: [Equation]
Revised: [Equation]

Original: (and the other deﬁnitions are symmetric for k b and k b ).
Revised: (and the other deﬁnitions are symmetric for k b and k b ).

Original: Now note that because we assume that signs are completely recoverable, |
Revised: Now note that because we assume that signs are completely recoverable, |

Original: [Equation]
Revised: [Equation]

Original: Next note that
Revised: Next note that

Original: [Equation]
Revised: [Equation]

Original: By the reverse triangle inequality,
Revised: By the reverse triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne c as min a | ˆ E [ g ( λ a ) g ( y )]
Revised: where we deﬁne c as min a | ˆ E [ g ( λ a ) g ( y )]

Original: + E [ g ( λ a ) g
Revised: + E [ g ( λ a ) g

Original: ( y )] | .
Revised: ( y )] | .

Original: (With high probability c
Revised: (With high probability c

Original: = ˆ a | min | | + | a | min | , but there is a chance that ˆ E [ g ( λ a ) g
Revised: = ˆ a | min | | + | a | min | , but there is a chance that ˆ E [ g ( λ a ) g

Original: ( y )] and E
Revised: ( y )] and E

Original: [ g ( λ a ) g ( y )] have opposite signs.)
Revised: [ g ( λ a ) g ( y )] have opposite signs.)

Original: For ease of notation, suppose we examine a particular ( a, b, c ) =
Revised: For ease of notation, suppose we examine a particular ( a, b, c ) =

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: | ˆ e 12 || ˆ e 13 | | ˆ e 23 | − | ˆ e 12 || ˆ e 13 | | e 23 | + | ˆ e 12 || ˆ e 13 | | e 23 | − | ˆ e 12 || e 13 | | e 23 | + | ˆ e 12 || e 13 | | e 23 | − | e 12 || e 13 | | e 23 | (cid:12) (cid:12)(cid:12) (cid:12)(cid:12) ≤c

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: (cid:18)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: e

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: e 13 ˆ e 23 e 23 (cid:12)(cid:12)(cid:12) ||

Original: [Equation]
Revised: ˆ e 23 | − | e 23 ||

Original: [Table]
Revised: + (cid:12)(cid:12)(cid:12) ˆ e 12 e 23 (cid:12)(cid:12)(cid:12) || ˆ e 13 | − | e 13 || + (cid:12)(cid:12)(cid:12) e 13 e 23 (cid:12)(cid:12)(cid:12) || ˆ e 12 | − | e 12 || (cid:19)≤

Original: [Equation]
Revised: [Equation]

Revised: 13 ˆ e 23 e 23 (cid:12)(cid:12)(cid:12)

Original: [Table]
Revised: | ˆ e 23 − e 23 | + (cid:12)(cid:12)(cid:12) ˆ e 12 e 23 (cid:12)(cid:12)(cid:12) | ˆ e 13 − e 13 | + (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: 23 (cid:12)(cid:12)(cid:12) | ˆ e 12 − e 12 | (cid:19).

Original: With high probability, all elements of ˆ e and e must be less than e max = max j,k e j,k .
Revised: With high probability, all elements of ˆ e and e must be less than e max = max j,k e j,k .

Original: We further know that elements of | e | are at least a 2 | min | /E
Revised: We further know that elements of | e | are at least a 2 | min | /E

Original: [ Y 2 ] .
Revised: [ Y 2 ] .

Original: Now suppose (with high probability) that elements of | ˆ e | are at least ˆ a 2 | min | > 0 , and deﬁne (cid:52) a,b
Revised: Now suppose (with high probability) that elements of | ˆ e | are at least ˆ a 2 | min | > 0 , and deﬁne (cid:52) a,b

Original: = ˆ e a,b − e a,b .
Revised: = ˆ e a,b − e a,b .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: The original expression is now
Revised: The original expression is now

Original: [Equation]
Revised: [Equation]

Original: To bound the maximum absolute value between elements of ˆ e and e , note that
Revised: To bound the maximum absolute value between elements of ˆ e and e [Equation] note that

Original: [Equation]
Revised: [Equation]

Original: where e a,b,c is a 3 × 3 matrix with e i,j in the ( i, j ) -th position.
Revised: where e a,b,c is a 3 × 3 matrix with e i,j in the ( i, j ) -th position.

Original: Moreover, it is a fact that ||
Revised: Moreover, it is a fact that ||

Original: ˆ e a,b,c − e a,b,c || F ≤ √ r
Revised: ˆ e a,b,c − e a,b,c || F ≤ √ r

Original: ||
Revised: ||

Original: ˆ
Revised: ˆ

Original: e a,b,c − e a,b,c || 2 ≤ √ 

Revised: e a,b,c − e a,b,c || 2 ≤ √

Original: ||
Revised: ||

Original: ˆ e a,b,c − e a,b,c
Revised: ˆ e a,b,c − e a,b,c

Original: || 2 where r is the rank of ˆ e a,b,c − e a,b,c .
Revised: || 2 where r is the rank of ˆ e a,b,c − e a,b,c .

Original: Putting everything together, 
Revised: Putting everything together, |

Original: [Equation]
Revised: [Equation]

Original: Lastly, to compute E [ || ˆ E [ g ( λ a ) g
Revised: Lastly, to compute E [ || ˆ E [ g ( λ a ) g

Original: ( y )]
Revised: ( y )]

Original: − E [ g ( λ a ) g ( y )]
Revised: − E [ g ( λ a ) g ( y )]

Original: || 2 ] , we use Jensen’s inequality (concave version, due to the square root) and linearity of expectation:
Revised: || 2 ] , we use Jensen’s inequality (concave version, due to the square root) and linearity of expectation:

Original: [Equation]
Revised: [Equation]

Original: We use the covariance matrix inequality as described in Tropp (2014), which states that
Revised: We use the covariance matrix inequality as described in Tropp (2014), which states that

Original: [Equation]
Revised: [Equation]

Original: where σ = max a e aa and C > 0 is a universal constant.
Revised: where σ = max a e aa and C > 0 is a universal constant.

Original: To get the probability distribution over ||
Revised: To get the probability distribution over ||

Original: ˆ e a,b,c − e a,b,c || 22 , we just note that P ( || ˆ e a,b,c − e a,b,c || 2 ≥ γ ) =
Revised: ˆ e a,b,c − e a,b,c || 22 , we just note that P ( || ˆ e a,b,c − e a,b,c || 2 ≥ γ ) =

Original: P ( || ˆ e − e || 22 ≥ γ 2 ) to get P 
Revised: P ( || ˆ e − e || 22 ≥ γ 2 ) to get P ( ||

Original: [Equation]
Revised: [Equation]

Original: From this we can integrate to get
Revised: From this we can integrate to get

Original: E [ || ˆ e a,b,c − e a,b,c || 22 ] =

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: (cid:19)

Original: [Equation]

Original: n ) .

Original: Substituting this back in, we get
Revised: Substituting this back in, we get

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: max [Equation] (cid:19)(cid:33) [Equation]
Revised: with high probability.

Original: O (cid:18) [Equation] n (cid:19)(cid:33)≤ (cid:32) [Equation]
Revised: Finally

Original: (cid:33)

Original: [Equation]

Original: O (cid:18)

Original: [Equation]
Revised: [Equation]

Original: n (cid:19)(cid:33)with high probability.

Original: Finally, we clean up the a | min | and ˆ a | min | terms.
Revised: we clean up the a | min | and ˆ a | min | terms.

Original: The terms involving a and ˆ a can be rewritten as
Revised: The terms involving a and ˆ a can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: We suppose that ˆ a | min | a | min | ∈ [1 − (cid:15), 1 + (cid:15) ] for some (cid:15) > 0 with high probability.
Revised: We suppose that ˆ a | min | a | min | ∈ [1 − (cid:15), 1 + (cid:15) ] for some (cid:15) > 0 with high probability.

Original: Then, this becomes less than1 + 2ˆ a 4 | min | (1 − (cid:15) ) 4 a 10 | min | + 2(1 − (cid:15) ) 5 a 10 | min | + (1 − (cid:15) ) 6 a 10 | min | ≤
Revised: Then, this becomes less than

Original: [Equation]
Revised: [Equation]

Original: Therefore, with high probability, the sampling error for the accuracy is bounded by
Revised: Therefore, with high probability, the sampling error for the accuracy is bounded by

Original: [Equation]
Revised: [Equation]

Original: Note that if all label functions are conditionally independent, we only need to know the sign of one accuracy to recover the rest.
Revised: Note that if all label functions are conditionally independent, we only need to know the sign of one accuracy to recover the rest.

Original: For example, if we know if E [ g ( λ 1 ) g ( y )] is positive or negative, we can use E
Revised: For example, if we know if E [ g ( λ 1 ) g ( y )] is positive or negative, we can use E

Original: [ g ( λ 1 ) g ( y )]
Revised: [ g ( λ 1 ) g ( y )]

Original: E [ g ( λ 2 ) g ( y )] 
Revised: E [ g ( λ 2 ) g ( y )] =

Original: =
Revised: [Equation]

Original: e 1 , 2 E [ Y 2 ] 2 , E [ g ( λ 1 ) g ( y )] E [ g ( λ 3 ) g ( y )]
Revised: [Equation] [Equation]

Original: = e 1 ,3 E 
Revised: [Equation] , E [

Original: [ Y 2 ] 2 , . . .
Revised: [Equation]

Original: , E [ g ( λ 1 ) g ( y )] 
Revised: ( λ 1 ) g ( y )] E [ g ( λ

Original: E [ g ( λ m ) g( y )] =
Revised: [Equation] ) g

Original: e 1 ,m E [ Y 2 ] 2 to recover all other signs.
Revised: ( y

Original: F E XTENDED E XPERIMENTAL D ETAILS
Revised: H E XTENDED E XPERIMENTAL D ETAILS

Original: In this section, we provide some additional experimental results and details.
Revised: In this section, we provide some additional experimental results and details.

Original: All experiments were conducted on a machine with Intel Broadwell 2.7GHz CPU and NVIDIA GK210 GPU.
Revised: All experiments were conducted on a machine with Intel Broadwell 2.7GHz CPU and NVIDIA GK210 GPU.

Original: Each experiments take fromminutes up to a few hours depending on the experiment conditions.
Revised: Each experiment takes fromminutes up to a few hours depending on the experiment conditions.

Original: Hyperbolic Space and Geodesic Regression
Revised: Hyperbolic Space and Geodesic Regression

Original: The following is basic background useful for understanding our hyperbolic space models.
Revised: The following is basic background useful for understanding our hyperbolic space models.

Original: Hyperbolic space is a Riemannian manifold.
Revised: Hyperbolic space is a Riemannian manifold.

Original: Unlike Euclidean space, it does not have a vector space structure.
Revised: Unlike Euclidean space, it does not have a vector space structure.

Original: Therefore it is not possible to directly add points.
Revised: Therefore it is not possible to directly add points.

Original: However, geometric notions like length, distance, and angles do exist; these are obtained through the Riemannian metric for the space.
Revised: However, geometric notions like length, distance, and angles do exist; these are obtained through the Riemannian metric for the space.

Original: Points p in Smooth manifolds M are equipped with a tangent space denoted T p M .
Revised: Points p in Smooth manifolds M are equipped with a tangent space denoted T p M .

Original: The elements (tangent vectors v ∈ T p M ) in this space are used for linear approximations of a function at a point.
Revised: The elements (tangent vectors v ∈ T p M ) in this space are used for linear approximations of a function at a point.

Original: The shortest paths in a Riemannian manifold are called geodesics.
Revised: The shortest paths in a Riemannian manifold are called geodesics.

Original: Each tangent vector x ∈ T p M is equivalent to a particular geodesic that takes p to a point q .
Revised: Each tangent vector x ∈ T p M is equivalent to a particular geodesic that takes p to a point q .

Original: Speciﬁcally, the exponential map is the operation exp : T p M → M that takes p to the point q = exp p ( v ) that the tangent vector v points to.
Revised: Speciﬁcally, the exponential map is the operation exp : T p M → M that takes p to the point q = exp p ( v ) that the tangent vector v points to.

Original: In addition, the length (cid:107) v (cid:107) of the tangent vector is also the distance d ( p, q ) on the manifold.
Revised: In addition, the length (cid:107) v (cid:107) of the tangent vector is also the distance d ( p, q ) on the manifold.

Original: This operation can be reversed with the log map, which, given q ∈ M , provides the tangent vector v = log p ( q ) .
Revised: This operation can be reversed with the log map, which, given q ∈ M , provides the tangent vector v = log p ( q ) .

Original: The geodesic regression model is the following.
Revised: The geodesic regression model is the following.

Original: Set some intercept p ∈ M .
Revised: Set some intercept p ∈ M .

Original: Scalars x i ∈ R are selected according to some distribution.
Revised: Scalars x i ∈ R are selected according to some distribution.

Original: Without noise, the output points are selected along a geodesic through p parametrized by β : y i = exp p
Revised: Without noise, the output points are selected along a geodesic through p parametrized by β : y i = exp p

Original: ( x i × β ) , where β is the weight vector, a tangent vector in T p ( M ) that is not known.
Revised: ( x i × β ) , where β is the weight vector, a tangent vector in T p ( M ) that is not known.

Original: This is a noiseless model; the more interesting problem allows for noise, generalizing the typical situation in linear regression.
Revised: This is a noiseless model; the more interesting problem allows for noise, generalizing the typical situation in linear regression.

Original: This noise is added using the following approach.
Revised: This noise is added using the following approach.

Original: For notational convenience, we write exp p ( v ) as exp( p, v ) .
Revised: For notational convenience, we write exp p ( v ) as exp( p, v ) .

Original: Then, the noisy y are y i =
Revised: Then, the noisy y are y i =

Original: exp(exp
Revised: exp(exp

Original: ( p, x i × β ) , ε i ) , where ε i ∈ T exp( p,x i ) M .
Revised: ( p, x i × β ) , ε i ) , where ε i ∈ T exp( p,x i ) M .

Original: This notion generalizes adding zero-mean Gaussian noise to the y ’s in conventional linear regression.
Revised: This notion generalizes adding zero-mean Gaussian noise to the y ’s in conventional linear regression.

Original: The equivalent of least squares estimation is then given by ˆ p, ˆ β = arg min q,α (cid:80) ni
Revised: The equivalent of least squares estimation is then given by ˆ p, ˆ β = arg min q,α (cid:80) ni

Original: =1 d (exp( q, αx i ) , y i ) 2 .
Revised: =1 d (exp( q, αx i ) , y i ) 2 .

Original: Semantic Dependency Parsing We used datasets on Czech and English taken from the Universal Dependencies Nivre et al.
Revised: Semantic Dependency Parsing We used datasets on Czech and English taken from the Universal Dependencies Nivre et al.

Original: (2020) repository.
Revised: (2020) repository.

Original: The labeling functions were Stanza Qi et al.
Revised: The labeling functions were Stanza Qi et al.

Original: (2020) pre-trained semantic dependency parsing models.
Revised: (2020) pre-trained semantic dependency parsing models.

Original: These pre-trained models were trained over other datasets from the same language.
Revised: These pre-trained models were trained over other datasets from the same language.

Original: For the Czech experiments, these were the models cs.cltt, cs.fictree, cs.pdt , while for English, they were taken from en.gum, en.lines, en.partut, en.ewt .
Revised: For the Czech experiments, these were the models cs.cltt, cs.fictree, cs.pdt , while for English, they were taken from en.gum, en.lines, en.partut, en.ewt .

Original: The metric is the standard unlabeled attachment score (UAS) metric used for unlabeled dependency parsing.
Revised: The metric is the standard unlabeled attachment score (UAS) metric used for unlabeled dependency parsing.

Revised: Finally, we used access to a subset of labels to compute expected distances, as in the label model variant in Chen et al.

Original: Movies dataset processing In order to have access to ample features for learning to rank and regression tasks, we combined IMDb (imd) and TMDb(tmd, 2017) datasets based on movie id.
Revised: Movies dataset processing In order to have access to ample features for learning to rank and regression tasks, we combined IMDb (imd) and TMDb(tmd, 2017) datasets based on movie id.

Original: In the IMDb dataset, we mainly used movie metadata, which has information chieﬂy about the indirect index of the popularity (e.g. the number of director facebook likes) and movie (e.g. genres).
Revised: In the IMDb dataset, we mainly used movie metadata, which has information chieﬂy about the indirect index of the popularity (e.g. the number of director facebook likes) and movie (e.g. genres).

Original: The TMDb dataset gives information mainly about the movie, such as runtime and production country.
Revised: The TMDb dataset gives information mainly about the movie, such as runtime and production country.

Original: For ranking and regression, we chose vote average column from TMDb dataset as a target feature.
Revised: For ranking and regression, we chose vote average column from TMDb dataset as a target feature.

Original: In rankings, we converted the vote average column into rankings so the movie with the highest review has a top ranking.
Revised: In rankings, we converted the vote average column into rankings so the movie with the highest review has a top ranking.

Original: We excluded the movie scores in IMDb dataset from input features since it is directly related to TMDb vote average .
Revised: We excluded the movie scores in IMDb dataset from input features since it is directly related to TMDb vote average .

Original: But we later included
Revised: But we later included

Original: IMDb, Rotten Tomatoes, MovieLens ratings as weak labels in Table 2.
Revised: IMDb, Rotten Tomatoes, MovieLens ratings as weak labels in Table 2.

Original: After merging the two datasets, we performed feature engineering as follows.
Revised: After merging the two datasets, we performed feature engineering as follows.

Original: One-hot encoding: color, content rating, original language, genres, status2.
Revised: One-hot encoding: color, content rating, original language, genres, status

Original: Top (frequency) 0.5% one-hot encoding: plotkeywords, keywords, production companies, actors 3.
Revised: Top (frequency) 0.5% one-hot encoding: plot

Original: Count: productioncountries, spoken languages 4.
Revised: production

Original: Merge (add) actor 1 facebook likes, actor 2 facebook likes, actor 3 facebook likes into actor facebook likes 5.
Revised: Merge (add) actor 1 facebook likes, actor 2 facebook likes, actor 3 facebook likes into actor facebook likes 5.

Original: Make a binary feature regarding whether the movie’s homepage is missing or not.
Revised: Make a binary feature regarding whether the movie’s homepage is missing or not.

Original: Transform date into one-hot features such as month, the day of week.
Revised: Transform date into one-hot features such as month, the day of week.

Original: We included the preprocessed dataset in supplementary. By adding  features above, we were able to enhance 
Revised: By adding the features above, we were able to enhance performance signiﬁcantly in the

Original: MSE from 0.3626 to 0.2106 in the fully-supervised regression task used asa baseline.
Revised: fully-supervised regression task used as

Original:  In the ranking experiments, we randomly sampled 5000 sets of movies as the training set, and 1000 sets of movies as the test set.
Revised: a baseline. In the ranking experiments, we randomly sampled 5000 sets of movies as the training set, and 1000 sets of movies as the test set.

Original: The number of items in an item set (ranking set) was 5.
Revised: The number of items in an item set (ranking set) was 5.

Original: Note that the movies in the training set and test set are strictly separated, i.e. if a movie appears in the training set, it is not included in the test set.
Revised: Note that the movies in the training set and test set are strictly separated, i.e. if a movie appears in the training set, it is not included in the test set.

Original: Model and hyperparameters In the ranking setup, we used 4-layer MLP with ReLU activations.
Revised: Model and hyperparameters In the ranking setup, we used 4-layer MLP with ReLU activations.

Original: Each hidden layer had 30 units and batch normalization(Ioffe & Szegedy, 2015) was applied for all hidden layers.
Revised: Each hidden layer had 30 units and batch normalization(Ioffe & Szegedy, 2015) was applied for all hidden layers.

Original: We used the SGD optimizer with ListMLE loss (Xia et al., 2008); the learning rate was 0.01.
Revised: We used the SGD optimizer with ListMLE loss (Xia et al., 2008); the learning rate was 0.01.

Original: In the regression experiments, we used gradient boosting regression implemented in sklearn with n estimators =250.
Revised: In the regression experiments, we used gradient boosting regression implemented in sklearn with n estimators =250.

Original: Other than n estimators , we used the default hyperparameters of sklearn implementation.
Revised: Other than n estimators , we used the default hyperparameters in sklearn’s implementation.

Original: BoardGameGeek data processing In the BoardGameGeek dataset (boa, 2020), we used metadata of board games only.
Revised: BoardGameGeek data processing In the BoardGameGeek dataset (boa, 2020), we used metadata of board games only.

Original: Since this dataset showed enough model performance without additional feature engineering, we used the existing features: yearpublished, minplayers, maxplayers, playingtime, minplaytime, maxplaytime, minage, owned, trading, wanting, wishing, numcomments, numweights, averageweight .
Revised: Since this dataset showed enough model performance without additional feature engineering, we used the existing features: yearpublished, minplayers, maxplayers, playingtime, minplaytime, maxplaytime, minage, owned, trading, wanting, wishing, numcomments, numweights, averageweight .

Original: The target variable was average ratings ( average ) for regression, and board game ranking ( Board Game Rank ).
Revised: The target variable was average ratings ( average ) for regression, and board game ranking ( Board Game Rank ).

Original: Note that ‘Board Game Rank‘ cannot be directly calculated from average ratings.
Revised: Note that ‘Board Game Rank‘ cannot be directly calculated from average ratings.

Original: BoardGameGeek has its own internal equation to determine the ranking of board games.
Revised: BoardGameGeek has its own internal formula to determine the ranking of board games.

Original: When we used the features above, we obtained a MSE Of 0.2837 in the fully-supervised regression task.

Original: In the ranking setup, we randomly sampled 5000 board game sets as the training set, and 1000 board game sets as the test set.
Revised: In the ranking setup, we randomly sampled 5000 board game sets as the training set, and 1000 board game sets as the test set.

Original: Simulated LFs generation in ranking and regression In rankings, we sampled from λ a ( i ) ∼
Revised: Simulated LFs generation in ranking and regression In rankings, we sampled from λ a ( i ) ∼

Original: Z e − β a d τ ( π,Y i ) with β a .
Revised: Z e − β a d τ ( π,Y i ) with β a .

Original: For more realistic heterogeneous LFs, 1/3 (good) LFs β a are sampled from
Revised: For more realistic heterogeneous LFs, 1/3 (good) LFs β a are sampled from

Original: Uniform (0 . 2 , 1) , and 2/3 (bad) LFs’ β a are sampled from Uniform (0 .
Revised: Uniform (0 . 2 , 1) , and 2/3 (bad) LFs’ β a are sampled from Uniform (0 .

Original: Note that higher value of β a yields less noisy weak labels.
Revised: Note that higher value of β a yields less noisy weak labels.

Original: In regression, we used the conditional distribution Λ | Y to generate[Figure] samples of Λ .
Revised: In regression, we used the conditional distribution Λ | Y to generate samples of Λ .

Original: Speciﬁcally, where Λ | Y ∼ N (¯ µ, ¯Σ) , where ¯ µ = Σ Λ Y Σ − 1 Y y and ¯Σ =
Revised: Speciﬁcally, where Λ | Y ∼ N (¯ µ, ¯Σ) , where ¯ µ = Σ Λ Y Σ − 1 Y y and ¯Σ =

Original: Σ Λ − Σ Λ Y Σ − 1 Y Σ Y Λ , from the assumption (Λ , Y ) ∼ N (0 , Σ) .
Revised: Σ Λ − Σ Λ Y Σ − 1 Y Σ Y Λ , from the assumption (Λ , Y ) ∼ N (0 , Σ) .

Original: G A DDITIONAL S YNTHETIC E XPERIMENTSG. R ANKING
Revised: I A DDITIONAL S YNTHETIC E XPERIMENTS

Revised: We present several additional synthetic experiments, including results for partial ranking labeling functions and for parameter recovery in regression settings.

Revised: I. R ANKING

Original: To check whether our algorithm works well under different conditions,  additional experimentswere performed with slightly different parameters.
Revised: To check whether our algorithm works well under different conditions, we performed additional experiments with varied parameters.

Original: Also, we included partial ranking experiment here.
Revised: In addition, we performed a partial ranking experiment.

Revised: [Figure]

Original:  Ranking synthetic data generation First, n true rankings Y are sampled uniformly at random.
Revised: [Figure] Ranking synthetic data generation First, n true rankings Y are sampled uniformly at random.

Original: In the full ranking setup, each LF π i is sampled from the Mallows λ a ( i ) ∼ 1 Z e − β a d τ ( π,Y i ) with parameters β a , Y i and in the partial ranking setup it is sampled from a selective Mallows with parameters
Revised: In the full ranking setup, each LF π i is sampled from the Mallows λ a ( i ) ∼ 1 Z e − β a d τ ( π,Y i ) with parameters β a , Y i and in the partial ranking setup it is sampled from a selective Mallows with parameters

Original: β a , Y i , S i
Revised: β a , Y i , S i

Original: where each
Revised: where each

Original: S i ⊆
Revised: S i ⊆

Original: [ d ] is chosen randomly while ensuring that each x ∈ [ d ] appears in at least  p fraction of these subsets.
Revised: [ d ] is chosen randomly while ensuring that each x ∈ [ d ] appears in at least a p fraction of these subsets.

Original: Higher p corresponds to dense partial rankings and smaller p leads to sparse partial rankings.
Revised: Higher p corresponds to dense partial rankings and smaller p leads to sparse partial rankings.

Original: We generate 18 LFs with 10 of then having β a ∼ Uniform (0 . 1 , 0 . 2) and rest have β a ∼ Uniform (2 , 5) .
Revised: We generate 18 LFs with 10 of then having β a ∼ Uniform (0 . 1 , 0 . 2) and rest have β a ∼ Uniform (2 , 5) .

Original: This was done in order to model LFs of different quality.
Revised: This was done in order to model LFs of different quality.

Original: These LFs are then randomly shufﬂed so that the order in which they are added is not a factor.
Revised: These LFs are then randomly shufﬂed so that the order in which they are added is not a factor.

Original: For the partial rankings setup we use the same process to get β a and randomly generate S i according to the sparsity parameter p .
Revised: For the partial rankings setup we use the same process to get β a and randomly generate S i according to the sparsity parameter p .

Original: Keeping the LFs parameters we run the experiments forrandom trials and observe mean and standard deviations.
Revised: For a set of LFs parameters we run the experiments forrandom trials and record the mean and standard deviation.

Original: Full Ranking Experiments Figure 6 shows synthetic data results without an end model, i.e., just using the inference procedure as an estimate of the label.
Revised: Full Ranking Experiments Figure 6 shows synthetic data results without an end model, i.e., just using the inference procedure as an estimate of the label.

Original: We report the ‘Unweighted’ Kemeny baseline that ignores differing accuracies.
Revised: We report the ‘Unweighted’ Kemeny baseline that ignores differing accuracies.

Original: ‘Estimated Weights’ uses our approach, while ‘Optimal Weights’ is based on an oracle with access to the true θ a parameters.
Revised: ‘Estimated Weights’ uses our approach, while ‘Optimal Weights’ is based on an oracle with access to the true θ a parameters.

Original: As expected, synthesis with estimated parameters improves on the standard Kemeny baseline.
Revised: As expected, synthesis with estimated parameters improves on the standard Kemeny baseline.

Original: The improvement for the full rankings case (top) is higher for fewer LFs; this is intuitive, as adding more LFs globally improves estimation even when accuracies differ.
Revised: The improvement for the full rankings case (top) is higher for fewer LFs; this is intuitive, as adding more LFs globally improves estimation even when accuracies differ.

Original: Partial Ranking Experiments In the synthetic data partial ranking setup, we vary the value of p (observation probability) from 0 .
Revised: Partial Ranking Experiments In the synthetic data partial ranking setup, we vary the value of p (observation probability) from 0 .

Original: 9 (dense) to 0 .
Revised: 9 (dense) to 0 .

Original: 2 (sparse) and apply our extension of the inference method tp partial rankings.
Revised: 2 (sparse) and apply our extension of the inference method to partial rankings.

Original: Figure 7 shows the plots obtained.
Revised: Figure 7 shows the results obtained.

Original: Our observations in terms of unweighted vs weighted[Figure] aggregation remain consistent here with the full rankings setup.
Revised: Our observations in terms of unweighted vs weighted aggregation remain consistent here with the full rankings setup.

Original: This suggests that the universal approach can provide the same type of gains in the partial rankings.
Revised: This suggests that the universal approach can provide the same type of gains in the partial rankings.

Original: G. R EGRESSION
Revised: I. R EGRESSION

Original: For regression, we reports a synthetic experiment to show how our algorithm works well by properly estimating parameteres necessary in inference.
Revised: Similarly, we performed a synthetic experiment to show how our algorithm performs in parameter recovery.

Original: Regression synthetic data generation For simplicity, we generated synthetic data with a linear relationship.
Revised: Regression synthetic data generation The data generation model is linear Y =

Original: Denote the data as ( X, Y ) , where X ∈ R q and Y ∈ R .The linear relationship is [Equation]
Revised: β T X , where our data is given by ( X, Y ) with X ∈ R q and Y ∈ R .

Original: We generate n such samples.
Revised: We generate n such samples.

Original: Note that we did not add a noise variable ε ∼ N  here since we do not directly interact with Y ; the noise exists instead in the labeling functions (i.e., the weak labels).
Revised: Note that we did not add a noise variable ε ∼ N (0 , σ 2 ) here since we do not directly interact with Y ; the noise exists instead in the labeling functions (i.e., the weak labels).

Original:  Parameter estimation in synthetic regression experiments Figure 8 reports results on synthetic data capturing label model estimation error for the accuracy and correlation parameters ( µ, σ ) and for directly estimating the label Y .
Revised: [Figure] Parameter estimation in synthetic regression experiments Figure 8 reports results on synthetic data capturing label model estimation error for the accuracy and correlation parameters ( µ, σ ) and for directly estimating the label Y .

Original: As expected, estimation improves as the number of samples increases.
Revised: As expected, estimation improves as the number of samples increases.

Original: The top-right plot is particularly intuitive: here, our improved inference procedure vastly improves over naive averaging [-Samples 
 M u p a r a m e rr o r , Lestimation error 
-]
Revised: The top-right plot is particularly intuitive: here, our improved inference procedure vastly improves over naive averaging [-Samples 
 M u p a r a m e rr o r , Lestimation error 
-]as it accesses sufﬁciently many samples to estimate the label model itself.

Original: [-Samples 
-]
Revised: [-Samples 
-]

Original: T o t , p a r a m e rr o r ,

Original: [-Lestimation error 
-]
Revised: [-Lestimation error 
-]

Original: [-Samples 
-]
Revised: [-Samples 
-]

Original: Y v a l u e e rr o r ,

Original: M S E

Original: Label estimation error

Original: Baseline

Original: [-Our Approach (a) Number of samples 
-]
Revised: [-Our Approach (a) Number of samples 
-]

Original: [-Num LFs (m) 2 × 103 × 104 × 106 × 
-]
Revised: [-Num LFs (m) 2 × 103 × 104 × 106 × 
-]

Original: M u p a r a m e rr o r ,

Original: [-Lestimation errors 
-]
Revised: [-Lestimation errors 
-]

Original: [-Num LFs (m) 
-]
Revised: [-Num LFs (m) 
-]

Original: T o t , p a r a m e rr o r ,

Original: [-Lestimation error 
-]
Revised: [-Lestimation error 
-]

Original: [-Num LFs (m) 
-]
Revised: [-Num LFs (m) 
-]

Original: Y v a l u e e rr o r ,

Original: M S E

Original: Label estimation error

Original: Baseline

Original: Our Approach (

Original: [Figure]

Original: Number of LFs

Original: [Figure]

Original: [Figure]

Original: as it accesses sufﬁciently many samples to estimate the label model itself.

Original: On the bottom, we observe, as expected, that label estimation signiﬁcantly improves with access to more labels.
Revised: On the bottom, we observe, as expected, that label estimation signiﬁcantly improves with access to more labels.

Original: H A DDITIONAL REAL LF E XPERIMENTS AND R ESULTSH. R ESULT ON R EGRESSION
Revised: J A DDITIONAL R EAL LF E XPERIMENTS AND R ESULTS

Original: In Figure [Figure] we show the results of end-model performance on the regression setting.
Revised: We present a few more experiments with different types of labeling functions.

Original: H. B OARD G AME G EEK DATASET
Revised: J. B OARD G AME G EEK DATASET

Original: In the board games dataset, we built labeling functions using simple programs expressing heuristics.
Revised: In the board games dataset, we built labeling functions using simple programs expressing heuristics.

Original: For regression, we picked several continuous features and scaled them to the rangeof and removed outliers.
Revised: For regression, we picked several continuous features and scaled them to a range and removed outliers.

Original: Similarly, for rankings, we picked what we expected to be meaningful features and produced rankings based on them.
Revised: Similarly, for rankings, we picked what we expected to be meaningful features and produced rankings based on them.

Original: The selected features were [’owned’, ’trading’, ’wanting’, ’wishing’, ’numcomments’].
Revised: The selected features were [’owned’, ’trading’, ’wanting’, ’wishing’, ’numcomments’].

Original: [Figure]
Revised: [Figure]

Original: examples
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Instead, when the training data was picked based on the residual of linear regression (bad subset scenario)
Revised: [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: we can show the performance with
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: In this case, we observed that despite not having access to any labels, we can produce performance similar to fully-supervised training on 5-10% of the true labels.
Revised: In this case, we observed that despite not having access to any labels, we can produce performance similar to fully-supervised training on 5-10% of the true labels.

Original: We expect that further LF development will produce even better performance, and are working on developing these labeling functions.
Revised: We expect that further LF development will produce even better performance.

Original: H. MSLR-WEB10K
Revised: J. MSLR-WEB10K

Original: To  illustrate the strength of our approach we were able to run anadditional experiment using unsupervised learning methods in information retrieval (such as BM25) as weak supervision sources.
Revised: To further illustrate the strength of our approach we ran an experiment using unsupervised learning methods in information retrieval (such as BM25) as weak supervision sources.

Original: The task is information retrieval, the dataset is MSLR-WEB10Kmsl, and the model and training details are identical to our paper’s experiments.
Revised: The task is information retrieval, the dataset is MSLR-WEB10Kmsl, and the model and training details are identical to the other experiments.

Original: We used several labeling functions including BM25 and relied on our framework to integrate these.
Revised: We used several labeling functions including BM25 and relied on our framework to integrate these.

Original: The labeling functions were written over BM25 and features such as covered query term number, covered query term ratio, boolean model, vector space model, LMIR.ABS, LMIR.DIR, LMIR.JM, and query-url click count.
Revised: The labeling functions were written over BM25 and features such as covered query term number, covered query [Table] [Table]

Original: As expected, the synthesis of multiple sources produced better performance than BM25 Dehghani et al.
Revised: As expected, the synthesis of multiple sources produced better performance than BM25 Dehghani et al.

Original: (2017) alone.
Revised: (2017) alone.

Original: Despite not using any labels, we outperform training on 10% of the data with true labels.
Revised: Despite not using any labels, we outperform training on 10% of the data with true labels.

Original: This suggests that our framework for 
Revised: This suggests that our framework for integrating multiple sources is a better choice than either hand-labeling or using a single source of weak supervision to provide weak labels. Below, for Kendall tau, lower is better. [Table] a lot of tie and the number of items is not uniform examples, we sampled the examples with unique scores (0, 1, 2, 3, 4).

Revised: Also, in each example, items are randomly chosen so that each score occurs only once in each item set.