Lifting Weak Supervision To Structured Prediction
Abstract
Weak supervision (WS) is a rich set of techniques that produce pseudolabels by aggregating easily obtained but potentially noisy label estimates from a variety of sources.
WS is theoretically well understood for binary classification, where simple approaches enable consistent estimation of pseudolabel noise rates.
Using this result, it has been shown that downstream models trained on the pseudolabels have generalization guarantees nearly identical to those trained on clean labels.
While this is exciting, users often wish to use WS for structured prediction , where the output space consists of more than a binary or multi-class label set: e.g. rankings, graphs, manifolds, and more.
Do the favorable theoretical properties of WS for binary classification lift to this setting?
We answer this question in the affirmative for a wide range of scenarios.
For labels taking values in a finite metric space, we introduce techniques new to weak supervision based on pseudo-Euclidean embeddings and tensor decompositions, providing a nearly-consistent noise rate estimator.
For labels in constant-curvature Riemannian manifolds, we introduce new invariants that also yield consistent noise rate estimation.
In both cases, when using the resulting pseudolabels in concert with a flexible downstream model, we obtain generalization guarantees nearly identical to those for models trained on clean data.
Several of our results, which can be viewed as robustness guarantees in structured prediction with noisy labels, may be of independent interest.
Introduction
Weak supervision (WS) is an array of methods used to construct pseudolabels for training supervised models in label-constrained settings.
The standard workflow [RSW + 16, RBE + 18, FCS + 20] is to assemble a set of cheaply-acquired labeling functions—simple heuristics, small programs, pretrained models, knowledge base lookups—that produce multiple noisy estimates of what the true label is for each unlabeled point in a training set.
These noisy outputs are modeled and aggregated into a single higher-quality pseudolabel.
Any conventional supervised end model can be trained on these pseudolabels.
This pattern has been used to deliver excellent performance in a range of domains in both research and industry settings [DRS + 20, RNGS20, SLB20], bypassing the need to invest in large-scale manual labeling.
Importantly, these successes are usually found in binary or small-cardinality classification settings.
While exciting, users often wish to use weak supervision in structured prediction (SP) settings, where the output space consists of more than a binary or multiclass label set [BHS + 07, KL15].
In such cases, there exists meaningful algebraic or geometric structure to exploit.
Structured prediction includes, for example, learning rankings used for recommendation systems [KAG18], regression in metric spaces [PM19], learning on manifolds [RCMR18], graph-based learning [GS19], and more.
An important advantage of WS in the standard setting of binary classification is that it sometimes yields models with nearly the same generalization guarantees as their fully-supervised counterparts.
Indeed, the penalty for using pseudolabels instead of clean labels is only a multiplicative constant.
This is a highly favorable tradeoff since acquiring more unlabeled data is easy.
This property leads us to ask the key question for this work: does weak supervision for structured prediction preserve generalization guarantees?
We answer this question in the affirmative, justifying the application of
WS to settings far from its current use.
Generalization results in WS rely on two steps [RHD + 19, FCS + 20]: (i) showing that the estimator used to learn the model of the labeling functions is consistent, thus recovering the noise rates for these noisy voters, and (ii) using a noise-aware loss to de-bias end-model training [NDRT13].
Lifting these two results to structured prediction is challenging.
The only available weak supervision technique suitable for SP is that of [SLV + 22].
It suffers from several limitations.
First, it relies on the availability of isometric embeddings of metric spaces into R d —but does not explain how to find these.
Second, it does not tackle downstream generalization at all.
We resolve these two challenges.
We introduce results for a wide variety of structured prediction problems, requiring only that the labels live in some metric space.
We consider both finite and continuous (manifold-valued) settings.
For finite spaces, we apply two tools that are new to weak supervision.
The approach we propose combines isometric pseudo-Euclidean embeddings with tensor decompositions —resulting in a nearlyconsistent noise rate estimator.
In the continuous case, we introduce a label model suitable for the so-called model spaces —Riemannian manifolds of constant curvature—along with extensions to even more general spaces.
In both cases, we show generalization results when using the resulting pseudolabels in concert with a flexible end model from [CRR16, RCMR18].
Contributions:
• New techniques for performing weak supervision in finite metric spaces based on isometric pseudo-Euclidean embeddings and tensor decomposition algorithms, • Generalizations to manifold-valued regression in constant-curvature manifolds, • Finite-sample error bounds for noise rate estimation in each scenario, • Generalization error guarantees for training downstream models on pseudolabels, • Experiments confirming the theoretical results and showing improvements over [SLV + 22].
Background and Problem Setup
Our goal is to theoretically characterize how well learning with pseudolabels (built with weak supervision techniques) performs in structured prediction.
We seek to understand the interplay between the noise in WS sources and the generalization performance of the downstream structured prediction model.
We provide brief background and introduce our problem and some useful notation.
2.1 Structured Prediction
Structured prediction (SP) involves predicting labels in spaces with rich structure.
Denote the label space by Y .
Conventionally Y is a set, e.g., Y = {− 1 , +1 } for binary classification.
In the SP setting,
Y has some additional algebraic or geometric structure.
In this work we assume that Y is a metric space with metric (distance)
d Y .
This covers many types of problems, including
• Rankings, where Y = S ρ , the symmetric group on { 1 , . . .
, ρ } , i.e., labels are permutations, • Graphs, where Y =
G ρ , the space of graphs with vertex set V = { 1 , . . .
, ρ } , • Riemannian manifolds, including Y = S d , the sphere, or H d , the hyperboloid.
Learning and Generalization in Structured Prediction In conventional supervised learning we have a dataset { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } of i.i.d. samples drawn from distribution ρ over X × Y .
As usual, we seek to learn a model that generalizes well to points not seen during training.
Let F = { f : X (cid:55)→ Y} be a family of functions from X to Y .
Define the risk R ( f ) for f ∈ F and f ∗ as
[Equation]
[Equation]
R [Equation] [Equation] For a large class of settings (including all of those we consider in this paper), [CRR16, RCMR18] have shown that the estimator ˆ f is consistent:
[Equation]
where α ( x ) = ( K + ν I ) − 1 K x .
Here, K is the kernel matrix for a p.d. kernel k : X × X → R , so that K i,j = k ( x i , x j ) , ( K x ) i
= k ( x, x i ) ,
and ν is a regularization parameter.
The procedure here is to first compute the weights α
and then to perform the optimization in (2) to make a prediction.
An exciting contribution of [CRR16, RCMR18] is the generalization bound
[Equation]
that holds with high probability, as long as there is no label noise.
The key question we tackle is does the use of pseudolabels instead of true labels y
i affect the generalization rate?
2.2 Weak Supervision
In WS, we cannot access any of the ground-truth labels y i .
Instead we observe for each x i the noisy votes λ 1 ,i , . . .
, λ m,i .
These are m weak supervision outputs provided by labeling functions (LFs) s a , where s a : X → Y and λ a,i =
s a ( x i ) .
A two step process is used to construct pseudolabels.
First, we learn a noise model (also called a label model) that determines how reliable each source s a is.
That is, we must learn θ for P θ ( λ 1 , λ 2 , . . .
, λ m | y ) —without having access to any samples of y .
Second, the noise model is used to infer a distribution (or its mode) for each point: P θ ( y i | λ 1 ,i , . . .
, λ m,i ) .
We adopt the noise model from [SLV + 22], which is suitable for our SP setting:
[Equation]
Z is the normalizing partition function, θ = [ θ 1 , . .
, θ m ] T > 0 are canonical parameters, and E is a set of correlations.
The model can be described in terms of the mean parameters
E [ d 2 Y ( λ a , y )] .
Intuitively, if θ a is large, the typical distance from λ a to y is small and the LF is reliable; if θ a is small, the LF is unreliable.
This model is appropriate for several reasons. It is an exponential family model with useful theoretical properties. It subsumes popular special cases of noise, including, for regression, zero-mean multivariate Gaussian noise; for permutations, a generalization of the popular Mallows model; for the binary case, it produces a close relative of the Ising model.
Our goal is to form estimates ˆ θ in order to construct pseudolabels.
One way to build such pseudolabels is to compute ˜ y = arg min
z ∈Y 1 /m (cid:80) ma
=
ˆ θ a d 2 Y ( z, λ a ) .
Observe how the estimated parameters ˆ θ a are used to weight the labeling functions, ensuring that more reliable votes receive a larger weight.
We are now in a position to state the main research question for this work:
Do there exist estimation approaches yielding ˆ θ that produce pseudolabels ˜ y
that maintain the same generalization error rate O ( n − 1 / 4 ) when used in (2) , or a modified version of (2) ?
Noise Rate Recovery in Finite Metric Spaces
In the next two sections we handle finite metric spaces.
Afterwards we tackle continuous (manifoldvalued) spaces.
We first discuss learning the noise parameters θ , then the use of pseudolabels. Roadmap For finite metric spaces with |Y| = r , we apply two tools new to weak supervision.
First, we embed Y into a pseudo-Euclidean space [Gol85].
These spaces generalize Euclidean space, enabling isometric (distance-preserving) embeddings for any metric.
Using pseudo-Euclidean spaces make our analysis slightly more complex, but we gain the isometry property, which is critical.
Second, we form three-way tensors from embeddings of observed labeling functions. Applying tensor product decomposition algorithms [AGH + 14], we can recover estimates of the mean parameters ˆ E [ d 2 Y ( λ a , y )] and ultimately ˆ θ a . Finally, we reweight the model (2) to preserve generalization. Tensor Decomposition
Pseudo-Label Inference Pseudo-Euclidean Embeddings ℝ d + ℝ d − {( λ a , i λ b , i … λ m , i )} ni =
=
+ = + λ a λ b λ m
Y ̂ θ ̂ T + λ j ˜ z
i =
Y | λ a , i ,… λ m , i ; ̂ θ ̂ T − ˜ z
i
End Model Training {( x i ,˜
z
i )} ni =
LF Outputs {( λ a , i λ b , i … λ m , i )} ni = λ
The intuition behind this approach is the following.
First, we need a technique that can provide consistent or nearly-consistent estimates of the parameters in the noise model. Second, we need to handle any finite metric space.
Techniques like the one introduced in [FCS + 20] handle the first—but do not work for generic finite metric spaces, only binary labels and certain sequences.
Techniques like the one in [SLV + 22] handle any metric space—but only have consistency guarantees in highly restrictive settings (e.g., it requires an isometric embedding, that the distribution over the resulting embeddings is isomorphic to certain distributions, the true label only takes on two values). Pseudo-Euclidean embeddings used with tensor decomposition algorithms meet both requirements
3.1 Pseudo-Euclidean Embeddings
Our first task is to embed the metric space into a continuous space—enabling easier computation and potential dimensionality reduction.
A standard approach is multi-dimensional scaling (MDS) [KW78], which embeds Y into R d .
A downside of MDS is that not all metric spaces embed (isometrically) into Euclidean space, as the square distance matrix D must be positive semi-definite.
A simple and elegant way to overcome this difficulty is to instead use pseudo-Euclidean spaces for embeddings.
These pseudo-spaces do not require a p.s.d.
inner product.
As an outcome, any finite metric space can be embedded into a pseudo-Euclidean space with no distortion [Gol85]—so that distances are exactly preserved.
Such spaces have been applied to similarity-based learning methods [PPD01, LRBM06, PHD + 06].
A vector u in a pseudo-Euclidean space R d + ,d − has two parts: u + ∈ R d + and u − ∈ R d − .
The dot product and the squared distance between any two vectors u , v are ⟨ u , v ⟩ ϕ = ⟨ u
+ , v + ⟩ − ⟨ u − , v − ⟩ and d 2 ϕ ( u , v ) =
|| u + − v +
||
−
||
u − − v − || 22 .
These properties enable isometric embeddings: the distance can be decomposed into two components that are individually induced from p.s.d.
inner products—and can thus be embedded via MDS.
Indeed, pseudo-Euclidean embeddings effectively run MDS for each component (see Algorithm 1 steps 4-9).
To recover the original distance, we obtain || u + − v
+ ||
and || u − − v −
|| 22 and subtract.
Example: To see why such embeddings are advantageous, we compare with a one-hot vector representation (whose dimension is |Y| ).
Consider a tree with a root node and three branches, each of which is a path with t nodes.
Let Y be the nodes in the tree with the shortest-hops distance as the metric.
The pseudo-Euclidean embedding dimension is just d = 3 ; see Appendix for more details.
The one-hot embedding dimension is d = |Y| = 3 t + 1 —arbitrarily larger!
Now we are ready to apply these embeddings to our problem.
Abusing notation, we write λ a and y for the pseudo-Euclidean embeddings of λ a , y , respectively.
We have that d 2 Y ( λ a , y ) =
d 2 ϕ
( λ a , y ) , so that there is no loss of information from working with these spaces.
In addition, we write the mean as µ a,y =
E [ λ a | y ] and the covariance as Σ a,y .
Our goal is to obtain an accurate estimate ˆ µ a,y
= ˆ E [ λ a | y ] , which we will use to estimate the mean parameters
E [ d 2 Y ( λ a , y )] .
If we could observe y , it would be easy to empirically estimate µ a,y —but we do not have access to it.
Our approach will be to apply tensor decomposition for multi-view mixtures [AGJ14].
3.2 Multi-View Mixtures and Tensor Decompositions
In a multi-view mixture model, multiple views { λ a } ma =1 of a latent variable Y are observed.
These views are independent when conditioned on Y .
We treat the positive and negative components λ + a ∈ R d + and λ − a ∈ R d − of our pseudo-Euclidean embedding as separate multi-view mixtures:
[Equation]
and
[Equation]
where µ + a,y =
E [ λ + a | y ] , µ − a,y =
E
[ λ − a | y ] and ϵ + a , ϵ − a are mean zero random vectors with covariancesd + I d + , 1 d − I d − respectively.
Here σ 2 is a proxy variance whose use is described in Assumption 3. [Equation] [Equation] [Equation]
[Equation]
( z · ˆ µ + a,y , y + ) and similarly s + b,y , s + c,y , s − a,y , s − b,y , s − c,y 14: ˆ µ + a,y = s + a,y · ˆ µ + a,y and similarly correct signs of ˆ µ + b,y , ˆ µ + c,y , ˆ µ − a,y , ˆ µ − b,y , ˆ µ − c,y 15: end for : ˜ [Equation]
return
We cannot directly estimate these parameters from observations of λ a , due to the fact that y is not observed.
However, we can observe various moments of the outputs of the LFs such as tensors of outer products of LF triplets.
We require that for each a such a triplet exists.
Then,
[Equation]
Here w y are the mixture probabilities (prior probabilities of Y ) and Y s =
{ y : w y > 0 } .
We similarly define T − and ˆ T − .
We then obtain estimates ˆ µ + a,y , ˆ µ − a,y using an algorithm from [AGH + 14] with minor modifications to handle pseudo-Euclidean rather than Euclidean space.
The overall approach is shown in Algorithm 1.
We have three key assumptions for our analysis,
Assumption 1.
The support of P Y , i.e., k = |{ y : w y > 0 }| and the label space Y is such that min [Equation] d
[Equation] , d − ) ≥ k , || µ + a,y || 2 = 1 , || µ − a,y || 2 = 1 for a ∈ [ m ] , y ∈ Y . Assumption 2. (Bounded angle between µ and y ) Let ϕ ( u , v ) denote the angle between any two vectors u , v in a Euclidean space.
We assume that ϕ ( µ + a,y , y + ) ∈ [0 , π/ 2 − c ) , ϕ ( µ − a,y , y − ) ∈ [0 , π/ 2 − c ) ∀ a ∈ [ m ] , and y ∈ Y s , for some sufficiently small c ∈ (0 , π/ 4] such that sin( c ) ≥ max( ϵ 0 ( d + ) , ϵ 0 ( d − )) , where ϵ 0 ( d ) is defined for some n > n 0 samples in (6) .
Assumption 3.
σ is such that the recovery error with model (4) is at least as large as with (3) .
These enable providing guarantees on recovering the mean vector magnitudes (1) and signs (2) and simplify the analysis (1), (3); all three can be relaxed at the expense of a more complex analysis.
Our first theoretical result shows that we have near-consistency in estimating the mean parameters in (3).
We use standard notation ˜
O ignoring logarithmic factors.
Theorem 1.
Let ˆ µ + a,y , ˆ µ − a,y be the estimates of µ + a,y , µ − a,y returned by Algorithm 1 with input ˆ T + , ˆ T − constructed using isometric pseudo-Euclidean embeddings (in R d + ,d − ).
Suppose Assumptions 1 and 2 are met, a sufficiently large number of samples n are drawn from the model in (3) , and k =
|Y s | . Then there exists a constant C 0 > 0 such that with high probability ∀ a ∈ [ m ] and y ∈ Y s ,
[Equation]
where
[Equation]
We interpret Theorem 1.
It is a nearly direct application of [AGJ14].
There are two noise cases for σ .
In the high-noise case, σ is independent of dimension d (and thus |Y| ).
Intuitively, this means the average distance balls around each LF begin to overlap as the number of points grows—explaining the multiplicative k term.
If the noise scales down as we add more embedded points, this problem is removed, as in the low-noise case.
In both cases, the second error term comes from using the algorithm of [AGH + 14] and is independent of the sampling error.
Since k = Θ( d ) , this term goes down with d .
The first error term is due to sampling noise and goes to zero in the number of samples n .
Note the tradeoffs of using the embeddings.
If we used one-hot encoding, d = |Y| , and in the high-noise case, we would pay a very heavy cost for (cid:112) d/n .
However, while sampling error is minimized when using a very small d , we pay a cost in the second error term.
This leads to a tradeoff in selecting the appropriate embedding dimension.
Generalization Error for Structured Prediction in Finite Metric Spaces
We have access to labeling function outputs
λ a,i , . . .
, λ m,i for points x i and noise rate estimates ˆ θ a , . .
, ˆ θ m .
How can we use these to infer unobserved labels y in (2)?
Our approach is based on [NDRT13, vRW18],where the underlying loss function is modified to deal with noise.
Analogously, we modify (2) in such a way that the generalization guarantee is nearly preserved.
4.1 Prediction with Pseudolabels
First, we construct the posterior distribution P ˆ θ ( Y = y | λ ) .
We use our estimated noise model
P ˆ θ ( λ | Y ) and the prior P ( Y = y ) .
We create pseudo-labels for each data point by drawing a random sample from the posterior distribution conditioned on the output of labeling functions: ˜ Z ( i ) =
˜
z i ∼ Y | λ a = λ ( i ) a , . . .
, λ m =
λ ( i ) m ; ˆ θ .
We thus observe ( x 1 , ˜ z 1 ) , . . .
, ( x n , ˜ z n ) where ˜ z i is sampled as above.
To overcome the effect of noise we create a perturbed version of the distance function using the noise rates, generalizing [NDRT13].
This requires us to characterize the noise distribution induced by our inference procedure.
In particular we seek the probability that ˜ Z = y j when the true label is y j .
This can be expressed as follows.
Let Y m denote the m -fold Cartesian product of Y and let Λ u = ( λ ( u ) 1 , . .
, λ ( u ) m ) denote its u th entry.
We write
[Equation]
We define Q ij =
P ˆ θ ( ˜ Z =
y j | Y =
y i ) using ˆ θ .
P is the noise distribution induced by the true parameters θ and Q is an approximation obtained from inference with the estimated parameters ˆ θ .
With this terminology, we can define the perturbed version of the distance function and a corresponding replacement of (2):
[Equation]
We similarly define ˜ d p , ˜ F p , ˆ f p using the true noise distribution P .
The perturbed distance ˜
d p is an unbiased estimator of the true distance.
However we do not know the true noise distribution P hence we cannot use it for prediction.
Instead we use ˜ d q .
Note that ˜
d q is no longer an unbiased estimator—its bias can be expressed as function of the parameter recovery error bound in Theorem 1.
4.2 Bounding the Generalization Error
What can we say about the excess risk R ( ˆ f q ) − R ( f ∗ ) ?
Note that compared to the prediction based on clean labels, there are two additional sources of error.
One is the noise in the labels (i.e., even if we know the true P , the quality of the pseudolabels is imperfect).
The other is our estimation procedure for the noise distribution.
We must address both sources of error.
Our analysis uses the following assumptions on the minimum and maximum singular values σ min ( P ) , σ max ( P ) and the condition number κ ( P ) of true noise matrix P and the function F .
Additional detail is provided in the Appendix.
Assumption 4.
(Noise model is not arbitrary)
The true parameters θ are such that σ min ( P ) > 0 , and the condition number κ ( P ) is sufficiently small.
Assumption 5.
(Normalized features) | α ( x ) | ≤ 1 , for all x ∈ X .
Assumption 6.
(Proxy strong convexity) The function F in (2) satisfies the following property with some β > 0 . As we move away from the minimizer of F , the function increases and the rate of increase is proportional to the distance between the points:
[Equation]
With these assumptions, we provide a generalization result for prediction with pseudolabels,
Theorem 2.
(Generalization Error )
Let ˆ f be the minimizer as defined in (2) over the clean labels and let ˆ f q (defined in (9) ) be the minimizer over the noisy labels obtained from inference in Algorithm 1.
Suppose Assumptions 4,5,6 hold.
Then for ϵ 2 = k 5 / 2 · ˜ O ( ϵ ( d + ) + ϵ ( d − )) · (cid:16) 1 + κ ( P ) σ min ( P ) (cid:17) and c 1 = 1 + √ k σ min ( P ) , with high probability,
[Equation]
Implications and Tradeoffs: We interpret each term in the bound.
The first term is present even with access to the clean labels and hence unavoidable.
The second term is the additional error we incur if we learn with the knowledge of the true noise distribution.
The third term is due to the use of the estimated noise model.
It is dominated by the noise rate recovery result in Theorem 1.
If the third term goes to 0 (perfect recovery) then we obtain the rate O ( n − 1 / 4 ) , the same as in the case of access to clean labels.
The third term is introduced by our noise rate recovery algorithm and has two terms: one dominated by ˜ O ( n − 1 / 2 ) and the other on ˜ O ( √ k/d ) (see discussion of Theorem 1).
Thus we only pay an extra additive factor O ( √ k/d ) in the excess risk when using pseudolabels.
Manifold-Valued Label Spaces: Noise Recovery and Generalization
We introduce a simple recovery method for weak supervision in constant-curvature Riemannian manifolds.
First we briefly introduce some background notation on these spaces, then provide our estimator and consistency result, then the downstream generalization result.
Finally, we discuss extensions to symmetric Riemannian manifolds, an even more general class of spaces.
Background on Riemannian manifolds The following is necessarily a very abridged background; more detail can be found in [Lee00, Tu11].
A smooth manifold M is a space where each point is located in a neighborhood diffeomorphic to R d .
Attached to each point p ∈ M is a tangent space
T M tangent space d dimensional vector space enabling use of calculus
A Riemannian manifold equips a smooth manifold with a Riemannian metric: a smoothly-varying inner product ⟨· , ·⟩ p at each point p .
This tool allows us to compute angles, lengths, and ultimately, distances d M ( p, q ) between points on the manifold as shortest-path distances.
These shortest paths are called geodesics and can be parametrized as curves γ ( t ) , where γ (0) = p , or by tangent vectors v ∈ T p M .
The exponential map operation exp : T p M → M takes tangent vectors to manifold points.
It enables switching between these tangent vectors: exp p ( v ) =
q implies that d M ( p, q ) =
∥ v ∥ .
Invariant Our first contribution is a simple invariant that enables us to recover the error parameters.
Note that we cannot rely on the finite metric-space technique, since the manifolds we consider have an infinite number of points.
Nor do we need an embedding—we have a continuous representation as-is.
Instead, we propose a simple idea based on the law of cosines.
Essentially, on average, the geodesic triangle formed by the latent variable y ∈ M and two observed LFs λ a , λ b , is a right triangle.
This means it can be characterized by the (Riemannian) version of the Pythagorean theorem:
Lemma 1.
For Y = M , a hyperbolic manifold, y ∼ P for some distribution P on M and labeling functions λ a , λ b drawn from (3) , E cosh d Y ( λ a , λ b ) =
E cosh d Y ( λ b ,
y ) E cosh d Y ( λ b , y ) , while for
Y =
M a spherical manifold, E cos d Y ( λ a , λ b ) =
E cos d Y ( λ b , y ) E cos d Y ( λ b , y ) .
These invariants enable us to easily learn by forming a triplet system.
Suppose we construct the equation in Lemma 1 for three pairs of labeling functions.
The resulting system can be solved to express E [cosh( d Y ( λ a , y ))] in terms of E cosh( d Y ( λ a , λ b )) , E cosh( d Y ( λ a , λ c )) , E cosh( d Y ( λ b , λ c )) .
Specifically,
[Equation]
Note that we can estimate ˆ E via the empirical versions of terms on the right , as these are based on observable quantities.
This is a generalization of the binary case in [FCS + 20] and the Gaussian (Euclidean) case in [SLV + 22] to hyperbolic manifolds.
A similar estimator can be obtained for spherical manifolds by replacing cosh with cos .
Using this tool, we can obtain a consistent estimator for θ a for each of a = 1 , . . .
, m .
Let Csatisfy E | ˆ E cosh( d Y ( λ a , λ b )) − E cosh( d Y ( λ a , λ b ))
| ≥ C 0 E | ˆ E d 2 Y ( λ a , λ b ))
− E d 2 Y ( λ a , λ b ) | ; that is,
C 0 reflects the preservation of concentration when moving from distribution cosh( d ) to d 2 .
Then,
Theorem 3.
Let M be a hyperbolic manifold.
Fix 0 < δ < 1 and let ∆( δ ) =
[Equation] ,i , λ b,i [Equation] ≤ ρ
[Equation] ≥ 1 − δ . Then, there exists a constant C 1 so that with probability at least 1 − δ , E |
[-− E d 2 Y ( λ a , y ) | ≤ C 1 cosh(∆( δ )) 3 / 2 /C 
 √ 2 n.-][Equation]
As we hoped, our estimator is consistent.
Note that we pay a price for a tighter bound: ∆( δ ) is large for smaller probability δ .
It is possible to estimate the size of ∆( δ ) (more generally, it is a function of the curvature).
We provide more details in the Appendix.
Next, we adapt the downstream model predictor (2) in the following way.
Let ˆ µ 2 a = ˆ E
[ d 2 Y ( λ a , y )] .
Let β = [ β 1 , . . .
, β m ] T be such that (cid:80) a β a = 1 and β minimizes (cid:80) a β 2 a ˆ µ 2 a .
Then, we set
[Equation]
We simply replace each of the true labels with a combination of the labeling functions.
With this, we can state our final result.
First, we introduce our assumptions.
Let q = arg min z ∈Y E
[ α ( x )( y ) d 2 Y ( z, y )] , where the expectation is taken over the population level distribution and α ( x )( y ) denotes the kernel at y .
Assumption 7.
(Bounded Hugging Function c.f.
[Str20]) Let q be defined as above.
For all a, b ∈ M , the hugging function at q is given by k bq ( a ) = 1 − ( ∥ log q ( a ) − log q ( b ) ∥ 2 − d 2 Y ( a, b )) /d
2 Y ( q, b ) .
We assume that k bq ( a ) is lower bounded by k min .
Assumption 8.
(Kernel Symmetry)
We assume that for all x and all v ∈ T q M , α ( x )(exp q ( v ))
=
α ( x )(exp q ( − v )) .
[Figure] The first condition provides control on how geodesic triangles behave; it relates to the curvature.
We provide more details on this in the Appendix.
The second assumption restricts us to kernels symmetric about the minimizers of the objective F .
Finally, suppose we draw ( x, y ) and ( x ′ , y ′ ) independently from P XY .
Set σ 2 o =
α
( x )( y ) E d 2 Y ( y, y ′ ) .
Theorem 4.
Let M be a complete manifold and suppose the assumptions above hold.
Then, there exist constants C 3 , C[-
-]
[Equation]
Note that as both m and n grow, as long as our worst-quality LF has bounded variance, our estimator of the true predictor is consistent.
Moreover, we also have favorable dependence on the noise rate.
This is because the only error we incur is in computing suboptimal β coefficients.
We comment on this suboptimality in the Appendix.
A simple corollary of Theorem 5 provides the generalization guarantees we sought,
Corollary 1.
Let M be a complete manifold and suppose the assumptions above hold.
Then, with high probability, R ( ˜ f ) ≤ R ( f ∗ )
+ O ( n − 14 ) .
Extensions to Other Manifolds
First, we note that all of our approaches almost immediately lift to products of constant-curvature spaces.
For example, we have that M 1 × M 2 has metric d 2 Y ( p, q ) =
d 2 M 1 ( p 1 , q 1 ) + d 2 M 2 ( p 2 , q 2 ) , where p i , q i are the projections of p, q onto the i th component.
We can go beyond products of constant-curvature spaces as well.
To do so, we can build generalizations of the law of cosines (as needed for the invariance in Lemma 1).
For example, it is possible to do so for symmetric Riemannian manifolds using the tools in [AH91]. [Figure]
Experiments
Finally, we validate our theoretical claims with experimental results demonstrating improved parameter recovery and end model generalization using our techniques over that of prior work [SLV + 22].
We illustrate both the finite metric space and continuous space cases by targeting rankings (i.e., permutations) and hyperbolic spaces.
In the case of rankings we show that our pseudo-Euclidean embeddings with tensor decomposition estimator yields stronger parameter recovery and downstream generalization than [SLV + 22]. In the case of hyperbolic regression (an example of a Riemannian manifold), we show that our estimator yields improved parameter recovery over [SLV + 22]. Finite metric spaces: Learning to rank To experimentally evaluate our tensor decomposition estimator for finite metric spaces, we consider the problem of learning to rank. We construct a synthetic dataset whose ground truth comprises n samples of two distinct rankings among the finite metric space of all length-four permutations.
We construct three labeling functions by sampling rankings according to a Mallows model, for which we obtain pseudo-Euclidean embeddings to use with our tensor decomposition estimator. In Figure 2 (top left), we show that as we increase the number of samples, we can obtain an increasingly accurate estimate of T —exhibiting the nearly-consistent behavior predicted by our theoretical claims. This leads to downstream improvements in parameter estimates, which also become more accurate as n increases. In contrast, we find that the estimates of the same parameters given by [SLV + 22] do not improve substantially as n increases, and are ultimately worse (see
Figure 2, top right).
Finally, this leads to improvements in the label model accuracy as compared to that of [SLV + 22], and translates to improved accuracy of an end model trained using synthetic samples (see Figure 2, bottom).
Riemannian manifolds: Hyperbolic regression We similarly evaluate our estimator using synthetic labels from a hyperbolic manifold, matching the setting of Section 5.
As shown in Figure 3, we find that our estimator consistently outperforms that of [SLV + 22], often by an order of magnitude.
We studied the theoretical properties of weak supervision applied to structured prediction in two general scenarios: label spaces that are finite metric spaces or constant-curvature manifolds.
We introduced ways to estimate the noise rates of labeling functions, achieving consistency or nearconsistency.
Using these tools, we established that with suitable modifications downstream structured prediction models maintain generalization guarantees.
Future directions include extending these results to even more general manifolds and removing some of the assumptions needed in our analysis. Acknowledgments
We are grateful for the support of the NSF (CCF2106707), the American Family Funding Initiative and the Wisconsin Alumni Research Foundation (WARF).
Checklist
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] 2.
Did you describe the limitations of your work?
[Yes] 3.
Did you discuss any potential negative societal impacts of your work?
[N/A] 4.
Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 5.
Did you state the full set of assumptions of all theoretical results?
[Yes] 6.
Did you include complete proofs of all theoretical results?
[Yes] See appendix