C REATING T RAINING S ETS VIA W EAK I NDIRECT S UPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
Creating labeled training sets has become one of the major roadblocks in machine learning.
To address this, recent Weak Supervision (WS) frameworks synthesize training labels from multiple potentially noisy supervision sources.
However, existing frameworks are restricted to supervision sources that share the same output space as the target task.
To extend the scope of usable sources, we formulate Weak
Indirect Supervision (WIS) , a new research problem for automatically synthesizing training labels based on indirect supervision sources that have different output label spaces.
To overcome the challenge of mismatched output spaces, we develop a probabilistic modeling approach, PLRM, which uses user-provided label relations to model and leverage indirect supervision sources.
Moreover, we provide a theoretically-principled test of the distinguishability of PLRM for unseen labels, along with a generalization bound.
On both image and text classification tasks as well as an industrial advertising application, we demonstrate the advantages of
PLRM by outperforming baselines by a margin of 2%-9%.
I NTRODUCTION
One of the greatest bottlenecks of using modern machine learning models is the need for substantial amounts of manually-labeled training data.
In real-world applications, such manual annotations are typically time-consuming, labor-intensive and static.
To reduce the efforts of annotation, researchers have proposed Weak Supervision (WS) frameworks (Ratner et al., 2016; 2018; 2019; Fu et al., 2020) for synthesizing labels from multiple weak supervision sources , e.g. , heuristics, knowledge bases, or pre-trained classifiers.
These frameworks have been widely applied on various machine learning tasks (Dunnmon et al., 2020; Fries et al., 2021; Safranchik et al., 2020; Lison et al., 2020; Zhou et al., 2020; Hooper et al., 2021; Zhan et al., 2019; Varma et al., 2019) and industrial data (Bach et al., 2019).
Among them, data programming (Ratner et al., 2016), one representative example that generalizes many approaches in the literature, represents weak supervision sources as labeling functions (LFs) and synthesizes training labels using Probabilistic Graphical Model (PGM).
Given both the increasing popularity of WS and the general increase in open-source availability of machine learning models and tools, there is a rising tide of available supervision sources that WS frameworks and practitioners could potentially leverage, including pre-trained machine learning models or prediction APIs (Chen et al., 2020; d’Andrea & Mintz, 2019; Yao et al., 2017).
However, existing WS frameworks only utilize weak supervision sources with the same label space as the target task.
This incompatibility largely limits the scope of usable sources, necessitating manual effort from domain experts to provide supervision for unseen labels.
For example, consider target task of classifying {“ dog ”, “ wolf ”, “ cat ”, “ lion ”} and a set of three weak supervision sources (e.g. trained classifiers or expert heuristics) with disjoint output spaces {“ caninae ”, “ felidae ”}, {“ domestic animals ”, “ wild animals ”} and {“ husky ”, “ bengal cat ”} respectively.
We call these types of sources indirect supervision sources .
For concreteness, we follow the general convention of data programming (Ratner et al., 2016) and refer to these sources as indirect labeling functions (ILFs) .
Despite their apparent utility, existing weak supervision methods could not directly leverage such
ILFs, as their output spaces have no overlap with the target one.
In this paper, we formulate a novel research problem that aims to leverage such ILFs automatically, minimizing the manual efforts to develop and deploy new models.
We refer to this as the Weak
Indirect Supervision (WIS) setting, a new Weak Supervision paradigm which leverages ILFs, along with the relational structures between individual labels, to automatically create training labels.
The key difficulty of leveraging ILFs is due to the mismatched label spaces.
To overcome this, we introduce pairwise relations between individual labels to the WIS setup, which are often available in structured sources (e.g. off-the-shelf Knowledge Bases (Miller, 1995; Sinha et al., 2015;
Dong et al., 2020) or large scale label hierarchies (Murty et al., 2017; The Gene Ontology Consortium, 2018; Partalas et al., 2015) for various domains), or can be provided by subject matter experts in far less time than generating entirely new sets of weak supervision sources.
For example, in the aforementioned example, we could rely on a biological species ontology to see that the unseen labels “ dog ” and “ cat ” are both subsumed by the seen label “ domestic animals ”.
Based on the label relations, we can automatically leverage the supervision sources as ILFs.
Notably, previous work (Qu et al., 2020) also leveraged a label relation graph but was focused on relation extraction task in a few-shot learning setting, while You et al.
(2020) proposed to learn label relations given data for each label in a transfer learning scenario.
In contrast, we aim to solve the target task directly and without clean labeled data.
The remaining questions are (1) how to synthesize labels based on pair-wise label relations and ILFs?
and (2) How can we know whether, given a set of ILFs and label relations, the unseen labels are distinguishable or not?
To address the first question, we develop a probabilistic label relation model (PLRM) , the first PGM for WIS which aggregates the output of ILFs and models the label relations as dependencies between random variables.
In turn, we use the learned PLRM to produce labels for training an end model.
Furthermore, we derive the generalization error bound of PLRM based on assumptions similar to previous work (Ratner et al., 2016).
The second question presents an important stumbling block when dealing with unseen labels, as we may not be able to distinguish the unseen labels given existing label relations and ILFs, resulting in an unsatisfactory synthesized training set.
To address this issue, we formally introduce the notion of distinguishability in WIS setting and theoretically establish an equivalence between: (1) the distinguishability of the label relation structure as well as the ILFs, and (2) the capability of PLRM to distinguish unseen labels.
This result then leads to a simple sanity test for preventing the model from failing to distinguish unseen labels.
In preliminary experiments, we observe a significant drop in model performance when the condition is violated.
In experiments, we make non-trivial adaptations for baselines from related settings to the new WIS problem.
On both text and image classification tasks, we demonstrate the advantages of PLRM over adapted baselines.
Finally, in a commercial advertising system where developers need to collect annotations for new ads tags, we illustrate how to formulate the training label collection as a WIS problem and apply PLRM to achieve an effective performance.
Summary of Contributions.
Our contributions are summarized as follows:
• We formulate Weak Indirect Supervision (WIS), a new research problem which synthesizes training labels based on indirect supervision sources and label relations, minimizing human efforts of both data annotation and weak supervision sources construction; • We develop the first model for WIS, the Probabilistic Label Relation Model (PLRM) with comparable statistical efficiency to previous WS frameworks and standard supervised learning; •
We introduce a new notion of distinguishability in WIS setting, and provide a simple test of the distinguishability of PLRM for unseen labels by theoretically establishing the connection between the label relation structures and distinguishability; • We showcase the potential of the WIS formulation and the effectiveness of PLRM in a commercial advertising system for synthesizing training labels of new ads tags.
On academic image and text classification tasks, we demonstrate the advantages of PLRM over baselines by quantitative experiments.
Overall, PLRM outperforms baselines by a margin of 2%-9%.
R ELATED W ORK
We briefly review related settings.
The comparison between WIS and related tasks is in Table 1.
[Table] Weak Supervision: We draw motivation from recent work which model and integrate weak supervision sources using PGMs (Ratner et al., 2016; 2018; 2019;
Fu et al., 2020) and other methods (Guan et al., 2018; Khetan et al., 2018) to create training sets.
While they assume supervision sources share the same label space as the new tasks, we aim to leverage indirect supervision sources with mismatched label spaces in a labor-free way.
Indirect Supervision: Indirect supervision arises more generally in latent-variable models for various domains (Brown et al., 1993; Liang et al., 2013; Quattoni et al., 2004; Chang et al., 2010; Zhang et al., 2019).
Very recently, Raghunathan et al.
(2016) proposed to use the linear moment method for indirect supervision, wherein the transition between desired label space Y and indirect supervision space O is known, as well as the ground truth of indirect supervisions for training.
In contrast, both are unavailable in WIS.
Theoretically, Wang et al.
(2020) developed a unified framework for analyzing the learnability of indirect supervision with shared or superset label spaces, while we focus on disjoint label spaces and the consequent unique challenge of distinguishability of unseen classes.
Zero-Shot Learning: Zero-Shot Learning (ZSL)
(Lampert et al., 2009; Wang et al., 2019) aims to learn a classifier that is able to generalize to unseen classes.
The WIS problem differentiates from
ZSL by (1) in ZSL setting, the training and test data belong to seen and unseen classes, respectively, and training data is labeled, while for WIS, both training and test data belong to unseen classes and unlabeled; (2) ZSL tends to render a classifier that could predict unseen classes given certain label information, e.g. , label attributes (Romera-Paredes & Torr, 2015), label descriptions (Srivastava et al., 2018) or label similarities (Frome et al., 2013), while WIS aims to provide training labels for unlabeled training data, allowing users to train any machine learning models, and the label relations are used only in synthesizing training labels.
P RELIMINARY : W EAK S UPERVISION
We first describe the Weak Supervision (WS) setting.
A glossary of notations used is in App. A.
Definitions and notations.
We assume a k -way classification task, and have an unlabeled dataset D consisting of m data points.
Denote by X i ∈
X the individual data point and Y i ∈ Y =
{ y 1 , . . .
, y k } the unobserved interested label of X i .
We also have n sources, each represented by a labeling function (LF) and denoted by λ j .
Each λ j outputs a label ˆ Y ji ∈ Y λ j = { ˆ y j 1 , . . . , ˆ y jk λj } on X i , where Y λ j is the label space associated with λ j and |Y λ
j | = k
λ j .
We denote the concatenation of LFs’ output as ˆ Y i =
[ ˆ Y 1 i , ˆ Y 2 i , . . .
, ˆ Y ni ] , and the union set of LFs’ label spaces as ˆ Y with | ˆ Y| = ˆ k .
Note that ˆ k is not necessarily equal to the sum over k λ j , since LFs may have overlapping label spaces.
We call ˆ y ∈ ˆ Y seen label and y ∈ Y desired labels.
In WS settings, we have Y ⊂ ˆ Y .
Notably, we assume all the involved labels come from the same semantic space.
The goal of WS.
The goal is to infer the training labels for the dataset D based on LFs, and to use them to train an end discriminative classifier f W : X → Y , all without ground truth training labels . [Table] I n p u t [Table]
M/rJSa89lMm48RQSRaLwoQjE6HsezRkihLDp5Zgopi9FZExVpgYm1HJhuAtv7xKWhdV77Jau69V6jd5HEU4gVM4Bw+uoA530IAmEBDwDK/w5ijnxXl3PhatBSefOYY/cD5/AKCakE4=</latexit> ¯ Y<latexit sha1_base64="/FJiOgBeNaC3A4eIpgMQnF9LPaY=">AAAB8HicbVBNSwMxEJ2tX7V+VT16CRbBU9ktpXosevFYwX7RriWbZtvQJLskWaEs [Figure] +O7mNza3tnfxuYW//4PCoeHzS0lGiCG2SiEeqE2BNOZO0aZjhtBMrikXAaTuY3M799hNVmkXywUxj6gs8kixkBBsrdfsBVml39lgZFEtu2V0ArRMvIyXI0BgUv/rDiCSCSkM41rrnubHxU6wMI5zOCv1E0xiTCR7RnqUSC6r9dHHwDF1YZYjCSNmSBi3U3xMpFlpPRWA7BTZjverNxf+8XmLCaz9lMk4MlWS5KEw4MhGaf4+GTFFi+NQSTBSztyIyxgoTYzMq2BC81ZfXSatS9mrl6n21VL/J4sjDGZzDJXhwBXW4gwY0gYCAZ3iFN0c5L86787FszTnZzCn8gfP5A6IekE8=</latexit> ¯ [Figure]
visualize the subsuming relation.
Finally, a label model, instantiated as a PGM, takes the ILF’s outputs and produces probabilistic labels in the target output space, which are in turn used to train an end machine learning model that can generalize beyond them.
W EAK I NDIRECT S UPERVISION
Now, we introduce the new Weak Indirect Supervision (WIS) problem.
Unlike the standard WS setting, we only have indirect labeling functions (ILFs) instead of LFs, and an additional label graph is given.
The goal of WIS remains the same as WS.
An example of WIS problem is in Fig.
Indirect Labeling Function.
In WIS, we only have indirect labeling functions (ILFs), which cannot directly predict any desired labels, i.e. , ˆ Y ∩ Y = ∅ .
Therefore, we refer to the desired labels as unseen labels.
To make it possible to leverage the ILFs, a label graph is given, which encodes pair-wise label relations between different seen and unseen labels.
Label Graph.
Concretely, a label graph G =
( V , E ) consists of (1) a set of all the labels as nodes, i.e. , V = ˆ Y ∪ Y , and (2) a set of pair-wise label relations as typed edges, i.e. , E = { ( y i , y j , t y i y j )
| t y i y j ∈
T , i < j, ∀ y i , y j ∈ V} .
Here, T is the set of label relation types and, similar to Deng et al.
(2014), there are four types of label relations: exclusive , overlapping , subsuming , subsumed , notated by t o , t e , t sg , t sd , respectively.
Notably, for any ordered pair of labels ( y i , y j ) , their label relation should fall into one of the four types.
The rationale behind these label relations is that when treating each label as a set, there are four unique set relations and each corresponds to one defined label relation respectively as shown in Fig.
For convenience, we denote the set of non-exclusive neighbors of a given label y in ˆ Y as N ( y, ˆ Y ) , i.e. , N ( y, ˆ Y ) =
{ ˆ y ∈ ˆ Y| t y ˆ y ̸ = t e } . A
B Exclusive Overlap
A
B Subsuming A B B
A
Subsumed Set relation (cid:29) <latexit sha1_base64="QqIKplT5Xx6/rl97+eSsY2psxnk=">AAACAXicbVC7SkNBEN0bjcb4umoj2Cwmgo3hXgtNo0RttItgHpCEsHczSZbsfbA7VwghNv6KjYUitjZ+g51/4+ZRaOKBgcM5M8zM8SIpNDrOt5VYWEwuLadW0qtr6xub9tZ2WYex4lDioQxV1WMapAighAIlVCMFzPckVLze1civ3IPSIgzusB9Bw2edQLQFZ2ikpr2bzV7UOYvoJT2jdfAj7GtAms027YyTc8ag88Sdkkwhlf9Mnh95xab9VW+FPPYhQC6Z1jXXibAxYAoFlzBM12MNEeM91oGaoQHzQTcG4w+G9MAoLdoOlakA6Vj9PTFgvtZ93zOdPsOunvVG4n9eLcZ2vjEQQRQjBHyyqB1LiiEdxUFbQgFH2TeEcSXMrZR3mWIcTWhpE4I7+/I8KR/n3JOcc2vSuCETpMge2SeHxCWnpECuSZGUCCcP5Im8kFfr0Xq23qz3SWvCms7skD+wPn4ABruW7A==</latexit>
A \ B = ; <latexit sha1_base64="WbehnkOmVK1pYY8rpVPPb/9EDeY=">AAACIXicbVDLTgJBEJz1ifhCPXqZCCYeCNn1gBwFL3rTRNSEJWR2aGDC7Ow602tCiBc/xIs/4EfowYPGeDN+hX/gACYqWMkk1VXd6ekKYikMuu67MzU9Mzs3n1pILy4tr6xm1tbPTJRoDlUeyUhfBMyAFAqqKFDCRayBhYGE86B7MPDPr0AbEalT7MVQD1lbiZbgDK3UyJRyubLPWUwrvoJLH8IYewYwT8u+itA3SWArWskP/J+6nMs1Mlm34A5BJ4n3TbL72c/Hm+p98biRefObEU9CUMglM6bmuTHW+0yj4BKu035iIGa8y9pQs1SxEEy9P7zwmm5bpUlbkbZPIR2qvyf6LDSmFwa2M2TYMePeQPzPqyXYKtX7QsUJguKjRa1EUozoIC7aFBo4yp4ljGth/0p5h2nG0YaatiF44ydPkrPdglcsuCc2jSMyQopski2yQzyyR/bJITkmVcLJLXkgz+TFuXOenFfnbdQ65XzPbJA
[Figure]
P ROBABILISTIC L ABEL R ELATION M ODEL
One of the key difficulties in both WS and WIS is that we do not observe the true label Y i .
Following prior work (Ratner et al., 2016; 2019; Fu et al., 2020), we use a latent variable Probabilistic Graphical
Model (PGM) for estimating Y i based on the ˆ Y i output by ILFs.
Specifically, the PGM is instantiated as a factor graph model.
This standard technique lets us describe the family of generative distributions in terms of M known dependencies/factor functions { ϕ } , and an unknown parameter Θ ∈ R M as P Θ ( · ) ∝ exp(Θ ⊤ Φ( · )) , where Φ is the concatenation of { ϕ } .
However, the unique challenge for WIS is that the dependencies { ϕ } between Y i and ˆ Y i are unknown due to the mismatches of label spaces.
We overcome these by leveraging the label graph G to build the dependencies for the PGM.
5. A B ASELINE PGM FOR WIS
In prior work (Ratner et al., 2016; Bach et al., 2017), the PGM for WS is governed by accuracy dependencies:
[Equation]
which is defined for each λ j and y ∈ Y λ j ∩Y .
However, in WIS, the ILFs cannot predict desired label y ∈ Y .
As a simple baseline approach to start, we leverage the coarse-grained exclusive/non-exclusive label relation to build a corresponding "accuracy" factor.
Specifically, for an ILF λ j and one label ˆ y ∈ Y λ j , given a desired label y ∈ Y , if ˆ y and y have non-exclusive label relation, i.e. , ˆ y ∈ N ( y, Y λ j ) we expect a certain portion of data assigned ˆ y should be labeled as y .
Thus, we treat ˆ Y j = ˆ y as a pseudo indicator of Y = y and add a pseudo accuracy dependency between them:
[Equation]
We call the PGM governed by pseudo accuracy dependencies Weak Supervision with Label Graph (WS-LG).
Notably, it can be treated as a simple adaptation of PGM for WS (Ratner et al., 2016; 2019; Fu et al., 2020) to the WIS problem.
However, such a naïve adaptation might have two drawbacks:
It does not model specific dependencies ILFs with different undesired labels.
For example, two ILFs outputting “ Husky ” and “ bulldog ” respectively would be naively modeled the same as if they both output “ Dog ”.
It can only directly model exclusive/non-exclusive label relations, ignoring the prior knowledge encoded in other relation types,
i
e. , subsuming, subsumed, or overlapping.
For example, given an unseen label “ Dog ” and some ILFs outputting “ Husky ” or “ Domestic Animals ”, WS-LG would treat all ILFs as indicators of “ Dog ”.
However, we know a “ Husky ” is of course a “ Dog ” (subsumed relation) while a “ Domestic Animals ” is not necessarily a “ Dog ” (subsuming relation).
5. P ROBABILISTIC L ABEL R ELATION M ODEL
To more directly model the full range and nuance of label relations, we propose a new probabilistic label relation model (PLRM) .
In PLRM, we explicitly model both (1) the dependency between ILF outputs and the true labels in their output spaces, i.e. their direct accuracy, and (2) the dependencies between these labels and the target unseen labels, as separate dependency types, thus explicitly incorporating the full label relation graph into our model and learning its corresponding weights.
Concretely, we augment the WS-LG model with (1) latent variables representing the assignment of the data to each seen label, and (2) label relation dependencies which capture fine-grained label relations between these output labels and desired labels.
To model seen label in ˆ Y , we introduce a binary latent random vector ¯ Y = [ ¯ Y 1 , . . , ¯ Y ˆ k ] , where ¯ Y i indicating whether the data should be assigned ˆ y i .
Then, for ILF λ j that could predict ˆ y i , we have accuracy dependency:
[Equation]
To model fine-grained label relations, for a desired label y ∈ Y and seen label ˆ y i ∈ ˆ Y , we add label relation dependencies.
We enumerate the label relation dependencies corresponding to the four label relation types, i.e. , exclusive, overlapping, subsuming, subsumed, as follows:
[Equation]
The above dependencies encode the prior knowledge of the label relations, but also allow the model to learn corresponding parameters.
For example, an exclusive label relation dependency ϕ e outputs {+-
+} when two exclusive labels are activated at the same time for the same data, otherwise 0, which reflects our prior knowledge of the exclusive label relation; and the corresponding parameter can be treated as the strength of the label relation.
Likewise, for any pair of seen labels, we add label relation dependency following the same convention.
Finally, we specify the model as:
[Equation]
Recall that Y is the unobserved true label, ¯ Y is the binary random vector, each of whose binary value ¯ Y
i reflects whether the data should be assigned seen label ˆ
y i ∈
ˆ Y , and ˆ Y is the concatenated outputs of ILFs.
Learning Objective.
We estimate the parameters ˆΘ by minimizing the negative log marginal likelihood P Θ ( ˆ Y ) for observed ILF outputs ˆ Y 1: m :
[Equation]
We follow Ratner et al.
(2016) to optimize the objective using stochastic gradient descent.
Training an End Model.
Let p ˆΘ ( Y | ˆ Y ) be the probabilistic label (i.e. distribution) predicted by learned PLRM.
We then train an end model f W : X → Y parameterized by W , by minimizing the empirical noise-aware loss (Ratner et al., 2019) with respect to ˆΘ over m unlabeled data points:
[Equation]
where ℓ ( Y, f W ( X i )) is a standard cross entropy loss.
Generalization Error Bound.
We extend previous results from (Ratner et al., 2016) to bound both the expected error of learned parameter ˆΘ and the expected risk for ˆ W .
All the proof details and description of assumptions can be found in Appendix.
Theorem 1.
Suppose that we run stochastic gradient descent to produce ˆΘ and ˆ W based on Eqs.
(2) and (3), respectively, and that our setup satisfies certain assumptions (App D.2).
Let | D | be the size of the unlabeled dataset.
Then we have
[Equation]
Interpreting the Bound.
By Theorem 1, the two errors decrease by the rate ˜ O
(1 /D ) and ˜ O
(1 /D 1 / 2 ) respectively as | D | increases.
This shows that although we trade computational efficiency for the reduction of human efforts by using complex dependencies and more latent variables, we maintain comparable statistical efficiency as previous WS frameworks and supervised learning theoretically.
D ISTINGUISHABILITY OF U NSEEN L ABELS
[Figure]
[Figure]
One unique challenge of WIS is that there may exist pairs of unseen labels which cannot be distinguished by the learned model.
For example, as shown in Fig. 3, where “ Dog ” is a seen label for which
LFs could predict for and “ Husky ” and “ Bulldog ” are unseen labels for which we want to generate training labels; however, we could not distinguish between “ Husky ” and “ Bulldog ” even though the LFs make correct predictions of seen label “ Dog ”, because both “ Husky ” and “ Bulldog ” share the same label relation to “ Dog ”.
To tackle this issue, we theoretically connect the distinguishability of unseen labels to the label relation structures and provide a testable condition for the distinguishability.
Intuitively, same label relation structures could lead to indistinguishable unseen labels as shown in Fig.
3; however, it turns out to be challenging to prove that different label relation structures could guarantee the distinguishability with respect to the model .
To illustrate, we formally define the distinguishability as below.
Definition 1 (Distinguishability) .
For any model P Θ ( Y, ¯ Y , ˆ Y ) with parameters Θ , any pair of unseen labels y i , y j ∈ Y are distinguishable w.r.t.
the model, if for a.e. Θ > 0 (element-wisely), there does NOT exist such a ˜Θ > 0 that, for ∀ ¯ Y , ˆ Y , the following equations hold
[Equation]
From the definition, we can see that the opposite of distinguishability, i.e. , indistinguishability, describes an undesired model: for any learned parameter Θ > 0 , we can always find another ˜Θ which optimizes the loss equally well (Eq.
(6)), but Eqs.
(4-5) implies whenever P Θ predict y i , P ˜Θ will predict y j instead , which reflects that the model cannot distinguish the two unseen labels.
Note that the notion of distinguishability is different from the identifiability in PGMs: the generic identifiability (Allman et al., 2015), the strongest notion of identifiability, requires the model to be identifiable up to label swapping , while the distinguishability aims to avoid the label swapping.
However, distinguishability is hard to verify since Eqs.
(4-5) and (6) need to hold for any possible configuration of ¯ Y , ˆ Y , and any pair of unseen labels.
Fortunately, for the proposed PLRM, we prove that distinguishability is equivalent to the asymmetry of the label relation structures when two conditions hold.
To state the required conditions, we first introduce the notations of consistency and informativeness to characterize the label graph and ILFs.
Consistency.
We discuss the consistency of a label graph to avoid an ambiguous or unrealistic label graph.
We interpret semantic labels
y a , y b as sets A , B , and then connect the label relations to the set relations (Fig. 2).
Given the set interpretations, we define the consistency of label graph as:
Definition 2 (Consistent Label Graph) .
A label graph G =
( Y , E ) is consistent if the induced set relations are consistent.
For example, assume Y =
{ y a , y b , y c } , and t ab = t bc =
t ca = t sg .
From t ab , t bc , we can observe that A ⫌ B ⫌ C , which contradicts to C ⫌ A implied by t ca =
t sg .
Thus, G is inconsistent.
Informativeness.
In addition, we try to describe what kind of ILF is desired.
Intuitively, an ILF is uninformative if it always "votes" for one of the desired labels.
For example, if the desired label space Y is {“ Dog ”, “ Bird ”}, then for an ILF λ 1 outputting {“ Husky ”, “ Bulldog ”}, we know “ Dog ” is non-exclusive to “ Husky ” and “ Bulldog ”, while “ Bird ” exclusive to both.
In such case, λ 1 can hardly provide information to help distinguish “ Dog ” from “ Bird ”, because it always votes for “ Dog ”.
On the other hand, a binary classifier of “ Husky ”, i.e. , λ 2 , is favorable since it could output “ Not a Husky ” to avoid consistently voting for “ Dog ”.
We can see an undesired ILF always votes for a single desired label.
To formally describe this, we define an informative ILF as:
Definition 3 (Informative ILF) .
An ILF λ j is informative if, for ∀ y ∈ Y , there exists X i ∈
D s.t.
the output of λ j on X i is not in N ( y, Y λ j ) , i.e. , ˆ Y ji ̸∈ N ( y, Y λ j ) .
Testable Conditions for Distinguishability.
Based on the introduced notations, we prove the necessary and sufficient condition for learned PLRM being able to distinguish unseen labels:
Theorem 2.
For PLRM induced from a consistent label graph, as well as informative ILFs, for any pair of y i , y j ∈ Y , they are indistinguishable, if and only if t ik = t jk for ∀ y k ∈
ˆ Y .
Theorem 2 provides users with a testable condition: for any pair of unseen labels
y i , y j , there should exist at least one seen labels y k
such that y k has different label relations to y i and y j , i.e., t ik ̸ = t jk , so that PLRM is able to distinguish y i and y j .
In preliminary experiments, we observe the violation of this condition causes a dramatic drop in overall performance (about 10 points).
Notably, based on
Theorem 2, users could theoretically guarantee the distinguishability of a pair of unseen labels by adding only one seen label and corresponding ILFs to break the symmetry.
E XPERIMENTS
We demonstrate the applicability and performance of our method on image classification tasks derived from ILSVRC2012 (Russakovsky et al., 2015) and text classification tasks derived from
LSHTC-3 (Partalas et al., 2015).
Both datasets have off-the-shelf label relation structure (Deng et al., 2014;
Partalas et al., 2015), which are directed acyclic graphs (DAGS) and from which we could query pairwise label relations.
Indeed, there is a one-to-one mapping between a DAG structure of labels and a consistent label graph (See App.
E.1 for an example). The ILSVRC2012 dataset consists of 1.2M training images from 1,000 leave classes; for non-leave classes, we follow Deng et al.
to aggregate images belonging to its descendent classes as its data points.
The LSHTC-3 dataset consists of 456,886 documents and 36,504 labels organized in a DAG.
7. S ETUP
For each dataset, we randomly sample 100 different label graphs, each of which consists of 8 classes, and use each label graph to construct a WIS task.
For each label graph, we treat 3 of the sampled classes as unseen classes and the other 5 as seen classes.
The distinguishable condition in Sec.is ensured for all the WIS tasks, and the performance drop when it is violated can be found in
App. G.1.
We sample data belonging to unseen classes for our experiments and split them into train, validation and test set.
For image classification tasks, we follow Mazzetto et al.
(2021b;a) to train a branch of image classifiers as supervision sources of seen classes.
For text classification tasks, we made keyword-based labeling functions as supervision sources of seen classes following Zhang et al.
(2021); each of the labeling functions returns its associated label when a certain keyword exists in the text, otherwise abstains.
Notably, all the involved supervision sources are "weak" because they cannot predict the desired unseen classes.
Experimental details and additional results are in App.
F.
[Equation] M ETHODS AND R ESULTS
In addition to the WS-LG baseline, which is an adaptation of Data Programming (Ratner et al., 2019) to WIS task, and PLRM, we also include the following baselines.
Note that all compared methods input the same data, ILFs, and label relations throughout our experiments for fair comparisons.
Label Relation Majority Voting (LR-MV).
We modify the majority voting method based on the label’s non-exclusive neighbors: we replace ˆ y predicted by any ILF with the set of desired labels
N (ˆ y, Y ) , i.e. , the desired labels with non-exclusive relation to ˆ y , then aggregate the modified votes.
Weighted Label Relation Majority Voting (W-LR-MV).
LR-MV only leverages exclusive/nonexclusive label relations.
To leverage fine-grained label relations, W-LR-MV attaches a weight to each replaced label.
Specifically, if the ILF’s output ˆ y is replaced with its ancestor label y (subsumed relation), then the weight of y equals 1, while for the other relations, the weight is [Equation] | , where
[Equation]
For the above methods, we compare the performance of (1) directly applying included models on the test set and (2) the end models (classifiers) trained with inferred training labels.
Zero-Shot Learning (ZSL).
It is non-trivial to apply ZSL methods, because ZSL assumes label attributes for all classes and a labeled training set of seen classes, while WIS input an unlabeled dataset of unseen classes, label relations and ILFs.
Fortunately, the Direct Attribute Prediction (DAP) (Lampert et al., 2013) method is able to make predictions solely based on attributes without labeled data, by training attribute classifier p ( a i | x ) for each attribute a i .
Therefore we include it in our experiments.
The details of applying DAP can be found in App.
F.2.
Evaluation Results.
For a fair comparison, we fix the network architecture of the classifiers for all the methods.
For image classification, we use ResNet-32 (He et al., 2016) and for text classification, we use logistic regression with pre-trained text embedding (Reimers & Gurevych, 2019).
The overall results for both datasets can be found in Table 2.
From the results, we can see that PLRM consistently outperforms baselines.
The advantages of PLRM show the effect of not just leveraging the label graph, as the baselines do, but modeling the accuracy of ILFs and the strengths of label relations [Table] as PLRM does.
The reported results have high variance [Table] which actually indicates the 100 different WIS tasks are diverse and have varying difficulty.
Also
[Table]
we can see the end models are much better than directly applying the label models on the test set; this shows that the end models are able to generalize beyond the training labels produced by label models
[Table]
7. R EAL - WORLD A PPLICATION
In this section, on a commercial advertising system (CAS), we showcase how to reduce human annotation efforts of new labeling tasks by formulating them as WIS problems.
In a CAS, ads tagging (classification) is a critical application for understanding the semantics of ads copy.
When new ads and tags are added to the system, manual annotations need to be collected for training a new classifier.
As tags are commonly organized as taxonomies, the label relations between existing and new tags are readily available or can be trivially figured out by humans; Existing classifiers and the heuristic rules previously used for annotating existing tags could serve as ILFs.
Therefore, given (1) an unlabeled dataset of new tags, (2) the label relations, and (3) ILFs, we formulate it as a WIS problem.
On such WIS formulation, we apply our method and baselines, to synthesize training labels of new tags.
Specifically, we have two WIS tasks where the tags are under the “ Car Accessories ” and “ Furniture ” categories respectively.
For both tasks, we have 3 new tags and leverage 5 existing tags related to the new ones with given relations.
On a test set, we evaluate the performance of DAP and the quality of labels produced by label models, as shown in Table 3.
Note that since we re-use the existing labeling sources tailored for existing tags as ILFs and obtain label relations from an existing taxonomy, we achieve these results without any manual annotation or creation of new labeling functions.
This demonstrates the potential of the proposed WIS task in real-world scenarios.
[Table]
C ONCLUSION
We propose Weak Indirect Supervision (WIS), a new research problem which leverages indirect supervision sources and label relations to synthesize training labels for training machine learning models.
We develop the first method for WIS called Probabilistic Label Relation Model (PLRM) with the generalization error bound of both PLRM and end model.
We provide a theoretically-principled sanity test to ensure the distinguishability of unseen labels.
Finally, we provide experiments to demonstrate the effectiveness of PLRM and its advantages over baselines on both academic datasets and industrial scenario.
Reproducibility Statement.
All the assumptions and proofs of our theory can be found in App.
C & D. Examples and illustrations of label graph are in App.
E. Experimental details can be found in App.
F. Additional experiments are in App.
G.
S UPPLEMENTARY MATERIALS FOR “C REATING T RAINING S ETS VIA W EAK I NDIRECT S UPERVISION ”
The supplementary materials are organized as follows.
In Appendix A, we provide a glossary of variables and symbols used in this paper.
In Appendix B, we provide the details of PLRM model.
In Appendix C and D, we provide the detailed proofs of Theorem 2 and Theorem 1 respectively.
In
Appendix E, we provide the detailed examples and illustrations of label graph in WIS.
In Appendix F and G, we provide experimental details and additional experiment resulst respectively.
A G LOSSARY OF S YMBOLS
[Table]
Concatenation
[Table] binary variable [Table] ¯ Y =
[ ¯ Y
[Table]
Y
[Table]
The set of label relations, E = { ( y i , y j , t y i y j ) | t y i y j ∈ T , i < j, ∀ y i , y j ∈ V} [Table] The set of [Table]
relation types, T [Table] [Table] e ,
[Table]
o
[Table]
t sd [Table] sg [Table]
B D ETAILS OF THE PLRM
We use Y, ¯ Y , and ˆ Y to represent random vector.
Then, we give the formal form of the PLRM as:
[Equation]
Recall that Y is the unobserved true label
¯ Y is the binary random vector, each of whose binary value ¯ Y
i reflects whether the data should be assigned seen label ˆ
y i ∈
ˆ Y and ˆ Y is the concatenated outputs of ILFs.
Specifically, we enumerate Φ as below:
(Pseudo accuracy dependency): ∀ j ∈ [ n ] , y ∈ Y , ˆ y ∈ Y
λ j , we have ϕ
Acc y, ˆ y,j ( Y, ˆ Y j ) :
= 1 { Y = y
∧ ˆ Y j
= ˆ y ∧
ˆ y ∈ N ( y, Y λ j ) }2.
(Accuracy dependency): ∀ j ∈ [ n ] , ˆ y i ∈
ˆ Y ∩ Y j we have ϕ
Acc ˆ y i ,j ( ¯ Y i , ˆ Y j ) :
= 1 { ¯ Y i = 1 ∧ ˆ Y j
= ˆ y i } 3.
(Label relation dependency between seen labels):
∀ ˆ
y i , ˆ
y j ∈
ˆ Y , i < j (a) if t ˆ y
i ˆ y j = t e , we have ϕ
e ˆ y
i , ˆ y j ( ¯ Y i , ¯ Y j ) :
{+= − 1 { ¯ Y i = 
+}
∧ ¯ Y j = 1 } (b) if t ˆ y
i ˆ y j
= t o , we have ϕ
o ˆ
y
i , ˆ y j ( ¯ Y i , ¯ Y j ) :
{+= 1 { ¯ Y i = 
+}
∧ ¯ Y j = 1 } (c) if t ˆ y
i ˆ y j = t sg , we have ϕ sg ˆ y i , ˆ y j ( ¯ Y i , ¯ Y j ) :
{+= − 1 { ¯ Y i = 
+}
∧ ¯ Y j = 1 } (d) if t ˆ y
i
ˆ
y j = t sd , we have ϕ sd
ˆ
y
i , ˆ y
[Equation]
[Equation]
(a) if t y ˆ
y
i
=
t e , we have
[Equation]
(b) if t y ˆ y i =
t o , we have [Equation] (c) if t y ˆ y
i
=
t sg , we have [Equation] (d) if t y ˆ y i = t
[Equation]
, we have [Equation] And example of our PLRM is shown in Fig. [Equation]
where square with difference colors corresond to different dependency/factor functions in PLRM. [Equation] [Table] [Figure]
To simplify the indexing of dependencies, we use Φ 1 to represent the concatenation of ϕ which involves both Y and ¯ Y , Φ 2 to represent the concatenation of ϕ which involve both Y and ˆΛ , and Φto represent the concatenation of remaining ϕ which do not involve Y .
Specifically, Φ 1 consists of k × ˆ k components corresponding to the dependency between the k desired labels and the ˆ k seen labels. We use the subscript i, j to denote the dependency function between the desired label y i and seen label ˆ y j , i.e., [Equation]
where ? is the corresponding relation. Similarly, Φ 2 consists of k × ( (cid:80) nj =1 k j ) components corresponding to the dependency between the k desired labels and the k j seen labels output by the ILF λ j ( j ∈ [ n ] ), and we use Φ 2 i,j,l to denote the dependency of y i and ˆ y j l , and Φ 2 i,j = (Φ 2 i,j,l ) k j l =1 to denote the dependency of y i and ˆ y j . According to Φ 1 , Φ 2 , and Φ 3 , we also divide the parameter Θ into Θ 1 (with elements being Θ 1 i,j correspondingly), Θ 2 (with elements being Θ 2 i,j = (Θ 2 i,j,l ) k
j l =1 correspondingly), and Θ 3 , and the joint probability is then given as:
[Equation]
Also, for notation convenience, we adopt following simplifications:
∀ y i ∈ Y
→ ∀ i ∈ [ k ] since |Y| =
k , similarly, ∀ ˆ y i ∈ Y j
→ ∀ i ∈ [ k j ] and ∀ ˆ
y i ∈
ˆ Y → ∀ i ∈
[ˆ k ] ; 2.
∀ λ j → ∀ j ∈ [ n ] since we have n ILFs in total; 3.
ϕ t yiyj y i ,y j
→ ϕ ty i ,y j where t = t y i y j
and can be seen from the subscript of the dependency.
C. P ROPOSITIONS AND L EMMAS
First, we state some propositions and lemmas that will be useful in the proof to come.
Proposition 1 (Multi-class classification) .
For a multi-class classification task, ∀ y i , y j ∈ Y , we have t y i y
j
= t e .
Similarly, ∀ ˆ y a , ˆ y b ∈ ˆ Y , we have t ˆ y a ˆ y b = t e .
Lemma 1.
For a consistent label graph G and ∀ ˆ y l ∈ ˆ Y , ∀ y i , y j ∈ Y , if t y i ˆ y l = t o , we have t y j ˆ y l ̸
= t sg .
Proof.
For ∀ y i , y j ∈ Y , based on Proposition 1, we know t y i y j
= t e , which implies (1) the intersection of the sets labeled by y i and y j is empty.
For ∀ ˆ y l ∈ ˆ Y , if t y i ˆ y l = t o , we have (2) the intersection of the sets labeled by y i and ˆ y l is not empty.
If t y j ˆ y l = t sg , which implies (3) y j ⫌
ˆ y l . Based on (2)(3), we have y i ∩ y j ̸ = ∅ , which is contradictory to (1).
Thus, we prove when t y i ˆ
y l = t o , t y j ˆ y l ̸
= t sg .
Lemma 2.
For an informative ILF λ j and given any y d ∈ Y , there exists some ˆ y l ∈ Y j , such that, Φ 2 d,j,l ( y d , ˆ y )
= 0 , ∀ l ∈ [ k j ] .
Proof.
Because ILF λ j is informative, we know there exists one ˆ y a ∈ Y
j
such that ˆ y a is exclusive to y d , i.e., ˆ y a / ∈ N ( y d , Y j ) . Therefore, for any ˆ y l ∈ ˆ Y , either ˆ y a ̸ = ˆ y l , or ˆ y l
=
ˆ y a / ∈ Y j , which leads to the conclusion by the definition of Φ 2 d,j,l = ϕ Acc y d , ˆ y l ,j .
C. D EFINITIONS
Before the main proof, we connect the indistinguishablity of label relation structure with the dependency structure of PLRM by introducing the concept of symmetry as follows:
Definition 4 (Symmetry) .
For y i , y j ∈ Y , we say y i and y j have symmetric dependency structure if the following equation holds:
[Equation]
Based on the construction of PLRM, we know that for ∀
y i , y j ∈ Y , ∀ ˆ y b ∈ ˆ Y , t y i ˆ y b
= t y j ˆ y b (the statement in Theorem 2) is equivalent to y i and y j have symmetric dependency structure.
C. E QUIVALENT S TATEMENT OF T HEOREM
Our main result states that asymmetric is equivalent to distinguishable as in the following theorem, which can readily be seen to be identical to Theorem 2 in the main body of the paper:
Theorem 3.
For a probability model defined as Eq.
(8) induced from a consistent label graph and informative ILFs, for any pair of y i , y j ∈ Y , y i and y j are distinguishable if and only if they have asymmetric dependency structure.
C. P ROOF OF THE NECESSITY IN T HEOREM 3: N ECESSARY C ONDITION
We first prove that for any y i , y j ∈ Y , y i and y j have asymmetric dependency structure is the necessary condition of that they are distinguishable.
Proof of Theorem 3.
We prove this theorem by reduction to absurdity.
Suppose y i and y j are symmetric.
Then, by Eq.
(8), the distribution of Y condition on any ¯ Y and ˆ Y can be calculated as follows:
[Equation]
On the other hand, applying Y = y i in the definition of Φ 2 leads to Φ
[Equation]
We further separate Φ 1 into (Φ 1 i ) ki =1 , where Φ 1 i collects all the dependency in Φ 1 with y i involved, i.e.,
[Equation]
with the corresponding parameters respectively denoted as Θ 1 i with Θ 1 = (Θ 1 i ) ki =1 . Similarly, Φ 2 is also divided into (Φ 2 i ) ki =1 following the same routine and Θ 2 is respectively divided into (Θ 2 i ) ki =1 .
Specifically, if y i and y j are symmetric, we further have
[Equation]
Based on the notation, P Θ ( Y = y i
| ¯ Y , ˆ Y ) can then be represented as
[Equation]
which further leads to
[Equation]
which is independent of Θ 3 . Similarly,
[Equation]
and ∀ l ∈ [ k ] [-= 
 | ˆ Y = ˆ Y-]/ { i, j } ,
[Equation]
Let ˜Θ be defined as follows:
[Equation]
2 j , ˜Θ 2 j = Θ 2 i , ˜Θ 2 l = Θ 2 l , ∀ l / ∈ { i, j } , and ˜Θ 3 = Θ 3 .
We then have
[Equation]
=
[Equation]
(cid:16)(cid:80)
[Equation]
= (cid:16)(cid:80)
[Equation]
(cid:16)(cid:80)
[Equation]
¯
[Equation] [Equation]
( Y
[Equation]
ˆ Y )
[Equation]
Similarly,
[Equation]
and ∀ l ∈ [ k ] / { i, j } ,
[Equation]
Therefore
[Equation]
we have
[Equation]
Since
P
[Equation]
and
[Equation]
we obtain that
[Equation]
which indicates y i and y j indistinguishable, and leads to a contradictory.
The proof is completed.
C. P ROOF OF T HEOREM 3: S UFFICIENT C ONDITION
We then prove that for any y i , y j ∈ Y , y i and y j have asymmetric dependency structure is the sufficient condition of that they are distinguishable.
Proof.
We use the same notations (Θ 1 i ) ki
=1 , (Θ 2 i ) ki =1 , and Θ 3 in Appendix C.5 to denote the separation of the parameter Θ . Let Θ be any parameter satisfying that there exists a parameter ˜Θ , such that Eq.
(4-5) holds.
By Eqs.
(10), (11), and Eq.
(12) together with Eqs. (4-5), we have ∀ r ∈ [ k ] , r ̸
= i, j , exp
[Equation]
i , ¯
[Equation]
[Equation]
T
[Equation] [Equation]
i ,
[Equation]
T
[Equation]
y i , ˆ
[Equation]
[Equation]
) =
[Equation] ) = [Equation]
[Equation] By simple rearranging [Equation] we have
[Equation]
By the equality between the second term and the third term in Eq.
(13), we obtain that
[Equation]
We further set ¯ Y in Eq.
(14) respectively to e l (the one hot vector with its l -th position being 1 ) and 0 for any fixed l ∈ [ˆ k ] , i [Equation] e.,
[Equation]
which by simple rearranging further leads to
[Equation]
Since Θ 1 j,l , ˜Θ 1 i,l > [Equation] and by definition we have
[Equation]
and
[Equation]
we obtain Θ 1 j,l = ˜Θ 1 i,l , and
[Equation]
Therefore, either t y j ˆ y l ∈ { t o , t sd , t sg } and t y
i ˆ y l ∈ { t o , t sd , t sg } , or t y j ˆ y l
= t e and t y
i ˆ y l = [-
 ( ∀ b (cid:54)-]t e , which by definition further indicates that Φ 2 i =
Φ 2 j (recall the way we build dependency between Y and ˆ Y ).
As l is arbitrarily picked, we then have Θ 1 j is equal to ˜Θ 1 i component-wisely.
By the equality between the first term and the third term in Eq.
(13) and following exact the same routine, we also have ˜Θ 1 j = Θ 1 i .
On the other hand, for any r ∈ [ˆ k ] , fixing ¯ Y and ˆ Y s ( ∀ s ̸ = r ), and setting ˆ Y r
= ˆ y rl ( l ∈ k r , ˆ y rl ∈ N ( y j , Y l ) ) in Eq.
(14), we have
[Equation]
On the other hand, by Lemma 2, there exists some p , s.t., ˆ y rp / ∈ N ( y j , Y r ) (which by Φ 2 i =[-Φ 2 j , we have Λ l [ b ] = 
-]
Φ 2 j further leads to ˆ y rp / ∈ N ( y i , Y r ) ).
Setting ˆ Y r
= ˆ y rl leads to
[Equation]
Subtracting the above two equations leads to Θ 2 j,a,l = ˜Θ 2 i,a,l .
Since a and l are arbitrarily picked, we conclude that Θ 2 j = ˜Θ 2 i .
Following the same routine, we also have Θ 2 i = ˜Θ 2 j .
Therefore, by applying Θ 1 j = ˜Θ 1 i , Θ 1 i = ˜Θ 1 j , Θ 2 j = ˜Θ 2 i , and Θ 2 i = ˜Θ 2 j in Eq.
(13), we have
[Equation]
i
[Equation]
Let ¯ Y {+= 
+}
ˆ k
(i.e., the ˆ k -dimension all 1 vector), we have
[Equation]
[-Let ˆ Y = 
-]
[Equation]
Since y i and y j are asymmetric, we have that there exists l , such that t y i ˆ y l ̸ =
t y j ˆ y l .
Concretely, by Eq.
(15), we have t y i ˆ y l ∈ { t o , t sd , t sg } , t y j ˆ y l = { t o , t sd , t sg } , and t y i ˆ y l ̸
= t y j ˆ y l .
On the other hand, Φ 1 i,l [Equation] y i ,
[Equation]
if and only if t y i ˆ y l = t o , t y j ˆ y l = t sg , or t y j ˆ y l = t o , t y
i ˆ y l = t sg , which contradicts Lemma 1.
Therefore,
[Equation]
In this case, solutions of Θ 1 i , Θ 1 j subject to respectively Eqs.
(16) and (17) lie along a zero-measure set.
The proof is completed.
D P ROOF OF T HEOREMD. L EARNING A LGORITHM
We first present the algorithm for producing ˆΘ and ˆ W in Algorithm 1.
Algorithm 1 WIS
Require:
Step size η , dataset D ⊂ X , and initial parameter Θ 0 .
ˆΘ → Θ 0 .
for all X ∈ D do Independently sample ( Y, ¯ Y , ˆ Y ) from π ˆΘ , and ( Y ′ , ¯ Y ′ , ˆ Y ′ ) from π ˆΘ conditionally given ˆ Y ′ =
ˆ Y ( X ) .
[Equation]
Compute ˆ W as described in (3) using ˆΘ .
output (ˆΘ , ˆ W )
D
A SSUMPTIONS
First, the problem distribution π ∗ needs to be accurately modeled by some distribution Θ ∗ in the family that we are trying to learn:
[Equation]
Secondly, given an example ( X, Y ) ∼ π ∗ , we assume Y is independent of X given ˆ Y ( X ) :
[Equation]
This assumption encodes the idea that while the ILFs can be arbitrarily dependent on the features, they provide sufficient information to accurately identify the true label vector.
Then, for any Θ , accurately learning Θ from data distribution is possible.
That is, there exists an unbiased estimator ˆΘ( D ) which is a function of the dataset D of i.i.d from π Θ , such that, for any Θ and some c > 0 ,
[Equation]
And we are reasonably certain in our guess of latent variables, i.e. , Y and ¯ Y .
That is, for any Θ , Θ ∗ ,
E ˆ Y ∗ ∼ Θ ∗ (cid:34) k (cid:88) i =
( n i
[Equation]
k ) Var ( Y
[Equation]
¯ Y
[Equation] ˆ Y ) ∼ [Equation] Θ (
[Equation]
Y =
y i
| ˆ Y =
ˆ Y
[Equation]
) 2 + ˆ k (cid:88)
[Equation]
≤ [Equation] We also assume that the output of the last layer of end model h W has bounded ℓ ∞ norm, that is, for any possible parameter W , [Equation]
[Equation]
Finally, we assume that solving Eq.
(3) has bounded generalization risk such that for some χ > 0 , solution ˆ W satisfies
[Equation]
D. P ROOF OF T HEOREM
To begin with, we state two basic lemmas needed for proofs throughout this section: Lemma D.1. Let x 1 , x 2 be two binary random variable.
Then we have variance of product of xand x 2 can be bounded as
[Equation]
Lemma D.2. Let Y be a random vector and ∥·∥ s be the spectral norm.
Then we have
[Equation]
Then, we borrow two lemmas from (Ratner et al., 2016), which are slightly different from the original ones but can be easily proved following the same derivations: Lemma D.3. [Lemma D.1 in (Ratner et al., 2016)]
Given a family of maximum-entropy distributions
[Equation]
If we let J be the maximum expected log-likelihood objective, under another distribution π ∗ , for the event associated with the observed labeling function values ˆ Y ,
[Equation]
then its Hessian can be calculated as ∇
[Equation]
( Y, ¯ Y , ˆ Y ) ∼ π Θ (cid:16) ϕ
( Y, ¯ Y , ˆ Y ) | ˆ Y
=
ˆ Y ∗ (cid:17)(cid:105) − Cov
( Y, ¯ Y , ˆ Y ) ∼ π Θ ( ϕ ( Y, ¯ Y , ˆ Y )) .
Lemma D.4. [Lemma D.4 in (Ratner et al., 2016)] Suppose that we are looking at a WIS maximum likelihood estimation problem and the objective function J (Θ) is strongly concave with concavity parameter c > 0 .
If we run stochastic gradient descent using unbiased samples from a true distribution π Θ ∗ , then if we set step size as
[Equation]
and run (using a fresh sample at each iteration) for T steps, where
[Equation]
We can bound the expected parameter estimation error with
[Equation]
where M is the dimension of Θ .
Based on Lemma D.4, in order to obtain the optimization error with respect to the estimated ˆΘ produced by Algorithm 1, we only need to show that the WIS object function J (Θ) 2 is strongly concave.
We prove this through the following lemma, which is a non-trivial extension of Lemma D. in (Ratner et al., 2016) given the fact that we have multiple latent variables and relatively complex dependency structures with comparison to (Ratner et al., 2016):
Lemma D.5.
[Extension of Lemma D.3 in (Ratner et al., 2016)] With conditions (20) and (21), the WIS objective function J (Θ) is strongly concave with strong convexity c.
We then come to bound the generalization error of ˆ W produced by Algorithm 1, using the following non-trivial extension of Lemma D.5 in (Ratner et al., 2016): Lemma D.6.
[Extension of Lemma D.5 in (Ratner et al., 2016)] Suppose that conditions (18)-(23) hold.
Let ˆ W be the learned parameters of the end model produced by Algorithm 1, and ℓ ( W ∗ ) be the minimum of cross entropy loss function ℓ .
Then, we can bound the expected risk with
[Equation]
Finally, we conclude Lemmas (D.4), (D.5) and (D.6) as the following theorem, which is identical to the Theorem 1 in the main body of the paper:
Theorem 4 (Extension of Theorem 2 in (Ratner et al., 2016)) .
Suppose that we run Algoirthm 1 on a
WIS specification to produce ˆΘ and ˆ W , and all conditions of Lemmas (D.5) and (D.6) are satisfied.
Then, for any ϵ > 0 , if we set the step size to be
[Equation]
and the input dataset D is large enough such that
[Equation]
then we can bound the expected parameter error and the expected risk as:
[Equation]
D. P ROOFS OF L EMMAS
Lemma D.1.
Let x 1 , x 2 be two binary random variable.
Then we have variance of product of xand x 2 can be bounded as
[Equation]
Proof.
Joint distribution of x 1 and x 2 can be listed as the following table: (where p 1 + p 2 + p 3 + p 4 = 1 ) x 1 / x
[Equation]
Then we have
[Equation]
while
[Equation]
The proof is completed.
Lemma D.2. Let Y be a random vector and ∥·∥ s be the spectral norm.
Then we have
[Equation]
Proof.
By definition of spectral norm, we have
[Equation]
Where x is a constant vector.
And by Cauchy-Schwarz inequality,
[Equation]
Because x is a constant vector and ∥ x ∥ ≤ 1 ,
[Equation]
The proof is completed.
Lemma D.5. [Extension of Lemma D.3 in (Ratner et al., 2016)] With conditions (20) and (21), the WIS objective function J (Θ) is strongly concave with strong convexity c.
Proof.
By Lemma D.3, hessian matrix of J can be decomposed as follows:
[Equation]
Basically, to prove that J (Θ) is strongly concave with strong convexity c , we need to show for a real number c > 0 ,
[Equation]
We calculate each term separately: for the first term
[Equation]
since A is symmetric, for any real number c , A ⪯ c I , if and only if its spectral norm ∥
A ∥ s [-and by Lemma D.
 we have-]≤ c , where ∥
A ∥ s equals to the eigenvalue of A with largest absolute value.
Since by definition, vector function Φ( Y, ¯ Y , ˆ Y ) can be represented as:
[Equation]
by Lemma D.2, we have A can be further bounded by
[Equation]
where
[Equation]
We then bound the four terms respectively
[Equation]
As for A 1 , for fixed ˆ Y ∗ , we have
[Equation]
where n i is the number of ILFs whose label space contains label that is non-exclusive to label y i , i.e. , n i = |{ j ∈ [ n ]
|N ( y i , Y j ) ̸ = ∅}| .
Therefore, we have
A 1 ≤
[Equation]
Similarly, for A 2 , we have
[Equation]
where m i is the number of ILFs whose label space contains the label ˆ y i .
As for A 3 , for fixed ˆ Y ∗ and any ˆ y i , ˆ y j ∈ ˆ Y , we further separate the proof into subcases by t ˆ y i ˆ y j which is simplified as t : (1).
t =
t o .
In this case,
[Equation]
where Eq.
( ∗ ) is due to Lemma D
[Equation]
t = t e .
Similarly,
[Equation]
≤
[Equation]
t =
t sg .
In this case,
[Equation]
t =
t sd .
Similar to (3).
[Equation]
, ¯ Y , ˆ Y ) ∼
[Equation]
Θ
[Equation]
Var
[Equation]
Combining (1), (2), (3), and (4), we have
[Equation]
As for A
[Equation]
by similar discussion of A
[Equation]
Combining estimation of A 1 , A 2 , A 3 , A 4 , and by condition (21) we have
[Equation]
which further leads to
A ⪯ c I .
For the second term B =
Cov ( Y, ¯ Y , ˆ Y ) ∼ π Θ (Φ( Y, ¯ Y , ˆ Y )) ,
B = E
( Y, ¯ Y , ˆ Y ) ∼ π Θ
[Equation]
Y
[Equation]
Θ
[Equation]
Θ
[Equation]
, where E ( Y, ¯ Y , ˆ Y ) ∼ π Θ (cid:20)(cid:16) ∇ Θ log π Θ ( Y, ¯ Y , ˆ Y ) (cid:17) 2 (cid:21) is the Fisher Information of Θ .
By the Cramér-Rao bound and the condition (20),
[Equation]
which further leads to
[Equation]
The proof is completed by putting estimation of terms A and B together.
Lemma D.6. [Extension of Lemma D.5 in (Ratner et al., 2016)] Suppose that conditions (18)-(23) hold.
Let ˆ W be the learned parameters of the end model produced by Algorithm [Equation] and ℓ ( W ∗ ) be the minimum of cross entropy loss function ℓ .
Then, we can bound the expected risk with
[Equation]
Proof.
We begin by rewriting objective of expected loss minimization problem using law of total expectation as follows:
[Equation]
and by our conditional independence assumption (condition (19))
[Equation]
we have
[Equation]
which further leads to
[Equation]
On the other hand, if we are minimizing the model with learned parameter ˆΘ , we will be actually minimizing
[Equation] where for any X ′ , E ( Y, ˆ Y ) ∼ π ˆΘ (cid:104) H ( Y, σ ( h ( X ′ , W )))
(cid:12)(cid:12)(cid:12) ˆ Y = ˆ Y ( X ′ ) (cid:105) can be further calculated as
[Equation]
For simplification, we rewrite P ( Y, ˆ Y ) ∼ π ˆΘ ( Y = y l | ˆ Y =
ˆ Y ( X ′ )) as follows with slight abuse of notations:
[Equation]
and similarly
[Equation]
Let l X ′ △ =
arg min l log ( σ ( h ( X ′ , W )) l ) .
The difference between the loss functions will be
[Equation]
Furthermore,
[Table]
[Equation]
[Equation]
Let ¯ h
[Equation]
1 , l
[Equation]
= [Equation] X [Equation]
X
[Equation]
By Eq.
(22), we have for any l ∈ [ k ] ,
[Equation]
2 H. For any fixed X ′ , define g X ′ (Θ) as follows:
[Equation]
based on which we have
[Equation]
By First Mean Value Theorem,
[Equation]
We then bound ∇ g X ( ξ ) element-wisely: (1).
For any i ∈ [ k ] , j ∈ [ n ] , ˆ y jl ∈ N ( y i , Y j ) , if i = l X ′ , ˆ Y j
( X ′ )
=
ˆ y jl ,
[Equation]
[Equation]
If ˆ Y j ( X ′ ) ̸
= ˆ y jl , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
jl ,j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
[Equation]
For j ∈ [ n ] , ˆ y r ∈
Y j , if ˆ Y j ( X ′ )
=
ˆ y r , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ∂g X ′ ( ξ )
∂θ Acc ˆ y r ,j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:88) l ̸
[Equation] l X ′
[Equation]
∂θ
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
(cid:12) (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:88) l ̸ = l X ′ ¯ [Equation] [Equation]
[Equation]
X
[Equation]
( ¯
[Equation]
= 1 | ˆ
[Equation]
( X ′ ))
(cid:17) (cid:12) (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) .
Let
[Equation]
and
[Equation]
Therefore,
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
On the other hand, (cid:88)
[Equation]
Y r | ˆ Y ( X ′ ) (cid:105) .
Similarly, we have (cid:88)
[Equation]
≤ 2 H Var (cid:104)
¯ Y r | ˆ Y ( X ′ ) (cid:105) .
Conclusively, we have
[Equation]
If ˆ Y j = ˆ y r , similar to (1), we have
[Equation]
For any ˆ y i , ˆ y j ∈ ˆ Y , by the definition of ϕ t ˆ y i , ˆ y j , there exists ( a, b ) ∈ { 0 , 1 } 2 , such that ϕ t ˆ
y
i , ˆ y j ( a, b ) ̸ = 0 .
Similar to (2) , let
[Equation]
and
[Equation]
we have
[Equation]
where inequality ( ∗ ) comes from Lemma D.1.
For any y i ∈ Y , ˆ y r ∈ ˆ Y , by the definition of ϕ ty i , ˆ y r , there exists a ∈ { 0 , 1 } , y j ∈ Y , s.t.,
ϕ ty i , ˆ y r ( y j , a ) ̸ = 0 . We further divide the proof into two cases: ϕ ty i , ˆ y r ( y i , a ) = 0 , and ϕ ty i , ˆ y r ( y i , a ) ̸ = 0 .
(4a).
If ϕ ty i , ˆ y r ( y i , a ) = 0 , we have t y i
ˆ y r
= t sg and consequently a = 1 .
Similar to (1-3).
, we have
[Equation]
where Eq. ( • ) is due to Let
[Equation]
and
[Equation]
Then we have
[Equation]
∂θ t y i
[Equation]
ˆ y r
[Equation] max (cid:40)(cid:88) [Equation] [Table]
[Equation]
On the other hand
[Equation]
( Y = y i | ˆ Y ( X ′ )) P π ξ ( Y ̸ = y i , ¯ Y r = 1 | ˆ Y ( X ′ ))
{+(cid:17) ≤ 2 H (cid:16) P π ξ ( ¯ Y r = 
+}
| ˆ Y ( X ′ )) P π ξ ( ¯ Y r = 1 | ˆ Y ( X ′ )) + P π ξ ( Y =
y i | ˆ Y ( X ′ )) P π ξ ( Y ̸ = y i , ¯ Y r = 1 | ˆ Y ( X ′ ))
(cid:17) ≤ 2 H Var π ξ (cid:104) 1 Y = y i | ˆ Y ( X ′ ) (cid:105)
+ 2 H Var π ξ (cid:104) ¯ Y r | ˆ Y ( X ′ ) (cid:105) . Therefore, in this case, we have
[Equation]
(4b).
If ϕ ty i , ˆ y r ( y i , a ) ̸ = 0 , similar to (4 a ) .
, we have
[Equation]
∂θ ty
i
[Equation]
ˆ y r
[Equation]
[Table]
Since
[Equation]
≥
[Equation]
, we have
[Equation]
Combining (4a).
and (4b).
, we have that
[Equation]
Combining (1-4).
, we then have
[Equation]
Therefore, by Eqs.
[Equation]
[Equation]
and (28)
[Equation]
and Assumption Eq.
(21), we have
[Equation]
Now
[Equation]
we apply the assumption that we are able to solve the empirical problem
[Equation]
producing an estimate ˆ W that satisfies
[Equation]
where W ∗ ˆΘ is the true solution to
[Equation]
Θ ( W ) .
Therefore,
[Equation]
where Eq.
( ∗ ) comes from condition (23).
With Eqs.
(20) and (21), we have Eq.
(24) by Lemma D.5, i.e.,
[Equation]
We can now bound this using the result of Lemma D.6, which results in
[Equation]
The proof is completed.
E E XAMPLES AND I LLUSTRATIONS
[Figure] Fig 5 shows the mapping between a label hierarchy and the corresponding label graph.
Indeed, given the order of labels, any label structure represented as a (directed acyclic graph) DAG can be converted to exact one consistent label graph based on the four types of label relations.
[Figure]
and a label graph.
E. A N E XAMPLE OF I NCONSISTENT L ABEL G RAPH
Fig.
6 shows an example of an inconsistent label graph.
We can see that the label graph is unrealistic and ambiguous because “ Husky ” subsumes “ Canidae ”, but (1) “ Canidae ” subsumes “ Dog ” and (2) “ Dog ” subsuems “ Husyky ” combined imply that “ Husky ” should be subsumed by “ Canidae ”.
Also, from the example, we can see that label graph induced from cyclic label hierarchy must be inconsistent.
[Figure]
E. E NUMERATION OF I NCONSISTENT T RIANGLE L ABEL G RAPH
For a triangle label graph G , we list all inconsistent label relation structures.
The consistency of larger label graph with more labels can be verified by checking the consistency of every triangle inside.
One example proof of {Exclusive, Overlap, Subsuming} can be found in Lemma 1.
[Table]
Overlap
Subsumed
Subsuming
Overlap
Subsumed
Exclusive
Overlap
Subsuming
Subsumed
Overlap
Exclusive
Subsumed
Exclusive
Subsumed
Subsuming
Exclusive
Overlap
Subsuming
Exclusive
Subsuming
Subsuming
Exclusive
Subsuming
Subsumed
Exclusive
Subsuming
Overlap
Subsuming
Exclusive
Subsumed
Subsuming
Subsumed
Exclusive
Subsuming
Overlap
Subsumed
Subsuming
Overlap
Exclusive Subsuming Subsuming Exclusive
Subsuming
Subsuming Subsumed
Subsuming Subsuming
Overlap
Subsumed
Overlap
Subsuming
Subsumed
Subsumed
Exclusive
Subsumed
Subsumed
Subsuming
Subsumed Subsumed
Overlap
Subsumed
Exclusive
Subsuming
Subsumed
Exclusive
Subsumed Subsumed
Exclusive
Overlap
Fig.
7 shows an example label graph with indistinguishable label relation structure.
Again, red labels represent desired unseen labels, while gray labels are undesired and seen.
We can see that unseen label “ Husky ” and “ Bulldog ” have indistinguishable label relation structures because for all seen labels, their label relations are equal.
For example, seen label “ Dog ” subsumes both “ Husky ” and “ Bulldog ”.
In contrast, for “ Husky ” and “ Bengal Cat ”, seen label “ Cat ” subsumes the latter but exclusive to the former, which indicates that “ Husky ” and “ Bengal Cat ” have distinguishable label relation structure.
Note that “ Bengal Cat ” and “ Persian Cat ” also have indistinguishable label relation structure, but the former is unseen desired label while the latter is seen and can be predicted by some ILF(s).
We are only interested in the distinguishablity of a pair of unseen labels.
In practice, users could "break the symmetry" by adding new ILFs with new labels.
For example, if we add an ILF that could predict “ Arctic Animals ”, then the new seen label “ Arctic Animals ” will be added into label graph as shown in Fig.
We know that “ Arctic Animals ” subsumes “ Husky ” but not “ Bulldog ”, so we break the indistinguishable label relation structure of “ Husky ” and “ Bulldog ” successfully.
[Figure] [Figure]
[Figure]
F E XPERIMENTAL D ETAILS F. D ATASET
Large scale Text Classification Dataset 3 : LSHTC-3 (Partalas et al., 2015), a large scale hierarchical text classification dataset, which consists of 456,886 documents and 36,504 categories organized in a label hierarchy.
We filter out the documents with multiple labels, and preserve categories with more than 500 documents.
We use a pre-trained sentence transformer (Reimers & Gurevych, 2019) to obtain document embeddings for classification.
We follow Zhang et al.
(2021) to generatekeyword-based labeling functions for each seen label as ILFs.
Large scale Image Classification Dataset 4 : {+ILSVRC
+} (Russakovsky et al., 2015), a large scale image classification dataset, which consists of 1.2M training images from 1000 object classes based 3 http://lshtc.iit on ImageNet.
Following Deng et al.
we use WordNet as the label hierarchy, and because all the images are assigned to leave labels in WordNet, for each non-leave label, we aggregate images belonging to its descendants as its data points (Deng et al., 2014).
For weak supervision sources creation, we follow Mazzetto et al.
(2021b;a) to train 10 image classifiers as ILFs.
We randomly sampling 2 or 3 exclusive seen labels from the label graph as well as 500 images for each label to train a ResNet-32 classifier.
F. D ESCRIPTION OF A PPLYING DAP
To apply DAP, we use both label relations and ILFs to construct attributes for both unseen classes and unlabeled data points.
Then, we train the attribute classifiers, which in turn are used to predict unseen labels on the test set as in Lampert et al.
To construct attributes for unseen labels and data points, we leverage the outputs of ILFs and label relations.
First, based on the label relations and basic logistic rules, we enumerate all the possible assignments of seen labels given a data point.
For example, if label A is subsumed by label B , then for a data point, when it belongs to label A , it must also belong to B ; And if label A and B are exclusive, then one data cannot belong to both at the same time.
Let s ∈ S denote one possible label assignment and
S is the set of all possible s .
Then we define the attribute as a vector of | S | dimension where each dimension corresponds to one s .
Second, we define the attribute of unseen labels.
For an unseen label A and a label assignment s , if A is not exclusive to any label in s then we set the corresponding attribute a s = 1 for label A , other wise 0.
The intuition is that, if A is not exclusive to labels in s , it’s likely that when a data belongs to assignment s , it also belongs to label A .
For each data point, we use the labels assigned by ILFs to build their attributes.
If a data belongs to assignment s then its corresponding attribute a s = 1 , otherwise 0.
Then, we can train attribute classifier p ( a | x ) for each attribute based on data point attributes.
During inference, we use unseen label attribute as well as attribute classifier as in Lampert et al.
[Equation]
F. H YPER - PARAMETERS
For the training of PGMs, we set the learning rate to be 1 n where n is the number of training data.
For training logistic regression model, we use the default parameters in scikit-learn library.
For training
ResNet model, we set batch size as 256 and use Adam optimizer with learning rate being 1e-3 and weight decay being 5e-5.
F. H ARDWARE AND I MPLEMENTATION D ETAILS
All experiments ran on a machine with an Intel(R) Xeon(R) CPU E5-2678 v3 with a 512G memory and a GeForce GTX 1080Ti-11GB GPU.
All the code was implemented in Python.
We use the standard implementation of the logistic regression model from Python scikit-learn library 5 and the ResNet model from torchvision library 6 .
Our code will be released upon the acceptance.
F. D ATASET D ETAILS OF R EAL - WORLD A PPLICATIONS
We list the tags we used in the real-world application (Sec. 7.3) and examples of label relations we query from the existing product category taxonomy.
[Table]
[Table]
G A DDITIONAL E XPERIMENTS G. P ERFORMANCE D ROP W HEN THE D ISTINGUISHABLE C ONDITION IS V IOLATED
To validate the effectiveness of the distinguishable condition
[Table]
we drive another 100 WIS tasks from
LSHTC-3 dataset where each task has at least one pair of unseen labels sharing exactly the same label relation structure.
In Table
[Table]
we report the performance drop on the averaged evaluation results over the 100 WIS tasks with comparison to the numbers in Table 2.
Although the two sets of WIS tasks are different and therefore are not individually comparable
[Table]
the averaged performance drop does indicates that the violation of the distinguishable condition results in undesirable synthesized training labels, which implicitly demonstrates the effectiveness of the distinguishable condition
[Table]
[Table]
G. T HE P ERCENTAGE OF E ACH M ETHOD B EING THE B EST
As we have 100 WIS tasks for each dataset, we are also curious about how many times the proposed
PLRM method performs the best.
To see this, we plot in Fig.
9 the percentage of each method being the best one among the 100 WIS tasks for both datasets.
From the figures we can see that (1)
PLRM is the one with the most perform-the-best times and (2) in around or more than 50% cases,
PLRM is the best method.
We do not expect the PLRM to be the best method across all the 100 WIS tasks since the WIS task could be very diverse dependending on the label relation structure and the accuracy/coverage of ILFs.
However, results here indicate the advantages of PLRM over baseline methods.
G. E XPLANATORY E XPERIMENTS ON S YNTHETIC D ATASETS
In this section, we study how different set-up parameters influence both the WIS problem and the methods applied.
We conduct a series of explanatory experiments on synthetic dataset.
First, we
[Figure] [Figure]
study how the accuracy of ILFs affect the WIS problem.
We introduce a parameter r , which controls the accuracy of ILF by randomly corrupting r % of the labels used to train ILF.
Larger r results in less accurate ILFs.
Then, we fix the number of unseen labels k and study the effect of (1) the number of ILFs n and (2) the number of seen labels ˆ k on WIS.
These two parameters correspond to the following scenarios: (1) users have varying amount of ILFs, and (2) there are varying amount of undesired labels, respectively.
Where r is the label corruption ratio used to control the accuracy of input models only for synthetic dataset.
We generate 100 WIS tasks with each set of parameters and report average accuracy of the inferred training labels in Fig.
Overall, we can see that PLRM consistently surpasses baselines in varying scenarios.
Study of r : how the accuracy of ILFs affect the WIS?
From the results in Fig.
10 (c), we can see when ILFs become less accurate, the performances of all the methods decrease as expected.
Also, with inaccurate ILFs, the PLRM has almost the same performance as the WS-LG model, which indicates adding complex dependencies do not confer performance gain.
The result aligns with the intuition that with limited number of inaccurate ILFs, it is hard to learn undesired labels well and transit them to desired labels via label relations.
Study of nk : how the number of ILFs affects the WIS?
From the results in Fig.
10 (b), we can see with more ILFs, the overall performances increase as expected.
This seems reasonable as with more ILFs, we are now able to learn each undesired label and the label relations well, so the PLRM surpasses the WS-LG model and the gap increases along with the number of models.
Study of ˆ kk : how the number of seen labels affects the WIS?
From the results in Fig.
10 (a), we observe a decrease in overall performance when involving more undesired labels, since ILFs have more diverse label space and it is harder to aggregate their output.
Surprisingly, we observe that adding the complex label relation dependencies still boosts the performance, as the PLRM outperforms the WS-LG model by a large margin.
We speculate the reason is: the complex label relation dependencies serve as a regularization to help aggregate diverse ILFs.