Foreseeing the Benefits of Incidental Supervision (2021)

Abstract: Real-world applications often require improved models by leveraging *a range of cheap incidental supervision signals*. These could include partial labels, noisy labels, knowledge-based constraints, and cross-domain or cross-task annotations – all having statistical associations with gold annotations but not exactly the same. However, we currently lack a principled way to measure the benefits of these signals to a given target task, and the common practice of evaluating these benefits is through exhaustive experiments with various models and hyperparameters. This paper studies whether we can, *in a single framework, quantify the benefits of various types of incidental signals for a given target task without going through combinatorial experiments*. We propose a unified PAC-Bayesian motivated informativeness measure, PABI, that characterizes the uncertainty reduction provided by incidental supervision signals. We demonstrate PABI’s effectiveness by quantifying the value added by various types of incidental signals to sequence tagging tasks. Experiments on named entity recognition (NER) and question answering (QA) show that PABI’s predictions correlate well with learning performance, providing a promising way to determine, ahead of learning, which supervision signals would be beneficial.

## introduction the supervised learning paradigm, where direct supervision signals are available in high-quality and large amounts, has been struggling to fulfill needs in many real-world ai applications. as a result, researchers and practitioners often resort to datasets that are not collected directly for the target task but capture some phenomena useful for it * part of this work was done while the author was at allen institute for ai. 1 our code is publicly available at https://github. com/cogcomp/pabi . ( @xcite @xcite . however, it remains unclear how to predict the benefits of these incidental signals on our target task beforehand. for example, given two incidental signals that are relevant to the target task, it is still difficult for us to predict which one is more beneficial. therefore, the common practice is often trial-and-error: perform experiments with different combinations of datasets and learning protocols, often exhaustively, to measure the impact on a target task @xcite . not only is this very costly, but this trial-and-error approach can also be hard to interpret: if we don't see improvements, is it because the incidental signals themselves are not useful for our target task, or is it because the learning protocols we have tried are inappropriate? the difficulties of foreseeing the benefits of various incidental supervision signals, including partial labels, noisy labels, constraints foot_0 , and cross-domain signals, are two-fold. first, it is hard to provide a unified measure because of the intrinsic differences among different signals (e.g., how do we incidental supervision signals unified measure support measures partial noisy constraints auxiliary cross-domain cross-type mixed-type theoretical empirical cst'11, hh'12, ld'14 × × × × × × al'88, ndrt'13, rvbs'17 × × × × × × vw'17, wnr'20 × × × × × × nhfr'19 × × × × × × b'17 × × × × × × × gmslbds'20 × × × × × × × pabi (ours) we have been in need of a unified informativeness measure like pabi. for instance, it might be obvious that we can expect better learning performance if the training data is less noisy and more completely annotated, but what if we want to compare the benefits of a noisy dataset to those of a dataset that is only partially labeled? pabi enables this kind of comparisons to be done analytically, without the need for experiments on a wide range of incidental signals such as partial labels, noisy labels, constraints, auxiliary labels (labels of another task), and cross-domain signals, for sequence tagging tasks in nlp. a specific example of ner is shown in fig. 1 , and the advantages of pabi can be found in finally, our experiments on two nlp tasks, ner and qa, show that there is a strong positive correlation between pabi and the relative improvement for various incidental signals. this strong positive correlation indicates that the proposed unified, theory-motivated measure pabi can serve as a good indicator of the final learning performance, providing a promising way to know which signals are helpful for a target task beforehand. ## related work there are lots of practical measures proposed to quantify the benefits of specific types of signals. for example, a widely used measure for partial signals in structured learning is the partial rate @xcite @xcite @xcite ; a widely used measure for noisy signals is the noise ratio @xcite @xcite ; @xcite propose to use the concaveness of the mutual information with different percentage of annotations to quantify the strength of constraints in the structured learning; others, in nlp, have quantified the contribution of constraints experimentally @xcite . @xcite proposes to use conditional entropy or mutual information to quantify the value for auxiliary signals. as for domain adaptation, domain similarity can be measured by the performance gap between domains @xcite or measures based on the language model in nlp, such as the vocabulary overlap @xcite . among them, the most relevant work is @xcite . however, their conditional entropy or mutual information is based on token-level label distribution, which cannot be used for incidental signals involving multiple tokens or inputs, such as constraints and cross-domain signals. at the same time, for the cases where both pabi and mutual information can handle, pabi works similar to the mutual information as shown in fig. 2 , and pabi can further be shown to be a strictly increasing function of the mutual information. the key advantage of our proposed measure pabi is that pabi is a unified measure motivated by the pac-bayesian theory for a broader range of incidental signals compared to these practical measures for specific types of incidental signals. there also has been a line of theoretical work that attempts to exploit incidental supervision signals. among them, the most relevant part is task relatedness. @xcite define task relatedness based on the richness of transformations between inputs for different tasks, but their analysis is limited to cases where data is from the same classification problem but the inputs are in different subspace. @xcite proposes to use the joint kolmogorov complexity @xcite to characterize relatedness, but it is still unclear how to compute the joint kolmogorov complexity in real-world applications. @xcite further propose to use conditional kolmogorov complexity to measure the task relatedness and provide empirical analysis for decision trees, but it is unclear how to use their relatedness for other models, such as deep neural networks. @xcite propose to cluster tasks based on the similarity between the task-optimal distance metric of k-nearest neighbors (knn), but their analysis is based on knn and it is unclear how to use their relatedness for other models. a lot of other works provide quite good qualitative analysis to show various incidental signals are helpful but they did not provide quantitative analysis to quantify to what extent these types of incidental signals can help @xcite @xcite @xcite @xcite @xcite . compared to these theoretical analyses, pabi can be easily used in practice to quantify the benefits of a broader range of incidental signals. ## pabi: a unified pac-bayesian informativeness measure we start with notations and preliminaries. let x be the input space, y be the label space, and ŷ be the prediction space. let d denote the underlying distribution on x × y. let ℓ : y × ŷ → r + be the loss function that we use to evaluate learning algorithms. a set of training samples @xmath0, y i } m @xmath1.i.d. from d. in the common supervised learning setting, we usually assume the concept that generates data comes from the concept class c. in this paper, we assume c is finite, and its size is | c |, which is common in sequence tagging tasks because of finite vocabulary and label set. we want to choose a predictor c : @xmath2, measured by the generalization error when the posterior is one-hot (exactly one entry of the distribution is 1), we have the original definitions of training error and generalization error, as in the pac framework @xcite . we are now ready to derive the proposed informativeness measure pabi motivated by pac-bayes. the generalization error bound in the pac-bayesian framework @xcite says that with probability 1 -δ over s, the following bound holds as shown in the generalization bound, the generalization error is bounded by the kl divergence d kl (π * ||π 0 ) from the prior distribution to the gold posterior distribution. therefore, we propose to utilize incidental signals to improve the prior distribution from π 0 to π0 so that it is closer to the gold posterior distribution π * . correspondingly, we can define pabi, the informativeness measure for incidental supervision signals, by measuring the improvement with regard to the gold posterior, in the kl divergence sense. definition 3.1 (pabi). suppose we use incidental signals to improve the prior distribution from π 0 to π0 . the informativeness measure for incidental signals, pabi, is defined as remark. note that s(π 0 , π0 ) = 0 if π0 = π 0 , while if π0 = π * , then s(π 0 , π0 ) = 1. this result is consistent with our intuition that the closer π0 is to π * , the more benefits we can gain from incidental signals. the square root function is used in pabi for two reasons: first, the generalization bounds in both pac-bayesian and pac (see appx. a.1) frameworks have the square root function; second, in our later experiments, we find that square root function can significantly improve the pearson correlation between the relative performance improvement and pabi. it is worthwhile to note that the square root is not crucial for our framework, because our goal is to compare the benefits among different incidental supervision signals, where the relative values are expressive enough. in this sense, any strictly increasing function in [0, 1] over the current formulation would be acceptable. we focus on the setting that the gold posterior π * is one-hot, which means π * concentrates on the true concept c * ∈ c, though the definition of pabi can handle general gold posterior. however, π * is unknown in practice, which makes eq. ( 1 ) hard to be computed in reality. in the following, we provide an approximation ŝ of pabi. definition 3.2 (approximation of pabi). assume that the original prior π 0 is uniform, and the gold posterior π * is one-hot concentrated on the true concept c * in c, as we have assumed that c is finite. let h(•) be the entropy function. the approximation ŝ of pabi is defined as the uniform prior π 0 is usually used when we do not have information about the prior on which concept in the class that generates data. the intuition behind ŝ is that, it measures how much entropy incidental signals reduce, compared with noninformative prior π 0 . ŝ can be computed through data and thus is practical. to see how this approximation works, first note d kl (π * ||π 0 ) = ln | c | because π * is one-hot and π 0 is uniform over the finite concept class c. let π c be the one-hot distribution concentrated on concept c for each @xmath3. the approximation is that we estimate π * by π c , where c follows π0 : as shown in appx. a.1, the approximation of pabi and pabi are equivalent in the nonprobabilistic cases with the finite concept class, indicating the quality of this approximation. some analysis on the extensions and general limitations of pabi can be found in appx. a.2 and a.3. ## examples for pabi in this section, we show some examples of sequence tagging tasks foot_1 in nlp for pabi. we notations descriptions d target domain with gold signals d source domain with incidental signals c(x) gold system on gold signals c(x) perfect system on incidental signals ĉ(x) ## experiments in this section, we verify the effectiveness of pabi for various inductive signals and transductive sigfor individual inductive signals, the baselines (gray points) are, i.e. one minus partial rate for partial labels @xcite , one minus noisy rate for noisy labels @xcite , and entropy normalized mutual information for auxiliary labels @xcite . nals on ner and qa. more details about experimental settings are in appx. a.8. learning with various inductive signals. in this part, we analyze the informativeness of inductive signals for ner. we use ontonotes ner (18 types of named entities) (hovy et al., 2006) as the main task. we randomly sample 10% sentences (30716 words) of the development set as the small gold signals, 90% sentences (273985 words) of the development set as the large incidental signals. we use two-layer nns with 5-gram features as our basic model. the lower bound for our experiments is the result of the model with small gold ontonotes ner annotations and bootstrapped on the unlabeled texts of the large gold ontonotes ner, which is 38% f1, and the upper bound is the result of the model with both small gold ontonotes ner annotations and the large gold ontonotes ner annotations, which is 61% f1. to utilize inductive signals, we propose a new bootstrapping-based algorithm cwbpp, as shown in algorithm 1, where inductive signals are used to improve the inference stage by approximating a better prior. it is an extension of codl (chang et al., 2007) by covering various inductive signals. we experiment on ner with various inductive signals, including three types of individual signals, partial labels, noisy labels, auxiliary labels, and two types of mixed signals: signals with both partial and noisy labels, and signals with both partial labels and constraints. as shown in fig. 2 , we find that there is a strong correlation between the relative improvement and pabi for various inductive signals. for individual signals in fig. 2 (a)-2(c), we find that pabi has a similar foreseeing ability comparing to the measures for specific signals, i.e., 1 -η p for partial labels, 1 -η n for noisy la- (3 ); gray points indicate the results with the naive informativeness measure η 2 ; black points indicate the results with the vocabulary overlap baseline @xcite . the specific correlations can be found in learning with cross-domain signals. in this part, we consider the benefits of cross-domain signals for ner and qa. for ner, we consider four ner datasets, ontonotes, conll, twitter @xcite , and gmb @xcite . we aim to detect the person names here because the only shared type of the four datasets is the person foot_5 . in our experiments, the twitter ner serves as the main dataset and the other three datasets are cross-domain datasets. there are 85 sentences in the small gold training set, 756 sentences (9 times of the gold signals) in the large incidental training set, and 851 sentences in the test set. for qa, we consider squad @xcite , qamr @xcite , qa-srl bank 2.0 @xcite , qa-re @xcite , newsqa @xcite , triv-iaqa @xcite . in our experiments, the squad dataset serves as the main dataset and other datasets are cross-domain datasets. we randomly sample 700 qa pairs as the small gold signals, about 6.2k qa pairs as the large incidental signals (9 times of the small gold signals), and 21k qa pairs as the test data. we tried larger datasets for both gold and incidental signals (keeping the ratio between two sizes as 9), and the results are similar as long as the size of gold signals is not too large. we use bert as our basic model and consider two strategies to make use of incidental signals: joint training and pre-training. for ner, the lower bound is the result with only small gold twitter annotations, which is 61.51% f1, and the upper bound is the result with both small gold twitter annotations and large gold twitter annotations, which is 78.31% f1. for qa, the lower bound is the result with only small gold squad annotations, which is a 26.45% exact match. the upper bound for the joint training is the result with both small gold squad annotations and large squad annotations, which is a 50.72% exact match. similarly, the upper bound for the pre-training is a 49.24% exact match. the relation between the relative improvement (pre-training or joint training) and informativeness measures (baselines or the pabi) is shown in fig. 3 and ## conclusion and future work motivated by pac-bayesian theory, this paper proposes a unified framework, pabi, to characterize the benefits of incidental supervision signals by how much uncertainty they can reduce in the hypothesis space. we demonstrate the effectiveness of pabi in foreseeing the benefits of various signals, i.e., partial labels, noisy labels, auxiliary labels, constraints, cross-domain signals and combinations of them, for solving ner and qa. to our best knowledge, pabi is the first informativeness measure that can handle various incidental signals and combinations of them; pabi is motivated by the pac-bayesian framework and can be easily computed in real-world tasks. because the recent success of natural language modeling has given rise to many explorations in knowledge transfer across tasks and corpora @xcite @xcite @xcite , pabi is a concrete step towards explaining some of these observations. we conclude our work by pointing out several interesting directions for our future work. first, pabi can also provide guidance in designing learning protocols. for instance, in a b/i/o sequence chunking task, foot_6 missing labels make it a partial annotation problem while treating missing labels as o introduces noise. since the informativeness of partial signals is larger than that of noisy signals with the same partial/noisy rate (see details in sec. 4.1), pabi suggests not treating missing labels as o, and this is exactly what @xcite prove to us via their experiments. we plan to explore more in this direction to apply pabi in designing better learning protocols. second, we need to acknowledge that our current exploration for auxiliary labels is still limited. the results for auxiliary labels with a different label set (fig. 2(c )) is blocked by the imbalanced label distribution (appx. a.4). for more complex cases, such as part-of-speech tagging (pos) for ner, we can only treat them as cross-sentence constraints now and the results are also limited (appx. a.5). in the future, we will work more in this direction to better quantify the value of auxiliary signals. another interesting direction is to link pabi with the generalization bound. it might be too hard to directly link pabi with the generalization bound for all types of incidental signals, but it is possible to link it to the generalization bound for some specific types. for example, for partial and noisy labels, pabi can directly be expressed in the generalization bound as in @xcite ; van rooyen and williamson (2017); @xcite . the main difficulties are in constraints and auxiliary signals, and we postpone it as our future work. finally, we plan to evaluate pabi in more applications, such as textual entailment and image classification, and more types of signals, such as cross-lingual and cross-modal signals.