Denoising Multi-Source Weak Supervision for Neural Text Classification (2020)

Abstract: We study the problem of learning neural text classifiers without using any labeled data, but only easy-to-provide rules as multiple weak supervision sources. This problem is challenging because rule-induced weak labels are often noisy and incomplete. To address these two challenges, we design a label denoiser, which estimates the source reliability using a conditional soft attention mechanism and then reduces label noise by aggregating rule-annotated weak labels. The denoised pseudo labels then supervise a neural classifier to predicts soft labels for unmatched samples, which address the rule coverage issue. We evaluate our model on five benchmarks for sentiment, topic, and relation classifications. The results show that our model outperforms state-of-the-art weakly-supervised and semi-supervised methods consistently, and achieves comparable performance with fully-supervised methods even without any labeled data. Our code can be found at https://github.com/weakrules/Denoise-multi-weak-sources .

## introduction many nlp tasks can be formulated as text classification problems, such as sentiment analysis @xcite , topic classification @xcite , relation extraction @xcite and question answering like slot filling @xcite . recent years have witnessed the rapid development of deep neural networks (dnns) for this problem, from convolutional neural network @xcite , recurrent neural network @xcite to extra-large pre-trained language models @xcite @xcite . dnns' power comes from their capabilities of fitting complex functions based on largescale training data. however, in many scenarios, labeled data are limited, and manually annotating them at a large scale is prohibitively expensive. weakly-supervised learning is an attractive approach to address the data sparsity problem. it labels massive data with cheap labeling sources such as heuristic rules or knowledge bases. however, the major challenges of using weak supervision for text classification are two-fold: 1) the created labels are highly noisy and imprecise. the label noise issue arises because heuristic rules are often too simple to capture rich contexts and complex semantics for texts; 2) each source only covers a small portion of the data, leaving the labels incomplete. seed rules have limited coverage because they are defined over the most frequent keywords but real-life text corpora often have long-tail distributions, so the instances containing only long-tail keywords cannot be annotated. existing works @xcite @xcite attempt to use weak supervision for deep text classification. @xcite proposes a data programming method that uses labeling functions to automatically label data and then trains discriminative models with these labels. however, data annotated in this way only cover instances directly matched by the rules, leading to limited model performance on unmatched data. @xcite proposes a deep self-training method that uses weak supervision to learn an initial model and updates the model by its own confident predictions. however, the self-training procedure can overfit the label noise and is prone to error propagation. @xcite solves query performance prediction (qpp) by boosting multiple weak supervision signals in an unsupervised way. however, they choose the most informative labelers by an ad-hoc user-defined criterion, which may not generalize to all the domains. @xcite assumes that human labelers are over-generalized to increase the coverage, and they learn restrictions on the rules to address learning wrongly generalized labels. however, their method requires the specific formulation process of rules to indicate which rules are generated by which samples, so that it cannot deal with other kinds of labeling sources like knowledge bases or third-party tools. we study the problem of using multiple weak supervision sources (e.g., domain experts, pattern matching) to address the challenges in weaklysupervised text classification. while each source is weak, multiple sources can provide complementary information for each other. there is thus potential to leverage these multiple sources to infer the correct labels by estimating source reliability in different feature regimes and then aggregating weak labels. moreover, since each source covers different instances, it is more promising to leverage multiple sources to bootstrap on unlabeled data and address the label coverage issue. motivated by the above, we propose a model with two reciprocal components. the first is a label denoiser with the conditional soft attention mechanism @xcite ( § 3.2). conditioned on input text features and weak labels, it first learns reliability scores for labeling sources, emphasizing the annotators whose opinions are informative for the particular corpus. it then denoises rule-based labels with these scores. the other is a neural classifier that learns the distributed feature representations for all samples ( § 3.3). to leverage unmatched samples, it is supervised by both the denoised labels and its confident predictions on unmatched data. these two components are integrated into an end-to-end co-training framework, benefiting each other through cross-supervision losses, including the rule denoiser loss, the neural classifier loss, and the self-training loss( § 3.4). we evaluate our model on four classification tasks, including sentiment analysis, topic classification, spam classification, and information extraction. the results on five benchmarks show that: 1) the soft-attention module effectively denoises the noisy training data induced from weak supervision sources, achieving 84% accuracy for denoising; and 2) the co-training design improves prediction accuracy for unmatched samples, achieving at least 9% accuracy increase on them. in terms of the overall performance, our model consistently outperforms sota weakly supervised methods @xcite @xcite , source 3: price cheap inexpensive economical reasonable pos pos pos pos pricey expensive overpriced costly neg neg neg neg source 2: service prompt attentive knowledgeable informative pos pos pos pos indifferent rude slow inattentive neg neg neg neg source 1: food delicious tasty flavorful yummy pos pos pos pos disgusting gross dry overcooked neg neg neg neg the margaritas were good and strong but maybe . it is worth coming here just for the goat cheese. service is above average as our server amanda was very know and and seemed well trained. the chicken was sort of and not so . . the beef one could have been more but was ok. the vegetable (corn & rajas) taco was full of just mooch as it lacked flavor and texture. the wild caught fish taco was yellowtail on my visit. the tiny taco was tasty but the fish was and it seemed to me for such a tiny taco with very little fish in it. pricey knowledgeable informative dry tasty flavorful overcooked expensive figure 1 : the annotation process for three weak supervision sources. "pos" and "neg" are the labels for the sentiment analysis task. ## our method we begin with an overview of our method and then introduce its two key components as well as the model learning procedure. ## related work learning from noisy supervision. our work is closely related to existing work on learning from noisy supervision. to deal with label noise, several studies @xcite @xcite adopt a data cleaning approach that detects and removes mislabeled instances. this is achieved by outlier detection @xcite , a-priori heuristics @xcite , self-training @xcite , or reinforcement learning @xcite . one drawback of this data cleaning approach is that it can discard many samples and incur information loss. different from data cleaning, some works adopt a data correction approach. the most prominent idea in this line is to estimate the noise transition matrix among labels @xcite @xcite @xcite and then use the transition matrices to re-label the instances or adapt the loss functions. specifically, @xcite and @xcite generate label noise by flipping clean labels based on such noise transition matrices. they are thus not applicable to our weak supervision setting where no clean labels are given. meanwhile, reweighting strategies have been explored to adjust the input training data. these techniques weigh training samples according to the predictions confidence @xcite , one-sided noise assumption @xcite , a clean set @xcite or the similarity of their descent directions @xcite . recently, a few studies @xcite have also explored designing denoising modules for neural networks. however, our method differs from them in that: (1) our method learns conditional reliability scores for multiple sources; and (2) these methods still require clean data for denoising, while ours does not. ## conclusion we have proposed a deep neural text classifier learned not from excessive labeled data, but only unlabeled data plus weak supervisions. our model learns from multiple weak supervision sources using two components that co-train each other: (1) a label denoiser that estimates source reliability to reduce label noise on the matched samples, (2) a neural classifier that learns distributed representations and predicts over all the samples. the two components are integrated into a co-training framework to benefit from each other. in our experiments, we find our model not only outperforms state-ofthe-art weakly supervised models, but also benefits supervised models with its denoised labeled data. our model makes it possible to train accurate deep text classifiers using easy-to-provide rules, thus appealing in low-resource text classification scenarios. as future work, we are interested in denoising the weak supervision further with automatic rule discovery, as well as extending the co-training framework to other tasks beyond text classification. definition 2 (keywords rules). has(x, l) ⇒ c if x matches one of the words in the list l. for the remaining third and fourth types, each dataset has specific definitions. we then state all the labeling rules for each dataset from