WALNUT : A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding (2022)

Abstract: Building machine learning models for natural language understanding (NLU) tasks relies heavily on labeled data. Weak supervision has been proven valuable when large amount of labeled data is unavailable or expensive to obtain. Existing works studying weak supervision for NLU either mostly focus on a specific task or simulate weak supervision signals from ground-truth labels. It is thus hard to compare different approaches and evaluate the benefit of weak supervision without access to a unified and systematic benchmark with diverse tasks and real-world weak labeling rules. In this paper, we propose such a benchmark, named WALNUT, to advocate and facilitate research on weak supervision for NLU. WALNUT consists of NLU tasks with different types, including document-level and token-level prediction tasks. WALNUT is the first semi-weakly supervised learning benchmark for NLU, where each task contains weak labels generated by multiple real-world weak sources, together with a small set of clean labels. We conduct baseline evaluations on WALNUT to systematically evaluate the effectiveness of various weak supervision methods and model architectures. Our results demonstrate the benefit of weak supervision for low-resource NLU tasks and highlight interesting patterns across tasks. We expect WALNUT to stimulate further research on methodologies to leverage weak supervision more effectively. The benchmark and code for baselines are available at aka.ms/walnut_benchmark.

## introduction to tackle natural language understanding (nlu) tasks via supervised learning, high-quality labeled examples are crucial. recent advances on large pre-trained language models @xcite @xcite lead to impressive gains on nlu benchmarks, including 1 walnut: semi-weakly supervised learning for natural language understanding testbed weak rules aggregated label rule 1 rule 2 ... rule 50 text 1 1 0 ... 1 1 text 2 0 -1 ... -1 0 ... ... ... ... ... ... text 1000 1 1 ... 1 1 labels text 1 1 text 2 0 text 3 1 walnut semi-weakly supervised learning few labeled data weakly labeled data • 8 nlu tasks with weak rules ○ doc-level classification ○ token-level classification • 5 clean/weak splits per task glue @xcite and superglue @xcite , at the assumption that large amount of labeled examples are available. for many realworld applications, however, it is expensive and time-consuming to manually obtain large-scale high-quality labels, while it is relatively easier to obtain auxiliary supervision signals, or weak supervision, as a viable source to boost model performance without expensive data annotation process. learning from weak supervision for nlu tasks is attracting increasing attention. various types of weak supervision have been considered, such as knowledge bases @xcite , keywords @xcite , regular expression patterns @xcite , and other metadata such as user interactions in social media @xcite . also, inspired by recent advances from semi-supervised learning, semi-weakly supervised learning methods which leverage both a small set of clean labels and a larger set of weak supervision @xcite @xcite @xcite @xcite are emerging to further boost task performance. however, a unified and systematic evaluation benchmark supporting both weakly and semiweakly supervised learning for nlu tasks is rather limited. on the one hand, many existing works only study specific nlu tasks with weak supervision, thus evaluations of proposed techniques leveraging weak supervision on a small set of tasks do not necessarily generalized onto other nlu tasks. on the other hand, some works rely on simulated weak supervision, such as weak labels corrupted from ground-truth labels @xcite , while real-world weak supervision signals can be far more complex than simulated ones. furthermore, existing weakly and semi-weakly supervised approaches are evaluated on different data with different metrics and weak supervision sources, making it difficult to understand and compare. to better advocate and facilitate research on leveraging weak supervision for nlu, in this paper we propose walnut (figure 1 ), a semi-weakly supervised learning benchmark of nlu tasks with real-world weak supervision signals. following the tradition of existing benchmarks (e.g., glue), we propose to cover different types of nlu tasks and domains, including document-level classification tasks (e.g., sentiment analysis on online reviews, fake news detection on news articles), and token-level classification tasks (e.g., named entity recognition in news and biomedical documents). walnut provides few labeled and many weakly labeled examples (figure 1 ) and encourages a consistent and robust evaluation of different techniques, as we will describe in section 3. in addition to the proposed benchmark, in section 4.2 we shed light on the benefit of weak supervision for nlu tasks in a collective manner, by evaluating several representative weak and semiweak supervision methods for and several base models of various sizes (e.g., bilstm, bert, roberta), leading to more than 2,000 groups of experiments. our large-scale analysis demonstrates that weak supervision is valuable for low-resource nlu tasks and that there is large room for performance improvement, thus motivating future research. also, by computing the average performance across tasks and model architectures, we show surprising new findings. first, simple techniques for aggregating multiple weak labels (such as unweighted majority voting) achieve better per-formance than more complex weak supervision paradigms. second, weak supervision has smaller benefit in larger base models such as roberta, because larger pre-trained models can already achieve impressively high performance using just a few clean labeled data and no weakly labeled data at all. we identify several more challenges on leveraging weak supervision for nlu tasks and shed light on possible future work based on walnut. the main contributions of this paper are: (1) we propose a new benchmark on semi-weakly supervised learning for nlu, which covers eight established annotated datasets and various text genres, dataset sizes, and degrees of task difficulty; (2) we conduct an exploratory analysis from different perspectives to demonstrate and analyze the results for several major existing weak supervision approaches across tasks; and (3) we discuss the benefits and provide insights for potential weak supervision studies for representative nlu tasks. token-level classification weak supervision has also been studied for token-level classification (sequence tagging) tasks, focusing on named entity recognition (ner). one of the most common approaches is distant supervision @xcite , which uses knowledge bases to heuristically annotate training data. besides distant supervision, several weak supervision approaches have recently addressed ner by introducing various types of labeling rules, for example based on keywords, lexicons, and regular expressions @xcite @xcite @xcite . walnut integrates existing weak rules into a unified representation and evaluation format. ## baseline evaluation in walnut in this section, we describe the baselines and evaluation procedure (section 4.1), and discuss evalua-tion results in walnut (section 4.2). our results highlight the value of weak supervision, important differences across different baselines, and the potential utility of walnut for future research on weak supervision. ## conclusions motivated by the lack of a unified evaluation platform for semi-weakly supervised learning for lowresource nlu, in this paper we propose a new benchmark walnut covering a broad range of data domains to advocate research on leveraging both weak supervision and few-shot clean supervision. we evaluate a series of different semi-weakly supervised learning methods with different model architecture on both document-level and tokenlevel classification tasks, and demonstrate the utility of weak supervision in real-world nlu tasks. we find that no single semi-weakly supervised learning method wins on walnut and there is still gap between semi-weakly supervised learning and fully supervised training. we expect walnut to enable systematic evaluations of semi-weakly supervised learning methods and stimulate further research in directions such as more effective learning paradigms leveraging weak supervision. we fine-tune these models (via the huggingface library) using task-specific classification heads on top of the encoder and a learning rate of 0.00001.