Partially Humanizing Weak Supervision: Towards a Better Low Resource Pipeline for Spoken Language Understanding (2022)

Abstract: Weak Supervised Learning (WSL) is a popular technique to develop machine learning models in absence of labeled training data. WSL involves training over noisy labels which are traditionally obtained from hand-engineered semantic rules and task-specific pre-trained models. Such rules offer limited coverage and generalization over tasks. On the other hand, pre-trained models are available only for limited tasks. Thus, obtaining weak labels is a bottleneck in weak supervised learning. In this work, we propose to utilize the prompting paradigm to generate weak labels for the underlying tasks. We show that task-agnostic prompts are generalizable and can be used to obtain noisy labels for different Spoken Language Understanding (SLU) tasks such as sentiment classification, disfluency detection and emotion classification. These prompts can additionally be updated with human-in-the-loop to add task-specific contexts, thus providing flexibility to design task-specific prompts. Our proposed WSL pipeline outperforms other competitive low-resource benchmarks on zero and few-shot learning by more than 4% on Macro-F1 and a conventional rule-based WSL baseline by more than 5% across all the benchmark datasets. We demonstrate that prompt-based methods save nearly 75% of time in a weak-supervised framework and generate more reliable labels for the above SLU tasks and thus can be used as a universal strategy to obtain weak labels.

## introduction weak supervised learning (wsl) @xcite has gained interest in the research community because of the success shown by leveraging the high availability of large volumes of unlabeled data @xcite . in these weak supervision setups, the unlabeled samples are pseudo-annotated by noisy labels and a noise correction strategy is applied to train the model over such labels. these noisy labels are derived using source(s), commonly known as weak source(s). most common forms of weak sources observed in the area of weak supervision are: rule-based weak sources @xcite and task-specific finetuned models @xcite . rule-based weak sources require designing the labeling functions using the heuristics, lexicons and external knowledge bases (like sentiwordnet @xcite ) to map an input to the class labels expected in the task. it is a challenge to extend such rules to dataset from different domains or to perform a different task. additionally, designing rules for an individual dataset is a manually time intensive task. while, there are a few rule-based sources available for common tasks like sentiment, it is hard to find such readily available weak sources for tasks like disfluency detection @xcite . the other type of weak-sources utilize a task specific fine-tuned language model. for example, bert-ner, which is bert @xcite fine-tuned for ner task, can be utilized as a weak source to identify named entities from the data. such task-specific models cannot be used to predict class labels for a task different than what the model is trained on. the common challenge imposed by both of these weak sources is the lack of generalizability across a wide variety of tasks. in this paper, we propose a universal weak source which not only generates better quality noisy class labels for wide range of tasks, but can also be tweaked to write 'human-in-theloop' task-specific details with minimal efforts. we present prompt-based weak source, a hybrid source which utilizes a pre-trained language model (plm) as a knowledge base and limited human intervention as a prompter to address the labeling problems observed in traditionally used weak sources. a prompt-based weak source requires prompting a plm @xcite @xcite to derive weak class labels. a prompt refers to a pattern string that is designed to coax the model into producing an output corresponding to a given class @xcite . we study different ways to prompt plms in 3.1.1 and 3.1.2. we study 3 key features of prompt-based sources as: generalizability (utilize task-agnostic prompts to cater to various tasks and effectively create prompts for multiple tasks within same generic framework.), flexibility (to modify the prompts to add task and class-label specific contexts in an easy manner to improve over task-agnostic prompts), potency (to derive weak labels with reliable source performance). our major contributions in this work are: ## related work existing works @xcite in wsl learns on a few gold data, while another group of work @xcite @xcite assumes that no labeled data is available. in the scope of our work, we explore approaches that do not rely on labeled data to train a weak supervised model (wsm). @xcite utilized bert to learn conditional reliability scores between multiple weak sources using an attention mechanism, while @xcite proposed a generative model to combine outputs from various weak sources. @xcite proposed a contrastive self-training strategy to learn over weak labels and outperformed prior works @xcite . hence, our work borrows ideas from @xcite to train a weak supervised model (wsm) considering its robustness towards high intensities of label noise. prompt-based methods utilize templates structured as natural language inference (nli)-style prompts (section 3.1.1) or cloze-style prompts (section 3.1.2) in a zero-shot and/or few-shot setup to predict the labels for the downstream task. works such as @xcite demonstrated fewshot training using cloze-style task-agnostic nullprompt, while flan @xcite utilized nli-style instruction templates and performed instruction tuning to improve the zeroshot performance. however, due to the large size of the model (137b parameters), we find the work unsuitable to be used in creating a low resource pipeline. on the other hand, lmbff @xcite and pattern exploiting training (pet) @xcite utilized a relatively smaller plm (340m parameters) on cloze-style prompts. lmbff showed that a few demonstrative examples during task finetuning provide additional context to better learn the prompts and report improvements over pet @xcite . considering the benefits of lmbff @xcite over other methods in creating a low resource pipeline, we utilize this approach to perform prompt-based fine-tuning. ## methodology the proposed methodology is a two-step process (figure 1 ). first, we prompt the plms with 'humanin-the-loop' as a strategy to produce weak labels for the unlabeled training data. next, we train a wsm on these weak labels. in the subsequent sections, we describe the two steps in detail. ## results and analysis we compare our proposed framework (wsl-tap, wsl-tsp) with other baselines in proposed wsl vs low-resource baselines: the results show that a wsm trained on prompt based weak labels (wsl-tap, wsl-tsp) outperforms other baselines including state-of-the-art zero-shot (meta-tuning) and few-shot (dnnc) approaches. wsl-tsp outperforms the few-shot method by more than 4% on mosi, 7% on swbd-d and 9% on iemocap dataset. further results show that training a weak supervision model (wsl-tap, wsl-tsp) over prompt-based weak labels bridges the gap with the oracle model. specifically, training a wsm improves the f1 scores by 1% for mosi, 8.2% for swbd-d and 2.1% for iemocap. this shows the effectiveness of proposed pipelines for training a low-resource model against few-shot and zero-shot methods. proposed wsl vs rule-based wsl: the proposed weak supervision pipeline on prompt-based weak source also outperforms a traditional rule based weak supervision pipeline (wsl-rule). the distinction between the performance of rule and prompt-based wsl pipeline is more evident for mosi and iemocap datasets. the proposed method outperforms rule based pipeline by 10% in mosi, 5% in swbd-d and 31% in iemocap dataset (wsl-tsp vs wsl-rule in task-agnostic vs task-specific prompts: we observe that weak labels obtained from taskspecific prompts (tsp) are consistently better than task-agnostic prompts (tap). likewise, wsl-tsp outperforms wsl-tap on all tasks. while the best results are obtained from task-specific prompts, even task-agnostic prompts outperform a rulebased pipeline. hence, the proposed pipeline (wsl-tap, wsl-tsp) could solve the bottleneck of labeling the data while training a wsm. best wsl scores: finally, we report the best scores obtained with proposed pipeline: @xmath0.26%; @xmath1.86%; @xmath2.47%. we note that the best score obtained on mosi dataset is competitive with the oracle results, which could be explained by a highly reliable weak labels obtained from prompt-based method on mosi. however, the performance on swbd-d and iemocap are lower than oracle (refer to section a.4 for explanation) but are strongly better than state-of-the-art low resource methods on zero-shot and few-shot methods. thus, we show that low resource pipeline for slu tasks could be effectively trained via proposed pipeline. quantifying the importance of human efforts: we outsource the task of curating prompts to 5 proficient english speakers. we observe that it takes around 17.0±6.0 minutes on an average across the speakers to come-up with the first good prompt for the benchmark slu tasks. while creating rules for the same under the restriction of given guidelines (section a.5), it takes roughly around an hour to come up with pattern-based heuristics for a certain dataset. this observation shows that we can save around 72% of annotation time by avoiding rulebased heuristics and relying only on prompt-based weak source. the reason why designing prompts takes relatively lesser time compared to designing rules is due to the fact that rule-based baselines have to be balanced between coverage and precision, along with the higher complexity of coding the rules against the designing the prompts. we consider rules (or a set of rules unioned together) to be good if it performs better than a baseline of majority class in terms of precision and cover at least 25% of the unlabeled samples. additionally, these rules have to written in a framework which is a time-consuming step as well against simply writing a prompt which is nothing but a textual sentence. so, iterating with prompts is much faster as compared to iterating on rules. furthermore, in many cases, a bunch of rules have to be unioned to achieve this criteria on coverage and precision and hence, it takes a longer cycle to identify the set of rules required to generate weakly labeled training data. as it is evident in ## conclusion in this work, we show effectiveness of utilizing prompt-based methods as universal weak sources to develop low-resource models for wide range of benchmark slu tasks. we show that the proposed method outperforms traditional methods of rule based wsl as well as state-of-the-art methods on other low resource settings like zsl and fsl. in future, we would like to study the application of automatic and soft prompts to generate the weak labels and the extension of work to multilingual tasks where limited training data is available. we would also like to explore areas around incorporating the human-in-the-loop feedback during the training process instead of only restricting it to improve on the quality of weak labels.