skweak: Weak Supervision Made Easy for NLP (2021)

Abstract: We present skweak, a versatile, Python-based software toolkit enabling NLP developers to apply weak supervision to a wide range of NLP tasks. Weak supervision is an emerging machine learning paradigm based on a simple idea: instead of labelling data points by hand, we use labelling functions derived from domain knowledge to automatically obtain annotations for a given dataset. The resulting labels are then aggregated with a generative model that estimates the accuracy (and possible confusions) of each labelling function. The skweak toolkit makes it easy to implement a large spectrum of labelling functions (such as heuristics, gazetteers, neural models or linguistic constraints) on text data, apply them on a corpus, and aggregate their results in a fully unsupervised fashion. skweak is especially designed to facilitate the use of weak supervision for NLP tasks such as text classification and sequence labelling. We illustrate the use of skweak for NER and sentiment analysis. skweak is released under an open-source license and is available at https://github.com/NorskRegnesentral/skweak

## introduction despite ever-increasing volumes of text documents available online, labelled data remains a scarce resource in many practical nlp scenarios. this scarcity is especially acute when dealing with resource-poor languages and/or uncommon textual domains. this lack of labelled datasets is also common in industry-driven nlp projects that rely on domain-specific labels defined in-house and cannot make use of pre-existing resources. large pretrained language models and transfer learning @xcite @xcite can to some extent alleviate this need for labelled data, by making it possible to reuse generic language representations instead of learning models from scratch. however, except for zero-shot learning approaches @xcite @xcite , they still require some amounts of labelled data from the target domain to fine-tune the neural models to the task at hand. the skweak framework (pronounced /skwi:k/) is a new python-based toolkit that provides solutions to this scarcity problem. skweak makes it possible to bootstrap nlp models without requiring any handannotated data from the target domain. instead of labelling data by hand, skweak relies on weak supervision to programmatically label data points through a collection of labelling functions @xcite @xcite . the skweak framework allows nlp practitioners to easily construct, apply and aggregate such labelling functions for classification and sequence labelling tasks. skweak comes with a robust and scalable aggregation model that extends the hmm model of @xcite . as detailed in section 4, the model now includes a feature weighting mechanism to capture the correlations that may exist between labelling functions. the general procedure is illustrated in figure 1 . another novel feature of skweak is the ability to create labelling functions that produce underspecified labels. for instance, a labelling function may predict that a token is part of a named entity (but without committing to a specific label), or that a sentence does not express a particular sentiment (but without committing to a specific sentiment category). this ability greatly extends the expressive power of labelling functions and makes it possible to define complex hierarchies between categoriesfor instance, company may be a sub-category of org, which may be itself a sub-category of ent. it also enables the expression of "negative" signals that indicate that the output should not be a particular label. based on our experience applying weak supervision to various nlp tasks, we expect this ability to underspecify output labels to be very useful in nlp applications. ## related work weak supervision aims to replace hand-annotated 'ground truths' with labelling functions that are programmatically applied to data points -in our case, texts -from the target domain @xcite @xcite @xcite . those functions may take the form of rule-based heuristics, gazetteers, annotations from crowd-workers, external databases, data-driven models trained from related domains, or linguistic constraints. a particular form of weak supervision is distant supervision, which relies on knowledge bases to automatically label documents with entities @xcite @xcite . weak supervision is also related to models for aggregating crowd-sourced annotations @xcite @xcite . crucially, labelling functions do not need to provide a prediction for every data point and may "abstain" whenever certain conditions are not met. they may also rely on external data sources that are unavailable at runtime, as is the case for labels obtained by crowd-workers. after being applied to a dataset, the results of those labelling functions are aggregated into a single, probabilistic annotation layer. this aggregation is often implemented with a generative model connecting the latent (un-observed) labels to the outputs of each labelling function @xcite @xcite . based on those aggregated labels, a discriminative model (often a neural architecture) is then trained for the task. weak supervision shifts the focus away from collecting manual annotations and concentrates the effort on developing good labelling functions for the target domain. this approach has been shown to be much more efficient than traditional annotation efforts @xcite . weak supervision allows domain experts to directly inject their domain knowledge in the form of various heuristics. another benefit is the possibility to modify/extend the label set during development, which is a common situation in industrial r&d projects. several software frameworks for weak supervision have been released in recent years. one such framework is snorkel @xcite which combines various supervision sources using a generative model. however, snorkel requires data points to be independent, making it difficult to apply to sequence labelling tasks as done in skweak. swellshark @xcite is another framework optimised for biomedical ner. swellshark, is however, limited to classifying already segmented entities, and relies on a separate, ad-hoc mechanism to generate candidate spans. flyingsquid @xcite presents a novel approach based on triplet methods, which is shown to be fast enough to be applicable to structured prediction problems such as sequence labelling. however, compared to skweak, the aggregation model of fly-ingsquid focuses on estimating the accuracies of each labelling function, and is therefore difficult to apply to problems where labelling sources may exhibit very different precision/recall trade-offs. a labelling function may for instance rely on a pattern that has a high precision but a low recall, while the opposite may be true for other labelling functions. such difference is lost if accuracy is the only metric associated for each labelling function. finally @xcite describe a weak supervision model based on an extension of hmms called linked hidden markov models. although their aggregation model is related to skweak, they provide a more limited choice of labelling functions, in particular regarding the inclusion of document-level constraints or underspecified labels. ## labelling functions labelling functions in skweak can be grouped in four main categories: heuristics, gazetteers, machine learning models, and document-level functions. each labelling function is defined in skweak as a method that takes spacy doc objects as inputs and returns text spans associated with labels. for text classification tasks, the span simply corresponds to the full document itself. the use of spacy greatly facilitates downstream processing, as it allows labelling functions to operate on texts that are already tokenised and include linguistic features such as lemma, pos tags and dependency relations. foot_0 skweak integrates several functionalities on top of spacy to easily create, manipulate, label and store text documents. ## aggregation model after being applied to a collection of texts, the outputs of labelling functions are aggregated using a generative model. for sequence labelling, this model is expressed as a hidden markov model where the states correspond to the "true" (unob-served) labels, and the observations are the predictions of each labelling function @xcite . for document classification, this model reduces to naive bayes since there are no transitions. this generative model is estimated using the baum-welch algorithm @xcite , which a variant of em that uses the forward-backward algorithm to compute the statistics for the expectation step. for efficient inference, skweak combines python with c-compiled routines from the hmmlearn package foot_1 employed for both parameter estimation and decoding. ## example with skweak, one can apply and aggregrate labelling functions with a few lines of code: import spacy, re from skweak import heuristics, gazetteers, aggregation, utils # first heuristic (see section 3) @xmath1.functionannotator ("money", money_detector) # detection of years @xmath2.tokenconstraintannotator ("years", lambda tok: re.match ("(19|20)\d{2}@xmath0750 in federal income taxes in 2016") # apply the labelling functions @xmath3.hmm("hmm", ["person", "date", "money"]) hmm.fit_and_aggregate([doc]) # and visualise the result (in jupyter) utils.display_entities(doc, "hmm") skweak's repository provides jupyter notebooks with additional examples and explanations. ## experimental results we describe below two experiments demonstrating how skweak can be applied to sequence labelling and text classification. we refer the reader to @xcite for more results on ner. 4 it should be stressed that the results below are all obtained without using any gold labels. ## conclusion the skweak toolkit provides a practical solution to a problem encountered by virtually every nlp practitioner: how can i obtain labelled data for my nlp task? using weak supervision, skweak makes it possible to create training data programmatically instead of labelling data by hand. the toolkit provides a python api to apply labelling functions and aggregate their results in a few lines of code. the aggregation relies on a generative model that express the relative accuracy (and redundancies) of each labelling function. the toolkit can be applied to both sequence labelling and text classification and comes along a range of novel functionalities such as the integration of underspecified labels and the creation of document-level labelling functions.