SPEAR : Semi-supervised Data Programming in Python (2022)

Abstract: We present SPEAR, an open-source python library for data programming with semi supervision. The package implements several recent data programming approaches including facility to programmatically label and build training data. SPEAR facilitates weak supervision in the form of heuristics (or rules) and association of noisy labels to the training dataset. These noisy labels are aggregated to assign labels to the unlabeled data for downstream tasks. We have implemented several label aggregation approaches that aggregate the noisy labels and then train using the noisily labeled set in a cascaded manner. Our implementation also includes other approaches that jointly aggregate and train the model for text classification tasks. Thus, in our python package, we integrate several cascade and joint data-programming approaches while also providing the facility of data programming by letting the user define labeling functions or rules. The code and tutorial notebooks are available at https://github.com/decile-team/spear . Further, extensive documentation can be found at https://spear-decile.readthedocs.io/ . Video tutorials demonstrating the usage of our package are available https://youtube.com/playlist?list=PLW8agt_HvkVnOJoJAqBpaerFb-z-ZlqlP . We also present some real-world use cases of SPEAR.

## introduction supervised machine learning approaches require large amounts of labeled data to train robust machine learning models. for classification tasks such as spam detection, (movie) genre categorization, sequence labelling, and so on, modern machine learning systems rely heavily on humanannotated gold labels. creating labeled data can be a time-consuming and expensive procedure that necessitates a significant amount of human effort. to reduce dependence on human-annotated labels, various techniques such as semi-supervision, distant supervision, and crowdsourcing have been proposed. in order to help reduce the subjectivity and drudgery in the labeling process, several recent data programming approaches @xcite @xcite have proposed the use of human-crafted labelling functions or automatic lfs @xcite to weakly associate labels with the training data. users encode supervision in the form of labelling functions (lfs), which assign noisy labels to unlabeled data, reducing dependence on human labeled data. lfs can defined as first-order logic rules as a composition of semantic role attributes @xcite or syntactic grammar rules @xcite . while most data-programming approaches cited above provide their source code in the public domain, a unified package providing access to all data programming approaches is however missing. in this work, we describe spear, a python package that implements several existing data programming approaches while also providing a platform for integrating and benchmarking newer ones. inspired by frameworks such as snorkel @xcite @xcite and algorithm based labeling in matlab foot_0 , we provide a facility for users to define lfs. further, we develop and integrate several recent data programming models that uses these lfs. we provide many easy-to-use jupyter notebooks and video tutorials for helping new users get quickly started. though we provide implementation on 5 text datasets, our package can be easily integrated with vision and speech datasets as well. the users can get started by installing the package using the below command. ## package flow the spear package consists of three components (and they are applied in the same order): (i) designing lfs, (ii) applying lfs, and (iii) applying a label aggregator (la). initially, the user is expected to declare an enum class listing all the class labels. the enum class associates the numeric class label with the readable class name. as part of (i), spear provides the facility for manually creating lfs. lfs can be in the form of regex rules as well. additionally, we also provide the facility of declaring a @preprocessor decorator to use an external library such as spacy foot_1 , nltk, etc. which can be optionally invoked by the lfs. thereafter, as part of (ii), the lfs can be applied on the unlabeled (and labeled) set using an apply function that returns a matrix of dimension #lfs × #instances. the matrix is then provided as input to the selected label aggregator (la) in (iii), as shown in figure1. we integrate several la options into spear. each la aggregates multiple noisy labels (obtained from the lfs) to associate a single class label with an instance. additionally, we have also implemented in spear, several joint learning approaches that employ semi-supervision and feature information. the high-level flow of the spear library is presented in figure 1 . ## designing and applying lfs user interacts with the library by designing labeling functions. similar to @xcite , labeling functions are python functions which take a candidate as an input and either associates class label or abstains. however, continuous lfs returns a continuous score in addition to the class label. these continuous lfs are more natural to program and lead to improved recall @xcite . ## models we implement several data-programming approaches in this demonstration that includes simple baselines such as fully-supervised, semisupervised and unsupervised approaches. as another unique feature (c.f. ## experiments we prepared jupyter tutorial notebooks for two standard text classification datasets, namely sms, youtube and trec. we took lfs on these datasets from @xcite and train using approaches implemented in this paper. figure 3 shows performance of various approaches implemented using our package on additional two datasets, mit-r and imdb. we can integrate image classification tasks by defining appropriate feature extraction module and rules. ## conclusion and future work spear is a unified package for semi-supervised data programming that enables quick annotation of training data and facilitates training of machine learning models. it eases the use of developing lfs and label aggregation approaches. this allows for better reproducibility, benchmarking and easier ml development in low-resource settings such as in textual post-editing. presently, we are integrating automatic lf induction approaches such as snuba @xcite ) that employ a small labeled set to induce lfs automatically. this will significantly increase the scope of labeling datasets, reducing the extent of human intervention in designing lfs. the package is written in python3 and open-sourced with a mit license 7 , open for community contribution.