Knodle: Modular Weakly Supervised Learning with P y T orch (2021)

Abstract: Strategies for improving the training and prediction quality of weakly supervised machine learning models vary in how much they are tailored to a specific task or integrated with a specific model architecture. In this work, we introduce Knodle, a software framework that treats weak data annotations, deep learning models, and methods for improving weakly supervised training as separate, modular components. This modularization gives the training process access to fine-grained information such as data set characteristics, matches of heuristic rules, or elements of the deep learning model ultimately used for prediction. Hence, our framework can encompass a wide range of training methods for improving weak supervision, ranging from methods that only look at correlations of rules and output classes (independently of the machine learning model trained with the resulting labels), to those that harness the interplay of neural networks and weakly labeled data. We illustrate the benchmarking potential of the framework with a performance comparison of several reference implementations on a selection of datasets that are already available in Knodle.

## introduction most of today's machine learning success stories are built on top of huge labeled data sets. creating and maintaining such data sources manually is a time-consuming, complicated and thus an expensive and error-prone process. various research directions address the hunger for bigger and better datasets. one of the most popular approaches that has recently gained traction is weak supervision. the learning algorithm is confronted with labels which are easy to obtain but are not guaranteed to be correct, and as such often demand denoising. such weak labels are created, for example, with the use of regular expressions, keyword lists or external databases. typically, methods for improving weakly supervised learning (and their respective implementations) are tailored towards domain-specific tasks or integrated with a specific model architecture. examples include the attention-over-instances architecture introduced for relation extraction @xcite , an em-based algorithm used for event extraction @xcite or models of systematic label flips for named entity recognition @xcite . such diversity and specificity of approaches makes it difficult to compare or transfer them across tasks without extensive adjustments dictated by the implementation, the task or the data set. we introduce knodle: a framework for knowledgesupervised deep learning, i.e weak supervision with neural networks. the framework provides a simple tensor-driven abstraction based on pytorch allowing researchers to efficiently develop methods for improving weakly supervised machine learning models and try them interchangeably to find the one that works the best for a given task. within this work, we refer to a denoising method as any method that helps to improve weakly supervised learning regardless the type of noise or bias and the exact level of denoising (weak labels, weak rules etc). the following points summarize knodle's main design goals: ## related work many strategies have been introduced to reduce the need for large amounts of manually labeled data. among these are active learning @xcite , where automatically selected instances are manually annotated by experts, and semi-supervised learning @xcite , where a small annotated dataset is combined with a large unlabeled one. fine-tuning pretrained language models such as bert (?) shows good results if moderate to small amounts of annotations are available. ## weakly supervised learning with knodle the knodle architecture provides a layer of abstraction that allows integrated label improvement and model training with weakly supervised learning signals in pytorch. on the one hand, since knodle has access to the information which rules matched for each sample, it is not restricted to methods that denoise only weak labels, such as cleanlab @xcite . on the other hand, the knodle abstraction also provides access to input and learned representations, and thus does not restrict denoising methods to rely on rule match correlations alone (as snorkel @xcite ). moreover, access to the deep learning model enables the integration of denoising methods that use or manipulate the prediction model itself. to the best of our knowledge, knodle is the first framework to provide a modular architecture for interchangable application of a wide spectrum of denoising algorithms. for that reason we believe that it can become a testbed where different algorithms for improving the weakly supervised data are implemented and compared with each other to find the most fruitful task-to-denoising-method combination or to use it as a foundation for further studies. the framework follows two main design principles, outlined below: formally, assume we have n samples, r rules and k classes. rule matches are gathered in a binary matrix @xmath0,1} n×r , where z @xmath1. the initial mapping from rules to the corresponding classes is given by another binary matrix @xmath2,1} r×k , t @xmath3. this separation between one tensor that contains rule matches and another tensor that translates them to labels allows knodle to access this fine-grained information during training for certain denoising algorithms. this is in contrast to other approaches that treat weak supervision as learning from a noisy heuristic label matrix y @xmath4. ## trainers knodle currently provides several out-of-the-box baselines and trainers, which we outline in the following section. all trainer classes are compatible with any pytorch model. as examples for pytorch classifiers, knodle provides code using logistic regression and huggingface's transformers @xcite . ## datasets apart from denoising methods, knodle includes a few datasets from previous works in the knodle-specific tensor format in order to demonstrate the abilities of the framework. all datasets are rather simple, but have their own peculiarities with respect to the respective z and t matrices, that are worth investigating. the overview of dataset statistics is provided in spam dataset. the first task uses the youtube comments dataset @xcite . here, the task is to classify whether a text is relevant to the video or holds spam, such as advertisement. the dataset has a small size of both train and test sets. thus, a single wrongly labeled instance might have quite a big impact on the learning algorithm. we use the preprocessed version by the snorkel team (snorkel, 2020b). among others, the rules were created based on keywords and regular expressions. spouse dataset. this relation extraction dataset is based on the signal media one-million news articles dataset @xcite . the task is to decide whether a sentence holds a spouse relation or not. again, the preprocessed version by the snorkel team is used (snorkel, 2020a), so the results can be related to previous studies @xcite . the rules are created via a set of known spouse relationships from dbpedia @xcite as well as keywords and encoded language patterns. the difficulty of the spouse dataset is its skewness: over 90% of samples in the test set hold a no-spouse relation. ## experiments the aim of knodle is not to find the best denoising method in general. rather, the goal is to find the method that improves weak labels most for a given task or dataset and its specific properties. thus, knodle supports experimentation to get a better understanding in which settings a certain method works well and when it does not. ## conclusion this work introduces the knowledge-supervised deep learning framework knodle. knodle provides a unified interface to work with multiple weak labeling sources, so that they can be seamlessly integrated with the training of deep neural networks. this is achieved by a tensor-based input format and a intuitive separation of weak supervision aspects and model training. the framework facilitates experimentation that helps researchers to gain better insights into the correspondence between characteristics of weak supervision problems, and the effectiveness of methods for improving weakly supervised learning. from a practical perspective, knodle can be used to compare different denoising methods and select the one that gives the best result for a specific task. knodle's modular approach makes it easy to add new data sets and denoising algorithms. adding functionality to knodle is straightforward, and we do hope that it will encourage researchers to create their own algorithms to improve learning with weakly annotated data, and incorporate them into the knodle framework.