Ruler: Data Programming by Demonstration for Document Labeling (2020)

Abstract: Data programming aims to reduce the cost of curating training data by encoding domain knowledge as labeling functions over source data. As such it not only requires domain expertise but also programming experience, a skill that many subject matter experts lack. Additionally, generating functions by enumerating rules is not only time consuming but also inherently difficult, even for people with programming experience. In this paper we introduce Ruler, an interactive system that synthesizes labeling rules using span-level interactive demonstrations over document examples. Ruler is a first-of-a-kind implementation of data programming by demonstration (DPBD). This new framework aims to relieve users from the burden of writing labeling functions, enabling them to focus on higher-level semantic analysis, such as identifying relevant signals for the labeling task. We compare Ruler with conventional data programming through a user study conducted with 10 data scientists who were asked to create labeling functions for sentiment and spam classification tasks. Results show Ruler is easier to learn and to use, and that it offers higher overall user-satisfaction while providing model performances comparable to those achieved by conventional data programming.

## introduction machine learning (ml) models used today are predominantly supervised and rely on large datasets labeled for training. however, the cost of collecting and maintaining labeled training data remains a bottleneck for training high-capacity supervised models @xcite . weak supervision methods such as crowdsourcing @xcite , distant supervision @xcite , and user-defined heuristics @xcite enable the use of noisy or imprecise sources to gather large training datasets. data ## dpbd framework problem statement given a dataset @xmath0,...,d m } of data records and a set of labels @xmath1,...,l n }, we aim to develop a framework that enables human labelers to assign a label from l for each data record intelligently sampled from d ⊂ d (|d | |d|), while demonstrating their rationales for label assignments through visual interaction. given a triplet (d i ,v i ,l j ) of a data record, a visual interaction from the labeler, and the label assigned, we want this framework to effectively synthesize and propose labeling rules r @xmath2,...,r k } for the labeler to choose from. finally, we want the framework to optimally aggregate all the chosen rules (labeling functions) in order to create a labeled training set from d\d with probabilistic labels in order to subsequently train discriminative models on it. ## ruler interface ruler is a web-based interactive system that builds on the data programming by demonstration (dpbd) framework introduced above to facilitate labeled training data preparation for document-level text classification models. for this, ruler leverages span-level features and relations in text documents demonstrated through visual interactions by users (labelers), as formalized by the dpbd framework. to begin a labeling task, the data owner needs to upload their unlabelled dataset, in addition to a small labeled development set, and optionally a small test and validation set. this mirrors the data requirements of snorkel, which the underlying dpbd modeler encapsulates. in the rest of this section, we discuss the user interface and interactions of ruler along with its implementation details in operationalizing dpbd for text document classification. recall that the purpose of the labeling interface in dpbd ( section 2.1) is to enable the labeler to encode domain knowledge into rules through visual interaction. to this end, ruler interface provides affordances through 6 basic views (figure 3 ), which we briefly describe below-the letters a-f refer to annotations in figure 3 . labeling pane (a) is the main view where the user interacts with document text. the labeling pane (figure 4 ) shows a single document at a time and supports all the labeling operations defined by the glm in the context of text data. the user can annotate spans by highlighting them directly with the cursor or adding them to a concept. these spans can be linked together if the relationship between them is significant to the user. once the user selects a document label (class) from the options displayed, the system generates a diverse set of labeling functions to suggest to the user. concepts pane (b) allows users to create concepts, add and edit tokens (whole words surrounded by non-alphabetical characters) or regular expressions, and see annotations over their text automatically added when a match is found (figure 5 ). this interaction allows users to abstract away details about specific language use by grouping tokens or regular expressions into concepts. suggested functions (c) shows the labeling functions suggested by the system. the user can select any functions that seem reasonable, and only then are they added to the underlying labeling model that is iteratively built. labeling statistics (d) displays current statistics of the label model computed over the development set, and differential changes incurred by the last data interaction. because this panel updates as the user interacts, the user can explore the space of labeling functions efficiently in terms of time, computation, and human effort. end-model statistics (e) shows the performance statistics for an end-discriminative model for which the user intends to collect training data. for example, in our user study, we used a logistic regression model with a bag of words features on the generated training data. we evaluate this model on the small held-out test set and show the statistics in this pane. this panel updates only when the user chooses to retrain the model, to deter from overfitting to the development set. selected functions (f) lists of currently selected labeling rules that make up the labeling model and shows each rule's performance statistics based on the development set. the user can click to open a details panel showing observed incorrect labels and sample texts labeled by this function. ## evaluation we evaluated ruler alongside manual data programming using snorkel @xcite . although nonprogrammer domain experts are a target audience for this technology, we wanted our evaluation to show that the ruler labeling language is expressive enough to create models comparable to manual data programming. we also wanted to understand the trade-offs afforded by each method in order to help programming-proficient users decide which is best for their situation. in order to make these comparisons, we conducted annotations are color coded by the concepts they are assigned to. this example is from the amazon reviews dataset @xcite . a user study with 10 programming-proficient data scientists and measured their task performance accuracy in completing two labeling tasks using the two methods. in addition to task performance, we analyzed both accessibility and expressivity using the qualitative feedback elicited from participants. in an initial pilot study, we included a third condition, babblelabble @xcite . for this method, users express labeling rationales in natural language, which the tool then parses into labeling rules. participants found babblelabble to be limited in terms of what patterns they could express and how to express them, as they "tried to express it in a parsable sentence" and faced errors. the preliminary results led us to believe that although babblelabble may be suitable for high-volume approaches like crowd-sourcing, it can be frustrating for a domain expert or lay user who is both providing the explanations and creating and debugging the label model. based on these observations, we removed babblelabble from our evaluation. participants we recruited participants with python programming experience through our professional network (none were involved in this project). note that ruler can be used by programmers and non-programmer domain experts alike, but a fair comparison with snorkel requires proficiency in conventional programming. all participants had significant programming experience (@xmath3.1 years, @xmath4.5). their experience with python programming ranged from 2 to 10 years, with an average of 5.2 years (@xmath5.8). ## results to analyze the performance of participants' labeling functions for each condition, we select the labeling model that achieves the highest f1 score on the development set. for example, if a user performs the spam classification task using ruler, and the set of functions that they created 20 minutes into their 30-minute session attains the highest score on the development set, we use that model to evaluate this condition, rather than strictly using whatever model the participant has at the end of the 30-minute session. we use each label model to generate a training dataset, which we then use to train a logistic regression model with bag-of-words features. finally, we evaluate the logistic regression model's performance on a held-out test set (400 examples). we also analyze participants' subjective ratings on a likert scale of 5 (1-5, higher is better) in their exit surveys. we use the paired wilcoxon signed-rank test to assess the significance of differences in prediction metrics and subjective ratings between ruler and snorkel. we also report the effect size r for all our statistical comparisons. ## related work we build on earlier work in weak supervision, programming by demonstration, and learning from feature annotations provided by users. weak supervision in order to reduce the cost of labeled data collection, weak supervision methods leverage noisy, limited, or low precision sources such as crowdsourcing @xcite , distant supervision @xcite , and user-defined heuristics @xcite to gather large training data for supervised learning. data programming @xcite is a programmatic approach to weak supervision by heuristics, where domain experts provide functions which are then used to label training data at scale and train ml models using probabilistic labels. ruler aims to make data programming easier and more accessible for document classification tasks. ## discussion ruler prioritizes accessibility over expressivity. is this trade-off inevitable? there are many ways we could improve the expressivity of ruler, for example, by extending analysis on the context of user demonstrations, or by giving users more direct control over the synthesized labeling functions. however, many of these improvements can only be informed by how people use ruler in real-world applications. with this in mind, deriving additional insights into how users without programming proficiency use ruler is an essential area to explore, and open-sourcing ruler is a step forward in this direction. we especially hope that ruler can be beneficial for domains like healthcare, where the domain expert's time is very valuable and there is little tolerance for low-quality models. future research also includes developing fast search and ranking algorithms, and experimenting with different active learning strategies to effectively search and navigate the vast joint space of labeling functions and data examples. accessibility is a key to wider adoption of any technology and machine learning (ml), especially in data-hungry supervised forms, is no exception. in this paper, we presented ruler, a data programming by demonstration (dpbd) system for quickly generating labeling functions to create training datasets for document-level classification tasks. ruler uses the dpbd framework to convert user rationales, interactively expressed as span-level annotations and relations, to labeling rules. dpdb is a general human-in-the-loop framework that aims to ease writing labeling functions, improving data programming's accessibility and efficiency. through a user study with 10 data scientists performing real-world labeling tasks for classification, we evaluated ruler together with conventional data programming and found that ruler enables more accessible data programming without any loss of performance in the final models. our study results also suggest that ruler may benefit even skilled programmers, as many functions can be captured more easily through visual interactions using our system than by coding them from scratch. we release ruler as open-source software to support future applications and extended research.