LEAN - LIFE : A Label-Efficient Annotation Framework Towards Learning from Explanation (2020)

Abstract: Successfully training a deep neural network demands a huge corpus of labeled data. However, each label only provides limited information to learn from, and collecting the requisite number of labels involves massive human effort. In this work, we introduce LEAN-LIFE, a web-based, Label-Efficient AnnotatioN framework for sequence labeling and classification tasks, with an easy-to-use UI that not only allows an annotator to provide the needed labels for a task but also enables LearnIng From Explanations for each labeling decision. Such explanations enable us to generate useful additional labeled data from unlabeled instances, bolstering the pool of available training data. On three popular NLP tasks (named entity recognition, relation extraction, sentiment analysis), we find that using this enhanced supervision allows our models to surpass competitive baseline F1 scores by more than 5-10 percentage points, while using 2X times fewer labeled instances. Our framework is the first to utilize this enhanced supervision technique and does so for three important tasks – thus providing improved annotation recommendations to users and an ability to build datasets of (data, label, explanation) triples instead of the regular (data, label) pair.

## introduction deep neural networks have achieved state-of-theart performance on a wide range of sequence labeling and classification tasks such as named entity recognition (ner) @xcite , relation extraction (re) @xcite @xcite , and sentiment analysis (sa) @xcite . however, they only yield such performance levels the increase is caused by the absorption of uv radiation by the oxygen and ozone. ## system overview as shown in fig. 2 , our framework consists of two main components, a user-friendly web-ui that can capture labels and explanations for labeling decisions, and a weak supervision framework that parses explanations for the creation of weakly labeled data. the framework then uses this weakly labeled data in conjunction with user-provided labels to train models for improved annotation recommendations. our ui shows annotators unlabeled instances (can be sampled using active learning), along with annotation recommendations in an effort to reduce annotation costs. we use py-torch to build our models and implement an api for communication between the web-ui and our weak supervision framework. the learned parameters of our framework are updated in an online fashion, thus improving in near real time. we will first touch on the annotation ui ( §3) and then go into our weak supervision framework ( §4). ## ui for capturing human explanation the emphasis of our front-end design is to simplify the capture of both label and explanation for each labeling decision, while reducing annotation effort via accessible annotation recommendation. our framework supports two forms of explanations, triggers and natural language. a trigger is a group of words in the sentence being annotated that aided the annotator's labeling decision, while natural language is a written explanation of the labeling decision. this section presents first the ui for capturing triggers ( §3.1) and then the ui for capturing natural language explanations ( §3.2). ## lean-life framework our weak supervision framework is composed of two main components, a weak labeling module that parses explanations to create labeling rules and a downstream model. the framework parses user-provided explanations to generate weakly labeled data and then trains the appropriate downstream model with this augmented training data. our weak labeling module supports both explanation formats provided to the annotator in the ui--triggers and natural language. this section first introduces how the module utilizes triggers ( §4.1) and then presents how the module deals with natural language( §4.2). ## experiments we conduct extensive experiments investigating label efficiency to prove the effectiveness of our annotation models. we found that using natural language explanations for re and sa, and trigger explanations for ner provided the best results. for the downstream model portion of our weak supervision framework, we use common supervised method for each task: (1-re) blstm+att @xcite adds an attention layer onto lstm to encode an sequence. tasks and datasets we test our implementation on three tasks: re, sa, ner. we use ta-cred @xcite for re, restaurant review from semeval 2014 task 4 for sa, and laptop reviews @xcite for ner. ## related works leveraging natural language explanations for additional supervision has been explored by many works. @xcite first demonstrated the idea of using natural language explanations for weak labeling by jointly training a task-specific semantic parser and label classifier to generate weak labels. this method is limited though, as the parser is too tightly coupled to the already labeled data, thus their weak learning framework is not able to build a much larger dataset than the one it already has. to address this issue, @xcite proposed a weak supervision framework that utilizes a more practical rule-based semantic parser. the parser constructs a logical form for an explanation that is then used as a labeling function--this resulted in a significant increase of the training set. another effort can be found in @xcite work to extend the stanford natural language inference dataset with natural language explanations--this extension was done for the important textual entailment recognition task. they demonstrate the usefulness of explanations as an additional training signal for learning more comprehensive sentence representations. even earlier @xcite explored breaking down natural language explanation into linguistic sub-structures for learning collections of neural modules which can be assembled into neural networks. our framework is very related to the above weak supervision methods via explanation. another approach to weak supervision is attempting to transfer knowledge from a related source to the target domain corpus @xcite . @xcite and @xcite proposed using a domain-specific dictionary for matching on the unannotated target corpus. both efforts employ partial crfs @xcite which assign all possible labels to unlabeled words and maximize the total probability. this approach addresses the incomplete annotation problem, but heavily relies on a domain-specific seed dictionary. ## conclusion in this paper, we propose an open-source webbased annotation framework lean-life that not only allows an annotator to provide the needed labels for a task, but can also capture explanation for each labeling decision. such explanations enable a significant improvement in model training while only doubling per instance annotation time. this increase in per instance annotation time is greatly outweighed by the benefits in model training, especially in a low resource settings, as proven by our experiments. this is an important consideration for any annotation framework, as the quicker the framework is able to train annotation recommendation models to reach high performance, the sooner the user receives useful annotation recommendations, which in turn cut down on the annotation time required per instance. better training methods also allow us to fight the potential generation of noisy data due to inaccurate annotation recommendations. we hope that our work on lean-life will allow for researches and practitioners alike to more easily obtain useful labeled datasets and models for the various nlp tasks they face.