Programmable Annotation with Diversed Heuristics and Data Denoising (2022)

Abstract: Neural natural language generation (NLG) and understanding (NLU) models are costly and require massive amounts of annotated data to be competitive. Recent data programming frameworks address this bottleneck by allowing human supervision to be provided as a set of labeling functions to construct generative models that synthesize weak labels at scale. However, these labeling functions are difficult to build from scratch for NLG/NLU models, as they often require complex rule sets to be specified. To this end, we propose a novel data programming framework that can jointly construct labeled data for language generation and understanding tasks – by allowing the annotators to modify an automatically-inferred alignment rule set between sequence labels and text, instead of writing rules from scratch. Further, to mitigate the effect of poor quality labels, we propose a dually-regularized denoising mechanism for optimizing the NLU and NLG models. On two benchmarks we show that the framework can generate high-quality data that comes within a 1.48 BLEU and 6.42 slot F1 of the 100% human-labeled data (42k instances) with just 100 labeled data samples – outperforming benchmark annotation frameworks and other semi-supervised approaches.

## introduction modern machine learning systems require large amounts of labeled data. for many applications, such labeled data is created by getting humans to explicitly label each training example. however, the standard labeling process that involves wizardof-oz @xcite and other crowd-sourcing approaches (e.g. @xcite @xcite ) is restricted to the level of individual examples, and so are slow and static @xcite . as a result, they are not only costly but require relabeling for any fine-grained domain revisions. data programming is a successful paradigm where humans provide low cost labeling rules written in programming languages to build imperfect training sets, which are then denoised for further improvements @xcite . however, two caveats exist: (1) heuristic rules are costly to construct from scratch as exhaustive alignments between text and sequence labels have to be specified manually @xcite , and especially strenuous for language generation that transforms meaning representations (mr) or slots into textual descriptions (e.g. @xcite ) where nlg is a one-tomany process (i.e. ). (2) labeling functions make decisions based on discrete rules that heavily limit the framework capability in making fine-grained decisions due to the lack of probabilistic information that guides the rule inference @xcite . to this end, we present a new data programming framework where language understanding and generation data can be jointly labeled. we argue that joint nlg/nlu annotation not only improves the overall data quality, but provide a greater degree of compositionality where semantic units such as slotvalue pairs can be individually controlled. we tar-get a weak supervision scenario (shown in figure 1 ) consisting of small, high-quality expert-labeled data and a large set of unlabeled mr or text instances. in this framework, subject-matter experts are to use labeling rules to modify the automaticallyinferred semantic alignments between mr and text, which are probabilistic rules that can be used for joint nlu/nlg labeling. the rules help to construct weak data via iterative denoising, before the dually-regularized nlu and nlg models can learn from the clean seed data to generate high-quality data. this work makes the following contributions: ## annotation framework summary here we formally describe the joint annotation framework. let x denote the set of meaning representation 2 (mr) instances and y denote the text sequences. in our setting, we have (1) a seed dataset s which consists of k labeled pairs, and (2) a large unlabeled mr or text set u where |u | ≫ @xmath0. the annotation framework targets the creation of labeled samples @xmath1, y 1 ), . . . , (x n , y n )} where x @xmath2. to do so, we construct a probabilistic rule set based on the seed data s (see §4) and outline this process in figure 2 . in particular, we draw the connection between the rule set compositionality and coverage to data denoising and highlight the framework advantages. the resulting rule set allows to create a large set of noisily-labeled data, where the framework then learns from the mixture of clean (seed) and noisy data in the process of data denoising (see §5) to create a higher quality set of data.