Learning with Different Amounts of Annotation: From Zero to Many Labels (2021)

Abstract: Training NLP systems typically assumes access to annotated data that has a single human label per example. Given imperfect labeling from annotators and inherent ambiguity of language, we hypothesize that single label is not sufficient to learn the spectrum of language interpretation. We explore new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples. Introducing such multi label examples at the cost of annotating fewer examples brings clear gains on natural language inference task and entity typing task, even when we simply first train with a single label data and then fine tune with multi label examples. Extending a MixUp data augmentation framework, we propose a learning algorithm that can learn from training examples with different amount of annotation (with zero, one, or multiple labels). This algorithm efficiently combines signals from uneven training data and brings additional gains in low annotation budget and cross domain settings. Together, our method achieves consistent gains in two tasks, suggesting distributing labels unevenly among training examples can be beneficial for many NLP tasks.

## introduction crowdsourcing annotations @xcite has become a common practice for developing natural language processing benchmark datasets. even after thorough quality control, it is often infeasible to reach complete annotator agreement, as annotators make mistakes @xcite and ambiguity is a key feature of human communication @xcite . rich prior works @xcite @xcite @xcite show 1 code and data split is available at https://github. com/szhang42/uneven_training_data . i have 100 unlabeled examples and a budget to collect 100 labels to build a training corpus. how should i distribute the labels? ## data configuration we first describe our training data configuration and then discuss our learning algorithms. we notate the input feature vector as x and output label distribution as y. we have three types of training example: unlabeled data set x @xmath0, x 2 u . . . , x un u )}, where u n is the total number of unlabeled examples, single label data set x @xmath1, y 1 s ), (x 2 s , y 2 s ) . . . , (x sn s , y sn s )} where s n is the total number of single label examples, and multi label data set we propose a set up where we distribute annotation label budget unevenly across training examples, resulting in unlabeled examples, single label examples, and multi label examples. we do not collect any new annotations in this work, and re-use dataset from prior work @xcite by resplitting existing datasets to simulate different label distribution scenarios. for each task, we study x s setting, which consider a fixed number of supervised, single label example. then, we introduce x s + x m setting, which includes multi label examples and single label examples (but fixing the amount of total annotation same as the x s setting). lastly, we study adding unlabeled examples x u to both settings. ## learning we introduce learning algorithms that can handle different types of training data. we describe feature extractors for both tasks, which maps natural language to a dense vector representation x then discuss learning algorithms. in the learning algorithms, we first discuss learning with annotated examples only (single label and multi label) and describe learning strategy to integrate unlabeled data. all learning configurations are optimized with the cross entropy (ce) loss. ## experiments we present performances of our labeling scheme and learning framework in this section. all experimental results are rerun three times with different random seeds to determine the variance, which is small. foot_2 ## related work assessing the annotation cost associated with learning has long been studied @xcite . @xcite studies the tradeoff between collecting multiple labels per example vs. annotating more examples. researchers have also explored different data labeling strategies, such as active learning @xcite , providing fine-grained rationales @xcite , retrospectively studying the amount of training data necessary for generalization @xcite , and the policy learning approach @xcite . in this work, we study uneven distribution of label annotation budget for training examples, which has not been explored to our knowledge. label propagation has been extensively used to infer pseudo-labels for unlabeled data, which are used to train the classifier @xcite . our use of mixup can be viewed as a way to propagate label information between the single labeled, multi labeled, and unlabeled data. rich prior work studies ambiguity in language interpretations @xcite . a few studies @xcite frame diverging, subjective interpretations as a multi label classification, and few studies @xcite @xcite introduce graded human responses. mayhew et al. (2020) studies training machine translation system with the goal of generating diverse set of reference translations. pavlick and kwiatkowski (2019) examines the distribution behind human references for nli and nie et al. (2020) presents a larger-scale data collection that we build on. earlier version of this paper @xcite ) study capturing inherent human disagreement in the nli task through calibration and using a small amount of multi-annotated training examples. this paper expands upon it, introducing a new learning framework for such uneven label distribution schemes. concurrent to our work, @xcite introduces distributed nli, a new nlu task with a goal to predict the distribution of human judgements by applying additional distribution estimation methods such as monte carlo (mc) dropout and deep ensemble methods. while we share a similar goal, our work focuses on how to distribute training labels across examples and how to learn under this new label distribution scheme. ## conclusion our work demonstrates the benefits from introducing a small amount of multi label examples at the cost of annotating fewer examples. the proposed learning algorithm, extended from mixup, flexibly takes signals from different types of training examples (single label data, multi label data, and unlabeled data) and show gains upon simply combining different datasets in low annotation budget settings. in this work, we retrospectively study with existing data to question original annotation collection designs. exploring reinforcement learning or active learning to predict an optimal distribution of annotation budget will be an exciting avenue for future work.