Want To Reduce Labeling Cost? GPT -3 Can Help (2021)

Abstract: Data annotation is a time-consuming and labor-intensive process for many NLP tasks. Although there exist various methods to produce pseudo data labels, they are often task-specific and require a decent amount of labeled data to start with. Recently, the immense language model GPT-3 with 170 billion parameters has achieved tremendous improvement across many few-shot learning tasks. In this paper, we explore ways to leverage GPT-3 as a low-cost data labeler to train other models. We find that to make the downstream model achieve the same performance on a variety of NLU and NLG tasks, it costs 50% to 96% less to use labels from GPT-3 than using labels from humans. Furthermore, we propose a novel framework of combining pseudo labels from GPT-3 with human labels, which leads to even better performance. These results present a cost-effective data labeling methodology that is generalizable to many practical applications.

## introduction data always plays a crucial role in developing machine learning models. however, collecting humanlabeled data is a costly and time-consuming process, especially in multi-task scenarios. with the success of pre-trained models @xcite @xcite on unlabeled data, the performance of models under few-shot and zero-shot settings has been greatly enhanced. in particular, the large-scale language model gpt-3 @xcite , with 175 billion parameters, is the state-of-the-art few shot learner on many nlp tasks. however, gpt-3 is constrained on its immense model size and requires a large amount of resource to be deployed for real applications. moreover, gpt-3 doesn't provide a free lunch, and its public api has a charge correlated with the number of processed tokens foot_0 . thus, an interesting problem arises: instead of directly deploying gpt-3 for downstream tasks, how can we leverage gpt-3 to achieve a more cost-effective and efficient training of other models? in this paper, we employ gpt-3 to label unannotated data to train smaller models which are deployed for inference. although the data labeled by gpt-3 is usually more noisy than human-labeled data, the process is much cheaper, faster and generalizable to multiple tasks. for example, for the stanford sentiment treebank (sst-2) task @xcite , it takes as low as 0.002 dollars on average to use the gpt-3 api to annotate one label. however, it costs 0.11 dollars to label an instance on crowd-sourcing platforms. plus, the gpt-3 api can label data non-stoppingly at a much faster speed than human labelers. in our extensive empirical analysis, we find that to make in-house models (e.g. pegasus @xcite , roberta @xcite ) to achieve the same performance on various nlu and nlg tasks, data labeled by gpt-3 incurs a much lower cost (e.g. 50%-95% lower) than data labeled by humans, especially in low-resource settings. moreover, we also find that these in-house models trained with data labeled by gpt-3 can outperform gpt-3 itself under the fewshot setting, which we give theoretical justifications. in addition to using labeled data from a single source, we explore ways to smartly assign unlabeled data to different labelers, i.e. gpt-3 and human, under a fixed budget. we frame this as a dual supervision problem @xcite with cost and budget constraints. in detail, we tried mixing data labeled by gpt-3 and humans with different ratios: 25%, 50%, 75% of the budget. moreover, we propose an active labeling strategy to have humans re-annotate data labeled by gpt-3 with the lowest confidence scores. both strategies manifest clear improvement over using a single source of labeler. we conduct comprehensive empirical analysis of our proposed cost-effective labeling strategies on 9 nlp tasks, including text entailment @xcite , sentiment analysis @xcite , topic classification @xcite , answer type classification (voorhees and tice, 2000), summarization @xcite , and question generation @xcite . we show that our labeling strategy can significantly reduce labeling cost while achieving the same performance with human-labeled data. for instance, our method saves 96% cost on the sentence classification task sst-2, 93.8% cost on the summarization task gigaword, and 50-75% cost on other tasks. we summarize our contributions as follows: ## method in this section, we introduce how gpt-3 can help reduce labeling costs. first, we present a cost analysis of gpt-3 and human labeling. next, we introduce how to use gpt-3 to label unannotated data. then, we theoretically explain why a downstream model trained with gpt-3 labels can outperform gpt-3 itself. finally, we show how to mix up labels from gpt-3 and humans to further boost performance at a lower cost. ## conclusion in conclusion, we investigate how to use gpt-3 to label unannotated data in a cost-efficient way. we show that our strategies can significantly reduce the labeling cost by achieving the same performance with human-labeled data. we also find that models trained with gpt-3 labels can achieve better performance than raw gpt-3. moreover, we introduce the gpt3-human labeling strategy, which outperforms both fully human and fully gpt-3 labeling. finally, we propose active labeling to leverage the advantages from human and gpt-3, which works better than randomly selecting data to label on multiple nlp tasks. our work shows the potential in cost-efficient data labeling with few-shot learners. for future work, we plan to extend our methods to data augmentation to produce both instances and labels. and it is worth noting that gpt-3 is not reliable enough yet at labeling "high-stakes" cases, e.g. identifying toxic language, but is more suitable for low-stakes labeling foot_5 .