Is GPT -3 a Good Data Annotator? (2023)

Abstract: Data annotation is the process of labeling data that could be used to train machine learning models. Having high quality annotation is crucial, as it allows the model to learn the relationship between the input data and the desired output. GPT-3, a large-scale language model developed by OpenAI, has demonstrated im- impressive zero- and few-shot performance on a wide range of NLP tasks. It is therefore natural to wonder whether it can be used to effectively annotate data for NLP tasks. In this paper, we evaluate the performance of GPT-3 as a data annotator by comparing it with traditional data annotation methods and analyzing its output on a range of tasks. Through this analysis, we aim to provide insight into the potential of GPT-3 as a general-purpose data annotator in NLP.

## introduction the democratization of artificial intelligence (ai) @xcite aims to provide access to ai technologies to all members of society, including individuals, small-and mediumsized enterprises (smes), academic research labs, and nonprofit organizations. achieving this goal is crucial for the promotion of innovation, economic growth, and fairness and equality. as typical ai models are usually data-hungry, one significant obstacle of ai democratization is the preparation of well-annotated data for training ai models. specifically, supervised learning critically depends on sufficient training data with accurate annotation, but data annotation can be a costly endeavor, particularly for small-scale companies and organizations @xcite . the cost of data annotation typically includes the labor costs associated with the labeling process, as well as the time and resources required to hire, train and manage annotators. additionally, there may be costs associated with the annotation tools and infrastructure needed to support the annotation process. individuals or small-scale organizations may not have resources to annotate sufficient training data, thereby are unable to reap the benefits of contemporary ai technologies. although the development of pretrained language models such as bert @xcite , xlnet @xcite , gpt-2 @xcite and roberta @xcite eases the data-hungry issue to some extent, data annotation remains an unavoidable challenge for supervised model training. gpt-3 @xcite foot_0 is a powerful large language model developed by openai. evaluations show that gpt-3 has gained through pretraining a surprisingly wide range of knowledge, which can be transferred to downstream tasks through knowledge distillation @xcite . we present some examples in appendix a.12. due to the model architecture and pretraining tasks designed for auto-regressive generation, gpt-3 is capable of generating human-like text and performing a broad array of nlp tasks, such as machine translation, summarization, and question-answering. however, the direct use of gpt-3 for inference in a production setting remains challenging due to its size and computational requirements. moreover, such large language models often lack the flexibility of local deployment, since their parameters are usually not publicly available. in contrast, it is often more feasible to use smaller language model models, such as bert base @xcite , in production environments. in this paper, we investigate the ability of gpt-3 to annotate training data for training machine learn-ing models, which can substantially lower the annotation cost and level the playing field for individuals or small organizations, so that they can harness the power of ai in their own missions. the process can be considered as distilling the knowledge of gpt-3 to small networks that can be straightforwardly deployed in production environments. we conduct extensive experiments to evaluate the performance, time, and cost-effectiveness of 3 different gpt-3 based data annotation approaches for both sequence-and token-level nlp tasks. our main contributions can be summarized as follows: ## related work large language models large language models (llms) have made significant progress on natural language processing tasks in recent years. these models are trained with self-supervision on large, general corpora and demonstrate excellent performance on numerous tasks @xcite @xcite @xcite @xcite @xcite . llms possess the ability to learn in context through few-shot learning @xcite . their capabilities expand with scale, and recent research has highlighted their ability to reason at larger scales with an appropriate prompting strategy @xcite @xcite @xcite @xcite @xcite . @xcite investigate methods to utilize gpt-3 to annotate unlabeled data. however, they mainly focus on the generation and sequence classification tasks. in this work, we conduct more comprehensive experiments and analysis on a wider range of settings, covering both sequenceand token-level tasks. in a recent work, @xcite demonstrate a worker-and-ai collaborative approach for dataset creation with a few seed examples, while we also analyze approaches that support zero-shot training data generation, which do not require any seed examples. prompt-learning prompt-learning, also known as prompting, offers insight into what the future of nlp may look like @xcite @xcite . by mimicking the process of pre-training, prompt-learning intuitively connects pre-training and model tuning @xcite . in practice, this paradigm has proven remarkably effective in low-data regimes @xcite @xcite . for instance, with an appropriate template, zeroshot prompt-learning can even outperform 32-shot fine-tuning @xcite . another promising characteristic of prompt-learning is its potential to stimulate large-scale pre-trained language models (plms). when applied to a 10b model, optimizing prompts alone (while keeping the parameters of the model fixed) can yield comparable performance to full parameter fine-tuning @xcite . these practical studies suggest that prompts can be used to more effectively and efficiently extract knowledge from plms, leading to a deeper understanding of the underlying principles of their mechanisms @xcite . ## methodology we study 3 different approaches to utilize gpt-3 for data annotation: 1) prompt-guided unlabeled data annotation (pgda); 2) prompt-guided training data generation (pgdg); and 3) dictionaryassisted training data generation (dadg). illustrations are shown in figure 1 . overall, these 3 approaches can be regarded as in-context learning @xcite , a new paradigm that is getting popular in nlp. under this paradigm, a language model "learns" to do a task simply by conditioning on l iop , a list of input-output pairs (iop). 3 more formally, ## conclusions in this work, we investigate the effectiveness of gpt-3 as a data annotator for various natural language processing (nlp) tasks using three main ap-proaches. our experimental results show that gpt-3 has the potential to annotate data for different tasks at a relatively lower cost, especially for individuals or organizations with limited budgets. with the limited budget, performance of model trained on the gpt-3 annotated data is often comparable to or even better than that trained on human-annotated data. however, it should be noted that the quality of data annotated by gpt-3 still has room for improvement when compared to human-annotated data. we hope the findings in this work can shed the light on automatic data annotation using large language models and provide some insights so that more methods can be proposed to enhance the quality of data created by these models. with everyone being able to create data for their model training, we can pave the way for the democratization of ai. i (bosheng) would like to express my heartfelt gratitude and deepest appreciation to the memory of my beloved father, whose untimely departure last year has left an indelible void in my life. his unwavering support, encouragement, and wisdom were instrumental in shaping my character and nurturing my academic pursuits. i am forever indebted to his enduring love and belief in my abilities. though he is no longer physically present, his spirit continues to guide and inspire me as i embark on this research journey. this work is dedicated to his cherished memory, a constant reminder of his profound impact on my life.