A nno LLM : Making Large Language Models to Be Better Crowdsourced Annotators (2024)

Abstract: Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance. However, data annotation is time-consuming and expensive, especially when the task involves a large amount of data or requires specialized domains. Recently, GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. In this paper, we first claim that large language models (LLMs), such as GPT-3.5, can serve as an excellent crowdsourced annotator when provided with sufficient guidance and demonstrated examples. Accordingly, we propose AnnoLLM, an annotation system powered by LLMs, which adopts a two-step approach, explain-then-annotate. Concretely, we first prompt LLMs to provide explanations for why the specific ground truth answer/label was assigned for a given example. Then, we construct the few-shot chain-of-thought prompt with the self-generated explanation and employ it to annotate the unlabeled data with LLMs. Our experiment results on three tasks, including user input and keyword relevance assessment, BoolQ, and WiC, demonstrate that AnnoLLM surpasses or performs on par with crowdsourced annotators. Furthermore, we build the first conversation-based information retrieval dataset employing AnnoLLM. This dataset is designed to facilitate the development of retrieval models capable of retrieving pertinent documents for conversational text. Human evaluation has validated the dataset’s high quality.

## introduction labeled data refers to a dataset that has been manually annotated with predefined target labels or categories. it is crucial to develop machine learning models for many nlp tasks, such as sentiment analysis @xcite , machine translation * work done during internship at microsoft research asia. † corresponding author. @xcite and word sense disambiguation @xcite . the process of labeling data is typically done by human annotators under specific guidelines and criteria on how to assign labels to each instance in the dataset. for example, in sentiment analysis, each sentence or document may be labeled with a polarity score such as "positive", "negative", or "neutral". however, it is very labor-intensive and time-consuming to create a large dataset with human annotation, which limits the availability of such data in various nlp tasks. previous works have shown that llms, such as gpt-3 @xcite and palm @xcite , achieve impressive results in many downstream tasks without requiring largescale task-specific data or parameter tuning, but only with a few examples as instructions. ope-nai has recently launched the gpt-3.5 series models, the upgraded versions of gpt-3. shortly after, openai also unveiled chatgpt, another fine-tuned version of gpt-3.5, which has gained significant global attention since its launch. augmenting manually labeled data with pseudolabeled data from gpt-3 is helpful for many nlp tasks, particularly when the labeling budget is restricted @xcite . however, the quality of gpt-3's labeled data still lags behind that of manually labeled data. considering the gpt-3.5 models' remarkable zero/few-shot capabilities, we raise an essential and significant inquiry: can gpt-3.5 potentially replace crowdsourced annotators? before answering this question, let us go over the process of crowdsourced data annotation. first, we need to provide annotators with a specific definition of the task. then, for classification tasks, we need to tell annotators the specific meanings of each category. finally, we need to provide annotators with a few examples that have already been annotated as references. naturally, we can guide gpt-3.5 to annotate data using the same approach as with human annotators by providing task defini-1. create the few-shot cot prompt with annotation guidances and rationales. 2. annotate the data for the requester. ## approach providing detailed instructions is crucial for crowdsourced workers to annotate data, as it helps them better understand task requirements and annotation standards, ultimately improving the quality and accuracy of annotated data. the instructions for each 1 conir is available at: https://github.com/nlpcode/ annollm . motivated by the guidance to human annotators, we will introduce how to convert gpt-3.5 into a zero-shot data annotator by providing guidance on the task description and category definitions in section 2.1. then, we will show how to transform gpt-3.5 into a few-shot data annotator using demonstrated examples in section 2.2. to make it easier to understand, we have provided a visual representation of the crowdsourcing annotation and annollm in figure 1 . finally, in section 2.3, we will demonstrate the utilization of annollm for constructing the conversation-based information retrieval dataset. ## experiment on data creation datasets. we construct the conversation-based information retrieval (conir) dataset based on the ms-marco passage ranking dataset @xcite . the sizes of the training and test sets for conir are 71,557 and 3,000 respectively. implementation details. since chatgpt is optimized for chat, we use it to create conir, namely using it to enrich paragraphs, generate and filter out irrelevant conversations in appendix f. following previous work @xcite , we resort to mrr@10 and recall of top-k (r@k) to evaluate the retrieval performance on different models. zero-shot performance. we train two typical dense retrieval models, dpr @xcite (initialized with distilbert @xcite ) and prod @xcite , on ms-marco, and then evaluate them on the test set of conir. notably, both models exhibit poor performance on conir, as demonstrated in in-domain performance. as shown in human evaluation. we randomly select 100 generated conversations and their paired paragraphs. three annotators are asked to assess the fluency of conversations on a 5-point likert scale from 1 (not fluent) to 5 (extremely fluent), and their relevance and factual consistency with the paired passages on a 3-point likert scale. recent research has shown that gpt-3.5 has strong few-shot and zero-shot learning abilities on various nlp tasks @xcite . in this paper, we first propose that we can readily change gpt-3.5 to a good data annotator for a specific task by providing the detailed annotation instructions similar to human annotators. ## conclusion in this paper, we present annollm, a novel annotation system powered by llms that has the potential to replace traditional crowdsourced annotators. an-nollm adopts a two-step approach, explain-thenannotate. in this method, llms are initially employed to generate a few-shot cot prompt, which is subsequently utilized to prompt llms in annotating unlabeled data. our experimental results on three datasets demonstrate the feasibility of using annollm to substitute crowdsourced annotators. moreover, we introduce the conir dataset, which is created using annollm, to facilitate the research on conversation-based information retrieval.