MEGA nno+: A Human- LLM Collaborative Annotation System (2024)

Abstract: Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels. We present MEGAnno+, a human-LLM collaborative annotation system that offers effective LLM agent and annotation management, convenient and robust LLM annotation, and exploratory verification of LLM labels by humans.

## introduction data annotation has long been an essential step in training machine learning (ml) models. accurate and abundant annotations significantly contribute to improved model performance. despite the recent advancements of pre-trained large language models (llm), high-quality labeled data remains crucial in various use cases requiring retraining. for instance, distilled models are often deployed in scenarios where repeated usage of llms for inference can be too costly (e.g., api calls) or timeconsuming (e.g., hosting on-premise). in specialized domains like medical and human resources, organizations often need customized models to meet heightened accuracy requirements and ensure the privacy of sensitive customer data. in addition to the training step, accurate labeled data is also necessary for evaluating and understanding of model performance. recent explorations @xcite have showcased the potential of llms in automating the data annotation process. unlike previous task-specific machine learning models, llms exhibit remarkable flexibility to handle any 1 demo & video: https://meganno.github.io textual labeling task as long as suitable prompts are provided. besides, compared to traditional annotation relying solely on human labor, llms can usually generate labels faster and at a lower cost. for example, hiring crowd workers for labeling may encounter problems such as delays, higher cost, difficulty in quality control @xcite @xcite . studies @xcite show that llms can achieve near-human or even betterthan-human accuracy in some tasks. furthermore, downstream models trained with llm-generated labels may outperform directly using an llm for inference @xcite . despite these advancements, it is essential to acknowledge that llms have limitations, necessitating human intervention in the data annotation process. one challenge is that the performance of llms varies extensively across different tasks, datasets, and labels @xcite . llms often struggle to comprehend subtle nuances or contexts in natural language, making involvement of humans with social and cultural understanding or domain expertise crucial. additionally, llms may produce biased labels due to potentially biased training data @xcite . in such cases, humans can recognize potential biases and make ethical judgements to correct them. in this work, we present meganno+, an annotation system facilitating human-llm collaboration through efficient llm annotation and selective human verification. while llm annotations are gaining interest rapidly, a comprehensive investigation on how to onboard llms as annotators within a human-in-the-loop framework in labeling tools has not been conducted yet. for example, supporting llm annotation requires not only user-friendly communications with llms, but also a unified backend capable of storing and managing llm models, labels, and additional artifacts. efficient human verification calls for a flexible search and recommendation feature to steer human efforts towards problematic llm labels, along with a mechanism for humans to review and rectify llm labels. throughout the paper, we explain how we achieve this in our system, showcase a use case, and discuss our findings. we summarize our contributions as below: ## related work llms as annotators there is growing interest in utilizing llms as general-purpose annotators for natural language tasks @xcite @xcite . wang et al. (2021) find that gpt-3 can reduce labeling cost by up to 96% for classification and generation tasks. similarly, ding et al. (2023) evaluate gpt-3 for labeling and augmenting data in classification and token-level tasks. other studies show that for some classification tasks, llms can even outperform crowdsourced annotators @xcite @xcite . ## design considerations let us start with a motivating example of moana, a data scientist working at a popular newspaper. moana is tasked with training a model to analyze the degree of agreement between user comments and political opinion pieces -e.g., whether the comments entail the opinion. moana opts for llm annotation, but she encounters various challenges in the process. firstly, without any guidance for prompting, she resorts to trial-and-error to eventually identify a suitable prompt for the task. even so, she must perform additional validations to ensure that the annotated labels are within the space of pre-defined labels. moreover, the api calls to the llm can be unreliable, throwing exceptions such as timing out and rate limit violations, requiring her to handle such errors manually. next, moana lacks the confidence to train a downstream model without verifying the llm annotations. however, without any assistance in reviewing potential annotation candidates for verification, she has to go through all the annotations, which can be timeconsuming. finally, she has to manually save used model configurations to reuse the model for additional datasets. from moana's example, we summarize our design requirements for a human-llm collaborative annotation system as follows: to satisfy these design requirements, we implement our system as an extension to meganno @xcite , an in-notebook exploratory annotation tool. its flexible search and intelligent recommendations enable efficient allocation of human and llm resources toward crucial data points (r 2a,2b). additionally, meganno provides a cohesive backend for the storage of data, annotations, and auxiliary information (r 1d,1e). ## use case: natural language inference moana, the aforementioned data scientist who needs to collect training data quickly, decides to use meganno+ to leverage llm-powered annotation. first, she imports her unlabeled data and sets the labeling schema as entailment or not entailment. she selects a gpt-3 davinci model with the default parameters and prompt template. to test this setting, she runs the model on 10 samples. after the job is finished, the annotation summary (fig. 4 ) shows that all samples are successfully an-notated by gpt-3 and 40% are entailment. also, one response is annotated with 'notentailed', exemplifying the instability of llms even with clear instructions. with meganno+'s table view widget, she examines data and labels (fig. 5 ). she realizes that some of the records labeled as 'not entailment' are contradictory whereas the rest are neutral. she updates the labeling schema to contain entailment, neutral, and contradiction. next, she wonders if changing the model's temperature would improve the accuracy of annotation. she creates another agent, gpt-3 with temperature with zero and re-runs annotation on the same subset. 1 @xmath0: 'davinci', ' temperature': 0} 2 @xmath1.create_agent(model_config2, template) 3 @xmath2.run_job(agent2, subset) she exports the annotations from both jobs and compares them. she concludes that the second model is good enough for her project. she imports her entire data and uses the agent to label them. since the size of the data is huge, she has to wait till the annotations are done. fortunately, with meganno+, she can track the progress in the output cell while the job is running. to review the annotations, she sorts the annotations in an ascending order of confidence and manually verifies low confidence (< 95%) annotations. ## discussion how to design an annotation task? based on our experience, we find that designing an annotation task and a prompt similar to more widely used and standardized nlp tasks is beneficial. for example, framing moana's problem as a natural language inference task is more effective than framing it as a binary classification of agreement and disagreement. also, the selection of label options may work better if it is similar to common options for given tasks, such as [positive, neutral, negative] > [super positive, positive, ..., negative] for sentiment classification. lastly, it is recommended that the format of a prompt be similar to the one used in training as some llms have different prompt format than the others. we plan to conduct more systematic test to discover reasonable default prompts for different models. ## conclusion meganno+ is a text annotation system for human-llm collaborative data labeling. with our llm @xmath3, reliable and high-quality labels can be collected efficiently. our tool supports robust llm annotation, selective human verification, and effective management of llms, labels, and metadata. as future work, we are currently working on adding more llm agents (e.g., open-source llms), supporting customized extraction of metadata (e.g., custom uncertainty metric), and improving prompt template ui for data-aware in-context learning. additionally, we plan to incorporate diverse annotation workflows such as multi-agent llm @xmath4; and llm @xmath5.