Enhancing Text Classification through LLM -Driven Active Learning and Human Annotation (2024)

Abstract: In the context of text classification, the financial burden of annotation exercises for creating training data is a critical issue. Active learning techniques, particularly those rooted in uncertainty sampling, offer a cost-effective solution by pinpointing the most instructive samples for manual annotation. Similarly, Large Language Models (LLMs) such as GPT-3.5 provide an alternative for automated annotation but come with concerns regarding their reliability. This study introduces a novel methodology that integrates human annotators and LLMs within an Active Learning framework. We conducted evaluations on three public datasets. IMDB for sentiment analysis, a Fake News dataset for authenticity discernment, and a Movie Genres dataset for multi-label classification.The proposed framework integrates human annotation with the output of LLMs, depending on the model uncertainty levels. This strategy achieves an optimal balance between cost efficiency and classification performance. The empirical results show a substantial decrease in the costs associated with data annotation while either maintaining or improving model accuracy.

## introduction active learning allows machine learning algorithms to choose their learning data selectively. this methodology optimizes learning efficiency while reducing the need for labor-intensive labeled instances, often leading to enhancing the performance metrics with less training @xcite . while active learning has been explored in research for over two decades, it has seen a resurgence of interest in natural language processing (nlp), particularly around 2009-2010. this resurgence has been aligned with adopting neural models in nlp research @xcite . recent trends suggest the advantageous pairing of active learning techniques with deep learning methodologies @xcite . a prevalent strategy within active learning is uncertainty sampling. this technique involves the algorithm selecting instances in which the model might be least certain about their labels. for instance, in binary classification tasks, it might select an instance whose probability of belonging to the positive class is nearest to 0.5. this often employs a 'pool-based' approach where a human expert validates and assigns true labels to the selected samples, which are then used to update the classifier iteratively @xcite . additional approaches like query-by-committee (qbc) @xcite and expected gradient length (egl) @xcite offer alternative techniques in active learning. broadly speaking, active learning aims to reduce the expenses associated with human annotation, achieving this by strategically selecting the most informative data points for labeling @xcite . however, active learning is not the only strategy employed to minimize human annotation costs. llms have been applied to various text annotation tasks such as political twitter message categorization @xcite , relevance and topic detection in tweets @xcite , and hate speech classification @xcite . some recent studies have also examined the potential of utilizing active learning in prompt-based uncertainty sampling with llms @xcite . in this study, we propose a novel pipeline for text classification focusing on three distinct, opensource datasets: imdb for sentiment analysis, a dataset for identifying fake news, and another for classifying movie genres. we introduce a framework that integrates active learning based on uncertainty sampling with human and gpt-3.5 annotations. this integration is tailored to adaptively choose between human and machine annotators based on the uncertainty levels estimated from gpt-3.5's annotation. to the best of our knowledge, this is the first study that comprehensively evaluates the utility and efficiency of combining human annotators, active learning, and gpt-3.5 in a text classification task. we extend the traditional active learning methodologies by integrating uncertainty measurements from llm, such as gpt-3.5, into our annotation selection process. this not only minimizes the costs of manual annotation but also capitalizes on the strengths of machine learning models for efficient and accurate text classification. our approach offers a nuanced trade-off analysis between cost and accuracy, using real-world pricing models for both human and machine annotation. the goal is to provide a robust, cost-effective, and scalable text classification pipeline that leverages the best of both human expertise and advanced machine learning techniques. the code provide in anonymous github. 1 in the subsequent sections, we explore a methodology that combines human expertise with llms in an active learning framework. we detail the experimental setup and methodologies, including uncertainty-based sampling and llms for data annotation. the results are analyzed for their accuracy and efficiency, leading to a discussion on the broader implications and future potential of this approach in text classification. ## related works text classification is an instrumental task in natural language processing (nlp) that uses methods ranging from traditional machine learning techniques to advanced neural networks such as long short-term memory (lstm) (qaisar, 2020) and convolutional neural networks (cnn) @xcite . although these methods primarily focus on improving model accuracy, our study diverges by emphasizing the reduction of labeled data through selective annotation, subsequently enhancing the model performance. we also manage the pool of unlabeled data by removing instances that have been labeled, thus continuously refining our dataset for model training. ## methodology this section details the methodology employed for text classification, emphasizing the integration of large language models in data annotation. the approach encompasses an active learning framework, applying uncertainty-based sampling across varied datasets to enhance accuracy and efficiency. algorithm 1 procedure for collecting sentiment label and confidence require: movie review text x[i] ensure: sentiment label and confidence level in json format 1: prompt: "what is the sentiment of the following movie review, and how confident are you about this 'sentiment'?" 2: instructions: "give your answer as a single word, either 'positive' or 'negative' and a single percentage in json format delimited with space." 3: display: "review text: "' x[i] "'" ## experimental setup in the experimental setup, we explore the impact of using gpt-3.5 for data annotation under various scenarios, including different confidence thresholds and combinations of human and gpt-3.5 annotations. the experiments were conducted across three datasets: imdb, movie genress, and fake news. gpt-3.5 labels only our first experiment investigates the feasibility of employing an llm for data annotation using only the labels provided by gpt-3.5. human labels only as a baseline, our second experiment involves using human annotations exclusively. this experiment serves as a control to measure the effectiveness of gpt-3.5's annotations against traditional human annotation. hybrid labels: confidence threshold exper ## analysis and results the methodology exhibited notable scalability across three distinct datasets, each presenting unique classification challenges: imdb reviews: this dataset involved a binary classification task, determining the sentiment of movie reviews as either positive or negative. fake news: another binary classification task where the variability in text lengths presented an additional layer of complexity. movie genres: this dataset represented a more intricate four-class classification based on movie plots. the consistent application of the methodology across these datasets underscores its adaptability and versatility in handling varying text lengths and classification complexities. confidence thresholds played a crucial role in balancing automated and manual annotations. in the imdb dataset, a 70% threshold was used, where data points above this threshold received one-shot learning, and those below it were processed with few-shot learning. for the movie genres and fake news datasets, an 80% threshold was applied. these thresholds were determined based on the datasets' characteristics, targeting approximately 10 to 15 percent of the imdb and movie genres datasets and about 4 percent of the fake news dataset for few-shot learning. this strategy effectively replicated the essence of active learning, focusing on the most informative or uncertain samples. the cost implications of each setup were a focal point. for instance, the hybrid 80 model for imdb and the hybrid 90 for movie genres demonstrated significant cost-efficiency while achieving comparable accuracies to human-only labels but at a fraction of the cost. this was effectively illustrated through logarithmic scale representations, figure 2 highlighting stark cost disparities and indicating the practicality of the chosen thresholds and annotation strategies. the concept of proxy validation emerged as a crucial aspect of the study. analysis revealed a notable correlation between the f1 score of the proxy validation and the remaining pool, indicating that proxy validation serves as a reliable indicator of the overall pool quality. though some variations between the f1 score of proxy validation and the actual pool were observed, these discrepancies were minimal. detailed examples and a deeper analysis of this phenomenon are provided in the appendix. increasing the size of the proxy validation compared to pool data might lead to even more similar f1 scores between proxy validation and the pool, enhancing the reliability of this method as an indicator. the study also delved into the analysis of gpt-3.5's output confidence scores. in the imdb dataset, 11% of the annotations were found to be incorrect overall. however, in instances where gpt-3.5's confidence was below 70%, the rate of incorrect annotations rose to nearly 50%. similar trends were observed in the movie genres dataset (33% overall inaccuracies, rising to 50% for data under 80% confidence) and the fake news dataset (27% overall inaccuracies, increasing to nearly 50% for data under 80% confidence). these findings suggest that the model's confidence score can be a reliable indicator of uncertainty and the likelihood of annotation errors. these results have significant implications for the application of active learning models, particularly in how confidence scores can be interpreted and used. the observed correlation between lower confidence scores and higher rates of annotation errors supports the idea that gpt-3.5's confidence can be treated similarly to uncertainty measures in traditional active learning models. this insight opens up new avenues for utilizing llms in active learning frameworks, where confidence scores can guide the annotation process more effectively.