Can Active Label Correction Improve LLM -based Modular AI Systems? (2024)

Abstract: Modular AI systems can be developed using LLM-prompts-based modules to minimize deployment time even for complex tasks. However, these systems do not always perform well and improving them using the data traces collected from a deployment remains an open challenge. The data traces contain LLM inputs and outputs, but the annotations from LLMs are noisy. We hypothesize that Active Label Correction (ALC) can be use on the collected data to train smaller task-specific improved models that can replace LLM-based modules. In this paper, we study the noise in three GPT-3.5-annotated datasets and their denoising with human feedback. We also propose a novel method ALC3 that iteratively applies three updates to the training dataset: auto-correction, correction using human feedback and filtering. Our results show that ALC3 can lead to oracle performance with feedback on 17-24% fewer examples than the number of noisy examples in the dataset across three different NLP tasks.

## introduction large language models (llms) such as openai's chatgpt (openai, 2022), google's bard based on lamda @xcite , and meta's llama @xcite are powerful zero or few-shot learners as they can generalize to a wide range of tasks simply through task explanations or demonstrations without any model fine-tuning @xcite . this led to the development of many modular ai systems that leverage these llms through api services to decompose complex tasks and to automate workflows @xcite @xcite . these ai systems rely on several discriminative and generative sub-tasks performed by llms, without any fine-tuning, to achieve the desired behavior. llm-based modular ai systems work reasonably well in practice, but their robustness is still in question because (i) they are limited by the quality of zero-shot learning of the llms that they employ @xcite , (ii) an error in one module can trigger a cascade effect, propagating the error throughout the modular system, (iii) ai developers cannot rely on future improvements in the quality of underlying llms as the performance on certain end-tasks can also worsen @xcite due to the fine-tuning focused on safety and conversationality of @xcite domain shift between llm training data and deployed environment. further, llms cannot be easily finetuned for all sub-tasks in every ai system because (i) there is a lack of training data, and (ii) the computational cost required to train and maintain an online service for multiple fine-tuned models can be expensive and has a large carbon footprint. to address these challenges, we propose collecting data traces containing llm inputs and outputs from deployed ai systems, and training smaller task-specific models that can replace llm calls (see figure 1 ). we propose a method based on active label correction (alc) @xcite to use human feedback to improve the quality of data obtained from modular ai systems and eval-uate its efficacy for three publicly-available nlp datasets. the data traces contain examples with low-quality llm annotations, but alc can utilize human expertise in an efficient manner by seeking annotations for examples that are most likely misannotated. we show that synthetic noise models employed by previous works are not adequate to capture noisy llm annotations which makes it challenging to predict misannotated examples in alc. the proposed method alc3 (i) autocorrects potentially misannotated examples using model predictions, (ii) filters out the confusing examples in addition to (iii) using human feedback. perfect misannotation prediction on a dataset with n % noisy examples will require human annotation on n % of data to clean it. our results show that the three-step process of alc3 can lead to oracle test performance with feedback on 17-24% fewer examples than n % (perfect predictor) for three nlp tasks with different data sizes (5k-100k) and complexity. the proposed method promotes the idea of leveraging the powerful zero-shot learning capabilities of llms for rapidly deploying modular ai systems and improving their quality as they accumulate data from the real world. we focus on discriminative tasks but the proposed method also extends to generative tasks in principle (see §a). we make three main contributions: (i) we propose an alc-based pipeline and our method alc3 to continually improve llm-based modular ai systems while minimizing annotation costs, (ii) we compare different noise models with llmannotation noise and the performance-cost curves of different alc methods for gpt-3.5-annotated datasets, and (iii) we present experiments for three publicly-available nlp datasets with tasks that are common to modular ai systems including natural language inference, named entity recognition, and intent classification. semi-supervised, weakly-supervised and active learning: in semi-supervised learning, training data consists of a set of labeled examples and a typically much larger set of unlabeled examples @xcite . weakly supervised learning more generally assumes that annotations are partial, noisy, or provided at a higher level of abstraction @xcite . self-training is a common method used to train a teacher model on the supervised subset to predict soft pseudo-labels for the unsupervised subset. then, the supervised subset and pseudo-labeled subset are used to train a student model @xcite @xcite . active learning strategy selects the most informative examples, typically by construction or by choosing from a pool of unlabeled examples, and querying an oracle such as human annotator for supervision @xcite . our problem is different from above but lies at the intersection since we begin with a fully-annotated noisy dataset and need to seek human expertise for improving data and model quality. active label correction (alc): alc (rebbapragada et al., 2012) uses self-training to clean a noisy training set by iteratively predicting most likely mislabeled examples using a trained model and receiving human feedback to correct them. simple disagreement measures based on the output probabilities work well for identifying mislabeled examples. robust alc @xcite assumes that label-conditioned noise and uses a latent model for the true label. dual alc or dalc @xcite additionally uses model predictions with high confidence for automatic correction. we note that previous works have primarily focused on image classification problems in vision and experimented with synthetic noise. we study and explore the use of alc for correcting real noise observed in gpt-3.5-annotated datasets. human-ai collaborative annotation: coannotating @xcite is a data annotation process that uses variations in llm prompts to estimate model uncertainty for deciding allocation to annotators. in our work, we boost data quality towards the goal of continual model improvement to inputs, noisy labels model predictions trained model autocorrection human annotation filtering x y y y y y ai system output input prompt & llm call inference update y retain human annotations from previous iterations y y y y iterations 1 2 3 4 5 6 7 8 9 iteration 1: 2, 3, 4, 5 iteration 2: 6, 7, 8, 9 replace llm-based modules in ai systems. as a result, we use an iterative process where human feedback helps increase model and misannotation prediction quality i.e. better prediction of examples most likely to be misannotated, along with auto-correction and filtering. coannotating is noniterative and does not leverage fine-tuned models or previously annotated human examples, but relies on numerous llms calls for each example. machine teaching @xcite ) is another framework for human-ai collaborative annotation where annotators work with an ml model and knowledge base to teach concepts to machines in a selective labeling process. in our proposed process, llms can be interpreted as a source of knowledge as they provide initial annotations, but the presence of annotations transforms the problem into one of label correction. ## methods modular ai systems rely on the zero/few-shot learning power of llms to process the input, perform sub-tasks, and generate output. these ai systems can accumulate data traces that include inputs, intermediate outputs of each module, and final outputs through user interactions. to evaluate a module that relies on llms as a zero-shot learner, one can analyze the amassed inputs and outputs. we propose collecting these input-output pairs for each module to create noisy annotated datasets, and improving data quality using human feedback with the goal of training a replacement model. the datasets above are assumed to be noisy because llm annotations have lower quality than human annotations. alc @xcite iteratively predicts and flag the examples that are most likely to be noisy or misannotated, and seeks correction from human annotators on the flagged examples. the iterative approach aims to jointly (i) minimize noise in training examples which is also a proxy for test error, and (ii) minimize human feedback by effectively predicting misannotated examples. our proposed method alc3, outlined in figure 2 , uses two additional steps beyond the basic alc. we first auto-correct the examples where model prediction contradicts the label with very high confidence, leveraging the trained model for labelling, similar to dual alc @xcite . after human annotation, we filter out examples with low label probabilities to reduce the noise in the training dataset. in each iteration, we retain the human annotations, but reset and obtain a new set of autocorrected and filtered examples. we describe each step in more details in the following subsections. ## conclusion modular ai systems can leverage llms as zero/few-shot learners for rapid deployment without any training or fine-tuning but there is a large performance gap when compared to fine-tuned models. the proposed pipeline utilizes our method alc3 to improve modular ai systems by leveraging llm-annotated data and improving its quality to train a replacement model for each module. compared to random or label-conditioned synthetic noise models used in previous work, we found the noise characteristics of gpt-3.5-annotated text data to be far more complex. with alc3, we achieve the same performance as ground truth data using human feedback on a fraction of the dataset. we found this fraction to be 17-24% less than the fraction of misannotated data for three unique tasks spanning different levels of complexity and data sizes, and ≈ 16% better than previous methods.