Identifying Noise in Human-Created Datasets using Training Dynamics from Generative Models (2025)

Abstract: Instruction fine-tuning enhances the alignment of autoregressive language models (ArLMs) with human intent but relies on large-scale annotated datasets prone to label and text noise. In this paper, we show that existing noise detection techniques designed for autoencoder models (AeLMs) do not directly generalize to ArLMs due to differences in learning dynamics. We propose TDRanker, a novel approach leveraging training dynamics to rank datapoints from easy-to-learn to hard-to-learn, effectively identifying noisy instances. Our method demonstrates robustness across multiple model architectures covering both autoencoder and autoregressive language models (GPT-2, BERT, LaMini-Cerebras-256M) and across various dataset noise levels, achieving at least 2x faster denoising than previous techniques. Applied to real-world classification and generative tasks, TDRanker significantly improves data quality and model performance. These findings suggest that TDRanker provides a scalable solution for refining instruction-tuning datasets, enhancing the reliability of fine-tuned ArLMs in practical applications.

## introduction autoregressive language models (arlms), also known as generative models, have recently achieved significant progress across various natural language tasks, including understanding, mathematical reasoning, and coding @xcite @xcite @xcite . these arlms, trained with a causal language modeling objective to predict the next token in a sequence, excel at generating coherent text but often lack alignment with human preferences @xcite . to address this, 'instruction fine-tuning' adapts base models to better meet user needs, creating 'instruction models' @xcite . however, instruction fine-tuning is resourceintensive, requiring substantial labeled data-e.g., 10m annotated examples for llama 3 instruct model @xcite . this makes data collection, annotation, and training both costly and time-consuming. moreover, the complexity of the annotation process often introduces various types of noise into the labeled datasets. such noise can originate from human errors during labeling (label noise) or inherent issues in the datapoints themselves, such as grammatical mistakes or poorly constructed sentences that can even lead to a change in the intended meaning of the sentence (text noise). figure 1 illustrates the various types of noise commonly found in instruction tuning datasets, including errors introduced by annotators, inconsistencies in labeling, and issues within the datapoints. while deep learning models are generally considered robust to a certain degree of label noise @xcite , studies on autoencoder language models (aelms), which include discriminative models like bert, have revealed that these models face significant challenges when fine-tuned on datasets with noisy labels @xcite and are susceptible to overfitting to label noise present in the dataset @xcite leading to significant performance loss on downstream tasks. this made it essential to either (1) develop strategies for fine-tuning aelms with such noisy instances @xcite @xcite , or (2) develop techniques to filter noisy instruction instances @xcite . unlike aelms, generative models (arlms) are pretrained on substantially larger datasets-often involving approximately 100 times more train- last_turn_question label how much does it cost to visit? non-standalone what should i do if i smell burning insulation? non-standalone what time will it take to get a reply on an appeal? standalone what is the court trial going to be held? non-standalone … question: <@xmath0, otherwise output 1. instruction template question examples figure 1 : an example of the noisy training dataset, containing both the text noise and the label noise. the phrase, "what time will it take...", is incorrect for expressing duration; "how long will it take..." is appropriate, and denotes text noise. the phrase, "what should i do...", can be answered without previous context, and denotes label noise. last_turn_question denotes the current user utterance in multi-turn dialogue with an ai assistant. the remainder of the paper is structured as follows: ( §2) presents the literature review, followed by a detailed description of tdranker ( §3) and the experimental setup ( §4). we conclude with an in-depth analysis and discussion of the results ( §5). ## related works identifying and mitigating label noise in datasets is crucial for improving model generalization. previous research has explored various approaches to detecting and handling noisy labels in both aelms and arlms. we categorize related work into two key areas: handling label noise in autoencoderbased models (discriminative) and addressing label noise in autoregressive models (generative). ## methodology in this section, we describe our method, tdranker, in detail. to identify noisy data in a dataset, the process involves two main steps. first, an arlm like gpt-2 is trained on noisy data. during training, the model's behavior is tracked for each data point, including its predictions (adjusted to match the expected labels), whether the predictions are correct, and its confidence in the predictions (see §3.2). this information helps determine how easy/hard the model learns each data point. second, the data points are ranked based on their learning difficulty, from easiest to hardest (see §3.3). experiments show that data points that are harder to learn often have problems like incorrect labels or poor-quality text. ## experiments this section describes experimental settings. ## results in this section, we evaluate different aspects of our proposed method: the impact of ranking functions on data selection ( §5.1), the impact of different model choices ( §5.2), the effectiveness of iterative application for dataset denoising ( §5.4), the impact of different noise levels ( §5.3), and the applicability of our method to real-world noisy datasets ( §5.5). ## conclusion in this paper, we propose an approach for identifying subsets of data containing a high percentage of noise, including label noise and text noise. training dynamics are captured during the fine-tuning stage and are used to rank datapoints from easy-tolearn and hard-to-learn, where hard-to-learn datapoints often contain a higher percentage of noise. our experiments show that this approach can be used regardless of whether the model is an aelm (bert) and arlm (gpt-2, lamini) architecture. our approach identified subsets of noise in realworld datasets, with the noisier subsets containing more fatal noise that cascades into other attributes of the data.