Donkii: Characterizing and Detecting Errors in Instruction-Tuning Datasets (2024)

Abstract: Instruction tuning has become an integral part of training pipelines for Large Language Models (LLMs) and has been shown to yield strong performance gains. In an orthogonal line of research, Annotation Error Detection (AED) has emerged as a tool for detecting quality problems in gold standard labels. So far, however, the application of AED methods has been limited to classification tasks. It is an open question how well AED methods generalize to language generation settings, which are becoming more widespread via LLMs. In this paper, we present a first and novel benchmark for AED on instruction tuning data: Donkii.It comprises three instruction-tuning datasets enriched with error annotations by experts and semi-automatic methods. We also provide a novel taxonomy of error types for instruction-tuning data.We find that all three datasets contain clear errors, which sometimes propagate directly into instruction-tuned LLMs. We propose four AED baselines for the generative setting and evaluate them extensively on the newly introduced dataset. Our results show that the choice of the right AED method and model size is indeed crucial and derive practical recommendations for how to use AED methods to clean instruction-tuning data.

## introduction recent successes in instruction tuning (instt) have shown that large language models (llms) can generalize to a wide range of tasks in the zeroshot setting @xcite @xcite . instt achieves this by training an llm on instruction-output pairs, where the instruction describes the task and the output contains the expected solution to the task. after fine-tuning on the instt dataset and an optional reinforcement learning phase @xcite , llms are able to generalize to instructions not * now at toloka.ai seen during fine-tuning. in an orthogonal line of inquiry, researchers have studied annotation error detection (aed), which allows to detect erroneous annotations in labelled datasets. these low quality instances are then corrected or removed in a semi-automated process @xcite @xcite . however, how to best apply aed for natural language generation has so far not been studied. in this work, we combine both strands of research and ask whether aed methods can help to detect errors in instt datasets and thus help to improve model quality by improving data quality. applying aed methods to instt datasets presents a number of challenges. (1) the systematic development and comparison of aed methods requires datasets with annotations indicating which instances contain annotation errors. such datatsets are not yet available for instt. (2) to our knowledge, researchers have applied aed methods only in the discriminative setting @xcite and it is not immediately clear how existing methods can be adapted to generative problems. (3) it is not obvious what even constitutes an error in instt. in this work, we address these three challenges; see also figure 1 for an illustration of our contributions: foot_0 (1) we present donkii, the first instruction tuning benchmark to enable the evaluation of aed methods. donkii contains error annotations on top of three existing instt datasets derived from manual error annotation efforts. we also introduce a hierarchy of error types for instt datasets; see figure 2. (2) we derive four aed baselines for generative problems based on recent work on training dynamics for aed @xcite . (3) we use donkii to evaluate the proposed aed baselines and study the effects of model size, different types of errors, and different types of instt data. the results show that there is a clear best-performing aed method for instt data among the four evaluated. ## proposed aed baselines for text generation datasets we present four aed baselines for text generation datasets. for this, we adapt methods based on training dynamics that were previously used for aed in classification problems @xcite . we chose these methods because they performed well in earlier work @xcite and because their adaptation to generative settings is relatively straight-forward. all four methods assign an error score to an instance, with a higher score ideally reflecting a higher probability of an incorrect annotation. all scores use the probabilities p e,l that the model assigned to the token l of the instance's output sequence at epoch e during training. we propose the following measures: (1) perplexity, which is the epoch-averaged perplexity of the instance based on p e,l : ## datasets and error types we describe donkii's three different data sources that we have enriched with annotations of erroneous instances: p3-donkii, sni-donkii, and adc-donkii. 2 each is based on an existing in-stt dataset and on manual inspection of the errors in that dataset: p3-donkii is derived from the meta-dataset public pool of prompts @xcite , sni-donkii from the meta-dataset super-natural instructions @xcite , and adc-donkii from the llm-authored dataset alpaca @xcite and its partially corrected version alpacadatasetcleaned (ruebsamen and contributors, 2023). we enrich each of these datasets with labels indicating which instances contain errors, using different mixtures of expert annotation and programmatic analysis of the source data. for each dataset, we construct three different sets of instances: x * which contains no errors, x err , for which we know that it contains errors, and x unk , for which we do not know if it contains errors. we evaluate aed methods by their ability of discriminating x * from x err . we exclude x unk from evaluation, as we do not exhaustively annotate the datasets with errors due to their sheer size and resource availability; see §5.1 for details. ## how well does aed do in instruction tuning data? in this section, we provide baseline results for error detection in instt datasets. to do this, we evaluate the baselines introduced in §3 using the donkii datasets proposed in §4. ## conclusion this work presents the first study on annotation error detection for generation tasks, in particular, instruction tuning data. despite the popularity of inst, there are no evaluation datasets for aed with marked errors. therefore, we present donkii, a suite of three existing instt datasets enriched with novel error annotations and an error taxonomy derived from manual annotation efforts. we propose four different aed methods for generative models and systematically evaluate them on the donkii datasets. we find that there is a clear best performing method for single instances with p µ and for task-level aed with ppl. in any case, the choice of model size is critical for optimal aed performance. in appendix f we report on preliminary experiments in which we investigated how annotation errors impact downstream performance. for future work, we plan to apply aed methods to more structured generative meta-datasets such as huguet cabot and navigli (2021) or fries et al. @xcite .