Contrastive Error Attribution for Finetuned Language Models (2023)

Abstract: Recent work has identified noisy and misannotated data as a core cause of hallucinations and unfaithful outputs in Natural Language Generation (NLG) tasks. Consequently, identifying and removing these examples is a key open challenge in creating reliable NLG systems. In this work, we introduce a framework to identify and remove low-quality training instances that lead to undesirable outputs, such as faithfulness errors in text summarization. We show that existing approaches for error tracing, such as gradient-based influence measures, do not perform reliably for detecting faithfulness errors in NLG datasets. We overcome the drawbacks of existing error tracing methods through a new, contrast-based estimate that compares undesired generations to human-corrected outputs. Our proposed method can achieve a mean average precision of 0.93 at detecting known data errors across synthetic tasks with known ground truth, substantially outperforming existing approaches. Using this approach and re-training models on cleaned data leads to a 70% reduction in entity hallucinations on the NYT dataset and a 55% reduction in semantic errors on the E2E dataset.

## introduction recent analyses of natural language generation systems have identified that data errors are a key cause of failures ranging from unfaithfulness @xcite to bias @xcite . while better data collection procedures @xcite and noise-robust training methods @xcite can help address some of these problems, neither of these approaches serves as a complete solution. the large-scale datasets needed to train modern neural methods will inevitably contain at least a few annotation mistakes in these datasets, and some of these will affect even the most robust model training procedures. data cleaning methods provide an alternative approach, where data errors are identified by tracing model errors back to the training dataset. this post-hoc approach allows practitioners to enforce desired properties such as faithfulness by repeatedly identifying and removing rare data errors that cause undesired behavior. existing work from the machine learning literature has proposed measuring the "influence" of training examples on generated outputs as a way to trace such errors @xcite @xcite @xcite . however, these influence-based approaches are often brittle, and we find that they fail in complex, real-world tasks such as text summarization or data-to-text generation. in a synthetic evaluation inspired by prior work in the memorization literature @xcite , we inject targeted hallucinations in the training data and evaluate error tracing methods on how well they identify these errors and reduce downstream hallucination. we show that existing gradient-based and embeddingbased influence estimation methods cannot reliably identify the inserted hallucinations and even perform worse than a standard retrieval-based baseline (bm25) @xcite . to address this, we develop a method called contrastive error attribution (cea), which combines three new techniques for error tracing: we develop a new contrast-based error tracing method that identifies training examples that cause the model to assign higher probabilities to undesired model outputs than human post-edited versions of the output; we distill these contrast-based scores into a neural net classifier to learn a generalizable model of data errors, and we replace standard gradient dotproduct approximations for influence with more exact loss difference estimates. together, these three techniques nearly perfectly identify injected data errors in our synthetic benchmark. 1 our approach performs well beyond synthetic benchmarks, and we find that error tracing can be used to substantially reduce errors when training neural systems on real generation tasks. we find that our approach reduces entity hallucinations by 70% on the new york times news summarization dataset, and substantially outperforms our strongest baseline, which only manages to reduce 20% of the hallucinations. similarly, our approach can reduce semantic errors @xcite on the e2e dataset by 55% compared to 16% for the strongest baseline. ## problem statement error tracing we define the general error tracing problem as the task of identifying a set of error examples u in a training set d train such that a learning algorithm a produces a model f that behaves correctly on a set of examples d err := {(x i , y i )} m @xmath0. more formally, the error tracing problem is defined by three components ## proposed method we propose and develop three ideas that address the shortcomings of influence-based error tracing. first, we replace the similarity function s with a contrast function that identifies training examples that are responsible for making the incorrect generation ŷ more likely, and the correct generation y less likely. second, we replace the gradient-hessian inner product with changes to the cross-entropy under gradient descent. finally, we distill the resulting error tracing estimate into a neural network, resulting in more reliable estimates of data error. we name our approach contrastive error attribution (cea), and describe each of the components below. ## experimental setup we carefully compare our proposed error tracing method (cae) to existing baselines on both synthetic and real summarization tasks. ## conclusion we explore whether error attribution can be used to produce cleaner datasets that lead to fewer errors in model generation. prior approaches to data cleaning, such as gradient-based influence measures, do not work well for generation tasks. we propose a novel contrastive error attribution approach that addresses the shortcomings that make existing gradient-based approximation methods unreliable in text generation settings. we benchmark our method on a synthetic dataset, as well as two real-world generation tasks. we find that our approach dramatically outperforms existing error attribution approaches on all benchmarks, and leads to substantial reduction in generation error using only a few examples.