Improving Model Evaluation using SMART Filtering of Benchmark Datasets (2025)

Abstract: One of the most challenging problems facing NLP today is evaluation. Some of the most pressing issues pertain to benchmark saturation, data contamination, and diversity in the quality of test examples. To address these concerns, we propose Selection Methodology for Accurate, Reduced, and Targeted (SMART) filtering, a novel approach to select a high-quality subset of examples from existing benchmark datasets by systematically removing less informative and lower quality examples. Our approach applies three filtering criteria, removing (i) easy examples, (ii) data-contaminated examples, and (iii) examples that are similar to each other based on distance in an embedding space. We demonstrate the effectiveness of SMART Filtering on three multiple choice QA datasets, where our methodology increases efficiency by reducing dataset size by 48% on average, while increasing Pearson correlation with rankings from ChatBot Arena, a more open-ended human evaluation setting. Our method enables us to be more efficient, whether we are using SMART Filtering to make new benchmarks more challenging, or to revitalize older, human generated datasets, while still preserving the relative model rankings.

## introduction with the recent rise of interest in natural language generation and language modeling as a general purpose pretraining task, model benchmarkingevaluating several models on a standard test set to derive a strict ranking-has become much less straightforward. this is in part due to the increasing speed of evaluation dataset saturation @xcite @xcite , whereby current state-of-the-art models are able to match human-level performance within the margin of error @xcite . figure 1 : this figure illustrates our methodology. we remove easy, data contaminated, and/or similar examples from datasets to find a high-quality subset. the examples selected are from mmlu dataset. this signals the need for new, potentially more challenging benchmarks; however, generating high quality human-annotated datasets is difficult @xcite . it is time-consuming to devise new evaluation datasets, which can require training and coordinating human annotators, cleaning data, and assessing test validity @xcite . moreover, the increasing speed of benchmark saturation leads to a difficult value proposition for benchmark creators who have only so much time to make important research contributions. as if the speed of progress in nlp benchmark saturation weren't enough of a challenge, the increasingly wide availability of text-generation systems has made the classical approach to collecting high quality, human-baselined evaluation datacrowd-sourcing-much less tenable. since detecting model-generated text is an open area of research @xcite , benchmark developers can no longer trust that a new, purportedly human-created evaluation dataset was indeed created by humans, and not humans using automatic tools as assistants. this state of affairs raises an important question about past evaluation datasets: if they weren't saturated, could they still help us rank models based on task performance? can we recapture some of the utility in saturated evaluation datasets that appear to be saturated? one common approach to dataset saturation is filtering @xcite . just as renovating an older home is generally more efficient than building a new home from scratch, dataset filtering can rejuvenate a high-quality older benchmark, without the need to rely on novel rounds of costly human data collection. along these lines, we propose selection methodology for accurate, reduced and targeted (smart) filtering, a novel approach for finding a high-quality subset of evaluation benchmarks that doesn't need human intervention, illustrated in figure 1. we achieve this by algorithmically removing the low-quality examples from existing evaluation datasets, preserving the high-quality examples that are most informative and reliable as our basis for benchmarking. our methodology applies three filtering criteria: 1) removing easy examples, 2) removing data-contaminated examples that are highly likely to have been leaked into the training datasets, and 3) removing similar examples. all smart filtering steps have their conceptual origin in dataset filtering, which can reduce the data size and hence compute costs @xcite . in particular, we utilize filtering methods used for training-time deduplication @xcite @xcite , a technique often used during pretraining to increase efficiency. we contend that pretraining deduplication approaches will also be effective for benchmarking filtering. they can increase headroom on standard, already vetted, human-written datasets that have been saturated, and re-enable meaningful model rankings. dataset filtering, if correctly applied (c.f., @xcite , can retain or even improve benchmark dataset quality @xcite , meaning that smart filtering might be able to contribute to addressing reliability issues arising from current day benchmarking practices @xcite @xcite . each of the steps in smart filtering aims to improve usefulness and reliability by removing examples that are not informative for differentiating models. smart filtering could also be applied to new datasets at the earlier stages of the benchmark lifecycle before they become standards. when used before new benchmarks premiere, smart filtering can ensure that the final version is as challenging, high quality, and efficient as possible. we test smart filtering on the classical multiple choice qa setting. we characterize the impact of smart filtering using three datasets-arc @xcite , mmlu @xcite and commonsenseqa @xcite . in tables 1 and 2 , we show that our methodology increases efficiency without hits to leaderboard utility. for example, on the arc dataset, we shrink the dataset by 68.9%, increasing computational efficiency drastically, while observing practically no effect on the relative ranking of models. we also compare model rankings of a dataset with openended human evaluated chatbot arena @xcite rankings, and found that datasets using smart filtering have a stronger correlation with chatbot arena indicating better correlation with human preferences. ## related works recently, many benchmark datasets such as mmlu @xcite , gsm8k @xcite , math @xcite and gpqa @xcite have been proposed to measure the capabilities of language models (lms). however, recent works have highlighted issues that one proposal suggests modifying the self-attention matrix to prevent answer options from paying attention to each other by masking their scores to zero @xcite , but nonetheless, benchmark validity and gameability remains an open area of ongoing research. further complicating evaluation, some models have a prior bias towards a particular option id (e.g. 'a') @xcite @xcite @xcite . additionally, models can perform well above random chance even when question text is removed and only answer options are given, potentially highlighting possible data contamination issues @xcite . recent works have shown that replacing the correct option with "none of the above" leads to a drastic decline in performance across all models @xcite . this calls into question the reliability of datasets and leaderboard ranking as the accuracy of a model may not be representative of its capabilities @xcite . under the assumption that leaked, gameable, spuriously correlated, and/or biased examples are often easier for models to do well on, we aim to help address such issues by filtering "easier" examples from benchmarks to increase reliability. other works have explored related approaches to improve evaluation @xcite . @xcite tested models only on difficult examples of a dataset, showing that evaluating on just 5% examples can achieve a very high correlation with the full dataset. however, their approach requires retraining the models on various subsets of training data and uses humans to verify the filtered dataset. in the current nlp paradigm, access to training data is often unavailable for most models @xcite and even when accessible, it is very expensive to retrain models on those datasets. contrasting with @xcite , our approach emphasizes computational efficiency and identifies a high-quality subset that correlates well with the original dataset, without requiring access to training data or expensive human verification. ## smart filtering methodology our methodology has three main filtering steps, each of which is applied independently; the order of filtering steps does not impact the final subset. all three steps in smart filtering are aimed at deduplicating and making nlp leaderboarding more efficient. we contend that examples in the dataset that all tested models get correct with very high confidence are not useful for establishing model ranking (although they might be useful for data exploration or other things). therefore, including such "easy" examples merely increases the computation cost without providing ability to distinguish between models in leaderboard settings @xcite . relatedly, examples that are present in pretraining datasets should not be used for model testing @xcite . leaked examples can give some models unwanted and difficult-to-interpret advantages when they are used for a leaderboard @xcite . finally, examples that are overly similar to one another effectively contribute less information about model performance than examples that meaningfully differ. example overlap can also contribute towards idiosyncratically favoring particular kinds of information and potentially even double counting. in what follows, we will describe each step in the smart filtering pipeline, to enable others to apply our methodology to benchmarks from any nlp task. ## results: efficiency & informativeness in this section, we present the results of applying smart filtering on popular datasets, focusing on three well-established multiple-choice question answering datasets: arc @xcite , mmlu @xcite and commonsenseqa @xcite . while we show results on mcqa datasets, our methodology is generic and can be applied to various dataset types. given that all other classification tasks can be, and often are, recast into mcqa format, it strikes us as likely that smart-filtering would work just as well for classical nlu tasks (like nli, coreference resolution, sentiment analysis) without needing novel strategies. although these tasks have conceptual differences, with respect to the format, these variations boil down to having different numbers of answer options. going into specifics of why we selected these datasets, we selected mmlu and arc datasets because of their widespread adoption in benchmarking by state-of-the-art models @xcite and their coverage of diverse topics and difficulties. these two datasets are 4-choice mcqa, so to broaden the scope, we included commonsenseqa, which offers 5-choice mcqa. we think our inclusion of different numbers of mcqa answer options between 4 and 5 gives some indication that the methodology will be adaptable to smaller numbers of answer options (2 or 3) too. we release our code and provide smart filtering version of each dataset foot_0 . to identify easy and data contaminated examples, we evaluate 7 open source models from the top of the open llm leaderboard (hugging-face, [2024]) on each qa dataset: llama 3.1-70b (meta ai, 2024), phi-3-14b (abdin et al., 2024), yi-34b (01.ai et al., 2024), qwen2-72b (yang et al., 2024), mixtral-8x22b (mistral ai, 2023), gemma2-27b (google, 2024), and dbrx-132b @xcite . we use instruction tuned ('instruct') versions when available, as they should have better question-answering capabilities. moreover, as these models came from different organizations, they were likely trained on different datasets. ideally, this minimizes the risk of bias towards any particular model when identifying easy and contaminated examples (see section 5.1 for more discussion on this). we used open-source models, as we aim to develop a methodology that is accessible to any ai practitioner with sufficient computation resources, thereby avoiding the substantial closed-source api costs. the effectiveness of smart filtering is shown in the impact of smart filtering varies significantly across datasets. in arc, 64.4% of the examples were classified to be 'easy', compared to only 27.6% in commonsenseqa. in mmlu, a significant 4.37% of the dataset was found to be data contaminated, while commonsenseqa had none. additionally, 12.57% of arc and 4.56% of mmlu examples were identified as similar. these findings highlight that different datasets might suffer from different types of low-quality examples. for mmlu, a widely accepted dataset for measuring llm performance, we were able to filter out 43% of the dataset while maintaining almost the same model ranking. this suggests that mmlu might not be saturated yet, and there is still room for further hill-climbing for current models. while relative model ranking is maintained, smart filtering leads to a notable decrease in inter-model agreement across all datasets, as mea-sured by the percentage of samples yielding identical predictions among models (figure 4 ). this reduction in model agreement is beneficial, as it enables better differentiation among models based on their capabilities, and provides more headroom for improvement by decreasing their correlation. in the following subsections, we present category-specific results for select datasets for detailed analysis, measure the correlation in model rankings before and after smart filtering, and compare model accuracies with chatbot arena scores @xcite . ## robustness of our approach in this section, we show that smart filtering is robust to altering settings, such as (i) the number of models used for identifying easy and datacontaminated examples and (ii) choice of embedding model used to find similar examples. ## conclusion in this work, we proposed smart filtering, a methodology for identifying a challenging and high-quality subset of any benchmark, existing or new, that can better capture the capabilities of models and rank them. to achieve this we remove easy, data contaminated, and similar examples from a dataset. smart filtering achieved significant increases in computational efficiency and better correlation with human preference data than the original datasets. we anticipate our approach will be useful for improving current benchmarking practices as well as for dataset creators to find high-quality subsets before dataset release in the future.