Is your benchmark truly adversarial? A dv S core: Evaluating Human-Grounded Adversarialness (2025)

Abstract: Adversarial datasets should validate AI robustness by providing samples on which humans perform well, but models do not. However, as models evolve, datasets can become obsolete. Measuring whether a dataset remains adversarial is hindered by the lack of a standardized metric for measuring adversarialness. We propose ADVSCORE, a human-grounded evaluation metric that assesses a dataset’s adversarialness by capturing models’ and humans’ varying abilities, while also identifying poor examples. We then use ADVSCORE to motivate a new dataset creation pipeline for realistic and high-quality adversarial samples, enabling us to collect an adversarial question answering (QA) dataset, ADVQA. We apply ADVSCORE using 9,347 human responses and ten language models’ predictions to track model improvement over five years (2020–2024). ADVSCORE thus provides guidance for achieving robustness comparable with human capabilities. Furthermore, it helps determine to what extent adversarial datasets continue to pose challenges, ensuring that, rather than reflecting outdated or overly artificial difficulties, they effectively test model capabilities.

## introduction: evaluating adversarial datasets requires human answers as language models attain near-perfect performance on existing benchmarks, there is an increasing demand for unexpected and challenging tasks to evaluate them. adversarial datasets contain examples that cause models to generate harmful @xcite , unsafe @xcite , or incorrect @xcite responses. an ideal adversarial example should be much easier for a human to answer correctly than for a model on realistic tasks @xcite @xcite . however, as models improve, these adversarial datasets can become outdated @xcite -what was hard for a model in 2020 can become trivial in five years-requiring periodic updates @xcite . on the other hand, it is difficult to recognize at what point have these adversarial datasets outlived their usefulness systematically, nor is there an established metric to measure which datasets best captures the gap between human and model ability. to fill this gap, we formulate advscore ( § 3). this metric measures two critical aspects: (i) adversarialness, which captures the performance gap between models and humans, while penalizing "illposed" examples (i.e., ambiguity), and (ii) discriminability-how effectively can a dataset rank models by their abilities. measuring whether a dataset is truly adversarial requires human answers; thus, advscore builds on item response theory @xcite , a framework widely used in psychometrics and educational testing. it captures the diversity of human and model abilities and identifies poor examples ( § 2). advscore is the first metric that evaluates an example's "adversarialness" grounded in human abilities: it can measure whether the dataset's adversarial challenge becomes weaker or stronger as language models improve. we apply advscore to motivate authors to contribute to a new human-in-the-loop hitl benchmark of adversarial questions, advqa. advqa's creation pipeline (figure 1 ) produces high-quality and realistic questions that are adversarial. moreover, advscore helps make advqa discriminative, ensuring that the captured adversarialness reflects the varying skills of humans and models. advqa exhibits the least decline in adversarialness over recent years compared to other adversarial benchmarks ( § 4). this minimal, but meaningful decline in advqa reveals that current models (e.g., gpt4) continue to struggle with tasks requiring commonsense reasoning and multistep reason-language models @xcite v.s. @xcite "what cardinal direction on a compass cannot be found in the name of a us state?" language models language models lms lms lms lms lms lms lms lms tactic: "multi-step reasoning" "not easy for humans", "ambiguous" advscore (0.250) ü margin (𝜇): 0.142 ü disc. (𝜅): 0.760 ü ambig. (𝛿): 0.097 advscore (-0.331) ü margin (𝜇): -0.192 ü disc. (𝜅): 0.98 ü ambig. (𝛿): 0.147 figure 1 : advscore diagnoses when a question is adversarial (top) and difficult for computers to answer for other reasons (bottom). after collecting candidate questions, we ask humans and computers to answer the questions. the top question (from advqa) has a higher advscore because it is specific, adversarial, discriminative, high-quality, and realistic. in contrast, the bottom question is ambiguous (e.g., none of humans or models correctly answered due to its ambiguity), which is confirmed by its low advscore. ## preliminaries of advscore: irt prior metrics for evaluating adversarial question generation strategies, such as attack success rate @xcite , distributional similarity @xcite , and proximity measurement @xcite assess algorithmic adversarialness without human validation. in contrast, we identify adversarial examples that pose realistic challenges aligned with human skills, not just pathological cases that break models. this requires evaluating how well the examples align with varying levels of human performance, particularly where models fall short, while ensuring that the examples are unambiguous. to capture this, we adopt item response theory (irt), which models the interactions between subjects' skills-in the qa setting, the subject answering the question could be either a human or a model-and example difficulty. this framework, widely used in psychometrics and educational testing @xcite , provides insights beyond accuracy: it can diagnose question quality as well as skilled subjects. the final latent variable is the question discriminability γ j which models how sensitive this probability is to changes in skill gap. 2 this encodes how strongly the question rewards the skill being higher or lower than the difficulty level. the objective of irt is to estimate the parameters that maximize the correctness probability p(r ij ). 3 advantages of irt over question success rate while question success rate (qsr)-the percentage of subjects answering a question correctly-may seem like a reliable measure of difficulty, it can be misleading. a good yet difficult question and an easy yet poorly written question could yield the same qsr, obscuring the true measure of difficulty. in contrast, irt evaluates subject responses. not only does irt consider the number of humans who answer a question correctly, but it also accounts for 2 perfect discriminability means that any subjects with a positive skill gap will answer the question correctly @xcite but negative skill gap will never answer the question correctly. ## advscore this section introduces advscore, a metric that evaluates how adversarial and discriminative a dataset is. we measure these two key criteria: (i) adversarialness, how much more challenging a question is for ai models compared to humans while being well-posed; and (ii) discriminability, how informative is the question in effectively distinguishing between different skill levels. ## adversarial benchmark evaluation we compare adversarial benchmarks across different domains using advscore. our evaluation includes advqa, a new qa dataset developed through a human-in-the-loop (hitl) process to align adversarial data with human capabilities. this section, analyzes advscore as a metric, while § 5 details the creation of advqa, and § 6 examines what makes advqa questions adversarial. ## advqa creation pipeline in the previous sections, we showed that advqa is more adversarial and discriminative than other datasets, suggesting its creation process contributed to these qualities. here, we discuss the advqa collection process as a case study to guide future high-quality adversarial datasets. ## discussion and analysis on advqa in this section, we show how advscore can help identify factors that encourage high-quality adversarial datasets. effective strategies in ad-vqa may guide the creation of more adversarial questions, and we analyze how the dataset's realistic aspect can help incorporate human variability during model evaluation. ## related work adversarial samples expose and evaluate model capabilities @xcite . recently, the natural language processing (nlp) community has questioned whether models trained on benchmarks learn to solve tasks in robust and generalizable ways @xcite @xcite @xcite . thus, evaluation of adversarial samples has been active in areas of reading comprehension @xcite and neural translation tasks @xcite . @xcite postulates that the abilities of many "superhuman" models may be overestimated due to poorly annotated datasets and biases embedded in the evaluation process (e.g., fixed test sets). an alternative is to provide more challenging benchmarks that require a stronger form of generalization and diversity @xcite @xcite ; hitl adversarial generation framework enables humans create examples while interacting with the model @xcite . for qa tasks, it is crucial to validate the model's ability to correctly answer easy and natural questions that are likely to be expressed by humans. for hitl adversarial generation for qa, @xcite and @xcite uses a synthetic generation method to amplify small set of human-authored adversaries. @xcite introduces a benchmark in which the humans interact with a visual qa model, and write an adversarial question for each of a set of images. @xcite and @xcite both use hitl incentive mechanisms to create adversarial questions. for evaluation of these adversarial datasets, @xcite introduces an irt-based ranking method to remedy the issue that current evaluation treats each model independently rather than considering relative differences. @xcite also redesigns the leaderboard framework with a bayesian approach where latent subject skill and item difficulty predict correct responses. our advscore can systematically probe models to understand their capabilities, and provide a measure to understand which also contribute in hitl adversarial dataset framework to help to create the next generation of data. ## conclusion adversarial datasets offer practical benefits for evaluating models to improve robustness and performance. grounded in human feedback, advscore ensures that evaluations of adversarial benchmarks align with human capabilities by post-hoc assessment of adversarial robustness and model improvements. thus, applying advscore in real-time benchmark construction can aid in evaluating the robustness of the models, and integrating advs-core into model training can improve their adaptability to real-world applications.