What Has Been Lost with Synthetic Evaluation? (2025)

Abstract: Large language models (LLMs) are increasingly used for data generation. However, creating evaluation benchmarks raises the bar for this emerging paradigm. Benchmarks must target specific phenomena, penalize exploiting shortcuts, and be challenging. Through two case studies, we ask whether LLMs are ready to meet these demands—by generating reasoning-over-text benchmarks and comparing them to those that were created through careful crowdsourcing. Specifically, we evaluate both the *validity* and *difficulty* of LLM-generated versions of two high-quality reading comprehension datasets: CondaQA, which evaluates reasoning about negation, and DROP, which targets reasoning about quantities. We find that prompting LLMs can produce variants of these datasets that are often valid according to the annotation guidelines, at a fraction of the cost of the original crowdsourcing effort. However, we show that they are *less challenging for LLMs* than their human-authored counterparts. This finding sheds light on what may have been lost by generating evaluation data with LLMs, and calls for critically reassessing the immediate use of this increasingly prevalent approach to benchmark creation.

## introduction the development of benchmarks that test frontier models in reasoning over text has been at the heart of nlp research @xcite . such benchmarks aim to cover a broad range of input variations and related skills, and prevent models from performing well by exploiting data shortcuts @xcite . they are created to be challenging, so that even the most capable emerging models cannot solve them immediately and they remain a meaningful yardstick for measuring progress. yet these goals are often hindered by crowdsourcing these results show that these reasoning benchmarks remain challenging. we study whether comparable variants can be created using llms. † cf. @xcite . ‡ cf. @xcite . △ o3-mini-2025-01-31 we address this question through the lens of reading comprehension, which is a common format in nlp for probing reasoning over text capabilities of frontier models @xcite . specifically, we study compositional reasoning over numbers as in drop @xcite and reasoning about the implications of negated statements as in condaqa @xcite . these reasoning types continue to challenge llms, as shown in what would llms-as-annotators need to be able to generate to create datasets like these? first, questions that are complex and require deeper understanding or inference, rather than being answerable by a phrase matching the paragraph's wording. second, minimal edits to the passages or questions to test that the model has the skills related to a specific type of reasoning. for example, an edit that only changes what is negated in a passage enables testing whether the model has the skill of understanding negation scope. in turn, this skill enables reasoning about both the original and edited passage, and answering questions about the negated content in the context of both passages correctly. we extensively prompt an llm to generate complex questions and contrastive edits, as would likely be done by dataset creators opting to use llms as annotators. prompt engineering we conduct ( §4) illustrates that using llms as annotators does not eliminate the need for iterative instruction refinement and manual review, much like traditional human annotation. that is, although an llm creates questions and edits faster and cheaper than people, the process of obtaining them may still require multiple rounds of manual review. our next findings are of practical importance. first, we manually analyze generations and find that llm-generated data is moderately to highly valid, though rarely perfect ( §4). second, in a user study with nlp researchers, we compare human and generated edits and find that generated edits are generally preferred to human-authored ones for better meeting the annotation guidelines ( §5). we attribute this result to their apparent simplicity, which allows them to adhere more closely to the guidelines. however, when benchmarking various models on human and generated versions, we find that human-authored edits still pose a greater challenge and rank models differently than generated versions, especially when bundle consistency is considered ( §6). these findings suggest that the core challenge is not validity, but difficulty. however, optimizing prompts for difficulty would require substantial human-in-the-loop effort because llms cannot yet reliably validate and answer their own generations. moreover, this would effectively turn evaluation-data generation into a form of adversarial dataset construction whose "naturalness" has been questioned @xcite . therefore, how to generate challenging instances without relying on excessive human effort and producing artificially-sounding text remains an open question. in light of this, using llms as annotators for any new benchmark does not seem advisable as of yet, especially if the validity of the generated data is the only measure of dataset quality. 1 2 background reading comprehension to probe reasoning. squad @xcite popularized using reading comprehension to probe models. @xcite show that squad questions are often answerable with a span in the paragraph that matches the expected answer type and is lexically related to the key words in the question. this has led to the other datasets posing greater reasoning challenges @xcite @xcite . we focus on two such datasets: drop @xcite and condaqa @xcite , which still challenge current models ( drop and condaqa questions. see contrast sets. models can solve benchmarks by exploiting data shortcuts @xcite . thus, they are better evaluated based on consistency -their ability to solve an entire set of slightly different examples with different ground truth, known as contrast sets @xcite or counterfactual data @xcite . contrastive edits are valid if minimal changes give a different ground truth, while maintaining fluency. however, edits targeting specific changes help evaluate whether models have the necessary skills for a specific type of reasoning. for example, an edit that requires changing only what is negated tests models' understanding of the negation scope: only with this skill can models consistently reason about negation and answer questions as "was parsons able to use his guitar after the fire?" in both contexts. we introduce other edit types below. ## generating drop and condaqa in this section, we outline our prompting workflow for generating questions ( §4.1) and edits ( §4.2). we also recap some of the prompting strategies we found most useful, and the data they were used to create in ## whose annotations are preferred? we next study whether nlp researchers prefer human or generated questions and edits based on how closely they follow the original guidelines for producing questions and edits with desired properties. there is a concern about potential dataset contamination when doing such studies with synthetic data, as models may simply memorize and reproduce the human-authored dataset from their training data. to mitigate against the effects of straightforward reproduction, we analyze the generated data to eliminate the possibility that generations are paraphrased instances of their human-authored counterparts; see appendix b. setup. we recruited 16 annotators who are nlp researchers and doctoral students to play the role of "dataset creators". 11 these annotators were selected to represent the typical demographic of researchers who develop evaluations for llms. we recruited them through social media platforms and personal networks and compensated them at a rate of @xmath0256. screenshots of this study are shown in figures 5-12 for condaqa and figures 19-26 for drop (appendix). 12 annotators were asked to imagine themselves as dataset creators who were designing a benchmark to probe llms' capabilities to reason and were provided background about the format of the datasets (without mentioning their names). they were then asked to choose edits and questions that better meet the dataset specification. they were also asked if the question or edit was invalid. for condaqa, annotators are shown one question pair and three edit pairs corresponding to the three edit types. for drop, annotators are shown one question pair and one compositional-edit pair. 13 since the compositional edits in §4 were made on generated questions, we created compositional edits for human-authored questions here to enable a valid comparison. the order of human and generated annotations for both studies is randomized. other details are in appendix §c. results. tables 2 3 show the results. we observe that annotators assess both generated and human edits and questions as valid at a much lower rate than one author of this paper, with the exception of drop compositional edits. they also rate the original condaqa paraphrase/scope edits and questions, along with the drop questions, as less valid than the generated ones. moreover, annotators are often shown to prefer model-generated questions and edits to human-written questions and edits from the original dataset. we include example agreement and disagreement cases between annotators and one author of this paper for generated instances of condaqa and drop questions and edits in tables 16-21 in the appendix. the annotators' higher validity and preference for generations likely stems from the fact that the generated edits tend to be simpler and, due to the way the prompts were designed, follow the guidelines for each type of edit more closely. in contrast, original annotators often interpret the question and edit definitions in slightly different ways and sometimes introduce extra flair to the annotations. while 12 html templates for the study are available at our github. 13 we excluded temporal edits because we could identify only ≈20 human-authored ones. ## conclusion we ask how benchmarks with llm-generated content compare to human-authored versions. we summarize our main findings and conclusions here, and provide a more detailed discussion in appendix e. we find that while generated content is often valid ( §4, §5), it tends to produce easier benchmarks ( §6). further, we find that these syntheticallygenerated datasets do not necessarily preserve model ordering, which could consequently lead practitioners to make different decisions around model development and deployment. finally, we also find that these quality differences between synthetic and human-authored data, may not be perceptible to researchers inspecting the data. thus, while llms are promising for data creation where complexity is less critical, reliable human annotators remain vital for new benchmarks assessing real-world generalization, corner cases, and testing models on nuanced, complex scenarios.