  Datasets are not only resources for training accurate, deployable systems,
but are also benchmarks for developing new modeling approaches. While large,
natural datasets are necessary for training accurate systems, are they
necessary for driving modeling innovation? For example, while the popular SQuAD
question answering benchmark has driven the development of new modeling
approaches, could synthetic or smaller benchmarks have led to similar
innovations?
  This counterfactual question is impossible to answer, but we can study a
necessary condition: the ability for a benchmark to recapitulate findings made
on SQuAD. We conduct a retrospective study of 20 SQuAD modeling approaches,
investigating how well 32 existing and synthesized benchmarks concur with SQuAD
-- i.e., do they rank the approaches similarly? We carefully construct small,
targeted synthetic benchmarks that do not resemble natural language, yet have
high concurrence with SQuAD, demonstrating that naturalness and size are not
necessary for reflecting historical modeling improvements on SQuAD. Our results
raise the intriguing possibility that small and carefully designed synthetic
benchmarks may be useful for driving the development of new modeling
approaches.
