  We present the ARC-DA dataset, a direct-answer ("open response", "freeform")
version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset. While ARC
has been influential in the community, its multiple-choice format is
unrepresentative of real-world questions, and multiple choice formats can be
particularly susceptible to artifacts. The ARC-DA dataset addresses these
concerns by converting questions to direct-answer format using a combination of
crowdsourcing and expert review. The resulting dataset contains 2985 questions
with a total of 8436 valid answers (questions typically have more than one
valid answer). ARC-DA is one of the first DA datasets of natural questions that
often require reasoning, and where appropriate question decompositions are not
evident from the questions themselves. We describe the conversion approach
taken, appropriate evaluation metrics, and several strong models. Although
high, the best scores (81% GENIE, 61.4% F1, 63.2% ROUGE-L) still leave
considerable room for improvement. In addition, the dataset provides a natural
setting for new research on explanation, as many questions require reasoning to
construct answers. We hope the dataset spurs further advances in complex
question-answering by the community. ARC-DA is available at
https://allenai.org/data/arc-da
