COVID-QA : A Question Answering Dataset for COVID -19 (2020)

Abstract: We present COVID-QA, a Question Answering dataset consisting of 2,019 question/answer pairs annotated by volunteer biomedical experts on scientific articles related to COVID-19. To evaluate the dataset we compared a RoBERTa base model fine-tuned on SQuAD with the same model trained on SQuAD and our COVID-QA dataset. We found that the additional training on this domain-specific data leads to significant gains in performance. Both the trained model and the annotated dataset have been open-sourced at: https://github.com/deepset-ai/COVID-QA

## summary we selected 147 scientific articles mostly related to covid-19 from the cord-19 (the white house office of science and technology policy, 2020 (accessed may 9, 2020) collection to be annotated by 15 experts. although the annotators were volunteers, it was required that all have at least a master's degree in biomedical sciences. the annotation team was led by a medical doctor (g.a.r.) who vetted the volunteer's credentials and manually verified each question/answer pair produced. we used an existing, web-based annotation tool that had been created by deepset 1 and is available at their neural search framework haystack 2 . the annotations were created in squad @xcite style fashion where annotators mark text as answers and formulate corresponding questions. covid-qa differs from squad in that answers come from longer texts (6118.5 vs 153.2 tokens), answers are generally longer (13.9 vs. 3.2 words) and it does not contain n-way annotated development nor test sets. we chose a roberta-base architecture @xcite squad dataset as a baseline model. we continued training the baseline model using our covid-qa annotations in 5-fold cross validation manner.