What Are People Asking About COVID-19 ? A Question Classification Dataset (2020)

Abstract: We present COVID-Q, a set of 1,690 questions about COVID-19 from 13 sources, which we annotate into 15 question categories and 207 question clusters. The most common questions in our dataset asked about transmission, prevention, and societal effects of COVID, and we found that many questions that appeared in multiple sources were not answered by any FAQ websites of reputable organizations such as the CDC and FDA. We post our dataset publicly at https://github.com/JerryWei03/COVID-Q . For classifying questions into 15 categories, a BERT baseline scored 58.1% accuracy when trained on 20 examples per category, and for a question clustering task, a BERT + triplet loss baseline achieved 49.5% accuracy. We hope COVID-Q can help either for direct use in developing applied systems or as a domain-specific resource for model evaluation.

## introduction a major challenge during fast-developing pandemics such as covid-19 is keeping people updated with the latest and most relevant information. since the beginning of covid, several websites have created frequently asked questions (faq) pages that they regularly update. but even so, users might struggle to find their questions on faq pages, and many questions remain unanswered. in this paper, we ask-what are people really asking about covid, and how can we use nlp to better understand questions and retrieve relevant content? we present covid-q, a dataset of 1,690 questions about covid from 13 online sources. we annotate covid-q by classifying questions into 15 general question categories 1 (see figure 1 ) and by grouping questions into question clusters, for which all questions in a cluster ask the same thing 1 we do not count the "other" category. other ( 6 and can be answered by the same answer, for a total of 207 clusters. throughout §2, we analyze the distribution of covid-q in terms of question category, cluster, and source. covid-q facilitates several question understanding tasks. first, the question categories can be used for a vanilla text classification task to determine the general category of information a question is asking about. second, the question clusters can be used for retrieval question answering (since the cluster annotations indicate questions of same intent), where given a new question, a system aims to find a question in an existing database that asks the same thing and returns the corresponding answer @xcite . we provide baselines for these two tasks in §3.1 and §3.2. in addition to directly aiding the development of potential applied systems, covid-q could also serve as a domain-specific resource for evaluating nlp models trained on covid data. 1 shows the distribution of collected questions from each source. we also post the original scraped websites for each source. ## question understanding tasks we provide baselines for two tasks: questioncategory classification, where each question belongs to one of 15 categories, and question clustering, where questions asking the same thing belong to the same cluster. as our dataset is small when split into training and test sets, we manually generate an additional author-generated evaluation set of 249 questions. for these questions, the first author wrote new questions for question clusters with 4 or 5 questions per cluster until those clusters had 6 questions. these questions were checked in the same fashion as the real questions. for clarity, we only refer to them in §3.1 unless explicitly stated. ## discussion use cases. we imagine several use cases for covid-q. our question clusters could help train and evaluate retrieval-qa systems, such as covid.deepset.ai or covid19.dialogue.co, which, given a new question, aim to retrieve the corresponding qa pair in an existing database. another relevant context is query understanding, as clusters identify queries of the same intent, and categories identify queries asking about the same topic. finally, covid-q could be used broadly to evaluate covid-specific models-our baseline (huggingface's bert-base-uncased) does not even have covid in the vocabulary, and so we suspect that models pre-trained on scientific or covidspecific data will outperform our baseline. more related areas include covid-related query expansion, suggestion, and rewriting. limitations. our dataset was collected in may 2020, and we see it as a snapshot in time of questions asked up until then. as the covid situation further develops, a host of new questions will arise, and the content of these new questions will potentially not be covered by any existing clusters in our dataset. the question categories, on the other hand, are more likely to remain static (i.e., new questions would likely map to an existing category), but the current way that we came up with the categories might be considered subjective-we leave that determination to the reader (refer to