COUGH : A Challenge Dataset and Models for COVID -19 FAQ Retrieval (2021)

Abstract: We present a large, challenging dataset, COUGH, for COVID-19 FAQ retrieval. Similar to a standard FAQ dataset, COUGH consists of three parts: FAQ Bank, Query Bank and Relevance Set. The FAQ Bank contains ~16K FAQ items scraped from 55 credible websites (e.g., CDC and WHO). For evaluation, we introduce Query Bank and Relevance Set, where the former contains 1,236 human-paraphrased queries while the latter contains ~32 human-annotated FAQ items for each query. We analyze COUGH by testing different FAQ retrieval models built on top of BM25 and BERT, among which the best model achieves 48.8 under P@5, indicating a great challenge presented by COUGH and encouraging future research for further improvement. Our COUGH dataset is available at https://github.com/sunlab-osu/covid-faq .

## introduction many institutional websites today maintain an faq page to help users find relevant information for commonly asked questions. the faq retrieval task is defined as ranking faq items {(q i , a i )} 1 from a collection given a user query q @xcite . in contrast to common information retrieval (ir), faq retrieval often introduces 3 new challenges: 1) brevity of faq texts in comparison with ir documents; 2) need for topic-specific knowledge; 3) usage of the new question field in faq items @xcite . however, faq retrieval is under-studied compared with other ir applications such as opendomain qa @xcite . in this work, we specifically study faq retrieval for covid-19, a contagious and fatal pandemic which is still evolving on a daily basis. many websites like cdc and who provide quality information on covid-19 and update faq pages regularly. to gain better insights into faq retrieval research and advance covid-19 information search, we present an faq dataset, cough foot_0 , consisting of faq bank, query bank and relevance set, following the standard of constructing an faq dataset @xcite . the faq bank contains 15919 faq items scraped from 55 authoritative institutional websites (see a full list in table a4 and a5 ). cough covers a wide range of topics on covid-19, from general information about the virus to specific covid-related instructions for a healthy diet. for evaluation, we further construct query bank and relevance set, including 1,236 crowd-sourced queries and their relevance to a set of faq items judged by annotators. examples from cough are shown in figure 1 . our dataset poses several new challenges (e.g., answer fields are longer and noisier, and harder to match, than question fields) to existing methods. the diversity of faq items, reflected in varying query forms and lengths as well as in narrative styles, also contributes to these challenges. the contribution of this work is two-fold. first, we construct a challenging dataset cough to aid the development of covid-19 faq retrieval models. second, we evaluate various faq retrieval models across different settings, explore their limitations, and encourage future work along this line. the most related works to ours are @xcite and @xcite , both of which constructed a collection of covid-19 faqs by scraping authoritative websites. however, the dataset in the former work is not available yet and the latter work does not evaluate models on their dataset, and there is still a great need to understand how existing models would perform on the covid-19 faq retrieval task. in the open domain, several faq datasets appeared recently, such as faqir @xcite , stackfaq @xcite and localgov @xcite . unfortunately, as shown in ## dataset analysis besides the generic goal of large size, diversity, and low noise, cough features 5 additional aspects. varying query forms: as indicated in however, due to budget limit, we did not proceed to the annotation phase for non-english data, so there is no non-english human-paraphrased user query or relevance judgement. annotation quality: we discard low-quality paraphrased queries (∼24%) and relevance annotations (∼11%). further, we show that ∼74% of annotated tuples have high agreements where multiple people vote for the same relevance class. more details of quality checking can be found in section 8.1. ## conclusion in this paper, we introduce cough, a large challenging dataset for covid-19 faq retrieval. cough features varying query forms, long and noisy answers, and multilinguality. cough also serves as a better evaluation benchmark since it has quality larger-scale relevance annotations. we discuss the limitations of current faq retrieval models via comprehensive experiments, and encourage future research to further improve faq retrieval. task details base cost base cost per unit cognitive complexity user query bank construction reference qa annotation identify 3 qa pairs (write questions and then find answers). 24 8 high question annotation on an audio clip identify 3 questions (write questions and then find answers), each of which has additional requirements (e.g., originality, creativeness). 30 10 extremely high ours* paraphrase queries give 3 paraphrases for the original query template. 12 4 medium annotated relevance set construction reference image labeling locate 5 required objects in a given image. 7 1.4 medium website class identification identify the type of niche of a twitter account. select from 6 classes. 2 2 low identify an item given an image, fill out a form with 6 required fields. 9 1.5 medium ours* relevance judgements identify the relevance. select from 4 classes. 2 2 low after crowdsourcing, we conducted post-hoc quality checking on both query bank and relevance set. we manually checked all 1,236 user queries and found that all of them make sense, are related to covid-19 and properly written. due to the subjectivity of the relevance judgement task, we evaluated the quality of the relevance annotations in two ways: 1) we find that 73.5% of query, faq item tuples have high agreements where multiple people vote for the same relevance class; 2) we re-judge the relevance on randomly sampled 1000 tuples by hiring two research assistants and it turns out that the matching level 12 is 76.5%. overall, the post-hoc checking confirms that our cough dataset is of high quality. annotation protocols. to further help ethics com-mittees and the public judge the fairness of our annotation process, the annotation protocols for both annotation tasks are listed in appendix a. figure a1 and a2 show the interfaces designed for the annotation process. d. for each annotated query, faq item tuple, we convert "matched" and "useful" to positive annotations, and "useless" and "non-relevant" to negative annotations. we then apply majority voting using converted binary annotations.