Using the Poly-encoder for a COVID -19 Question Answering System (2020)

Abstract: To combat misinformation regarding COVID- 19 during this unprecedented pandemic, we propose a conversational agent that answers questions related to COVID-19. We adapt the Poly-encoder (Humeau et al., 2020) model for informational retrieval from FAQs. We show that after fine-tuning, the Poly-encoder can achieve a higher F1 score. We make our code publicly available for other researchers to use.

## introduction at the beginning of 2020, the covid-19 pandemic spread quickly across the world as did harmful misinformation (e.g., about how the disease spreads) leading to preventable mistakes in managing its spread. information regarding covid-19 must be communicated by verified sources such as the center for disease control and prevention (cdc) and the world health organization (who). furthermore, covid-19 information is rapidly changing, making it difficult to distinguish evidence-based guidance from misinformation. to help society respond to covid-19, people need easy access to accurate information about covid-19. for some people, a conversational interface can provide a quick and accessible interface to trusted sources. we adapt the poly-encoder @xcite ) for the covid-19 qa task. the performance of a qa model is assessed not only by prediction quality but also by prediction speed, as scoring many candidates can be extremely slow. recently, the fine-tuning of deep pre-trained language models @xcite has been shown to achieve state-of-the-art benchmarks on a qa task @xcite . the poly-encoder extends these methods for conversational information retrieval. another challenge for the covid-19 qa system is to return trusted information to user questions. hence, we use the dataset of @xcite which has over two-thousand verified questions and answers from trusted online sources (e.g., who, cdc). the answers are verified by researchers at the johns hopkins bloomberg school of public health (jhsph). finally, a third challenge is how to evaluate the covid-19 qa system. to research this important problem, experts from jhsph annotated a dataset of similarity scores between faqs from trusted sources and user questions from online sources (e.g. qorona foot_0 , twitter, and dialoguemd foot_1 ). 3 this similarity set allows us to evaluate our qa system. we contribute the following: ## covid-19 qa we formulate our covid-19 qa task as a problem of how to choose answers from a set of pre-selected sentences @xcite . specifically, we want to respond to a user query with a response from frequently asked questions (faqs) @xcite . the poly-encoder model can use the qa context to respond both quickly and accurately to user queries. ## experimental setup we fine-tuned the poly-encoder with the redditpre-trained transformer using the covid-19 qa dataset and evaluated it on the q-a and q-q datasets. ## results and analysis we conducted experiments using two types of evaluation sets. one is covid-19 qa dataset (q-a) for the evaluation, which is composed of questionanswer pairs and includes randomly selected distractor candidate answers. to evaluate the model's ability relative to the number of candidates, we evaluated models with a different number of candidates in we evaluated our model and the baseline using both 10 foot_10 and 20 candidates in q-a ( another avenue of evaluation is the question to question (q-q) similarity set. in this case, we evaluated the jhu-covid-qa@20 model with q-q set and attained an accuracy of 72.05%. this demonstrated that a model only fine-tuned on the q-a dataset worked well on q-q evaluation; however, we only saw improvement in recall over the baseline. the performance can be improved by also including q-q as well as q-a in fine-tuning. we leave this for future work.