Automatic Evaluation vs. User Preference in Neural Textual Q uestion A nswering over COVID -19 Scientific Literature (2020)

Abstract: We present a Question Answering (QA) system that won one of the tasks of the Kaggle CORD-19 Challenge, according to the qualitative evaluation of experts. The system is a combination of an Information Retrieval module and a reading comprehension module that finds the answers in the retrieved passages. In this paper we present a quantitative and qualitative analysis of the system. The quantitative evaluation using manually annotated datasets contradicted some of our design choices, e.g. the fact that using QuAC for fine-tuning provided better answers over just using SQuAD. We analyzed this mismatch with an additional A/B test which showed that the system using QuAC was indeed preferred by users, confirming our intuition. Our analysis puts in question the suitability of automatic metrics and its correlation to user preferences. We also show that automatic metrics are highly dependent on the characteristics of the gold standard, such as the average length of the answers.

## introduction the global health crisis caused by covid-19 raised several challenges not only to medical research community but also to the natural language processing (nlp) community. the pace of publication of scientific articles on this topic has caused an information overload for health experts, highlighting the need for information systems based on nlp and artificial intelligence that assist in finding relevant and accurate information in publications. faced with this situation, the white house and several leading research groups released the covid-19 open research dataset (cord-19) @xcite , which contains over 200k articles about covid-19, sars-cov-2 and related coronaviruses. additionally, they launched the kaggle cord-19 challenge where they defined 10 tasks with the aim of developing text and data mining tools on top of this dataset that can help the medical community to find answers to high priority scientific questions. the challenge has been a real success with more than 500 teams participating in the first round of the challenge @xcite . in this paper we present a question answering (qa) system that won one of the tasks in round one of the kaggle cord-19 challenge. our goal when building the system was to fight information overload. for this purpose, the system highlights sub-sentence replies, and only answers when the quality of retrieval and answers is satisfactory. the system uses neural textual qa techniques to directly find specific answers to the scientific questions listed in the tasks of the challenge. the system is not tailored towards specific questions, and can be readily used to answer any other question. one of the challenges during development was the lack of resources for automatically evaluating the system. after submission we have been able to partially evaluate the main components of our system, using related shared-tasks and datasets that have been recently released and were not available at the time. the automatic evaluation revealed that the submitted system was not optimal, contradicting some design choices, such as using the quac dataset @xcite to fine-tune the qa system in addition to the squad 2.0 @xcite dataset. we thus performed an additional a/b test to check whether users preferred the system using quac or not. the test confirmed our intuition, and raises questions on the suitability of automatic metrics and its correlation to human satisfaction. our findings are in line with de vries et al. (2020), who claim that due to the tradition of prioritizing quantity of data to naturalness and real-world use cases, researchers could end up with findings that are not relevant for the real world. in this sense the system trained on the squad 2.0 dataset, which is not built with prospective users, achieves highest automatic metrics but falls behind when attending to user preference. there have been recent efforts for building qa datasets closer to real-world use cases @xcite @xcite . among all of them, we show that the system trained also on the quac dataset is preferred by the users. the contributions of our paper are two-fold. first, we describe the qa system that is able to answer covid related questions to aid clinical scientists to find the information they need effectively among thousands of scientific papers. second, we report some analysis on the shortcomings that automatic evaluation of qa systems suffer from. ## related work the release of the cord-19 dataset has attracted the interest of a wide variety of researchers, who have focused on developing systems that help other scientists to obtain relevant information about the covid-19. some of these systems, as the one developed by verizon media foot_0 , focus themselves on the retrieval step. in this case they propose a search engine over the whole cord-19 dataset using the vespa data serving engine. more similar to ours we have the cord-19 search foot_1 developed with amazon comprehend medical and the covid19 research explorer foot_2 developed by google. in both of these systems, apart from retrieving articles with the search engine, they also highlight the most probable answer in the retrieved paper. development of all these systems is contemporary to the development of ours so we all share similar ideas but with different design patterns. apart from that, all the previously mentioned systems lack any quantitative evaluation. ## system description in this section we describe each of the modules of our neural qa system which answers covid-19 related queries. the system is composed of two main sub-systems. the first is an information retrieval (ir) module, which, given a user question, searches among the whole dataset and selects a datasets the system relies on the freely available cord-19 dataset of scientific papers about covid-19, sars-cov-2 and related coronaviruses @xcite to extract the answers. two datasets are used to fine-tune the neural qa module: squad2.0 @xcite , which is a reading comprehension dataset widely used in the qa research community, and quac @xcite , which is a conversational qa dataset containing a higher rate of non-factoid questions than squad. paper filtering as we are mostly interested in papers related to covid-19, we filter out papers that are about coronaviruses other than covid-19 (for example, sars-cov and mers) from the cord-19 dataset. for that purpose, we created a list of synonyms of covid-19 ("coronavirus 2019", "ncov 2019", "sarscov2", "wuhan coronavirus", etc), and we check if a synonym appears in the title or the abstract of a paper. in that way, we filter out those papers that do not include any of the synonyms. from now on, we will consider only the papers that we keep after filtering. information retrieval the ir module indexes not only the abstracts, but also the full text of the papers. as shorter piece of texts are more favorable for the following qa component in the pipeline, the indexing unit is an abstract or each of the paragraphs of the full text. the text is tokenized, stemmed and lower cased, and a stopword filter is applied before indexing as common practice. the classical bm25f search algorithm @xcite is used to retrieve the most relevant paragraphs given a natural language question. question answering given a question in natural language and a paragraph, the qa module returns the answer to the question in the paragraph or "no answer" otherwise. the implemented system is based on neural network techniques. more specifically, we have used the scibert language representation model, which is a pretrained language model based on bert, but trained on a large corpus of scientific text, including text from biomedical domain @xcite . following the usual reading comprehension method we use scibert as a pointer network, which selects an answer start and end index given a question and a paragraph. we used both squad and quac to fine-tune scibert for qa. manual inspection revealed that a system fine-tuned on squad2.0 produced answers that were specially good for covid related questions seeking short answers. however, we also observed that a fine-tuning with squad2.0 and quac produced answers of better quality, particularly for questions which require longer answers. we thus decided to use a scibert model fine-tuned first on squad2.0 and then on quac 4 as our final model. pipeline the final system 5 combines both ir and qa modules in a pipeline, as follows. first, we refined (split or simplify) manually the questions that were too large and complex for the qa system. then we retrieved the 20 most relevant paragraphs using the ir module for each of the questions. finally, we ran the qa module over the relevant paragraphs to select specific answers from them. if more than %85 of these 20 answers are of "no answer" type, the current question will not be answered to not overload the users with bad answers. in the other cases, the best answer for each of the best five paragraphs will be shown. additionally, next to each answer, we show extra information such as the journal and title of the paper (with a link to access online version on the web). moreover, we also show the paragraph from which each answer has been extracted. each paragraph is highlighted with the best 3 answers, using different lightness of color (the darker the better the answer). appendix a shows an example of the output of the system for one of the questions of a task from kaggle cord-19 challenge. ## discussion on evaluation methods the systems in the kaggle challenge were judged by an evaluation committee based on the accuracy, as well as on the documentation and the overall presentation of the submissions. no automatic evaluation was performed, due to the lack of manually annotated datasets available. according to this manual evaluation, our system was the best on one of the tasks of the cord-19 challenge. ## conclusions and future work we have presented a qa system that is able to answer covid-19 related questions with a high degree of success, as shown by the prize obtained at the kaggle cord-19 challenge. in this challenge a qualitative evaluation was conducted by a committee formed by experts, but no automatic evaluation was performed. we present an automatic evaluation of both the ir and qa modules, using different resources that have been created recently. in the case of the use (or not) of quac training data, the automatic evaluation contradicted our intuitions, and we thus performed a/b test between the system submitted to kaggle (using quac) and the system that performed best according to the automatic evaluation (not using quac). this a/b test confirms our initial intuitions, and shows that the automatic evaluation metrics do not correlate well with human preference. the analysis also shows that automatic metrics are highly dependent on the characteristics of the gold standard, such as the average length of the answers. as future work, we plan to perform more thorough studies on the contradictions between automatic evaluation and user preferences, in line with @xcite . we would also like to devise automatic metrics for qa in scenarios where complex and self-explanatory answers are expected by the user, hopefully showing higher correlation with user preferences. these metrics would be essential for the research community in order to build tools that are helpful for end users.