Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval (2024)

Abstract: In today’s digital world, seeking answers to health questions on the Internet is a common practice. However, existing question answering (QA) systems often rely on using pre-selected and annotated evidence documents, thus making them inadequate for addressing novel questions. Our study focuses on the open-domain QA setting, where the key challenge is to first uncover relevant evidence in large knowledge bases. By utilizing the common retrieve-then-read QA pipeline and PubMed as a trustworthy collection of medical research documents, we answer health questions from three diverse datasets. We modify different retrieval settings to observe their influence on the QA pipeline’s performance, including the number of retrieved documents, sentence selection process, the publication year of articles, and their number of citations. Our results reveal that cutting down on the amount of retrieved documents and favoring more recent and highly cited documents can improve the final macro F1 score up to 10%. We discuss the results, highlight interesting examples, and outline challenges for future research, like managing evidence disagreement and crafting user-friendly explanations.

## introduction in the digital era, using the internet to search for health information has become a prevalent behavior @xcite . users turn to seek health advice online due to its ease of access, wide coverage of information, convenience of searching, interactivity, and anonymity @xcite . health information sought online includes anything regarding the symptoms and treatments of different diseases. in general, health information seeking can lead to enhanced patient involvement in medical decision-making, improved communication with care providers, and improved quality of life @xcite . nevertheless, finding trustworthy and relevant evidence in abundant online content remains an open challenge @xcite . especially in the medical field, clinical recommendations can change with time, so finding the latest evidence is essential for reliable answers. interacting with online search engines and conversational systems is done with question answering (qa). technical solutions based on machine learning (ml) and natural language processing (nlp) aim to automate this task and provide users with reliable answers to their inquiries. the purpose of qa systems is multi-fold: it helps scientists verify their research hypothesis by finding related studies, it allows lay users to find answers to their everyday health concerns, and enables factuality assessment of generative language models by factchecking their responses over trusted evidence @xcite . while qa can work with answering questions over a provided document, we are focusing on the more realistic and challenging problem of opendomain question answering, where extensive collections of documents with diverse topics have to be quried to find the relevant evidence @xcite . the open-domain qa systems usually consist of two main components, a retriever and a reader @xcite . the retriever's task is finding the relevant documents that will serve as the main source of evidence for answering the question. the reader (qa module) performs the reasoning process between the question and found evidence, and produces the final answer. while both components are essential for the system, we posit that the retrieval is a more challenging part, considering that the qa module receives the input from it and the quality of the final answer depends on the retrieved documents @xcite . retrieving credible evidence documents relevant to the query ensures the final output's quality. this is true for both the retrieval-powered text classification tasks and recently popular retrieval-augmented generation (rag) approaches @xcite . while much progress has been made in open-domain qa, addressing the challenges in retrieval settings still needs to be explored. these include assessing the quantity of documents needed to be retrieved for a reliable answer, the amount of evidence passages extracted from them, and the quality of the documents themselves, such as their recency and strength of findings. figure 1 shows an example of a health question answered with two different retrieved documents -the more recent one has more up-to-date knowledge and findings. to bridge this research gap, in this study, we perform an array of experiments to test the predictive performance of an open-domain qa system with different evidence retrieval strategies. we use three diverse datasets of biomedical and health questions, which contain discreet labels like yes and no as their final answers, and use their gold labels as ground truth. we use the large collection of 20 million biomedical research abstracts from pubmed as the knowledge base for evidence retrieval. by keeping the reader (answering module) fixed, we only vary the different retrieval aspects and measure the change in the qa performance as measured by classification metrics precision, recall, and f1. the settings we explore include the number of documents retrieved and sentences extracted from them, the articles' publication year, and their number of citations. our findings demonstrate that the qa performance is improved by accounting for the amount and quality of the retrieved documents. to summarize, our research contributions are: we make our code and datasets publicly available in a github repository. foot_0 ## related work in this section, we outline the work related to our study. ## foundations in this section, we explain the foundations of the study, including the used datasets, the evidence corpus, and the structure of the used qa system. ## experiments we conduct three groups of experiments to test the influence of different retrieval parameters on the performance of our qa system. ## results in this section, we present and describe the results of the conducted experiments. ## discussion in this section, we discuss and provide deeper insights uncovered in the results. ## conclusion in this paper, we conducted a number of experiments assessing the performance of a health question-answering system in an open-domain setting. moving away from the standard setup of working with a small evidence corpus, we expand the knowledge sources to a large corpus of more than 20 million biomedical research abstracts. we measured the answer prediction performance over three diverse datasets of health questions while varying four aspects: number of retrieved documents, number of extracted sentences, and evidence quality in the form of year of publication and number of citations. our results show that a lower number of documents retrieved leads to better performance, with the ideal spot in the 1-5 range. we also show that the performance is improved and made more reliable by using a time-aware evidence retrieval process, i.e., retrieving only the highly cited and more recent papers. our research leaves room for exploration of disagreeing and conflicting evidence, generating explanations for end users, and including further metadata. we hope our research will encourage more exploration of the open-domain health question answering setting and addressing real-world user needs.