Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision (2022)

Abstract: Current medical question answering systems have difficulty processing long, detailed and informally worded questions submitted by patients, called Consumer Health Questions (CHQs). To address this issue, we introduce a medical question understanding and answering system with knowledge grounding and semantic self-supervision. Our system is a pipeline that first summarizes a long, medical, user-written question, using a supervised summarization loss. Then, our system performs a two-step retrieval to return answers. The system first matches the summarized user question with an FAQ from a trusted medical knowledge base, and then retrieves a fixed number of relevant sentences from the corresponding answer document. In the absence of labels for question matching or answer relevance, we design 3 novel, self-supervised and semantically-guided losses. We evaluate our model against two strong retrieval-based question answering baselines. Evaluators ask their own questions and rate the answers retrieved by our baselines and own system according to their relevance. They find that our system retrieves more relevant answers, while achieving speeds 20 times faster. Our self-supervised losses also help the summarizer achieve higher scores in ROUGE, as well as in human evaluation metrics.

## introduction motivation. users of medical question answering systems often write long questions, called @xcite . several aspects of chqs hinder the capacity of current question answering (qa) systems to process them: long medical questions may contain peripheral information like patient history @xcite that are not necessary to retrieve relevant answers. consumer health questions may also use a distinct vocabulary from the one used by medical providers to describe the same health concepts @xcite . a growing number of approaches attempt to enhance the processing of consumer health questions -or medical question understanding. these approaches include query relaxation @xcite , question entailment @xcite @xcite , question summarization @xcite , and question similarity @xcite . however, the above medical question understanding approaches stop short of retrieving answers after processing consumer health questions. the medical question answering task at trec 2017 liveqa @xcite attempts to fill the gap by proposing the task of consumer health question answering. the goal is to retrieve relevant answers obtained using online search for the corresponding chq. as part of their participation in this task, @xcite find that online search engine queries introduce noise in performance, and that even collected and curated medical knowledge available offline can fare better. contributions. to enable the use of a curated medical knowledge base for answering long user questions, we introduce a novel, knowledge-grounded and semantically self-supervised system for consumer health question understanding and answering (chqua). we tackle a challenging aspect of chqua: providing answers when no relevance labels are available. our contributions are as follows: finally, we conduct an evaluation to compare the relevance of our system with two strong baselines of retrieval-based question answering. we ask evaluators to ask their own questions, and then perform a blind evaluation of the retrieved answers by each system. seven evaluators find that our system retrieves more relevant answers compared to the two baselines, while achieving significantly faster processing speeds. we also find that the self-supervised losses help achieve better scores in rouge and human evaluation metrics. how-ever, we find that the task remains challenging, with room for improvement. we release our code, model, and matched datasets to encourage further research in consumer health question understanding and answering. many participating teams adopt a question matching approach, and train their models on question similarity datasets like the quora question pair dataset @xcite , or other datasets collected from community question answering websites. todo @xcite in the mediqa 2019 shared task, ben abacha and demner-fushman (2019a) introduce a differently defined consumer health question answering task. here, the goal is to rank a given list of answers according to their relevance with regard to a chq. @xcite introduce a new disease knowledge infusion training procedure for bert @xcite that scores well in this task. medical question answering. medical qa approaches include translating questions to sparql queries @xcite , semantic similarity between questions and candidate answers @xcite , knowledge representations @xcite , ranking candidate answers @xcite , summarization of questions and/or answers @xcite , and medical entity linking @xcite . there is a variety of definitions for the task of medical qa and related sub-tasks in the literature. @xcite define medical qa as the task of finding the correct answer from a set of candidates and a body of evidence documents. they propose to work on two datasets: the national medical li-censing examination of china (nmlec) @xcite , and clinical diagnosis based on electronic medical records (cd-emr), where the goal is to predict the correct diagnosis based on patient history. @xcite propose to tackle three kinds of medical questions found in the bioasq challenge @xcite : factoid questions where answers are single entities, list-type questions where answers are a set of entities, and yes/no questions. retrieval-based question answering. recent methods for retrieval-based qa systems use contextual text embeddings to evaluate a candidate answer's relevance to a given question. @xcite propose to use multi-cast attention networks (mcan), a new attention mechanism, to model question-answer pairs. mrini et al. (2021e) introduce a recursive, treestructured model that models sentences according to their syntactic tree. their results show that tree structure sets a new state of the art in conventional, formally worded qa benchmarks like trecqa and wikiqa @xcite , but does not fare well in informally worded, user-written datasets. @xcite introduce dense passage retrieval (dpr): a dual-encoder based on bert @xcite , that predicts relevance scores of passages with regard to a question. dpr encoders are trained on the relevance of passages from datasets containing such labels, using a supervised negative log-likelihood loss based on the semantic similarity of questions and relevant passages. @xcite modify the query part of retrieval-based qa: they propose to use language models to generate context for queries. they then feed the extended queries to retrieval systems, such as dpr or bm-25. ## problem definition we define knowledge-grounded consumer health question understanding and answering (chqua) as the problem of retrieving a fixed number of answer sentences from a medical knowledge base that are the most relevant given a long and informal user question -called a consumer health question (chq). there are three steps in chqua: question summarization, matching the summarized user question with a relevant faq from the knowledge base, and retrieval of the relevant answer sentences from the corresponding answer document. knowledge-grounded chqua is comprised of three elements used for training. first, the chq is the input of the task. second, the reference faq (frequently asked question) is the golden or expert-written summary corresponding to the chq. whereas the chq is a long and informally worded question, the reference faq is the corresponding short, one-sentence, formally worded question. at inference time, the reference faq is not available, and we will therefore use a summary generated by the model. third, the medical knowledge base is comprised of faqs, where each faq has a corresponding answer document with at least one sentence. faqs in the knowledge base are also short, one-sentence, formally worded questions. the goal of knowledge-grounded chqua is to find a set r of n relevant answer sentences, from a document comprised of answer sentences a i , such that a i corresponds to question q i from the knowledge base. we call q i the retrieved or matching faq, such that q i is the most similar question to the user's summarized question q u : ## our pipeline our proposed pipeline for consumer health question understanding and answering has three main components. in the first step, our approach learns to understand the intent of user questions (chqs) by summarizing them. we use an encoder-decoder-based summarization model for this step. the second step is question matching, or the retrieval of the relevant faq from the knowledge base: we ground the generated summary to a medical knowledge base of faqs and corresponding ## experiments and results in this section, we evaluate our proposed pipeline for consumer health question understanding and answering, and we propose to compare our proposed pipeline against two strong baselines. seven medical experts judge the performance of our system and baselines by asking their own questions, and rating the relevance of the answers retrieved. then, we analyze the results through the lens of summarization metrics, human evaluation, and computational speed. ## conclusions we introduce an end-to-end pipeline for knowledgegrounded consumer health question answering and understanding (chqua). our challenge is that we have no labels for question matching or answer relevance. we propose to use semantic similarity as a proxy for those labels, and we design three novel self-supervised losses: one works to match the user's summarized question to a knowledge base question, and the other two losses work complementarily to teach our model to select a fixed number of relevant answer sentences. we compare our proposed system against two strong baselines of retrieval-based question answering. we hire seven medical experts to ask their questions, and they find that our system provides more relevant answers. our system also achieves processing times that are more than 20 times faster. finally, we find that our proposed self-supervised losses enable the summarizer model to achieve higher scores in rouge and human evaluation metrics, compared to a summarization-only baseline. however, we find that this task remains challenging and that there is still room for improvement. we release our code and model to encourage further research.