Efficient Medical Question Answering with Knowledge-Augmented Question Generation (2024)

Abstract: In the expanding field of language model applications, medical knowledge representation remains a significant challenge due to the specialized nature of the domain. Large language models, such as GPT-4, obtain reasonable scores on medical question-answering tasks, but smaller models are far behind.In this work, we introduce a method to improve the proficiency of a small language model in the medical domain by employing a two-fold approach. We first fine-tune the model on a corpus of medical textbooks. Then, we use GPT-4 to generate questions similar to the downstream task, prompted with textbook knowledge, and use them to fine-tune the model. Additionally, we introduce ECN-QA, a novel Medical QA dataset containing “progressive questions” composed of related sequential questions. We show the benefits of our training strategy on this dataset.The study’s findings highlight the potential of small language models in the medical domain when appropriately fine-tuned.

## introduction deep learning led to a breakthrough in natural language processing, reaching human performances on many tasks like question answering or translation. however, their performances are still subpar in complex domains, such as medicine. this domain presents unique challenges, mainly due to its specialized vocabulary, complex concepts, and fast-changing medical literature. language-based medical tasks, such as medical question answering, require vast knowledge and reasoning abilities to make correct diagnoses. traditional language models (lms), while effective in general language processing, struggle when faced with medical knowledge learning mainly because sufficient data for medical knowledge is not necessarily readily avail-* equal contribution able for training. moreover, in the context of language models, their number of parameters often plays a pivotal role in performances. large models, although powerful, come with high computational costs and resource requirements, both for training and inference, making them less accessible and practical for widespread use. on the other hand, small models, which are more economical, face challenges in generalization and adapting to specialized domains like medicine. these models require careful fine-tuning to grasp the depth and breadth of medical knowledge effectively. the diversity of general, non-medical datasets on which lms are trained poses another challenge. these datasets, encompassing a wide array of topics and styles, do not specifically cater to the medical domain. as a result, small models trained on such datasets might fail to develop the necessary understanding for answering more specialized medical questions. therefore, we tackle these issues for medical question answering tasks. first, we design a new dataset, ecn-qa. existing medical question answering (qa) datasets such as medqa @xcite and others @xcite , @xcite are usually single-question multiple answers, which do not encompass the complexity of making a medical diagnosis, which requires multiple turns of questions. our dataset is based on the french medical residency examination and contains multiple related questions that require models to remember previous questions and reasoning over multiple steps. we then propose a method to train small to mid-size language models for medical question answering. we leverage a corpus of medical textbooks for pre-training. the pre-training set is enriched with specialized questions generated by large language models prompted with medical data from books. this helps to specialize the model on the target task with a small amount of original data. our code will be made available online. ## method we detail our training strategy in this section. the strategy is depicted in figure 1 . we detail related works in the appendix a. ## conclusion we introduced ecn-qa, a novel dataset for medical question answering that contains a novel type of exercise: progressive questions. we proposed a training strategy based on prompted question generation that improves results over our baseline model, enabling the model to surpass gpt-3.5 accuracy with a much lower parameter count. potential avenues for improving efficient medical question answering include increasing the size of the pre-training dataset and the number of generated questions and investigating retrieval-based answering (open-book exam). a model with significant capabilities in medical answering can aid in making informed decisions, especially in timesensitive situations where rapid response is crucial. such a model can offer up-to-date information, suggest potential diagnoses, and recommend treatment options based on the latest research and clinical guidelines.