R e MAG - KR : Retrieval and Medically Assisted Generation with Knowledge Reduction for Medical Question Answering (2024)

Abstract: Large Language Models (LLMs) have significant potential for facilitating intelligent end-user applications in healthcare. However, hallucinations remain an inherent problem with LLMs, making it crucial to address this issue with extensive medical knowledge and data. In this work, we propose a Retrieve-and-Medically-Augmented-Generation with Knowledge Reduction (ReMAG-KR) pipeline, employing a carefully curated knowledge base using cross-encoder re-ranking strategies. The pipeline is tested on medical MCQ-based QA datasets as well as general QA datasets. It was observed that when the knowledge base is reduced, the model’s performance decreases by 2-8%, while the inference time improves by 47%.

## methodology the model m utilizes the query q and relevant document d to produce a predicted answer (as per eq. in the keyword extraction phase, at least three keywords are extracted from all queries. for each query q, keybert is used to extract at least three related medical keywords or key phrases k q . this ensures that individual words and phrases relevant 121 to the medical context are captured accurately. fol-122 lowing this, in the knowledge base indexing and 123 storage phase, the pubmed api is employed to re-124 trieve 15 relevant articles for each identified keyword 125 or keyphrase. this retrieval process results in a sub-126 stantial corpus of about 600,000 articles, providing 127 a focused subset of the extensive pubmed database 128 (canese and weis, 2013), consisting of 24.9 million 129 articles. the collected articles are then transformed 130 into embedding vectors through the baai embed-131 2 63 ding model (zhang et al., 2024). this model converts text data into a format that can be efficiently processed for similarity searches. finally, the faiss (facebook ai similarity search) vectorstore index is used to store these embedding vectors. faiss is optimized for high-speed similarity searches on large datasets, making it suitable for handling the extensive medical corpus generated. for facilitating retrieval, medcpt was used in the process of document retrieval. for this, we compute the cosine similarity score between the query embedding q and each document embedding d i . the top k documents were chosen based on query similarity. medcpt was utilized to streamline the retrieval process and ensure that the most relevant documents were retrieved. for this, the cosine similarity score was computed between the query embedding q and the embedding of each document d i . the cos-sim score measures how similar two vectors are in orientation and magnitude, with a higher score indicating greater similarity. by calculating this score for each document in the database, the system can effectively rank them based on their relevance to the query. using the computed scores, the top k documents were selected for retrieval, ensuring that the retrieved documents are closely aligned with the user's query. the cross-encoder re-ranker medcpt is advantageous in re-ranking the top k extracted articles for generating the top n articles (where @xmath0, enhancing the relevance of the information produced. med-cpt was chosen as retriever and re-ranker due to its first-stage dense retriever (medcpt retriever) and the second-stage re-ranker (medcpt re-ranker). the medcpt retriever contains a query encoder (qenc) and an article encoder (denc), both initialized by pubmedbert. it is trained on 255m query-article pairs from pubmed search logs and in-batch negatives. the medcpt re-ranker is a transformer crossencoder (crossenc) initialized by pubmedbert. it is trained on 18m semantic query-article pairs and localized negatives derived from the pre-trained med-cpt retriever. upon re-ranking retrieved articles, they are merged with the original query and provided as input to the llm, which generates a response, represented as a, based on the amalgamation of the query and the re-ranked articles. the generated response is then evaluated by comparing it with the ground-truth answers, serving as a metric for assessing the perfor-180 mance of the llm in understanding and responding 181 to the given query. we have utilized two specific 182 llms for our experiments, namely llama-2 and 183 llama-3. these models have been selected based 184 on their capabilities and suitability for the task at 185 hand. we aim to evaluate these llms' effectiveness 186 in generating accurate responses when presented 187 with queries and relevant document contexts. 2024 ) was considered for subjective tasks due to its 206 novelty and outstanding scores.