How do you know that? Teaching Generative Language Models to Reference Answers to Biomedical Questions (2024)

Abstract: Large language models (LLMs) have recently become the leading source of answers for users’ questions online. Despite their ability to offer eloquent answers, their accuracy and reliability can pose a significant challenge. This is especially true for sensitive domains such as biomedicine, where there is a higher need for factually correct answers. This paper introduces a biomedical retrieval-augmented generation (RAG) system designed to enhance the reliability of generated responses. The system is based on a fine-tuned LLM for the referenced question-answering, where retrieved relevant abstracts from PubMed are passed to LLM’s context as input through a prompt. Its output is an answer based on PubMed abstracts, where each statement is referenced accordingly, allowing the users to verify the answer. Our retrieval system achieves an absolute improvement of 23% compared to the PubMed search engine. Based on the manual evaluation on a small sample, our fine-tuned LLM component achieves comparable results to GPT-4 Turbo in referencing relevant abstracts. We make the dataset used to fine-tune the models and the fine-tuned models based on Mistral-7B-instruct-v0.1 and v0.2 publicly available.

## introduction the idea of automated referencing dates back to 1970 when @xcite ) proposed an automatic system where a computer evaluates the appropriateness of references within an article. with the emergence of generative large language models (llms), numerous systems are being developed to answer specific questions, supported by relevant references @xcite @xcite . generative llms can produce answers that appear coherent, confident and articulate. however, the information conveyed may not be correct or verifiable. furthermore, the limited internal knowledge of generative llms can hinder their ability to deliver factually accurate answers, particularly within specialized fields @xcite . this issue is notably concerning in the biomedical domain, where accurate and factual answers are critical. the scientific community has recognized the dangers of factually incorrect or nonsensical information and has been reluctant to utilize these models to their potential. providing an opportunity for scientists to obtain correct and verifiable answers to questions is an opportunity to increase scientific productivity and its impact. moreover, privacy, sovereignty and security concerns in pharma and biomedicine often necessitate building systems where all components are controllable (e.g. deployed in-house), to avoid reliance on third-party apis such as openai foot_0 , especially when secret data is concerned. incorporating domain-specific external knowledge beyond llm data is essential for mitigating hallucinations in llms. the retrieval-augmented generation (rag) approach, which integrates the generative capabilities of an llm with a specialized retrieval system, enhances the model's accuracy and relevance by grounding its responses in verified information. in this paper, we present a biomedical rag system consisting of a hybrid search based on pubmed foot_1 and fine-tuned generative models for referenced question-answering (qa). we make both the models and the dataset used to fine-tune the models publicly available. the remainder of this paper is organized as follows: section 2 provides a review of related work on reliability and verifiability of the llm generated content and the approaches to generating texts with references. section 3 describes the design of the ir and generative components. we evaluate the components in section 4, first individually and then jointly. we end the paper with conclusions and some future work remarks in section 5. ## related work generative llms, such as gpt and similar architectures, have enabled question-answering (qa) tasks across various domains, including medicine. the current state of these models is characterized by several challenges, particularly regarding the verifiability and reliability of the information they generate. by evaluating chatgpt responses and references in the medical domain, @xcite found that 69% of generated references were fabricated, while professionals rated the answers at a median quality of 60%. similarly, when @xcite conducted manual evaluations of four prominent generative search engines bing chat, neevaai, perplexity.ai, and youchat, they found that while the responses of these engines were fluent and seemingly informative, only 51.5% of sentences generated by these engines were fully supported by their citations, and merely 74.5% of citations accurately supported the statements they were linked to. these results leave space for improvement. in general, there are two approaches to generating text with references @xcite . the first assumes training llms to produce references from parametric knowledge (information internalized from the training data). the second one assumes producing references from non-parametric knowledge (content retrieved from external sources). the first approach, integrating citations directly from llm's parametric knowledge, poses a significant technical challenge. unlike search engines and ir systems that rely on indices for data retrieval, llms encode information into hidden representations during training, lacking a direct index. therefore, referencing the sources of information becomes intricate. despite these challenges, approaches have been suggested to train llms to include references using source identifiers @xcite . however, these methods exhibit certain limitations, including citation inaccuracies and being restricted to academic citations. the second approach, known as retrievalaugmented generation (rag), combines generative llms with ir systems to form a hybrid system @xcite . here, the model is trained to recognize instances requiring citations, and the ir system retrieves suitable sources to provide context to the llm. as a result, the llm incorporates these sources as citations into its outputs, improving the credibility and accuracy of responses. while pre-trained and fine-tuned llms rely solely on their parametric knowledge, rag integrates a customized external knowledge base without additional training, thus reducing hallucinations. moreover, annotators often perceive rag-enhanced answers to be more factual and specific compared to those from fine-tuned models @xcite . ## method the rag system we propose in this paper is designed to perform referenced qa in the biomedical domain. it consists of two main components. the ir component, based on hybrid semantic and lexical search, retrieves relevant pubmed abstracts and provides a context for the generative llm. the final system output is an answer to the user query, which contains a reference for each of the claims extracted from the relevant abstracts. the overview of the system architecture can be seen in figure 1 . ## conclusions and future work in this paper, we provide an overview of biomedical generative search with answers grounded in pubmed and referenced claims. our aim was to develop a system capable of generating accurate and verifiable answers to biomedical questions while maintaining user sovereignty and leveraging opensource models. starting with our ir component, we discovered that employing a combination of lexical and semantic searches yields the highest precision score. our system demonstrates an absolute improvement of 23.4% map@10 measure compared to the pubmed search engine. through separate evaluations, we found that lexical search alone outperforms semantic search. however, integrating both approaches is advantageous for identifying instances lacking exact term matches, where semantic search contributes significantly. to enhance semantic search performance in ir, one future direction is to fine-tune these models on domainspecific data. this approach aims to improve the quality of embeddings in the biomedical domain, enabling them to encode domain-specific knowledge better, enhance contextual understanding, and ultimately improve ir performance. overall, the mistral 7b instruct models performed comparatively to gpt-4 turbo in terms of the task of referenced qa. based on the evaluation of the whole pqaref test set, m1 and m2 showed superior performance over 0-m1 and 0-m2 in referencing the most relevant abstracts, with m2 showing an improved performance of 2.3% over m1, 21.3% over 0-m2 and 59.2% over 0-m1. as a general trend, m2 includes more information in its answers. all four models showed hallucinations when generating ids of references. once again, m2 performed best in this respect with only 3 mismatches in id digits, followed by 0-m1 (11) and 0-m2 (26), with the worst performance of 79 hallucinated answers coming from m1. while m2 was still using correct information from the corresponding abstract, this point needs further attention. exchanging the ids with numerals (1-10) for each abstract during fine-tuning could potentially solve this issue. this is something we plan to try in the next iteration of the dataset and training. in terms of recall values for relevant abstracts, based on the manual evaluation of 10 examples from pqaref test set both fine-tuned models performed better, exhibiting a 47% and 5% improvement over their versions in zero-shot mode. the situation is slightly different for the same 10 questions with abstracts retrieved using our ir. while recall values of m1 were still superior, especially compared to 0-m1 with 27% improvement, 0-m2 performed slightly better than m2 (1% difference). however, considering the number of hallucinations and answers with no references, we give clear preference to the fine-tuned models. it is worth noting that on a small test set, we have achieved a comparable and sometimes even better performance than gpt-4 turbo with much smaller, open-source and fine-tuned models. having an automated evaluation method of referenced qa quality would allow us to make a more comprehensive comparison of our models to models such as gpt-4 turbo. to supplement manual evaluation and speed up the evaluation process, @xcite created a benchmark for automatic evaluation of citation quality and proposed an entailment model as a method for automatic quality check of references. we intend to further check the quality of answers on a larger sample using the same method. at this point, m2 gives more reliable answers, but the decision on which model to use in our final pipeline will be made after this evaluation.