Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Models (2024)

Abstract: Recent surge in the accessibility of large language models (LLMs) to the general population can lead to untrackable use of such models for medical-related recommendations. Language generation via LLMs models has two key problems: firstly, they are prone to hallucination and therefore, for any medical purpose they require scientific and factual grounding; secondly, LLMs pose tremendous challenge to computational resources due to their gigantic model size. In this work, we introduce pRAGe, a Pipeline for Retrieval Augmented Generation and Evaluation of medical paraphrases generation using Small Language Models (SLM). We study the effectiveness of SLMs and the impact of external knowledge base for medical paraphrase generation in French.

## introduction large language models (llms) are used for a big variety of nlp tasks and are found to be very effective, but they exhibit a specific trait that can make them unreliable: hallucinations @xcite . llm hallucinations are incorrect output generations that do not correspond to the input prompt, or are not factual information reflecting world knowledge. llms such as chat-gpt (openai, 2022) became widely used by lay people and the risk of incorrect information spreading within medical text generation persists. in very specialized and sensitive fields such as medicine, hallucinations can have dangerous consequences for the patient via wrong prognosis or treatment recommended by @xcite . this leads to the necessity that the information patients receive to be scientifically and factually grounded, either by human expert or external knowledge base. additionally, recent state-of-the-art (sota) results *these authors contributed equally to this work. ## related work sota large language models such as gpt-3.5 (openai, 2022), gpt-4 @xcite and llama-2 @xcite can give impressive results on question-answering tasks @xcite , or radiology reports simplification @xcite . however, these models are either not opensource (gpt) or may require access to powerful servers. smaller models (<7b) are easier and feasible to implement in downstream tasks, but they tend to hallucinate more compared to llms. in health related applications, we need to retrieve correct information, thus rag systems are essential. different types of rag systems have been developed @xcite , going from the original naive rag (simple structure of a retriever and a generator) @xcite to more advanced or modular rag such as ra-dit @xcite . there are several types of advanced rag architectures such as the self-reflective self-rag @xcite , self-biorag @xcite , black box rag like replug @xcite , pretrained model rag such as realm @xcite , retro @xcite or atlas @xcite . however, these advanced rag systems have very complex implementation architectures and require servers with many gpus. almanac @xcite ) is a rag system developed for the clinical domain , but it is not open source and it is available only for english. we are interested in developing easy to implement rag systems for languages other than english, in our case for french. there are some french language models, such as camembert @xcite for the general domain and drbert @xcite for the medical domain, but they are not adapted for text-to-text generation tasks. multilingual decoder models, such as mixtral, falcon, bloom, and french models such as vigogna (based on vicuna) or claire are available for general language. as for encoder-decoder french models trained for nlg task (more precisely for summarization tasks), we cite barthez (eddine et al., 2020), fine-tuned on bart. many opensource medical llms for nlg tasks are available for english, such as pmc-llama @xcite , meditron @xcite (both based on llama-2), clinicalgpt @xcite , but there is no nlg medical model trained exclusively on the french language. multilingual medical models such as medical-mt5 @xcite and biomistral @xcite are of interest for our research. one important aspect when working with llms on medical text is the quality of the generation in order to minimize the hallucination risk. llms can generate content according to a specific prompt. however, we want to improve the quality of the output and obtain a short paraphrase of the term and not a complete description of the term (as shown in french sentential paraphrase databases are scarce. one such resource for the medical domain is the wikilarge fr and clear, a parallel corpus for text simplification built with translation slm fr encoder-decoder baseslm barthez-orangesum-abstract biomistral-7b-slerp-gptq prage fr encoder fr decoder biomistral drbert biomistral-7b-slerp-gptq prage sent-camembert biomistral-7b-slerp-gptq barthez drbert barthez-orangesum-abstract prage sent-camembert barthez-orangesum-abstract ## methodology we illustrate our method in figure 1 . prage is built on a encoder-retriever-decoder framework. we designed prage to embed medical query and to generate an output in a style that translates medical knowledge for patients in a simpler language, e.g. @xmath0. therefore, we pair models for the general language with medical models. we tested different configurations of non proprietary encoders and decoders, as shown in ## experiments in this section, we describe the experiments we conducted to study the comparison between slms and prage models for generating paraphrases for medical terms. we also describe the fine-grained evaluation process in the subsections below. ## results and discussion we present the summarized automatic evaluation in next, we analyse the results observed from the manual evaluation, presented in while the hallucination @xcite percentage is low for biomistral alone in zeroshot setting (6% for fine-tuned and 4% for vanilla, in the best settings foot_11 ), barthez is not adapted for zero-shot inference, as it is a summarization model. in most analyzed cases, it only summarizes the prompt, as the following example shows: expliquez-moi le terme médical en mots simples : phobie (explain the medical term in simple words: phobia). however, in prage setting, barthez is capable of summarizing the retrieved documents in a coherent answer. nevertheless, its hallucination rate is still higher: 50% for both finetuned and vanilla. this can be explained by the fact that the model is not adapted for q&a tasks, contrary to biomistral. in prage, biomistral had a lower hallucination rate (10% in fine-tuned version and 8% in vanilla version). however, we observed that biomistral alone outperforms its prage counterpart in both finetuning and zero-shot setting (+4% improvement). this results could be explained by the work of @xcite on the impact of parametric an non-parametric knowledge of gpt language models (lms) in vanilla and rag setting depending on fact popularity. the authors' experiments showed that non-parametric memories are more effective for less popular facts (end-tail knowledge) than base lms. however, they also proved that nonparametric memories can mislead lms with less factual information. in our experiments, we hypothesize that the popularity of the medical knowledge to be paraphrased and the relevance of the information retrieved from refomed-kb can explain the lower performance of biomistral in the prage pipeline. further qualitative analysis is needed to test this hypothesis for medical knowledge. short and concise answers. we analyzed if the generated medical paraphrase or explanation is accurate and concise in two settings: a constraint of 25 and 50 tokens of generated text. in a 50 token setting, the human analysis shows that the best model is biomistral accuracy in correct answers in a relaxed and strict setting (96% ; 94%), while the second best is its fine-tuned version (94% ; 90%). however, the model is not as good in strict correctness in a 25 token setting (68%). our finetuned version of biomistral is better (90%, up to 22% increase in performance) on strict correctness and conciseness in a 25 tokens setting. in the prage pipeline, camembert biomistral, both non fine-tuned and fintuned, gave better answers in terms of strict correctness for the 25 token setting (82%; 81%). readability for laypeople. we analyzed how the readability score was influenced by our different setting experiments. in the short answer setting (25 token), the readability is better with base biomistral (1.08, lower is better). however, even if the readability is good, the answers are incomplete (10% completeness-strict). our fine-tuned version of base biomistral improves the completeness of the answer (16% completeness-strict), while the best prage model, camembert biomistral fine-tuned, increases it even further up to 33%. thus, we see a +23% improvement in performance (in completeness-strict) with our fine-tuned model in prage. one aspect that explains the decrease in readability in our fine-tuned models is the higher use of medical terms in the generated answers, as the fine-tuning step with the refomed dataset focuses on medical terms. as the goal of this study was to give short and concise paraphrases to a user query, we see that there are advantages of the fine-tuned model: it generates subsentential paraphrases, thus shorter and complete units of meaning (in the 25 tokens setting). moreover, the fine-tuned model also generates simplified text generations, as observed in the following example where the medical term "osteophyte" is explained by using a subsentential paraphrase in very simple language: "deposits of bone tissue that form on the edges of bones" (original in french ostéophyte - des dépôts de tissu osseux qui se forment sur les bords des os). inter-annotator agreement score. we computed a krippendorff's alpha score @xcite for two criteria of the human evaluation: completeness and readability. the annotation was conducted by 3 french linguists annotators: 1 linguist completed a full annotation and 2 other linguists contributed to the second annotation (one annotated @xmath1. we show the krippendorff's alpha score and the percentage agreement in ## conclusion and future work we presented prage, a pipeline for retrieval, generation and evaluation of medical paraphrases in a patient/lay person oriented downstream application. we showed that finetuning biomistral with the refomed dataset increases the ability of the model to generate short, concise and correct subsentential paraphrases. the prage pipeline helps increase the scientific grounding of text generation via llms for medical domain. our work intends to help bridge the gap between scientific medical knowledge and lay people. further work will include testing another french language models, such as vigogne or claire in similar settings, finetuned and in prage. we are also planning to conduct an extensive annotation campaign with specialists of the medical domain to asses a fair inter-agreement score on correctness and further explore how the generation is affected by the choice of knowledge base used.