SYNFAC - EDIT : Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization (2024)

Abstract: Large Language Models (LLMs) such as GPT & Llama have demonstrated significant achievements in summarization tasks but struggle with factual inaccuracies, a critical issue in clinical NLP applications where errors could lead to serious consequences. To counter the high costs and limited availability of expert-annotated data for factual alignment, this study introduces an innovative pipeline that utilizes >100B parameter GPT variants like GPT-3.5 & GPT-4 to act as synthetic experts to generate high-quality synthetics feedback aimed at enhancing factual consistency in clinical note summarization. Our research primarily focuses on edit feedback generated by these synthetic feedback experts without additional human annotations, mirroring and optimizing the practical scenario in which medical professionals refine AI system outputs. Although such 100B+ parameter GPT variants have proven to demonstrate expertise in various clinical NLP tasks, such as the Medical Licensing Examination, there is scant research on their capacity to act as synthetic feedback experts and deliver expert-level edit feedback for improving the generation quality of weaker (<10B parameter) LLMs like GPT-2 (1.5B) & Llama 2 (7B) in clinical domain. So in this work, we leverage 100B+ GPT variants to act as synthetic feedback experts offering expert-level edit feedback, that is used to reduce hallucinations and align weaker (<10B parameter) LLMs with medical facts using two distinct alignment algorithms (DPO & SALT), endeavoring to narrow the divide between AI-generated content and factual accuracy. This highlights the substantial potential of LLM-based synthetic edits in enhancing the alignment of clinical factuality.

## introduction the advent of generative artificial intelligence (ai) has been markedly accelerated by the development of large language models (llms) such as gpt-3 @xcite , @xcite , llama @xcite . these models have demonstrated superior capabilities in natural language understanding and natural language generation, outperforming language model predecessors (lms) like t5 @xcite and @xcite in a variety of linguistic tasks. despite these advancements, llms confront significant challenges, primarily their propensity for generating hallucinations-fabricated information not grounded in source text-and producing factually inconsistent outputs @xcite @xcite . such limitations critically undermine the models' reliability, particularly critical in clinical nlp applications, where inaccuracies could result in serious misdiagnoses. the nlp community has discussed many reasons for the hallucination problem, including some limitations stemming from traditional supervised fine-tuning (sft). sft fails to differentiate between significant errors, such as hallucinations, and minor inaccuracies, like grammatical mistakes, treating all errors equally in their loss calculations. moreover, sft applies a uniform loss weighting across all data, regardless of its type, quality, or complexity, potentially diluting the training signal for more critical learning objectives. in response to these limitations, recent research has explored learning paradigms incorporating human feedback, such as rlhf @xcite @xcite , rlaif @xcite , rrhf (yuan et al., 2023), and raft (teed and deng, 2020). techniques including ppo @xcite , dpo @xcite , and salt @xcite have demonstrated effectiveness in aligning these hallucination-prone models. however, these alignment methods require substantial amounts of human-annotated data to illustrate human preferences during training, which can be challenging to obtain in clinical domains @xcite @xcite . in this work, we mainly focus on previously less-studied edit feedback data to better align lms (gpt-2 (1.5b)) & llms (llama-2 (7b)) to generate factually correct clinical note summaries. recent works @xcite @xcite discussed some limitations of current common feedback types (comparison or rating feedback) and the advantages of adding edit feedback for better human alignment. human edits are a more natural way to collect feedback from clinicians as they fix ai-generated text for their workflow to improve generation @xcite . collecting other forms of feedback that are not directly tied to the clinician's workflow will not scale as much, this is especially true in domains requiring expert domain knowledge and with nuanced user goals. considering the cost, time, and availability of the experts, it is important to collect edit feedback from the expert's daily workflow. however, it is challenging to collect real-world clinician's edit feedback due to privacy protection and strict data regulations like hipaa @xcite . generating a synthetic imitation edit feedback dataset by leveraging large (>100b parameters) gpt variants like gpt-3.5 & gpt-4 @xcite @xcite ) is one potential solution 2 . such dataset can then be used for alignment training. although these large gpt variants have reached expert-level performance in many clinical nlp tasks (e.g., medical licensing examination) @xcite @xcite , there is not much previous work discussing whether they can generate expert-level edit feedback for lms and llms in the clinical nlp tasks. therefore, we propose to use these large gpt variants as synthetic experts for generating high-quality edit feedback for finetuning llms and lms using the recent sota alignment methods like dpo @xcite & salt @xcite for improving factuality in the clinical domain for the clinical note summarization task. specifically, we propose a new pipeline to generate synthetic preference-based data in two directions for alignment training: 1) @xmath0: where we use these large gpt variants as synthetic experts to add factual hallucinations to generate factuality-based low-quality dispreferred summaries given the original factuality-based highquality preferred ground-truth summaries & the corresponding clinical notes. 2) @xmath1: where we use these large gpt variants as synthetic experts to add factual information to generate factualitybased high-quality preferred summaries given the factuality-based low-quality dispreferred unaligned model generated summaries and the corresponding clinical notes. we then treat the high-quality summaries as the preferred ones and the low-quality summaries as the dispreferred ones used in our synthetic preference data pairs. our experiments demonstrate the efficacy of utilizing synthetic edit feedback to enhance the factual accuracy of model-generated summaries. specifically, for llama2 (7b), we observed a 2.44% ↑ in rougel and a 1.35% ↑ in factuality using the dpo. similarly, salt resulted in a 2.47% ↑ in rougel and a 2.04% ↑ in factuality. for gpt-2, dpo led to a 3.04% ↑ in rougel and a 2.93% ↑ in factuality, while salt yielded a 4.04% ↑ in rougel and a 4.64% ↑ in factuality. moreover, our topperforming model garnered a 78% preference rate for factuality among human evaluators, highlighting its superior performance. ## problem statement given an available dataset d : {x, y } of c clinical notes x : {x 1 , x 2 , ...x c }, their corresponding ground truth reference discharge summaries y : {y 1 , y 2 , ...y c }, and a reference model π ref , the aim of the clinical note summarization task t is to train the model π ref (y i |x i ). here the i th clinical note x i :{x i 1 , x i 2 , ...x i n } consists of n tokens (j th token represented by x i j ) and the i th reference summary y i : {y i 1 , y i 2 , ...y i m } consists of m tokens (j th token represented by y i j & @xmath2. the standard way to fine-tune π ref on t is to simply fine-tune π ref using the cross-entropy loss over the original training dataset d, as shown in figure 1(a) . aligning π ref using alignment training requires the need for preference-based data d pref : {x, y w , y l }, where y w is a set of preferred summaries, and y l are the dispreferred ones. such preference-based data is usually gathered through human annotation or is generated synthetically. as previously explained, not only gathering human annotations is expensive in the clinical domain, but even generating synthetic data using standard approaches like corruption @xcite can be challenging. so in this work, we (1) propose a new pipeline to generate high-quality synthetic preference data d pref : {x, y w , y l }; (2) use d pref to align π ref to generate factually consistent outputs using alignment methods like dpo training & salt loss. in the following subsections, we describe the synthetic edit-based preference data generation pipeline and the edit feedback-based alignment training method in detail. algorithm 1: synthetic preference data generation (@xmath3. dataset: d:{x, y } clinical note articles: x:{x 1 , ..., x c } reference summaries: y :{y 1 , ..., y c } small lm: πsm function get_d @xmath4,f @xmath5 ## synthetic imitation edit feedback for summarization alignment, the model learns from the preference data pairs (y w ,y l )) in d pref by learning to increase the likelihood of y w and to decrease the likelihood of y l . usually, y w is easy to get from the ground truth labels (reference summaries) in d, but on the other hand, y l is usually not readily available. heuristic-based data augmentation functions have been previously explored to tackle this problem in low-resource settings @xcite . in this work, we propose to use llms as synthetic experts (specifically acting as edit functions (f e ) imitating as domain experts) to synthetically generate d pref : {x, y w , y l } in two directions, i.e., (1) @xmath6. the synthetic preference data generation procedure for the above-mentioned two directions is explained in detail in the following two sections. ## related work recent studies have demonstrated the efficacy of llms in enhancing data augmentation processes @xcite @xcite ## conclusion this study leverages synthetic edit feedback to improve factual accuracy in clinical summarization using dpo and salt techniques. our approach demonstrates the effectiveness of gpt-generated edits in enhancing the reliability of clinical nlp applications.