Aligning Factual Consistency for Clinical Studies Summarization through Reinforcement Learning (2023)

Abstract: In the rapidly evolving landscape of medical research, accurate and concise summarization of clinical studies is crucial to support evidence-based practice. This paper presents a novel approach to clinical studies summarization, leveraging reinforcement learning to enhance factual consistency and align with human annotator preferences. Our work focuses on two tasks: Conclusion Generation and Review Generation. We train a CONFIT summarization model that outperforms GPT-3 and previous state-of-the-art models on the same datasets and collects expert and crowd-worker annotations to evaluate the quality and factual consistency of the generated summaries. These annotations enable us to measure the correlation of various automatic metrics, including modern factual evaluation metrics like QAFactEval, with human-assessed factual consistency. By employing top-correlated metrics as objectives for a reinforcement learning model, we demonstrate improved factuality in generated summaries that are preferred by human annotators.

## introduction recently, the exponential growth of medical literature, specifically in the realm of clinical studies such as randomized controlled trials (rcts), has underscored the necessity for efficient summarization techniques @xcite @xcite @xcite . clinicians and researchers face the arduous task of sifting through vast amounts of information daily to remain abreast of the latest findings and advancements in their respective fields @xcite . summarizing clinical studies enables healthcare professionals to access crucial information more rapidly, ensuring that their decisions and treatment plans are informed by the most recent, evidence-based knowledge. as a result, the development of effective and accurate summa-rization techniques for clinical studies has become an essential area of research in the medical domain @xcite @xcite @xcite @xcite . automatic summarization of clinical studies is fundamental for systems that aim to interpret the vast array of available medical literature @xcite @xcite . randomized controlled trials (rcts) are considered the gold standard of clinical evidence among various study types, including cohort studies, observational studies, and case studies @xcite . the ability to efficiently process and summarize the massive volume of rcts holds great potential for enhancing clinical decision-making @xcite . to delve deeper into clinical study summarization, we simultaneously explore single-document and multi-document summarization techniques. for single-document summarization, we propose an rct conclusion generation task based on the pubmed 200k rct sentence classification dataset @xcite . we utilize the pubmed rct200k dataset @xcite , with original annotations for concluding sentences, meaning our summarization system's objective is to generate concluding sentences for a clinical study. in the case of multi-document summarization, we examine the challenge of automatically generating a narrative biomedical summary from multiple trial reports. here inputs are titles and abstracts from systematic reviews previously conducted by members of the cochrane collaboration 1 (wallace et al., 2021), using the review abstract as our target, shown as figure . 1. ensuring the factual consistency of summaries is vital in the medical field, as they must precisely preliminary studies suggest the benefit of blood alkalinization with nahco3 in op poisoning, but there is insufficient evidence to support its routine clinical use. further research is required to determine the method of alkalinization that will optimize outcomes and the regimen which will produce the target arterial ph of 7.50 (range 7.45 to 7.55). conclusion: "antithrombin administration in patients with low at activity after surgery with cardiopulmonary bypass reduces postoperative thrombin generation and fibrinolysis with no effects on platelet activation and inflammatory response." figure 1 : the diagram provides an overview of our tasks, which include single-document summarization and multi-document summarization. for the single-document summarization task, the input consists of a title and an abstract of a clinical study, with the goal being to generate concluding sentences. on the other hand, the input for the multi-document summarization task consists of the titles and abstracts from a corresponding review. highlighted in red are various specialized medical concepts, logical reasoning, and numerical understanding, which introduce new challenges for clinical study summarization. in our approach, we utilize the top-correlated metrics from the previous experiment as the objective for a reinforcement learning (rl) model, like previous work @xcite . by doing so, we aim to guide the model toward generating more factually consistent summaries. our results show that the rl-based models exhibit improved factuality and are preferred by human annotators, demonstrating the effectiveness of using rl for enhancing factual consistency in clinical study summarization systems. our main contributions: we emphasize our focus on clinical studies and discuss the unique challenges associated with their summarization. our experiments feature comprehensive benchmarks and modern factual evaluation metrics, such as qafacteval @xcite . we gathered annotations from both crowd workers and domain experts to assess the factual correctness of summaries generated by state-of-the-art models. by utilizing the top-correlated metrics as the objective for a reinforcement learning (rl) model, our results demonstrate improved factuality that is preferred by human annotators, showcasing the effectiveness of our approach. ## methodology we have observed that dae, questeval, and qafacteval exhibit high correlations with factual consistency across the two datasets. therefore, our goal is to incorporate dae and qafacteval as reinforcement learning objectives to enhance the performance of the base model in text summarization. to achieve this, we can augment the base model's loss function with qafacteval using reinforcement learning, specifically the policy gradient approach. consider the base model's loss function l(θ), where θ represents the model's parameters. the reinforcement learning objective, e.g. using qafacteval, is to maximize the reward function j(θ) for each trajectory τ under the policy θ. the reward function can be defined as: here, r(τ ) denotes the qafacteval reward for a given trajectory τ , and p θ (τ ) represents the probability of the trajectory under the policy θ. to incorporate the qafacteval reward into the base model's loss function, we can modify the original loss function l(θ) as follows: in this new loss function l (θ), α is a hyperparameter that balances the contribution of the original loss function and the reinforcement learning objective. by optimizing this new loss function, the base model can generate summaries that better align with the qafacteval metric. to approximate the expected reward j t (θ) at each time step using a single sample, we can employ the following monte carlo estimation: here, r i,t denotes the qafacteval reward for a single trajectory τ i at time step t, and m is the number of samples (trajectories) used to approximate the expected reward. using this approximation, we can update the bart model's loss function at each time step as follows: was used, a batch size of 32, and gpt-3 we use openai api. and here we have beam search for decoding with beam size 3. we use their original code base for all metrics foot_1 . ## future work we find that gpt-3-generated summaries, though scoring poorly on automatic evaluation metrics, were considered superior in quality through manual evaluation. future studies could focus on this to offer clearer insights and establish a stronger basis for this assertion. this paper also motivates a potential avenue for improvement, suggesting a shift from traditional automatic metrics towards utilizing advanced models like gpt-4 as evaluators. the premise behind this is to test the correlation between the scores assigned by gpt-4 and human evaluators. if a high correlation exists (which we hypothesize might be the case), it would be feasible to employ gpt-4 scores to guide the tuning process in reinforcement learning. this approach essentially distills the capabilities of gpt-4, leveraging its advanced understanding and evaluative capacity to enhance the performance and efficiency of the summarization models. this proposition could herald a novel direction in the evaluation and tuning of such models, potentially offering more reliable and nuanced performance metrics. ## conclusion in the field of clinical studies summarization, there has been limited research on factual consistency so far. we have demonstrated that, for singledocument and multi-document summarization of clinical studies, there are two main issues: 1) the accuracy of automatic metrics for evaluating factual consistency is limited, and 2) existing models have their limitations. we employed human evaluation for assessing factual consistency, and this analysis has been conducted over a larger set of automatic metrics to provide a more comprehensive picture. furthermore, we demonstrate that further optimizing the model using reinforcement learn-ing (rl) with the metric as a reward can result in significant improvements in factual consistency. our contributions include a simple yet effective approach for two medical summarization tasks, validation of several automatic evaluation metrics for their correlation with expert-assessed factualness, and the identification of the best-correlating metric to guide generation models toward enhanced summary correctness. this work lays the foundation for the development of more robust clinical trial summarization systems, facilitating the efficient dissemination of medical knowledge to practitioners and researchers.