Improving Factuality in Clinical Abstractive Multi-Document Summarization by Guided Continued Pre-training (2024)

Abstract: Factual accuracy is an important property of neural abstractive summarization models, especially in fact-critical domains such as the clinical literature. In this work, we introduce a guided continued pre-training stage for encoder-decoder models that improves their understanding of the factual attributes of documents, which is followed by supervised fine-tuning on summarization. Our approach extends the pre-training recipe of BART to incorporate 3 additional objectives based on PICO spans, which capture the population, intervention, comparison, and outcomes related to a clinical study. Experiments on multi-document summarization in the clinical domain demonstrate that our approach is competitive with prior work, improving the quality and factuality of the summaries and achieving the best-published results in factual accuracy on the MSLR task.

## introduction neural abstractive multi-document summarization (mds) is an active area in natural language processing. it requires comprehension of several input documents by resolving the shared and potentially redundant information among them, and the generation of fluent salient summaries @xcite @xcite . while advances in sequence-to-sequence models have improved the fluency and cohesion in abstractive summarization, the generation of unfaithful or non-factual summaries is still a major issue @xcite . this is critical in many settings like clinical document summarization, where factual accuracy is more valued than other summary qualities @xcite ). an effective method for improving the factuality of summaries is to guide the model using additional guidance signals extracted from the input @xcite . models are then trained to either generate the signals prior to the summary or condition the generation of the signals by prepending them to the source text. signals included entity chains @xcite , keywords and custom prompts @xcite , legal arguments @xcite , and a mixture of human-annotated attributes @xcite . @xcite introduced a generalized framework for guided summarization (gsum) where they use a secondary encoder for the guidance signals, and combine it with the original document encoder in bart @xcite . the decoder then uses both encoders' outputs for generation. however, all these approaches operate at the supervised fine-tuning stage, and are thus constrained by the amount of labeled data available. in addition, they assume fine-tuning will inherently make the model pay higher attention to the factual attributes in input documents, which is neither guaranteed nor sufficient for generating faithful summaries @xcite . in this work, we propose a guided continued pre-training approach for improving the model's understanding of the factual constitution of the documents using guidance signals. our method operates over unlabeled corpora from the target domain, and the model is then fine-tuned directly on the summarization task without additional inputs or attribute highlights. our pre-training objective extends bart @xcite to help the model understand meta-information about the factual attributes of the documents based on pico spans. our main contributions are: 1) we introduce a set of continued pre-training objectives addressing factual attributes; 2) we show that the proposed method outperforms baselines and prior methods improving both the summary quality and factuality, with ablation of the contribution of each objective to the performance gain; and 3) we show that the proposed technique improves the faithfulness of the summaries in few-shot and zero-shot settings. to the time of writing, our method ranks 1 st in the cochrane task leaderboard. our proposed approach consists of 3 steps: (i) pretraining a general-purpose encoder-decoder model ( §2.1), (ii) specializing the model in the target domain using a continued pre-training stage ( §2.2), and (iii) supervised fine-tuning of the domainspecialized model for mds ( §2.3). ## experimental settings dataset. we experiment with allenai's multidocument summarization of literature review (mslr) task. the task consists of two datasets: ms^2 @xcite and @xcite . in both datasets, the inputs are abstracts of clinical papers, and the targets are either the abstract of their corresponding review paper in ms^2, or the author's conclusion in cochrane. we choose this dataset because it provides a quantitative metric for evaluating factual consistency. results are reported on the validation sets using the official task's evaluation script. appendix a reports additional results on the test set. continued pre-training. we sample 360k nonempty english abstracts from the pubmed dataset on huggingface foot_2 on the same clinical domain as the mslr datasets. this is the same number of input abstracts in the ms2 and cochrane training sets combined. pre-training is done for 50 epochs, using a learning rate of 3e-05. the maximum sequence length is set to 2,048 tokens. batch size of 32 was used. we used the default setting of fairseq @xcite for the rest of hyperparameters. fine-tuning. in our setup, input abstracts are grouped by review id and concatenated to a maximum of 2048 tokens. to assess the generalization capabilities achieved by our suggested pretraining method, we experimented with full-shot fine-tuning, few-shot fine-tuning (using 10% of the training data), and zero-shot learning. 4 fine-tuning is done for 20,000 and 5,000 total number of updates for full and few-shot settings, respectively, with a learning rate of 3e-05. we use a dropout of ## conclusions we propose a continued pre-training stage that combines several objectives designed to improve the understanding of clinical documents. the resulting model improves the factual accuracy of summaries, even in few-and zero-shot settings. with limited resources, our system based on bart-large out-performs models of the same scale and achieves competitive results with larger ones. all in all, our work demonstrates that it is possible to adapt a general purpose model to a specific task and domain without any labeled data by continuing pre-training with a carefully designed objective.