CUED at P rob S um 2023: Hierarchical Ensemble of Summarization Models (2023)

Abstract: In this paper, we consider the challenge of summarizing patients medical progress notes in a limited data setting. For the Problem List Summarization (shared task 1A) at the BioNLP Workshop 2023, we demonstrate that ClinicalT5 fine-tuned to 765 medical clinic notes outperforms other extractive, abstractive and zero-shot baselines, yielding reasonable baseline systems for medical note summarization. Further, we introduce Hierarchical Ensemble of Summarization Models (HESM), consisting of token-level ensembles of diverse fine-tuned ClinicalT5 models, followed by Minimum Bayes Risk (MBR) decoding. Our HESM approach lead to a considerable summarization performance boost, and when evaluated on held-out challenge data achieved a ROUGE-L of 32.77, which was the best-performing system at the top of the shared task leaderboard.

## introduction summarization is a common natural language generation (nlg) task with growing recent interest @xcite . the 1a shared task of bionlp 2023 considers medical problem list summarization @xcite , where patient notes are summarized to assist medical diagnosis applications. there are several challenges faced in designing systems for this task: first, the challenge is low-resource, with only 765 examples available for training/validation. second, a high-stake application with specialized medical terms requires systems that can deal with domain-specific terms and find relevant diagnoses from patient documents. this paper introduces hierarchical ensemble of summarization models (hesm), an approach that is composed of two sequential ensembling layers, 1 our code is available at https://github.com/ potsawee/hierarchical_ensemble_summ . of token-level ensembles, followed by minimum bayes risk (mbr) decoding, as shown in figure 1 . ensembling methods combine the predictions of various models @xcite and have been effective in nlg tasks, such as in summarization @xcite . for low-resource settings, this method allows outputs to be composed from multiple different ensemble members, which can reduce the influence of noise and eliminate spurious signals, reducing the chance of medically inaccurate summaries. we demonstrate that using hesm with clinical-t5 models @xcite leads to systems that have a good grasp of medical knowledge, and that are able to generate outputs that are consistently closer to ground-truth summaries. our proposed hesm method was submitted to the bionlp problem list summarization challenge, and achieved the top position of the shared task leaderboard, out of 9 teams. ## hierarchical ensemble of summarization models when working with a small dataset, individual models are prone to overfit specific aspects of the data due to the limited number of training samples. by training multiple models on the same dataset, each model can potentially capture different aspects of the data that are generalizable and not prone to overfitting. combining these diverse models together can then create a more robust and accurate prediction model @xcite . various approaches can be used to create diverse individual systems. a simple approach is to use different weights' initialization @xcite for different seeds. alternatively for pre-trained systems (as considered in this work), one can set different random seeds, which will influence training dropout and stochastic gradient descent batch creation, resulting in variability in the final models' weights. one can also use a form of data bagging, where a different subset of the data is used to train each model @xcite . for example for the clinical notes, one model can be trained using only the assessment section of the notes, whilst another can be trained using the assessment+subjective sections. given an ensemble of diverse models, one may then combine them for a more robust ensemble system. a possible model combination method is weight averaging. although weight averaging across training runs has shown success in image classification @xcite , weight averaging across different training runs is expected to work only when individual runs operate in similar weight spaces. this limits the types of combinations for weight averaging to only models with the same architecture and the same input format. as a result, we focus instead on two other methods of combination: token-level ensembling and minimum bayes risk decoding @xcite . ## conclusions in low-resource and medical-domain summarization, our work has demonstrated that abstractive summarization outperforms extractive and zeroshot methods. furthermore, both token-level ensemble and mbr decoding improve the overall performance. our hesm, which utilizes both ensembling techniques, achieves state-of-the-art performance with the highest rouge-l score in the bionlp 2023's shared task 1a leaderboard.