Evaluation of LLM s in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings (2025)

Abstract: Large Language Models (LLMs) recently achieved great success in medical text summarization by simply using in-context learning. However, these recent efforts do not perform fine-grained evaluations under difficult settings where LLMs might fail. They typically report performance scores over the entire dataset. Through our benchmarking study, we show that LLMs show a significant performance drop for data points with high concentration of out-of-vocabulary (OOV) words or with high novelty. Vocabulary adaptation is an intuitive solution to this vocabulary mismatch issue where the LLM vocabulary gets updated with certain expert domain (here, medical) words or subwords. An interesting finding from our study is that Llama-3.1, even with a vocabulary size of around 128K tokens, still faces _over-fragmentation_ issue with medical words. To that end, we show vocabulary adaptation helps improve the LLM summarization performance even in difficult settings. Through extensive experimentation of multiple vocabulary adaptation strategies, two continual pretraining strategies, and three benchmark medical summarization datasets, we gain valuable insights into the role of vocabulary adaptation strategies for customizing LLMs to the medical domain. We also performed a human evaluation study with medical experts where they found that vocabulary adaptation results in more relevant and faithful summaries. Our codebase is made publicly available at https://github.com/gb-kgp/LLM-MedicalSummarization-Benchmark.

## introduction recent works like clinsumm @xcite explore various strategies to adapt llms in the task of medical text summarization. these strategies are (i) in-context learning @xcite where a fixed number of exemplars is added in the prompt at infer-* corresponding author ## benchmarking vocabulary adaptation strategies for llms a vocabulary adaptation technique consists of three steps: (i) generating candidate vocabulary tokens from target downstream datasets, (ii) selecting important vocabulary tokens from candidate set using some utility function (e.g., fragment score @xcite , corpus entropy @xcite ) to form added vocabulary, and (iii) learning the embeddings of the added vocabulary and integration into llm. in this work, we benchmark the effect of all the three steps, combining the effect of first two steps: constructing the vocabulary and adding candidate tokens from the target domain to finalize the vocabulary for integration into the llm as one, and then checking effect of last step: on how to train the embeddings. ## experimental setup we use in-context learning (icl) @xcite along with greedy decoding for generating summaries from llms. we follow the approach similar to clin-summ @xcite , where examples are sampled from the train set using the similarity computed using pubmedbert foot_2 with the given test data point. the template for prompting is shown in appendix a ( next, we describe the benchmark medical text summarization datasets used, followed by details on evaluation metrics, baseline models, and training details. ## experimental results ## conclusion this work is a first step towards understanding how vocabulary adaptation strategies effect the performance of general-purpose llms in medical text summarization. we first show that general-purpose llms fail in certain challenging generation scenarios where reference summaries have high oov concentration and high novelty. we then benchmark the performance of three vocabulary adaptation strategies on four models: llama-2 7b, mistral 7b, qwen2 7b, and llama-3.1 8b model; over three biomedical summarization datasets and two continual pretraining strategies. llama-3.1 and qwen2 even with a vocabulary size of 128k and 151k tokens, still faces the over-fragmentation issue with medical words, and vocabulary adaptation is shown to help improve the llm summarization performance. medical experts find that vocabulary adaptation improves the relevance, coherence, and faithfulness of medical summaries. we make the codebase available for reproducibility purposes foot_3 .