DACP : Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization (2025)

Abstract: Large language models (LLMs) have achieved impressive performance in text summarization, yet their performance often falls short when applied to specialized domains that differ from their original pre-training distribution. While fine-tuning can improve summarization quality, it typically relies on costly and scarce high-quality labeled data. In this work, we explore continual pre-training as a scalable, self-supervised approach to adapt LLMs for downstream summarization tasks, particularly in the context of noisy real-world conversation transcripts. We conduct extensive experiments using large-scale, unlabeled business conversation data to investigate whether continual pre-training enhances model capabilities in conversational summarization. Our results demonstrate that continual pre-training yields substantial gains in both in-domain and out-of-domain summarization benchmarks, while maintaining strong generalization and robustness. We also analyze the effects of data selection strategies, providing practical guidelines for applying continual pre-training in summarization-focused industrial applications.

## introduction llms have demonstrated remarkable performance in text summarization, even outperforming humanwritten summaries in various publicly available datasets @xcite . this impressive capability of llms in generating highquality summaries has led to the development of various llm-powered summarization applications for practical use cases @xcite . however, real-world deployment of llms is associated with high inference costs @xcite . therefore, smaller llms 1 are often preferred over their larger counterparts to reduce production costs @xcite . note that, despite the recent advances of llms in text summarization, recent research has found that the performance of llms, especially the cost-effective smaller ones, can drop sharply in downstream summarization tasks when the input differs from the initial data used during their pre-training @xcite . thus, it is important to adapt the smaller llms in the targeted domain before deploying them for real-world inference. although smaller llms can be adapted to downstream tasks related to a certain domain by leveraging techniques like fine-tuning or instruction-tuning @xcite , this process requires the availability of human-annotated data, which can be challenging to obtain @xcite . while this limitation can be addressed by leveraging larger closed-source llms for data annotation, their applicability in real-world scenarios is limited due to the privacy concerns of the customer data and the high cost of manually verifying llm-annotated labels. in this regard, continual pre-training of smaller open-sourced llms on a vast amount of unlabeled internal data in a selfsupervised fashion could be a potential solution for domain adaptation @xcite . to this end, in this paper, we study the continual pre-training in the context of llms on real-world business conversational data. our goal is to apply a data-centric solution and investigate whether they can help improve the performance in downstream summarization tasks related to real-world business conversations (e.g., meeting recaps, call summary and action items generation, etc.). our extensive experiments demonstrate that continual pre-training @xcite helps llms to improve their performance in downstream summarization tasks in the business conversational domain. our major contributions in this paper are summarized below: ## related work existing llms are massively pre-trained on vast amounts of publicly available internet data using the self-supervised next token prediction (ntp) objective @xcite @xcite . however, these public datasets can be significantly different than the proprietary data used in the realworld industrial scenario @xcite . as demonstrated by @xcite , llms often underperform on real-world, domain-specific summarization compared to public benchmarks that reflect their pre-training data . to address this, continual pre-training via leveraging self-supervised learning on internal datasets could be useful to adapt existing llms to a specific domain @xcite , as demonstrated by @xcite @xcite . nonetheless, prior research on continual pre-training of llms is mostly limited to certain domains, such as biomedicine @xcite @xcite or finance @xcite . no prior research has studied the effectiveness of domain adaptation via continual pre-training on noisy conversational data. since utilization of llms on conversational data is on the rise foot_0 for real-world use cases @xcite , it is important to investigate how to effectively utilize vast amounts of unlabeled asr-generated conversation transcripts to successfully adapt llms to downstream tasks related to real-world business conversations. in this paper, we aim to address the gap in the prior research. our focus is to investigate the effectiveness of continual pre-training for domain adaptation by leveraging large amounts of unlabeled business conversations. based on our extensive experiments, we provide our insights on (i) how we select the data for continual pre-training and why we choose a particular strategy, (ii) what pretraining strategy is followed and why, and (iii) how helpful continual pre-training is to adapt llms to various summarization tasks related to business conversations. these findings will help industries working with conversational data to effectively utilize llms for real-world use cases. ## methodology an overview of our methodology is shown in figure 1 . below, we describe the overall process. ## conclusion and future work in this paper, we study how to effectively leverage vast amounts of unlabeled asr-generated transcripts to adapt llms to handle real-world business conversational tasks. based on extensive experiments, we observe that our proposed dacp technique helps llms to adapt effectively across downstream summarization tasks, demonstrating strong generalizability and robustness. this suggests that strategic data curation and processing, focusing on quality and diversity can lead to better model adaptation, a key consideration when dealing with large unlabeled industrial datasets. in the future, we will explore the interplay between the model size and the data size in dacp-style training, alongside developing a new domain-specific benchmark with a broader task selection.