Efficient Continual Pre-training for Building Domain Specific Large Language Models (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable open-domain capabilities. LLMs tailored for a domain are typically trained entirely on domain corpus to excel at handling domain-specific tasks. In this work, we explore an alternative strategy of continual pre-training as a means to develop domain-specific LLMs over an existing open-domain LLM. We introduce FinPythia-6.9B , developed through domain-adaptive continual pre-training on the financial domain.Continual pre-trained FinPythia showcases consistent improvements on financial tasks over the original foundational model. We further explore simple but effective data selection strategies for continual pre-training. Our data selection strategies outperform vanilla continual pre-training’s performance with just 10% of corpus size and cost, without any degradation on open-domain standard tasks. Our work proposes an alternative solution to building domain-specific LLMs cost-effectively.

## introduction large language models (llms) have exhibited a profound understanding of natural language, improving performance on various tasks @xcite . open web data helps create general-purpose llms with a broad range of capabilities. general-purpose llms are however not "specialists". for example, while llms could write good news articles, it would be hard-pressed to write specialized legal documents. in order to create a specialist domainspecific llm, they need to be trained on domain data. approaches for building domainspecific llms fall into two categories: training domain-specific llms from scratch or continually pre-training existing llms with domain data. most researchers have taken the first approach of building domain-specific llms from scratch including the med-palm family @xcite for the medical domain and bloomberggpt @xcite for finance. little attention is paid to domainadaptive continual pre-training, despite being a much cheaper alternative. notably, pmc-llama @xcite , a medical llm was trained through continual pre-training of llama @xcite on medical papers. continual pre-training can also be used for updating a llm with the latest knowledge in an evolving environment. in this work, we explore the following: 1) is domain-adaptive continual pre-training helpful in building domain-specific llms?; 2) can we employ data selection strategies for a more effective domain-adaptive continual pretraining?; and 3) does domain-adaptive continual pre-training hurt llm's open-domain capabilities? we answer these questions in the confines of finance domain by training a continually pre-trained model, finpythia, built on top of pythia @xcite . we report a boost on financial benchmarks @xcite after continual pretraining on domain data of size 8% of what pythia was trained on to answer the first question. we see an evidence of latest financial domain knowledge acquisition in fin-pythia during qualitative analysis. to answer the second question, we propose two simple data selection techniques, task-aware efficient task-similar domain-adaptive continual pre-training (ets-dacp) and efficient task-agnostic domain-adaptive continual pretraining (eta-dacp). these methods outperform vanilla domain-adaptive continual pre-training with just 10% of selected domain data or 0.8% of pythia's training corpus. we use three metrics for data selection: similarity, perplexity, and token type entropy. while similarity needs task data as seed data, the latter two metrics are task-agnostic metrics. to answer the third question, we benchmark on four open-domain standard tasks and observe that continually pre-trained llm retains its general capabilities while adapting to the domain. the main contributions of this paper are threefold: ## methodology in this section, we describe the curation of our financial corpus used for continual pre-training, background concepts, and our proposed taskaware domain-adaptive continual pre-training. ## related work domain specific large language models. while the majority of released llms are general-purpose models, domain-specific llms have emerged as valuable counterparts. med-palm, trained on a medical corpus, achieved state-of-the-art results on medical benchmarks @xcite while bloomberg developed a financial llm from scratch on a financial corpus @xcite . continual pre-training presents an alternative approach to building domain-specific llms from scratch. @xcite build medical llms through vanilla continual pre-training llama @xcite on medical papers. continual pre-training of language models. continual pre-training of lms on unlabeled data for a given task has been demonstrated to be beneficial for task performance @xcite . @xcite document that continual pre-training on a similar domain contributes to task performance. notably, the work closest to ours is @xcite , that selects the data based on an importance sampling scheme with weighting using relevance to the target task data distribution. while these works exclusively use task data, we also propose a task agnostic method, eta-dacp, as task similarity is not always feasible. other methods like das @xcite , do not perform a pre-selection of data as they perform the importance sampling on the fly, making it 3x more expensive than vanilla pretraining. further, das uses contrastive learning between two copies of the model elevating memory requirements. in contrast to all these methods which have done experiments on small language models like bert/roberta, we are the first ones to explore data selection for continual pre-training for large language models (llms), to best of our knowledge. ## conclusion in this paper, we demonstrate the efficacy of domain-adaptive continual pre-training for developing domain-specific llms. our results show that domain-adaptive continual pretraining improves the llms' performance on financial tasks and it enables the llms to acquire domain knowledge at a much lower cost. furthermore, we propose efficient domainadaptive continual pre-training methods, ets-dacp and eta-dacp, to enhance continual pre-training. by being selective during the training data curation, our methods refine continual pre-training, yielding even better results with just 10% of the data (cost) of vanilla continual pre-training. entropy based task-agnostic data selection works almost at par with the task-aware data selection strategy. this finding can be used on data selection for continual pre-training even in the absence of a task. we also observe no degradation in performance on open-domain standard tasks, implying that domain continual pre-training does not hurt open-domain capabilities. our findings place domain continual pretraining as a strong alternative to building domain-specific llms from scratch. by being smarter about data selection, we can surpass vanilla continual pre-training at a fraction of the cost. contrary to the wide belief that just adding more data helps, our results suggest that quality of data matters too. while domain continual pre-training has been studied widely in small lm literature, we provide a unique insight in llms given the scale and costs involved. overall, our work paves the way for developing domain-specific llms at a reduced cost, with implications for a wide range of llm applications.