Multilevel Analysis of Biomedical Domain Adaptation of Llama 2: What Matters the Most? A Case Study (2024)

Abstract: Domain adaptation of Large Language Models (LLMs) leads to models better suited for a particular domain by capturing patterns from domain text which leads to improvements in downstream tasks. To the naked eye, these improvements are visible; however, the patterns are not so. How can we know which patterns and how much they contribute to changes in downstream scores? Through a Multilevel Analysis we discover and quantify the effect of text patterns on downstream scores of domain-adapted Llama 2 for the task of sentence similarity (BIOSSES dataset). We show that text patterns from PubMed abstracts such as clear writing and simplicity, as well as the amount of biomedical information, are the key for improving downstream scores. Also, we show how another factor not usually quantified contributes equally to downstream scores: choice of hyperparameters for both domain adaptation and fine-tuning.

## introduction domain adaptive pretraining (dapt) is an effective method to adapt a model to a particular domain via continual pretraining @xcite , with biobert @xcite being a successful case in the biomedical domain. from our side, we have widely used dapt not only to adapt llms to biomedicine, but as a part of a bigger pipeline to create customeroriented, intelligent agents which can faithfully recover domain knowledge from both their parameters and external databases. however, domain adapting llama 2 @xcite brought us a puzzle: huge variability on downstream scores. given both the size of llama 2 and gpu memory restrictions, the domain adaptation of llama 2 was restricted to a sample (subset) of pubmed abstracts. after domain-adapting and fine-tuning llama 2 on pubmed abstracts and on the biosses dataset @xcite , respectively, we obtained highly different downstream scores depending on the choice of the sample used for dapt, as displayed in fig. 1 . this figure shows a huge variability in pearson (downstream) scores: from 67 points up to an almost perfect score of 98 points. surely, hyperparameter choice for both dapt and fine-tuning has an impact on the scores, but, by comparing the patterns of score variation across samples used for dapt we observe that this variation does not seem to be explained only by the choice of hyperparameter values. clearly, data used for dapt has also an impact on the scores. thus, we asked: what features from the samples used for dapt impact on the downstream scores and to what extent? and to what extent is the impact of the choice of hyperparameter values? we hypothesized that text patterns, such as sentence length, syntactic dependencies, or text complexity, among others, impact on the downstream scores. in order to analyze the importance of each text feature, and the effect of hyperparameters, we used multilevel modeling, a regression model widely used in the social sciences to explain social phenomena such as the effect of both school and student features on the students' performance scores. our results are simple: clarity and simplicity of writing, and amount of biomedical information, are key text features from pubmed abstracts for improving downstream scores. moreover, variation in scores is also largely due to the choice of hyperparameters, contributing approximately as equal as the text features. these results not only explain important features from domain adaptation but can also serve for designing better document sampling strategies and hyperparameter search methods. moreover, the use of multilevel models (mlms) is key for a deeper understanding of nlp models which we hope the nlp community will adopt. ## conclusions how important is to analyze the data used for dapt? working in the trenches has allowed us to see the paramount importance that data plays on the right adjustment of llms to a target domain. from a customer-oriented perspective, dapt plays a vital role for the correct adjustment of llms not only to parametric knowledge but also to human alignment via sft and to databases via rag (retrieval augmented generation). thus, as in a snowball effect, studying the factors that matter for biomedical dapt -clarity and simplicity of writing as well as biomedical information-assures us to provide better adapted llms for customer applications.