A Multilevel Analysis of P ub M ed-only BERT -based Biomedical Models (2024)

Abstract: Biomedical NLP models play a big role in the automatic extraction of information from biomedical documents, such as COVID research papers. Three landmark models have led the way in this area: BioBERT, MSR BiomedBERT, and BioLinkBERT. However, their shallow evaluation –a single mean score– forbid us to better understand how the contributions proposed in each model advance the Biomedical NLP field. We show through a Multilevel Analysis how we can assess these contributions. Our analyses across 5000 fine-tuned models show that, actually, BiomedBERT’s true effect is bigger than BioLinkBERT’s effect, and the success of BioLinkBERT does not seem to be due to its contribution –the Link function– but due to an unknown factor.

## introduction machine reading of biomedical texts has greatly advanced due to pretrained nlp models such as bert. biomedical nlp applications are of great value due to their utility in real-world scenarios such as answering questions which require background knowledge or the extraction of complex biomedical entities from astonishing volumes of academic papers related to covid, for example. of special acknowledgement, three bert-based biomedical models, trained on pubmed abstracts (their only source of biomedical knowledge), led the way to concise research contributions on biomedical nlp, namely, biobert @xcite proposing domain adaptive pretraining (dapt), msr biomedbert 1 @xcite we refer to as biomedbert) which challenged dapt by pretraining bert from scratch with pubmed abstracts, and biolinkbert @xcite which implemented a way to link hyperlinked documents at pretraining timethe link function. these 3 contributions resulted 1 previously known as pubmedbert. however, we claim, current evaluation methods are oversimplistic. they reduce to a simple mean score across datasets in the blurb suite -a single estimate. this forbid us to better understand the contributions proposed by each work such as their effect on scores and interaction with downstream datasets. moreover, from this single estimate, how can we disentangle the contributions' effects from the effects of other variables such as random seeds, learning rates, or number of epochs? we cannot. and while some works show ablation studies to see the particular effects of the proposed contribution, doing so to isolate it from all possible variables (including those mentioned above) leads to an exponential number of ablations which results in a non-environmentally friendly, unfeasible approach if pretraining is necessary for each ablation. in this paper, we propose a regression analysis widely used in the fields of psychology and the social sciences -multilevel analysis-to account for the effects of all measurable variables, without the need for ablations or further pretraining experiments, in order to disentangle their effects from the true effect of the proposed contributions from biobert, biomedbert, and biolinkbert. our analyses show that, actually, random seeds have a big effect on downstream scores. also, while biobert's and biomedbert's contributions have a big and significant effect by improving on vanilla bert's score by 2.25 and 4.36 points, respectively, on average across blurb datasets, biolinkbert's link function shows only a big effect for qa datasets but not for any other dataset. ## multilevel analysis and results we show the results of fitting our mlm. for tables 2 , 3 , and 4, the statistical significance code is: @xmath0, @xmath1.001 '**', @xmath2.01 '*'. mlm results for level-1 variables: we first test for the statistical significance of fixed-and randomeffects of level-1 variables. we observe in mlm results for level-2 variables: we observe that most level-2 variables are statistically significant ( effects from pretrained models: if we fit our mlm with indicator variables for each pretrained model, instead of their contributions, we obtain the following effects: biobert (1.79**), biomed-bert (4.52***), biolinkbert (3.47***). the result for biolinkbert seems to contradict the nonsignificant effect of the link function. it does not. the effect of the biolinkbert variable takes into account all possible functions inside biolinkbert (without disentangling them) including the variable of pretrain_pubmed since biolinkbert was pretrained from scratch with pubmed data. this means that, overall, biolinkbert is highly useful: it surpasses vanilla bert, on average, by 3.47 points across datasets, though the link function does not seem to be the main reason for this result due to its small effect size and lack of statistical significance. surprisingly, we see that biomedbert has the biggest mean effect of all models: 4.52 points improvement over bert. r-squared effects: as shown in figure 1 , fixedeffects of level-1 and level-2 variables account for around 70% of all the variability in the test scores; however, given that most of the level-1 coefficients are non-significant and moderately small, we would expect them to contribute little to this explanation of variability. surprisingly, though, level-1 random coefficients (slope variation) account for around 10% of the variance in test scores, a considerable portion of the variability. finally, we note that around 20% of the variance remains unexplained (the residual part) which may mean two things. first, there is still room for adding variables at either level to better explain the test scores; we hypothesized that other pretraining features, such as batch size, could impact on the scores, however, it was not possible to add them to the analysis since they perfectly correlate with variables already added, leading to the problem of collinearity. and second, fully understanding nlp models is a complex task which requires detailed analyses of several variables. robust estimates: as shown in ## conclusions our multilevel analysis of biomedical models can disentangle the effects from fine-tuning and pretraining by providing particular effects of each variable with respective statistical significance. as we saw, contrary to expectation, biomedbert has the biggest mean effect across datasets from all models. moreover, even though biolinkbert holds as a useful model, its main contribution -the link function-does not seem to be the main reason for its success, except for qa datasets where the link function excels. furthermore, we showed that all fine-tuning variables behave differently for each pretrained model, giving some advantage to some models purely by chance. and this figure, according to r-squared tests, accounts for 10% of all the test scores; thus, we suggest using several random seeds to counterbalance their effects. finally, we note that it would be nearly impossible to see all these figures with current evaluation methods -a