Pre-training data selection for biomedical domain adaptation using journal impact metrics (2024)

Abstract: Domain adaptation is a widely used method in natural language processing (NLP) to improve the performance of a language model within a specific domain. This method is particularly common in the biomedical domain, which sees regular publication of numerous scientific articles. PubMed, a significant corpus of text, is frequently used in the biomedical domain. The primary objective of this study is to explore whether refining a pre-training dataset using specific quality metrics for scientific papers can enhance the performance of the resulting model. To accomplish this, we employ two straightforward journal impact metrics and conduct experiments by continually pre-training BERT on various subsets of the complete PubMed training set, we then evaluate the resulting models on biomedical language understanding tasks from the BLURB benchmark. Our results show that pruning using journal impact metrics is not efficient. But we also show that pre-training using fewer abstracts (but with the same number of training steps) does not necessarily decrease the resulting model’s performance.

## introduction advances in deep learning for natural language processing (nlp) in recent years have enabled transfer learning to develop @xcite , particularly since the creation of transformers @xcite . one type of transfer learning aims to start with a pre-training phase where the model learns the general language structure and then a second phase where the model can be fine-tuned for a specific task. in the context of deep learning for nlp, this method avoids re-training a model from scratch for each new task, starting with a model that already has general language knowledge. these pre-trained models generally use a large corpus of text. a specialized domain, such as finance or the biomedical domain, may contain numerous tasks. in the case of language, a specialized domain has a specific vocabulary containing terms more rarely found in general texts. we can observe this phenomenon when looking at tokens produced by a biomedical tokenizer against a general tokenizer @xcite . moreover, tasks may require domain-specific knowledge not found in general sources. so, to improve the performance of a model previously trained on a general domain to a specific domain, it is interesting to use a corpus specific to the domain to which we wish to adapt our model. most of the data used for pre-training in the biomedical field are research articles and papers that can be either abstracts, full texts, or a combination of both. this data generally originates from large public databases such as pubmed or pubmedcentral (for full-text articles). however, to our knowledge, no study has examined the selecting subsets of these large databases for pre-training using metrics specific to scientific papers. that leads us to our research question: can a language model be adapted to the biomedical domain by efficiently selecting scientific documents in the pretraining data while maintaining or improving its performance? this paper presents our experiments on adapting the pretrained bert-base model to the biomedical domain. we get the pubmed january 2024 baseline corpus and define different subset configurations using journal impact metrics: h-index @xcite and scimago journal rank or sjr (guerrero-bote and moya-anegón, 2012). we then perform continual pre-training from the bert-base model @xcite and evaluate it on several tasks from the blurb benchmark @xcite . ## methods we use the same methodology as @xcite , with some small modifications : let d be a large dataset containing documents and ξ a metric assigning a score to a document. we build a subset p cξ by adding instances that fit our selection criteria c : where c 0ξ and c 1ξ are the lower and upper bound for the criteria c and the metric ξ. for each metric, we consider two selection criteria: keeping top or middle percentiles foot_0 of d as the data to be kept. this serves as verifying if the model learns better with high quality documents (defined by the metric, for our metrics, higher is better). we keep either 25% or 50% of the documents in d. so for instance, if we take the 25 % middle for the metric ξ, we should compute the 37.5 % and 62.5 % percentiles with respect to metric ξ, which corresponds to c 0ξ and c 1ξ , and keep the documents between these two percentiles. then, we tokenize each document in the subset, and we concatenate them into sequences of length equal to the model's context length. this differs from @xcite as we do the filtering before tokenization (because our metrics are applied on a document, not on a sequence of tokens). these sequences are then used to pre-train a model. the goal is then to pre-train a model on a subset of the whole training set while retaining or improving the model's performance. ## results and discussion to limit random effects, we perform the fine-tuning multiple times with different random seeds, as described in the blurb paper: using five seeds for all datasets except for bioasq and pubmedqa, for which we use ten seeds (because they are smaller in size). we then report the average performance across the different seeds for each dataset in the ## conclusion this paper presents our early experiments on selecting the pre-training data for the biomedical domain. we show that the journal impact metrics are not better than the random selection at a fixed number of training steps. we also observe that reducing the number of abstracts in the training set does not necessarily decrease the final model performance and show the need to investigate how many documents we need to pre-train a model without losing performance. further directions include finding better metrics (or combinations of metrics) to assess the quality of a document in the pre-training corpus, investigating metrics at a different level (at the corpus level using various mixtures of biomedical domains), and using a corpus of full-text articles.