B io M -Transformers: Building Large Biomedical Language Models with BERT , ALBERT and ELECTRA (2021)

Abstract: The impact of design choices on the performance of biomedical language models recently has been a subject for investigation. In this paper, we empirically study biomedical domain adaptation with large transformer models using different design choices. We evaluate the performance of our pretrained models against other existing biomedical language models in the literature. Our results show that we achieve state-of-the-art results on several biomedical domain tasks despite using similar or less computational cost compared to other models in the literature. Our findings highlight the significant effect of design choices on improving the performance of biomedical language models.

## introduction the amount of biomedical literature has grown substantially in recent years. this growth created a demand for powerful biomedical language models. transformer-based language models, such as bert @xcite , have shown effectiveness in capturing the contextual representation of corpora at large volume. to address the lack of biomedical contextual representation, both biobert @xcite , and scibert @xcite have adapted bert to the biomedical domain. recently, several transformer-based models have been introduced, including megatron @xcite , roberta @xcite , albert @xcite and electra @xcite . these models show impressive performance gains over bert in the general domain leading most nlp leader boards. however, these models have been evaluated with environmental design factors varying in several dimensions (e.g., vocabulary and corpora domain, loss function, training steps, batch size, and model's scale). understanding the contribution of these factors to the performance of the language models is challenging, especially when our goal is to shift the contextual representations to the biomedical domain. this challenge motivates us to investigate the impact of design choices on the performance of biomedical language models. moreover, highlighting this impact is critical when evaluating new applications in bionlp, where each application may evaluate its performance against other models that use different design setups. in this work, we pretrain and evaluate different variants of large biomedical transformer-based models across different design factors. thus, our contributions in this paper includes : ## pretraining our language models we pretrain all our models using the original implementation of bert, albert, and electra. we use tensorflow 1.15 and tpuv3-512 units to pretrain our large models and tpuv3-32 to pretrain our biom-electra b model. ## results and discussion ## conclusion we introduce four biomedical transformer-based language models. our results show that language models with general domain vocabulary and pubmed+pmc corpora perform better on the model h time ratio biom-electra b 768 03:01 0.35x biom-electra l 1024 08:27 1.00x biom-albert xxlarge 4096 31:15 3. 67x