B io M egatron: Larger Biomedical Domain Language Model (2020)

Abstract: There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books. Yet, most works do not study the factors affecting each domain language application deeply. Additionally, the study of model size on domain-specific models has been mostly missing. We empirically study and evaluate several factors that can affect performance on domain language applications, such as the sub-word vocabulary set, model size, pre-training corpus, and domain transfer. We show consistent improvements on benchmarks with our larger BioMegatron model trained on a larger domain corpus, contributing to our understanding of domain language model applications. We demonstrate noticeable improvements over the previous state-of-the-art (SOTA) on standard biomedical NLP benchmarks of question answering, named entity recognition, and relation extraction. Code and checkpoints to reproduce our experiments are available at [github.com/NVIDIA/NeMo].

## introduction effectively transferring the success of bert @xcite to the biomedical domain, most notably @xcite (biobert) and @xcite (scibert) inspired a large number of similar works last year. for example, @xcite ; @xcite ; @xcite added clinical text to the pubmed biomedical pretraining corpus and tested on standard biomedical and clinical nlp benchmarks. many other similar works appeared at the acl bionlp workshop @xcite . more recently, @xcite performed a comprehensive study on the pre-training corpus domain, language model masking method, and adversarial training, benchmarking on a number of different datasets for token classification, sequence classification, and sequence regression. compared to the previous works, we perform a more detailed study on (1) subword vocabulary, (2) labeling method, (2) model size, and (3) domain transfer, showing gains in token classification, sequence classification, and question answering. ## related works a prime example of language models (lms) in the biomedical domain is biobert @xcite . it is a transformer lm pre-trained on the pubmed ( www.ncbi.nlm.nih.gov/pubmed ) biomedical text corpus comprised of biomedical literature abstracts. their pre-training started from the checkpoint of @xcite trained on wikipedia and books-corpus. independently, @xcite (scibert) pre-trained bert from scratch using their vocabulary set on scientific text corpora, including pubmed abstracts and computer science papers. both demonstrated increased performance over the previous non-bert sota on biomedical benchmarks, including named entity recognition (ner), relation extraction (re), and question answering (qa). biobert and scib-ert report similar results on ner and re, while only biobert report qa results. they inspired other follow-up works @xcite @xcite , most notably translating their success to the clinical domain, adding the mimic-iii @xcite ) clinical text corpus. @xcite (pub-medbert) used the pubmed full-text for pretraining in addition to the abstracts, and use a domain vocabulary set learned from pubmed corpus. meanwhile, they mostly report similar ner and re tests and results, and only biobert reports qa results. additionally, most use a bert base with 110m parameters. @xcite report slightly improved performance on re using bert large while reporting worse results on ner, compared to bert base . these results on biomedical tasks do not benefit from scaling model size to the same degree as standard nlp benchmarks such as glue or squad @xcite . ## language model pre-training bert base & large we compare our models to the pre-trained bert base & large models of biobert @xcite and pubmedbert @xcite ) (bert base ) for fine-tuning and evaluation. for qa we use the bert large variant of biobert following the authors' recommendation. biomegatron megatron-lm @xcite was introduced for efficient model parallel training of large lms, with up to 8.3b parameters. @xcite showed that rearranging the order of the layer normalization and the residual connections is critical to enabling the scaling of the bert-style models beyond 336m parameters, and we use the same architecture. megatron-lm also used a larger pre-training text corpus, comprised of wikipedia @xcite , cc-stories (trinh and le, 2018), real-news @xcite , and openwebtext @xcite . for our lm training, we use the 4.5 billion-word pubmed abstract set and the 1.6 billion-word cc0-licensed commercial use collection of the pmc full-text corpus ( www.ncbi.nlm.nih.gov/pmc ). we train three sizes of biomegatron: with 345 million, 800 million, and 1.2 billion number of parameters. we compare four pre-training scenarios in the smallest 345m model -using bert-cased/uncased vocabularies, each pre-trained from scratch and finetuned from general domain lm. we also compare two sets of domain vocabularies learned on pubmed text corpus using sentencepiece (github.com/google/sentencepiece) library, each containing 30k and 50k subword units. we train the larger biomegatron models with less variation: 800m models from scratch on pubmed with bert -cased/-uncased vocabularies; and 1.2b model starting from general domain lm checkpoint using bert-uncased vocabulary. ## downstream benchmark tasks we use the most widely used downstream biomedical benchmark datasets for ner, re, and qa. named entity recognition the bc5cdr @xcite ner dataset annotated disease and chemical terms with iob tagging @xcite . in ncbi-disease @xcite , only disease entities are iob-tagged. ## results and discussion the evaluation results on ner and re are shown in ## conclusion we review and test several factors that can affect the performance of domain language models. we find that a language model targeted for a domain and application performs best. for example, model size is a secondary factor to vocabulary set for token classification task. larger model size does not necessarily translate to better performance on a cross-domain benchmark task. this probably indicates that there is no master model that can "do it all", at least well enough as a targeted one. the model size is a secondary factor; larger model size can probably further improve the performance of a a domain-and applicationspecific language model.