Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art (2020)

Abstract: A large array of pretrained models are available to the biomedical NLP (BioNLP) community. Finding the best model for a particular task can be difficult and time-consuming. For many applications in the biomedical and clinical domains, it is crucial that models can be built quickly and are highly accurate. We present a large-scale study across 18 established biomedical and clinical NLP tasks to determine which of several popular open-source biomedical and clinical NLP models work well in different settings. Furthermore, we apply recent advances in pretraining to train new biomedical language models, and carefully investigate the effect of various design choices on downstream performance. Our best models perform well in all of our benchmarks, and set new State-of-the-Art in 9 tasks. We release these models in the hope that they can help the community to speed up and increase the accuracy of BioNLP and text mining applications.

## introduction the pretrain-and-finetune approach has become the dominant paradigm for nlp applications in the last few years @xcite @xcite , bringing significant performance gains in many areas of nlp. models trained on wikipedia and webtext @xcite generally perform well on a variety of target domains, but various works have noted that pretraining on in-domain text is an effective method for boosting downstream performance further @xcite @xcite . several pretrained models are available specifically in the domain of biomedical and clinical nlp driving forward the state of the art including biobert @xcite , scibert @xcite , clinicalbert @xcite and biomedroberta @xcite . while it is great to have multiple options, it can be difficult to make sense of what model to use in what case -different models are often compared on different tasks. to further complicate matters, more powerful general-purpose models are being released continuously. it is unclear whether it is better to use a more powerful general-purpose model like roberta, or a domain-specific model derived from an earlier model such as biobert. and given the opportunity to pretrain a new model, it is unclear what are the best practices to do that efficiently. our goal is to understand better the landscape of pretrained biomedical and clinical nlp models. to that effect, we perform a large-scale study across 18 established biomedical and clinical nlp tasks. we evaluate four popular bionlp models using the same experimental setup. we compare them to general purpose roberta checkpoints. we find that biobert performs best overall on biomedical tasks, but the general-purpose roberta-large model performs best on clinical tasks. we then take advantage of recent advances in pretraining by adapting roberta @xcite to biomedical and clinical text. we investigate what choices are important in pretraining for strong downstream bionlp performance, including model size, vocabulary/tokenization choices and training corpora. our best models perform well across all of the tasks, establishing a new state of the art on 9 tasks. finally, we apply knowledge distillation to train a smaller model that outperforms all other models with similar computational requirements. we will release our pretrained models and the code used to run our experiments. 1 2 tasks and datasets we select a broad range of datasets to cover both scientific and clinical textual domains, and common modelling tasks -namely i) sequence labelling tasks, covering named entity recognition (ner) and de-identification (de-id) and ii) classification tasks, covering relation extraction, multi-class and multi-label classification and natural language inference (nli)-style tasks. these tasks were also selected to optimize overlap with previous work in the space, drawing tasks from the blue benchmark @xcite , biobert @xcite , scibert @xcite and clinicalbert @xcite . the tasks are summarized in jnlpba @xcite ) is an ner task requiring the identification of entities of interest in micro-biology, with 2,000 training pubmed abstracts. ## pretraining corpora there is a wide range of text corpora in the biomedical and clinical domains. we limit our options to data that is freely available to the public so that models can be open-sourced. task name domain task metric task name domain task metric bc5-cdr-chemical pubmed n.e.r. f1 i2b2-2012 clinical n.e.r. f1 bc5-cdr-disease pubmed n.e.r. f1 i2b2-2014 clinical de-id f1 jnlpba pubmed n.e.r. f1 hoc pubmed multi-label classif. macro-f1 ncbi-d pubmed n.e.r. f1 chemprot pubmed rel. extract. macro-f1 bc4chemd pubmed n.e.r. f1 gad pubmed binary rel. extract. f1 bc2gm pubmed n.e.r. f1 eu-adr pubmed binary rel. extract. f1 linneaeus pubmed n.e.r. f1 ddi-2013 pubmed rel. extract. micro-f1 species-800 pubmed n.e.r. f1 i2b2-2010-re clinical rel. extract. f1 i2b2-2010 clinical n.e.r. f1 mednli clinical nli acc pubmed central full-text pubmed central foot_2 (pmc) is an open access collection of over 5 million full-text articles from biomedical and life science research, which has been used in past scientific language modeling work @xcite . following past work, we obtained all pubmed central full-text articles published as of march 2020. we use the pubmed parser package foot_3 to extract plain text from each article. after removing empty paragraphs and articles with parsing failures we retained 60gb of text from 3.4 million articles, consisting of approximately 9.6 billion words. ## pretrained models we compare five publicly-available language models which together form a representative picture of the state-of-the-art in biomedical and clinical nlp. we use the huggingface transformers library to access the model checkpoints @xcite . scibert @xcite is a masked language model (mlm) pretrained from scratch on a corpus of 1.14m papers from semantic scholar @xcite , of which 82% are in the biomedical domain. scibert uses a specialized vocabulary built using sentence-piece @xcite foot_4 on their pretraining corpus. we use the uncased scibert variant. biobert @xcite is based on the bert-base model @xcite , with additional pretraining in the biomedical domain. we use biobert-v1.1. this model was was trained for 200k steps on pubmed and pmc for 270k steps, followed by an additional 1m steps of training on pubmed, using the same hyperparameter settings as bert-base. clinicalbert @xcite is also based on bert-base, but with a focus on clinical tasks. we use the "bio+clinical bert" checkpoint, which is initialized from biobert, and then trained using texts from mimic-iii for 150k steps using a batch size of 32. roberta @xcite is a state-of-theart general purpose model. we experiment with roberta-base and roberta-large to understand how general domain models perform on biomedical tasks. both models are pretrained with much larger batch sizes than bert, and use dynamic masking strategies to prevent the model from overmemorization of the training corpus. roberta outperforms bert on general-domain tasks @xcite . biomed-roberta @xcite ) is a recent model based on roberta-base. biomed-roberta is initialized from roberta-base, with an additional pretraining of 12.5k steps with a batch size of 2048, using a corpus of 2.7m scientific papers from semantic scholar @xcite . ## results digging deeper, we note that standard robertalarge is competitive with the four specialized models on sequence labelling tasks (85.8 vs 85.9) and outperforms them on clinical tasks (84.0 vs 83.3), despite having no specialized biomedical or clinical pretraining. this suggests that larger, more powerful general-purpose models could be a good default choice compared to smaller, less powerful domain-specific models. nevertheless, applying domain-specific training to otherwise-comparable models results in significant performance gains in our experiments, as shown by comparing ours-base and ours-large to roberta-base and roberta-large in ## distillation in section 6.1.3, we noted that larger models result in better accuracy. however, they also require more computational resources to run, limiting their applicability. recent work addresses this issue by distilling larger models into smaller ones while retaining performance. next, we investigate whether distillation works well in the bionlp space. ## related work pretrained word representations have been used in nlp modelling for many years @xcite @xcite , and have been specialised for bionlp applications @xcite @xcite . more recently, contextual embeddings have led to robust improvements across most nlp tasks, notably, elmo @xcite and bert @xcite , followed more recently by models such as xlnet @xcite , roberta @xcite , xlm and xlm-roberta @xcite amongst others. several works adapt such models to scientific and biomedical domains. four such models -scib-ert @xcite , biobert @xcite , clinicalbert @xcite and biomed-roberta @xcite -are extensively covered in section 4. others include bluebert @xcite , which continues to pretrain bert with data from pubmed and mimic-iii. @xcite and @xcite train elmo and bert models on clinical data. in concurrent work, @xcite train models for pubmed-like text, but do not consider clinical text. methods for training or finetuning models on downstream tasks is also an active area of research. we focus on well-established single-task finetuning techniques for bert-like models using standard hyperparameter settings. @xcite use complex task-specific models to yield strong results on clinical tasks, and @xcite investigate stilts methods @xcite on a suite of bionlp tasks, achieving gains over baselines. in this work, we build a suite of 18 tasks to evaluate our models. aggregated benchmarks have become a common tool in nlp research, popularized by the glue benchmark @xcite for language understanding and its successor super-glue @xcite . evaluating on a suite of tasks is common in bionlp too. @xcite al. (2019 2020 ) evaluate on 7 and 2 biomedical tasks respectively. unfortunately, often there is little overlap between efforts, and different metrics and dataset splits are often used, making cross-model comparisons challenging, hence our efforts to evaluate all models on a single testbed. in concurrent work, @xcite also note this problem, and release a similar suite of tasks, referred to as blurb, but do not include clinical tasks. we plan to evaluate our models on the "blurb" benchmarks in future work. ## conclusion we have thoroughly evaluated 6 open-source language models on 18 biomedical and clinical tasks. of these models, we found that biobert was the best on biomedical tasks, but general-purpose roberta-large performed best on clinical tasks. we then pretrained 6 of our own large-scale specialized biomedical and clinical language models. we determined that the most effective models were larger, used a dedicated biomedical vocabulary and included both biomedical and clinical pretraining. these models outperform all the other models in our experiments. finally, we demonstrate that our base model can be further improved by knowledge distillation from our large model, although there remains a gap between the distillation-improved base model and our large model.