B io ELECTRA :Pretrained Biomedical text Encoder using Discriminators (2021)

Abstract: Recent advancements in pretraining strategies in NLP have shown a significant improvement in the performance of models on various text mining tasks. We apply ‘replaced token detection’ pretraining technique proposed by ELECTRA and pretrain a biomedical language model from scratch using biomedical text and vocabulary. We introduce BioELECTRA, a biomedical domain-specific language encoder model that adapts ELECTRA for the Biomedical domain. WE evaluate our model on the BLURB and BLUE biomedical NLP benchmarks. BioELECTRA outperforms the previous models and achieves state of the art (SOTA) on all the 13 datasets in BLURB benchmark and on all the 4 Clinical datasets from BLUE Benchmark across 7 different NLP tasks. BioELECTRA pretrained on PubMed and PMC full text articles performs very well on Clinical datasets as well. BioELECTRA achieves new SOTA 86.34%(1.39% accuracy improvement) on MedNLI and 64% (2.98% accuracy improvement) on PubMedQA dataset.

## introduction following the success of bert @xcite (bidirectional encoder representations from transformers) in the general domain, the pretrain-andfinetune approach has been used in the biomedical domain. with large scale free text available from pubmed and pubmed central (millions of articles), biomedical domain has large unlabelled domainspecific corpus. however, the biomedical domain has labelled datasets that are very small compared to the general domain. thus the transfer learning approach is well suited for biomedical domain. in the biomedical domain, biobert @xcite , bluebert @xcite and clinical-bert @xcite are the initial models based on bert. these models follow continual pretraining approach where the model weights are initialised with weights from bert trained on wikipedia and book corpus and uses the same vocabulary. recent models scibert @xcite , pubmedbert @xcite and biolm @xcite have shown that pretrain-ing from scratch using domain specific corpora along with domain specific vocabulary improves the model performance significantly. in this work, we adapt electra @xcite , a recent and powerful general domain model for the biomedical domain and we release bio-electra -a biomedical domain specific language encoder model. we follow the domain specific pretraining approach where the electra model is pretrained on pubmed and pubmed central (pmc) full text articles. electra outperforms bert, albert @xcite , xlnet @xcite and roberta @xcite on the glue @xcite benchmark and squad @xcite . in particular, we make the following contributions. ## related work pretrained word embeddings @xcite , @xcite and contextualised word embeddings @xcite have helped the deep learning algorithms to improve their performance in nlp tasks. ulmfit @xcite , introduces the transfer learning approach to natural language processing and ope-nai gpt @xcite , pretrains a transformer @xcite for learning general language representations. similar to ulm-fit and openai gpt, bert @xcite follows this fine tuning approach and introduces a powerful bidirectional language representation model using the transformer based model architecture. bert achieves sota on most nlp tasks without any heavily-engineered task specific architectures. following the success of bert, xlnet @xcite with generalized autoregressive pretraining and roberta @xcite with robust pretraining techniques experiment with different pretraining objectives. albert @xcite uses weight sharing and embedding factorisation to reduce memory consumption and increase the training speed. electra @xcite introduces sample-efficient 'replaced token detection' pretraining technique. electrasmall, trained with very little compute outperforms gpt and performs comparably with larger models like roberta and xlnet. recent works adapt bert to scientific, biomedical and clinical domains. biobert @xcite pretrains bert with data from pubmed and pubmed central (pmc) articles. bluebert @xcite pretrains bert on pubmed, pmc and mimic iii @xcite data. clinicalbert @xcite initialises with biobert weights and pretrains on data from mimic iii. scibert @xcite , pub-medbert @xcite and bio-lm @xcite pretrain bert based models from scratch with domain specific data. scibert pretrains on 1.14m papers from semantic scholar @xcite , pubmedbert on pubmed and pmc data and bio-lm @xcite ## results our models pretrained on domain specific text along with domain specific vocabulary have consistently shown that the pretraining from scratch with domain specific data enables the model to capture the contextual representations of the language better. the results clearly show that all bioelectra models pretrained from scratch with biomedical domain text and domain specific vocabulary perform better than the model pretrained on both general and biomedical domain text with general domain vocabulary. however it is interesting to note that bioelectra (w+p) model has significantly better results for i2b2-2010, share/clefe and clin-icalsts datasets than pubmedbert. bioelec-tra (w+p)'s score for mednli is comparable to that of pubmedbert @xcite . ## conclusion and future work we release bioelectra-base models pretrained from scratch on biomedical domain specific text and evaluate the performance on seven different biomedical nlp tasks with 17 datasets. we achieve sota on all the datasets in the blurb @xcite benchmark and all four clinical datasets in the blue @xcite benchmark. our results show that pretraining from scratch with biomedical domain text helps the model to learn better contextual representations. we release the pretrained weights for all our models and the code for reproducibility. we plan to explore and experiment with our domain specific pretraining approach on electra-large models. we also intend to train electra-base and electra-large mod-els on mimic iii @xcite clinical notes and evaluate the performance of the models on biomedical nlp tasks. as electra shows a significant improvement on squad @xcite , we want to focus on biomedical qa tasks (span prediction) and evaluate domain specific pretrained electra models performance.