On the effectiveness of small, discriminatively pre-trained language representation models for biomedical text mining (2020)

Abstract: Neural language representation models such as BERT have recently shown state of the art performance in downstream NLP tasks and bio-medical domain adaptation of BERT (Bio-BERT) has shown same behavior on biomedical text mining tasks. However, due to their large model size and resulting increased computational need, practical application of models such as BERT is challenging making smaller models with comparable performance desirable for real word applications. Recently, a new language transformers based language representation model named ELECTRA is introduced, that makes efficient usage of training data in a generative-discriminative neural model setting that shows performance gains over BERT. These gains are especially impressive for smaller models. Here, we introduce two small ELECTRA based model named Bio-ELECTRA and Bio-ELECTRA++ that are eight times smaller than BERT Base and Bio-BERT and achieves comparable or better performance on biomedical question answering, yes/no question answer classification, question answer candidate ranking and relation extraction tasks. Bio-ELECTRA is pre-trained from scratch on PubMed abstracts using a consumer grade GPU with only 8GB memory. Bio-ELECTRA++ is the further pre-trained version of Bio-ELECTRA trained on a corpus of open access full papers from PubMed Central. While, for biomedical named entity recognition, large BERT Base model outperforms Bio-ELECTRA++, Bio-ELECTRA and ELECTRA-Small++, with hyperparameter tuning Bio-ELECTRA++ achieves results comparable to BERT.

## introduction transformers based language representation learning methods such as bidirectional encoder rep-resentations from transformers (bert) @xcite are becoming increasingly popular for downstream biomedical nlp tasks due to their performance advantages @xcite . the performance of these models comes at a steep increase in computation cost both at training and inference time. for example, we use a bert based re-ranker as the final step in our biomedical question answering system @xcite , where 60% of the question answering time latency is due to the bert classifier with 110 million parameters. the increased size of the transformer models is correlated with the increased performance @xcite . since the computational cost involved at inference time for large models is a bottleneck in their practical applications in the real world especially for real time applications such as semantic search and question answering, new approaches to achieve similar performance on smaller models are getting increasingly popular. a popular approach on this end is distilling bert to a smaller classifier such as distillbert @xcite , tinybert @xcite and mobilebert @xcite . however, a small and efficient model without going through the trouble of training a large model and mimicking it in a smaller model is more preferable. bert uses a masked language modeling (mlm) approach by masking 15% of the training sentences and learning to guess the masked tokens in a generative manner. this results bert using only 15% of the training data. a recent approach called elec-tra @xcite , introduced a new language modeling approach where a discriminative model is trained to detect whether each token in the corrupted input was replaced by a co-trained generator model sample or not. electra is computationally more efficient than bert and outperforms bert given the same model size, data and computation resources @xcite . the improvements over bert is most impressive at small model sizes, which makes it an excellent candidate in pursuit of small and efficient language representation models for biomedical text mining. in this paper, we introduce two small and efficient electra based domain-specific language representation models trained on pubmed abstracts and on pubmed central (pmc) open-access full papers, respectively, with a domain specific vocabulary achieving comparable or (in some cases) better results on several biomedical text mining tasks to bert base model that have 8 times more parameters resulting in 8 times decrease in inference time. the models are trained on a modest consumer grade gpu with only 8gb ram which is much lower bar for pre-training of domain-specific language representation models than for bert and variants. the performance on biomedical named entity recognition (ner) of small electra models are not as impressive as in the question answering related tasks compared to bert. however, bio-electra++ ner performance can be significantly improved by hyperparameter tuning to achieve comparable performance to bert. ## conclusion in this paper, we have shown that small domainspecific language representation models that make more efficient use of pre-training data can achieve comparable or better (in some cases) downstream performance on several biomedical text mining tasks to bert base with eight times more parameters. two domain-specific biomedical language representation models based on recently introduced electra architecture named bio-electra and bio-electra++ were pre-trained on a consumer grade gpu with only 8gb memory. while, bio-electra performance is highly competitive to bert base for question answering and classification tasks, its performance lags behing bert base for ner tasks. to further improve the performance of bio-electra, we pre-trained it further with a second biomedical corpus of full papers from pmc open access initiative. the resulting biomedical language representation model, bio-electra++, outperformed bio-electra in 8 out of 9 datasets. after hyperparameter finetuning, the performance lead of bert base over bio-electra++ on ner tasks is drastically decreased making bio-electra++ competitive or superior to bert in all biomedical text mining tasks tested.