Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SCIBERT, a pretrained language model based on BERT (Devlin et al., 2019) to address the lack of highquality, large-scale labeled scientific data. SCIBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github. com/allenai/scibert/.
SciBert Due to the scarcity of annotated corpus in the scientific domain, SciBert REF is present to improve downstream scientific NLP tasks.
SciBERT: A Pretrained Language Model for Scientific Text
EMNLP/IJCNLP
Computer Science