B io BART : Pretraining and Evaluation of A Biomedical Generative Language Model (2022)

Abstract: Pretrained language models have served as important backbones for natural language processing. Recently, in-domain pretraining has been shown to benefit various domain-specific downstream tasks. In the biomedical domain, natural language generation (NLG) tasks are of critical importance, while understudied. Approaching natural language understanding (NLU) tasks as NLG achieves satisfying performance in the general domain through constrained language generation or language prompting. We emphasize the lack of in-domain generative language models and the unsystematic generative downstream benchmarks in the biomedical domain, hindering the development of the research community. In this work, we introduce the generative language model BioBART that adapts BART to the biomedical domain. We collate various biomedical language generation tasks including dialogue, summarization, entity linking, and named entity recognition. BioBART pretrained on PubMed abstracts has enhanced performance compared to BART and set strong baselines on several tasks. Furthermore, we conduct ablation studies on the pretraining tasks for BioBART and find that sentence permutation has negative effects on downstream tasks.

## introduction since the advent of elmo @xcite and bert @xcite , the new pretrain-thenfinetune paradigm has brought great performance improvement and dominated the methodology research of the natural language processing (nlp) field. previous research has illustrated that pretraining language models on the domain-specific corpora can improve the model performance on domain-specific tasks further @xcite . with the large-scale publicly accessible corpora from pubmed, researchers have already proposed biomedical domain pretrained language models such as biobert @xcite and pubmedbert @xcite to aid the later research. natural language generation (nlg) tasks such as dialogue system @xcite and question answering @xcite are of critical importance for the biomedical artificial intelligence research, and there is also a trend to approach natural language understanding as nlg tasks in the general domain @xcite . for example, an entity retrieval task can be solved by constrained natural language generation @xcite . however, there exist two gaps in the research of the biomedical nlg. on the one hand, the architectures of the biomedical pretrained language models are almost all encoder-only transformers. such architecture is incapable of generating natural languages auto-regressively. a decoder is necessary for language generation @xcite . on the other hand, there are very few in-domain generative language models for biomedicine @xcite . models pretrained on biomedical corpora may further enhance the performance of current biomedical nlg methods. to bridge the gaps mentioned above, we propose a biomedical auto-regressive generative language model, biobart, pretrained on the biomedical corpora. in our work, we adopt bart (bidirectional and auto-regressive transformers), a generative pretrained language model which achieves state-of-the-art results on different nlg tasks in the general domain @xcite . we continuously pretrain bart on pubmed abstracts to achieve biomedical domain adaption only using the text-infilling task. we also collate and evaluate bio-bart on the existing biomedical nlg tasks. the in-domain biobart outperforms bart model and sets strong baselines for several nlg tasks. the main contributions of our work are summa-rized as follows foot_0 : ## biomedical domain pretraining bart is a sequence-to-sequence model with a bi-directional encoder and a left-to-right autoregressive decoder. the model architecture is consistent with the transformers @xcite except for changing the relu activation functions to gelus @xcite . bart is pretrained by denoising the corrupted input documents. the work ablates five different types of corruption noise: text masking, text deletion, text infilling, sentence permutation, and document rotation. as a result, the pretraining documents are corrupted in two ways: 1) text infilling: for each document, a number of token spans are sampled, and each sample span is replaced with a single mask token. 2) sentence permutation: a document is split into sentences and sentences are shuffled in random orders. the pretraining objective is to minimize the negative log-likelihood of the original documents. prior work has shown that continuous-pretrained models can get competitive results compared with those trained from scratch @xcite . in our work, we continuously pretrain bart on the biomedical domain corpora. we revisit the methods to corrupt input texts. bart keeps the sentence permutation noise because of the significant performance gain on the summarization task, although this noise may lead to slight degradation on other tasks. we run further ablation studies on various biomedical nlg tasks. we show that the model pretrained without sentence permutation has better performance. further details are listed in section 5.5. therefore we only implement the text infilling task to corrupt input texts for pretraining biobart. ## generative downstream task in this section, we introduce the generative downstream tasks in the biomedical domain. we will conduct experiments on these tasks to illustrate the performance of the domain-specific biobart. ## conclusions in this work, we pretrain the biomedical domain generative language model biobart. we also collect various publicly available benchmarks for biomedical generative tasks to prompt future research. our experimental results show that continuous pretraining on pubmed abstracts helps the model with domain adaption. biobart shows great improvements on different benchmarks and achieves competitive or superior results over the current state-of-the-art methods. we also release our pretraining and fine-tuning codes to facilitate future research for reproducibility. we will explore pretraining generative language models 1) on in-domain vocabularies and from scratch, 2) and with clinical corpora such as emrs in mimic-iii @xcite or pmc-patients @xcite in the future studies.