B io NART : A Biomedical Non- A uto R egressive Transformer for Natural Language Generation (2023)

Abstract: We propose a novel Biomedical domain-specific Non-AutoRegressive Transformer model for natural language generation: BioNART. Our BioNART is based on an encoder-decoder model, and both encoder and decoder are compatible with widely used BERT architecture, which allows benefiting from publicly available pre-trained biomedical language model checkpoints. We performed additional pre-training and fine-tuned BioNART on biomedical summarization and doctor-patient dialogue tasks. Experimental results show that our BioNART achieves about 94% of the ROUGE score to the pre-trained autoregressive model while realizing an 18 times faster inference speed on the iCliniq dataset.

## introduction in biomedical natural language processing, several domain-specific pre-trained language models (plms) such as biobert @xcite , scib-ert @xcite and pubmedbert @xcite have been proposed by training on domain-specific text in pubmed/medline, and these domain-specific models have shown remarkable performance on the in-domain downstream tasks. similarly, following the success of generative encoder-decoder-based sequence-to-sequence plms such as bart @xcite , t5 @xcite , and bert2bert @xcite , pre-trained biomedical generative models such as scifive @xcite and bio-bart @xcite have also been proposed, and these models can tackle the generation tasks in the biomedical domain, such as clinical dialogue, biomedical question answering and biomedical text summarization. the generative models usually adopt an au-toregressive (ar) strategy to decode the target sequence token-by-token from left to right. ar decoding can perform high-quality inference, but it has the limitation that it cannot be easily parallelized and requires significant time and computational cost for inference. to alleviate this problem, several non-autoregressive (nar) methods @xcite that can simultaneously predict all target tokens have been proposed in the general domain. however, there has been no discussion and verification of nar methods in the biomedical domain. this paper proposes a novel biomedical domainspecific nar model called bionart. in constructing the biomedical nar model, we achieve nar generation with only minimal modifications from the bert @xcite -based encoderdecoder model bert2bert @xcite . this allows our model to benefit from already performed large-scale pre-training by initializing both the encoder and decoder of our model with parameters checkpoints of plms such as biobert @xcite and scibert @xcite . in addition, we further verify the effect of additional pre-training to fit the parameters from checkpoints to the encoder-decoder nar. finally, we perform our pre-training on pubmed/medline abstracts with permutation language modeling @xcite task. our contributions are summarized as follows: ## method in this section, we first describe our bionart model architecture and then explain the objective function used during training and nar text generation for inference. finally, we show the parameter initialization of our bionart. the overview of bionart is shown in figure 1 . ## related work existing work has shown that pre-training the language models on the domain-specific texts can bring better performance on corresponding domain downstream tasks. biobert @xcite is pre-trained on the pubmed abstracts and pubmed central (pmc) full-text articles. instead of continuous training from the general bert checkpoint, scibert @xcite and pubmed-bert @xcite are trained from scratch from scientific papers on semantic scholar @xcite and pubmed papers, respectively. biobart @xcite further analyzed the influence of the pre-training task on the biomedical natural language generation (nlg) tasks and collated existing biomedical nlg tasks along with corresponding data and experimental settings, but all biomedical nlg models are ar-based, and there has been no discussion and verification of the nar methods in the biomedical domain. ## conclusion in this study, we proposed a non-autoregressive language model for the biomedical domain for the first time. our method is based on the encoder-decoder model, and both the encoder and decoder are compatible with widely used bert architecture, which allows for boosting generation performance and reducing additional pre-training costs. in the experiments on biomedical summarization tasks, bionart achieved an 18 times inference speedup without a large performance drop compared to ar models. our code and models are available at https://github.com/aistairc/bionart in future work, we would like to approach other downstream tasks such as relation extraction and entity linking as sequence-to-sequence problems and realize a high-speed and versatile model for the biomedical domain.