Relation Extraction Using Multiple Pre-Training Models in Biomedical Domain (2021)

Abstract: The number of biomedical documents is increasing rapidly. Accordingly, a demand for extracting knowledge from large-scale biomedical texts is also increasing. BERT-based models are known for their high performance in various tasks. However, it is often computationally expensive. A high-end GPU environment is not available in many situations. To attain both high accuracy and fast extraction speed, we propose combinations of simpler pre-trained models. Our method outperforms the latest state-of-the-art model and BERT-based models on the GAD corpus. In addition, our method shows approximately three times faster extraction speed than the BERT-based models on the ChemProt corpus and reduces the memory size to one sixth of the BERT ones.

## introduction the amount of biomedical documents is increasing rapidly. the documents contain valuable knowledge, such as chemical compound names and their relations. however, the current knowledge extraction is considerably manual. therefore, a demand for extracting knowledge automatically from largescale biomedical text data is increasing. biomedical relation extraction (re) models based on bert @xcite have shown great performance @xcite . the methods using bert models pretrained on biomedical corpora achieved state-ofthe-art (sota) performance on several biomedical re datasets. however, bert models require a huge amount of computational resources and generally need a long time for extraction processes. by processing data in parallel with multiple computational resources, we can process larger text as compared with a single resource. however, a high-end gpu or a distributed environment for efficient computation is not available in many situations. even if we can utilize such computational resources, substantial energy consumption becomes a problem @xcite . therefore, more lightweight and accurate re models are expected. in this paper, we construct a biomedical re model by combining word embeddings obtained from multiple lightweight models. the re model can be executed in a wide range of environments, such as a cpu and a middle-class gpu, by reducing memory consumption during the learning process or the inference process. during the inference process of our proposed model, the amount of calculation can be suppressed and the model can process documents at high speed. since the calculation in each lightweight model is small scale, memory consumption can be suppressed. furthermore, by selecting whether or not to utilize each word embeddings, we can customize the model to be suitable for the computer environment. hence, our model can process faster than the bert-based models in the inference process. we adopt more lightweight pre-trained models: elmo @xcite and contextual string embeddings (cse) @xcite . elmo is contextualized word-level embeddings from the language model (lm) based on multiple layers of bidirectional long-short term memories (bi-lstms) @xcite . on the other hand, cse is character-level embeddings of each input word from lm based on single-layer bi-lstm. subword information of words plays an important role in the estimation of kinds and features of chemicals since chemical names tend to contain characteristic sub-word patterns such as prefixes and suffixes. therefore, we propose re models that combine elmo and cse to utilize both word-level and character-level features effectively. we investigate the effectiveness of our re mod-els in terms of not only the accuracy but also the processing speed and the memory size. the contributions of this paper are as follows: beltagy et al. (beltagy et al., 2019) have proposed a method with pre-trained the bert model called scibert. they pre-trained the bert model on the large scale computer science and biomedical corpora. they constructed a new vocabulary of the bert model for the tasks of science and biomedical domains. the scibert model with their vocabulary achieved sota performance on several benchmark tasks on the domains. lee et al. (lee et al., 2019) have proposed a model called biobert. they pre-trained the bert model on a large scale biomedical corpus containing 4.5 billion words. they applied the model to biomedical ner, re, and question answering tasks and achieved high performance on the benchmark tasks. for the re tasks, they utilized the sentence classifier of the original version of bert, which uses a [cls] token for the classification of relations. they used pre-defined strings such as @gene@xmath0 to express a pair of target entities. for instance, a sentence with two target entities (gene and disease in this case) is represented as example 1. example 1 serine at position 986 of @gene@xmath1. zhou et al. (zhou et al., 2016) have proposed a relation classification model with an attentionbased bi-lstm model. they used pre-defined indicator tags to express a pair of target entities. for instance, a sentence with a pair of target entities is represented as example 2. … nonselective mao inhibitors, selegiline does … cse language model bi-lstm+attention model predicted relation class elmo glove example 2 <@xmath2. entity pairs were anonymized using the predefined strings in the method of lee et al. @xcite . in contrast, this model can predict a relation class using surface information of the target entity pair. sub-word information such as prefixes and suffixes plays an important role in estimations of kinds and features of chemicals. therefore, we express the entity pair using tags in our method. bert is constructed by multiple layers of multihead self-attention layers and requires large-scale computational resources. more lightweight pretraining lm models have also been proposed. @xcite have proposed models for biomedical nli tasks using elmo pre-trained on large-scale in-domain text data. elmo is an lm based on multi-layer bi-lstms for aiming at obtaining contextualized word-level embeddings. cse is generated by a character-level lm. the lm is lightweight since it is constructed with a single layer of bi-lstm. sharma et al. @xcite have proposed a biomedical ner method with cse generated from the lm pre-trained on a biomedical corpus. watanabe et al. @xcite have proposed a method with a multi-task learning model using cse. their method achieved sota performance on the biomedical ner task. @xcite evaluated the effectiveness of cse on the biomedical ner tasks. however, they did not evaluate the effectiveness of cse on the biomedical re task. we evaluate the effectiveness of elmo and cse on the biomedical re tasks. ## proposed method figure 1 shows an overview of our method. we incorporate three types of word embeddings into the re model: glove @xcite , cse, and elmo. first, we train a character-level lm for cse and a word-level lm for elmo using large-scale biomedical corpora. then we obtain glove, cse, and elmo vectors corresponding to each word in a sentence as shown in the middle of figure 1 . next, we construct an re model as shown in the top of figure 1 . we explain the pretraining procedure for the glove embeddings and the language models for elmo and cse vectors in section 3.1. then, we explain the re model based on the combinations of multiple word embeddings in section 3.2. ## discussion we showed the effectiveness of our method in the previous section. in this section, we discuss our method from various perspectives, including some negative results. first, we discuss a comparison between the bert-based models and the proposed method in section 6.1. then, we discuss the experimental settings of the proposed method in section 6.2 and 6.3. ## conclusions in this paper, we reported the effectiveness of lightweight and high-performance re models for the biomedical domain. our method used the combination of word embeddings generated by the pretrained lms (the elmo model and the cse model). the elmo model is a word-level lm and the cse model is a character-level lm. we proposed re models that combined elmo and cse to utilize both word-level and character-level features effectively. we evaluated the proposed methods on the biomedical re datasets. we used the chemprot dataset and the gad dataset. we compared our methods with bert-based methods (biobert and scibert) and the sota methods. we also evaluated the model performance and the inference time. experimental results showed the effectiveness of the combinations of the lms. for the gad dataset, we obtained the sota score. for the chemprot dataset, our model showed approximately three times faster extraction speed than biobert. in addition, our model reduced the memory size to one sixth of the bert-based models. however, the f1 score of our method was lower than that of biobert for the chemprot dataset. in future work, we analyze the causes of the high extraction speed and the low performance of our model for the chemprot dataset in terms of the parameter size and architectures. taiki watanabe, akihiro tamura, takashi ninomiya, takuya makino, and tomoya iwakura. 2019. multitask learning for chemical named entity recognition with chemical compound paraphrasing. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6243-6248. peng zhou, wei shi, jun tian, zhenyu qi, bingchen li, hongwei hao, and bo xu. 2016. attention-based bidirectional long short-term memory networks for relation classification. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 207-212.