Comparing Encoder-Only and Encoder-Decoder Transformers for Relation Extraction from Biomedical Texts: An Empirical Study on Ten Benchmark Datasets (2022)

Abstract: Biomedical relation extraction, aiming to automatically discover high-quality and semantic relations between the entities from free text, is becoming a vital step for automated knowledge discovery. Pretrained language models have achieved impressive performance on various natural language processing tasks, including relation extraction. In this paper, we perform extensive empirical comparisons of encoder-only transformers with the encoder-decoder transformer, specifically T5, on ten public biomedical relation extraction datasets. We study the relation extraction task from four major biomedical tasks, namely chemical-protein relation extraction, disease-protein relation extraction, drug-drug interaction, and protein-protein interaction. We also explore the use of multi-task fine-tuning to investigate the correlation among major biomedical relation extraction tasks. We report performance (micro F-score) using T5, BioBERT and PubMedBERT, demonstrating that T5 and multi-task learning can improve the performance of the biomedical relation extraction task.

## introduction the scientific literature provides a rich source of biomedical knowledge (e.g., drug-drug interactions), and due to its rapid growth, it becomes increasingly difficult for scientists to keep up-to-date with the most recent discoveries hidden in literature @xcite . moreover, manual curation of information from biomedical literature is time-consuming, costly, and insufficient to keep up with the rapid growth of the literature @xcite . hence, there has been growing interest in using natural language processing (nlp) techniques for automatic relation extraction (re) between biomedical entities from texts. recently, a variety of approaches based on pretrained language models such as bert @xcite and other variants have shown promising results in various nlp tasks such as relation extraction @xcite , question answering @xcite , text summarization @xcite , and misinformation detection @xcite . in particular, re with classification-based encoder-only pretrained transformers (bert and variants) has been extensively studied @xcite @xcite . in contrast, re with pretrained language models based on encoder-decoder architecture, specifically textto-text transfer transformer (t5) @xcite , has not been well-studied. unlike encoderonly transformers, which are designed to predict a single prediction for an input sequence, t5 generates target tokens based on an encoder-decoder architecture. in this paper, our goal is to compare pretrained sequence-to-sequence transformers with the encoder-only transformers for re from biomedical texts. in order to satisfy this aim, we compare t5 with in-domain bert-based models such as biobert and pubmedbert on ten biomedical re benchmark datasets. we also explore the use of multi-task fine-tuning (mtft) on ten biomedical re datasets (each with different entities and relation types) to investigate the correlation among four major biomedical re tasks, namely chemicalprotein relation extraction, disease-protein relation extraction, drug-drug interaction, and proteinprotein interaction. our experiments show that t5 performs better than in domain bert-based models (encoder-only) such as biobert and pubmed-bert. the results also show that fine-tuning t5 with multi-task learning substantially improves the performance compared to single task fine-tuning. ## related work there has been a recent surge in interest from the nlp community to automatically extract re-lations between biomedical entities (proteins, gene, diseases, etc.) from the biomedical literature @xcite @xcite . recently, with the success of pretrained language models, several techniques based on transformers are widely utilized for extracting the relationships between entities from biomedical literature @xcite @xcite @xcite @xcite @xcite . the success of these systems has primarily been a result of encoder-only transformers such as bert @xcite and its variants like scibert @xcite , biobert @xcite , and pubmedbert @xcite . unlike re with classification-based encoder-only transformers which have been widely studied, re with encoder-decoder transformers has not been wellexplored. encoder-decoder-based transformer, specifically t5, @xcite has shown strong performance in various nlp tasks such as question answering and text summarization, etc. in this work, we perform comprehensive comparisons of encoder-only transformers with the encoder-decoder transformer, specifically t5, on ten public biomedical relation extraction datasets. we also explore the use of multi-task learning to learn the shared complementary features across multiple biomedical relation extraction datasets. ## conclusion in this paper, we present a comprehensive evaluation of encoder-only and encoder-decoder transformers on four benchmark biomedical re tasks. we also explored the use of mtft to investigate the correlation among these biomedical re tasks. for that, we used ten popular datasets, namely aimed, bioinfer, hprd50, iepa, lll, ddi, chemprot, drugprot, gad, and eu-adr. the experiments showed that t5 and mtft achieved better performance than bert-based models (biobert and pubmedbert) in extracting relations between bio-entities from texts. in the future, we plan to study the impact of each re task/dataset on the downstream performance.