EARA : Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation (2023)

Abstract: Measuring Semantic Textual Similarity (STS) is a fundamental task in biomedical text processing, which aims at quantifying the similarity between two input biomedical sentences. Unfortunately, the STS datasets in the biomedical domain are relatively smaller but more complex in semantics than common domain, often leading to overfitting issues and insufficient text representation even based on Pre-trained Language Models (PLMs) due to too many biomedical entities. In this paper, we propose EARA, an entity-aligned, attention-based and retrieval-augmented PLMs. Our proposed EARA first aligns the same type of fine-grained entity information in each sentence pair with an entity alignment matrix. Then, EARA regularizes the attention mechanism with an entity alignment matrix with an auxiliary loss. Finally, we add a retrieval module that retrieves similar instances to expand the scope of entity pairs and improve the model’s generalization. The comprehensive experiments reflect that EARA can achieve state-of-the-art performance on both in-domain and out-of-domain datasets. Source code is available.

## introduction biomedical semantic textual similarity (sts) involves measuring the similarity between biomedical texts, such as scientific articles, clinical notes, and electronic health records (ehrs). this measurement can be utilized to identify the relevant information, reduce excessive redundant information in ehrs @xcite @xcite , and improve the accuracy as well as the efficiency of evidence-based medicine (ebm) practices @xcite . unfortunately, naively applying plms to biomedical datasets may not yield promising results, owing to the greater semantic complexity of such datasets than those in the general domain. a sentence is deemed semantically intricate when it contains fine-grained information of multiple types, such as entities, which may distract the plms. for instance, in figure 1 (a), we observe a decline in the representational capacity of bluebert (as evidenced by an increase in the prediction error ratio) as the number of entities in sentences increases on the n2c2sts datasets. upon visualizing the at-tention matrix of the '[cls]' token and 'atenolol' token of a specific instance in the n2c2sts with baseline bluebert, we discover that the model tends to focus on punctuation and the sentence containing the selected token. besides, during the training process, we also observed that the pearson correlation coefficient (pcc) of models on the training set could often approach 1.0. however, as the pcc of models on the validation set increased, the pcc of models on the test set decreased. the reason might lie in that the size of the biomedical dataset is quite small compared to the size of the general-domain dataset. and the complexity of plms is also too excessive for the biomedical dataset, which can easily result in overfitting. in this paper, we propose eara, which improves the biomedical semantic textual similarity with entity-aligned attention and retrieval augmentation. eara first makes plms pay more attention to important entities by aligning the sametype entities in sentence pairs. secondly, eara builds a knowledge retriever on the mimic-iii database @xcite . thirdly, eara retrieves similar sentences to expand the scope of entity pairs and feeds them into the entity-aware transformer. finally, eara fuses the logits of retrievals and inputs to improve the model's generalization. the experiments show that eara can achieve state-of-the-art performance on multiple biomedical tasks. in summary, our work has the following contributions: in this section, we mainly introduce the related work from three perspectives: 1) biomedical sts, 2) plm-enhanced models beyond biomedical sts and 3) retrieval-based methods. ## discussion in this section, we conduct further analysis to better understand the results of the experiments. ## conclusion in this work, we present an entity-aligned and retrieval augmentation plm for the biomedical semantic textual similarity tasks. we propose an entity alignment matrix and an auxiliary loss to regularize the attention matrix of plms and introduce retrieval augmentation to improve the generalization ability of models further. the comprehensive experiments show that the eara achieves state-ofthe-art performance on the biomedical sts data. besides, eara also improves the performance of out-of-domain datasets.