B io S im CSE : B io M edical Sentence Embeddings using Contrastive learning (2022)

Abstract: Sentence embeddings in the form of fixed-size vectors that capture the information in the sentence as well as the context are critical components of Natural Language Processing systems. With transformer model based sentence encoders outperforming the other sentence embedding methods in the general domain, we explore the transformer based architectures to generate dense sentence embeddings in the biomedical domain. In this work, we present BioSimCSE, where we train sentence embeddings with domain specific transformer based models with biomedical texts. We assess our model’s performance with zero-shot and fine-tuned settings on Semantic Textual Similarity (STS) and Recognizing Question Entailment (RQE) tasks. Our BioSimCSE model using BioLinkBERT achieves state of the art (SOTA) performance on both tasks.

## introduction word embeddings or vector representations of words generated by neural network architectures, capture the semantic relationships between words much better than traditional methods such as one hot encoding, bag of words, and so on. when dealing with large texts in real-world situations, it is essential to capture the semantic relationship between words as well as between sentences. thus, rich sentence embeddings that capture the overall sentence semantics are critical for nlp systems. sentence embeddings play a significant role in various nlp tasks such as information retrieval, semantic search, intent detection, natural language inference tasks. in recent years, pre-trained models with transformer architecture for the general domain have grown in popularity. the advent of @xcite based models has made generating high-quality vector representations for natural language text much more manageable. these embeddings act as feature inputs for several down-stream tasks. however, these models only generate word-level embeddings, from which we can derive sentence-level embeddings by averaging over the word-level embeddings. another method is to use a cross encoder network from bert to directly fine-tune for the task.although this approach outperforms the averaging approach, it is computationally expensive and unsuitable for semantic similarity search and clustering tasks. the biomedical domain with its corpora, significantly different from the general domain corpora, needs sophisticated and domain-specific models for effective knowledge representation. in this paper, we adapt simcse @xcite , a state-of-theart contrastive learning-based sentence embedding method, and release biosimcse -a biomedical domain-specific sentence embedding model. in summary, our contributions are 1. we train and release foot_0 biomedical sentence embeddings with supervised and unsupervised training objectives from simcse. ## background transformer-based language representations produced by universal sentence encoder @xcite and bert has aided nlp practitioners and researchers in various nlp tasks. using bert, sentence embeddings can either be generated by averaging the context embeddings of the last few layers or from the output of the last layer. @xcite shows that sentence embeddings produced by averaging word-level embeddings from bert-like transformer models are unsuitable for standard similarity measurements such as cosine similarity. sbert uses the siamese network @xcite , a modified bert network for the generation of fixed-size sentence embeddings. though sbert significantly reduces the time during inference and produces quality sentence embeddings, it follows a supervised approach. it heavily relies on labelled data to train sentence embedding models that might not be suitable for domains without labelled corpora. natural language inference (nli) datasets are commonly used for supervised training of sentence embeddings models. the multi-genre natural language inference (multinli) @xcite corpus is mainly used to train general domain sentence embeddings. multinli has 433k sentence pairs that have textual entailment information annotated. in the biomedical domain, large corpora of text are publicly available as research papers and articles. however, the availability of annotated datasets is lower than that of the general domain, and the number of samples is also low. medical natural language inference (mednli) @xcite and radiology natural language inference ( radnli 2021 ) propose methods to generate sentence embeddings using a unsupervised approach with different training objectives. a domain like biomedical, where supervised datasets are unavailable, has to rely on unsupervised techniques. in this work, we selected simcse because its unsupervised training is comparable to that of its supervised competitors for training sentence embeddings; in addition, simcse performed better in our initial experiment with the other unsupervised training objectives described above. ## methods the training objective for simcse @xcite utilises the contrastive learning approach, which has a cross-entropy loss function with in-batch negatives. in unsupervised learning, positive pairs are made by giving the same sentence to the pre-trained encoder twice with regular dropout as noise, all other sentences in a mini-batch act as negative pairs. the nli dataset has a contradiction hypothesis for each premise and its entailment hypothesis. for supervised sentence embeddings training, simcse uses entailment pairs as positive cases and adds matching contradiction pairs and other sentences in the mini-batch as negatives. as in the original simcse implementation, we use the using biomedical corpora detailed in 3.1, we train biomedical domain-specific unsupervised and supervised sentence transformer models. the sentence embeddings training is done only with the model base architecture -12 layers of transformers block with a hidden dimension of 768 and multihead attention over 12 layers. hyper-parameters used for training are provided in appendix a. the trained sentence embeddings are then evaluated in zero-shot and fine-tuned settings on the three datasets outlined in 4.2. ## evaluation we use sts and rqe tasks in the biomedical domain to evaluate the performance of our biosim-cse sentence embeddings model. the datasets used for evaluation are detailed in 4.2. the similarity between two sentence embeddings is determined using cosine similarity. we determine a threshold in cosine similarity using the development set to classify entailment or not for rqe (binary classification) dataset. using spearman's correlation, we evaluate sts in line with the original simcse research. for rqe, accuracy is used. we evaluate biosimcse sentence embeddings under zero-shot and fine-tuned settings. ## results the biosimcse unsupervised biolinkbert model achieves remarkable results on all three datasets during the zero-shot evaluation. the zero-shot performs even better than biolinkbert fine-tuned with task-specific data using cross-encoder architecture. from the t-sne of sentence representation figure 2 , we can see that the similar sentence pairs (denoted by the same shapes) are closely aligned after training the biolinkbert model with simcse and also the average cosine similarity increased from 86.5 to 90.1 for the same. we can also observe that the transformer-based models have outperformed biosentvec chen et al. (2019), a nontransformer-based model with a large margin for both biosses and clinicalsts. biosentvec utilizes word vectors and n-grams approach to generate sentence embeddings using sent2vec pagliardini et al. (2018) model. from the results, we can see that the supervised training of simcse is not practical as the unsupervised training, as the biomedical domain has a limited no.of samples in the nli dataset. when fine-tuned with task-specific data biosimcse unsupervised biolinkbert model sets new state-of-the-art results for all three datasets. for biosses, spearman's correlation is improved by +4.62 points, compared to the previous sota of 91.75. for clinicalsts the current sota is by bioelectra kanakarajan et al. (2021) 82.11, biosimcse improve the sota by +1.65 points. biosimcse improve the rqe baseline 54.1 accuracy score abacha et al. (2019) by +5.94 points and sets a new sota. we have omitted the rqe sota result from panlp zhu et al. (2019) (accuracy of 74.9), as this score is achieved using multitask and ensemble methods. performance on the evaluation datasets has steadily improved for both biolinkbert and pub-medbert following training with simcse compared to the cross-encoder approach. ## conclusion in our work, we have explored simcse for training sentence embeddings in the biomedical domain. we utilize the publicly available biomedical literature and nli dataset for training the network in an unsupervised and supervised fashion and further fine-tune them with the task-specific datasets to adapt better to the task's requirements. our biosimcse model has achieved sota on all three evaluation datasets. our results demonstrate that simcse unsupervised training objectives can be able to train high-quality biomedical domainspecific sentence embeddings. we make the code and weights available for all of our models for reproducibility.