Augmented Bio- SBERT : Improving Performance for Pairwise Sentence Tasks in Bio-medical Domain (2022)

Abstract: One of the modern challenges in AI is the access to high-quality and annotated data, especially in NLP; that is why augmentation is gaining importance. In computer vision, where image data augmentation is standard, text data augmentation in NLP is complex due to the high complexity of language. Moreover, we have seen the advantages of augmentation where there are fewer data available, which can significantly improve the model’s accuracy and performance. We have implemented Augmentation in Pairwise sentence scoring in the biomedical domain. By experimenting with our approach to downstream tasks on biomedical data, we have looked into the solution to improve Bi-encoders’ sentence transformer performance using an augmented dataset generated by cross-encoders fine-tuned on Biosses and MedNLI on the pre-trained Bio-BERT model. It has significantly improved the results with respect to the model only trained on Gold data for the respective tasks.

## introduction language models are data hungry; they consume massive amounts of data to identify patterns. for many niches, low-resource domains like that of bio domain nlp, manually finding or annotating a substantial dataset is complicated. bio-domain language models comparison @xcite . fortunately, we don't need to label this new data; we can automatically generate or label data using one or more data augmentation techniques. pairwise sentence scoring tasks are used widely in nlp applications. @xcite like information retrieval, question answering, duplicate question detection, or clustering. pre-trained transformers have led to remarkable progress in several tasks, especially bert @xcite is an approach that sets new state-of-the-art performance for many tasks, including pairwise sentence scoring. for tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: cross-encoders and bi-encoders. we pre-trained cross-encoders on the gold dataset, then outputs of different pairs from random sampling and semantic sampling were fed to the cross-encoder. the silver dataset produced was then provided to bi-encoder. there is a new approach like poly-encoder, which mostly fits tasks around conversational ai. we have used biobert, a biomedical language representation model designed for biomedical text mining tasks such as biomedical named entity recognition, relation extraction, question answering, etc. cross-and bi-encoders, details on pretrained model architecture have been discussed in section 4.2 and 4.3. we worked on two tasks for pairwise sentences-semantic similarity and language inferences based on medical data such as biosses and mednli. we have evaluated the results on the test set of each data set. in the end, we have compared our model results with results of textual similarity and inference tasks on a blue benchmark and have included the ## related work there have been many nlp-augmentation methods based on paraphrasing models and non-paraphrasing models. a survey of data augmentation approaches for nlp highlights techniques used for popular nlp applications and tasks, @xcite like mitigating biases and fixing class imbalance. augmented sbert: data augmentation method for improving bi-encoders for pairwise sentence scoring tasks have worked on sentence pair scor-ing through cross-encoder and biencoder on general language(english). but available language evaluation doesn't fit bio-medical uses, given it can't relate to the biomedical domain. probing biomedical embeddings from language models , (jin, 2019) this paper deals with the contrast in general, english and biomedical on any down-streaming tasks, and the results show that even pre-trained by in-domain corpus as a fixed feature extractor, biobert still cannot effectively encode biomedical relations compared to bert. bioelmo is significantly better than elmo in representing same relations closer to each other. augmentation with general language bert will perform poorly, as it has been compared in figure 1 . ## downstreaming tasks we have used two tasks for pairwise sentences: natural language inference and semantic textual similarity. the dataset for training and testing have been described in section 5.1. natural language inference (nli): is the task @xcite of determining whether a given hypothesis can be inferred from a given premise. we are using, mednli -a publicly available, expertly annotated dataset for nli in the clinical domain. semantic textual similarity: semantic textual similarity deals with determining how similar two pieces of texts are. related tasks are paraphrase or duplicate identification. we have used the biosses (gizem sogancioglu, 2017) dataset for the same. ## methods in this section, we will describe the pre-trained model used, the fine-tuning, cross-and bi-encoders, and the step-by-step method of getting the predictions from the bi-encoder sentence transformer after fine-tuning it on a silver dataset. we will also discuss the sampling techniques used for creating a silver dataset. 4.1 model pretrained model: biobert largely outperforms bert and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. due to this, we have decided to use this sentence transformer based on this model. sbert: given a pre-trained, well-performing cross-encoder, we sample sentence pairs according to a specific sampling strategy (discussed later) and label these using the cross-encoder. we call these weakly labeled examples the silver dataset, and will merge both with the gold training dataset. we then train the bi-encoder on this extended training dataset. we refer to this model as augmented sbert (augsbert). in figure 2 . we have illustrated the process of augmented sbert fine-tuning model: fine-tuning sentence transformer models requires pairs of labeled data, crossencoder model fine-tuned on gold dataset, and the bi-encoder fine-tuned on gold and silver data 1 . ## results the results section includes experimentation on the only gold dataset and the gold and silver dataset, using both methods, random sampling, and semantic similarity sampling, given in the depicting the silver dataset helps improve the model, produced by using sbert augmentation and mixed with gold labels. we have also compared it with the results of elmo and biobert results given in ## conclusion as language models get bigger, so do datasets. and although we have seen an explosion of data in the past decade, it is often not accessible, especially in niche domains like bio-domain. and there are datasets, like biosses, which has only 100 pairs of sentences. thus finding a substantial annotated dataset becomes difficult. sentence-bert augmentation can be used to improve pairwise sentence models and sentence similarity datasets. like in our case, it has improved results by up to 23.9 percent in the case of the biosses and 482 percent in case of mednli, as discussed in