Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction (2021)

Abstract: Contrastive learning has been used to learn a high-quality representation of the image in computer vision. However, contrastive learning is not widely utilized in natural language processing due to the lack of a general method of data augmentation for text data. In this work, we explore the method of employing contrastive learning to improve the text representation from the BERT model for relation extraction. The key knob of our framework is a unique contrastive pre-training step tailored for the relation extraction tasks by seamlessly integrating linguistic knowledge into the data augmentation. Furthermore, we investigate how large-scale data constructed from the external knowledge bases can enhance the generality of contrastive pre-training of BERT. The experimental results on three relation extraction benchmark datasets demonstrate that our method can improve the BERT model representation and achieve state-of-the-art performance. In addition, we explore the interpretability of models by showing that BERT with contrastive pre-training relies more on rationales for prediction. Our code and data are publicly available at: https://github.com/AnonymousForNow .

## introduction contrastive learning is a family of methods to learn a discriminative model by comparing input pairs @xcite . the comparison is performed between positive pairs of "similar" inputs and negative pairs of "dissimilar" inputs. the positive pairs can be generated in an automatic way by transforming the original data to variants without changing the key information (e.g., rotate an image). contrastive learning can encode general properties (e.g. invariance) in the learned representation while it is relatively hard for other representation learning methods to achieve (bengio et al., 1 these authors contributed equally. 2013; @xcite . therefore, contrastive learning provides a powerful approach to learn representations in a self-supervised manner and has shown great promise and achieved the state of the art results in recent years @xcite . despite its advancement, contrastive learning has not been well studied in biomedical natural language processing (bionlp), especially for relation extraction (re) tasks. one obstacle lies in the discrete characteristics of text data. compared to computer vision, it is more challenging to design a general and efficient data augmentation method to construct positive pairs. instead, there have been significant advances in the development of pretrained language models to facilitate downstream bionlp tasks @xcite @xcite . therefore, leveraging contrastive learning in the large pre-trained language models to learn more general representation for re tasks remains unexplored. to bridge this gap, this paper presents an innovative method of contrastive pre-training to improve the language model representation for biomedical relation extraction. as the main difference from the existing contrastive learning framework, we augment the datasets for re tasks by randomly changing the words that do not affect the relation expression. here, we hypothesize that the shortest dependency path (sdp) between two entities @xcite captures the required knowledge for the relation expression. we hence keep words on sdp fixed during the data augmentation. in addition, we utilize external knowledge bases to construct more data to make the learned representation generalize better, which is a method that is frequently used in distant supervision @xcite . to verify the effectiveness of the proposed method, we use the transformer-based bert model as a backbone @xcite and evaluate our method on three widely studied re tasks in the biomedical domain: the chemical-protein interactions (chemprot) @xcite , the drug-drug interactions (ddi) @xcite , and the protein-protein interactions (ppi) @xcite . the experimental results show that our method boosts the bert model performance and achieves state-of-the-art results on all three tasks. interest has also grown in designing interpretable bionlp models that are both plausible (accurate) and rely on a specific part of the input (faithful rationales) @xcite . here rationale is defined as the supporting evidence in the inputs for the model to make correct predictions. in this direction, we propose a new metric, "prediction shift", to measure the sensitivity degree to which the small changes (out of the sdp) of the inputs will make model change its predictions. we show that the contrastively pre-trained model is more robust than the original model, suggesting that our model is more likely to make predictions based on the rationales of the inputs. in sum, the contribution of this work is fourfold. ( 1 ) we propose a new method that utilizes contrastive learning to improve the bert model on biomedical relation extraction tasks. (2) we utilize external knowledge to generate more data for learning more generalized text representation. (3) we achieve state-of-the-art performance on three benchmark datasets of relation extraction tasks. (4) we propose a new metric that aims to reveal the rationales that the model uses for predicting relations. the code and the new rationale test datasets are available at https://github. com/udel-biotm-lab/bert-clre . ## related work the history of contrastive representation learning can be traced back to @xcite , in which the authors explore the method of representation learning that similar inputs are mapped to nearby points in the representation space. recently, with the development of data augmentation techniques, deep neural network architectures, contrastive learning regains attention and achieves superior performance on visual representation learning @xcite . in @xcite , the momentum contrast (moco) framework is designed to learn representation using the mechanism of dictionary look-up: an encoded example (the query) should be similar to its matching key (augmented sample from the same data example) and dissimilar to others. in @xcite , the authors propose the simclr frame to learn the representations by maximizing the agreement between augmented views of the same data point. the contrastive representation has all the properties that a good representation should have: 1) distributed property; 2) abstraction and invariant property; 3) disentangled representation @xcite . the distributed property emphasizes the expressive aspect of the representation (different data points should have distinguishable representations). the capture of abstract concepts and the invariance to small and local changes are concerned in the abstraction and invariant property. from the disentangled representation's perspective, it should encode as much information as possible. in this work, we will show contrastive learning can improve the invariant aspect of the representation. in the natural language processing (nlp) field, several works have utilized the contrastive learning technique. @xcite propose a pre-trained language representation model (cert) using contrastive learning at the sentence level to benefit the language understanding tasks. @xcite employ contrastive self-supervised learning to solve the commonsense reasoning problem. peng et al. (2020) propose a self-supervised pretraining framework for relation extraction to explore the encoded information for the textual context and entity type. compared with the previous works, we employ different data augmentation techniques and utilize data from external knowledge bases in contrastive learning to improve the model for relation extraction tasks. relation extraction is usually seen as a classification problem when the entity mentions are given in the text. many different methods have been proposed to solve the relation extraction problem @xcite @xcite @xcite . however, the language model methods redefine this field with their superior performance @xcite @xcite @xcite ). among all the language models, bert @xcite -a language representation model based on bidirectional transformer @xcite , attracts lots of attention in different fields. several bert models have been adapted for biomedical domain: biobert @xcite , scibert @xcite , blue-bert @xcite and pubmedbert @xcite . biobert, scibert and bluebert are pre-trained based on the general-domain bert using different pre-training data. in contrast, pub-medbert @xcite is pre-trained from scratch using pubmed abstracts. in recent years, there is increasing interest in designing more interpretable nlp models that reveal the logic behind model predictions. in (deyoung et al., 2020), multiple datasets of rationales (from human experts) are collected to facilitate the research on interpretable models in nlp. in @xcite , the authors propose an encoder-generator framework to automatically generate candidate rationales to justify the predictions of neural network models. ## experiments as discussed before, we will utilize the bert model as the encoder for the inputs. in particular, we will employ two bert models pre-trained for the biomedical domain in our experiments: biobert @xcite and pubmedbert @xcite . ## conclusion and future directions in this work, we propose a contrastive pre-training method to improve the text representation of the bert model. our approach differs from previous studies in the choice of text data augmentation with linguistic knowledge and the use of the external knowledge bases to construct large-scale data to facilitate contrastive learning. the experimental results demonstrate that our method outperforms the original bert model on three relation extraction benchmarks. additionally, our method shows robustness to slightly changed inputs over the bert models. in the future, we will investigate different settings of data augmentation and contrastive pre-training to exploit their capability on language models. we also hope that our work can inspire researchers to design better metrics and create highquality datasets for the exploration of model interpretability.