Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning (2021)

Abstract: Though language model text embeddings have revolutionized NLP research, their ability to capture high-level semantic information, such as relations between entities in text, is limited. In this paper, we propose a novel contrastive learning framework that trains sentence embeddings to encode the relations in a graph structure. Given a sentence (unstructured text) and its graph, we use contrastive learning to impose relation-related structure on the token level representations of the sentence obtained with a CharacterBERT (El Boukkouri et al., 2020) model. The resulting relation-aware sentence embeddings achieve state-of-the-art results on the relation extraction task using only a simple KNN classifier, thereby demonstrating the success of the proposed method. Additional visualization by a tSNE analysis shows the effectiveness of the learned representation space compared to baselines. Furthermore, we show that we can learn a different space for named entity recognition, again using a contrastive learning objective, and demonstrate how to successfully combine both representation spaces in an entity-relation task.

## introduction pretrained language models (lms), such as bert @xcite , roberta @xcite and gpt-3 @xcite , capture contextualized information effectively and are used in a wide variety of natural language processing (nlp) tasks. they have revolutionized nlp research. the main mechanism of these models is multi-head self-attention @xcite , which enables capturing patterns of semantic and syntactic interest in text. however, their ability to encapsulate high level semantic information, such as relations in the text, and domain-specific knowledge, is limited because they are trained on very large corpora using the main objectives of language modeling. in many nlp tasks, pretrained lm embeddings are used as model input. a common strategy is to concatenate the embeddings that are extracted from different lms and let the model decide which part of the information is useful for the task. this empirical approach does not provide strong intuition and results in poor explainability capabilities because most of the task-specific models are black boxes. in this study, we present a novel contrastive learning (cl) framework to leverage the embedding space of characterbert and impose a relation structure on the embeddings. the proposed framework receives a sentence and a graph that represents the text relations in a structured way, and the cl paradigm is applied to impose this structure on the token embeddings of the character-bert text encoder. different graph formulations that represent the text relations are explored. the main goal is to create a common embedding space where relations can be easily detected. to evaluate progress towards this goal, we use the ade dataset @xcite , which is widely used for relation extraction (re) @xcite @xcite and named entity recognition (ner) tasks @xcite @xcite in the challenging field of information extraction (ie) from biomedical text. to evaluate the efficacy of our approach, a simple baseline neural network classifier for re, us-ing the pretrained characterbert medical version representations, is trained. the representations of the characterbert tuned version after applying cl are used to train the same classifier, which vastly outperforms the baseline classifier. a tsne @xcite analysis illustrates that meaningful relation-related clusters can be identified in the learned embedding space. this provides a second strong indication that structure can be effectively imposed on lm embeddings using our proposed framework. even if the main focus of this work is not solving the ie problem directly, to further explore the capabilities of the relation-aware representation space, we train a simple knn classifier for re that is competitive with state-of-the-art performance. strict evaluation @xcite of the re task presupposes correct detection of the boundaries and the entity type of each argument in the relation. hence, we apply the cl paradigm to learn a distinct embedding space for the entities and use a knn classifier to solve the ner task. finally, we perform a strict evaluation of the complete entity-relation extraction task. this transparent, computationally inexpensive and intuitively simple approach has comparable results to the state-of-the-art models. this achievement illustrates how informative and meaningful the learned embedding spaces are. in summary, our key contributions are: the paper is structured as follows. section 2 presents the ade dataset and the data preprocessing steps, and section 3 explains the framework in detail. in section 4, we evaluate the quality of the framework in baseline setups. the tsne analysis is presented in section 5. in section 6, we use the framework to solve the ie task and compare the results to state-of-the-art models. ## dataset this study focuses on biomedical text, and ade dataset is used. the sentences are annotated with labels for drugs and adverse effects, as well as the relations among these entities. adverse effects (aes) cover a range of signs, symptoms, diseases, disorders, abnormalities, organ damage and even death caused by that drug. the corpus is annotated at the sentence-level, so non-local relations (between entities of different sentences) do not exist. ## framework in essence, contrastive learning is a paradigm for learning representations which capture some auxiliary information by training them to distinguish positive from negative instances of this auxiliary information. our framework is inspired by the recent publications on image view-based cl of visual representation @xcite @xcite @xcite , but differs from the existing work by the application of cl to the graph and text modalities. our work is also inspired by the semantic bootstrapping hypothesis @xcite , which proposes that children acquire their native language through exposure to sentences of the language (i.e., a language model) paired with structured representations of their meaning @xcite . the main cl framework for imposing relationaware structure on the token embeddings is tested under two different settings. the difference in each setting is related to the modeling of the graph and the level of applying the cl paradigm. to solve the end-to-end ie task, a second model is proposed for learning a distinct embedding space where the named entities are projected. ## evaluation -baseline for the clgs model, the first evaluation step is a simple similarity check. we use the trained clgs model to extract the sentence representation and the positive and negative graph representations for all the sentences in the test set. then, a similarity check is applied using the extracted sentence and graph representations. the most similar graph is predicted as the positive sentence graph. given the positive and all the negative hard graphs extracted from each sentence, the model should be able to detect the correct graph. the different model variations perform well, but the mean pooling selection in the graph and sentence side results in better performance, as the accuracy is over 91%. the addition of the projection layers is not advantageous. graph pooling text pooling projection layer accuracy mean [cls] -88.39 mean mean -91.23 max max -89.1 mean [cls] yes 88.63 mean mean yes 87.68 the second evaluation step is applied to both models (clgs and cldr). following previous research on representation learning @xcite @xcite , we evaluate the tuned characterbert text encoder, taken from the trained clgs and cldr models, in a linear classification setting, where all the candidate relations (concatenation of the token embeddings) are created, and a linear classification layer is trained for the re task. as a baseline model, we use the pretrained medical character-bert to create the representation for the relations foot_5 . this linear setting directly provides insight into how successfully the relation-related structure is imposed at the token level of the text encoder, by evaluating the quality of the learned representations for re. ## tsne analysis a tsne analysis is performed to further explore the quality of the learned embedding spaces. using the tuned characterbert of the cldr model, the relation representation space is created. we project the positive (orange dots) and hard negative relations (blue dots), where one of the two relation tokens is correct. in the tsne plot (fig. 6 ), meaningful relation clusters can be easily identified, which demonstrates the efficiency of our framework (cldr model). the relation representations are asymmetric, as the drug and ae tokens have similar representations (fig. 7 ). this means that we cannot solve re and ner tasks using the same representation space. hence, we learn a different space for the named entities (clner model). figure 7: tsne plot -relation representation space obtained with characterbert of cldr model -named entities in the tsne plot in the entity representation space (fig. 8 ), we can detect insightful entity clusters. in particular, the clusters related to the drug tags (b-drug, i-drug) are very dense and well shaped. this is a strong finding that illustrates that the clner model can extract very good representations for the ner task. 6 entity-relation task the insights of the tsne analysis, with the welldefined clusters in the embedding spaces, lead us to approach the entity-relation task using intuitively simple and transparent knn classifiers. for the re task, we utilize the tuned characterbert of the cldr model to create the candidate relation representations. at the inference step, for each candidate relation, we decide whether it is positive based on the labels of the k-nearest neighbors in the learned embedding space. the value of k is chosen based on the performance in the randomly selected validation set (10% of training set) for each fold. we adopt the same strategy for the ner task using the clner model and project each token to the named entity representation space. to solve both ner and re tasks, we combine the two semantic spaces. first, we determine whether a candidate relation (concatenation of the tokens) is predicted as positive in the relation representation space, which is obtained by the tuned characterbert of the cldr model. then, we determine whether the boundaries and the types of the two entities in the candidate relation are predicted correctly in the entity representation space obtained by the clner model. all possible candidate relations and the named entities of the test set are classified. we strictly evaluate the performance of the ie task. as @xcite state, an entity is considered correct if its boundaries are detected correctly and the predicted type (drug or ae) matches the ground truth. in the same setup, a relation is considered correct if its type and the two entities (boundaries and type) involved in the relation are correctly predicted. we measure precision, recall and f1 score. following previous work on ie, we report the macro-averaged f1 score, and as 10-fold cross-validation is adopted, we average the scores over the folds. in the re task, we achieve very competitive results using a simple and transparent knn classifier. in contrast, the state-of-the-art models @xcite ) are very complex and computationally expensive. this fact highlights the high quality of the learned relation representation space (cldr model). in principle, the ner task is a sequence-tagging problem. however, we obtain good performance with a knn classifier that performs the inference in the learned entity representation space (clner model). notably, the last column of the above results reveal the quality of the representations for both ner and re tasks. hence, the proposed cl framework can be used as a preprocessing and representation learning step in the pipeline for ie models. the cl framework can be trained to leverage the embedding space and create meaningful, disentangled representations for the ie task. we successfully evaluated the representations with a simple knn classifier, but the learned representations can be used as input in complex models for entity and relation classification to achieve better results and faster convergence. we will explore this research direction in the future. ## conclusion we present a novel cl framework, which, in principle, is text encoder-agnostic, for effectively imposing relation-related structure to lms and leveraging the embedding space. we evaluate the quality of the learned representations using relative baselines and competitively solve an entity-relation task. the overall results indicate that the learned representations are very powerful. the performed tsne analysis illustrates that meaningful clusters can be easily identified in the learned embedding spaces. we note that the proposed framework can be used as a representation learning step for complex ie systems. in future work, we intend to explore the capabilities of our approach in continual learning settings and exploit external graph structured knowledge in representation learning of language data.