Learning Informative Representations of Biomedical Relations with Latent Variable Models (2020)

Abstract: Extracting biomedical relations from large corpora of scientific documents is a challenging natural language processing task. Existing approaches usually focus on identifying a relation either in a single sentence (mention-level) or across an entire corpus (pair-level). In both cases, recent methods have achieved strong results by learning a point estimate to represent the relation; this is then used as the input to a relation classifier. However, the relation expressed in text between a pair of biomedical entities is often more complex than can be captured by a point estimate. To address this issue, we propose a latent variable model with an arbitrarily flexible distribution to represent the relation between an entity pair. Additionally, our model provides a unified architecture for both mention-level and pair-level relation extraction. We demonstrate that our model achieves results competitive with strong baselines for both tasks while having fewer parameters and being significantly faster to train. We make our code publicly available.

## introduction the vast amounts of scientific literature can provide a significant source of information for biomedical research. using this literature to identify relations between entities is an important task in various applications @xcite @xcite . existing approaches to biomedical relation extraction usually fall into one of two categories. mention-level extraction aims to classify the relation between a pair of entities within a short span of text (usually a sentence). in contrast, pair-level extraction aims to classify the relation between a pair of entities across an entire paragraph, document or corpus. for both mention-level and pair-level relation extraction, recent work has been focused on representation learning. this is considered to be one of the major steps towards making progress in artificial intelligence @xcite . representations of relations which understand their context are particularly important in biomedical research, where identifying fruitful targets is crucial due to the high costs of experimentation. learning such representations is likely to require large amounts of unsupervised data due to the scarcity of labelled data in this domain. recent mention-level methods have been based on using large unsupervised models with transformer networks @xcite to learn representations of sentences containing pairs of entities. these representations are then used as the inputs to much smaller models, which perform supervised relation classification @xcite . recent pair-level methods have been based on encoding each mention of a pair of entities, and designing a mechanism to pool these encodings (across a paragraph, document, or corpus) into a single representation. this representation is then used to classify the relation between the entity pair @xcite . however, representation learning methods for both mention-level and pair-level extraction typically use a point estimate for each representation. as a result, they may struggle to capture the nature of the true, potentially complex relations between each pair of entities. for example, figure 1 shows sentences for two entity pairs which demonstrate that relation statements can be very different, typically depending on biological circumstances (e.g. anatomical location, experimental details, presence of a disease, etc). such nuanced relations can be difficult to capture with a single point estimate. protein akt and protein gsk3β: ". . . akt negatively regulates gsk3β activity. . . " ". . . akt phosphorylates gsk3β. . . " protein eaat2 and disease als: "eaat2/c1-4 were found to be equally expressed in als patients and controls." "eaat2 protein is significantly reduced in als in the motor cortex and spinal cord." figure 1: two sets of sentences demonstrating the potentially complex nature of the relation between a pair of entities. relation for each entity pair, and that this relation can be multimodal (because of the aforementioned complexities). the sentences containing each pair are textual observations of these underlying relations. we therefore propose a probabilistic model which uses a continuous latent variable to represent the true relation between each entity pair. the distribution of a sentence containing that pair is then conditioned on this latent variable. in order to be able to model the complex relations between each entity pair, we use an infinite mixture distribution for the latent representation. our model provides a unified architecture for learning representations of relations between entity pairs both at mention and pair level. we show that (an approximation to) the posterior distribution of the latent variable can be used for mention-level relation classification. we also demonstrate that the prior distribution from the same model can be used for pair-level classification. on both tasks, we achieve results competitive with strong baselines with a model which has fewer parameters and is significantly faster to train. the code is released at https://github.com/ benevolentai/relvm ## model in this section, we introduce our unified architecture for both mention-level and pair-level relation extraction. throughout, we use the following notation: approach large corpora of labelled relation statements are often scarce, whereas unlabelled sentences are usually plentiful. in order to leverage these unlabelled sentences, we first train an unsupervised model to learn representations of entity pairs and the contexts in which they occur. we then train much smaller models to classify relations using the representations from the unsupervised model. ## related work mention-level relation extraction is typically performed using supervised learning. in the general domain, @xcite combine an lstm with a position-aware attention mechanism to perform multiclass relation extraction. soares et al. (2019) fine-tune the bert @xcite architecture to relation extraction tasks by enforcing similarity between representations of sentences containing the same pair of entities across a corpus. @xcite ) construct a teacher model to generate soft labels which guide the optimisation of a student network via knowledge distillation. in the biomedical and scientific domains, biobert @xcite and scibert @xcite train the bert architecture on domain-specific corpora, achieving state of the art results on mentionlevel relation extraction tasks. @xcite combine an rnn over the sentence's words and a cnn over its dependency graph to classify drugdrug and protein-protein interactions. pair-level relation extraction usually relies on distant supervision @xcite . in the general domain, @xcite develop a latent variable model to perform multi-instance learning while handling overlapping relations. @xcite use an attention mechanism to pool the representations of sentences containing a given pair into a single representation, which is then used as the input to a classifier. @xcite poon (2017 contrary to our work, there does not appear to be prior research performing both mention-level and pair-level relation extraction with a unified model. ## conclusion we have presented a model for learning representations of pairs of biomedical entities from unlabelled text corpora. we use a latent variable with an arbitrarily flexible distribution in order to be able to capture the complex relations between each pair of entities. the unified architecture can be used for both mention-level and pair-level relation extraction. on both tasks, we achieve results competitive with strong baselines. we also show significant computational gains in terms of the number of parameters and training times. our model presents many avenues for future work. the results in other directions include evaluating the benefits of having a representation which explicitly captures uncertainty about the relations. for example, this can be done by assessing if the model is less confident when making predictions about entity pairs which do not occur frequently in the unlabelled corpus. additionally, since our model can produce a representation for any pair of entities (even those which do not occur together in the unlabelled corpus), it could be used in a link prediction setting to score unseen entity pairs.