G raph C ache: Message Passing as Caching for Sentence-Level Relation Extraction (2022)

Abstract: Entity types and textual context are essential properties for sentence-level relation extraction (RE). Existing work only encodes these properties within individual instances, which limits the performance of RE given the insufficient features in a single sentence. In contrast, we model these properties from the whole dataset and use the dataset-level information to enrich the semantics of every instance. We propose the GraphCache (Graph Neural Network as Caching) module, that propagates the features across sentences to learn better representations for RE. GraphCache aggregates the features from sentences in the whole dataset to learn global representations of properties, and use them to augment the local features within individual sentences. The global property features act as dataset-level prior knowledge for RE, and a complement to the sentence-level features. Inspired by the classical caching technique in computer systems, we develop GraphCache to update the property representations in an online manner. Overall, GraphCache yields significant effectiveness gains on RE and enables efficient message passing across all sentences in the dataset.

## introduction sentence-level relation extraction (re) aims at identifying the relationship between two entities mentioned in a sentence. re is crucial to the structural perception of human language, and also benefits many nlp applications such as automated knowledge base construction @xcite , event understanding @xcite , discourse understanding @xcite , and question answering @xcite . the modern tools of choice for re are the large-scale pretrained language models (plms) that are used to encode individual sentences, therefore obtaining the sentence-level representations @xcite @xcite . existing work considers entity types and textual context as essential properties for re @xcite @xcite . nonetheless, most existing re models only capture these properties locally within individual instances, while not globally modeling them from the whole dataset. given the insufficient features of a single sentence, it is beneficial to model these properties from the whole dataset and use them to enrich the semantics of individual instances. to overcome the aforementioned limitation, we propose to mine the entity and contextual information beyond individual instances so as to further improve the relation representations. particularly, we first construct a heterogeneous graph to connect the instances sharing common properties for re. this graph includes the sentences and property caches. each cache represents a property of entity types or contextual topics. we connect every sentence to the corresponding property caches (see fig. 1 ), and perform message passing over edges based on a graph neural network (gnn). in this way, the property caches aggregate the features from connected sentences, which will act as a complement to the sentence-level features and provide prior knowledge when identifying relations. the constructed graph connecting sentences has the same scale as the whole dataset, which leads to high computational complexity of the gnn. to address this issue, our idea is to view the message passing of gnns as data loading in computer systems, adapting the classical caching techniques to efficiently mining the property information from all sentences. we encapsulate this computational idea in a new gnn module, called graphcache @xcite , that uses an online updating strategy to refresh the property caches' representations. in addition, we design an attention-based global-local fusion module to augment the sentence-level representations using the property caches with adaptive weights. graphcache can be incorporated into popular re models to improve their effectiveness without increasing their time complexity, as analyzed in theory ( §3.2). as far as we know, ours is the first work to propagate the features across instances to enrich the semantics for sentence-level re. we evaluate graphcache on three public re benchmarks including tacred @xcite , semeval-2010 task 8 @xcite , and tacrev @xcite . empirical results show that graphcache consistently improves the effectiveness of popular re models by a significant margin and propagates features between all sentences in an efficient manner. graph neural networks for natural language processing. due to the large body of work on applying gnns to nlp, we refer readers to a recent survey @xcite for a general review. gnns have been explored in several nlp tasks such as semantic role labeling @xcite , machine translation @xcite , and text classification @xcite @xcite @xcite . gnns have also been widely adopted in various variants of relation extraction on the sentence level, @xcite @xcite , the document level @xcite @xcite , and the dialogue level @xcite . however, on the sentence-level relation extraction, most existing work @xcite @xcite uses the graph neural networks to encode the relation representations from individual instances instead of operating the message passing between instances. in contrast, we build a heterogeneous graph to connect the instances that share the properties for re, and design the caching updater to efficiently perform the message passing between instances. mary gave birth to jerry at the age of 21.' foot_0 , 'mary' and 'jerry' are the entities, the entity types are both person, and the ground-truth relation between 'jerry' and 'mary' is parent. we propose graphcache (graph neural networks as caching) as a message passing methodology to model the dataset-level property repre-sentations and use them to enrich every instance's semantics. graphcache creates a graph representation where sentences with shared property information are connected with property caches. graphcache first models the global semantic information by aggregating the features from the whole dataset, and then fuses the global and local features to augment the relational representations for every sentence. we analogize the message passing in gnns to caching in computer systems. caching is about loading data from high volume disks to low volume caches, so as to accelerate data loading. analogously, when gnns perform the message passing between sentences through a smaller number of bridge nodes, we can think of the massive sentences in the dataset as the disk data, and the properties, which aggregates the features from sentences, as caches. graphcache can be flexibly plugged into existing re models. as far as we know, ours is the first work to propagate the features between instances to enrich the semantics for re. graph-cache takes an existing re model as the backbone, e.g., bert, and takes the sentence-level representations given by the backbone as the inputs of message passing. a graphcache module consists of three key components: (i) a graph construction technique builds a few property caches. each cache represents a property for re: entity type or contextual topic. we connect each sentence to its corresponding properties, so that every property aggregates the features from its neighbor sentences. (ii) caching message passing aggregates the sentence-level representations to model the properties' representations in an online manner. (iii) global-local fusion fuses the global property representations and local sentence-level ones to augment the relation representations. next, we will discuss the three main components in more detail. ## experiments in this section, we evaluate the effectiveness of our graphcache method when incorporated into various re models. we compare our methods against a variety of strong baselines on the task of sentencelevel re. we closely follow the experimental setting of the previous work @xcite @xcite to ensure a fair comparison, as detailed below. ## conclusion in this paper, we study the efficient message passing to enhance the relation extraction models. we propose a novel method named graphcache, which provides efficient message passing between instances in the whole dataset. graphcache is a model-agnostic technique that can be incorporated into popular relation extraction models to enhance their effectiveness without increasing their time complexity. in our work, we present a simple yet effective implementation of graphcache, which models two universal and essential properties for relation extraction: entity information and textual context. our experimental results show that graphcache, with our heterogeneous graph, yields significant gains for the sentence-level relation extraction in an efficient manner.