Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction (2022)

Abstract: Bi-encoder architectures for distantly-supervised relation extraction are designed to make use of the complementary information found in text and knowledge graphs (KG).However, current architectures suffer from two drawbacks. They either do not allow any sharing between the text encoder and the KG encoder at all, or, in case of models with KG-to-text attention, only share information in one direction. Here, we introduce cross-stitch bi-encoders, which allow full interaction between the text encoder and the KG encoder via a cross-stitch mechanism. The cross-stitch mechanism allows sharing and updating representations between the two encoders at any layer, with the amount of sharing being dynamically controlled via cross-attention-based gates. Experimental results on two relation extraction benchmarks from two different domains show that enabling full interaction between the two encoders yields strong improvements.

## introduction identifying semantic relations between textual mentions of entities is a key task for information extraction systems. for example, consider the sentence: assuming an inventory of relations such as may_treat or founded_by, a relation extraction (re) system should recognize the predicate in (1) as an instance of a may_treat relation and extract a knowledge graph (kg) triple like (aspirin, may_treat, pain). re systems are commonly trained on data obtained via distant supervision @xcite : given a kg triple, i.e., a pair of entities and a relation, one assumes that * equal contribution all sentences mentioning both entities express the relation and collects all such sentences as positive examples. ds allows collecting large amounts of training data, but its assumption is often violated: a common approach for dealing with noisy sentences is to use the kg as a complementary source of information. models taking this approach are typically implemented as bi-encoders, with one encoder for textual input and one encoder for kg input. they are trained to rely more on the text encoder when given informative sentences and more on the kg encoder when faced with noisy ones @xcite @xcite @xcite @xcite . however, current bi-encoder models suffer from drawbacks. bi-encoders that encode text and kg separately and then concatenate each encoder's output, as illustrated in figure 1a and proposed by @xcite , i.a., cannot share information between the text encoder and the kg encoder during encoding. in contrast, bi-encoders whose text encoder can attend to the kg encoder's hidden states, as illustrated in figure 1b and proposed by @xcite ; @xcite ; @xcite , i.a., do allow information to flow from the kg encoder to the text encoder, but not in the opposite direction. here, we propose a cross-stitch bi-encoder (xbe, figure 1c ) that addresses both of these drawbacks by enabling information sharing between the text encoder and kg encoder at arbitrary layers in both directions. intuitively, such a "full interaction" between the two encoders is desirable because it is not a priori clear at which point in the encoding process an encoder's representation is best-suited for sharing with the other encoder. concretely, we equip a bi-encoder with a cross-stitch component @xcite to enable bi-directional information sharing and employ a gating mechanism based on cross-attention @xcite to dynamically control the amount of information shared between the text encoder and kg encoder. as we will show, allowing bi-directional information sharing during the encoding process, i.e., at intermediate layers, yields considerable performance improvements. in summary, our contributions are: ## terminology and notation throughout this work we use terminology and notation as follows. we assume access to a domainspecific knowledge graph (kg) which contains fact triples @xmath0, r, e 2 ), ...} consisting of entities e 1 , e 2 ∈ e and a relation @xmath1. the set of entities e and the inventory of relations r are closed and finite. given a corpus of entity-linked sentences and kg triples (e k 1 , r k , e k 2 ), distant supervision (ds) yields a bag of sentences b @xmath2, ..., s k n } where each sentence s k i mentions both entities in the pair (e k 1 , e k 2 ). given the entity pair (e k 1 , e k 2 ) and the sentence bag b k , a ds-re model is trained to predict the kg relation r k . ## cross-stitch bi-encoder (xbe) the cross-stitch bi-encoder model is designed to enable bidirectional information sharing among its two encoders. as illustrated in figure 1c , it consists of a text encoder, a kg encoder, and a cross-stitch component controlled by cross-attention. the following subsections describe these components. ## xbe for relation extraction in distantly supervised relation extraction, the automatically collected data consists of a set of sentence bags {b 1 , ..., b n } and set of corresponding kg triples {(e 1 1 , r 1 , e 1 2 ), ..., (e n 1 , r n , e n 2 )}. to create training instances, we mask the relation in the kg triples {(e 1 1 , [m], e 1 2 ), ..., (e n 1 , [m], e n 2 )} and provide these masked triples as input to the kg encoder, while the text encoder receives one sentence from the corresponding sentence bag. if the sentence bag contains k sentences, we pair each sentence with the same kg triple and run the biencoder for each pairing, i.e., k times, to obtain a sentence bag representation. during training, the loss of the model is calculated via equations 9, 10 and 11, during inference, we follow @xcite and use the mean of sentence embeddings as the bag embedding: as our bi-encoder consists of two transformerbased encoders, we make use of pre-training for each modality. for the text encoder, we employ an off-the-shelf model, as detailed in the next section. the kg encoder is pre-trained on a set of kg triples via a relation prediction task. specifically, given a relation masked triple (e 1 , [m], e 2 ), the kg encoder is asked to predict the masked symbolic token and pre-trained via the loss given by equation 11. ## additional related work in this section we discuss related work besides the approaches already mentioned in the introduction. to improve the performance of a ds-re model, recently, researchers introduce various attention mechanisms. lin et al. (2016) ## limitations we focus only on one particular nlp task (i.e., ds-re) to explore the effective way to jointly encoding kg and text, and thus further work is required to determine to what extend the proposed xbe can be generalized into multiple nlp tasks. therefore, our work carries the limitation that the performance gain in ds-re does not guarantee that it is effective in other nlp tasks such as knowledge graph completion and question answering, where the combination of kg and text is needed. for this reason, we empathize the importance of multi-tasking for exploring such research question. in addition, we only utilize monolingual datasets to conduct evaluation and thus further work is required to investigate the effectiveness of the proposed model on multi-lingual datasets. ## conclusions and future work we proposed a cross-stitch bi-encoder architecture, xbe, to leverage the complementary relation between kg and text for distantly supervised relation extraction. experimental results on both medline21 and nyt10 datasets prove the robustness of our model because the proposed model achieves significant and consistent improvement as compared with strong baselines and achieve a new state-of-the-art result on the widely used nyt10 dataset. possible future work includes a more thorough investigation of how communication between kg encoder and text encoder influences the performance, as well as a more complex kg encoder that can not only handle relation triples, but arbitrary kg subgraphs, which could have applications in, e.g., multi-hop relation extraction.