DREEAM : Guiding Attention with Evidence for Improving Document-Level Relation Extraction (2023)

Abstract: Document-level relation extraction (DocRE) is the task of identifying all relations between each entity pair in a document. Evidence, defined as sentences containing clues for the relationship between an entity pair, has been shown to help DocRE systems focus on relevant texts, thus improving relation extraction. However, evidence retrieval (ER) in DocRE faces two major issues: high memory consumption and limited availability of annotations. This work aims at addressing these issues to improve the usage of ER in DocRE. First, we propose DREEAM, a memory-efficient approach that adopts evidence information as the supervisory signal, thereby guiding the attention modules of the DocRE system to assign high weights to evidence. Second, we propose a self-training strategy for DREEAM to learn ER from automatically-generated evidence on massive data without evidence annotations. Experimental results reveal that our approach exhibits state-of-the-art performance on the DocRED benchmark for both DocRE and ER. To the best of our knowledge, DREEAM is the first approach to employ ER self-training.

## introduction document-level relation extraction (docre) has been recognized as a more realistic and challenging task compared with its sentence-level counterpart @xcite @xcite . in docre, an entity can have multiple mentions scattered throughout a document, and relationships can exist between entities in different sentences. therefore, docre models are expected to apply information filtering to long texts by focusing more on sentences relevant to the current decision of relation extraction (re) and less on irrelevant ones. to this end, existing studies retrieve supporting evidence (evidence hereafter, yao et al., edmund, faced with the threat of assassination, attempts to escape to france into self-imposed exile; and in a later scene, two drunk knights overhear king richard iv exclaiming "who will rid me of this turbulent priest?" @xcite the words attributed to king henry ii which led to becket's death in 1170, and embark on a mission to murder edmund. @xcite … the archbishop subject: prince edmund object: blackadder relation: present in work evidence: 1,2 figure 1 : example document and one of the relation triples from docred, where the i-th sentence is marked with @xcite in the beginning. mentions in bold italics are those of subjects and objects, whereas entity mentions other than subject and object are underlined. 2019), a set of sentences necessary for humans to identify the relation between an entity pair @xcite @xcite . as shown in figure 1 , to decide the present in work relation between prince edmund and blackadder, reading sentences 1 and 2 should be sufficient. although sentences 5 and 6 also mention the subject, they are irrelevant to the relation decision. evidence of the relation triple (prince edmund, present in work, blackadder) is thus defined as sentences 1 and 2. despite the usefulness of evidence, automatic evidence retrieval (er) faces two major issues. firstly, the existing approaches for er are memoryinefficient. previous systems tackle er and docre as separate tasks, introducing extra neural network layers to learn er with docre jointly @xcite @xcite . the er module typically involves a bilinear classifier that receives entity-pair-specific embeddings and sentence embeddings as the input. to compute the evidence score of each sentence for each entity pair, the module must walk through all (entity pair, sentence) combinations. the computations signifi-cantly increase memory consumption, particularly in documents with numerous sentences and entities. secondly, the availability of human annotations of evidence is limited. to make matters worse, gold training data for docre are more expensive to annotate than those for their sentence-level counterpart. despite the difficulty of obtaining human annotations, acquiring evidence annotations at a low cost has been underexplored. although automatically collecting silver training data for re by distant supervision @xcite , locating evidence for a sliver re instance in the document is nontrivial. this work aims at alleviating these issues to improve the usage of er in docre. to reduce the memory consumption, we propose documentlevel relation extraction with evidence-guided attention mechanism (dreeam), a memoryefficient approach for incorporating docre with er. we adopt atlop @xcite , a transformer-based docre system widely used in previous studies @xcite @xcite , as the backbone. instead of introducing an external er module, we directly guide the docre system to focus on evidence. specifically, we supervise the computation of entity-pairspecific local context embeddings. the local context embedding, formed as a weighted sum among all token embeddings based on attention from the encoder, is trained to assign higher weights to evidence and lower weights otherwise. to compensate for the shortage of evidence annotations, we propose performing er under a weaklysupervised setting. specifically, we design a strategy to perform self-training with dreeam on massive, unlabeled data. the data is obtained from distant supervision (distantly-supervised data hereafter) and thus is automatically annotated with relation labels but not evidence labels. we expect the knowledge about er learned from the human-annotated data to generate and grow on the distantly-supervised data. to enable self-training, we first adopt a teacher model trained on humanannotated data to retrieve silver evidence from distantly-supervised data. next, we train a student model on the data for re while learning er from the silver evidence. the student model is further finetuned on the human-annotated data to refine its knowledge. experiments on the docred benchmark @xcite show that with the help of er self-training, dreeam exhibits state-of-the-art performance on both re and er. in short, the contributions of this work are: (1) we propose dreeam, a memory-efficient approach to incorporate evidence information into transformer-based docre systems by directly guiding the attention. dreeam does not introduce any extra trainable parameters for er while achieving good performance on both re and er. ## proposed method: dreeam to perform information filtering, atlop computes a localized context embedding based on attention weights from the transformer-based encoder. the rationale is that cross-token dependencies are encoded as attention weights in transformer layers. in this work, we propose dreeam to enhance atlop with evidence. in addition to the automatically-learned cross-token dependencies, the attention modules are supervised to concentrate more on evidence sentences and less on others. dreeam can be employed for both supervised and self-training, sharing the same architecture with different supervisory signals, as shown in figure 2 first, we train a teacher model on human-annotated data with gold relations and evidence labels. next, we apply the trained teacher model to predict silver evidence for evidence 𝑒 ! 𝑟 ! 𝑒 " [1] 𝑒 ! 𝑟 " , 𝑟 # 𝑒 # [2,3] 𝑒 # 𝑟 ! 𝑒 @xmath0 𝑟 @xmath1 ] supervise (a) model architecture of dreeam. silver relation teacher model student model silver relation silver evidence gold relation gold evidence distantly-supervised data human-annotated data (2) (1) (2) (3) (4) the distantly-supervised data. then, we train a student model on the distantly-supervised data, with er supervised by the silver evidence. finally, we finetune the student model on the human-annotated data to refine its knowledge. the rest of this section introduces the training processes of the teacher and student models, followed by the inference process. ## experiments to evaluate dreeam, we conduct experiments under supervised and weakly-supervised settings. ## conclusion we have introduced methods to improve the usage of er in docre. first, we propose dreeam, a memory-efficient method to reduce the computation cost of er. unlike existing approaches that train an evidence classifier for er, dreeam directly supervises the attention to concentrate more on evidence than on others. next, we propose to employ dreeam in a weakly-supervised setting to compensate for the shortage of human annotations. instead of gold evidence annotated by humans, we adopt evidence predictions from a teacher model trained on human-annotated data as the supervisory signal to realize er self-training on unlabeled data. experiments on the docred benchmark show that dreeam exhibits state-of-the-art performance on both re and er, with the help of weakly-supervised training on data obtained from distant supervision of relations. compared with existing approaches, dreeam performs er with zero trainable parameters introduced, thereby reducing the memory usage to 27% or less. the soundness of dreeam is confirmed by conducting experiments on re-docred, a revised version of docred. in the future, we plan to transfer the evidence knowledge of dreeam trained on docred to other docre datasets. mike mintz, steven bills, rion snow, and daniel jurafsky. 2009. distant supervision for relation extraction without labeled data. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp, pages 1003-1011, suntec, singapore. association for computational linguistics. nanyun peng, hoifung poon, chris quirk, kristina toutanova, and wen-tau yih. 2017. cross-sentence n-ary relation extraction with graph lstms. transactions of the association for computational linguistics, 5:101-115. chris quirk and hoifung poon. 2017. distant supervision for relation extraction beyond the sentence boundary. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 1, long papers, pages 1171-1182, valencia, spain. association for computational linguistics. olaf ronneberger, philipp fischer, and thomas brox. 2015. u-net: convolutional networks for biomedical image segmentation. in medical image computing and computer-assisted intervention -miccai 2015, pages 234-241, cham. springer international publishing. qingyu tan, ruidan he, lidong bing, and hwee tou ng. 2022a. document-level relation extraction with adaptive focal loss and knowledge distillation. in findings of the association for computational linguistics: acl 2022, pages 1672-1681, dublin, ireland. association for computational linguistics. qingyu tan, lu xu, lidong bing, hwee tou ng, and sharifah mahani aljunied. 2022b. revisiting docred -addressing the false negative problem in relation extraction.