Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation (2022)

Abstract: Document-level Relation Extraction (DocRE) is a more challenging task compared to its sentence-level counterpart. It aims to extract relations from multiple sentences at once. In this paper, we propose a semi-supervised framework for DocRE with three novel components. Firstly, we use an axial attention module for learning the interdependency among entity-pairs, which improves the performance on two-hop relations. Secondly, we propose an adaptive focal loss to tackle the class imbalance problem of DocRE. Lastly, we use knowledge distillation to overcome the differences between human annotated data and distantly supervised data. We conducted experiments on two DocRE datasets. Our model consistently outperforms strong baselines and its performance exceeds the previous SOTA by 1.36 F1 and 1.46 Ign_F1 score on the DocRED leaderboard.

## introduction the problem of document-level relation extraction foot_0 (docre) is highly important for information extraction and nlp research. the docre task aims to extract relations among multiple entities within a document. the docre task is more challenging than its sentence-level counterpart in the following aspects: (1) the complexity of docre increases quadratically with the number of entities. if a document contains n entities, classification decisions must be made on n(n -1) entity pairs and most of them do not contain any relation. (2) aside from the imbalance of positive and negative examples, the distribution of relation types for the positive entity pairs is also highly imbalanced. considering the docred @xcite dataset as an example, there are 96 relation types in total, where the top 10 relations take up 59.4% of all the relation labels. this imbalance significantly increases the difficulty of the document-level re task. most existing approaches of docre leverage dependency information to construct a documentlevel graph @xcite , and then use graph neural networks for reasoning. another popular strand of this field uses transformer-only @xcite architecture @xcite @xcite . such models are able to achieve state-of-theart performance without explicit graph reasoning, showing that pre-trained language models (prlms) are able to implicitly capture long-distance relationships. however, there are three limitations of the existing docre methods. firstly, existing methods mainly focus on the syntactic features from prlms while neglecting the interactions between entity pairs. @xcite and @xcite have used cnn structure to encode the interaction between entity pairs, but cnn structure cannot capture all the elements within the two-hop reasoning paths. secondly, there is no prior work that explicitly tackles the class-imbalance problem for docre. existing works @xcite @xcite ) only focus on threshold learning for balancing the positive and negative examples, but the class-imbalance problem within positive examples is not addressed. lastly, there are very few works discussing the method of adapting distantly supervised data for the docre task. @xcite has shown that distantly supervised data is able to improve the performance of document-level relation extraction. however, it only uses the distantly supervised data to pre-train the re model in a naive manner. to overcome the limitations of existing works, we propose a semi-supervised learning framework for document-level relation extraction. firstly, to improve the reasoning for two-hop relations, we propose to use an axial attention module as feature extractor. this module enables us to attend to elements that are within two-hop logical paths and capture the interdependency among the relation triplets. secondly, we propose adaptive focal loss to address the imbalanced label distribution problem. the proposed loss function encourages the long-tail classes to contribute more to the overall loss. lastly, we use knowledge distillation to overcome the differences between the annotated data and the distantly supervised data. specifically, we first train a teacher model with a small amount of human annotated data. the teacher model will then be used to generate predictions on a large amount of distantly supervised data. the generated predictions are used as soft labels for pre-training our student model. finally, the pre-trained student model is further fine-tuned on the human annotated data. we conducted experiments on two datasets -the docred @xcite dataset and the ha-cred @xcite dataset. experimental results show that our model consistently outperforms competitive baselines. moreover, our model significantly outperforms the existing state-of-theart ssan-adapt @xcite on the docred leaderboard by 1.36 in f1 score and 1.46 in ign_f1 score. 3 besides, we provide a thorough ablation study and error analysis to identify the bottleneck of our method. ## error analysis even though our final model significantly outperforms the previous state of the art on the do-cred leaderboard, the absolute performance of our model still does not match human performance. in this section, we provide a detailed error analysis of our model on the development set of docred. ground truth predictions @xmath0: 8,273 (51.4%) mr: 3,814 (23.7%) w: 242 (1.5%) nr ms: 3,761 (23.4%) 380,703 we first construct the union of our model's predictions and the ground truth triples (without nr label). then, we categorize the union into four categories: (1) correct (c), where prediction triples are in the ground truth. (2) wrong (w), where the predicted head entity and tail entity are in the ground truth but the predicted relation is wrong. ## related work early works on relation extraction mainly focused on sentence-level re @xcite @xcite . however, prior works have shown that a large number of relations can only be extracted from multiple sentences @xcite @xcite . various methods have been proposed to tackle document-level relation extraction (docre). graph neural networks (gnns; @xcite have been widely used for the docre task. quirk and poon (2017) used words as nodes and dependency information as edges to construct document-level graphs. this graph will be used to extract features for each entity pair. later works extended this idea by applying different gnn architectures @xcite @xcite @xcite . in particular, @xcite proposed the latent stucture refinement (lsr) model, which used structured attention to induce the document-level graph. @xcite constructed the document-level graph by entity-mention nodes and sentence edges. besides the graph-based methods, transformer-only architectures have also proven to be highly effective for the docre task @xcite . specifically, @xcite proposed adaptive thresholding loss to tackle the multi-label classification problem in docre. on the other hand, learning from distant supervision is another important problem for relation extraction. @xcite used generative adversarial training for selecting informative examples @xcite used reinforcement learning to achieve the same goal. however, there are no existing works that jointly learn from annotated data and distant data. to this end, this paper is the first to overcome the differences between the human annotated and distantly supervised data. moreover, this paper also tackles the under-explored class imbalance problem and the two-hop logical reasoning problem with novel solutions to the shortcomings of existing approaches. ## conclusions in this paper, we have proposed a novel framework for document-level relation extraction, based on knowledge distillation, axial attention, and adaptive focal loss. our proposed method is able to significantly outperform the previous state of the art on the docred leaderboard. besides, we also conducted a thorough ablation study and error analysis to identify the bottleneck of the document-level relation extraction task.