Distantly-Supervised Joint Extraction with Noise-Robust Learning (2024)

Abstract: Joint entity and relation extraction is a process that identifies entity pairs and their relations using a single model. We focus on the problem of joint extraction in distantly-labeled data, whose labels are generated by aligning entity mentions with the corresponding entity and relation tags using a knowledge base (KB). One key challenge is the presence of noisy labels arising from both incorrect entity and relation annotations, which significantly impairs the quality of supervised learning. Existing approaches, either considering only one source of noise or making decisions using external knowledge, cannot well-utilize significant information in the training data. We propose DENRL, a generalizable framework that 1) incorporates a lightweight transformer backbone into a sequence labeling scheme for joint tagging, and 2) employs a noise-robust framework that regularizes the tagging model with significant relation patterns and entity-relation dependencies, then iteratively self-adapts to instances with less noise from both sources. Surprisingly, experiments on two benchmark datasets show that DENRL, using merely its own parametric distribution and simple data-driven heuristics, outperforms strong baselines by a large margin with better interpretability.

## introduction joint extraction aims to detect entities along with their relations using a single model (see figure 1 ), which is a critical step in automatic knowledge base construction @xcite . in order to cheaply acquire a large amount of labeled joint training data, distant supervision (ds) @xcite was proposed to automatically generate training data by aligning knowledge base (kb) with an unlabeled corpus. it assumes that if an entity pair has figure 1 : an example of joint extraction on a sentence with multiple relations that share the same entity, e.g., "microsoft" in both the third and the forth relations. a relationship in a kb, all sentences that contain this pair express the corresponding relation. nevertheless, ds brings plenty of noisy labels which significantly degrade the performance of the joint extraction models. for example, given a sentence "bill gates lived in albuquerque" and the sentence in figure 1 , ds may assign the relation type between "bill gates" and "albuquerque" as place_lived for both sentences. the words "lived in" in the first sentence is the pattern that explains the relation type, thus it is correctly labeled. while the second sentence is noisy due to the lack of corresponding relation pattern. moreover, due to the ambiguity and limited coverage over entities in open-domain kbs, ds also generates noisy and incomplete entity labels. in some cases, ds may lead to over 30% noisy instances @xcite , making it impossible to learn useful features. previous studies for handling such noisy labels consider either weakly-labeled entities, i.e., distantly-supervised named entity recognition (ner) @xcite , or noisy relation labels, i.e., distantly-supervised relation extraction (re) @xcite , where they focus on designing novel hand-crafted relation features @xcite , neural architectures @xcite , and tagging scheme @xcite to improve relation extraction performance. additionally, in-context learning (icl) using external knowledge of large language models (llms) @xcite is popular. however, they are resource-demanding, sensitive to prompt design, and may struggle with complex tasks. to cheaply mitigate both noise sources, we propose denrl-distantly-supervised joint extraction with noise-robust learning. denrl assumes that 1) reliable relation labels, whose relation patterns significantly indicate the relationship between entity pairs, should be explained by a model, and 2) reliable relation labels also implicitly indicate reliable entity tags of the corresponding entity pairs. specifically, denrl applies bag-of-word regularization (br) to guide a model to attend to significant relation patterns that explain correct relation labels, and ontologybased logic fusion (olf) that teaches underlying entity-relation dependencies with probabilistic soft logic (psl) @xcite . these two information sources are integrated to form a noiserobust loss, which regularizes a tagging model to learn from instances with correct entity and relation labels. next, if a learned model clearly locates the relation patterns and understands entityrelation logic of candidate instances, they are selected for subsequent adaptive learning. we further sample negative instances that contain corresponding head or tail entities of recognized patterns in those candidates to reduce entity noise. we iteratively learn an interpretable model and select highquality instances. these two-fold steps are mutually reinforced-a more interpretable model helps select a higher quality subset, and vice versa. given the superiority of unified joint extraction methods, we introduce a sequence labeling @xcite method to tag entities and their relations simultaneously as token classification. we incorporate a bert @xcite backbone that learns rich feature representations into the tagging scheme to benefit the information propagation between relations and entities. the transformer attention mechanism builds a direct connection between words and contributes to extracting long-range relations @xcite . its multi-head attention weights indicate interactions between each pair of words, which is further leveraged by self-matching to produce position-aware representations. these representations are finally used to decode different tagging results and extract all entities together with their relations. our contributions are summarized as follows: • our work introduces a novel framework for distantly-supervised joint extraction. this innovation lies in identifying and addressing multi-source noise arising from both entity and relation annotations, and a unified joint tagging scheme adaptable to various backbones. ## joint extraction architecture we incorporate a pre-trained bert backbone into our sequence tagging scheme to jointly extract entities and their relations (see figure 3 ). ## noise-robust learning to reduce the impact of noisy labels on tagging performance, we introduce bow regularization (br) to attend to confident relation patterns for reducing relation noise and ontology-based logic fusion (olf) to increase entity-relation coherence for reducing entity noise. finally, we employ self-adaptive learning (sal) to iteratively train on instances that can be explained by the model. ## conclusions we propose denrl, a noise-robust learning framework for distantly-supervised joint extraction, which consists of a transformer backbone, a new loss function and a self-adaptive learning step. specifically, we use bag-of-word regularization and logic fusion to learn important relation patterns and entity-relation dependencies. the regularized model is able to select trustable instances and build a versatile relation pattern set. a self-adaptive learning procedure then iteratively improves the model and dynamically maintains trustable pattern set to reduce both entity and relation noise. in the future, we aim to explore more complex patterns when configuring pattern sets. we will also evaluate our framework on other tasks such as event extraction and open information extraction.