Jointprop: Joint Semi-supervised Learning for Entity and Relation Extraction with Heterogeneous Graph-based Propagation (2023)

Abstract: Semi-supervised learning has been an important approach to address challenges in extracting entities and relations from limited data. However, current semi-supervised works handle the two tasks (i.e., Named Entity Recognition and Relation Extraction) separately and ignore the cross-correlation of entity and relation instances as well as the existence of similar instances across unlabeled data. To alleviate the issues, we propose Jointprop, a Heterogeneous Graph-based Propagation framework for joint semi-supervised entity and relation extraction, which captures the global structure information between individual tasks and exploits interactions within unlabeled data. Specifically, we construct a unified span-based heterogeneous graph from entity and relation candidates and propagate class labels based on confidence scores. We then employ a propagation learning scheme to leverage the affinities between labelled and unlabeled samples. Experiments on benchmark datasets show that our framework outperforms the state-of-the-art semi-supervised approaches on NER and RE tasks. We show that the joint semi-supervised learning of the two tasks benefits from their codependency and validates the importance of utilizing the shared information between unlabeled data.

## introduction named entity recognition (ner) and relation extraction (re) are two crucial tasks in information extraction. supervised learning schemes have made significant progress in ner and re research by leveraging rich annotated data (e.g., @xcite ; @xcite . however, high-quality data annotation still involves extensive and expensive labor. moreover, training ner and re models in various domains and applications require diverse annotated data. semi-supervised learning approaches (ssl) employ a small amount of annotated data as a source of supervision for learning powerful models at a lower cost. ssl in ner and re have performed very well in recent years by employing bootstrapping, distant supervision or graph-based approach @xcite @xcite . however, they train a ner model @xcite @xcite or a re model @xcite @xcite separately. therefore, they neglect the underlying connections between entity recognition and relation extraction under a semi-supervised learning scheme, making it harder for the model to assign accurate annotation to unla-beled data. for instance, in figure 1 , the annotated entity "generative model" in sentence s1 and the unannotated "probabilistic model" in sentence s2 are syntactically similar. likewise, the context phrases "uses... to" and "used in" are also similar. if such similarities are ignored, the model may fail to draw a syntactic analogy between "dependency parsing" and "alignment", and thereby miss labeling the latter as an entity that shares the same type with the former. to the best of our knowledge, there is no universal framework to integrate semi-supervised learning for different tasks in ie, despite evidence of the effectiveness of a joint or multi-task learning approach @xcite @xcite @xcite . in addition, existing semi-supervised approaches devote considerable effort to aligning labeled and unlabeled data but do not exploit similarities between instance pairs that are structurally parallel, which exist across unlabeled data. consequently, they do not perform classification from the perspective of global consistency. for example, given the sentences s1 to s3 in figure 1 , we expect a model to recognize the entities and relations as (method, used-for, task) in triplet form. however, it is hard to infer the correct pseudo label to the unlabeled entities "alignment" or "nli alignment" from the annotated entity "dependency parsing". because they are not semantically or lexically similar. likewise, the affinity between "uses to" and "apply" is not obvious; and hence it would be difficult to extract the relation used-for between entities. nonetheless, the "alignment" and "nli alignment" pair are alike, and so are the pair "probabilistic model" and "probabilistic model". exploiting the relationships between unlabeled data would integrate the information hidden in the text and make use of the large quantity of unlabeled data for semisupervised learning. to address the above limitations, we propose a semi-supervised method based on label propagation over a heterogeneous candidate graph to populate labels for the two tasks (see figure 3 ). more specifically, we introduce a joint semi-supervised algorithm for the two tasks, where unannotated and annotated candidates (entities and relations) are treated as nodes in a heterogeneous graph, and labels are propagated across the graph through similarity-scored edges. our framework jointprop considers the interactions among the unlabeled data by constructing the graph using the union of labeled and unlabeled data into one learning diagram. we evaluate jointprop on multiple benchmark datasets and our proposed framework achieve state-of-theart results on both semi-supervised ner and re tasks. to the best of our knowledge, this is the first work that performs semi-supervised learning for entity and relation extraction in a unified framework to leverage unannotated data for both tasks. our contributions are summarized as following: the framework generates candidate spans from the unlabeled data, automatically constructs a semantic similarity-based graph for all the candidates, and performs label propagation across the graph. semi-supervised learning the semi-supervised learning seeks to enhance limited labeled data by leveraging vast volumes of unlabeled data @xcite which mitigate data-hungry bottleneck and supervision cost. ssl has a rich history @xcite . there have been substantial works in semisupervised settings in nlp, such as bootstrapping @xcite @xcite , co-training @xcite , distant supervision @xcite , and graph-based methods @xcite @xcite . in particular, graph-based ssl algorithms have gained considerable attention @xcite @xcite . there are two underlying assumptions for the label propagation @xcite . first, similar training samples are more likely to belong to the same class. second, nodes in similar structures are likely to have the same label. label propagation exploits these assumptions by propagating label information to surrounding nodes based on their proximity. the metric-based method had been applied in a graph-based ssl setting for its ability to infer labels for unseen classes directly during inference. for example, @xcite propagates the label based on estimating the posterior probabilities of unlabeled data. meanwhile, @xcite sought to exploit the manifold structure of novel class space in a transitive setting. the purpose of this work is to predict a possible entity type y e (s i ) ∈ e for each span s @xmath0, s j ) ∈ r for every pair of spans s @xmath1, s @xmath2. the label can also be a 'null' label for a span (i.e. y e (s i ) = ϵ) or a span pair (i.e. y r (s i , s j ) = ϵ). the output of the task are y @xmath3, e) : @xmath4, @xmath5, s j , r) : s i , s @xmath6, @xmath7. ## experiments we evaluate the effectiveness of jointprop against models from two lines of work: semi-supervsied ner and semi-supervsied re. we also provided a detailed analysis to demonstrate the benefits of our framework. for implementation details and dataset descriptions please refer to appendix a and appendix b. ## conclusion in this paper, we propose a novel heterogeneous graph-based propagation mechanism for joint semisupervised learning of entity and relation extraction. for the first time, we explore the interrelation between different tasks in a semi-supervised learning setting. we show that the joint semi-supervised learning of two tasks benefits from their codependency and validates the importance of utilizing the shared information between unlabeled data. our experiments show that combining the two tasks boost the model performance. we also evaluate two public datasets over competitive baselines and achieve state-of-the-art performance. we also conduct ablation studies of our proposed framework, which demonstrate the effectiveness of our model. we further present case studies of our model output.