Generic and Trend-aware Curriculum Learning for Relation Extraction (2022)

Abstract: We present a generic and trend-aware curriculum learning approach that effectively integrates textual and structural information in text graphs for relation extraction between entities, which we consider as node pairs in graphs. The proposed model extends existing curriculum learning approaches by incorporating sample-level loss trends to better discriminate easier from harder samples and schedule them for training. The model results in a robust estimation of sample difficulty and shows sizable improvement over the state-of-the-art approaches across several datasets.

## introduction relation extraction is the task of detecting (often pre-defined) relations between entity pairs. it has been investigated in both natural language processing @xcite @xcite and network science @xcite . relation extraction is a challenging task, especially when data is scarce. nonetheless, the ability to automatically link entity pairs is a crucial task as it can reveal relations that have not been previously identified, e.g., informing clinicians about a causal relation between a gene and a phenotype or disease. figure 1 shows an example sentence from a pubmed article in the gene phenotype relation (pgr) dataset @xcite , which describes the application domain of the present work as well. previous research has extensively investigated relation extraction at both sentence @xcite @xcite and document @xcite levels. furthermore, effective graph-based neural network approaches have been developed for various prediction tasks on graphs, including link prediction between given node pairs @xcite @xcite . several recent approaches @xcite @xcite our study further emphasizes that ndufs6 sequence should be analyzed in patients presenting with lethal neonatal lactic acidemia because of isolated complex i deficiency phenotype gene causal figure 1 : an example showing the report of a causal relation between a gene and a phenotype (symptom) from the pgr dataset @xcite . @xcite illustrated the importance of enhancing graph neural networks using structurallyinformed features such as shortest paths, random walks and node position features. in this work, we develop a graph neural network titled graph text neural network (gtnn) that employs structurally-informed node embeddings as well as textual descriptions of nodes at prediction layer to avoid information loss for relation extraction. gtnn can be trained using a standard approach where data samples are fed to the network in a random order @xcite . however, nodes, edges or sub-graphs can significantly vary in their difficulty to learn, owing to frequent substructures, complicated topology and indistinct patterns in graph data. we tackle these challenges by presenting a generic and trend-aware curriculum learning approach that incorporates samplelevel loss trajectories (trends) to better discriminate easier from harder samples and schedule them for training graph neural networks. the contributions of this paper are: (a): a graph neural network that effectively integrates textual data and graph structure for relation extraction, illustrating the importance of direct use of text embeddings at prediction layer to avoid information loss in the iterative process of learning node embeddings for graph data; and (b): a novel curriculum learning approach that incorporates loss trends at sample-level to discover effective curricula for training graph neural networks. we conduct extensive experiments on real world datasets in both general and specific domains, and compare our model against a range of existing approaches including the state-of-the-art models for relation extraction. experimental results demonstrate the effectiveness of the proposed approach; the model achieves an average of 8.6 points improvement in f1 score against the best-performing graph neural network baseline that does not directly use text embeddings at its prediction layer. the proposed curriculum learning approach further improves this performance by 0.7 points, resulting in an average f1 score of 89.9 on our three datasets. we conduct extensive experiments to shed light on the improved performance of the model. code and data are available at https: //clu.cs.uml.edu/tools.html . ## method consider an undirected graph @xmath0, e) where v and e are nodes and edges respectively, and nodes carry text summaries as their descriptions. edges in the graph indicate "relations" between their end points, e.g., causal relations between genes and diseases, or links between concepts in an encyclopedia. our goal is to predict relations/links between given node pairs in g. ## trend model introspection we conduct several ablation studies to shed light on the improved performance of trend-sl. ## conclusion and future work we propose a novel graph neural network approach that effectively integrates textual and structural information and uses loss trajectories of samples during training to learn effective curricula for predicting relations between given entity pairs. our approach can be used for both sentence-and document-level relation extraction, and shows a sizable improvement over the state-of-the-art models across several datasets. in future, we will investigate curriculum learning approaches for other sub-tasks of relation extraction, develop more effective techniques to better fit trends to time series data, and investigate the effect of curricula on other graph neural networks for relation extraction.