Complexity-Guided Curriculum Learning for Text Graphs (2023)

Abstract: Curriculum learning provides a systematic approach to training. It refines training progressively, tailors training to task requirements, and improves generalization through exposure to diverse examples. We present a curriculum learning approach that builds on existing knowledge about text and graph complexity formalisms for training with text graph data. The core part of our approach is a novel data scheduler, which employs “spaced repetition” and complexity formalisms to guide the training process. We demonstrate the effectiveness of the proposed approach on several text graph tasks and graph neural network architectures. The proposed model gains more and uses less data; consistently prefers text over graph complexity indices throughout training, while the best curricula derived from text and graph complexity indices are equally effective; and it learns transferable curricula across GNN models and datasets. In addition, we find that both node-level (local) and graph-level (global) graph complexity indices, as well as shallow and traditional text complexity indices play a crucial role in effective curriculum learning.

## introduction message passing @xcite is a widely used framework for developing graph neural networks (gnns), where node representations are iteratively updated by aggregating the representations of neighbors (a subgraph) and applying neural network layers to perform non-linear transformation of the aggregated representations. we hypothesize that topological complexity of subgraphs or linguistic complexity of text data can affect the efficacy of message passing techniques in text graph data, and propose to employ such complexity formalisms in a novel curriculum learning framework for effective training of gnns. examples of graph and text complexity formalisms are node centrality and connectivity @xcite ; and word rarity and type token ratio @xcite respectively. in curriculum learning (cl) @xcite data samples are scheduled in a meaningful difficulty order, typically from easy to hard, for iterative training. cl approaches have been successful in various areas @xcite @xcite , including nlp @xcite @xcite @xcite @xcite . existing approaches use data properties such as sentence length, word rarity or syntactic features @xcite ; and model properties such as training loss and its variations @xcite to order data samples for training. however, other types of complexity formalisms such as those developed for graph data are largely underexplored. recently, @xcite proposed to estimate graph difficulty based on intra-and inter-class distributions of embeddings, realized through neural density estimators. @xcite employed a selective training strategy that targets nodes with diverse label distributions among their neighbors as difficult to learn nodes. @xcite used loss trajectories to estimate the emerging difficulty of subgraphs and weighted sample losses for data scheduling.we encourage readers to see @xcite for recent surveys on graph cl approaches. we propose that existing knowledge about text and graph complexity can inform better curriculum development for text graph data. for example, a training node pair that shares many common neighbors is expected to be easier for link prediction than a local bridge foot_0 that lacks common neighbors. motivated by this rationale, we present a complexityguided cl approach for text graphs (tgcl), which employs multiview complexity formalisms to space training samples over time for iterative training. it advances existing research as follows: subgraphs gnn trainer u v u v gene (u): armc5 the protein contains an armadillo repeat domain, suggesting that protein-protein interactions are important for its function. immunohistochemical analysis of human adrenal samples, hela cells, and adrenocortical cancer cells showed that localized to cytoplasm.. ## method a curriculum learning approach should estimate the complexity of input data, determine the pace of introducing samples based on difficulty, and schedule data samples for training. as figure 1 shows, tgcl tackles these tasks by quantifying sample difficulty through complexity formalisms ( §2.1), gradually introducing training samples to gnns based on a flexible "competence" function ( §2.2), and employing different data schedulers that order training samples for learning with respect to model behavior ( §2.3). by integrating these components, tgcl establishes curricula that are both data-driven and model-dependent. in what follows, we present approaches for addressing these tasks. ## curricula introspection we conduct several analysis on tgcl scheduler to study its behavior and shed light on the reasons for its improved performance. due to space limitations, we conduct these experiments on one representative dataset from each task, pgr for link prediction and ogbn-arxiv for node classification. ## conclusion and future work we introduce a novel curriculum learning approach for text graph data and graph neural networks, inspired by spaced repetition. by leveraging text and graph complexity formalisms, our approach determines the optimized timing and order of training samples. the model establishes curricula that are both data-driven and model-or learner-dependent. experimental results demonstrate significant performance improvements in node classification and link prediction tasks when compared to strong baseline methods. furthermore, our approach offers potential for further enhancements by incorporating additional complexity indices, exploring different scheduling functions and model transferability, and extending its applicability to other domains.