Curriculum Learning for Graph Neural Networks: A Multiview Competence-based Approach (2023)

Abstract: A curriculum is a planned sequence of learning materials and an effective one can make learning efficient and effective for both humans and machines. Recent studies developed effective data-driven curriculum learning approaches for training graph neural networks in language applications. However, existing curriculum learning approaches often employ a single criterion of difficulty in their training paradigms. In this paper, we propose a new perspective on curriculum learning by introducing a novel approach that builds on graph complexity formalisms (as difficulty criteria) and model competence during training. The model consists of a scheduling scheme which derives effective curricula by accounting for different views of sample difficulty and model competence during training. The proposed solution advances existing research in curriculum learning for graph neural networks with the ability to incorporate a fine-grained spectrum of graph difficulty criteria in their training paradigms. Experimental results on real-world link prediction and node classification tasks illustrate the effectiveness of the proposed approach.

## introduction graph neural networks (gnns) are generally trained using stochastic gradient descent (sgd), where the standard approach is to iteratively use the entire training data to optimize model's objective until convergence. curriculum learning techniques improve this training process by scheduling examples for training, e.g., by gradually learning from easier examples before training with harder ones. such curricula can be predefined by humans @xcite or dynamically derived from data during training @xcite . curriculum learning for graph neural networks is an emerging area of research. recently, @xcite employed a traditional curriculum learning approach introduced in @xcite to improve negative sampling for graph classification. @xcite proposed to estimate the difficulty of graph entities-nodes, edges or subgraphsbased on the intra-and inter-class distributions of their embeddings in supervised settings, and developed a smooth-step function to gradually introduce harder examples to gnns during training. @xcite developed a loss-based curriculum learning approach that dynamically adjusts the difficulty boundaries of training samples based on their sample-level loss trajectories obtained from recent training dynamics of gnn models. to the best of our knowledge, existing curriculum learning approaches often employ a single criterion of difficulty in their curriculum learning framework, e.g., prediction loss @xcite , consistency in prediction loss @xcite , moving average of loss @xcite or transformations of loss @xcite . we address this gap by developing a new curriculum learning approach for gnns titled multiview competence-based curriculum learning (mccl) that builds on the complexity formalisms of graph data. by leveraging rich graph structures, graph complexity formalisms and model competence (learning progress), we will design robust curricula for training gnns. ## competence-based multiview curricula we present a competence-based multiview curriculum learning framework for training gnns. at every training iteration, the framework selects a sub-set of training examples based on the best complexity index (view) and model's competence at that iteration. algorithm 1 describes the overall approach. we first introduce our complexity indices and then present the model. ## multiview curricula introspection we perform several ablation studies on the mccl model, investigating genuine complexity indices compared to random ordering, multiview curricula versus anti-curricula, the impact of complexity indices in training, and the model's time complexity. ## related work curriculum learning @xcite aims to improve the generalizability of a model by gradually training it with easy examples followed by hard ones. @xcite introduced a generic loss function called superloss (sl) which can be added on top of any target-task loss function to dynamically weight the training samples according to their difficulty for the model using a batch-wise threshold. @xcite proposed dynamic instance hardness to determine the difficulty of an instance with running average of the hardness metric over training history. curriculum learning has been investigated in nlp @xcite @xcite @xcite @xcite @xcite @xcite @xcite . specifically, settles and meeder ( 2016 et al. ( 2019 ) proposed an approach to use competency function using rarity of words or length of a sentence for neural machine translation and inspired @xcite to define a curriculum based on multi-modal (text and image) data to choose which modality should be used for training. the model uses sample perplexity at batch level to select the modality for training. linguistic features such as word rarity or length of sentence in @xcite and sample perplexity in @xcite were used as measures of difficulty. @xcite designed a curriculum learning approach for nlp tasks using cross-view of training data to identify easy and hard examples and rearrange the examples during training. other works of curriculum learning in nlp focused on machine translation and language understanding. @xcite developed a framework to train non-autoregressive sequence-to-sequence model to edit text where a curriculum is designed to first perform easy-to-learn edits followed by increasing difficulty of training samples. @xcite designed several curriculum learning approaches using teacher-student model where the teacher model calculates the difficulty of each training example using question-answering probability, variability, and out-of-distribution measures. curriculum learning for graph data is an emerging area of research. @xcite explored curriculum learning approach in the self-supervision settings where the difficulty measure evaluates the difficulty of negative samples which is calculated based in the embeddings's similarity between positive and negative examples. @xcite proposed a curriculum based subgraph classification approach, curgraph, which first obtains graphlevel embeddings via unsupervised gnn method and then uses neural density estimator to model embedding distributions. the difficulty scores of graphs are calculated by a predefined difficulty measure based on the inter-and intra-class distribution of sub-graph embeddings. in vakil and amiri (2022), we extended the superloss approach developed in @xcite by introducing a curriculum learning framework that dynamically adjusts the difficulty of samples during training with respect to the loss trajectory. we demonstrated the effectiveness of incorporating this strategy in graph curriculum learning settings. previous work in graph curriculum learning has employed a single criterion of difficulty in their curriculum learning framework. our work uses multiple criteria for curriculum learning on graphs. we encourage readers to see @xcite for a survey on graphs curriculum learning approaches. finally, in terms of datasets, @xcite developed the pgr dataset, which we used in our experiments. they developed a transformer model to identify the relation between biomedical entities, genes and phenotypes, from scientific pubmed articles. for relation classification authors considered a pair of entities and the context from the corresponding sentence in which both entities occur. ## conclusion and future work we present a novel curriculum learning approach for training graph neural networks. our approach combines well-established graph complexity indices (views) obtained from graph theory and demonstrates the effectiveness of learning from diverse difficulty views for the tasks of link prediction and node classification. our approach improves over the state-of-the-art techniques for curriculum learning on graphs across several datasets. ablation studies show that the model prioritizes genuine complexity indices over to random ordering, and effectively uses and learn multiview complexity indices in both curricula and anti-curricula settings, and has lower time complexity that competing models. in future, we will extend our approach to other graph processing tasks, focusing on nlp applications such as clustering and community detection, and investigate the effect of graph complexity indices in such tasks.