R - VGAE : Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning (2020)

Abstract: The task of concept prerequisite chain learning is to automatically determine the existence of prerequisite relationships among concept pairs. In this paper, we frame learning prerequisite relationships among concepts as an unsupervised task with no access to labeled concept pairs during training. We propose a model called the Relational-Variational Graph AutoEncoder (R-VGAE) to predict concept relations within a graph consisting of concept and resource nodes. Results show that our unsupervised approach outperforms graph-based semi-supervised methods and other baseline methods by up to 9.77% and 10.47% in terms of prerequisite relation prediction accuracy and F1 score. Our method is notably the first graph-based model that attempts to make use of deep learning representations for the task of unsupervised prerequisite learning. We also expand an existing corpus which totals 1,717 English Natural Language Processing (NLP)-related lecture slide files and manual concept pair annotations over 322 topics.

## introduction with the increasing amount of information available online, there is a rising need for structuring how one should process that information and learn knowledge efficiently in a reasonable order. as a result, recent work has tried to learn prerequisite relations among concepts, or which concept is needed to learn another concept within a concept graph @xcite @xcite . figure 1 shows an illustration of prerequisite chains as a directed graph. in such a graph, each node is a concept, and the direction of each edge indicates the prerequisite relation. consider two concepts p and q, we define @xmath0. for example, the concept variational autoencoders is a prerequisite concept of the concept variational graph autoencoders. if someone wants to learn about the concept variational graph autoencoders, the prerequisite concept variation autoencoder should appear in the prerequisite concept graph in order to create a proper study plan. recent work has attempted to extract such prerequisite relationships from various types of materials including wikipedia articles, university course dependencies or moocs (massive open online courses) @xcite @xcite . however, these materials either need additional steps for pre-processing and cleaning, or contain too many noisy free-texts, bringing more challenges to prerequisite relation learning or extracting. recently, @xcite presented a collection of university lecture slide files mainly in nlp lectures with related prerequisite concept annotations. we expanded this dataset as we believe these lecture slides offer a concise yet comprehensive description of advanced topics. deep models such as word embeddings @xcite and more recently contextualized word embeddings @xcite have achieved great success in the nlp tasks as they demonstrate a stronger ability to represent the semantics of the words than other traditional models. however, recent prerequisite learning approaches fail to make use of distributional semantics and advances in deep learning representations @xcite . in this paper, we investigate deep node embeddings within a graph structure to better capture the semantics of concepts and resources, in order to learn accurate prerequisite relations. in addition to learning node representations, there has been growing research in geometric deep learning @xcite and graph neural networks @xcite , which apply the representational power of neural networks to graph-structured data. notably, @xcite proposed graph convolutional networks (gcns) to perform deep learning on graphs, yielding competitive results in semi-supervised learning settings. textgcn was proposed by @xcite to model a corpus as a heterogeneous graph in order to jointly learn word and document embeddings for text classification. we build upon these ideas for constructing a resource-concept graph foot_0 . additionally, most of the mentioned methods require a subset of labels for training, a setting which is often infeasible in the real world. limited research has been investigated learning prerequisite relations without using human annotated relations during training @xcite . in practice, it is very challenging to obtain annotated concept-concept relations, as the complexity for annotating is o(n foot_1 ) given n concepts. to tackle this issue, we propose a method to learn prerequisite chains without any annotated concept-concept relations, which is more applicable in the real word. our contributions are two-fold: 1) we expand upon previous annotations to increase coverage for prerequisite chain learning in five categories, including ai (artificial intelligence), ml (machine learning), nlp, dl (deep learning) and ir (information retrieval). we also expand a previous corpus of lecture files to include an additional 5000 more lecture slides, totaling 1,717 files. more importantly, we add additional concepts, totaling 322 concepts, as well as the corresponding annotations of each concept pair, which totals 103,362 relations. 2) we present a novel graph neural model for learning prerequisite relations in an unsupervised way using deep representations as input. we model all concepts and resources in the corpus as nodes in a single heterogeneous graph and define a propagation rule to consider multiple edge types by eliminating concept-concept relations during training, making it possible to perform unsupervised learning. our model leads to improved performance over a number of baseline models. notably, it is the first graph-based model that attempts to make use of deep learning representations for the task of unsupervised prerequisite learning. resources, annotations and code are publicly available online 2 . ## evaluation we compare our proposed model with two groups of baseline models. we report accuracy, f1 scores, the macro averaged mean average precision (map) and area under the roc curve (auc) scores in ## analysis we then take the recovered concept relations from our best performed model r-vgae (ss+p2v) in ## conclusion in this paper we introduced an expanded dataset for prerequisite chain learning with additional an 5,000 lecture slides, totaling 1,717 files. we also provided prerequisite relation annotations for each concept pair among 322 concepts. additionally, we proposed an unsupervised learning method which makes use of advances in graph-based deep learning algorithms. our method avoids any feature engineering to learn concept representations. experimental results demonstrate that our model performs well in an unsupervised setting and is able to further benefit when labeled data is available. in future work, we would like to perform a more comprehensive model comparison and evaluation by bringing other possible variations of graph-based models to learn a concept graph. another interesting direction is to apply multi-task learning to the proposed model by adding a node classification task if there are node labels available. a part of the future work would also include developing educational applications for learners to find out their study path for certain concepts. it is also possible to extend our prerequisite chain learning framework to other domains rather than the five mentioned categories focused on nlp domain. we plan to expand our method to other tasks including matrix completion task, linking prediction task in knowledge graphs in an unsupervised way and other additional tasks.