E xpan RL : Hierarchical Reinforcement Learning for Course Concept Expansion in MOOC s (2020)

Abstract: Within the prosperity of Massive Open Online Courses (MOOCs), the education applications that automatically provide extracurricular knowledge for MOOC users become rising research topics. However, MOOC courses’ diversity and rapid updates make it more challenging to find suitable new knowledge for students. In this paper, we present ExpanRL, an end-to-end hierarchical reinforcement learning (HRL) model for concept expansion in MOOCs. Employing a two-level HRL mechanism of seed selection and concept expansion, ExpanRL is more feasible to adjust the expansion strategy to find new concepts based on the students’ feedback on expansion results. Our experiments on nine novel datasets from real MOOCs show that ExpanRL achieves significant improvements over existing methods and maintain competitive performance under different settings.

## introduction the cognitive-driven theory has been widely used in practical teaching since ausubel firstly proposed it in @xcite , which suggests educators provide new knowledge for students to motivate their learning continuously. in fact, in addition to the concepts taught in course, many related concepts are also attractive and worthy of learning. as shown in figure 1 , when a student studies the concept lstm in "deep learning" course from coursera 1 , many related concepts, including its prerequisite concepts (rnn), related scientists (jürgen schmidhuber) and its related applications (machine translation) can also benefit his/her further study. in traditional classrooms, these concepts are often considerately introduced by teachers. however, in the era of massive open online courses (moocs), thousands of courses are prerecorded for with millions of students with various backgrounds @xcite , which makes it infeasible to pick out these essential concepts manually. therefore, there is a clear need to automatically discover course-related concepts so that they can easily acquire additional knowledge and achieve better educational outcomes. this task is formally defined as course concept expansion @xcite , a special type of concept expansion or set expansion @xcite , which refers to the task of expanding a small set of seed concepts into a complete set of concepts that belong to the same course or subject from external resources. despite abundant efforts in related topics @xcite @xcite , existing methods still face three challenges when applied to moocs. first, distinct from the task of enriching a certain concept set, the purpose of course concept expansion is to benefit students' learning, making the context information insufficient to detect whether a concept is appropriate to be an expansion result. how to properly introduce student feedback in the model's loop is a crucial challenge. second, unlike the set expansion for a clear general category (e.g., countries), courses are often the combinations of multiple categories, especially in interdisciplinary courses like mathematics for computer science foot_0 . therefore, it isn't easy to model the course's semantic scope @xcite when applying existing expansion methods. third, moocs are updated continuously, and numerous new courses arise everyday @xcite , which requires a good generalization ability of the expansion model; otherwise, the frequent model retraining will cause severe waste of resources. to address the above problems, we construct a novel interactive environment on real moocs, which collects students' feedback on expansion results and provides new knowledge for mooc students in an interesting way for better education. and based on the feedback, we propose expanrl, a hierarchical reinforcement learning framework for course concept expansion in moocs, which decomposes the concept expansion task into a hierarchy of two subtasks: high-level seed selection and low-level expansion. boosted by user feedback on expansion results, expanrl jointly learns how to select seed concepts to model the semantic scope of the course better, and whether a concept is beneficial for students. moreover, the hierarchical reinforcement learning (hrl) structure enables expanrl to learn proper expansion strategies instead of the modeling of a particular course, making our model keep a high performance even in unobserved courses. the evaluation is conducted on 9 datasets from real mooc courses, compared with 5 representative baseline methods. we further conduct an online evaluation to investigate whether students admit the expanded concepts. our contributions include 1) an investigation on how to involve hrl framework into the task of concept expansion; 2) a paradigm that connects the nlp concept expansion task with the educational application; 3) an interactive mooc environment, consisting of 9 novel datasets of different subjects, 6,553 extracted course concepts, and 495,324 user behaviors from a real mooc website. ## the proposed model in this section, we first introduce our hierarchical reinforcement concept expansion framework, ex-panrl, then present our high-level seed selection model and low-level expansion model separately. ## related work our work follows the task of concept expansion in moocs @xcite , a particular type of set expansion problem, which takes several seeds as input and expands the entity set. set expansion was born to serve knowledge acquisition applications on the internet. google sets was a pioneer which leaded a series of early research, e.g. bayesian sets @xcite , seal @xcite , seisa @xcite and others @xcite @xcite . these efforts utilize web tables as a resource and mainly serves for search engines. recently, more related research has turned its attention to other application fields, such as news mining @xcite , knowledge graphs @xcite , education assistance @xcite , etc. meanwhile, corpus-based expansion methods snowball, and iterative bootstrapping became a common solution @xcite @xcite , which expands the set in round and select high-quality results to extract feature iteratively. expanrl is inspired by this type of method and is designed to optimize the existing iterative process. expanrl also benefits from hierarchical reinforcement learning (hrl), which has been employed in many nlp tasks @xcite and achieved impressive results. by decomposing complex tasks into multiple small tasks to reduce the complexity of decision making @xcite , hrl naturally matches the iterative set expansion tasks. ## conclusion and future work we investigate the task of course concept expansion, which utilizes the nlp approaches in improving mooc education. after constructing a novel interactive mooc environment to collect user feedback on expansion results, we design a paradigm, expanrl, which decomposes the concept expansion task into a hierarchy of two subtasks: highlevel seed selection and low-level concept expansion. experiment results on nine datasets from real moocs prove that expanrl can better serve students by recognizing the helpful expanded results and maintaining good performance in interdisciplinary courses and even new courses. promising future directions include detecting how to ensemble supervised learning and rl expansion models and applying the proposed model in related tasks. we also hope our design of interactive games can call for more fancy methods that utilize student feedback in nlp applications