Distantly Supervised Course Concept Extraction in MOOC s with Academic Discipline (2023)

Abstract: With the rapid growth of Massive Open Online Courses (MOOCs), it is expensive and time-consuming to extract high-quality knowledgeable concepts taught in the course by human effort to help learners grasp the essence of the course. In this paper, we propose to automatically extract course concepts using distant supervision to eliminate the heavy work of human annotations, which generates labels by matching them with an easily accessed dictionary. However, this matching process suffers from severe noisy and incomplete annotations because of the limited dictionary and diverse MOOCs. To tackle these challenges, we present a novel three-stage framework DS-MOCE, which leverages the power of pre-trained language models explicitly and implicitly and employs discipline-embedding models with a self-train strategy based on label generation refinement across different domains. We also provide an expert-labeled dataset spanning 20 academic disciplines. Experimental results demonstrate the superiority of DS-MOCE over the state-of-the-art distantly supervised methods (with 7% absolute F1 score improvement). Code and data are now available at https://github.com/THU-KEG/MOOC-NER .

## introduction course concept extraction in massive open online courses (moocs) aims to recognize high-quality knowledge concepts and subject terms taught in the course. automatically extracting course concepts can help students better understand knowledgeable concepts of the course and reduce the burden of teacher workloads @xcite . it is a core task in course content analysis and mooc knowledge graph construction, which is a fundamental step to building ai-driven mooc systems with various downstream applications such as course recommendation and question answering figure 1 : an example of distant labels obtained with a dictionary, suffering from noisy and incomplete annotations. che. corresponds to chemistry with red color, agr. to agriculture with green, and med. to medicine with yellow. @xcite . however, moocs' explosive growth, like the number of online courses, which grew from 13.5k in 2019 to 19.4k in 2021 foot_0 , makes it expensive and tedious to annotate course corpus manually. therefore, there is a clear need to achieve automatically consistent and accurate course concept extraction in moocs to eliminate the heavy work of human annotations. early works for course concept extraction in moocs include graph propagation @xcite and statistical ranking methods @xcite . recently, distant supervision has been proposed for the automatic generation of training labels. as shown in figure 1 , the labeling procedure matches the tokens in the course corpus with concepts in an easily accessed dictionary. however, this matching process suffers from two challenges: (1) noisy annotation where a mention can be low-quality (i.e., the mention of 'plant' and 'species' of the first instance) or unrelated to the field of the course (i.e., the mention of 'identification' from chemistry but this instance is about agriculture); and (2) incomplete annotation where a mention can be matched partially (i.e., the mention of 'cerebrovascular disease' and 'neurological event' of the second instance) or missed completely (i.e., the mention of 'life-threatening' ) our analysis yields that the limited dictionary and diverse moocs lead to more noise and incomplete annotations foot_1 . first, the dictionary lacks sufficiently extensive coverage because of moocs' rapid growth and missing criteria. therefore, the out-of-dictionary, low-quality concepts will consequently render more course concepts unmatched and false-positive noisy annotations during matching. second, moocs can span 20 or even more academic disciplines @xcite , producing unrelated noisy annotations across different open domains. additionally, the uneven concept distribution and semantic differences among varied disciplines are different, imposing significant challenges to training an effective and accurate model. to address the two challenges, we propose a novel three-stage framework ds-moce to distantly supervised extract course concepts in moocs across different domains. our framework consists of (1) discipline-aware dictionary empowerment which employs prompt-based learning to explicitly generate concept distribution over diverse mooc domains and implicitly enhance the dictionary's limited capability; (2) distant supervision refinement which removes unrelated noise with much higher precision annotations for model training; and (3) discipline-embedding models with self-training to deal with noise iteratively while finding incomplete mentions based on semantic knowledge and syntactic information of pre-trained language models (plms) and positiveunlabeled learning (pul). for evaluation, we provide an expert-labeled dataset spanning 20 academic disciplines, which contains 522 expert-annotated sentences from 17 courses with 15, 375 course concepts. our contributions include 1) a novel three-stage framework to distantly supervised extract course concepts in moocs across different domains to eliminate the heavy work of human annotations; 2) a distant supervision refinement method to discard unrelated field noise and discipline-embedding models with a self-training strategy to remove noise iteratively and address the incomplete challenge based on pul; 3) an expert-labeled dataset with the excellent performance of our ds-moce framework over existing distantly supervised methods, with one implementation report of 7% absolute f1 score improvement. ## problem formulation following @xcite , we give some necessary definitions and then formulate the problem of distantly supervised course concept extraction. a course corpus is composed of n courses from different academic disciplines, denoted as @xmath0, where c i is one course. each course c @xmath1, f i } consists of two parts, where f @xmath2, . . . , f ik i ] is course related academic disciplines, and s @xmath3,...,n i is composed of n i course video subtitles, where v ij stands for the j-th video subtitles. finally, we get all academic fields @xmath4,...,k related to course corpus d, so k is the number of academic disciplines. a dictionary @xmath5,...,k , where distantly supervised course concept extraction in moocs is formally defined as follows. distant supervision refinement dictionary empowerment u plant species identification using pollen characteristics plant species identification using pollen characteristics che. agr. agr. agr. discipline-aware dictionary discipline distribution x y p(y|x) lm prompt generator [mask] including [identification] dictionary agr. med. che. … concept e.g. identification discipline-embedding models with self-training plant species identification using pollen characteristics agr. lm classification head o o b i plant species … pollen characteristics u1 e1 u2 e2 u3 e3 u3 e4 course corpus co-training pu learning dc-moce(co) dc-moce(pul) figure 2: our proposed three-stage framework ds-moce for distantly supervised course concept extraction in moocs, which includes (1) discipline-aware dictionary empowerment; (2) distant supervision refinement and (3) discipline-embedding models with self-training. for two advanced implementations: ds-moce (co) means our framework with two student-teacher networks co-training self-train strategy, and ds-moce (pul) means our framework adding positive-unlabeled learning loss self-train strategy. ## the ds-moce framework considering the limited dictionary and diverse moocs, it is natural not to ignore the academic discipline characteristics for distantly supervised course concept extraction in moocs. as shown in figure 2 , we propose a three-stage framework ds-moce, which includes 1) discipline-aware dictionary empowerment to transfer the power of plms to the dictionary; 2) distant supervision refinement which considers academic disciplines to tackle the unrelated field noise explicitly; and 3) discipline-embedding models to fully exploit the power of plms with concept distribution to implicitly handle the noise and incomplete challenges, which then can be integrated with two advanced ds-ner implementations. one employs a co-training strategy to deal with the noise iteratively, denoted as ds-moce(co). the other employs pul to deal with the incomplete problem, denoted as ds-moce(pul). ## conclusion and future work in this paper, we attribute the increased noise and incomplete challenges of distantly supervised course concept extraction in moocs to the limited dictionary and diverse moocs. to tackle these challenges, we propose a three-stage framework ds-moce, which handles the unrelated noise through dic-matching refinement and disciplineembedding model training, and leverages the power of pre-trained language models for dictionary empowerment and incomplete mentions mining. we also provide an expert-labeled dataset spanning 20 academic disciplines. experimental results show that ds-moce is highly effective, outperforming the state-of-the-art distantly supervised methods. although achieving significant improvement, course concept extraction in moocs is still nontrivial. in the future, we plan to design a more robust training method to jointly deal with severe noisy and incomplete issues and apply it to other real-world open domains.