Improving Embeddings Representations for Comparing Higher Education Curricula: A Use Case in Computing (2022)

Abstract: We propose an approach for comparing curricula of study programs in higher education. Pre-trained word embeddings are fine-tuned in a study program classification task, where each curriculum is represented by the names and content of its courses. By combining metric learning with a novel course-guided attention mechanism, our method obtains more accurate curriculum representations than strong baselines. Experiments on a new dataset with curricula of computing programs demonstrate the intuitive power of our approach via attention weights, topic modeling, and embeddings visualizations. We also present a use case comparing computing curricula from USA and Latin America to showcase the capabilities of our improved embeddings representations.

## introduction several stakeholders in education compare and assess curricula, such as governments aiming to improve the competitiveness of their local programs @xcite , or academics looking to analyze market offer and suggest new courses and innovations @xcite @xcite . despite their benefits, manual comparisons can be time-consuming, and are prone to biases from human assessors due to their personal beliefs and perspectives @xcite . this situation becomes more challenging when studying curricula related to programs in computing due to their fast-paced evolution, which may prevent suitable analysis and evaluation by human intervention @xcite . to aid higher education stakeholders, we propose a method to automatically compare curricula based on the names and content of their courses, and test it on study programs in computing. previous work has represented curricula using bag-of-words, and relied on clustering algorithms to identify groups of commonly-studied topics in data science @xcite , or to find links between study programs in different countries and across computing disciplines @xcite . others have attempted to associate curricula to knowledge areas defined by international associations, such as acm or ieee @xcite , using standard tf-idf representations @xcite or topic modelling @xcite . in this work, we follow the current trend in nlp applications and use pre-trained word embeddings, such as bert @xcite , to obtain better representations of textual curricula. we fine-tune these embeddings on a computing discipline classification task, using a newly-collected dataset that includes study programs from accredited universities in the usa and latin america. we introduce a course-guided attention mechanism that allows the model to identify which courses are more relevant for each computing discipline, and pair it with metric learning @xcite to learn welldefined groups. experiments with different types of pre-trained word embeddings and classification algorithms, show that our proposed approach generates an accurate representation of computing curricula that allows human understanding. we also show qualitative results via attention weights, topic modeling, and embedding visualizations. these results highlight the benefits of our approach for identifying relevant words for each computing curriculum. finally, we exploit these embeddings to visualize how latin america computing programs relate to recognized abet disciplines. in summary, our main contributions are: ## computing curricula dataset we collected curricula from university study programs from different countries and categorized them into five computing disciplines: computer science (cs), computer engineering (ce), information technology (it), information science (is), and software engineering (se). each curriculum consists of a set of courses including their title and course description, as depicted in the usa portion of the dataset is used to train and evaluate embeddings representation in a computing discipline classification task (sec. 5), while the latam portion is exploited to analyse the degree of internationalization of latin american study programs in computing (sec. 7). ## approach in order to obtain better representations of textual curricula, we propose to use pre-trained bert @xcite embeddings that have been finetuned in a computing discipline classification task, using an approach that combines a novel coursebased attention mechanism and metric learning. figure 2 shows an overview of our method. coursebased attention identifies the most and the least important courses following the intuition of core and elective courses, while metric learning learns boundaries to form well-defined groups. course-based attention. our course-based attention approach aims to learn the relevance of each course associated with computing curricula. as shown in figure 2 , it receives a computing curriculum composed of courses and their bert embeddings curriculum emb . then, we compute att weights for each course. using these weights, we calculate a weighted average over the courses and generate a new curriculum embedding curriculum emb_avg . finally, we collapse the generated embedding into 128 features. metric learning. using these features, we learn well-defined groups among computing curricula. we employ metric learning with the following triplet loss @xcite , where n is the number of instances in a batch, α is the triplet margin with value 0.3, foot_2 and θ denotes the learnt parameters. given an anchor curriculum (c a i ) and using instances in the same batch, we select curriculums with the same category as positive annotations (c p i ), and curricula from different categories as negative annotations (c n i ). data points were randomly sampled with a batch size of 64 to ensure that every category is present in each iteration. ## quantitative experiments we aim to validate which approach generates a more precise representation for classification and human intuition. using the newly-computed embeddings, we trained four classifiers: k-nearest neighbour (knn), logistic regression (lr), linear support vector machine (lsvm), and radial support vector machine (rsvm) with a proper search range of parameters (detailed in appendix a.2.1). ## application: internationalization as a use case, we investigated how latam computing careers relate to international standards. we used bert met+att to project unseen cs latam computing curricula and try to relate them to usa standards in figure 3 (c ) using umap. latam curricula (in triangles) form two groups: one near cs, and one surrounding is and se. also, no latam curriculum is close to the ce profile. from this visualization, we infer that latam curricula are different from the us because none of them lay inside us groups. then, we perform a closer study on individual latam countries. brazil and mexico have a clear cs profile. also, mexico seems much more integrated with the us (a) cs (b) ce (c) it (d) is (e) se figure 4: word clouds from courses of top 5 attention weights obtained with bert met+att model on the test set with (a) computer science (cs), (b) computer engineering (ce), (c) information technology (it), (d) information system (is), and (e) software engineering (se). profile. on the other hand, peru has a mixed profile between cs, se, and is; which may suggest a better definition of courses per career. finally, colombia belongs to se. reducing subjectivity in curricula comparison. our approach helps reduce human bias by allowing an automatic comparison of a curriculum to international standards. discovering relevant courses via attention weights is particularly useful. for example, cs01 cs02 se01 algorithms and complexity design and analysis design and analysis of algorithms design and analysis of algorithms data structures and object-oriented design data structures database modeling systems programming computer systems and networks personal software process principles and practices parallel systems introduction to parallel systems computer network security individual software design and development introduction to machine learning computing theory introduction to database systems ## conclusion in this paper, we explored an intuitive way to generate accurate representations for understanding computing curricula, by combining course-guided attention and metric learning. our approach finds more cohesive groups with clear separations among them. these groupings are helpful for different machine learning models. we also analyzed what our approach learns via attention weights, topic modeling, and visualization techniques.