Facilitating learning outcome assessment– development of new datasets and analysis of pre-trained language models (2023)

Abstract: Student mobility reflects academic transfer from one postsecondary institution to another and facilitates students’ educational goals of obtaining multiple credentials and/or advanced training in their field. This process often relies on transfer credit assessment, based on the similarity between learning outcomes, to determine what knowledge and skills were obtained at the sending institution as well as what knowledge and skills need to still be acquired at the receiving institution. As human evaluation can be both a challenging and time-consuming process, algorithms based on natural language processing can be a reliable tool for assessing transfer credit. In this article, we propose two novel datasets in the fields of Anatomy and Computer Science. Our aim is to probe the similarity between learning outcomes utilising pre-trained embedding models and compare their performance to human-annotated results. We found that ALBERT, MPNeT and DistilRoBERTa demonstrated the best ability to predict the similarity between pairs of learning outcomes. However, Davinci - a GPT-3 model which is expected to predict better results - is only able to provide a good qualitative explanation and not an accurate similarity score. The codes and datasets are available at https://github.com/JAkriti/New-Dataset-and-Performance-of-Embedding-Models .

## introduction student mobility refers to the movement -or, "transfer" -of students from one post-secondary institution (i.e., college or university) to another. students might choose to transfer for any number of reasons; common motivating factors include the opportunity to obtain both advanced training and multiple credentials in order to increase the number of future employment options. additionally, students whose high school grades do not allow them to enter their program or institution of choice might instead enroll first in an institution with less strin-gent admission requirements. obtainment of the initial post-secondary credential (e.g., diploma) can then facilitate transfer into the desired credential (e.g., degree) @xcite , particularly when both are within related fields (e.g., computer programming diploma and computer science degree). transferring within similar fields of study often means that there is overlap in topics and/or courses required for both credentials; therefore, in order to effectively recognize students' previous learning, receiving institutions are often required to assess "transfer credit." although numerous factors might influence this assessment, learning outcomes are considered a particularly valuable tool in the process @xcite . learning outcomes are the measurable objectives defined at the end of an assignment, class, course, or program @xcite and indicate the skill or knowledge level that can be expected from a student who has successfully completed the task in question. when a student transfers between institutions, the receiving institution typically reviews course learning outcomes from the previous institution to determine whether they align with the learning outcomes of comparable courses offered at the receiving institution. generally, program coordinators or other domain experts (e.g., teaching faculty) are the trusted authority designated to determine whether credit is warranted; however, human evaluation can be a complex and challenging task @xcite . the process of assessing transfer credit can be facilitated through the use of natural language processing (nlp) based semantic similarity algorithms. nlp has wide applicability, with the main challenge of measuring textual semantic similarity @xcite . in the past few decades, there has been rich advancement in defining various measures for similarity between words, short texts, and sentences @xcite . word-embeddings have emerged as a well-known technique that represents text in the form of a real-valued vector that reasonably captures the syntactic and semantic resemblance between them @xcite . transformer-based pre-trained language models trained on large text corpora have successfully emerged to be paradigmatic models for building vector-based representations of texts @xcite . these models have applications in numerous fields such as text summarization (mohamed and oussalah, 2019), question/answering @xcite , sentiment analysis @xcite , and sentence prediction, among others. in this direction, this paper aims to propose two novel datasets consisting of course learning outcomes in postsecondary education. we determined the complexity of the outcomes (sentences) through readability analysis. we also implemented various embedding models to scrutinize the similarity between pairs of sentences and compared the models' performance with human-annotated results. among different models, we found that al-bert, mpnet and distilroberta demonstrated the best ability to predict the similarity between pairs of learning outcomes. however, davinci -a gpt-3 model which is expected to predict better results -is only able to provide a good qualitative explanation and not an accurate similarity score. ## results this section provides an extensive comparative analysis of various embedding techniques (discussed in section 3.6) implemented for evaluating the similarity scores of learning outcomes in proposed datasets. to evaluate the relationship between the similarity scores of human annotators and embedding models we employ pearson's and spearman's rank coefficients for the anatomy and computer science datasets (including the dataset proposed by @xcite and the dataset proposed in this paper). ## conclusion transfer credit assessment usually consists of course comparisons via the evaluation of learning outcomes, which represent an important tool for assessment but are also subject to potential inconsistencies and bias. therefore, an automated system to assess transfer credit based on learning outcomes across institutes can facilitate the process by providing a reliable and consistent measure of similarity. over the years there has been a rich advancement in the era of large language models to measure semantic similarity between texts. various pretrained embedding models have been developed to represent text for algorithms to understand and compare semantic similarity. in this paper, we aim to propose two novel datasets of learning outcomes for courses in human anatomy and computer science operating systems and perform an analysis using embedding models to assist in transfer credit assessment. we found that versions of albert, mpnet and distilroberta outperform davinci (a gpt-3 model) that only provides a good qualitative interpretation of the similarity between pairs of sentences. application of these models within the context of transfer credit assessment can contribute to greater efficiency and consistency when determining learning outcome similarity.