Improving Transfer Learning for Early Forecasting of Academic Performance by Contextualizing Language Models (2024)

Abstract: This paper presents a cutting-edge method that harnesses contextualized language models (LMs) to significantly enhance the prediction of early academic performance in STEM fields. Our approach uniquely tackles the challenge of transfer learning with limited-domain data. Specifically, we overcome this challenge by contextualizing students’ cognitive trajectory data through the integration of both distal background factors (comprising academic information, demographic details, and socioeconomic indicators) and proximal non-cognitive factors (such as emotional engagement). By tapping into the rich prior knowledge encoded within pre-trained LMs, we effectively reframe academic performance forecasting as a task ideally suited for natural language processing.Our research rigorously examines three key aspects: the impact of data contextualization on prediction improvement, the effectiveness of our approach compared to traditional numeric-based models, and the influence of LM capacity on prediction accuracy. The results underscore the significant advantages of utilizing larger LMs with contextualized inputs, representing a notable advancement in the precision of early performance forecasts. These findings emphasize the importance of employing contextualized LMs to enhance artificial intelligence-driven educational support systems and overcome data scarcity challenges.

## introduction modern artificial intelligence (ai) methods, such as deep learning (dl), have increasingly been deployed as cost-effective solutions to develop earlywarning systems across various sectors, including health @xcite @xcite @xcite @xcite and education @xcite @xcite . these systems leverage forecasting-based interventions to preemptively address potential issues, from medical conditions to academic performance. in the educational domain, specifically, ai-based interventions utilize cognitive data, like students' course-related assessment scores, to predict and improve academic outcomes @xcite @xcite . the efficacy of these interventions hinges on the precision of early forecasts-predicting course performance as early as possible @xcite . however, this poses a significant challenge when training data is scarce, leading to suboptimal model performance. transfer learning could offer a solution, yet the approach is hampered by the lack of relevant pre-trained models or sufficiently large, domain-specific datasets for pre-training @xcite . in this paper, we address the challenges associated with limited training data by introducing a novel transfer learning methodology specifically tailored for domain-specific data within stem (science, technology, engineering, and mathematics) education contexts. we propose leveraging transformer-based @xcite pretrained language models (lms) for early prediction of academic performance in undergraduate stem courses. our method exploits the extensive knowledge base @xcite and reasoning capabilities @xcite @xcite of lms, transforming end-of-the-semester performance forecasting into a natural language text generation task. to enhance knowledge transfer using limited domain data, we contextualize students' cognitive data by integrating both distal background factors and proximal non-cognitive factors. this multidimensional approach encompasses demographic, socioeconomic, and academic background factors, as well as non-cognitive features like emotional engagement, to enrich the predictive model. by transforming the ordinal (numeric or real-valued) features of our data into natural language text sequences, we tailor pre-trained lms to our specific task. additionally, we augment these sequences to increase the dataset size, thereby improving predictive accuracy through a more balanced representation of various performance outcomes. ## method to harness the nuanced understanding pre-trained lms offer regarding students' academic experiences, we assembled a detailed longitudinal dataset that examines the interplay among various factors, including background, cognitive, and non-cognitive elements in student learning. figure 1 illustrates the lm-based transfer learning framework, featuring the contextualization of proximal cognitive data followed by the preprocessing of the contextualized academic trajectory. data contextualization involves integrating distal background and proximal non-cognitive factors with cognitive trajectory data. below, we outline the process of compiling a language dataset, encompassing data collection and pre-processing methods, and conclude with a formal description of transfer learning through fine-tuning of lms. ## experiments to thoroughly investigate the research questions outlined in section 1, we performed a series of experiments focusing on the learning capabilities of lms. these experiments involved fine-tuning pretrained lms across multi-dimensional language datasets spanning 8 weeks, 4 weeks, and 2 weeks. this selection of timeframes facilitated an in-depth examination of lm adaptability over various periods. the effectiveness of the adapted lms was assessed through their ability to identify performance types based on matching keywords in the predicted output sequences. moreover, we explored the impact of lm size-small, medium, and large-on their performance. experimental setup. for the encoder-decoder lm, we used pre-trained flan-t5 @xcite , which is a variant of the t5 model @xcite . the flan-t5 model is instruction fine-tuned, making it suitable for our purposes. we employed flan-t5 with three different capacities, determined by the number of parameters: flan-t5-small (80m), flan-t5-base (250m), and flan-t5-large (770m). these lms have a context window limited to 512 tokens. as baseline comparisons, we utilized four models that work with only numeric features: three neural networks (nns) and one non-nn machine learning model. the neural networks include a long short-term memory (lstm) network @xcite , a convolutional neural network (cnn) with a one-dimensional (1d) convolutional kernel @xcite , and a transformer network @xcite . the non-nn machine learning model employed was a support vector machine (svm) with a linear kernel @xcite , which demonstrated superior performance over the gaussian radial basis function kernel. the baseline models were trained using 3 variably-length numeric datasets containing only the cognitive features. exploring baseline models with all three feature types is planned as future work. to ensure compatibility with the lm-based experiments, the numeric datasets were created from the augmented verbalized datasets by decoding the cognitive feature part of text sequences into numeric values. we used the same test sets to evaluate both model types, employing the following metrics: accuracy, precision, recall, and f1 score. a detailed description of the experimental setup is provided in the appendix. ## conclusion in this paper, we ventured into the realm of leveraging modern ai, particularly deep learning and transfer learning methodologies, to tackle the critical challenge of early performance forecasting in the educational sector. our investigation centered on the innovative use of transformer-based pretrained lms for predicting undergraduate stem course outcomes, marking a significant departure from traditional numeric feature-based models. by integrating a novel transfer learning approach tailored for small-domain data within stem education, we aimed to overcome the limitations posed by sparse training datasets, a common hurdle in the educational domain. our methodology hinged on the contextualization of academic trajectory data, incorporating a rich tapestry of both cognitive and non-cognitive factors. through this multi-dimensional approach, we enhanced the lms' capacity to understand and predict academic performance, achieving a notable improvement in forecasting accuracy. specifically, we demonstrated that: these insights underscore the transformative potential of ai-driven tools in proactively identifying and supporting students at risk, thereby enhancing educational outcomes. by leveraging the vast knowledge encapsulated within lms and enriching it with detailed contextual data across demographic, socioeconomic, academic, and emotional engagement dimensions, we not only tailored the pre-trained lms to our specific task but also enriched the predictive model with a comprehensive understanding of students' academic journeys. looking ahead, our work opens the door to future research in several key areas. integrating more detailed contextual signals such as real-time academic engagement and behavioral data could enhance lm predictive accuracy, leveraging advances in natural language processing and sentiment analysis to understand students' emotional and cognitive states better. expanding our approach to a wider range of educational contexts and disciplines would help validate its scalability and adaptability. additionally, exploring continual learning techniques for lms might illuminate how to improve forecasting systems' accuracy and reliability over time without extensive retraining. addressing the ethical and privacy concerns inherent in using detailed student data is also crucial, necessitating robust data governance and ethical ai frameworks to protect students' rights and ensure equitable benefits.