Response Construct Tagging: NLP -Aided Assessment for Engineering Education (2022)

Abstract: Recent advances in natural language processing (NLP) have greatly helped educational applications, for both teachers and students. In higher education, there is great potential to use NLP tools for advancing pedagogical research. In this paper, we focus on how NLP can help understand student experiences in engineering, thus facilitating engineering educators to carry out large scale analysis that is helpful for re-designing the curriculum. Here, we introduce a new task we call response construct tagging (RCT), in which student responses to tailored survey questions are automatically tagged for six constructs measuring transformative experiences and engineering identity of students. We experiment with state-of-the-art classification models for this task and investigate the effects of different sources of additional information. Our best model achieves an F1 score of 48. We further investigate multi-task training on the related task of sentiment classification, which improves our model’s performance to 55 F1. Finally, we provide a detailed qualitative analysis of model performance.

## introduction engineering education research (eer) investigates effective pedagogical practices in engineering through qualitative and quantitative methods. a major focus of this research is curriculum design, particularly, inculcating "engineering thinking" @xcite and identity @xcite along with technical skills. in order to develop and improve such curricula, one effective method is to evaluate student experiences in engineering courses in a subjective manner, assessing several attributes such as their perception towards engineering in daily life, and the impact of the curriculum on their self-identity as an engineer @xcite . a popular framework to carry out such assessments is to administer surveys before and after ## analysis on the survey as described in section 2.2. the cell values show the count of each affect label corresponding to a construct on the test set, and the color intensity shows the model's f1 on a scale of 0-1. overall, we see that the most frequent label is unavailable, on which the model's performance is also high across all constructs. this is particularly noticeable for the constructs of disciplinary knowledge, identification, and navigation, for which almost all annotations fall under unavailable except for q3. next, we look at the plots for individual questions. we observe that for q1, for the motivated use construct, the model does well on the positive and negative affect labels, but does not predict the other two classes correctly. for the affective value construct, the model predicts the neutral label correctly, but not positive or negative. for q2, which asks students about their feelings towards starting a project, both positive and negative affect labels are frequent, and the model also performs well on these classes. on q3, for both the constructs of expansion of perception and motivated use, f1 is high on both the positive and negative affect labels, but lower on the unavailable and neutral labels. finally, for q4, the model does well on the positive and negative labels for expansion of perception. on motivated use, the model achieves comparable performance on all labels, and on the affective value construct, the model does poorly at predicting neutral and negative labels. ## related work prior research has investigated how nlp can be used to analyze student feedback, with the goal of improving teaching and learning. similar to our work is sentiment analysis for classifying student's affective states after completing a course @xcite . more specifically, aspect-based sentiment analysis @xcite is used to determine sentiment towards distinct entities such as instructors or course material @xcite , as well as attributes such as teachers' helpfulness @xcite or quality of examples used @xcite . several methods have been investigated for this problem, including sentiment lexicons @xcite , probabilistic models @xcite , convolutional neural networks @xcite , and lstm models @xcite . however, our proposed task differs from aspect-based sentiment analysis since the constructs we are looking for are implicit, and are never explicitly mentioned in a student response. beyond sentiment classification, other applications have been studied for understanding student feedback: @xcite automatically summarize student responses to open-ended reflection prompts, and @xcite summarize student feedback on courses. @xcite analyze posts on mooc forums to determine student motivation and engagement. in engineering education research, nlp has been used for determining "disciplinary discourse" in student resumés @xcite , and for measuring metacognitive development of students in engineering classrooms @xcite . in our experiments, we use pretrained models for classification through fine-tuning, which have proven to be highly effective for nlp tasks @xcite . pretrained models have also been used successfully in educational applications @xcite @xcite . we also use multi-task learning @xcite , which has been investigated for tasks such as text classification @xcite and sequence labeling @xcite . multi-task learning has proven to be particularly effective in lowresource settings @xcite @xcite , which is applicable for our task as well. ## conclusion we introduce a new task, response construct tagging, to automatically tag student survey responses for the affective state of a student towards six predefined constructs. we present a classification model for this task based on the roberta pretrained model, that outperforms multiple baselines. on investigating the different information sources this model can utilize, we find that the best performance of 48.4 f1 can be attained by encoding a response, construct, and the corresponding question. we also demonstrate the benefits of training our model in a multitask learning setting on the related task of sentiment classification, which achieves a score of 55.1 f1, a 6.7 f1 improvement. our task, and corresponding model, enables educators to assess the effectiveness of their curriculum in influencing students' identity and perceptions of engineering, and thereby to design curricula that maximize positive influence. limitations and future work our proposed model can detect certain constructs and affects with high accuracy, such as the positive labels. however, rct is a challenging task -differences between affect labels are nuanced, and a single response can indicate different affective states towards different constructs. moreover, the sparsity of labels in our dataset makes it difficult to learn the rarer combinations of affect and constructs, such as negative identification. however, this is an inherent limitation with the classroom assessment framework, since students might be unwilling or unlikely to express feelings such as "not identifying as an engineer". one way to mitigate this problem might be to generate student responses artificially for constructs and affects that are not represented in the dataset. in future work, we will investigate how this can be done both manually, i.e., using human annotators, and automatically, such as conditionally generating responses that display a desired affect. life. students who expressed that aesthetics were generally unimportant were tagged as negative expansion of perception.