SIGHT : A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcripts (2023)

Abstract: Lectures are a learning experience for both students and teachers. Students learn from teachers about the subject material, while teachers learn from students about how to refine their instruction. Unfortunately, online student feedback is unstructured and abundant, making it challenging for teachers to learn and improve. We take a step towards tackling this challenge. First, we contribute a dataset for studying this problem: SIGHT is a large dataset of 288 math lecture transcripts and 15,784 comments collected from the Massachusetts Institute of Technology OpenCourseWare (MIT OCW) YouTube channel. Second, we develop a rubric for categorizing feedback types using qualitative analysis. Qualitative analysis methods are powerful in uncovering domain-specific insights, however they are costly to apply to large data sources. To overcome this challenge, we propose a set of best practices for using large language models (LLMs) to cheaply classify the comments at scale. We observe a striking correlation between the model’s and humans’ annotation: Categories with consistent human annotations (0.9 inter-rater reliability, IRR) also display higher human-model agreement (0.7), while categories with less consistent human annotations (0.7-0.8 IRR) correspondingly demonstrate lower human-model agreement (0.3-0.5). These techniques uncover useful student feedback from thousands of comments, costing around $0.002 per comment. We conclude by discussing exciting future directions on using online student feedback and improving automated annotation techniques for qualitative research.

## introduction lectures are a learning experience for both students and teachers. students learn from teachers about the subject material. teachers also learn from students about how to improve their instruction (for teaching @xcite @xcite . however, in the online education setting, student feedback is both abundant and unstructured. this makes it challenging for teachers with online content to synthesize and learn from available feedback. to take a step towards tackling this challenge, we contribute sight (student insights gathered from higher education transcripts), a large dataset of 288 math lecture transcripts and 15,784 comments collected from the massachusetts institute of technology opencourseware (mit ocw) youtube channel. mit ocw is a popular youtube channel that offers a collection of lecture content from real mit courses. their courses gather up to thousands of student comments @xcite @xcite , in which users express a range of feedback from excitement about the pedagogy to confusion about the course content. the dataset is a rich source of data for studying the relationship between teaching content and student commentary. second, we develop a rubric for categorizing different kinds of student feedback in the youtube comments using a qualitative analysis approach. qualitative analysis involves iteratively examining the data and accounting for the context @xcite @xcite . for example, we examine the student comments for useful feedback categories while accounting for the online context of the instruction. our rubric includes 9 categories of student youtube comments, spanning from general feedback useful for encouraging instructors (e.g., "amazing lectures!") to specific comments on the pedagogy or technical content. lecture transcript while qualitative analysis methods are effective in uncovering domain-specific insights, applying these methods to large sources of data is challenging @xcite @xcite . scaling annotation effectively is crucial for sifting through large amounts of unstructured data (e.g., the 15,784 comments in sight) and uncovering relevant student feedback. however, qualitative methodologies often require domain-expertise. this limits the pool of analysts, which makes it expensive to find this expertise or means that only a small sample of the data can be analyzed @xcite . additionally, the qualitative analysis process is time-consuming because it allows for the annotation rubric to be adapted and accommodate new categories. this means that data has to be re-annotated frequently and must be analyzed flexibly. therefore, our third contribution is proposing a set of best practices for using pretrained large language models (llms)-specifically chatgpt (openai, 2023)-to cheaply, quickly and flexibly annotate data at scale. we explore different prompting approaches (e.g., zero-shot, k-shot, and reasoning). we analyze the quality of the model annotation and the diversity of user feedback. categories with consistent human annotations (>0.9 inter-rater reliability, irr) also display higher human-model agreement (>0.7), while categories with less consistent human annotations (0.7-0.8) correspondingly demonstrate lower human-model agreement (0.3-0.5). albeit imperfect, annotating with chatgpt allows researchers to explore their entire dataset in a fast, cost-effective way. for example, we are able to sift through 15,784 comments and identify those related to student confusion and lecture pedagogy in a few hours, all under $0.002 per comment. these comments can be invaluable for instructors looking to improve their lecture content. in summary, we make the following contributions in this paper: ## sight this section details the dataset contents and data collection procedure. ## feedback rubric we develop a rubric that catalogs different types of student feedback found in sight. this rubric is used for annotating the comments at scale as well (rf. section 5). this section details how we developed this rubric. ## scaling annotation this section details how we scale annotations using our rubric and llms. scaling annotation is crucial for sifting through large amounts of unstructured data (e.g., the 15,784 comments in sight) and uncovering relevant student feedback. by using llms, we can cheaply and quickly classify comments, without the need for expensive human annotation. ## limitations while our work provides a useful starting point for understanding student feedback, there are limitations to our work. addressing these limitations will be an important area for future research. ## conclusion our work contributes sight, a large-scale dataset of lecture transcripts and student comments. we propose a rubric and different prompting methods for performing automated annotations on sight. while we find that there is still room for improvement on reliably automating the annotation process, the dataset and rubric provide a foundation for future research to address the challenges of discovering useful feedback from students at scale. for qualitative researchers, sight offers a unique opportunity to investigate and gain insights into the feedback provided by students in online learning environments. the comments cover students' perspectives and opinions related to math lectures. educators can also leverage sight as a valuable resource to learn from student comments and refine their teaching materials. by analyzing the feedback provided by students, educators can identify strengths and weaknesses in their instruction, discover areas that students find challenging or confusing, and gather valuable insights to enhance their teaching methodologies. we hope the dataset and methods building off of this dataset can aid educators in making data-informed decisions to optimize their instructional practices, thereby promoting a more effective learning environment.