Automatically Generated Summaries of Video Lectures May Enhance Students’ Learning Experience (2023)

Abstract: We introduce a novel technique for automatically summarizing lecture videos using large language models such as GPT-3 and we present a user study investigating the effects on the studying experience when automatic summaries are added to lecture videos. We test students under different conditions and find that the students who are shown a summary next to a lecture video perform better on quizzes designed to test the course materials than the students who have access only to the video or the summary. Our findings suggest that adding automatic summaries to lecture videos enhances the learning experience. Qualitatively, students preferred summaries when studying under time constraints.

## introduction video lectures have been an important part of scaled online courses and flipped classrooms for several years, and have become widely used for an increasingly larger range of courses as a substitute for students unable to attend class due to the covid-19 pandemic @xcite . past research in human-computer interaction aimed to improve educational videos via interactive transcripts, word clouds, keyword search, and highlight storyboards @xcite , or by segmenting the videos to present highlight moments with snapshots and transcripts @xcite . others have created video digests that are organized into a textbook-like format with chapters, titles, and sections with text summaries @xcite . @xcite 's system provides an authoring interface that allows video authors to manually write textual summaries of a video themselves or to send the video to a crowdsourcing service to have summaries written. textual summaries are believed to be effective in helping students review course materials. for example, @xcite find * these authors contributed equally. in this work, we investigate the feasibility of automatically summarizing lecture videos' transcripts using recent advances in large language models such as gpt-3 @xcite . we are encouraged by recent research in natural language processing demonstrating that people often prefer gpt-3 generated summaries over other methods of automatically generated summaries for news @xcite . the availability of high-quality automatic summaries would allow their use in a wide range of online courses. in this paper, we first detail our method for creating an automatic summarizer of video lectures. then, we report a controlled user study that we conducted with around 100 university students to investigate whether the automatic summaries were an effective study aid. ## automatic video lecture summarizer we designed an automatic lecture summarizer that leverages recent advances in large language models (llms). large language models are trained on massive language datasets, thus creating rich representations of language content. these models have been especially successful in generating fluent responses in a variety of natural language generation tasks with zero or few-shot prompting @xcite , instruction following @xcite @xcite , or fine-tuning @xcite . it is well-known, albeit not well-understood, that the quality of gpt-3 language generation is affected by the given prompt even if the differences in the way the prompt is articulated are not semantically very different. it is, also, unclear for which tasks it is sufficient to give prompts to achieve suc-cessful gpt-3 responses or whether there is a need for fine-tuning a model. for these reasons, we experimented with three approaches to generating lecture summaries. in all of our experiments, the input provided to the model is retrieved by transcribing video lectures. for the video transcription, we use amazon transcribe. the transcribed video content is then split into chunks of a maximum of 4,000 tokens. this is the maximum length that can be encoded by gpt-3. the final summary of the lecture is the aggregate of the summaries of its parts. for each chunk, we retain the corresponding start time in the video. text: <@xmath0: in this condition, the summary is generated without any training. we call this approach the "zero prompt" condition. below we provide an input example of a raw transcript from a natural language processing lecture given in an artificial intelligence university course. ## evaluation study in this section, we report a controlled study that we conducted with the goal of evaluating the potential benefits of offering students an automatic summary of transcribed video lectures. in what follows, we describe the participants of the study, the testing conditions, and the results. participants. we recruited 106 undergraduate and master's students who were taking an artificial intelligence course in fall 2022. students were given extra course credit for their participation. study design. we evaluated student performance on materials that students reviewed for two upcoming topics in the course presented in video lectures. these consisted of two short, prerecorded lecture videos on lexical semantic models (10 minutes) and stochastic gradient descent (12 minutes). for each topic, we evaluated the three learning conditions listed below. testing conditions 1) reviewing only the lecture video, 2) reviewing only the automatically generated summary, 3) reviewing both the video and the automatic summary. all students were randomly assigned to a different learning condition for each topic. many participants reviewed both lecture topics. for the second round, they were assigned to a different learning condition. prior to reviewing the course materials, students were given pre-test quizzes with four questions for each topic to test their initial understanding of the concepts. the answers were not shown to the students. after the pre-test, students reviewed the course materials using the materials associated with their randomly assigned learning condition. finally, they answered a 10-question quiz on the material that they had reviewed. these included the 4 questions from the pre-test, plus 6 previously unseen questions. the quiz questions consisted of a mix of true/false questions and multiple-choice questions. the quiz questions are given in appendix d. the students were not given a time constraint for reviewing the materials. however, once they started the quiz, they were no longer allowed to review the materials. the mean correctness on the pre-test quizzes is below 70%. after reviewing the learning materials, the condition in which students demonstrate the smallest improvement is condition (2) with only access to the automatic summaries improve: a relative improvement of 17.7% or an 11% absolute improvement. students who reviewed only the videos (condition 1) have a relative improvement of 18.5% (12.7% absolute). students who reviewed both the videos and the automatic summaries have a relative improvement of 24.4% (16% absolute). a finergrained breakdown of the students' performance on the quizzes for each lecture video is given in appendix f. we conducted a paired t-test to determine if there was a significant difference in the test correctness scores before and after the video+summary intervention. the results showed a calculated t-statistic of 2.12 and a p-value of 0.045 for the lexical semantic lecture, as well as a calculated t-statistic of -4.16 and a p-value of 0.0003 for the stochastic gradient descent lecture. these findings indicate a significant difference between the means. although we cannot conclusively determine that the video+summary approach is the most effective learning condition among those tested (as indicated by the kruskal-wallis test result of h(2) = 2.13 and a p-value of 0.34), we can observe that the results show a positive trend in the desired direction. ## conclusion our work shows that students reviewing both the lecture video and the automatically generated summary have a performance improvement from pretest to post-quiz. this suggests that accompanying lecture videos with automatically generated summaries does improve the studying experience. as online learning becomes more ubiquitous, incorporating automatically generated summaries with videos can enhance students' overall learning experience.