Content Extraction from Lecture Video via Speaker Action Classification Based on Pose Information Online lecture videos are increasingly important elearning materials for students. Automated content extraction from lecture videos facilitates information retrieval applications that improve access to the lecture material. A significant number of lecture videos include the speaker in the image. Speakers perform various semantically meaningful actions during the process of teaching. Among all the movements of the speaker, key actions such as writing or erasing potentially indicate important features directly related to the lecture content. In this paper, we present a methodology for lecture video content extraction using the speaker actions. Each lecture video is divided into small temporal units called action segments. Using a pose estimator, body and hands skeleton data are extracted and used to compute motionbased features describing each action segment. Then, the dominant speaker action of each of these segments is classified using Random forests and the motionbased features. With the temporal and spatial range of these actions, we implement an alternative way to draw keyframes of handwritten content from the video. In addition, for our fixed camera videos, we also use the skeleton data to compute a mask of the speaker writing locations for the subtraction of the background noise from the binarized keyframes. Our method has been tested on a publicly available lecture video dataset, and it shows reasonable recall and precision results, with a very good compression ratio which is better than previous methods based on content analysis.

Key frame extraction from unstructured consumer video clips We present a key frame extraction method dedicated to summarize unstructured consumer video clips acquired from digital cameras. Analysis of spatiotemporal changes over time provides meaningful information about the scene and the cameramanu0027s general intents. First, camera and object motion are estimated and used to derive motion descriptors. A video is segmented into homogeneous segments based on major types of camera motion (e.g., pan, zoom, pause, steady). Dedicated rules are used to extract candidate key frames from each segment. Confidence measures are computed for the candidates to enable ranking in semantic relevance. This method is scalable so that we can produce any desired number of key frames from the candidates. We demonstrated the effectiveness of our method by comparing results with the ground truth agreed by multiple judges.