We developed NameIt, a system that associates faces and names in news videos. It processes information from the videos and can infer possible name candidates for a given face or locate a face in news videos by name. To accomplish this task, the system takes a multimodal video analysis approach: face sequence extraction and similarity evaluation from videos, name extraction from transcripts, and video-caption recognition. T he Name-It system 1,2 associates names and faces in news videos. Assume that we're watching a TV news program. When persons we don't know appear in the news video, we can eventually identify most of them by watching only the video. To do this, we detect faces from a news video, locate names in the sound track, and then associate each face to the correct name. For face-name association, we use as many hints as possible based on structure, context, and meaning of the news video. We don't need any additional knowledge such as newspapers containing descriptions of the persons or biographical dictionaries with pictures. Similarly, Name-It can associate faces in news videos with their right names without using an a priori face-name association set. In other words, Name-It extracts face-name correspondences only from news videos. Name-It takes a multimodal approach to accomplish this task. For example, it uses several information sources available from news videosimage sequences, transcripts, and video captions. Name-It detects face sequences from image sequences and extracts name candidates from transcripts. It's possible to obtain transcripts from audio tracks by using the proper speech recognition technique with an allowance for recognition errors. However, most news broadcasts in the US already have closed captions. (In the near future, the worldwide trend will be for broadcasts to feature closed captions.) Thus we use closed-caption texts as transcripts for news videos. In addition, we employ video-caption detection and recognition. We used "CNN Headline News" as our primary source of news for our experiments. Given image sequences, transcripts, and video captions as information sources, Name-It associates extracted faces with extracted name candidates using the correlation of their timing information and face similarity information. Video captions are also taken into account as supplementary information. To associate faces and names, Name-It integrates several advanced image processing and natural-language processing techniques-face sequence extraction and similarity evaluation from videos, name extraction from transcripts, and video-caption recognition. Although these technologies aren't always highly accurate, integrating these results will help the system achieve more accurate output. With respect to face-name association, the Piction system 3 works similarly to Name-It. Piction identifies faces within a given captioned newspaper photograph by extracting faces from the photograph and analyzing the caption to obtain geometric constraints among faces. The system then labels each face with a name. A drawback of Piction is that face location information is assumed to be described in captions-for example, "top row, from left, are Michael, Brian …." On the other hand, Name-It doesn't assume such a description. Instead, while Piction deals with one photograph and caption at a time, Name-It processes many videos-including many news topics-to collect a fraction of a hint from each video fragment to infer face-name association. In doing this, Name-It uses face similarity while Piction doesn't. To realize Name-It, further analysis of video semantic content proves necessary. State-of-theart video analysis technologies automatically extract video structure information. 4, 5 Typically, once a video is given as a target, it's decomposed into segments or shots. These shots are then classified based on the video's structure. This process employs several techniques, such as cut detection, color histogram calculation and comparison, camera motion analysis, motion segmentation, and so on. Among them, cut detection and color his- 1070-986X/99/$10.00
Satoh et al. REF proposed the first approach to associate names with faces in news videos by measuring the frequency of faces and names occurring at the same time.
Name-It: Naming and Detecting Faces in News Videos
IEEE Multim.
IEEE Multim.
Computer Science