Face and eye detection algorithms are deployed in a wide variety of applications. Unfortunately, there has been no quantitative comparison of how these detectors perform un der difficult circumstances. We created a dataset of low light and long distance images which possess some of the problems encountered by face and eye detectors solving real world problems. The dataset we created is composed of re imaged images (photohead) and semi-synthetic heads im aged under varying conditions of low light, atmospheric blur, and distances of 3m, 50m, 80m, and 200m. This paper analyzes the detection and localization per formance of the participating face and eye algorithms com pared with the Viola Jones detector and four leading com mercial face detectors. Performance is characterized un der the dif f erent conditions and parameterized by per-image brightness and contrast. In localization accuracy for eyes, the groups/companies focusing on long-range face detec tion outperform leading commercial applications. Over the last several decades, face/eye detection has changed from being solely a topic for research to being commonplace in cheap point-and-shoot cameras. While this may lead one to believe that face detection is a solved problem, it is solved only for easy settings. Detec tion/localization in difficult settings is still an active field of research. Most researchers use controlled data sets such as FERET [14] and PIE [lI], which are captured under con trolled lighting and blur conditions. While these datasets are useful in the creation and testing of detectors, they give little indication of how these detectors will perform in diffi cult or uncontrolled circumstances. 978-1-4577-1359-0/11/$26.00 ©2011 IEEE In ongoing projects at UCCS and Securics addressing long-range and low-light biometrics, we found there were significant opportunities for improvement in the problems of face detection and localization. Face detection is just the first phase of a recognition pipeline and most recognition algorithms need to locate features, the most common being eyes. Until now, there has not been a quantitative compar ison of how well eye detectors perform under difficult cir cumstances. This work created a dataset of low light and long distance images which possess some of the problems face detectors encounter in difficult circumstances. By chal lenging the community in this way, we have helped iden tify state-of-the-art algorithms suitable for real-world face and eye detection and localization and we show directions where future work is needed. This paper discusses twelve algorithms. Participants include the Correlation-based Eye Detection algorithm (CBED), a submission from DEALTE, the Multi-Block Modified Census Transform algorithm (MBMCT), the Min imum Output Sum of Squared Error algorithm (MOSSE), the Robust Score Fusion-based Face Detection algorithm (RSFFD), SIANI, and a contribution from UCSD MPLab. In addition, we compare four leading commercial algo rithms along with the Viola Jones implementation from OpenCV 2.1. In Ta ble 1, algorithms are listed in alphabet ical order with participants on the top section and our own contributions in the bottom. While many toolkits, datasets, and evaluation metrics exist for evaluating face recognition and identification systems, [14, 1] these are not designed for evaluating simple face/eye detection/localization measures. Overall there has been littie focus on difficult detection/localization, despite the ob vious fact that a face not detected is a face not recognizedmultiple papers show that eye localization has a significant impact on recognition rates [10, 7] . The Conference on Intelligent Systems Design and Ap plications [8] performed a face detection competition with two contestants in 2010. Their data sets included a law en forcement mugshot set of 845 images, controlled digital camera captures, uncontrolled captures, and a "tiny face" set intended to mimic captures from surveillance cameras. All except the mugshot database had generally good qual ity. In their conclusions, they state "Obviously, the biggest improvement opportunity lies in the surveillance area with tiny faces." There have been a few good papers evaluating face de tectors. For example, [35] uses a subset of data from LFW, and also considered elliptical models of the ideal face lo cation. However, LFW is a dataset collected using auto mated face detection with refinement. Similarly, [3] lever ages parts of existing data and spends much of their discus sion about what is an ideal face model. The data in these is presented as being somewhat challenging but still most tested detectors did well. We note, however, that evaluating face detectors against an ideal model is not very appropri ate, and in this paper we evaluate detectors with a much more accepting model of a detection -we consider a detec tion correct if the reported model overlaps the ground truth. Many descriptions of face detection algorithms include a small evaluation of their performance, but they often eval uate only the effects of different changes within that algo rithm. [37, 28] Comparisons to others are usually done in the context of proving that the discussed algorithm is better than the state-of-the-art. Because of the inconsistent met rics used, it is often impossible to compare the results of these kinds of evaluations across papers. The results of this competition show that there is room for improvement on larger, blurry, and dark faces, and espe cially so for smaller faces. We set out to create a dataset which would highlight some of the problems presented by somewhat realistic but diffi cult detection/localization scenarios. To do this, we created four sub-sets, each of which presents a different scenario in order to isolate how a detector performs on specific chal lenges. Our naming scheme generally follows scenario width, where scenario is the capture conditions or distance and width is the approximate width of the face in pixels. Note that width alone is a very weak proxy for resolution and many of the images have significant blur within result ing in effective resolution sometimes being much lower. The experiments use the photohead approach for semi synthetic data discussed in [4, 5] allowing control over the conditions and including many faces and poses. The highest quality images, the 80m-500px sub-set, were obtained by imaging semi-synthetic head models generated from PIE. They are displayed on a projector and imaged at 80 meters indoors using a Canon 5D mark II with a Sigma EX 800Imn lens; see Figure lao This camera lens combina tion produced a controlled mid-distance dataset with min imal atmospherics and provides a useful base line for the long distance sub-sets. For the second sub-set, 200m-300px, we imaged the semi synthetic PIE models, this time from 200 meters outside; see Figure lc . We used a Canon 5D mark II with a Sigma EX 800mm lens with an added a Canon EFF 2x II Extender, resulting in an effective 1600mm lens. The captured faces suffered varying degrees of atmospheric blur. For the third sub-set, we re-imaged FERET from 200 me ters; see Figure ld for a zoomed sample. We used a Canon 5D mark II with a Sigma EX 800mm lens. The resulting faces were approximately 50 pixels wide and suffered at mospheric blur and loss of contrast. We chose a subset of these images, first filtered such that our configuration of Vi ola Jones correctly detected the face in 40% of the images. We further filtered by hand-picking only images that con tained discernible detail around the eyes, nose, and mouth. For the final sub-set, we captured displayed images (not models) from PIE[lI] at close range, approximately 3m, in a low light environment, with an example in Figure lb . We captured this set with a Salvador (now FUR) EMCCD cam era. While the Salvador can operate in extremely low light conditions, it produces a low resolution and high noise im age. The noise and low resolution create challenging faces that simulate long-range low-light conditions. To evaluate algorithm performance when given non-face images, we included a proportional number of images that did not contain faces. When evaluating the result, we also considered the false positives and true rejects of images in this non-face dataset. The "non-faces" were almost all nat ural scenes obtained from the web -most were very easily distinguished from faces. Given these datasets, we randomly selected 50 images of each subset to create 4 labeled training datasets. The train ing sets also included the groundtruth for the face bounding box and eye coordinates. The purpose of this set was not to provide a dataset to train new algorithms; 50 images is far too few for that. Instead, it allowed the participants to inter nally validate that their algorithm could process the images and the protocol with some reasonable parameter selection. For testing, we randomly selected 200 images of each subset to create the four testing sets. The location of the face within the image was randomized. An equal number of non-face images was added, and the order of images was then randomized.
For example, Parris et al. REF organized a "Face and Eye Detection on Hard Datasets" challenge at IJCB 2011.
Face and eye detection on hard datasets
2011 International Joint Conference on Biometrics (IJCB)
2011 International Joint Conference on Biometrics (IJCB)
Economics Computer Science