Abstract This paper presents a novel way to perform multi-modal face recognition. We use Partial Least Squares (PLS) to linearly map images in different modalities to a common In face recognition, one often seeks to compare gallery images taken under one set of conditions, to a probe image acquired differently. For example, in criminal investigations, we might need to compare mugshots to a sketch drawn by a sketch artist based on the verbal description of the suspect. Similarly, mug-shots or passport photos might be compared to surveillance images taken from a different viewpoint. The probe image might also be of lower resolution (LR) compared to a gallery of high resolution (HR) images. We propose a general framework that uses Partial Least Squares (PLS) [16] to perform recognition in a wide range of multi-modal scenarios. PLS has been used very effectively for face recognition, but in a different manner, with different motivation [17, 19, 20, 21, 22] ; our contribution is to show how and why PLS can be used for cross-modal recognition. More generally, we argue for the applicability of linear projection to an intermediate subspace for multi-modal recognition, also pointing out the value of the Bilinear Model (BLM) [14] for face recognition, which also achieves state-of-the art results on some problems. Experimental evaluation of our framework using PLS with pose variation has shown significant improvements in terms of accuracy and run-time over the state-of-art on the CMU PIE face data set [26] . For sketchphoto recognition, our method is comparable to the state of-art. We also illustrate the potential of our method to handle variation in resolution with a simple, synthetic example. In all three domains we apply exactly the same algorithm, and use the same, simple representation of images. Our generic approach performs either near or better than state-of-the-art approaches that have been designed for specific cross-modal conditions. Our approach matches probe and gallery images by linearly projecting them into an intermediate space where images with the same identity are highly correlated ( Figure  1 ). We argue that for a variety of cross-modality recognition problems, such projections will exist and can be found using PLS and BLM. One consequence of our approach is that we do not need to synthesize an artificial gallery image from the probe image. There has been a huge amount of prior work on comparing images taken in different modalities, which we
In work REF , Sharma and Jacobs use PLS to match probe and gallery images by linearly projecting them into an intermediate space where images with the same identity are highly correlated.
Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch
CVPR 2011
CVPR 2011
Computer Science