Abstract-Existing learning-based face super-resolution (hallucination) techniques generate high-resolution images of a single facial modality (i.e., at a fixed expression, pose and illumination) given one or set of low-resolution face images as probe. Here, we present a generalized approach based on a hierarchical tensor (multilinear) space representation for hallucinating high-resolution face images across multiple modalities, achieving generalization to variations in expression and pose. In particular, we formulate a unified tensor which can be reduced to two parts: a global image-based tensor for modeling the mappings among different facial modalities, and a local patch-based multiresolution tensor for incorporating high-resolution image details. For realistic hallucination of unregistered low-resolution faces contained in raw images, we develop an automatic face alignment algorithm capable of pixel-wise alignment by iteratively warping the probing face to its projection in the space of training face images. Our experiments show not only performance superiority over existing benchmark face super-resolution techniques on single modal face hallucination, but also novelty of our approach in coping with multimodal hallucination and its robustness in automatic alignment under practical imaging conditions. Index Terms-Face hallucination, super-resolution, tensor. I. PROBLEM STATEMENT UE to the intrinsic nonrigidness and extrinsic uncontrollable imaging conditions, face images of noncooperative human subjects captured by live surveillance cameras from a distance often consist of nonlinear variations caused by changes in expression, viewpoint (pose), or illumination. The difficulties in analyzing face images are further compounded when the resolutionoffaceimagesbecomeslow,whichistypicalinCCTVvideos. The missing high-resolution details of facial features and in appearance can deteriorate effective face image analysis and recognition. This raises two important questions need be addressed. 1) Given low-resolution face images, how do we recover or synthesize the lost high-frequency details at a predefined pose and expression but with unknown identity? 2) How do we build a unified model capable of coping with variations from different and multiple modalities where each modality defines one source of variation such as change in expression, pose, or illumination? There Reconstruction-based approach inherits limitations when magnification factor increases [27] . In our approach, we focus on learning-based super-resolution, which when applied to human face images, is also known as "face hallucination" [28] . In this paper, we are motivated by the desire to develop a generalized model capable of hallucinating face images across multiple modalities such as expression or pose variations, given any low-resolution face image input of a single modality. To this end, we formulate a unified tensor space representation incorporating both global and local tensors. Specifically, we model both the high-and low-resolution training face images of multiple modalities using a unified tensor space representation. We reduce this unified tensor to two components: a global image-based tensor that models the mappings among different facial modalities, and a local patch-based multiresolution tensor that incorporates high-resolution face image details into the cross-modality mapping process. Given any low-resolution face image input of a single modality, we first synthesize multiple low-resolution face images of different modalities using the trained global tensor. Based on these synthesized low-resolution face images, we then use the trained local tensor to construct the corresponding high-resolution image for each facial modality. Due to that the high-resolution face image constructed by the local tensor lacks the highest frequency visual information, we further add a high-frequency component residue using nonparametric patch learning from high-resolution training face images. We integrate this sequential statistical modeling process into a Bayesian framework, so that given any low-resolution face image of a single modality, we are able to obtain hallucinated high-resolution face images of multiple modalities. Another important underlying issue for face hallucination is the requirement for accurate pixel-wise face alignment (registration): even a small amount of misalignment can dramatically degrade the hallucination result. Most existing techniques require that both training and test face images have been manually registered to a predefined template [28] , [31] . Consequently, they are of limited use in many practical scenarios, where the faces contained in raw images are normally of nonfrontal views at low resolution. Recently, Liu et al. [37] proposed automatically registering low-resolution face images using an algorithm derived from the model of Lucas-Kanade [33] . However, this algorithm only considers a mean training face (although variance 1057-7149/$25.00
Jia and Gong REF performed face hallucination of a single modality (expression, pose and illumination) into a set of high-resolution (HR) images of different modalities, but used multiple people's images as priors.
Generalized Face Super-Resolution
IEEE Transactions on Image Processing
IEEE Transactions on Image Processing
Computer Science Medicine Mathematics