Towards Robust Blind Face Restoration with Codebook Lookup Transformer
Abstract
Blind face restoration is a highly ill-posed problem that often requires auxiliary guidance to 1) improve the mapping from degraded inputs to desired outputs, or 2) complement high-quality details lost in the inputs.
In this paper, we demonstrate that a learned discrete codebook prior in a small proxy space largely reduces the uncertainty and ambiguity of restoration mapping by casting blind face restoration as a code prediction task, while providing rich visual atoms for generating highquality faces.
Under this paradigm, we propose a Transformer-based prediction network, named CodeFormer , to model the global composition and context of the low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded.
To enhance the adaptiveness for different degradation, we also propose a controllable feature transformation module that allows a ﬂexible trade-off between ﬁdelity and quality.
Thanks to the expressive codebook prior and global modeling, CodeFormer outperforms the state of the arts in both quality and ﬁdelity, showing superior robustness to degradation.
Extensive experimental results on synthetic and realworld datasets verify the effectiveness of our method.
Introduction
Face images captured in the wild often suffer from various degradation, such as compression, blur, and noise.
Restoring such images is highly ill-posed as the information loss induced by the degradation leads to inﬁnite plausible high-quality (HQ) outputs given a low-quality (LQ) input.
The ill-posedness is further elevated in blind restoration, where the speciﬁc degradation is unknown.
Despite the progress made with the emergence of deep learning, learning a LQ-HQ mapping without additional guidance in the huge image space is still intractable, leading to the suboptimal restoration quality of earlier approaches.
To improve the output quality, auxiliary information that 1) reduces the uncertainty of LQ-HQ mapping and 2) complements high-quality details is indispensable.
Various priors have been used to mitigate the ill-posedness of this problem, including geometric priors [5, 6, 30, 44], reference priors [23–25], and generative priors [2, 37, 43].
Although improved textures and details are observed, these approaches often suffer from high sensitivity to degradation or limited prior expressiveness .
These priors provide insufﬁcient guidance for face restoration, thus their networks essentially resort to the information of LQ input images that are usually highly corrupted.
As a result, the LQ-HQ mapping uncertainty still exists, and the output quality is deteriorated by the degradation of the input images.
Most recently, based on generative prior, some methods project the degraded faces into a continuous inﬁnite space via iterative latent optimization [26] or direct latent encoding [29].
Despite great realness of outputs, it is difﬁcult to ﬁnd the accurate latent vectors in case of severe degradation, resulting in low-ﬁdelity results (Fig. 1(d)).
To enhance the ﬁdelity, skip connections between encoder and decoder are usually required in this kind of methods [37, 43, 2], as
E n c od e r
[Figure]
n c od e r
T r a n s f
[Figure]
r
[Figure]
illustrated in Fig. 1(a) (top), however, such designs meanwhile introduce artifacts in the results when inputs are severely degraded, as shown in Fig.
1(e).
Different from the aforementioned approaches, this work casts blind face restoration as a code prediction task in a small ﬁnite proxy space of the learned discrete codebook prior , which shows superior robustness to degradation as well as rich expressiveness. The codebook is learned by selfreconstruction of HQ faces using a vector-quantized autoencoder, which along with decoder stores the rich HQ details for face restoration. In contrast to continuous generative priors [11, 37, 43], the combinations of codebook items form a discrete prior space with only ﬁnite cardinality.
Through mapping the LQ images to a much smaller proxy space (e.g., 1024 codes), the uncertainty of the LQ-HQ mapping is signiﬁcantly attenuated, promoting robustness against the diverse degradation, as compared in Figs. 1(d-g). Besides, the codebook space possess greater expressiveness, which perceptually approximates the image space, as shown in Fig. 1(h).
This nature allows the network to reduce the reliance on inputs and even be free of skip connections. Though the discrete representation based on a codebook has been deployed for image generation [4, 11, 34], the accurate code composition for image restoration remains a non-trivial challenge. The existing works look up codebook via nearest-neighbor (NN) feature matching, which is less feasible for image restoration since the intrinsic textures of LQ inputs are usually corrupted. The information loss and diverse degradation in LQ images inevitably distort the feature distribution, prohibiting accurate feature matching.
As depicted in Fig. 1(b) (right), even after ﬁne-tuning the encoder on LQ images, the LQ features cannot cluster well to the exact code but spread into other nearby code clusters, thus the nearest-neighbor matching is unreliable in such cases.
Tailored for restoration, we propose a Transformer-based code prediction network, named CodeFormer , to exploit global compositions and long-range dependencies of LQ faces for better code prediction.
Speciﬁcally, taking the LQ features as input, the Transformer module predicts the code token sequence which is treated as the discrete representation of the face images in the codebook space. Thanks to the global modeling for remedying the local information loss in LQ images, the proposed
CodeFormer shows robustness to heavy degradation and keeps overall coherence.
Comparing the results presented in Figs.
1(f-g), the proposed CodeFormer is able to recover more details than the nearest-neighbor matching, such as the glasses, improving both quality and ﬁdelity of restoration.
Moreover, we propose a controllable feature transformation module with an adjustable coefﬁcient to control the information ﬂow from the LQ encoder to decoder.
Such design allows a ﬂexible trade-off between restoration quality and ﬁdelity so that the continuous image transitions between them can be achieved.
This module enhances the adaptiveness of CodeFormer under different degradations, e.g., in case of heavy degradation, one could manually reduce the information ﬂow of LQ features carrying degradation to produce high-quality results.
Equipped with the above components, the proposed CodeFormer demonstrates superior performance in existing datasets and also our newly introduced WIDER-Test dataset that consists of 970 severely degraded faces collected from the WIDER-Face dataset [42].
In addition to face restoration, our method also demonstrates its effectiveness on other challenging tasks such as face inpainting, where long-range clues from other regions are required.
Systematic studies and experiments are conducted to demonstrate the merits of our method over previous works.
Related Work
Blind Face Restoration.
Since face is highly structured, geometric priors of faces are exploited for blind face restoration.
Some methods introduce facial landmarks [6], face parsing maps [5, 30, 41], facial component heatmaps [44], or 3D shapes [16, 28, 48] in their designs.
However, such prior information cannot be accurately acquired from degraded faces.
Moreover, geometric priors are unable to provide rich details for high-quality face restoration.
Reference-based approaches [9, 23–25] have been proposed to circumvent the above limitations.
These methods generally require the references to possess same identity as the input degraded face.
For example, Li et al.
[25] propose a guided face restoration network that consists of a warping subnetwork and a reconstruction subnetwork, and a high-quality guided image of the same identity as input is used for better restoring the facial details.
However, such references are not always available.
DFDNet [23] pre-constructs dictionaries composed of high-quality facial component features.
However, the component-speciﬁc dictionary features are still insufﬁcient for high-quality face restoration, especially for the regions out of the dictionary scope ( e.g. , skin, hair).
To alleviate this issue, recent VQGAN-based methods [39, 46] explores a learned HQ dictionary, which contains more generic and rich details face restoration. Recently, the generative facial priors from pre-trained generators, e.g., StyleGAN2 [21], have been widely explored for blind face restoration.
It is utilized via different strategies of iterative latent optimization for effective GAN inversion [12, 26] or direct latent encoding of degraded faces [29].
However, preserving high ﬁdelity of the restored faces is challenging when one projects the degraded faces into the continuous inﬁnite latent space.
To relieve this issue, GLEAN [2, 3], GPEN [43], and
GFPGAN [37] embed the generative prior into encoder-decoder network structures, with additional structural information from input images as guidance.
Despite the improvement of ﬁdelity, these methods highly rely on inputs through the skip connections, which could introduce artifacts to results when inputs are severely corrupted.
Dictionary Learning.
Sparse representation with learned dictionaries has demonstrated its superiority in image restoration tasks, such as super-resolution [13, 32, 33, 40] and denoising [10].
However, these methods usually require an iterative optimization to learn the dictionaries and sparse coding, suffering from high computational cost.
Despite the inefﬁciency, their high-level insight into learning a dictionary to store HQ texture cues has inspired deep learning-based image restoration [18, 47] and synthesis methods [11, 34].
Jo and Kim [18] construct a look-up table (LUT) by transferring the network output values to a LUT, so that only a simple value retrieval is needed during inference.
However, storing HQ textures in the image domain usually requires a huge LUT, limiting its practicality.
VQVAE [34] is ﬁrst to introduce a highly compressed codebook learned by a vector-quantized autoencoder model.
VQGAN [11] further adopts the adversarial loss and perceptual loss to enhance perceptual quality at a high compression rate, signiﬁcantly reducing the codebook size without sacriﬁcing its expressiveness.
Unlike the large hand-crafted dictionary [18, 23], the learnable codebook automatically learns optimal elements for HQ image reconstruction, providing superior efﬁciency and expressiveness as well as circumventing the laborious dictionary design.
Inspired by the codebook learning, this paper investigates a discrete proxy space for blind face restoration.
Different from recent VQGAN-based approaches [39, 46], we exploit the discrete codebook prior by predicting code sequences via global modeling, and we secure prior effectiveness by ﬁxing the encoder. Such designs allow our method to take full advantage of the codebook so that it does not depend on the feature fusion with LQ cues, signiﬁcantly enhancing the robustness of face restoration.
HQDecoder ! !
HQEncoder 012…N
Codebook " " !
LQEncoder " # ! ! " ! "
"#$ " "#% " & !
$ $ !(#,%)
=argmin ! !
∈( $ )(#,%) −- *
Nearest-NeighborMatching 10 42 8 1 52 12 0 28 16 23 11 49 4 87 1(a) (b)
HQDecoder ! ! !
T r a n s f o r m e r
CodePrediction 012…N
Codebook " # … … 108 #! $
CFT $ # $ ' %$ ' ./ ×" & ! "
# " $ #" $
ControllableFeatureTransformation (CFT)
Fixedmodules + + "
Multiplication Addition "+ .
̂.
Methodology
The main focus of this work is to exploit a discrete representation space that reduces the uncertainty of restoration mapping and complements high-quality details for the degraded inputs.
Since local textures and details are lost and corrupted in low-quality inputs, we employ a Transformer module to model the global composition of natural faces, which remedies the local information loss, enabling high-quality restoration.
The overall framework is illustrated in Fig.
We ﬁrst incorporate the idea of vector quantization [11, 34] and pre-train a quantized autoencoder through self-reconstruction to obtain a discrete codebook and the corresponding decoder (Sec. 3.1).
The prior from the codebook combination and decoder is then used for face restoration.
Based on this codebook prior, we then employ a Transformer for accurate prediction of code combination from the low-quality inputs (Sec. 3.2).
In addition, a controllable feature transformation module is introduced to exploit a ﬂexible trade-off between restoration quality and ﬁdelity (Sec. 3.3).
The training of our method is divided into three stages accordingly.
3.1 Codebook Learning (Stage I)
To reduce uncertainty of the LQ-HQ mapping and complement high-quality details for restoration, we ﬁrst pre-train the quantized autoencoder to learn a context-rich codebook, which improves network expressiveness as well as robustness against degradation.
As shown in Fig.
2(a), the HQ face image I h ∈
R H
× W × 3 is ﬁrst embeded as a compressed feature
Z h ∈ R m × n × d by an encoder E H .
Following VQVAE [34] and VQGAN [11], we replace each “pixel” in Z h with the nearest item in the learnable codebook C = { c k ∈ R d } Nk =0 to obtain the quantized feature Z c ∈ R m × n × d and the corresponding code token sequence s ∈ { 0 , · · · , N − 1 } m · n :
[Equation]
The decoder D H then reconstructs the high-quality face image I rec given Z c .
The m · n code token sequence s forms a new latent discrete representation that speciﬁes the respective code index of the learned codebook, i.e., Z ( i,j ) c
= c k when s ( i,j ) = k .
Training Objectives.
To train the quantized autoencoder with a codebook, we adopt three image-level reconstruction losses: L1 loss L 1 , perceptual loss [Equation]
19, 45] L per , and adversarial loss [11]
[Equation]
adv :
[Equation]
where Φ denotes the feature extractor of VGG19 [31].
Since, image-level losses are underconstrained when updating the codebook items, we also adopt the intermediate code-level loss [11, 34]
L featcode to reduce the distance between codebook C and input feature embeddings Z h :
[Equation]
where sg ( · ) stands for the stop-gradient operator and β = 0 .
25 is a weight trade-off for the update rates of the encoder and codebook.
Since the quantization operation in Eq.
(1) is non-differentiable, we adopt straight-through gradient estimator [11, 34] to copy the gradients from the decoder to the encoder.
The complete objective of codebook prior learning L codebook is:
[Equation]
where λ adv is set to 0.8 in our experiments.
Codebook Settings.
Our encoder E H and decoder D H consist of 12 residual blocks and 5 resize layers for downsampling and upsampling, respectively.
Hence we obtain a large compression ratio of r = H/n = W/m = 32 , which leads to a great robustness against degradation and a tractable computational cost for our global modeling in Stage II.
Although more codebook items could ease reconstruction, the redundant elements could cause ambiguity in subsequent code predictions.
Hence, we set the item number N of codebook to 1024, which is sufﬁcient for accurate face reconstruction.
Besides, the code dimension d is set to 256.
3.2 Codebook Lookup Transformer Learning (Stage II)
Due to corruptions of textures in LQ faces, the nearest-neighbour (NN) matching in Eq.
(1) usually fails in ﬁnding accurate codes for face restoration.
As depicted in Fig. 1(b), LQ features with diverse degradation could deviate from the correct code and be grouped into nearby clusters, resulting in undesirable restoration results, as shown in Fig. 1(f).
To mitigate the problem, we employ a
Transformer to model global interrelations for better code prediction.
Built upon the learned autoencoder presented in Sec. 3.1, as shown in Fig.
2(b) , we insert a Transformer [36] module containing nine self-attention blocks following the encoder.
We ﬁx the codebook C and decoder D H and ﬁnetune the encoder E H for restoration.
The ﬁnetuned encoder is denoted as E L .
To obtain the LQ features Z l ∈ R m × n × d through E L , we ﬁrst unfold the features into m · n vectors Z vl ∈ R ( m · n ) × d , and then feed them to the Transformer module.
The s -th self-attention block of Transformer computes as the following:
[Equation]
where X
= Z vl .
The query Q , key K , and value V are obtained from X s through linear layers.
We add a sinusoidal positional embedding P ∈ R ( m · n ) ×
d [1, 7] on the queries Q and the keys K to increase the expressiveness of modeling sequential representation.
Following the self-attention blocks, a Linear layer is adopted to project features to the dimension of ( m · n ) × N . Overall, taking the encoding feature Z vl as an input, the Transformer predicts the m · n code sequence ˆ s ∈ { 0 , · · · , | N | − 1 } m · n in form of the probability of the N code items.
The predicted code sequence then retrieves the m · n respective code items from the learned codebook, forming the quantized feature ˆ Z c ∈ R m × n × d that produces a high-quality face image through the ﬁxed decoder
D H .
Thanks to our large compression ratio ( i.e. , 32), our Transformer is effective and efﬁcient in modeling global correlations of LQ face images.
Training Objectives.
We train Transformer module T as well as ﬁnetune the encoder E L for restoration, while the codebook C and decoder D H are kept ﬁxed.
Instead of employing reconstruction loss and adversarial loss in the image-level, only code-level losses are required in this stage: 1) crossentropy loss L tokencode for code token prediction supervision, and 2) L2 loss L feat (cid:48) code to force the LQ feature Z l to approach the quantized feature Z c from codebook, which eases the difﬁculty of token prediction learning:
[Equation]
(6) where the ground truth of latent code s is obtained from the pre-trained autoencoder in Stage I (Sec. 3.1), thus the quantized feature Z c is then retrieved from codebook according to the s .
The ﬁnal objective of Transformer learning is:
[Equation]
(7) where λ token is set to 0.5 in our method.
Note that our network after this stage has already equipped with superior robustness and effectiveness in face restoration, especially for severely degraded faces.
3.3 Controllable Feature Transformation (Stage III)
Despite our Stage II has obtained a great face restoration model, we also investigate a ﬂexible tradeoff between quality and ﬁdelity of face restoration.
Thus, we propose the controllable feature transformation (CFT) module to control information ﬂow from LQ encoder E L to decoder D H .
Speciﬁcally, as shown in Fig. 2, the LQ features F e are used to slightly tune the decoder features F d through spatial feature transformation [38] with the afﬁne parameters of α and β .
An adjustable coefﬁcient w ∈ [0 , 1] is then used to control the relative importance of the inputs:
[Equation]
where P θ denotes a stack of convolutions that predicts α and β from the concatenated features of c ( F e , F d ) .
We adopt the CFT modules at multiple scales s ∈ { 32 , 64 , 128 , 256 } between encoder and decoder.
Such a design allows our network to remain high ﬁdelity for mild degradation and high quality for heavy degradation. Speciﬁcally, one could use a small w to reduce the reliance on input LQ images with heavy degradation, thus producing high-quality outputs.
The larger w will introduce more information from LQ images to enhance the ﬁdelity in case of mild degradation.
Training Objectives.
To train the controllable modules and ﬁnetune the encoder E L in this stage, we keep the code-level losses of L tf in Stage II, and also add image-level losses of L 1 , L per , and
L adv , which are the same as that in Stage I except that I rec is replaced by restoration output I res .
The complete loss for this stage is the sum of above losses weighted with their original weight factors.
We set the w to 1 during training of this stage, which then allows network to achieve continuous transitions of results by adjusting w in [0 , 1] during inference.
Even so, unless otherwise stated, we set the w = 0 by default in inference to make quality a priority.
Experiments 4.1 Datasets
Training Dataset.
We train our models on the FFHQ dataset [21], which contains 70,000 high-quality (HQ) images, and all images are resized to 512 × 512 for training.
To form training pairs, we synthesize
LQ images I l from the HQ counterparts
I h with the following degradation model [23, 37, 43]:
[Equation]
where the HQ image I h is ﬁrst convolved with a Gaussian kernel k σ , followed by a downsampling of scale r .
After that, additive Gaussian noise n δ is added to the images, and then JPEG compression with quality factor q is applied.
Finally, the LQ image is resized back to 512 × 512 .
We randomly sample σ , r , δ , and q from [1 , 15] , [1 , 30] , [0 , 20] , and [30 , 90] , respectively.
Testing Datasets.
We evaluate our method on a synthetic dataset CelebA-Test and three real-world datasets: LFW-Test, WebPhoto-Test, and our proposed WIDER-Test.
CelebA-Test contains 3, images selected from the CelebA-HQ dataset [20], where LQ images are synthesized under the same degradation range as our training settings.
The three real-world datasets respectively contain three different degrees of degradation, i.e. , mild (LFW-Test), medium (WebPhoto-Test), and heavy (WIDER-Test).
LFW-Test consists of the ﬁrst image of each person whose name starting with A in
LFW dataset [17], containing 431 images.
WebPhoto-Test [37] consists of 407 low-quality faces collected from the Internet.
Our WIDER-Test comprises 970 severely degraded face images from the
WIDER Face dataset [42], providing a more challenging dataset to evaluate the generalizability and robustness of blind face restoration methods.
4.2 Experimental Settings and Metrics
Settings.
We represent a face image of 512 × 512 × 3 as a 16 × 16 code sequence.
For all stages of training, we use the Adam [22] optimizer with a batch size of 16.
We set the learning rate to 8 × 10 −for stages I and II, and adopt a smaller learning rate of 2 × 10 − 5 for stage III.
The three stages are trained with 1.5M, 200K, and 20K iterations, respectively.
Our method is implemented with the
PyTorch framework and trained using four NVIDIA Tesla V100 GPUs.
Metrics.
For the evaluation on CelebA-Test with ground truth, we adopt PSNR, SSIM, and LPIPS [45] as metrics.
We also evaluate the identity preservation using the cosine similarity of features from [Figure] [Figure] ArcFace network [8], denoted as IDS.
For the evaluation on real-world datasets without ground truth, we employ the widely-used non-reference perceptual metrics: FID [15] and NIQE [27].
4.3 Comparisons with State-of-the-Art Methods
We compare the proposed CodeFormer with state-of-the-art methods, including PULSE [26], DFDNet [23], PSFRGAN [5], GLEAN [3], GFP-GAN [37], and GPEN [43].
We conduct extensive comparisons on both synthetic and real-world datasets.
Evaluation on Synthetic Dataset.
We ﬁrst show the quantitative comparison on the CelebA-Test in Table 1.
In terms of the image quality metrics LPIPS, FID, and NIQE, our CodeFormer achieves the best scores than existing methods.
Besides, it also faithfully preserves the identity, reﬂected by the highest IDS score and PSNR.
Additionally, we present the qualitative comparison in Fig.
The compared methods fail to produce pleasant restoration results, e.g., DFDNet [23], PSFRGAN [5], [Figure]
[Table] [Table] [Table] GFP-GAN [37], and GPEN [43] introduce obvious artifacts and GLEAN [3] produces over-smoothed results that lack facial details.
Moreover, all compared methods are unable to preserve the identity.
Thanks to the expressive codebook prior and global modeling, CodeFormer not only produces high-quality faces but also preserves the identity well, even when inputs are highly degraded.
Evaluation on Real-world Datasets.
As presented in Table 2, our CodeFormer achieves comparable perceptual quality of FID score with the compared methods on the real-world testing datasets with mild and medium degradation, and the best score on the testing dataset with heavy degradation.
Although PULSE [26] also obtains good perceptual NIQE score, it cannot preserve the identity of input images, as the identity score of IDS and visual results respectively suggested in Table 1 and
Fig. From the visual comparisons in Fig.
4, it is observed that our method shows exceptional robustness to the real heavy degradation and produces most visually pleasing results.
Notably, CodeFormer successfully preserves the identity and produces natural results with rich details.
4.4 Ablation Studies
Effectiveness of Codebook Space.
We ﬁrst investigate the effectiveness of the codebook space.
As shown in Exp.
(a) of Table 3, removing the codebook ( i.e. , directly feeding the encoder features Z l to the decoder) results in worse LPIPS and IDS scores.
The results suggest that the discrete space of codebook is the key to ensure the robustness and effectiveness of our model.
Superiority of Transformer-based Code Prediction.
To verify the superiority of our Transformerbased code prediction for codebook lookup, we compare it with two different solutions, i.e. , nearestneighbour (NN) matching, i.e. , Exp. (b), and a CNN-based code prediction module, i.e. , Exp. (c),
[Figure]
that adopts a Linear layer for prediction following encoder E L .
As shown in Table 3, the comparison of Exps.
(b) and (c) indicates that adopting code prediction for codebook lookup is more effective than NN feature matching.
However, the local nature of convolution operation of CNNs restricts its modeling capability for long code sequence prediction.
In comparison to the pure CNN-based method, i.e. , Exp.
(c), our Transformer-based solution produces better-ﬁdelity results in terms of LPIPS and IDS scores, as well as higher accuracy of code prediction under all degradation degrees, as shown in Fig.
Besides, the superiority of CodeFormer is also demonstrated in visual comparisons shown in Fig.
5 and Fig.
Importance of Fixed Decoder.
Unlike the large dictionary ( ∼ 3.2G) in DFDNet [23], which aims to store a massive quantity of facial details, we deliberately adopt a compact codebook C ∈ R N × d with N =1024 and d =256 that only keeps the essential codes for face restoration, which then activate the detailed cues stored in the pre-trained decoder. Thus, the codebook must be used alongside the decoder to fully unleash its potential.
To vindicate our design, we conduct two studies: 1) ﬁxing both codebook and decoder, i.e. , Exp.
(g), and 2) ﬁxing codebook but ﬁne-tuning decoder, i.e. , Exp.
(e).
Table 3 shows ﬁne-tuning decoder deteriorates the performance, validating our statement.
This is because ﬁne-tuning the decoder destroys the learned prior that is held by the pre-trained codebook and decoder, resulting in suboptimal performance.
Therefore, we keep the decoder ﬁxed in our method. [Figure] =! = 0.
= 0. = 0. = 0. ! =
[Figure] fidelity Flexibility of Controllable Feature Transformation Module. Considering the diverse degradation in real-world LQ face images, we provide a controllable feature transformation module (CFT) to allow a ﬂexible trade-off between quality and ﬁdelity.
As shown in Fig.
7, a smaller w tends to produce a high-quality result while a larger w improves the ﬁdelity.
While such a ﬂexibility is rarely explored in previous work, here we show that it is an appealing strategy to improves the adaptiveness of our method for different scenarios.
As shown in Table 3, Exp.
(f), i.e. , setting the coefﬁcient w toincreases the identity score but decreases the quality score.
In this work, we prioritize quality, and hence the coefﬁcient w is set to 0 by default.
4.5 Running time
We compare the running time of state-of-the-art methods [26, 23, 5, 2, 37, 43] and the proposed
CodeFormer.
All existing methods are evaluated on 512 2 face images using their publicly available code. As shown in Table 5, the proposed CodeFormer has a similar running time as PSFRGAN [5] and GPEN [43] that can infer one image within 0.1s. Meanwhile, our method achieves the best performance in terms of LPIPS on the Celeb-Test dataset.
[Table]
[Figure]
4.6 Extensions
Face Color Enhancement.
We ﬁnetune our model on face color enhancement using the same color augmentations (random color jitter and grayscale conversion) as GFP-GAN (v1) [37].
We compare our method with GFP-GAN (v1) [37] on the real-world old photos (from CelebChild-Test dataset [37]) with color loss.
The proposed CodeFormer generates high-quality face images with more natural color and faithful details.
[Figure]
Face Inpainting. The proposed Codeformer can be easily extended to face inpainting, and it shows great performance even in large mask ratios.
To build training pairs, we use a publicly available script [43] to randomly draw irregular polyline masks for generating masked faces.
We compare our method with two state-of-the-art face inpainting methods CTSDG [14] and GPEN [43], as well as Nearest-Neighbor matching for codebook lookup.
As shown in Fig.
9, CTSDG and GPEN struggle in cases with large masks.
Using Nearest-Neighbor matching within our framework roughly reconstructs the face structures, but it also fails in restoring complete visual parts such as the glasses and the eyes.
In contrast, our CodeFormer generates high-quality natural faces without strokes and artifacts.
[Figure]
4.7 Limitation
Our method is built on a pre-trained autoencoder with a codebook.
Thus, the capability and expressiveness of the autoencoder could affect the performance of our method.
Though the identity inconsistency issue is signiﬁcantly relieved by the Transformer’s global modeling, inconsistency still exists in some rare visual parts such as accessories, where the current codebook space cannot seamlessly represent the image space.
Using multiple scales in the codebook space to explore more ﬁne-grained visual quantization may be a solution.
Although CodeFormer exhibits great robustness in most cases, when it comes to side faces, CodeFormer offers limited superiority to other methods and also cannot produce good results, as failure cases shown in Fig.
This is expected because there are only few side faces in the FFHQ training dataset, thus, the codebook is unable to learn sufﬁcient codes for this case, leading to less effectiveness in both reconstruction and restoration.
[Figure]
Conclusion
This paper aims to address the fundamental challenges in blind face restoration.
With a learned small discrete but expressive codebook space, we turn face restoration to code token prediction, signiﬁcantly reducing the uncertainty of restoration mapping and easing the learning of restoration network.
To remedy the local loss, we explore global composition and dependency from degraded faces via an expressive Transformer module for better code prediction.
Beneﬁting from these designs, our method shows great expressiveness and strong robustness against heavy degradation.
To enhance the adaptiveness of our method for different degradation, we also propose a controllable feature transformation module that allows a ﬂexible trade-off between ﬁdelity and quality.
Experimental results suggest the superiority and effectiveness of our method. Acknowledgement
This study is supported under the RIE2020 Industry Alignment