C ONTROLLING THE MEMORABILITY OF REAL AND UNREAL FACE IMAGES
Anonymous authors
Paper under double-blind review
A BSTRACT
Everyday, we are bombarded with many photographs of faces, whether on social media, television, or smartphones.
From an evolutionary perspective, faces are intended to be remembered, mainly due to survival and personal relevance.
However, all these faces do not have the equal opportunity to stick in our minds.
It has been shown that memorability is an intrinsic feature of an image
but yet, it is largely unknown what attributes make an image more memorable.
In this work, we aimed to address this question by proposing a fast approach to modify and control the memorability of face images.
In our proposed method, we ﬁrst found a hyperplane in the latent space of StyleGAN to separate high and low memorable images.
We then modiﬁed the image memorability (while maintaining the identity and other facial features such as age, emotion, etc.) by moving in the positive or negative direction of this hyperplane normal vector.
We further analyzed how different layers of the StyleGAN augmented latent space contribute to face memorability.
These analyses showed how each individual face attribute makes an image more or less memorable.
Most importantly, we evaluated our proposed method for both real and unreal (generated) face images.
The proposed method successfully modiﬁes and controls the memorability of real human faces as well as unreal (generated) faces.
Our proposed method can be employed in photograph editing applications for social media, learning aids, or advertisement purposes.
I NTRODUCTION
In our everyday life, we are exposed to many pictures of scenes, objects and faces.
Research has shown that all images do not have the same likelihood to be recalled later (Isola et al., 2011;
Bainbridge et al., 2013; Isola et al., 2014).
Although different people have different abilities in memorizing visual contents (image or video), it has been shown that memorability is an intrinsic feature of an image and it is consistent across different observers (Isola et al., 2011;
Bainbridge et al., 2013; Isola et al., 2014).
In other words, memorability of an image is an attribute of that image which can be measured, predicted or manipulated (Isola et al., 2011).
So far there have been several studies that have attempted to understand, predict, and even modify image memorability.
(Khosla et al., 2013a; 2015; Goetschalckx & Wagemans, 2019; Fajtl et al.; Needell & Bainbridge, 2021; Squalli-Houssaini et al., 2018; Almog et al.), and a few work attempted to modify image memorability (Khosla et al., 2013b;
Siarohin et al., 2017; Sidorov, 2019; Goetschalckx et al., 2019).
However, for practical applications (e.g., education and advertisement), it is most important to have methods for memorability modiﬁcations.
Moreover, such approaches will help understanding what constitute as image memorability, i.e. ”What makes an image memorable?”.
This work proposes a new framework based on Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) for modifying face memorability as a facial attribute.
Older approaches on modifying face memorability manually used face features (Khosla et al., 2013b), such as {+SIFT (Lowe, 2004), HOG2x
+}
(Dalal & Triggs, 2005), and Local Binary Pattern (LBP) (Ojala et al., 2002).
Recently, GANs have been used to modify the memorability of images. (Goetschalckx et al., 2019; Sidorov, 2019).
Goetschalckx et al.
(2019) leverage latent vector modiﬁcation to change the memorability of the fake food, scenes, and animal images generated by BigGAN (Brock et al., 2018).
This memorability modiﬁcation affects several attributes of the image, such as size, color, and shape.
In our work, we aim to modify memorability of face images of real people while keeping their identity, consequently, their method cannot be used here.
The largest dataset of the human faces with their memorability annotations is US 10k Face database (Bainbridge et al., 2013), which includes 2222 face images with their memorability scores acquired from human observers in an experiment. StyleGANs are the state-of-the-art models for generating real-looking faces.
Our utilization of StyleGANs is required to create a dataset of realistic-looking faces. Not only that, for modifying the memorability of real faces, we need StyleGANs to reconstruct real faces with high accuracy. To date, StyleGANs are the state-of-the-art models in reconstructing real-face images. StyleGANs provide an extended latent space which we leverage to derive a more accurate memorability hyperplane.
Also, the face attributes of StyleGANs are especially disentangled in comparison to other GANs, which is required to accurately modify faces for memorability and study the attributes contributing to this.
For this, we employed pre-trained StyleGAN1 (Karras et al., 2019) and StyleGAN2 (Karras et al., 2020) on the FFHQ dataset (Karras et al., 2019) to generate 100k fake faces.
Next, we adopted computational memorability models which we trained on the US 10k Face Database, to predict the memorability of the generated face images and organized them into faces with high or low memorability.
Inspired by Shen et al.
, we found a hyperplane in latent subspace to separate the highly-memorable and low-memorable faces.
We showed that both latent space and extended latent space can be used for ﬁnding the separating hyperplane.
After ﬁnding the hyperplane, we moved the latent vector of each image, in the positive or negative direction of the normal vector of that hyperplane and changed the distance of the latent vector from the separating hyperplane to manipulate the memorability of that image.
We name the normal vector of this hyperplane, memorability modiﬁcation vector.
With this proposed approach, we could control the amount of change in memorability by using different weights for the memorability modiﬁcation vector.
In contrast to the method proposed by Sidorov (2019), our method does not require training another auxiliary network for modifying face memorability and the amount of change in memorability is controlled by a hyperparameter.
Since different hyperplanes for different facial attributes in StyleGAN latent space (Shen et al., 2020; H ¨ ark ¨ onen et al., 2020) can be found, our method can be used to modify the memorability of the images conditionally.
For example, we are able to change the memorability of the face while maintaining the length of the hair and the existence of eyeglasses.
For this, we ﬁrst ﬁnd the corresponding hyperplanes for these attributes, and then leverage projection to those subspaces to have the desired attributes ﬁxed while changing memorability.
StyleGAN produces high quality real-looking images which are near impossible to differentiate from real images.
To make sure our memorabilitymodiﬁed faces still look real, we considered the Frechet Inception Distance (FID) (Heusel et al., 2017) and Kernel Inception Distance (KID) (Bi ´ nkowski et al., 2018) scores of the generated faces from the StyleGAN as the baseline.
Then by calculating the FID score and KID score of our memorability modiﬁed images, we showed that our modiﬁed faces still looked real.
For real faces, we ﬁrst embedded the face images into GAN latent space using Image2Style (Abdal et al., 2019) and the method provided by Karras et al.
After ﬁnding the image latent vector, we modiﬁed the real face memorability in the same way previously explained for synthetic faces.
Figure 1 illustrates the general idea of our approach.
Finally, we examined how different layers of extended latent space of StyleGAN affect the image of each face and its memorability.
1. R ELATED WORKS
Image Memorability.
People have different capabilities in memorizing different visual events (Hunt et al., 1981).
In spite of these differences, through a series of experiments, Isola et al.
(2011) showed that people consistently remember some images and forget others.
They designed an online memory game experiment and recruited a large number of participants through Amazon Mechanical Turk.
In this experiment, participants observed a series of images presented in a sequence and were tasked to detect repetitive images in the sequence.
Then Isola et al.
(2011) measured a memorability score for each image, which corresponded to the rate of people remembering an image after single exposure to that image in the sequence.
Khosla et al.
(2015) created the largest annotated image memorability dataset (LaMem), which consists of 60,000 images, mostly objects, scenes, and animals.
Using this dataset, they introduced the ﬁrst deep model for predicting image memorability.
This model uses AlexNet (Krizhevsky et al., 2012) as its backbone architecture.
Moreover, Needell & Bainbridge (2021) introduced new architectures based on residual networks (He et al., 2016) to improve the performance of memorability prediction.
In addition to these models, Fajtl et al.
leveraged Attention Maps to introduce AMNet for predicting memorability.
[Figure]
Dividing
[Figure]
low
[Figure]
𝑤=𝑤+𝛼.𝑤 ∗ or 𝑧=𝑤+𝛼.𝑧 ∗ Modifying the latent vector of the image - 𝛼 +
Face Generator
Modified image
Generated Face
[Figure]
𝒛 0 1 … 0
R ea l F ace
Gan Inversion
Finding the separating hyperplane and 𝑤 ∗
High-memorable faces low-memorable faces 𝑤 ∗ or 𝑧 ∗
F a k e F ace
Generative Adversarial Networks (GANs). With the development of Generative Adversarial Networks (Goodfellow et al., 2014), we are now capable of generating real-looking synthetic images that are indistinguishable from real images. Generally, these networks are composed of two parts; a generative and a discriminative network. The goal of the generative network is to generate reallooking images to fool the discriminator and the goal of the discriminator is to learn to distinguish generated images from real images.
These two networks are optimized through a minmax game where both sides compete to reach their speciﬁed goals.
In recent years, there have been huge improvements in this area and many different GANs have been introduced to produce natural-looking images such as Progressive GAN (Karras et al., 2017), DCGAN (Radford et al., 2015), and CycleGAN (Zhu et al., 2017).
In this work, we used pre-trained StyleGAN1 (Karras et al., 2019) and StyleGAN2 (Karras et al., 2020) on the FFHQ dataset (Karras et al., 2019) to generate real-looking faces.
Modifying Image Memorability.
While image memorability modiﬁcation has many potential applications (e.g. in education or advertisement), it has not been adequately investigated.
Khosla et al.
(2013b) proposed a pioneering method for changing face memorability.
It leveraged Active Appearance Models (AAMs (Cootes et al., 2001)) to represent faces by their shape and appearance.
Then the loss function was deﬁned based on the cost of modifying identity, modifying facial attributes, and memorability.
As a result, in their method, the identity was ﬁxed.
Another approach (Sidorov, 2019) used VAE/GAN (Larsen et al., 2016), StarGAN (Choi et al., 2018), and AttGAN (He et al., 2019) and trained them with three memorability levels (poorly memorable, moderately memorable, and highly memorable) of faces and modiﬁed the memorability of different faces to these three levels only.
Additionally, Siarohin et al.
(2017) utilized style transfer to increase the memorability of an input image.
However, the added style adversely affected the realness of the modiﬁed images, such that it barely could be used in real-world applications.
Most recently, Goetschalckx et al.
(2019) trained a transformer network to change the memorability of each generated image by BigGAN (Brock et al., 2018) through modifying their latent vectors.
Their method works on generated images of objects and scenes.
M ETHOD
The overview of our proposed method is depicted schematically in Figure 1.
Below we explain the approach step by step.
2. C REATING THE DATASET
For the purpose of analyzing the latent vectors of the GANs and their relation to memorability, we need a large dataset of face images with their memorability scores.
The largest dataset available for face images is the 10k US Adult Faces Database (Bainbridge et al., 2013).
This database contains 10,168 natural human face images and 2,222 of these images are annotated with memorability scores.
To create a larger dataset for face images with their corresponding memorability scores, we leveraged StyleGAN1 and StyleGAN2, which are the state-of-the-art models for creating realisticlooking face images.
These models were pre-trained on the FFHQ dataset (Karras et al., 2019) which consists of 70,000 high-quality face images with 1024 × 1024 resolution with variations in age, gender, and glasses.
We created two different datasets with StyleGAN1 and StyleGAN2.
We randomly sampled 100k z ∈ R 1 × 512 from a standard normal distribution with truncation to produce high-quality synthetic face images and saved their mappings in the extended latent space ( R 18 × 512 ) of both GANs.
2. P REPROCESSING STEP
The generated images from the StyleGAN have 1024 × 1024 resolution in three channels.
For the purpose of acquiring the memorability of the generated images, we had to preprocess them to predict their memorability with the computational memorability predicting models (assessor).
We leveraged VGG16 (Simonyan & Zisserman, 2014), ResNet50 (He et al., 2016), and SENet50 (Hu et al., 2018) that are pre-trained on VGGFace2 (Cao et al., 2018) and ﬁne-tuned them on the US 10k
Face Database to correctly estimate face memorability scores.
This dataset consists of oval-shaped human faces with white backgrounds.
10k US Adult Face images have the same height of 256 with different widths.
Hence, we had to ﬁrst ovalize the generated faces, then compute their memorability scores.
We used MTCNN (Zhang et al., 2016) for detecting the face in the generated images and masked an oval on it to make all the images similar to the US 10k Face Database (See Figure 2).
[Figure]
2. P REDICTING THE MEMORABILITY SCORES
As explained in previous section, memorability assessors are trained on 10K US face database which includes ovalized faces.
We ﬁrst, tested whether the ovalization process is necessary to obtain memorability scores for generated faces.
For this, we ﬁne-tuned the pre-trained SENet50 (Hu et al., 2018),
VGG16 (Simonyan & Zisserman, 2014), and ResNet50 (He et al., 2016) for predicting face memorability scores on 10K US face database.
We calculated the memorability of both the oval-shaped and square-shaped faces generated from StyleGAN1 and StyleGAN2.
The distributions of the memorability scores using SENet with oval and squared faces are shown in Figure 3.
As can be seen, the memorability distributions are very similar for oval and square faces.
We further calculated the
Kendall rank correlation (Schaeffer & Levitt, 1956) and Spearman’s rank correlation for the paired oval-shaped and square-shaped images.
We observed that SeNet50 and VGG16 have a high-rank score (see Table 1).
These results showed that these models can also be used for the square-shaped face images that contain a background.
The beneﬁt of using square-shaped faces is that we can have more control on the hair of the person.
Please note that the performance of memorability predicting models are measured with rank correlation (Khosla et al., 2015). [Figure] [Table]
2. T HE PROPOSED METHOD FOR MODIFYING FACE MEMORABILITY
Next we aimed to ﬁnd a hyperplane to separate the highly-memorable and low-memorable images.
First, we needed to label the faces into highly-memorable and low-memorable faces.
For this, we used the mean of memorability scores as a threshold.
We labeled an image as highly or low memorable if its memorability score was higher or lower than the mean.
In our experiments, we also tried using median of the memorability scores as the threshold for labeling high and low memorable groups.
Next, using logistic regression, we attempted to ﬁnd a hyperplane to separate low-memorable and highly-memorable images.
For this purpose, we used either z ∈ R 1 × 512 latent vectors or w ∈ R 18 × 512 extended latent space of StyleGAN1 and StyleGAN2.
After ﬁnding the separating hyperplane, we moved the latent vector or the augmented latent vector in the positive or negative direction of the normal vector of that hyperplane to control the image memorability.
This hyperplane denotes the moderately memorable images, as we chose it to separate the faces into highmemorable and low-memorable faces based on the mean (or median) of the memorability scores.
Memorability of each image is related to the distance of its latent vector (or extended latent vector if extended latent vector is used to ﬁnd the hyperplane) from this hyperplane.
Consider image i with corresponding latent vector z i ∈ R 1 × 512 or w i ∈ R 18 × 512 , and its memorability score mem i .
We note the normal vector of this hyperplane by z ∗ ∈ R 1 × 512 or w ∗ ∈ R 18 ×
and we show the distance from the hyperplane by function d .
We will have:
[Equation]
Hence, we can change the memorability of each image, by changing the distance of its latent vector (or extended latent vector) from the separating hyperplane; z edit =
z + α z ∗ or similarly w edit =
w + α w ∗
and we will have:
[Equation]
The results of this classiﬁcation task for ﬁnding the separating hyperplane are presented in Tablewhen different assessors are used for memorability score predictions.
In each case, the accuracy was about 10 percent higher when we used extended latent space ( w ), hence we showed the results for extended latent space in Table 4.
According to the results, we decided to use SENet50 as our assessor and mean memorability as the threshold for our further analysis. [Table]
The performance of logistic regression is higher when the extended latent space is used to ﬁnd the separating hyperplane.
Further, as we show in the next section working with the extended latent space yields better results in modifying face memorability.
(See A.4 for the latent space results.)
One of the beneﬁts of our approach is that we can ﬁx speciﬁc attributes while changing memorability.
It has been shown that latent vector play an important role in determining different attributes of a face and we can ﬁnd hyperplanes to separate faces based on those attributes, such as glasses, age, and smile.
Consider the norm vectors of these hyperplanes as A = { a 1 , a 2 , ..., a k } .
We can ﬁx these attributes by subspace projection as follows: w ∗ new = w ∗ − (cid:80) i
= k i =1 ( w ∗ T .
a i ) a i (See A.5).
2. L ATENT V ECTOR R ECOVERY FOR R EAL FACES
The efﬁciency of our method to modify real human faces depends on how well the latent vector of the real face can be obtained to reconstruct the original image.
After acquiring the latent vector of the real face image, we can repeat the process for the synthesized images and modify their memorability.
In this work, we used image2style (Abdal et al., 2019) to embed the real faces to latent space of StyleGAN1, whose loss function consists of a VGG-16 perceptual loss (Johnson et al., 2016) and a pixel-wise MSE loss term.
Furthermore, for projecting real faces to StyleGAN2 latent space, we employed the same algorithm described by Karras et al.
(2020) after using the dlib library (King, 2009) to align 68 face landmarks in the preprocessing step.
In this work, we assume that we have the latent vector of the generated faces, however, if the latent vector of a generated face is not available, a similar approach can be employed to retrieve the latent vector.
E XPERIMENTS 3. C HANGING MEMORABILITY OF SYNTHESIZED FACES
As described in Section 2.1, we created two different datasets with StyleGAN1 and StyleGAN2.
We randomly sampled 100k z ∈ R 1 × 512 from a standard normal distribution with truncation to produce high-quality synthetic face images and saved their mappings in R 18 × 512 augmented latent space of both GANs.
We calculated their memorability scores using SENet50, then labeled them highly or low-memorable images when compared to the mean memorability score in the dataset, ﬁnally we used the logistic regression to ﬁnd the separating hyperplane as described in Section 2.4.
After ﬁnding the separating hyperplane, we evaluated our method in modifying the memorability of generated faces with StyleGAN1 and StyleGAN2 using our proposed method.
As we discussed earlier in 2.4, we found separating hyperplanes in either latent space or extended latent space of
StyleGAN1 and StyleGAN2.
Figure 4 and Figure 5, demonstrates some samples of memorability modiﬁcation with StyleGAN1 and StyleGAN2 respectively (See A.1 for more examples).We observe that increasing the memorability scores causes some decreases in the facial weight, increases the presence of makeup and thickness of the lips, and makes the person look younger. Moreover, it will affect the skin tone by making it brighter. It makes the face more serious.
However, decreasing the memorability score has opposite effects. In addition to using our method to evaluate face images qualitatively, we also tested the performance of our method in modifying memorability, quantitatively.
For this, we used 10k synthesised faces and tried different weights of memorability modiﬁcation on them.
Figure 6 depicts the distributions of memorability scores when shifted by our method as well as the mean of the distributions, which suggests us we were successful in modifying the memorability scores of the faces. [Figure] [Figure]
3. R EALNESS OF THE MODIFIED FACES
In this experiment [Figure] we evaluated the realness of the faces when our memorability modiﬁcation was applied to them.
StyleGAN is a state-of-the-art model in generating real-looking faces.
Hence, we considered the realness of the generated images from StyleGAN (before memorability modiﬁcation) as our baseline and compared them to the modiﬁed faces. First we generated 10k synthesized faces, then appllied different weights of memorability modiﬁcation vector to their latent vectors.
We utilized two well-known measures for this purpose; FID and KID.
As demonstrated in Figure 7, the
FID and KID scores of the modiﬁed images are close to the unmodiﬁed faces and therefore, the realness of faces is not affected by our method.
[Figure]
3. C HANGING MEMORABILITY OF REAL FACES
[Figure] Next we evaluated our method on real human faces.
First, we computed their extended latent vector using the GAN inversion method previously described and then applied our proposed method to them to change their memorability.
For real human faces, we chose to work with the extended latent vectors, because the regenerated images from the extended latent space are more similar to the original images (See Figure 8).
3. L AYERWISE MEMORABILITY MODIFICATION
[Figure] Next we tried to identify which layers in extended latent vectors contributed the most to face memorability, i.e., which layers are most responsible for modifying a face memorability score.
In each experiment, we only changed one layer and kept other layers the same.
We plotted the faces after these layerwise changes to examine what kind of changes these layerwise modiﬁcations caused in the faces.
We observed that modiﬁcations in the ﬁrst 11 layers were mostly responsible for changes in a face, and the other 7 layers mostly just affected the color and background of the image.
Hence, we only focused on the ﬁrst 11 layers.
We then repeated the same layerwise changes on 3k synthesized faces and calculated the mean memorability of these images before layerwise and after layerwise modiﬁcations (See Figure 9, for the other layers see A.2).
C ONCLUSION
In this work, we proposed a new method for modifying the memorability of face images.
Our approach does not suffer from the limitations of the previous methods and is able to modify the memorability of faces (synthetic and real) within an arbitrary continuous range.
Moreover, we demonstrated these changes will not affect the realness of the images.
However, if a very large weight for the memorability modiﬁcation vector is chosen, it will affect the face identity and realness.
We showed that our method is effective by applying it to 10k synthesized faces.
Further, we employed our method on real human faces and showed it is effective in changing the memorability of real faces as well.
Finally, we studied how layerwise modiﬁcations will affect the face and its memorability score and discussed one of the beneﬁts of the proposed method is modifying the memorability score conditionally by leveraging subspace projection method. A. F ACE M EMORABILITY A SSESSORS
In order to train the models, we split the 10k US Face Database images into train, validation and test split.
We used 80 percent of the data as the training samples and used 10 percent of the data for each of the test and validation splits.
Euclidean distance was used as the loss function and the batch size was set to 64.
Moreover, we leveraged Adam optimizer to train our models.
Due to large false alarm rates in human face images, we trained our models both with raw memorability scores (computed by hit rate) and corrected memorability scores (considering false alarm rate).
We also tried some simple augmentations on the dataset and found, the score of the models will slightly increase if we use a simple augmentation like random horizontal ﬂipping( p = 0 . 5 ).
Consistent with Khosla et al.
Khosla et al.
(2013b)
, we observed when the corrected hit rate scores are used, all models outperform the case when only hit rate scores are used.
That is because false alarm rate introduces noise to memorability scores, therefore, the models perform better when we reduce the noise by correcting for false alarms.
We have brought the rank correlation scores using pretrained VGG16, ResNet50 and SENet50 using hit rate and true hit rate values.
[Table]
A. F INDING HYPERPLANE IN THE LATENT SPACE
In this work, we have provided the results of the separating hyperplane accuracy in extended latent space.
We utilized the latent space to ﬁnd the separating hypeprlane in the latent space of StyleGAN and reported the accuracy of the hyperplane.
[Table]
A. M ODIFYING THE FACES CONDITIONALLY
As we described previously, one of the beneﬁts of our work is that it makes it possible to modify the memorability scores of the faces conditionally.
Different hyperplanes for different face attributes could be used and then with projection, we can try to maintain the corresponding attributes unchanged.
As there is a correlation between different face attributes, choosing large weights and attempting to change the memorability scores drastically, may affect the attribute.
We showed the attempt to maintain smile and age attributes in Figure 27 as an example.
However, this method could be applied to a variety of face attributes.
[Figure]
A. M ODIFYING MEMORABILITY OF GENERATED NON - FACE IMAGES BY S TYLE GAN
We extended our experiments and tested our method on other non-face images.
We leveraged pretrained StyleGAN2 on churches, cats, horses and cars.
In here, we used MemNet as our memorability assessor.
(See Table 5 for accuracy of the hyperplanes in churches, cats, horses and cars augmented latent space.)
[Table]
[Figure]
[Figure]
A. M ODIFYING MEMORABILITY OF GENERATED OBJECT IMAGES BY B IG GAN
Lastly, we tried to show the effectiveness of our method on generated images by BigGAN.
We generated 200k images by 512 × 512 BigGAN-deep
(Brock et al., 2018), predicted their memorability scores by our assessor
[Figure]
and divided them into low-memorable and highly-memorable images.
We observed that the effect of the modiﬁcations.
is similar to Goetschalckx et al.
Increasing the
[Figure]
[Figure]
memorability scores, caused the images to become zoomed-in, in some cases the color changed and also in a few cases (Cheese burger and snake in Figure 35a, made the objects rounder.
[Figure]
[Figure]
[Figure]
[Figure]