U NCERTAINTY - ORIENTED O RDER L EARNING FOR F ACIAL B EAUTY P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Previous Facial Beauty Prediction (FBP) methods generally model FB feature of an image as a point on the latent space, and learn a mapping from the point to a precise score.
Although existing regression methods perform well on a single dataset, they are inclined to be sensitive to test data and have weak generalization ability.
We think they underestimate two inconsistencies existing in the FBP problem: 1. inconsistency of FB standards among multiple datasets, and 2. inconsistency of human cognition on FB of an image.
To address these issues, we propose a new
Uncertainty-oriented Order Learning (UOL), where the order learning addresses the inconsistency of FB standards by learning the FB order relations among face images rather than a mapping, and the uncertainty modeling represents the inconsistency in human cognition.
The key contribution of UOL is a designed distribution comparison module, which enables conventional order learning to learn the order of uncertain data.
Extensive experiments show that UOL outperforms the state-of-the-art methods on both accuracy and generalization ability.
I NTRODUCTION
Sociological and psychological studies (Laurentini & Bottino, 2014) have shown that Facial Beauty (FB) has a great impact on career development, interpersonal relationships, social status and social acceptance.
Thus, Facial Beauty Prediction (FBP), a challenging task in computer vision, has attracted much attention.
In the last decade, several methods had been applied for FBP (Altwaijry &
Belongie, 2013; Lin et al., 2019; Shi et al., 2019), automatic facial beautification (Liang et al., 2014), and makeup recommendation (Alashkar et al., 2017; Liu et al., 2014; Ou et al., 2016; Scherbaum et al., 2011).
The pioneer FBP methods focused on designing handcrafted features based on aesthetic knowledge, such as geometric features (Aarabi et al., 2001; Fan et al., 2012; Farkas & Cheung, 1981; Gunes &
Piccardi, 2006; Mao et al., 2009), holistic features (Gan et al., 2014; Mu, 2013; Suti ´ c et al., 2010;
Wang et al., 2014; White et al., 2004; Yan, 2014) or mixed features (Altwaijry & Belongie, 2013;
Eisenthal et al., 2006; Kagian et al., 2006; Whitehill & Movellan, 2008; Zhang et al., 2016), and performing prediction by classifiers (such as KNN (Aarabi et al., 2001; Suti´c et al., 2010; Eisenthal et al., 2006), SVM (Mao et al., 2009; Whitehill & Movellan, 2008; Bottino & Laurentini, 2012; Xie et al., 2015), decision trees (Mao et al., 2009), Adaboost (Suti´c et al., 2010), etc.) or regression algorithms (such as linear regression (Fan et al., 2012; Kagian et al., 2006; Schmid et al., 2008), ridge regression Mu (2013); White et al.
(2004), Gaussian regression (Xie et al., 2015), etc.).
Later, deep learning was introduced into FBP due to its superiority at various vision tasks.
Many studies tried to learn the mapping from FB features to FB scores (the mean of multiple ratings)
(Lin et al., 2019; Shi et al., 2019; Lin et al., 2022; Xu et al., 2018; Xu & Xiang, 2020).
Although existing regression methods perform well on a single dataset, they often show weak generalization when tested across datasets.
We think they underestimate two inconsistencies existing in the FBP: (1) inconsistency of FB standards among datasets due to the nonalignment of FB reference bases between datasets.
Even FB scores across different datasets are normalized to the same scale, obvious biases still exist, as shown in Fig.
1 (a); (2) inconsistency of human cognition.
Studies (Fan et al., 2017; Wang & Geng, 2021) pointed that the FB ratings made by different people were more likely to diverge.
Figure 1 (b) illustrates an example.
[Figure]
The inconsistency of FB standards among datasets has not been addressed in this field yet, but could be regarded as a nonlinear label shift in domain adaptation problem, in which different datasets are the overlapping subsets of the universal domain.
Existing domain adaptation methods aim to learn domain invariant representations from multiple datasets via minimizing domain shift measures (Sun et al., 2016), optimal distribution matching (Flamary et al., 2017; Li et al., 2020) and domain adversarial training (Ganin et al., 2016).
On the contrary, we expect to learn the invariant in FB from a single dataset, which can be easily applied to other datasets.
Our observation shows the order of FB is highly consistent across different datasets and can be learned from one FBP dataset.
A psychology study (Simon, 1955) has shown that human subconsciously cognize real-world scales by learning an order rather than measuring exact values.
An order pattern is essentially an awareness of relative relation that is independent on the reference base.
The research (Saaty, 1977) also pointed out that relative relations can be measured much easier than estimating precise quantities in many cases.
Lin et al.
(2022) introduced the relative ranking into the loss function of a regression model for FBP problem.
Lim et al.
(2019) proposed an order learning based on relative relations, and applied it to estimate precise facial ages, which demonstrated a better performance.
We then apply the idea of order learning to learn the FB order of instances in the dataset to address the problem of the inconsistency of FB standards.
For the inconsistency of human cognition, some studies (Fan et al., 2017; Wang & Geng, 2021) applied the label distribution learning (LDL) as the objective of regression model.
However, LDL essentially learns the mapping from a feature point on the latent space to a label distribution, thus still suffers from weak generalization.
In psychophysics, Thurstone (1927c) proposed A Law of
Comparative Judgment to address such inconsistency, which is also known as uncertainty problem.
Thurstone argued that the discriminal processes generated by a stimulus were not always equal, therefore, modeled it as a Gaussian distribution on a psychological scale, called discriminal dispersion 1 .
Inspired by this, we introduce an uncertain modeling method, which models the inconsistent cognition of FB as a multi-dimensional Gaussian distribution on a high-dimensional psychological scale space.
However, conventional order learning can only compare data with precise labels rather than uncertain data.
In order to address both inconsistencies, we design a module to compare distributions based on the Monte Carlo sampling, which enables order learning to learn the order relations of uncertain data.
To compare with competing methods, we compare the input face image with a set of reference images from the dataset to transfer the relative relations to FB score, whose labels cover the range of FB scores.
The label of the most similar references will be the score of the input.
We conduct extensive experiments on the FBP benchmark dataset SCUT-FBP5500 and related datasets, Color FERET, Hot-Or-Not, MEBeauty and MIFS.
The results show that our method outperforms six competing methods.
The source code will be available soon.
The main contributions of our work are threefold:
• We propose Uncertainty-oriented Order Learning (UOL), which enables order learning to learn the relative relations of uncertain data by a distribution comparison module.
[Figure]
Constraint
PairwiseConstraint
Uncertainty Constraint
Hinge Loss (Eq. 5)
KL Loss (Eq. 4)
Images in a batch
Distribution in a batch
Uncertainty degree in a batch
Variance of ratings (cid:1709) (cid:1709)
VGG
Uncertainty Modeling (cid:1710) (cid:1710) ( a ) T r a i n i n g P h
a s e
Reference Set
VGG
Uncertainty Modeling
Pair ( (cid:2208) (cid:2778)(cid:2202) , (cid:2208) (cid:2191)(cid:2202) ) (cid:2246) (cid:2206) (cid:2227) (cid:2206)
Concatenate (cid:2206) (cid:2778) > (cid:2206) (cid:2191) (cid:2206) (cid:2778) (cid:3406) (cid:2206) (cid:2191) (cid:2206) (cid:2778) <
(cid:2206) (cid:2191) Maximum Consistency (MC) Estimation Module
Mean of distribution
Covariance of distribution
Feature vectors
Comparator
Monte Carlo Sampling (cid:2208) (cid:2778)(cid:2202) (cid:2208) (cid:2191)(cid:2202)
Uncertainty Modeling
Distribution Comparison
Order Learning (cid:2206) ( b ) E s t i m a t i o n
P h a s e (cid:2246) (cid:2206) (cid:2227) (cid:2206) 2.
CE Loss (Eq. 9)
whose mean µ and diagonal covariance Σ are learned by VGG from the image.
In distribution comparison, we sample from both the distributions of test image and reference image to form a pair and predict its order by a comparator in order learning.
After having the order relations of T pairs between reference images and the test image, the Maxinum Consistency (MC) rule (Lim et al., 2019) is applied to estimate the score of the test image.
• To address the inconsistency of FB standards among datasets, we apply order learning to learn the relative relations between instances.
To address the inconsistency of human cognition, we model FB features as multi-dimensional Gaussian distributions on a psychological scale space, which can learn more robust relative relations of FB features.
• Extensive experiments demonstrate that our UOL has a better performance and generalization over the competing methods on SCUT-FBP5500 and other FBP datasets.
The code will be available soon.
M ETHODS
Our proposed UOL consists of four modules: an order learning model in section 3.1 ; an uncertainty modeling module in section 3.2 ; a distribution comparison module in section 3.3 ; a FB score estimation module in section 3.4 .
Figure 2 shows the overall framework.
2. O RDER L EARNING
Order learning aims to learn the order relations between instances.
As the order between FB is independent on the reference base, it can largely avoid the bias of FB standards introduced by different datasets.
Following is the principle of order learning.
Given two faces images, x i and x j , and their FB scores y i and y j respectively, the order between x i and x j can be defined and encoded by a one-hot label,
[Equation]
where θ is the threshold that represents the discrimination of FB.
[Figure]
The conventional order learning treats an instance as a fixed feature point on the latent space, which is learned by a network g ( · ) , shown in the left of Fig.3.
It then carries out pairwise feature comparisons between two instances.
The comparator f ( · , · ) in order learning consists of three fully connected layers and is formulated as Y ′ =
f ( g ( x 1 ) , g ( x 2 )) , which learns order relation from precise labels of two samples in the pair.
2. U NCERTAINTY M ODELING
Previous methods of FBP usually use the mean of human ratings as the FB score for regression, which underestimate the human cognition biases.
According to Thurstone’s discriminal dispersion theory, the discriminal processes to the same stimulus are not always equal, but rather present a
Gaussian distribution on the psychological scale.
This theory is also applicable to FBP.
We then design a high-dimensional psychological scale space to address the inconsistency of human cognition on FB.
Specifically, we model the human ratings of an instance x as a multi-dimensional Gaussian distribution z ∼ N ( µ ( x ) , Σ( x )) in the space, which is used as a feature for pairwise comparisons, as shown in the right of Fig.
A VGG16 is applied to encode mean vector µ ( x ) and covariance matrix Σ( x ) of the distribution.
Σ( x ) represents the dispersion of the rating distribution.
As Σ( x ) is a diagonal matrix, the degree of discriminal dispersion is the Frobenius norm of it, shown as ∥ Σ( x ) ∥ F = (cid:113)(cid:80) Di = j =1 | σ i,j | .
During training, the variance of the i -th instance’s ratings is set as the ground truth, η i , representing discriminal dispersion degree of the instance, and the network is optimized by minimizing the KL divergence between the predicted distribution of dispersion degrees ( ∥ Σ( x 1 )
∥ F , ∥ Σ( x 2 ) ∥ F , · · · , ∥ Σ( x M ) ∥ F ) and the ground truth distribution ( η 1 , η 2 , · · · , η M ) of the
M training instances.
Such operation can optimize the prediction to be as close as possible to the human ratings,
[Equation]
From another perspective, our uncertainty modeling can be considered as a feature-level data augmentation (Li et al., 2021a; Wang et al., 2019), which can improve the robustness of models.
The detailed discussion can be found in Appendix B.
There have been a few works to model the uncertainty differently for other tasks.
Probout (Gast & Roth, 2018) replaces the intermediate activations with low-dimensional Gaussian distributions by adjusting the activation function, and obtains uncertainty in a lightweight manner instead of traditional Bayesian approaches.
GP-DNNOR (Liu et al., 2019) considers the image quality as a distribution rather than a feature point, then models the uncertainty by a low-dimensional Gaussian distribution.
However, low-dimensional Gaussian distribution naturally limits the feature expressiveness.
By contrast, DUL (Chang et al., 2020) and POEs (Li et al., 2021b) model facial images as multi-dimensional Gaussian distributions, but this uncertainty aims to alleviate the effect of the inherent noise in the image, so only the “mean” of distribution is used for inference.
Also, the information bottleneck loss in POEs emphasizes that the covariance matrix of distribution is close to the
Identity matrix I .
Instead, the uncertainty in our UOL denotes the inconsistency of human cognition, which is different from the Identity matrix.
Thus, both the motivation and modeling approach of UOL have differences from these methods.
2. C OMPARISON OF D ISTRIBUTION
After modeling the FB uncertainty, an order should be established for these multi-dimensional
Gaussian distributions on the phycological scale space.
However, the conventional order learning cannot learn the order between distributions.
Thurstone compares the observations of multiple subjects as the order of two stimuli on the psychological scale.
To mimic this process, we design a uncertainty-oriented comparison module based on the Monte Carlo sampling.
( (cid:2208) (cid:2778)(cid:2778) , (cid:2208) (cid:2779)(cid:2778) ) ( (cid:2208) (cid:2778)(cid:2779) , (cid:2208) (cid:2779)(cid:2779) )
( (cid:2208) (cid:2778)(cid:2176) , (cid:2208) (cid:2779)(cid:2176) )
(cid:1709)
[Figure]
To have a better order relation, we first constrain the Wasserstein distance between distributions on the psychological scale space.
It allows that instances with similar scores have smaller distances between their distributions, while instances with significant different scores have larger distances.
To this end, we apply a hinge loss to constrain the ordinal property of the psychological scale space and form a triplet for any three instances ( x l , x m , x n ) from the dataset, who have ground truth ( y l , y m , y n ) and corresponding feature distributions ( z l , z m , z n ) ,
[Equation]
where S = {
[Equation]
l, m, n ) |
[Equation]
y l − y m
[Equation]
< | y l − y n |} and τ is the margin.
d ( · , · ) denotes the Wasserstein distance between two Gaussian distributions, specifically d ( z 1 , z 2 ) 2 = (cid:80)
Dj =1 ( µ j 1 − µ j 2 ) 2 + ( σ j 1 − σ j 2 ) 2 , where µ j 1 , µ j 2 , σ j 1 and σ j 2 are the j -th dimension of µ 1 , µ 2 , diag (Σ 1 ) and diag (Σ 2 ) respectively.
D is the dimensionality of the vector.
The construction procedure of triplets can be found in Appendix C.
Afterwards, we apply T times Monte Carlo sampling on the distribution of instance x i , which is analogous to the observations of multiple subjects on a stimulus.
To make network be backpropagated, the random sampling and forward propagation must be separated.
Thus, we apply the reparameterization sampling method (Liang et al., 2018a), to get the t -th sampling z ( t ) i from distribution
z i , where z ( t ) i = µ ( x i ) + diag ( (cid:112) Σ( x i )) · ε ( t ) , ε ( t ) ∼ N (0 , I ) .
N (0 , I ) denotes the multidimensional Gaussian distribution with zero mean and identity covariance matrix I .
The sampling process is shown in Fig.
The comparator f ( · , · ) in conventional order learning is applied to learn the order between two sampling feature points.
The relative relation Y ′ between two distributions of x 1 and x 2 is obtained by calculating the mean of T comparisons, Y ′
=
1 T (cid:80)
Tt =1 f
( z ( t ) 1 , z ( t )
[Equation]
A cross-entropy loss L Cls for triple classification is used to optimize the comparator f ( · , · ) ,
[Equation]
where Y ′ c and Y ′
r denote the c -th and r -th dimensions of the output vector Y ′ ,
c is the dimension where the ground truth is.
Thus, the entire loss of our UOL is
[Equation]
where α and β are weights to control the contribution of each loss function.
2. FB S CORE E STIMATION
After establishing the order of samples, network cannot predict the FB score yet because the order is independent on the reference base.
We apply the maximum consistency (MC) rule (Lim et al., 2019) to compare the input face image with a set of reference images.
Figure 5 shows the reference set for
SCUT-FBP5500 dataset.
These comparisons will find the most similar references to the input.
Their precise labels will determine the label of input.
Specifically, given a test image x and a reference r n , 1 ≤ n ≤ N , where N is the number of references, our trained comparator will predict one of three relative relations “ > , ≈ , < ”.
Let s ′ be an estimate of the true score
s ( x ) .
Then, the consistency between the comparator result and the estimate is defined as
[Equation]
where [ · ] is the indicator function.
The function ϕ ( x, r n , s ′ ) returns either 0 for an inconsistent case or 1 for a consistent case.
To maximize the consistency with all references, we estimate the FB score of x by
[Equation]
E XPERIMENT 3. I MPLEMENTATION D ETAILS
We use a pretrained VGG16 on ImageNet as the backbone of our method, and optimize it by Adam optimizer with a batch size of 32.
The learning rate is 1 e − 4 at the beginning and a Cosine Annealing scheduler with the minimal learning rate 1 e − 6 is applied.
We set the training epoch to 100 for all 5-fold cross validation.
For data preprocessing, all the facial images ( 350 × 350 ) are resized to 256 × 256 firstly.
Then a 224 × 224 center croping and a random horizontal flipping are performed, followed by per-pixel rescale to 0 ∼ 1 and mean value subtraction.
The hyperparameters α and β in the loss function are 1 e − 4 and 1 e − 3 , respectively.
We discretize the FB scores of training set at intervals of 0.1, and select min ( n i , 10) images in each score interval from training set as the reference images to estimate the final score, where n i denotes the total number of training data in the i -th score interval.
3. D ATASETS AND E VALUATION M ETRICS 3.2. D ATASETS
SCUT-FBP
(Liang et al., 2018b) has 5500 frontal facial images, which was scored byvolunteers among the range of 1 ∼ 5.
The data consist of male/female and Asian/Caucasian faces, and have diverse annotation information (facial feature annotation, ratings and mean rating for each face by different volunteers).
In this paper, we use the mean score and corresponding variance of different volunteers’ ratings to train and test our model.
Hot-Or-Not (Gray et al., 2010) contains 2056 frontal female face images aged 18-40 without constraint on race, lighting, pose or expression.
The data was scored by 30 volunteers among the range of -3 ∼ 3.
MEBeauty (Lebedeva et al., 2021) includes 1300 females and 1250 males facial images.
Each gender group is divided into six racial categories: Black, Asian, Caucasian, Hispanic, Indian and
Middle Eastern.
The data was scored by 300 volunteers among the range of 1 ∼ 10.
Color FERET (Phillips et al., 2000) is a dataset for face recognition.
It contains 11,338 color images of size 512×768 pixels captured in a semi-controlled environment with 13 different poses fromsubjects.
In this paper, 671 frontal images are selected to validate the generalization capability of our method.
MIFS (Makeup Induced Face Spoofing)
(Chen et al., 2017) is a facial image dataset collected from
YouTube videos of makeup impersonations, consisting of 107 makeup transformations.
Each subject has two images with and without makeup.
In real life, people usually believe faces who wear makeup are more attractive than those who do not.
Therefore, we select facial image of each subject with and without makeup to evaluate the discrimination of UOL and competing methods.
In this work, UOL is only trained on the training set of SCUT-FBP5500, and evaluated on all five datasets without any fine-tuning.
We do this for testing the generalization ability of UOL.
3.2. E VALUATION M ETRICS
To test the effectiveness of UOL, we follow the evaluation metrics in previous metheds (Lin et al., 2019; Xie et al., 2015; Shi et al., 2019; Lin et al., 2022; Xu et al., 2018; Bougourzi et al., 2022): mean absolute error (MAE) and root mean square error (RMSE).
As SCUT-FBP5500, Hot-Or-Not and MEBeauty have varied ranges of FB scores rated by different people, MAE and RMSE are infeasible to evaluate the generalization ability of a method (please refer to Appendix D).
The Pearson correlation coefficient (PC) (Benesty et al., 2009) is commonly used in psychological studies of FB for generalization testing.
Thus, PC is employed as a metric in this work.
For MIFS dataset, we apply the accuracy rate (ACC) as the metric, which measures if the estimation of a face with makeup by a model is higher than that of the face without makeup.
3. C OMPARISON WITH SOTA M ETHODS
To verify the performance and generalization ability of UOL
[Table]
we compare it with the state-of-the-art methods R 3 CNN (Lin et al., 2022), CRNet (Xie et al., 2015), Co-attention (Shi et al., 2019), AaNet (Lin et al., 2019), ComboLoss (Xu et al., 2018) and
CNN-ER (Bougourzi et al., 2022).
[Table]
P ERFORMANCE E VALUATION ON SCUT-FBP
We first test all methods on the large-scale dataset SCUT-FBP5500 and report the results in Table
[Table]
One can see that our UOL achieves the best on both MAE and RMSE, but slightly worse than
Co-attention and CNN-ER on PC, which demonstrates that UOL has reached the state-of-the-art performance on SCUT-FBP5500.
3.3. G ENERALIZATION C APABILITY E VALUATION
Our method aims to improve the generalization ability of the model by mimicking human cognition.
To this end, we train all methods on SCUT-FBP5500 and then test them on unseen datasets (Hot-OrNot, MEBeauty, Color FERET, MIFS).
The competing methods are strictly implemented according to their open codes and papers.
(1) Experiments on Datasets with Human Ratings
We normalize scores of Hot-Or-Not and MEBeauty to the range 1 ∼ 5 and compute PC with the estimated scores by each method.
Table 2 shows that the PC of UOL is considerably higher than those of other methods, which indicates that UOL has better generalization ability.
The evaluations of
MAE and RMSE show that UOL underperforms CNN-ER on Hot-Or-Not by 1% ∼ 2% , but outperform it on MEBeauty by 5% .
One can see that Co-attention performs worse than its performance on
SCUT-FBP5500, which shows it is sensitive to test data when FB standard shifts or image quality varies.
[Table]
Ratings
Color FERET and MIFS do not have human ratings.
So we design two different experiments to evaluate the generalization ability of these methods.
For Color FERET , we select all 671 frontal face images as the test data and apply UOL and six competing methods to give scores for each image for simulating human rating.
After having all seven ratings for each image, the lowest and highest ones are removed, and the mean of the remaining ratings is considered as FB score.
Afterward, we do the same process as above and list the results in
Table 3.
UOL also achieves the highest PC.
[Table]
For MIFS , facial images appear in pairs, in which one is with makeup and another one is not.
We employ 7 volunteers to compare the image pairs, and clean the data according to the consistency of the volunteers’ results.
Please refer to Appendix E for the detailed data cleaning process.
The volunteers’ results are used as the ground truth.
We apply each method to images in a pair.
If the estimated score of the image with makeup is higher than the one without makeup, the comparison is correct, otherwise incorrect.
Table 3 shows UOL is the second best.
After carefully examining the results, we find the two pairs misestimated by UOL are also misestimated by other four methods.
It indicates that some unknown FB features have not been explored by these methods.
CRNet just misestimates a pair, the best in this experiment, but performs worse than its upgrade version,
ComboLoss, in other experiments.
All above results demonstrate the generalization ability of UOL outperforms the competing FBP algorithms.
3. A BLATION S TUDIES 3.4. E FFECTIVENESS OF ORDER LEARNING AND UNCERTAINTY MODELING
To validate the effectiveness of order learning and uncertainty modeling respectively, we conduct ablation studies on three datasets with FB scores.
All versions are trained on the SCUT-FBP5500.
The backbone is VGG16.
The results under different settings are listed in Table 4.
One can see that order learning contributes a significant performance gain because it is more consistent with human cognition to order patterns than regression approaches.
Uncertainty modeling also boosts the performance of a regression model with a marginal gain.
Their integration, UOL, can further
[Table] [Table]
boost the performance.
These results demonstrate that order is a very valuable invariant in FB
[Table]
and uncertainty modeling is more feasible for order learning (a classification model) than a regression model.
Label distribution learning (LDL) could be also considered an uncertainty modeling.
We then apply
LDL to the backbone and report the results in Table 4.
It can be seen that LDL achieves a marginal gain on SCUT-FBP5500 and MEBeauty, but performs worse on Hot-Or-Not.
The possible reason is that SCUT-FBP5500 and MEBeauty have similar data distributions, but Hot-Or-Not has different distribution.
LDL essentially learns the mapping from fixed points on the latent space to certain distributions and is inclined to overfit the label distribution.
Please note that order learning has difficulty in using label distribution because its form is not comparable to learn order.
3.4. E FFECTIVENESS OF THREE LOSSES
UOL employs three loss functions that play different roles.
CE Loss aims at training the comparator in Fig.
2 for estimating the relative order between instances.
CE Loss is indispensable to our UOL.
Hinge Loss constrains the distance between the modeled distributions of instances on the latent space, which can improve the representation of order relation and further boost the order estimation.
KL Loss constrains the consistency between the modeled distribution of instances and the variance of human ratings for instances, which aims at a more accurate distance metric in Hinge Loss.
We also evaluate the effectiveness of them and report the results in Table 5.
We can see that
Hinge
Loss helps CE Loss achieve better performance, KL Loss further boosts the performance when CE
Loss and Hinge Loss work together.
But CE + KL Losses degrade the performance because KL
Loss cannot directly help CE Loss without Hinge Loss.
C ONCLUSION
In this paper, we propose a novel Uncertainty-oriented Order Learning for facial beauty prediction.
UOL enables order learning to learn the relative relations of uncertain data by a distribution comparison module, in which order learning addresses the inconsistency of FB standards between datasets, and uncertainty modeling tackles the inconsistency of human cognition of FB.
Extensive experiments demonstrate that our method outperforms state-of-the-art methods on SCUT-FBP in terms of FB score prediction, and better generalization on other datasets.
However, the improper use of FBP models might result in an unethical impact.
Devising better data forensics approaches could be countermeasures.
In the future work, we will explore the impact of face attributes on the
UOL.
A T HURSTONE ’ S T HEORY
Our UOL is inspired by the discriminal dispersion theory from Thurstone’s Law of Comparative
Judgment .
The following are the details that mainly come from Thurstone’s papers Psychophysical
Analysis (Thurstone, 1927a) and A Law of Comparative Judgment (Thurstone, 1927b).
A. D ISCRIMINAL D ISPERSION
Thurstone argues that a term is needed for the process by which the organism identifies, distinguishes, discriminates, or reacts to stimuli.
In order to avoid any implications, they call the psychological values of psychophysics discriminal processes.
In Fig. 6(a), let the circles
R 1 , R 2 , R 3 , · · · , R n represent a series of stimuli which constitutes a scale with regard to any prescribed stimulus attribute.
Psychologically some of these attributes can be measured, while physically the measurement may even be impossible.
Thurstone assumes that a series of stimuli have been arranged in a scale according to any attribute about which one can say “more” or “less” and that psychophysics need not be limited to stimuli which have magnitude or size, such as lifted weights and room size.
Suppose that each stimulus in the series has a discriminal process which is a psychic or physiological function of the organism.
Thus, the stimulus R n has a discriminal process S n .
It may be assumed that this relation is not so fixed as might be indicated by Fig. 6(a).
It undoubtedly happens that stimulus R n does not always produce the same discriminal process S n , shown in Fig.
6(b).
The present method of psychophysical analysis rests on the assumption that constant and repeated stimuli are not always associated with exactly the same discriminal process but that there is some qualitative fluctuation from one occasion to the next in this process for a given stimulus.
It should be recalled that each of these processes or qualities is identified by that stimulus which most frequently produces it so that S n is habitually associated with R n .
A given process S 5 would be associated frequently with R 5 but occasionally also with adjacent and closely similar stimuli in the stimulus continuum such as R 3 , R 4 , R 5 , R 6 .
Thurstone’s research focuses on the second case, namely the qualitative fluctuations in the discriminal processes that are associated with a constant and repeated stimulus.
The psychophysical relations may be summarized, so far, in the following propositions.
(1) A series of stimuli R 1 , R 2 , R 3 , · · · , R n can be arranged in a scale, with reference to any prescribed quantitative or qualitative stimulus attribute.
(2) These stimuli are differentiated by processes of the organism of unknown nature and they are designated S 1 , S 2 , S 3 , · · · , S n , respectively.
Every stimulus R n is identified by the organism with the process S n .
In this discussion they are referred to as the discriminal processes or qualities.
(3) It is assumed that the correspondence R n −→ S n is subject to noticeable fluctuation so that R n , does not always produce the exact process S n , but sometimes nearly similar processes S n +1 or S n − 1 and sometimes even S n +2 or S n − 2 .
This fluctuation among the discriminal processes for a uniform repeated stimulus will be designated the discriminal dispersion .
The relative frequencies of the different processes are shown in Fig.
6(b) for stimulus
R i in a rough diagrammatic way.
Thus, there is a thick line connecting R i with S i to indicate the relation between the stimulus and its modal discriminal process.
There are only thin lines connecting the adjacent processes with the same stimulus R i and this represents the relatively lower frequency of this association.
Since the assumption of a normal distribution for the discriminal dispersion can be experimentally verified and limited to those stimulus series where its reality can be tested, it will be reasonable to make this assumption subject to verification in every case.
The hypothesis can be stated as follows and shown in Fig.
The discriminal dispersion which any given repeated stimulus produces on the psychological continuum is usually normal.
The frequencies with which the discriminal processes occur for a given stimulus ordinarily describe a normal distribution when plotted on the psychological continuum as a base.
According to this hypothesis, assign scale values to the various processes as distances from S n as an origin.
These distances would be assigned in terms of the standard deviation of the distribution of process-frequencies.
Psychological measurement depends, then, on the adoption of one of
[Figure]
[Figure]
(cid:2175) (cid:2784) (cid:2175) (cid:2785) (cid:2175) (cid:2778)(cid:2777) (cid:2175) (cid:2782) (cid:2175) (cid:2781) (cid:2175) (cid:2780) (cid:2175) (cid:2779) (cid:2175) (cid:2778) (cid:2174) (cid:2784) (cid:2175) (cid:2778)(cid:2778) (cid:2175) (cid:2778)(cid:2779) (cid:2175) (cid:2778)(cid:2780) (cid:2175)
[Figure]
(cid:2175) (cid:2778)(cid:2780)
A. L AW OF C OMPARATIVE J UDGMENT
An ambiguous stimulus which is observed at widely different degrees of excellence or beauty on different occasions will have of course a large discriminal dispersion.
Some other stimulus or specimen which is provocative of relatively slight fluctuations in discriminal processes will have, similarly, a small discriminal dispersion.
The scale difference between the discriminal processes of two specimens which are involved in the same judgment will be called the discriminal difference on that occasion.
If the two stimuli be denoted A and B and if the discriminal processes corresponding to them be denoted a and b on any one occasion, then the discriminal difference will be the scale distance ( a − b ) which varies of course on different occasions.
If, in one of the comparative judgments, A seems to be better than B , then, on that occasion, the discriminal difference ( a − b ) is positive.
If, on another occasion, the stimulus
B seems to be better than A , then, on that occasion, the discriminal difference ( a − b ) is negative.
Finally, the scale distance between the modal discriminal process for any two specimens is the separation which is assigned to the two specimens on the scale that their separation is equal to the separation between their respective modal discriminal processes.
Thurstone states the law of comparative judgment as follows:
[Equation]
in which V 1 and V 2 are the psychological scale values of the two compared stimuli; x (1 > 2) is the sigma value corresponding to the proportion of judgments p (1 > 2) , when p (1 > 2) is greater than 0.
the numerical value of x (1 > 2) is positive, otherwise x (1 > 2) is negative; σ 1 and σ 2 are the discriminal dispersion of stimuli R 1 and R 2 ; γ is the correlation between the discriminal deviations of R 1 and
R 2 in the same judgment.
B A NOTHER P ERSPECTIVE OF U NCERTAINTY M ODELING
It has been generally accepted in machine learning that data augmentation can improve the robustness of models.
We think our uncertainty modeling can be considered as a specific form of data argumentation, because its process is similar to the feature-level data augmentation (Li et al., 2021a;
Wang et al., 2019).
Firstly, we build up a Gaussian distribution in the high-dimensional psychological scale space according to the human ratings.
Then, we randomly sample from these Gaussian distributions for pairwise comparisons.
This process can be considered as disturbing a single feature point on the latent space, which is the feature level augmentation.
As the disturbed features usually belong to the same class of the original feature, such augmentation is often applied to classification tasks (Li et al., 2021a;
Wang et al., 2019).
Our order learning just is a triple classification problem.
As a slight feature disturbance does not change the relative relation between instances, feature-level data augmentation is suitable for order learning.
In addition, our method uses the mean of multiple comparison results, which statistically lower the impact of extreme augmentations.
Moreover, the pairwise comparisons in a batch, which compare any two features in the same batch, can be considered as another form of feature-level augmentation.
Therefore, our uncertainty modeling in order learning can achieve better generalization.
C T RAINING S TRATEGY IN M ETHODS C. T RIPLET C ONSTRUCTION FOR H INGE L OSS
To model the order on psychological scale space described in Sec.3.3 of the paper, we construct a triplet of instances ( x l , x m , x n ) for the hinge loss, which ensures that their FB scores ( y l , y m , y n ) meet | y l − y m | < | y l − y n | .
Theoretically, selecting instances can be random, it makes training time consuming.
Therefore, we define a hard triplet, whose difference between | y l − y m | and | y l − y n | is small, and construct M hard triplets from a batch of training data with M instances.
We take the i - th instance as the anchor l , then set the instance with index ( i + 1) mod M as the second element m .
Finally, we select an instance n from the remaining M − 2 instances as the third element, which satisfies:
[Equation] [Equation]
Algorithm 1 describes the strategy of this hard triplets selection.
C. P AIR C ONSTRUCTION FOR P AIRWISE C OMPARISON
Assume FB scores are in the range of [1 , H ] .
In order learning, we define a pair of two instances, who meet | y i − y j | < θ , as a “ ≈ ” pair, where θ ≪ H .
If instances are selected randomly in each batch, the number of “ ≈ ” pairs will be much smaller than “ > ” pairs and “ < ” pairs.
Such data imbalance makes the trained comparator in Sec.3.3 overfit the “ > ” and “ < ” pairs.
Meanwhile, random selection may let an instance frequently appear in a batch.
It also makes networks overfit this instance.
To tackle these problems, we design a balanced pairs selection strategy, which ensures the proportion of “ ≈ ” pairs in a batch is close to 13 , and all instances appear almost equally in a batch.
Algorithm 2 describes our balanced pairs selection strategy.
D E VALUATION M ETRICS
In the paper, we use Pearson correlation coefficient (PC) as the main evaluation metric due to the following reasons:
(1) PC measures the correlation between prediction and ground truth, which quantifies the degree of interdependence of prediction and ground truth.
PC is a well-accepted metric for the evaluation of FB in psychological research.
(2) MAE measures the mean of absolute errors between the predictions and ground truth.
RMSE measures the deviation between the predictions and ground truth.
Lower MAE and RMSE do not guarantee a better correlation of predictions and ground truth.
(3) Hot-Or-Not, MEBeauty and SCUT-FBP5500 datasets have different score ranges, and were rated by different people.
Thus, the score normalization does not guarantee that the same score represents the same FB level across different datasets
Instead, the order of ratings from different people appears more consistent.
Thus, we employ PC as a metric to measure the generalization ability of a model trained on SCUT-FBP5500.
It also inspires us to improve the generalization ability of the model by learning the order relationship.
E MIFS D ATASET C LEANING P ROCESS
MIFS has no human rating of FB, but each ID contains at least one facial image with makeup and one without makeup in the dataset.
Commonly, face wearing makeup is more attractive than the one wearing no makeup.
Therefore, MIFS dataset is used to evaluate the generalization ability of all models after data cleaning.
MIFS cleaning process is as follows:
Step 1 : Manually select two frontal facial images of each face ID with and without makeup from MIFS, respectively.
Group them as a pair.
Step 2 : Show each pair to 7 volunteers.
They vote the more beautiful image in the pair.
Step 3 :
Calculate the consistency of the volunteers’ votes of each face ID, select pairs with higher consistency (more than 5 volunteers give the same vote) as the test data.
The ground truth is the majority voting.
[Table] [Table]
[Table] [Table]
do
.else
INSERT( prob , 2 3 ∗ unsim );endendr ← RANDOM CHOICE BY PROB ( candidates,prob ) ;// select an instance’s index r from candidates by their probabilities prob , we implement this by using numpy.
INSERT ( flag [ i ] ,r ) ;
INSERT ( flag [ r ] ,i ) ;
INSERT ( p, { i,r } ) ;order ← GENLABEL ( | y
i − y r | ,θ ) ;// GENLABEL denotes generating the order label for a pair by their ground truth and threshold.
INSERT ( Y,order ) ;
DELETE ( candidates,r ) ;end 47 end