A View From Somewhere: Human-Centric Face Representations
Abstract
We propose to implicitly learn a set of continuous face-varying dimensions, without ever asking an annotator to explicitly categorize a person.
We uncover the dimensions by learning on a novel dataset of 638,180 human judgments of face similarity (FAX).
We demonstrate the utility of our learned embedding space for predicting face similarity judgments, collecting continuous face attribute values, and attribute classification.
Moreover, using a novel conditional framework, we show that an annotator’s demographics influences the importance they place on different attributes when judging similarity, underscoring the need for diverse annotator groups to avoid biases.
Introduction
Most human-centric image datasets are web-scraped, lacking ground-truth information about the data subjects.
Moreover, data protection legislation considers demographic attributes to be personal information and limits their collection and use [4, 3].
Even when labels are known, evaluating diversity by examining counts across subgroups fails to reflect the continuous nature of human phenotypic diversity (e.g., skin tone is often reduced to light vs. dark ).
Furthermore, such an approach often denies multi-group membership [60] (e.g., erasing multi-ethnic and intersex individuals).
When labels are unknown, researchers typically choose certain attributes they consider to be relevant for human diversity and use human annotators to infer them [32, 66, 77].
In practice, this is difficult for ill-defined and highly changeable social constructs such as race and gender [22, 34, 9].
Observational labels from annotators risk not only encoding stereotypes , but reifying and propagating them beyond “their cultural context” [35, 18].
Furthermore, discrepancies between, e.g., observed and self-identified race, gender, or other sensitive attributes can induce psychological distress [11] by invalidating an individual’s self-image and identity [59].
Contributions.
As an image subject can exhibit certain traits to a greater or lesser extent than others, even within the same subpopulation, the goal of our work is to learn a similarity function , which measures the similarity between two faces in a way that is aligned with human perception .
We aim to do so without using categorical labels.
Concretely, we present a model of conditional decision-making for learning continuous, humaninterpretable face embeddings directly from human judgments of face similarity.
Underlying our approach is FAX, a novel dataset of 638,180 judgments over 4,921 faces, where each judgment corresponds to the odd-one-out (i.e., least similar) face in a triplet of faces.
Such contextual similarity judgments have been shown to offer a window into the dimensions (i.e., factors of variation) of object categories [27, 84] and reachspace environments [31] in the human mind.
Distinct from other computer vision datasets, each judgment in FAX is accompanied by the identifier and demographic attributes of the annotator who made the judgment.
[Figure]
Fig.
1 illustrates our method, which concurrently learns an embedding space and the importance distinct annotators place on each of the embedding dimensions when determining similarity. By learning importance scores, we induce subspaces that encode each annotator’s notion of similarity.
We show that predicting similarity judgments from an annotator in their respective subspace increases predictive accuracy. Significantly, we find that annotator subspaces are not interchangeable, but can be grouped wrt the sociocultural backgrounds (e.g., nationality, ancestry) of the annotators.
Our results underscore the need for diverse annotator groups to avoid biases. Additionally, we show that our embeddings are highly correlated with the human mental representational space of faces, as compared to face embeddings induced by learning to predict face identity or face attributes.
Importantly, we find that the individual embedding dimensions are: related to concepts of gender, race, age, as well as face and hair morphology.
We demonstrate the utility of such dimensions for: (i) collecting continuous (as opposed to discrete) face attribute values for novel faces from annotators; and, (ii) binary attribute classification.
1.1 Background
Disentangled Representation Learning.
Learning interpretable representations through semantically labeled concepts and prototypes has recently gained attention [2, 38, 43].
Others reveal meaningful directions inside generative models, where directional distances estimate the magnitude of attributes [53].
However, the identification of interpretable directions is performed post-hoc, requiring potentially biased labeled examples and/or pretrained models [81, 47], self-supervision [57, 30], or paired images sharing at least one factor of variation [48].
Our approach is similar in spirit to wholly unsupervised approaches [23, 55, 68, 79].
Rather than modeling the data generating distribution, our representations are trained to predict human similarity judgments, reflecting the human mental representational space of faces.
Our baselines primarily center on embeddings induced by training on facial analysis datasets, since in the face domain, post-hoc interpretation of latent dimensions [69, 53, 81, 47] predominantly rely on datasets such as CelebA [46]. Psychological Embeddings.
Multidimensional scaling (MDS) is often used to learn psychological embeddings from pairwise similarities [72, 84, 27, 58, 73, 31].
Zheng et al.
[84] developed an MDS approach for learning psychological object embeddings from odd-one-out judgments based on three assumptions.
First, embeddings can be learned solely from odd-one-judgments, where representations are constrained to be continuous, non-negative, and sparse.
Such properties support interpretability such that dimensions indicate both feature presence and feature magnitude.
Second, odd-one-out judgments are a function of sim( i, j ) , sim
( i, k ) , and sim( j, k ) , where x ∈ X and sim : ( x i , x j ) → sim( i, j ) ∈ R denote a face image and similarity function, resp.
Third, odd-one-out judgments are stochastic, where the probability of selecting x k as the odd-one-out is p ( k )
∝ exp(sim( i, j )) .
As
MDS cannot embed data outside of the training set, researchers have used pretrained models to predict MDS representations [64] or directly infer similarity judgments [56, 6].
Rather than relying on pretrained models which can introduce implicit biases [39, 71], we perform end-to-end learning from scratch.
Further, unlike recent work [27, 31], we focus on human perception of naturalistic face images, evidenced to be processed differently to objects in the human visual system [70].
Annotator Bias.
Categorization by observation depends not only on the subject being categorized, but also on the annotator’s sociocultural background, perception of the subject, and contextual cues [67, 7, 29, 59].
For instance, a subject’s clothing can influence their racial categorization [19].
Despite this, annotator positionality has only recently entered into discourse in computer vision [16, 83, 65], albeit predominantly in the context of natural language processing [1, 78, 80, 10, 62].
However, in order to mitigate bias, one must first measure bias [41].
Unlike previous works, our model of conditional decision-making elucidates the importance distinct annotators place on different attributes when judging similarity, thus permitting the mitigation of bias. Veit et al. [74] employ a similar procedure to learn subspaces which encode different notions of similarity (e.g., font style, character type).  Method
To learn a similarity function aligned with human perception, we collect a large-scale dataset, FAX, of odd-one-out similarity judgments.
An odd-one-out judgment corresponds to the least similar face in a triplet of face images, representing a three alternative forced choice task .
Data.
The benefits of odd-one-out judgments are threefold.
Most significantly, the judgments do not require an annotator to categorize people.
Thus, we do not encode, reify, or propagate stereotypes.
Second, for a triplet of faces ( x i , x j , x k ) , repeatedly varying x k permits the identification of the relevant dimensions that contribute to sim( i, j ) .
That is, w.l.o.g., x k provides context for which sim( i, j ) is determined, making the task easier than explicit pairwise similarity tasks (i.e., “Is x i similar to x j ?”).
This is because it is not always apparent to an annotator which dimensions are relevant when determining sim( i, j ) , especially when x i and x j are perceptually different.
Finally, there is no need to prespecify attribute lists hypothesized as relevant for comparison (e.g., “Is x i older than x j ?”).
The odd-one-task instead implicitly uncovers salient attributes.
For our proof of concept, the 4,921 faces were sampled from the CC-BY licensed dataset FFHQ [33].
The subset was obtained by first splitting FFHQ into 56 partitions based on estimated intersectional group labels, and then randomly sampling from each partition with equal probability.
All faces are near-frontal with little to no eye occlusions and an apparent age > 19 years old.
FAX contains 638,180 quality-controlled triplets over the 4,921 faces, representing 0.003% of all possible triplets.
Each triplet is labeled with an odd-one-out judgment and the identifier of the annotator who made the judgment.
As in previous non-facial odd-one-out datasets [31, 28], there is a single judgment per triplet.
In total, 1,645 annotators contributed to FAX over Amazon Mechanical
Turk (AMT), providing their consent, age, nationality, ancestry, and gender identity.
Compensation was 15 USD per hour.
Additional details regarding FAX are provided in Appendix A
Model of Conditional Decision-Making.
At a high-level, we want to apply Zheng et al.’s
MDS approach (Section 1.1) to learn face embeddings.
However, MDS cannot embed images outside of the training set, limiting its utility.
Moreover, MDS pools all judgments, disregarding intra- and inter-annotator stochasticity.
To resolve these issues, we propose to learn a conditional convolutional neural network (CNN).
Let { ( { x iℓ , x jℓ , x kℓ } , kℓ, a ) } nℓ =1 denote a training set of n (triplet, judgment, annotator) tuples, where a ∈ A .
To simplify notation, we assume that judgments always correspond to the index kℓ .
Suppose f : x (cid:55)→ f ( x ) = w ∈ R
d is a CNN, parametrized by Θ , where w ∈ R d is an embedding of x .
We model the probability of annotator a selecting kℓ as p ( kℓ | a ) ∝ exp(sim a ( iℓ, jℓ )) , where sim a ( · , · ) denotes the internal similarity function of a .
Given two images x i and x j , we define their similarity according to a as: sim a ( i, j ) =
( σ ( ϕ a ) ⊙ ReLU( w i ))
⊤ · ( σ ( ϕ a ) ⊙ ReLU( w j )) , where σ ( · ) and ReLU( · ) denote the sigmoid and ReLU functions, resp.
Note that σ ( ϕ a ) ∈ [0 , 1] d is a mask associated with a .
Each mask plays the role of an element-wise gating function, encoding the importance a places on each of the possible
d embedding dimensions when determining similarity.
By conditioning prediction on annotator identifiers, we induce subspaces that encode each annotator’s notion of similarity.
This allows us to study whether per-dimensional importance scores differ between annotators.
Let Φ =
[ ϕ ⊤ 1 , . . .
, ϕ ⊤|A| ]
∈ R
d
×|A| denote a trainable weight matrix, where each column vector ϕ ⊤ a corresponds to annotator a ’s mask prior to applying σ ( · ) .
In our conditional framework, we jointly optimize Θ and Φ by minimizing:
[Equation]
where ˆ p ( kℓ | a ) =
[Equation]
) is the predicted probability that kℓ is the odd-one-out conditioned on a .
The first term in Eq.
1 encourages similar face pairs to result in large dot products.
The second term (modulated by λ 1 ∈ R ) promotes sparsity.
The third term (modulated by λ 2 ∈ R ) supports interpretability by penalizing negative values.
The last term (modulated by λ 3 ∈ R ) penalizes large weights.
When λ 3 = 0 and σ ( ϕ a ) =
, 1] ( ∀ a ), Eq.
1 corresponds to the unconditional MDS objective proposed by Zheng et al.
In what follows, we refer to unconditional and conditional models trained on FAX as FAX-U and FAX-C , resp.
In addition, we refer to conditional models whose masks are learned post hoc as FAX-C-PH .
A FAX-C-PH model uses a fixed unconditional model (i.e., FAX-U ) to obtain face embeddings such that only Φ is trainable.
Significantly, our conditional decision-making framework is generally applicable to any task that involves mapping from inputs to human decisions; it only requires record-keeping during data collection such that each judgment (or annotation) is associated with the annotator who generated it.
Implementation.
All models have ResNet18 [25] architectures and output 128-dimensional embeddings.
We use the Adam [37] optimizer with default parameters, reserving 10% of FAX for validation.
Based on grid search, we empirically set λ 1 = 5 × 10 − 5 and λ 2 = 1 × 10 − 2 .
For
FAX-C and FAX-C-PH , we additionally set λ 3 = 1 × 10 − 5 .
Across independent runs, we find that: (i) only a fraction of the 128 dimensions are needed; and (ii) individual dimensions are reproducible.
Post-optimization, we remove dimensions with maximal values close to zero.
For FAX-U , this results in 22 dimensions (61.9% validation accuracy).
Further details are in Appendix B.
Experiments
We demonstrate the utility of FAX induced embedding spaces for predicting face similarity judgments, revealing annotator bias, collecting continuous face attribute values, and attribute classification.
Throughout this paper, statistical tests correspond to Mann-Whitney U -tests ( p -value < 0.05).
For comparison, we consider embeddings extracted from pretrained face recognition and facial analysis models.
In particular, an ArcFace [17] face recognition model trained on CASIA-Webface (CWF) [82]; and analysis models trained on CelebA (CA) [46] (40 binary face attribute labels), FairFace (FF) [32] (perceived binary gender expression, age, and race labels), and FFHQ (binary gender expression, age, and race labels).
All models have ResNet18 architectures and output 128-dimensional embeddings.
For the analysis models, we also consider unnormalized class logit representations, as well as the logit representations converted to one-hot encodings.
All representations are normalized to unit-length, and the dot product of two representations determines their similarity.
Additional details and experimental results are in Appendix C.
Predicting Human Similarity Judgments.
The utility of any face embedding method is typically determined by measuring whether similar faces are closer together in feature space than dissimilar faces.
We therefore employ two evaluation protocols, evaluating odd-one-out predictive accuracy.
In Protocol 1 , we test whether our embeddings are predictive of human judgments not observed during learning.
This is a typical MDS test setting, as MDS cannot embed images outside of the training set.
Using images from the stimulus set of 4,921 faces, we generate 1,000 novel triplets and collect 22– unique judgments on AMT per triplet (24,060 judgments).
In addition to predictive accuracy, we compute Spearman’s r correlation between the entropy in human- and model-generated triplet oddone-out probabilities.
Human-generated odd-one-out probabilities are of the form: ( n i , n j , n k )
/n , where, w.l.o.g., n k /n corresponds to the fraction of n odd-one-out votes for k .
As we have 22–25 judgments per triplet in Protocol 1 , we can reliably estimate odd-one-out probabilities.
The Bayes optimal classifier accuracy corresponds to the best possible accuracy any model could achieve given the stochasticity in the human judgments.
The classifier makes the most probable prediction, i.e., the majority judgment.
Its accuracy is equal to the mean majority judgment probability over the 1,000 triplets, corresponding to 65.5 ± 1% (95% CI).
In Protocol 2 , we sample 56 novel face images from FFHQ not contained in the stimulus set.
We then generate all (cid:0) 56 3 (cid:1) possible triplets and collect 2–3 unique judgments on AMT per triplet (80, judgments).
As an additional performance measure, we compute Spearman’s r between the strictly upper triangular model- and human-generated similarity matrices.
Entry ( i, j ) in the human-generated similarity matrix corresponds to the fraction of triplets containing ( i, j ) , where neither was judged as the odd-one-out.
Entry ( i, j ) in a model-generated similarity matrix corresponds to the mean ˆ p ( i, j ) over all triplets containing ( i, j ) . [Table]
Results for Protocol 1 and Protocol 2 are shown in Tbl.
First, embeddings trained on FAX outperform the baselines across all metrics.
In particular, as evidenced by the correlation tests, the human mental representational space is better represented by embedding spaces induced by learning on FAX.
This even holds for our FAX-U model trained on only 72K judgments.
Second, FAX-C and FAX-C-PH have increased performance over their unconditional counterparts.
This shows that the learned annotatorspecific masks generalize, i.e., the encoded importance an annotator places on each dimension extends to novel judgments.
Most significantly, in Protocol 1 , our conditional models attain a predictive accuracy at or above the upper bound of the Bayes optimal classifier.
Annotator Bias.
Conditioning prediction on annotator identifiers provides the best predictive accuracy, evidencing that knowledge of the annotator determining similarity assists in informing the outcome.
However, annotators are often framed as interchangeable [51, 14].
To test the validity of this assumption, in Protocol 2 , we randomly swap the annotator associated with each judgment and then recompute the predictive accuracy of FAX-C .
Repeating this process 100 times results in a performance drop from 61.7% to 52.8% ± 0.02% (95% CI) on average.
This shows that annotator subspaces are not interchangeable.
[Table]
An interesting question relates to whether an annotator’s sociocultural background influences their decision making.
To evaluate this, we create sets { ( σ ( v a ) , y ) } , where σ ( v a ) and y ∈ Y are annotator a ’s learned mask and self-identified demographic attribute, resp.
For a particular annotator attribute (e.g., nationality), we limit the dataset to annotators who contributed ≥judgments and belong to one of the two largest groups wrt the attribute.
Using 10-fold cross validation, we train linear support vector machines (SVMs) [26] with balanced class weights to predict y from σ ( v a ) .
Tbl.
2 shows the average area under the receiver operating characteristic (AUC) for each attribute.
Most significantly, none of the AUC confidence intervals include chance performance.
Moreover, the linear SVMs are able to discriminate between binary groups wrt nationality, regional ancestry, and subregional ancestry with high probability (86–88%).
Interpretability.
Since we aim to replace the explicit collection of problematic categorical labels, we evaluate whether the individual FAX-U (and FAX-C-PH ) dimensions are human-interpretable through a (qualitative) dimension labeling task and a (quantitative) dimension rating task [84, 27, 31].
For the dimension labeling task , we task annotators with writing descriptions about each of the 22 dimensions using continuous dimensional scales .
A dimensional scale corresponds to the the stimulus set sorted in descending order based on their values in the dimension.
Consistent descriptions evidence that a dimension is meaningful.
We collect 25–62 descriptions per dimension.
Based on the descriptions, we observe distinct dimensions coinciding with commonly defined demographic attribute concepts, i.e., Male , Female , Black , White , East Asian , South Asian , and Elderly .
In addition, there are separate dimensions for face and hair morphology, i.e., Wide Face , Long Face , Smiling Expression , Neutral Expression , Balding , Facial Hair , and Dyed Hair .
(See Fig. 2.)
For the dimension rating task , we task annotators with placing 20 novel faces on each of thedimensional scales.
We collect 20 unique judgments per dimension for each face. Separately for each face, we average the judgments per dimension such that we obtain human-generated embeddings.
Using the human-generated embeddings, we create a human-generated similarity matrix and compare
Dim.
[Figure] [Figure]
Smiling Dim
[Figure] Wide [Figure] Dim.
Elderly
Dim.
Balding Dimension Labeling Task. We show a subset of the 22 FAX-U (and FAX-C-PH ) dimensions. Each dimension is shown with the top 8 faces from the it to a model-generated similarity matrix.
The model-generated similarity matrix is created using model-generated embeddings.
Spearman’s r correlation between the strictly upper triangular modeland human-generated similarity matrices is 0.83 and 0.86 for FAX-U and FAX-C-PH embeddings, resp.
This shows that: (i) dimensional values correspond to the feature magnitude; and (ii) dimensional scales can be used to directly collect continuous attribute values for faces, sidestepping the limits of categorical definitions [34, 9, 35].
Note that the use of a dimensional scale is not restricted to any particular embedding method, as long as the dimension is human-interpretable [Figure] [Figure]
[Table]
Binary Attribute Classification.
Facial analysis models are trained for classification, therefore we test the utility of FAX-U dimensions for discriminating between binary-valued attributes.
For labeled face data, we use COCO [44],
OpenImages MIAP (MIAP) [66], CFD, FFHQ,
CelebA (CA) [46], and Casual Conversations (CC) [24].
Tbl.
3 shows that FAX-U dimensions are competitive with the baselines, even in challenging unconstrained settings as represented by COCO and MIAP.
Therefore, training on FAX results in interpretable dimensions that can serve as classifiers.
Notably, FAX-U dimensions have the highest AUC for self-identified CFD attributes Black , East Asian , and Indian .
Conclusion
Motivated by issues inherent to categorization by observation, we proposed a method for implicitly learning continuous face-varying dimensions, without ever asking an annotator to explicitly categorize a person. We uncovered the face embedding space by learning on a novel dataset of human judgments of face similarity (FAX).
We showed that the individual dimensions are humaninterpretable and related to concepts of gender, race, age, as well as face and hair morphology categories. We demonstrated the utility of our learned embedding space for predicting face similarity judgments, collecting continuous face attribute values, and attribute classification. Moreover, using our novel conditional framework, we showed that an annotator’s demographics influences the importance they place on different attributes when judging similarity, underscoring the need for diverse annotator groups to avoid biases. We hope that our work inspires others to pursue unorthodox tasks for learning face-varying dimensions, which do not encode, reify, and propagate stereotypes, or invalidate the self-image and identity of image subjects.
Future Work. As our dataset represents 0.003% of all possible triplets that can be sampled from 4,921 images. To go one step further, triplets should be sampled selectively so as to learn additional face attribute dimensions. That is, FAX can be extended using active learning approaches, e.g., focusing on triplets composed of faces that are close in embedding space whose similarity judgments are unlikely to be based on gender, skin color, or race, but rather finer-grained attributes such as hair color, eye color, nose shape, etc.
A Data A.1 Face image stimuli
[Table]
The stimulus set was sampled from the publicly available FFHQ dataset [33] of 70,000 permissively licensed face images.
The FFHQ dataset which includes JSON metadata and documentation is available from the following repository: https:/ / ffhq-dataset .
For face detection, as in [33], we used the dlib mmod human face detector [36] ( https://github.com/davisking/dlib-models ).
To align the detected faces, we followed the same procedure as detailed in [54], which uses the dlib 68 landmark shape predictor [36] ( https: //github.com/davisking/dlib-models ).
As stated in the main text, we used FFHQ-Aging annotations [54], obtained using Face++, for absolute head yaw angle, absolute head pitch angle, eyewear, and eye occlusion annotations to prune the dataset.
The purpose of these constraints was to reduce the influence of head pose on human behavior and to better allow for eye comparisons.
The stimulus set of 4,921 representative examples were then sampled from the remaining images such that each intersectional subgroup based on perceived binary gender expression, age group, and ethnicity would be included.
We only considered face images with an apparent age greater than 19 years old in an attempt to exclude images of minors.
Ethnicity was estimated using a
FairFace [32] trained model (implementation details are in Appendix B.2).
Perceived binary gender expressions and apparent age groups were obtained from FFHQ-Aging crowdsourced annotations.
To mitigate biasing our set toward stereotypical faces, we randomly sampled images from the 56 possible intersectional subgroups rather than selecting the most confidently predicted or annotated.
Table 4 shows the group counts at the level of age, gender, and ethnicity, whereas Table 6 reports the counts at the finer intersectional subgroup level.
Exact balancing was not possible due to the skewed composition of FFHQ.
Ideally, we would have sampled images from a dataset with self-reported attributes.
However, while such datasets do exist, they are typically small, tightly constrained, and largely unrepresentative.
A.2 Collection of human behavioral judgments
All participants were recruited on AMT and were required to have previously completed a minimum of 100 human intelligence tasks (HITs) on AMT with a 95% approval rating.
Eligibility was further determined through a prescreening survey (nominal fee of 0.01 USD), which included a short multiplechoice English language proficiency test.
Since we placed no restriction on participant location, in order to be eligible, we required participants to answer at least two out of three multiple-choice
English language proficiency questions correctly.
Collection of human
behavioral odd-one-out similarity judgments for training and validation.
Based on estimated face image labels, a triplet can comprise 1–2 binary gender expression group(s), 1– 3 age group(s), and 1–3 ethnicity group(s) resulting in 18 triplet types which were uniformly sampled.
We sampled 703,300 unique triplets in total.
Participants were presented with triplets alongside the following minimal instructions: “Choose the person that looks least similar to the two other people.
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
Focus your judgment on the people depicted, ignoring differences in head position/direction, facial expression/emotion, lighting, accessories, and background/objects”.
Each HIT consisted of 20 unique triplets, where each triplet was presented separately and in succession.
We collected 703,300 unique triplet similarity judgments (i.e., one judgment per triplet) on AMT from 1,887 eligible participants.
Each triplet was judged once by a single participant, resulting in a single judgment per triplet.
To control for quality, according to the criteria outlined in Table 5, we excluded triplet judgments obtained from any participant who provided overly fast, deterministic, or incomplete judgments.
This left 638,180 judgments from 1,645 participants.
Irrespective of estimated quality, all participants were compensated at a rate of 15 USD per hour.
Table 7 shows the age, ancestry, gender identity, and nationality group counts of the 1,645 participants.
of human behavioral odd -one-out similarity judgments for testing: Same images, novel triplets.
We sampled an additional 1,000 triplets (images were sampled from the stimulus set
[Table]
of 4,921 images)
We collected 25 unique judgments per triplet on AMT (total: 25,000 judgments).
After excluding responses based on the quality control criteria outlined in Table 5, we were left with 22–25 unique judgments per triplet (total: 24,060 judgments) obtained from 355 participants.
Tableshows the age, ancestry, gender identity, and nationality group counts of the 355 participants.
Collection of human behavioral odd-one-out similarity judgments for testing: Novel images, novel triplets.
We sampled all possible triplets from 56 novel images (sampled from FFHQ, but disjoint from the stimulus set) and collected 3 unique judgments per triplet on AMT (total: 83, judgments).
After excluding responses based on the quality control criteria outlined in Table 5, we were left with 2–3 unique judgments per triplet (total: 80,300 judgments) obtained fromparticipants.
Table 9 shows the age, ancestry, gender identity, and nationality group counts of theparticipants.
Collection of dimension labels.
For each of the 22 dimensions, we constructed a scale (measurement units corresponded to dimensional values which were converted to percentiles) and tasked participants with providing up to three labels (minimum one label) using words only.
Note that participants were not restricted to entering single word labels, they were provided with free-text boxes.
Participants were presented with a scale and were asked: “Which visual characteristics do you think the people at the high end of the scale have in common compared to the people at the low end of the scale?”.
In addition, participants were asked to view and interact with the entire scale before providing any labels.
Figure 5 shows an example of a scale and a novel face image shown to participants.
After quality control, there were 25–62 labels per dimension.
Quality control roughly entailed removing responses that did not correspond to words and those that were derogatory in nature.
The filtered labels were then converted into 35 broad topic categories: “Age”, “Age related”, “Ancestry”, “Ear shape”, “Eye color”, “Eye related”, “Eye shape”, “Eyebrow related”, “Eyebrow shape”, “Face related”, “Face shape”, “Facial expression”, “Facial hair”, “Forehead shape”, “Gender
[Table]
“Hair “Hair
Figure 4 provides evidence of the interpretability of the FAX-U (and FAX-C-PH ) dimensions. There is clear relationship between the dimension topics obtained from annotator descriptions and dimension labels generated using CelebA and FairFace models
[Table]
Table 10 shows the age, ancestry, gender identity, and nationality group counts of the 102 participants.
Collection of Dimension Ratings.
We tasked participants with placing 20 novel face images along each of the 22 dimensions using the unlabeled scales from the dimension labeling task.
We collected 8,800 judgments in total (20 per image-dimension tuple).
Participants were presented with a scale and a novel face image above the scale, and were asked: “Decide where on the scale you would place the person shown based on their similarity to the people shown beneath the scale”.
In addition, participants were asked to view and interact with the entire scale before making their decision.
Figure 6 shows an example of a scale and a novel face image shown to participants.
Table 11 shows the age, ancestry, gender identity, and nationality group counts of the 164 participants.
B Implementation Details B.1 FAX Models
All images were resized to 128 × 128 and normalized to [ − 1 , 1] .
Standard data augmentation was used (horizontal mirroring and 112 × 112 random crops).
For the unconditional FAX models, guided by the validation loss, we empirically set λ 1 = 5 × 10 − 5 and λ 2 = 1 × 10 − 2 .
(The conditional models used the same hyperparameters with the additional loss term’s weight set to λ 3 = 1 × 10 − 4 .)
All models were optimized for 40 epochs with Adam [37] (default parameters), learning rate 1 × 10 − 3 , and batch size 128 on a single Tesla T4 GPU.
Post-optimization, we obtained our core set of dimensions by dispensing with those whose maximal value was close to zero, resulting in a low-dimensional space.
(For the FAX trained model, this resulted in 22 dimensions.)
The threshold for all models was determined based on maximizing accuracy on the validation set.
Across five independent runs, the
[Table]
and that the 22 dimensions are approximately replicated across the independent runs.
Note that we discovered some redundancy in the dimensions, in particular 8 dimensions have a
Pearson r > 0 .
9 with another dimension.
The number of (near) nonzero dimensions depends on the L 1 sparsity penalty, which must be carefully chosen: Too high of a penalty will result in the entanglement of distinct factors of variation, whereas too low of a penalty will result in repeated dimensions.
We erred on the side of caution (lower penalty), since it is more difficult to localize the source of bias when factors are merged, i.e., since our main aim is auditing datasets and models.
B.2 Baseline models
All images were resized to 128 × 128 and normalized to [ − 1 , 1] .
Standard data augmentation was used (horizontal mirroring and 112 × 112 random crops).
For preprocessing, we used the same [Table] [Table] alignment procedure as in [54] to produce 128 × 128 face crops.
Image values were normalized to [ − 1 , 1] .
CelebA model.
Training was performed using a batch size of 512 (across 4 Tesla T4 GPUs) with
SGD for 45 epochs.
The initial learning rate 0 . 1 was divided by 10 at epoch 15, 30, and 40.
L 2 weight decay and SGD momentum were set to 0 .
[Table]
[Table]
5K 287K [Table] [Table]
5K [Table] [Table] 72K
[Table] Tripletmarginwithdistanceswap [Table] [Table]
FairFace model.
Training was performed using a batch size of 512 (across 4 Tesla T4 GPUs) with
SGD for 45 epochs.
The initial learning rate 0 . 1 was divided by 10 at epoch 15, 30, and 40.
L 2 weight decay and SGD momentum were set to 0 . 0005 and 0 .
9 , respectively.
Standard data augmentation was used (horizontal mirroring and 112 × 112 random crops).
CASIA-WebFace models.
We experimented with several face recognition models that differed only in terms of the loss function minimized: Softmax, ArcFace [17], CosFace [76], and SphereFace [45].
Training was performed using a batch size of 512 (across 4 Tesla T4 GPUs) with SGD for 55 epochs.
The initial learning rate 0 . 1 was divided by 10 at epoch 15, 30, and 40.
L 2 weight decay and SGD momentum were set to 0 . 0005 and 0 .
9 , respectively.
Standard data augmentation was used (horizontal mirroring and 112 × 112 random crops).
C Additional experiments
Predicting human similarity judgments.
When appropriate, as baselines, we compare to supervised and self-supervised representation learning methods.
Supervised models minimize a crossentropy, ArcFace [17], CosFace [76], SphereFace [45], or triplet margin with distance swap [8] loss.
Self-supervised approaches correspond to MoCo-v2 [15], SwAV [12], or DINO [13].
For data, supervised methods train on CASIA-Webface [82], CelebA [46], FairFace [32], FFHQ [33, 54], or
ImageNet-1K [61]. Self-supervised approaches learn on IG-1B [21], ImageNet-1K [61], or PASS [5].
As is standard, baseline representations are extracted from the final encoder layer of a model and then normalized to unit-length.
The dot product of two representations determines their similarity.
Results for Protocol 1 and Protocol 2 are shown in Tbl.
C.1 Prototypicality
To test whether dimensional values represent the typicality of faces, we use face images from Chicago
Face Database (CFD) [49, 40, 50] labeled with prototypicality ratings.
The ratings (obtained from human annotators) correspond to the average prototypicality of a face wrt a race category from one (less typical) to five (very typical), considering skin color, hair, eyes, nose, cheeks, lips, and other physical features.
For gender expression, ratings correspond to the typicality of the face relative to others of the same race and gender in the United States from one (not at all typical) to seven (extremely typical).
Tbl.
13 shows that relevant FAX-U dimensions are positively correlated with the typicality ratings according to Spearman’s r .
Although the odd-one-task does not require an annotator to explicitly categorize any person, category typicality appears to manifest from the similarity judgments for concepts related to race and gender. [Figure]
Comparative Dataset Diversity Auditing.
Drawing inspiration from biodiversity measures [42], we propose to use our learned dimensions for comparative dataset diversity auditing.
Given a set of candidate datasets { c k } nk =1 , we aim to find c
k ⋆ = arg
min c k sim( c k ) , where sim( c k ) measures the similarity of the faces in x ∼ c k .
We denote by q k ∈ [0 , 0 .
, 0 . 99 , 1] and {+
+} − q k the proportion of images in c k from D
= { ( x i , y 0 ) }
i and D 1 = { ( x j , y 1 ) } j , resp., where D 0 ∩ D 1 = ∅ .
Here y 0 , y 1 ∈ { 0 , 1 } are sensitive attribute labels.
Let ˆ y ∈ R denote a proxy continuous attribute value.
We define the similarity between ( x i , x j ) ∼ c
k as z ij = abs(ˆ y
i − ˆ y j ) − 1 .
Further, let
Z k = ( z ij ) ∈ R | c k |×| c k | denote c k ’s similarity matrix.
In biodiversity terms, the average ordinariness of faces in c k is ∝ (cid:80) i (cid:80)
j z ij ( ∀ i ).
This quantity is large when most faces in c k are concentrated into a few very similar faces.
Importantly, concentration is inversely connected to diversity.
Therefore, we can interpret the mean of Z k as a diversity score, which we denote by divscore( Z k ) .
Note that divscore( Z k ) → 0 for homogeneous sets of faces.
Auditing Model Behavior.
Beyond dataset auditing, as an example, we study the disparate impact of the face image restoration model PULSE [52].
We select PULSE due to its much discussed racial bias [63, 52, 75].
Let R : x L (cid:55)→ R ( x L ) = x
′ H denote the PULSE model, which maps a low-resolution face image to a high-resolution face image.
We denote by x H the ground-truth high-resolution face image.
Suppose z H and z ′ H denote face attribute representations of x H and x ′ H , resp.
Disparate outcomes in attribute changes due to R across demographic groups may be an indicator of data and/or algorithmic bias.
For face data, we use CFD [49], centering our analysis on self-identified sensitive labels: Black , White , Indian , and (East) Asian .
We create lowresolution faces by downsampling from 1024 × 1024 to 32 × 32 .
For each group, we calculate [Table] the mean cosine similarity between each ( z H , z ′ H ) pair and report the min-max group ratio, i
[Figure] e., worst-case scenario [20]. For the CelebA and FairFace models, attribute representations correspond to unnormalized classifier predictions (logits)
[Figure]
whereas for the FAX and CASIA-WebFace-AF models we use the face embeddings.
The min-max ratio for the FAX, CelebA, FairFace, and CASIA-WebFace-AF attribute representations are 0.78, 0.88, 0.58, and 0.98, resp.
All attribute representations result in the same min-max pair, i.e., Black - White .
Using FAX, CelebA, and FairFace attribute representations, we find statistically significant differences in 22/22, 36/40, and 18/18, resp., for Black individuals, whereas for White individuals, we find statistically significant differences in 16/22, 26/40, and 12/18, resp.
All methods show that Black individuals undergo more significant attribute changes.
Fig.
7 plots the mean of z ′ H − z H , highlighting FAX-model attribute changes per group.
To ease presentation, we combined
FAX-model dimensions measuring the same attribute.
First, Black individuals have highly reduced values in the Black dimension and magnified values in the White dimension, corroborating previous qualitative findings [75].
Second, the impact of PULSE on White individuals largely centers on changes in face width and length, representing a novel insight.
While neither type of modification is desirable, the inadvertent erasure of minority groups is extremely harmful.
FAX dimensions are therefore useful as a tool for gaining insight into the behavior of face-based models.