HYPE: H UMAN E Y E P ERCEPTUAL E VALUATION OF G ENERATIVE M ODELS
A BSTRACT
Generative models often use human evaluations to determine and justify progress.
Unfortunately, existing human evaluation methods are ad-hoc: there is currently no standardized, validated evaluation that: (1) measures perceptual ﬁdelity, (2) is reliable, (3) separates models into clear rank order and (4) ensures high-quality measurement without intractable cost.
In response, we construct H UMAN E Y E
P ERCEPTUAL E VALUATION (HYPE), a human metric that is (1) grounded in psy chophysics research in perception (2) reliable across different sets of randomly sampled outputs from a model results in separable model performances, and (4) efﬁcient in cost and time.
We introduce two methods
The ﬁrst , HYPE time , measures visual perception under adaptive time constraints to determine the mini mum length of time (e.g 250 ms) that model output such as a generated face needs to be visible for people to distinguish it as real or fake
The second , HYPE ∞ , measures human error rate on fake and real images with no time constraints, maintain - ing stability and drastically reducing time and cost.
We test HYPE across four state -of-the-art generative adversarial networks (GANs) on unconditional image generation using two datasets, the popular CelebA and the newer higher-resolution
FFHQ, and two sampling techniques of model outputs.
By simulating HYPE’s evaluation multiple times, we demonstrate consistent ranking of different models identifying StyleGAN with truncation trick sampling ( 27 .
6% HYPE ∞ deception rate with roughly one quarter of images being misclassiﬁed by humans) as superior to StyleGAN without truncation (
[Figure]
I NTRODUCTION
Historically, likelihood-based estimation techniques served as the de-facto evaluation metric for generative models (Hinton, 2002; Bishop, 2006).
But recently, with the application of generative models to complex tasks such as image and text generation (Goodfellow et al., 2014; Papineni et al., 2002), likelihood or density estimation grew no longer tractable (Theis et al., 2015).
Consequently, most generative tasks today resort to analyzing model outputs (R¨ossler et al., 2019; Salimans et al.,
Equal 2016; Denton et al., 2015; Karras et al., 2018; Brock et al., 2018; Radford et al., 2015).
These output evaluation metrics consist of either automatic algorithms that do not reach the ideals of likelihoodbased estimation, or ad-hoc human-derived methods that are unreliable and inconsistent (R¨ossler et al., 2019; Denton et al., 2015).
Consider the well-examined and popular computer vision task of realistic face generation (Goodfellow et al., 2014).
Automatic algorithms used for this task include Inception Score (IS) (Salimans et al., 2016) and Fr´echet Inception Distance (FID) (Heusel et al., 2017).
Both have been discredited for evaluation on non-ImageNet datasets such as faces (Barratt & Sharma, 2018; Rosca et al., 2017;
Borji, 2018; Ravuri et al., 2018).
They are also sensitive to visual corruptions such as salt and pepper noise and to semantic distortions such as swirled images — neither of which fool people (Heusel et al., 2017).
So, while automatic metrics are consistent and standardized, they cannot fully capture the semantic side of perceptual ﬁdelity (Borji, 2018).
Realizing the constraints of the available automatic metrics, many generative modeling challenges resort to summative assessments that are completely human (R¨ossler et al., 2019; Salimans et al., 2016; Denton et al., 2015).
These human measures are (1) ad-hoc, each executed in idiosyncrasy without proof of reliability or grounding to theory, and (2) high variance in their estimates (Salimans et al., 2016; Denton et al., 2015; Olsson et al., 2018).
These characteristics combine to a lack of reliability, and downstream, (3) a lack of clear separability between models.
Theoretically, given sufﬁciently large sample sizes of human evaluators and model outputs, the law of large numbers would smooth out the variance and reach eventual convergence; but this would occur at (4) a high cost and a long delay.
In this paper, we present HYPE (H UMAN E Y E P ERCEPTUAL E VALUATION ) that addresses these criteria in turn.
It: (1) measures the perceptual ﬁdelity of generative model outputs via a grounded method inspired by psychophysics methods in perceptual psychology, (2) is a reliable and consistent estimator, (3) is statistically separable to enable a comparative ranking, and (4) ensures a cost and time efﬁcient method through modern crowdsourcing techniques such as training and aggregation.
We present two methods of evaluation.
The ﬁrst, called HYPE time , is drawn directly from psychophysics literature (Klein, 2001) and displays images using adaptive time constraints to determine the time-limited perceptual threshold a person needs to distinguish real from fake (Cornsweet, 1962).
The HYPE time score is understood as the minimum time, in milliseconds, that a person needs to see the model’s output before they can distinguish it as real or fake.
Small HYPE time scores indicate that model outputs can be identiﬁed even at a glance; large scores suggest that people need to dedicate substantial time and attention.
The second method, called HYPE ∞ , is derived from the ﬁrst to make it simpler, faster, and cheaper while maintaining reliability.
It measures human deception from fake images with no time constraints.
The HYPE ∞ score is interpretable as the rate at which people mistake fake images and real images, given unlimited time to make their decisions.
We demonstrate HYPE’s performance on unconditional generation of human faces using generative adversarial networks (GANs) (Goodfellow et al., 2014).
We evaluate four state-of-the-art GANs:
WGAN-GP (Gulrajani et al., 2017), BEGAN (Berthelot et al., 2017), ProGAN (Karras et al., 2017), and the most recent StyleGAN (Karras et al., 2018).
First, we track progress across the years on the popular CelebA dataset (Liu et al., 2015).
We derive a ranking based on perception (HYPE time , in milliseconds) and error rate (HYPE ∞ , as a percentage) as follows:
StyleGAN ( 439 . 4 ms, 50 . 7% ),
ProGAN ( 363 . 7 ms, 40 .
, BEGAN ( 111 .
1 ms, 10 . 0% ), WGAN-GP ( 100 .
ms, 3 . 8% ).
A score of 500 ms on HYPE time indicates that outputs from the model become indistinguishable from real, when shown for 500 ms or less, but any more would start to reveal notable differences.
A score of 50% on HYPE ∞ represents indistinguishable results from real, conditioned on the real training set, while a score above 50% through 100% represents hyper-realism in which generated images appear more real than real ones when drawn from a mixed pool of both.
Next, we test StyleGAN trained on the newer FFHQ dataset (Karras et al., 2018), comparing between outputs generated when sampled with and without the truncation trick, a technique used to prune low-ﬁdelity generated images (Brock et al., 2018; Karras et al., 2018).
We ﬁnd that outputs generated with the truncation trick ( 363 . 2 ms, 27 . 6% ) signiﬁcantly outperforms those without it ( 240 .
7 ms, 19 . 0% ), which runs counter to scores reported by FID.
HYPE indicates that GANs have clear, measurable perceptual differences between them.
HYPE produces identical rankings between HYPE time and HYPE ∞ .
We also ﬁnd that even the best evaluated model, StyleGAN trained on FFHQ and sampled with the truncation trick, only performs at
6% HYPE ∞ , suggesting substantial opportunity for improvement.
Finally, we show that we can reliably reproduce these results with 95% conﬁdence intervals using 30 human evaluators at $60 in a task that takes 10 minutes.
While important measures, we do not focus on diversity, overﬁtting, entanglement, training stability, and computational and sample efﬁciency of the model (Borji, 2018;
Lucic et al., 2018)
and instead aim to construct the gold standard for human perceptual ﬁdelity.
We deploy HYPE as a rapid solution for researchers to measure their generative models, requiring just a single click to produce reliable scores and measure progress.
We deploy HYPE at https://hype.stanford.edu, where researchers can upload a model and retrieve a HYPE score inminutes for $60 .
Future work would extend HYPE to adapt to other generative tasks such as text generation or abstractive summarization.
HYPE: H UMAN E Y E P ERCEPTUAL E VALUATION
Model creators can choose to perform two different evaluations and receive two different scores: the HYPE time score, which gathers time-limited perceptual thresholds to measure the psychometric function and report the minimum time people need to make accurate classiﬁcations, and the HYPE ∞ score, a simpliﬁed approach which assesses people’s error rate under no time constraint.
HYPE displays a series of images one by one to crowdsourced evaluators on Amazon Mechanical Turk and asks the evaluators to assess whether each image is real or fake.
Half of the images are drawn from the model’s training set (e.g., FFHQ or CelebA), which constitute the real images.
The other half are drawn from the model’s output.
We use modern crowdsourcing training and quality control techniques to ensure high quality labels (Mitra et al., 2015).
2. P
: HYPE TIME
Our ﬁrst method, HYPE time , measures time-limited perceptual thresholds.
It is rooted in psychophysics literature, a ﬁeld devoted to the study of how humans perceive stimuli, to evaluate human time thresholds upon perceiving an image.
Our evaluation protocol follows the procedure known as the adaptive staircase method (Cornsweet, 1962)
(see Figure 3).
An image is ﬂashed for a limited length of time, after which the evaluator is asked to judge whether it is real or fake.
If the evaluator consistently answers correctly, the staircase descends and ﬂashes the next image with less time.
If the evaluator is incorrect, the staircase ascends and provides more time.
This process requires sufﬁcient iterations to converge on the minimum time needed for each evaluator to sustain correct guesses in a sample-efﬁcient manner (Cornsweet, 1962), producing what is known as the psychometric function (Wichmann & Hill, 2001), the relationship of timed stimulus exposure to accuracy.
For example, for an easily distinguishable set of generated images, a human evaluator would immediately drop to the lowest millisecond exposure.
However, for a harder set, it takes longer to converge and the person would remain at a longer exposure level in order to complete the task accurately.
The modal time value is the evaluator’s perceptual threshold: the shortest exposure time at which they can maintain effective performance (Cornsweet, 1962; Greene & Oliva, 2009).
HYPE time displays three blocks of staircases for each evaluator.
An image evaluation begins with a 3-2-1 countdown clock, each number displaying for 500 ms.
The sampled image is then displayed for the current exposure time.
Immediately after each image, four perceptual mask images are rapidly displayed for 30 ms each.
These noise masks are distorted to prevent visual afterimages and further sensory processing on the image afterwards (Greene & Oliva, 2009).
We generate masks from the test images, using an existing texture-synthesis algorithm (Portilla & Simoncelli, 2000).
Upon each submission, HYPE time reveals to the evaluator whether they were correct.
Image exposure times fall in ms, 1000 ms , which we derive from the perception literature (Fraisse, 1984).
All blocks begin at 500 ms and last for 150 images ( 50 % generated, 50 % real), values empirically tuned from prior work (Cornsweet, 1962; Dakin & Omigie, 2009).
Exposure times are raised at 10 ms increments and reduced at 30 ms decrements, following the 3 -up/ 1 -down adaptive staircase approach.
This 3 -up/ 1 -down approach theoretically leads to a 75% that the perceptual threshold (Levitt, 1971; Greene & Oliva, 2009; - sweet, 1962)
Every evaluator completes multiple staircases, called blocks , on different sets of images.
As a result, we observe multiple measures for the model.
We employ three blocks, to balance quality estimates against evaluators’ fatigue (Krueger, 1989; Rzeszotarski et al., 2013).
We average the modal exposure times across blocks to calculate a ﬁnal value for each evaluator.
Higher scores indicate a better model, whose outputs take longer time exposures to discern from real.
2. C OST - EFFECTIVE APPROXIMATION : HYPE ∞
Building on the previous method, we introduce HYPE ∞ : a simpler, faster, and cheaper method after ablating HYPE time to optimize for speed, cost, and ease of interpretation.
HYPE ∞
shifts from a measure of perceptual time to a measure of human deception rate, given inﬁnite evaluation time.
The HYPE ∞ score gauges total error on the task, enabling the measure to capture errors on both fake and real images, and effects of hyperrealistic generation when fake images look even more realistic than real images.
HYPE ∞ requires fewer images than HYPE time to ﬁnd a stable value, at a 6x reduction in time and cost (10 minutes per evaluator instead of 60 minutes, at the same rate of $ per hour).
Higher scores are better, like HYPE time : a HYPE ∞ value of 10% indicates that only 10% of images deceive people, whereas 50% indicates that people are mistaking real and fake images at chance, rendering fake images indistinguishable from real.
Scores above 50% suggest hyperrealistic images, as evaluators mistake images at a rate greater than chance, on average mistaking more fake images to be real than real ones and vice versa.
HYPE ∞ shows each evaluator a total of 100 images: 50 real and 50 fake.
Similar to HYPE time , we calculate the proportion of images that were judged incorrectly, and aggregate the judgments over the n evaluators on k images to produce the ﬁnal score for a given model.
2. C ONSISTENT AND R ELIABLE D ESIGN
To ensure that our reported scores are consistent and reliable, we need to sample sufﬁcient model outputs, select suitable real images for comparison, and hire, qualify, and appropriately pay enough evaluators.
To ensure a wide coverage of images, we randomly select the fake and real images provided to workers from a pool of 5000 images (see Sampling sufﬁcient model outputs, below).
Comparing results between single evaluators can be problematic.
To ensure HYPE is reliable, we must use a sufﬁciently large number of evaluators, n , which can be treated as a hyperparameter.
To determine a suitable number, we use our experimental results (further discussed in the Results section) to compute bootstrapped 95% conﬁdence intervals (CI) across various values of n evaluators.
Quality of evaluators.
To obtain a high-quality pool of evaluators, each is required to pass a qualiﬁcation task.
Such a pre-task ﬁltering approach, sometimes referred to as a person-oriented strategy, is known to outperform process-oriented strategies that perform post-task data ﬁltering or processing (Mitra et al., 2015).
Our qualiﬁcation task displays 100 images ( 50 real and 50 fake) with no time limits.
Evaluators pass if they correctly classify 65% of both real and fake images.
This threshold should be treated as a hyperparameter and may change depending upon the GANs used in the tutorial and the desired discernment ability of the chosen evaluators.
We choose 65% based on the cumulative binomial probability of 65 binary choice answers out of 100 total answers: there is only a one in one-thousand chance that an evaluator will qualify by random guessing.
Unlike in the staircase task itself, fake qualiﬁcation images are drawn equally from multiple different GANs.
This is to ensure an equitable qualiﬁcation across all GANs, as to avoid a qualiﬁcation that is biased towards evaluators who are particularly good at detecting one type of GAN.
The qualiﬁcation is designed to be taken occasionally, such that a pool of evaluators can assess new models on demand.
Payment.
Evaluators are paid a base rate of $1 for working on the qualiﬁcation task.
To incentivize evaluators to remained engaged throughout the task, all further pay after the qualiﬁcation comes from a bonus of $0 .
02 per correctly labeled image.
This pay rate typically results in a wage of approximately $12 per hour, which is above a minimum wage in our local state.
Sampling sufﬁcient model outputs.
The selection of K images to evaluate from a particular model is a critical component of a fair and useful evaluation.
We must sample a large enough number of images that fully capture a model’s generative diversity, yet balance that against tractable costs in the evaluation.
We follow existing work on evaluating generative output by sampling K =generated images from each model (Salimans et al., 2016;
Miyato et al., 2018; Warde-Farley &
Bengio, 2016) and K = 5000 real images from the training set.
From these samples, we randomly select images to give to each evaluator.
E XPERIMENTS
Datasets.
We evaluate on two datasets of human faces:
CelebA-64 (Liu et al., 2015) is popular dataset for unconditional image generation, used since 2015.
CelebA-64 includes 202,599 images of human faces, which we align and crop to be 64 × 64 pixel images using a standard mechanism.
We train all models without using attributes.
FFHQ-1024 (Karras et al., 2018) is a newer dataset released in 2018 with StyleGAN and includes 70,000 images of size 1024 × 1024 pixels.
Architectures.
We evaluate on four state-of-the-art models trained on CelebA-64: StyleGAN (Karras et al., 2018), ProGAN (Karras et al., 2017), BEGAN (Berthelot et al., 2017), and WGANGP (Gulrajani et al., 2017).
We also evaluate on two types of sampling from StyleGAN trained on FFHQ-1024: with and without the truncation trick, which we denote StyleGAN trunc and
StyleGAN no-trunc respectively.
For parity on our best models across datasets, StyleGAN trained on CelebA-64 is sampled with the truncation trick.
We train StyleGAN, ProGAN, BEGAN, and WGAN-GP on CelebA-64 using 8 Tesla V
GPUs for approximately 5 days.
We use the ofﬁcial released pretrained StyleGAN model on
FFHQ-1024 (Karras et al., 2018).
We sample noise vectors from the d -dimensional spherical Gaussian noise prior z ∈ R d ∼ N (0 , I ) during training and test times.
We speciﬁcally opted to use the same standard noise prior for comparison, yet are aware of other priors that optimize for FID and IS scores (Brock et al., 2018).
We select training hyperparameters published in the corresponding papers for each model.
We evaluate all models for each task with the two HYPE methods: (1) HYPE time and (2) HYPE ∞ .
Evaluator recruitment.
We recruit 360 total human evaluators across our 12 evaluations, each of which included 30 evaluators, from Amazon Mechanical Turk.
Each completed a single evaluation in { CelebA-64, FFHQ-1024 } × { HYPE time , HYPE ∞ } .
To maintain a between subjects study in this evaluation, we did not allow duplicate evaluators across tasks or methods.
In total, we recorded ( 4 CelebA-64 + 2 FFHQ-1024) models × 30 evaluators × 550 responses = 99 , 000 total responses for our HYPE time evaluation and ( 4 CelebA-64 + 2 FFHQ-1024) models × 30 evaluators × 100 responses = 18 , 000 total responses for our HYPE ∞ evaluation.
Metrics.
For HYPE time , we report the modal perceptual threshold in milliseconds.
For HYPE ∞ , we report the error rate as a percentage of images, as well as the breakdown of this rate on real and fake images individually.
To show that our results for each model are separable, we report a oneway ANOVA with Tukey pairwise post-hoc tests to compare all models within each {
CelebA-64,
FFHQ-1024 } × { HYPE time , HYPE ∞ } combination.
As mentioned previously, reliability is a critical component of HYPE, as an evaluation is not useful if a researcher can re-run it and get a different answer.
To show the reliability of HYPE, we use bootstrap (Felsenstein, 1985), a form of simulation, to simulate what the results would be if we resample with replacement from this set of labels.
Our goal is to see how much variation we may get in the outcome.
We therefore report evaluator 95% bootstrapped conﬁdence intervals, along with standard deviation of the bootstrap sample distribution.
Conﬁdence intervals (CIs) are deﬁned as the region that captures where the modal exposure might be estimated to be if the same sampling procedure were repeated many times.
For this and all following results, bootstrapped conﬁdence intervals were calculated by randomly sampling 30 evaluators with replacement from the original set of evaluators across 10 , 000 iterations.
Note that bootstrapped CIs do not represent that there necessarily exists substantial uncertainty—our reported modal exposure (for HYPE time ) or detection rate (for HYPE ∞ ) is still the best point estimate of the value.
We discuss bootstrapped CIs for other numbers of evaluators later on in the Cost Tradeoffs section.
R ESULTS
First, we report results using the above datasets, models and metrics using HYPE time .
Next, we demonstrate the HYPE ∞ ’s results approximates the ones from HYPE time at a fraction of the cost and time.
Next, we trade off the accuracy of our scores with time.
We end with comparisons to FID.
4. HYPE TIME
CelebA-64.
We ﬁnd that StyleGAN trunc resulted in the highest HYPE time score (modal exposure time), at a mean of 439 .
3 ms, indicating that evaluators required nearly a half-second of exposure to accurately classify StyleGAN trunc images (Table 1).
StyleGAN trunc is followed by ProGAN at 363 .
7 ms, a 17% drop in time.
BEGAN and WGAN-GP are both easily identiﬁable as fake, so they are tied in third place around the minimum possible exposure time available of 100 ms.
Both
BEGAN and WGAN-GP exhibit a bottoming out effect — reaching our minimum time exposure of 100 ms quickly and consistently 1 .
This means that humans can detect fake generated images at 100 ms and possibly lower.
Thus, their scores are identical and indistinguishable.
To demonstrate separability between StyleGAN trunc , ProGAN, BEGAN, and WGAN-GP together, we report results from a one-way analysis of variance (ANOVA) test between all four models, where each model’s input is the list of modes from each model’s 30 evaluators.
The ANOVA results conﬁrm that there is a statistically signiﬁcant omnibus difference ( F (3 , 29) = 83 .
5 , p < 0 . 0001 ).
Pairwise post-hoc analysis using Tukey tests conﬁrms that all pairs of models are separable (all p < 0 . 05 ), with the exception of BEGAN and WGAN-GP ( n.s. ).
FFHQ-1024 .
We ﬁnd that StyleGAN trunc resulted in a higher exposure time than StyleGAN no-trunc , at 363 . 2 ms and 240 .
7 ms, respectively (Table 2).
While the 95% conﬁdence intervals that represent a very conservative overlap of 2 .
7 ms, an unpaired t-test conﬁrms that the difference between the two models is signiﬁcant ( t (58)
= 2 . 3 , p = 0 .
[Table]
[Table]
4. HYPE
[Table]
CelebA-64 .
Table 3 reports results for HYPE ∞ on CelebA-64.
We ﬁnd that StyleGAN trunc resulted in the highest HYPE ∞ score, fooling evaluators 50.7% of the time.
StyleGAN trunc is followed by ProGAN at 40 .
3% , BEGAN at 10 . 0% , and WGAN-GP at 3 .
No conﬁdence intervals are overlapping and an ANOVA test is signiﬁcant ( F (3 , 29)
= 404 . 4 , p < 0 .
Pairwise post-hoc
Tukey tests show that all pairs of models are separable (all p < 0 . 05 ).
Notably, HYPE ∞ results in separable results for BEGAN and WGAN-GP, unlike in HYPE time where they were not separable due to a bottoming-out effect.
[Table] [Table]
We observe a consistently separable difference between StyleGAN trunc and
StyleGAN no-trunc and clear delineations between models (Table 4).
HYPE ranks trunc ( 27 .
6% ) above StyleGAN no-trunc ( 19 . 0% ) with no overlapping CIs.
Separability is conﬁrmed by an unpaired t-test ( t (58) = 8 . 3 , p < 0 . 001 ).
4. C OST TRADEOFFS WITH ACCURACY AND TIME
One of HYPE’s goals is to be cost and time efﬁcient.
When running HYPE, there is an inherent tradeoff between accuracy and time, as well as between accuracy and cost.
This is driven by the law of large numbers: recruiting additional evaluators in a crowdsourcing task often produces more consistent results, but at a higher cost (as each evaluator is paid for their work) and a longer amount of time until completion (as more evaluators must be recruited and they must complete their work).
To manage this tradeoff, we run an experiment with HYPE ∞ on StyleGAN trunc .
We completed an additional evaluation with 60 evaluators, and compute 95% bootstrapped conﬁdence intervals, [Table] choosing from 10 to 120 evaluators (Figure 4).
We see that the CI begins to converge aroundevaluators, our recommended number of evaluators to recruit and the default that we build into our system.
Payment to evaluators was calculated as described in the Approach section.
Atevaluators, the cost of running HYPE time on one model was approximately $360 , while the cost of running HYPE ∞ on the same model was approximately $60 .
Payment per evaluator for both tasks was approximately $12 /hr, and evaluators spent an average of one hour each on a HYPE time task and 10 minutes each on a HYPE ∞ task.
Thus, HYPE ∞ achieves its goals of being signiﬁcantly cheaper to run than HYPE time while maintaining consistency.
4. C OMPARISON TO FID
As FID is one of the most frequently used evaluation methods for unconditional image generation, it is imperative to compare HYPE against FID on the same models (Table 5).
We show through Spearman rank-order correlation coefﬁcients that FID is correlated with neither human judgment measure, not HYPE time ( ρ = − 0 . 0286 ) nor with HYPE ∞ ( ρ = − 0 . 0857 ), where a Spearman correlation of -1.0 is ideal because lower FID and higher HYPE scores indicate stronger models.
Meanwhile,
HYPE time and HYPE ∞ exhibit strong correlation ( ρ = 0 . 9429 ), where 1.0 is ideal because they are directly related.
We calculate FID across the standard protocol of evaluating 50K generated and 50K real images for both CelebA-64 and FFHQ-1024, reproducing scores for StyleGAN no-trunc .
[Table]
R ELATED WORK
Cognitive psychology.
We leverage decades of cognitive psychology to motivate how we use stimulus timing to gauge the perceptual realism of generated images.
It takes an average of 150 ms of focused visual attention for people to process and interpret an image, but only 120 ms to respond to faces because our inferotemporal cortex has dedicated neural resources for face detection (Rayner et al., 2009; Chellappa et al., 2010).
Perceptual masks are placed between a person’s response to a stimulus and their perception of it to eliminate post-processing of the stimuli after the desired time exposure (Sperling, 1963).
Prior work in determining human perceptual thresholds (Greene &
Oliva, 2009) generates masks from their test images using the texture-synthesis algorithm (Portilla & Simoncelli, 2000).
We leverage this literature to establish feasible lower bounds on the exposure time of images, the time between images, and the use of noise masks.
Success of automatic metrics.
Common generative modeling tasks include realistic image generation (Goodfellow et al., 2014), machine translation (Bahdanau et al., 2014), image captioning (Vinyals et al., 2015), and abstract summarization (Mani, 1999), among others.
These tasks often resort to automatic metrics like the Inception Score (IS) (Salimans et al., 2016) and Fr´echet
Inception Distance (FID) (Heusel et al., 2017) to evaluate images and BLEU (Papineni et al., 2002),
CIDEr (Vedantam et al., 2015) and METEOR (Banerjee & Lavie, 2005) scores to evaluate text.
While we focus on how realistic generated content appears, other automatic metrics also measure diversity of output, overﬁtting, entanglement, training stability, and computational and sample efﬁciency of the model (Borji, 2018;
Lucic et al., 2018; Barratt & Sharma, 2018).
Our metric may also capture one aspect of output diversity, insofar as human evaluators can detect similarities or patterns across images.
Our evaluation is not meant to replace existing methods but to complement them.
Limitations of automatic metrics.
Prior work has asserted that there exists coarse correlation of human judgment to FID (Heusel et al., 2017) and IS (Salimans et al., 2016), leading to their widespread adoption.
Both metrics depend on the Inception v3 Network (Szegedy et al., 2016), a pretrained ImageNet model, to calculate statistics on the generated output (for IS) and on the real and generated distributions (for FID).
The validity of these metrics when applied to other datasets has been repeatedly called into question (Barratt & Sharma, 2018; Rosca et al., 2017; Borji, 2018;
Ravuri et al., 2018).
Perturbations imperceptible to humans alter their values, similar to the behavior of adversarial examples (Kurakin et al., 2016).
Finally, similar to our metric, FID depends on a set of real examples and a set of generated examples to compute high-level differences between the distributions, and there is inherent variance to the metric depending on the number of images and which images were chosen—in fact, there exists a correlation between accuracy and budget (cost of computation) in improving FID scores, because spending a longer time and thus higher cost on compute will yield better FID scores (Lucic et al., 2018).
Nevertheless, this cost is still lower than paid human annotators per image.
Human evaluations.
Many human-based evaluations have been attempted to varying degrees of success in prior work, either to evaluate models directly (Denton et al., 2015; Olsson et al., 2018) or to motivate using automated metrics (Salimans et al., 2016; Heusel et al., 2017).
Prior work also used people to evaluate GAN outputs on CIFAR-10 and MNIST and even provided immediate feedback after every judgment (Salimans et al., 2016).
They found that generated MNIST samples have saturated human performance—that is, people cannot distinguish generated numbers from real
MNIST numbers, while still ﬁnding 21 .
3% error rate on CIFAR-10 with the same model (Salimans et al., 2016).
This suggests that different datasets will have different levels of complexity for crossing realistic or hyper-realistic thresholds.
The closest recent work to ours compares models using a tournament of discriminators (Olsson et al., 2018).
While reported to have anecdotal correlation with human judgment, this comparison was not yet rigorously evaluated and human discriminators were not presented experimentally.
The framework we present would enable such a tournament evaluation to be performed reliably and easily.
D ISCUSSION AND F UTURE W ORK
We develop HYPE (1) grounded in psychophysics to measure human perceptual ﬁdelity.
Through empirical analysis, we ﬁnd that (2) HYPE is reliable in its ability to replicate human perceptual performance measures across models and tasks, demonstrating non-overlapping 95% bootstrapped conﬁdence intervals across all models with 30 evaluators.
(3) HYPE also produces separable results and ranks models consistently using two methods: HYPE time and HYPE ∞ .
HYPE ∞ builds on
HYPE time to (4) optimize for cost and time, enabling an efﬁcient evaluation of a single model to occur within 10 minutes at $60 .
Envisioned Use.
We created HYPE to be a simple and rapid method of human evaluation of generative models.
We envision that researchers will use HYPE via https://hype.stanford.edu to upload their model, receive a score, and compare progress.
We also envision that HYPE, and speciﬁcally the variant HYPE ∞ or HYPE time that ﬁts best, to be used in generative model competitions, e.g. a
HYPE ∞ leaderboard on CelebA-64.
Competition organizers can have the license to choose whether the model alone should be compared, or that automatic sampling methods such as the truncation trick should be included as part of the system.
During periods of high usage, such as competitions, a retainer model (Bernstein et al., 2011) enables nearly instantaneous availability of evaluators, meaning evaluation using HYPE ∞ could take as few as 10 minutes.
Without the retainer model, however, we estimate that time rises to 30 minutes.
Limitations.
Extensions of HYPE may require different task designs.
For example, conditional image generation will need to isolate different categories into individual blocks (e.g. cats in one block, dogs in another, etc.)
(Krishna et al., 2016).
The design would likely affect humans’ absolute thresholds, as cognitive load may be of consideration; the number of humans required per task may require signiﬁcant increase if evaluating fairly across all possible categories.
Practically, the most valuable direction for the community to pursue with HYPE is likely one that includes the most difﬁcult categories, especially when progress on those is hard to measure using automatic metrics.
In the case of text generation (translation, caption generation), HYPE time may require much longer and much higher range adjustments to the perceptual time thresholds for text comprehensibility than those used in visual perception (Krishna et al., 2016).
Future Work.
We plan to extend HYPE to different imaging datasets and imaging tasks such as conditional image generation, as well as to text and video, such as translation (Papineni et al., 2002) and video captioning (Krishna et al., 2017).
Future work would also explore budget-optimal estimation of HYPE scores and adaptive estimation of evaluator quality (Karger et al., 2014).
Additional improvements involve identifying images that require more evaluators (Weld et al., 2015).
We also aim to build in faster time exposures under 100 ms — ideally down to 13 ms, the minimum time exposure of human perception (Potter et al., 2014) — for tasks that require that level of granularity.
Doing so requires careful engineering solution, since 100 ms appears to be the minimum time that is trustable before we are throttled by JavaScript paint and rendering times on modern browsers.
We will investigate the ecological validity of our methods – that is, whether HYPE’s evaluation is representative of how a person would perceive a GAN in everyday life.
For instance, HYPE shows evaluators whether they classiﬁed an image correctly immediately after they answer.
While this is standard practice in the psychophysics literature for staircase tasks, it likely does not reﬂect how one might encounter generated content in everyday life.
Notably, in pilot studies, we found that without such feedback, evaluators were far less consistent and our metric would not be stable.
Finally, we plan to investigate whether the reliability of HYPE may be impacted by the month or year at which it is run, as the population of available crowdsourced workers may differ across these factors.
Anecdotally, we have found HYPE to be reliable regardless of the time of day.
C ONCLUSION
HYPE provides researchers with two human evaluation methods for GANs that (1) are grounded in psychopisics to measure human perceptual ﬁdelity directly, (2) provide task designs that result in consistent and reliable results, (3) distinguishes between different model performances through separable results, (4) is cost and time efﬁcient .
We report two metrics: HYPE time and
HYPE ∞ .
HYPE time uses time perceptual thresholds where longer time constraints are more difﬁcult to achieve because they give humans more time to interpret the generated content and observe artifacts.
HYPE ∞ reports the error rate under unlimited time, where higher rates indicate a more realistic set of outputs.
We demonstrate the efﬁcacy of our approach on unconditional image generation across four GANs { StyleGAN, ProGAN, BEGAN, WGAN-GP } and two datasets of human faces { CelebA-64, FFHQ-1024 } , with two types of output sampling on StyleGAN { with the truncation trick, without the truncation trick } .
To encourage progress of generative models towards human-level visual ﬁdelity, we deploy our evaluation system at https://hype.stanford.edu, so anyone can upload and evaluate their models based on HYPE at the click of a button.
A PPENDIX A. C ONFIDENCE I NTERVALS
[Table]
StyleGAN 2.
StyleGAN 1. StyleGAN 1.
StyleGAN 1. StyleGAN 1.
StyleGAN 1. StyleGAN 1.
StyleGAN trunc 4.
StyleGAN trunc 3.
StyleGAN trunc 2. StyleGAN trunc 2.
StyleGAN trunc 2. StyleGAN trunc 2.
StyleGAN trunc 2.
StyleGAN no-trunc 3.
StyleGAN no-trunc 2. StyleGAN no-trunc 2.
StyleGAN no-trunc 1. StyleGAN no-trunc 1.
StyleGAN no-trunc 1. StyleGAN no-trunc 1.
Model
[Table]
Number
[Table]
WGAN-GP 0.
WGAN-GP 0. WGAN-GP 0.
WGAN-GP 0. WGAN-GP 0.
BEGAN 12.
BEGAN 10.
BEGAN 8.
BEGAN 7. BEGAN 7.
StyleGAN trunc 52.
StyleGAN trunc 42.
StyleGAN trunc 37.
StyleGAN trunc 33.
StyleGAN trunc 30.
StyleGAN trunc 28.
StyleGAN trunc 26.
StyleGAN no-trunc 49.
StyleGAN no-trunc 40.
StyleGAN no-trunc 34.
StyleGAN no-trunc 31.
StyleGAN no-trunc 28.
StyleGAN no-trunc 26.
StyleGAN no-trunc 24.
ProGAN 51.
ProGAN 41.
ProGAN 36.
ProGAN 32.
ProGAN 29.
StyleGAN 56.
StyleGAN 46.
StyleGAN 39.
StyleGAN 35.
StyleGAN 32.