Generative models often use human evaluations to determine and justify progress. Unfortunately, existing human evaluation methods are ad-hoc : there is currently no standardized, validated evaluation that: (1) measures perceptual fidelity, (2) is reliable, (3) separates models into clear rank order, and (4) ensures high-quality measurement without intractable cost. In response, we construct Human-eYe Perceptual Evaluation (HYPE) , a human metric that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) results in separable model performances, and (4) efficient in cost and time. We introduce two methods. The first, HYPE-Time, measures visual perception under adaptive time constraints to determine the minimum length of time (e.g. , 250ms) that model output such as a generated face needs to be visible for people to distinguish it as real or fake. The second, HYPE-Infinity ,  measures human error rate on fake and real images with no time constraints, maintaining stability and drastically reducing time and cost . We test HYPE across four state-of-the-art generative adversarial networks (GANs) on unconditional image generation using two datasets, the popular CelebAand the newer higher-resolution FFHQ, and two sampling techniques of model outputs. By simulating HYPE 's evaluation multiple times, we demonstrate consistent ranking of different models, identifying StyleGAN with truncation trick sampling (27.6\% HYPE-Infinity deception rate, with roughly one quarter of images being misclassified by humans) as superior to StyleGAN without truncation (19.0\%) on FFHQ. See URL for details .
Generative models often use human evaluations to measure the perceived quality of their outputs. Automated metrics are noisy indirect proxies, because they rely on heuristics or pretrained embeddings. However, up until now, direct human evaluation strategies have been ad-hoc , neither standardized nor validated. Our work establishes a gold standard human benchmark for generative realism. We construct Human eYe Perceptual Evaluation (HYPE) a human benchmark that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) able to produce separable model performances, and (4) efficient in cost and time. We introduce two variants: one that measures visual perception under adaptive time constraints to determine the threshold at which a model's outputs appear real (e.g. 250ms) , and the other a less expensive variant that measures human error rate on fake and real images sans time constraints . We test HYPE across six state-of-the-art generative adversarial networks and two sampling techniques on conditional and unconditional image generation using four datasets: CelebA, FFHQ, CIFAR-10, and ImageNet. We find that HYPE can track model improvements across training epochs, and we confirm via bootstrap sampling that HYPE rankings are consistent and replicable .
R
determine and justify progress. Unfortunately, existing human evaluation methods are
measure the perceived quality of their outputs. Automated metrics are noisy indirect proxies, because they rely on heuristics or pretrained embeddings. However, up until now, direct human evaluation strategies have been
R
: there is currently no standardized, validated evaluation that: (1) measures perceptual fidelity, (2) is reliable, (3) separates models into clear rank order, and (4) ensures high-quality measurement without intractable cost. In response, we construct Human-eYe
, neither standardized nor validated. Our work establishes a gold standard human benchmark for generative realism. We construct Human eYe
R
, a human metric
a human benchmark
R
results in
able to produce
R
methods. The first, HYPE-Time,
variants: one that
R
minimum length of time
threshold at which a model's outputs appear real
D
,
D
that model output such as a generated face needs to be visible for people to distinguish it as real or fake. The second, HYPE-Infinity
A
and the other a less expensive variant that
R
with no time constraints, maintaining stability and drastically reducing time and cost
sans time constraints
R
four
six
R
(GANs) on
and two sampling techniques on conditional and
R
two datasets, the popular CelebAand the newer higher-resolution FFHQ, and two sampling techniques of model outputs. By simulating HYPE 's evaluation multiple times, we demonstrate consistent ranking of different models, identifying StyleGAN with truncation trick sampling (27.6\% HYPE-Infinity deception rate, with roughly one quarter of images being misclassified by humans) as superior to StyleGAN without truncation (19.0\%) on FFHQ. See URL for details
four datasets: CelebA, FFHQ, CIFAR-10, and ImageNet. We find that HYPE can track model improvements across training epochs, and we confirm via bootstrap sampling that HYPE rankings are consistent and replicable