Distinct scientific theories can make similar predictions. To adjudicate between theories, we must design experiments for which the theories make distinct predictions. Here we consider the problem of comparing deep neural networks as models of human visual recognition. To efficiently determine which modelsbetter explain human responses, we synthesize controversial stimuli: images for which different models produce distinct responses. We tested nine different models, which employed different architectures and recognition algorithms, including discriminative and generative models, all trained to recognize handwritten digits ( from the MNISTset of digit images ). We synthesized controversial stimuli to maximize the disagreement among the models . Human subjects viewed hundreds of these stimuli  and judged the probability of presence of each digit  in each image. We quantified how accurately each model predicted the human judgements. We found that the generative models (which learn the distribution of images for each class) better predicted the human judgmentsthan the discriminative models (which learn to directly map from images to labels) . The best performing model was the generative Analysis-by-Synthesis model (based on variational autoencoders) . However, a simpler generative model (based on Gaussian-kernel-density estimation) also performed better than each of the discriminative models . None of the candidate models fully explained the human responses. We discuss the advantages and limitations of controversial stimuli as an experimental paradigm and how they generalize and improve onadversarial examples as probes of discrepancies between models and human perception.
Distinct scientific theories can make similar predictions. To adjudicate between theories, we must design experiments for which the theories make distinct predictions. Here we consider the problem of comparing deep neural networks as models of human visual recognition. To efficiently compare models' ability to predict human responses, we synthesize controversial stimuli: images for which different models produce distinct responses. We applied this approach to two visual recognition tasks, handwritten digits ( MNIST) and objects in small natural images (CIFAR-10). For each task, we synthesized controversial stimuli to maximize the disagreement among models which employed different architectures and recognition algorithms . Human subjects viewed hundreds of these stimuli , as well as natural examples, and judged the probability of presence of each digit /object category in each image. We quantified how accurately each model predicted the human judgments . The best performing models were a generative Analysis-by-Synthesis model (based on variational autoencoders) for MNIST and a hybrid discriminative-generative Joint Energy Model for CIFAR-10. These DNNs, which model the distribution of images, performed better than purely discriminative DNNs, which learn only to map images to labels . None of the candidate models fully explained the human responses. Controversial stimuli generalize the concept of adversarial examples, obviating the need to assume a ground-truth model. Unlike natural images, controversial stimuli are not constrained to the stimulus distribution models are trained on, thus providing severe out-of-distribution tests that reveal the models' inductive biases. Controversial stimuli therefore provide powerful probes of discrepancies between models and human perception.
R
determine which modelsbetter explain
compare models' ability to predict
R
tested nine different models, which employed different architectures and recognition algorithms, including discriminative and generative models, all trained to recognize
applied this approach to two visual recognition tasks,
R
from the MNISTset of digit images ). We
MNIST) and objects in small natural images (CIFAR-10). For each task, we
R
the models
models which employed different architectures and recognition algorithms
A
, as well as natural examples,
A
/object category
R
judgements. We found that the generative models (which learn the distribution of images for each class) better predicted the human judgmentsthan the discriminative models (which learn to directly map from images to labels)
judgments
R
model was the
models were a
R
. However, a simpler generative model (based on Gaussian-kernel-density estimation) also
for MNIST and a hybrid discriminative-generative Joint Energy Model for CIFAR-10. These DNNs, which model the distribution of images,
R
each of the discriminative models
purely discriminative DNNs, which learn only to map images to labels
R
We discuss the advantages and limitations of controversial stimuli as an experimental paradigm and how they generalize and improve onadversarial examples as
Controversial stimuli generalize the concept of adversarial examples, obviating the need to assume a ground-truth model. Unlike natural images, controversial stimuli are not constrained to the stimulus distribution models are trained on, thus providing severe out-of-distribution tests that reveal the models' inductive biases. Controversial stimuli therefore provide powerful