Paper:
Learning visual biases from human imagination
Although the human visual system can recognize many concepts under challenging conditions, it still has some biases. In this paper, we investigate whether we can extract these biases and transfer them into a machine recognition system. We introduce a novel method that, inspired by well-known tools in human psychophysics, estimates the biases that the human visual system might use for recognition, but in computer vision feature spaces. Our experiments are surprising, and suggest that classifiers from the human visual system can be transferred into a machine with some success. Since these classifiers seem to capture favorable biases in the human visual system, we further present an SVM formulation that constrains the orientation of the SVM hyperplane to agree with the bias from human visual system. Our results suggest that transferring this human bias into machines may help object recognition systems generalize across datasets and perform better when very little training data is available.

Review:
Summary: This paper draws inspiration from work on psychophysics on classification images.Large-scale human experiments were run, where people were asked to classify images generated from random noise (randomly generated by inverting HOG or CNN feature spaces to more closely approximate the distribution of natural images).The results were used to 1) visualize human perception of different classes, 2) see how well classifiers trained on datasets of random noise would work on real images, and 3) use the results as an additional source of information to regularize classifiers trained on a small number of images.Quality:This is a very unusual paper.It is overall a high quality and well written paper where interesting and novel experiments were carried out; however it is unclear if the results or methods of the paper are of practical value.Clarity:The paper is well written and clear.Originality:This is a really creative paper with an interesting and novel idea.It is nice that the authors drew in research from other disciplines (psychophysics) that most people in machine learning are unfamiliar with.Significance:I think the main value of this paper is that it will inspire readers to think about philosophical issues like what is dataset bias?, does the definition of a class come from human imagination or is inherent to the world?, how does one classify images that lie outside the distribution of natural images?I think in general people will find it fun and interesting to see how people perceive random images, as well as some of the qualitative results that people from India had different perceptions than people from the US.At the same time, the claims of the paper are controversial (see points 1 & 2 below) and the methods realistically have little practical value (see point 3 below).1) Does the proposed method alleviate problems of dataset bias? I don't think so.I think dataset bias emerges if one trains from datasets/images that come from a different distribution than natural data or problems we ultimately would like to solve (admittedly this is a fuzzy and probably misguided definition). Training a classifier (or generating a classification image) by having people label random white noise in pixel space is training from an extremely different (and also biased) synthetic distribution.Training from random noise in HOGgles space is similarly another type of biased synthetic dataset.I think it is an interesting finding to see how people classify images outside the domain of natural images, but this is an entirely different issue than dataset bias.A machine learning algorithm has biases because of the data that was used to train it as well as limitations of the particular learning model.Humans have biases due to their personal experiences as well as their learning models (ie., architecture of the brain).An algorithm trained from human labels will inherently (attempt to) learn to capture the biases of its human labelers.This isn't altered by using random images as opposed to natural images.Rather, using random images exacerbates the problem by using an even more unnatural distribution of data.2) Does the notion of classification images from psychophysics literature offer some new insight into human perception or dataset bias? I don't think so, or at least the paper doesn't go far enough in explaining this concept to communicate this to a NIPS audience that is unfamiliar with psychophysics research (such as myself).The 2 technical differences in the proposed classification image-based approach compared to methods used in machine learning are a) having people label randomly generated training data instead of natural images, and b) the method of generating classification images to visualize human classifiers.Looking at cited literature [23,9] classification image literature are making a lot of assumptions about human perception (e.g., a linear separator in feature space) and these assumptions are similar to some models used in machine learning (i.e., similar to a simplified version of Fisher's linear discriminant).I am unable to grasp why using a random noise training set is desirable or necessary, or if this is just a method of choice to acquire a dataset.3) The proposed method of regularizing classifiers trained from a small number of images with classification images is creative but impractical.It required tons of human annotation labelling random images for very small improvements in performance.