Today, computer vision systems are tested by their accuracy in detecting and localizing instances of objects. As an alternative, and motivated by the ability of humans to provide far richer descriptions, even tell a story about an image, we construct a "visual Turing test": an operator-assisted device that produces a stochastic sequence of binary questions from a given test image. The query engine proposes a question; the operator either provides the correct answer or rejects the question as ambiguous; the engine proposes the next question ("just-in-time truthing"). The test is then administered to the computer-vision system, one question at a time. After the system's answer is recorded, the system is provided the correct answer and the next question. Parsing is trivial and deterministic; the system being tested requires no natural language processing. The query engine employs statistical constraints, learned from a training set, to produce questions with essentially unpredictable answersthe answer to a question, given the history of questions and their correct answers, is nearly equally likely to be positive or negative. In this sense, the test is only about vision. The system is designed to produce streams of questions that follow natural story lines, from the instantiation of a unique object, through an exploration of its properties, and onto its relationships with other uniquely instantiated objects. Going back at least to the mid-twentieth century there has been an active debate about the state of progress in artificial intelligence and how to measure it. Alan Turing [1] proposed that the ultimate test of whether or not a machine could "think," or think at least as well as a person, was for a human judge to be unable to tell which was which based on natural language conversations in an appropriately cloaked scenario. In a much-discussed variation (sometimes called the "standard interpretation"), the objective is to measure how well a computer can imitate a human [2] in some circumscribed task normally associated with intelligent behavior, although the practical utility of "imitation" as a criterion for performance has also been questioned [3] . In fact, the overwhelming focus of the modern AI community has been to assess machine performance more directly by dedicated tests for specific tasks rather than debating about general "thinking" or Turing-like competitions between people and machines. In this paper we implement a new, query-based test for computer vision, one of the most vibrant areas of modern AI research. Throughout this paper we use "computer vision" more or less synonymously with semantic image interpretation -"images to words." But of course computer vision encompasses a great many other activities; it includes the theory and practice of image formation ("sensors to images"); image processing ("images to images"); mathematical representations; video processing; metric scene reconstruction; and so forth. In fact, it may not be possible to interpret scenes at a semantic level without taking at least some of these areas into account, especially the geometric relationship between an image and the underlying 3D scene. But our focus is how to evaluate a system, not how to build one. Besides successful commercial and industrial applications, such as face detectors in digital cameras and flaw detection in manufacturing, there has also been considerable progress in more generic tasks, such as detecting and localizing instances from multiple generic object classes in ordinary indoor and outdoor scenes, in "fine-grained" classification such as identifying plant and animal species, and in recognizing attributes of objects and activities of people. The results of challenges and competitions (see [4, 5] ) suggest that progress has been spurred by major advances in designing more computationally efficient and invariant image representations [6, 7, 8] ; in stochastic and hierarchical modeling [9, 10, 11, 12] ; in discovering latent structure by training multi-layer networks with large amounts of unsupervised data [13] ; and in parts-based statistical learning and modeling techniques [14, 15, 16] , especially combining discriminative part detectors with simple models of arrangements of parts [17] . Quite recently, sharp improvements in detecting objects and related tasks have been made by training convolutional neural networks with very large amounts of annotated data [18, 19, 20, 21, 22] . More generally, however, machines lag very far behind humans in "understanding images" in the sense of generating rich semantic annotation. For example, systems that attempt to deal with occlusion, context and unanticipated arrangements, all of which are easily handled by people, typically encounter problems. Consequently, there is no point in designing a "competition" between computer vision and human vision: interpreting real scenes (such as the ones in Figure 1 ) is virtually "trivial" (at least effortless and nearly instantaneous) for people whereas building a "description machine" that annotates raw image data remains a fundamental challenge. We seek a quantitative measure of how well a computer vision system can interpret ordinary images of natural scenes. Whereas we focus on urban street scenes, our implementation could easily be extended to other image populations and the basic logic and motivations remain the same. The "score" of our test is based on the responses of a system under evaluation to a series of binary questions about the existence of people and objects, their activities and attributes, and relationships among them, all relative to an image. We have chosen image-based rather than scene-based queries (see Scenes vs. images). Suppose an image sub-population I has been specified ("urban street scenes" in Figure 1 ), together with a "vocabulary" and a corresponding set of binary questions (see Vocabulary and Questions). Our prototype "visual Turing test" (VTT) is illustrated in Figure 2 . Questions are posed sequentially to the computer vision system using In computer vision, as in other fields of AI, the methods of evaluation largely define the scientific effort. Most current evaluations measure detection accuracy, emphasizing the classification of regions according to objects from a pre-defined library. But detection is not the same as understanding. We present here a different evaluation system, in which a query engine prepares a written test ("visual Turing test") that uses binary questions to probe a system's ability to identify attributes and relationships in addition to recognizing objects.
Geman et al. REF proposed a visual Turing test in which the questions are automatically generated and require no natural language processing.
Visual Turing test for computer vision systems
Proceedings of the National Academy of Sciences of the United States of America
Proceedings of the National Academy of Sciences of the United States of America
Computer Science Medicine