In this paper , we aim to understand whether current language and vision ( LaVi ) models truly grasp the interaction between the two modalities . To this end , we propose an extension of the MS-COCO dataset , FOIL-COCO , which associates images with both correct and ' foil ' captions , that is , descriptions of the image that are highly similar to the original ones , but contain one single mistake ( ' foil word ' ) . We show that current LaVi models fall into the traps of this data and perform badly on three tasks : a ) caption classification ( correct vs. foil ) ; b ) foil word detection ; c ) foil word correction . Humans , in contrast , have near-perfect performance on those tasks . We demonstrate that merely utilising language cues is not enough to model FOIL-COCO and that it challenges the state-of-the-art by requiring a fine-grained understanding of the relation between text and image . Most human language understanding is grounded in perception . There is thus growing interest in combining information from language and vision in the NLP and AI communities . So far , the primary testbeds of Language and Vision ( LaVi ) models have been ' Visual Question Answering ' ( VQA ) ( e.g. Antol et al . ( 2015 ) ; Malinowski and Fritz ( 2014 ) ; Malinowski et al . ( 2015 ) ; Gao et al . ( 2015 ) ; Ren et al . ( 2015 ) ) and ' Image Captioning ' ( IC ) ( e.g. Hodosh et al . ( 2013 ) ; Fang et al . ( 2015 ) ; Chen and Lawrence Zitnick ( 2015 ) ; Donahue et al . ( 2015 ) ; Karpathy and Fei-Fei ( 2015 ) ; Vinyals et al . ( 2015 ) ) . Whilst some models have seemed extremely successful on those tasks , it remains unclear how the reported results should be interpreted and what those Figure 1 : Is the caption correct or foil ( T1 ) ? If it is foil , where is the mistake ( T2 ) and which is the word to correct the foil one ( T3 ) ? models are actually learning . There is an emerging feeling in the community that the VQA task should be revisited , especially as many current dataset can be handled by ' blind ' models which use language input only , or by simple concatenation of language and vision features ( Agrawal et al . , 2016 ; Jabri et al . , 2016 ; Zhang et al . , 2016 ; Goyal et al . , 2016a ) . In IC too , Hodosh and Hockenmaier ( 2016 ) showed that , contrarily to what prior research had suggested , the task is far from been solved , since IC models are not able to distinguish between a correct and incorrect caption . Such results indicate that in current datasets , language provides priors that make LaVi models successful without truly understanding and integrating language and vision . But problems do not stop at biases . Johnson et al . ( 2016 ) also point out that current data ' conflate multiple sources of error , making it hard to pinpoint model weaknesses ' , thus highlighting the need for diagnostic datasets . Thirdly , existing IC evaluation metrics are sensitive to n-gram overlap and there is a need for measures that better simulate human judgments ( Hodosh et al . , 2013 ; Elliott and Keller , 2014 ; Anderson et al . , 2016 ) . Our paper tackles the identified issues by proposing an automatic method for creating a large dataset of real images with minimal language bias and some diagnostic abilities . Our dataset , FOIL ( Find One mismatch between Image and Language caption),1 consists of images associated with incorrect captions . The captions are produced by introducing one single error ( or ' foil ' ) per caption in existing , human-annotated data ( Figure 1 ) . This process results in a challenging error-detection / correction setting ( because the caption is ' nearly ' correct ) . It also provides us with a ground truth ( we know where the error is ) that can be used to objectively measure the performance of current models . We propose three tasks based on widely accepted evaluation measures : we test the ability of the system to a ) compute whether a caption is compatible with the image ( T1 ) ; b ) when it is incompatible , highlight the mismatch in the caption ( T2 ) ; c ) correct the mistake by replacing the foil word ( T3 ) . The dataset presented in this paper ( Section 3 ) is built on top of MS-COCO ( Lin et al . , 2014 ) , and contains 297,268 datapoints and 97,847 images . We will refer to it as FOIL-COCO . We evaluate two state-of-the-art VQA models : the popular one by Antol et al . ( 2015 ) , and the attention-based model by Lu et al . ( 2016 ) , and one popular IC model by ( Wang et al . , 2016 ) . We show that those models perform close to chance level , while humans can perform the tasks accurately ( Section 4 ) . Section 5 provides an analysis of our results , allowing us to diagnose three failures of LaVi models . First , their coarse representations of language and visual input do not encode suitably structured information to spot mismatches between an utterance and the corresponding scene ( tested by T1 ) . Second , their language representation is not finegrained enough to identify the part of an utterance that causes a mismatch with the image as it is ( T2 ) . Third , their visual representation is also too poor to spot and name the visual area that corresponds to a captioning error ( T3 ) . We have introduced FOIL-COCO , a large dataset of images associated with both correct and foil captions . The error production is automatically generated , but carefully thought out , making the task of spotting foils particularly challenging . By associating the dataset with a series of tasks , we allow for diagnosing various failures of current LaVi systems , from their coarse understanding of the correspondence between text and vision to their grasp of language and image structure . Our hypothesis is that systems which , like humans , deeply integrate the language and vision modalities , should spot foil captions quite easily . The SoA LaVi models we have tested fall through that test , implying that they fail to integrate the two modalities . To complete the analysis of these results , we plan to carry out a further task , namely ask the system to detect in the image the area that produces the mismatch with the foil word ( the red box around the bird in Figure 1 . ) This extra step would allow us to fully diagnose the failure of the tested systems and confirm what is implicit in our results from task 3 : that the algorithms are unable to map particular elements of the text to their visual counterparts . We note that the addition of this extra step will move this work closer to the textual / visual explanation research ( e.g. , ( Park et al . , 2016 ; Selvaraju et al . , 2016 ) ) . We will then have a pipeline able to not only test whether a mistake can be detected , but also whether the system can explain its decision : ' the wrong word is dog because the cyclists are in fact approaching a bird , there , in the image ' . LaVi models are a great success of recent research , and we are impressed by the amount of ideas , data and models produced in this stimulating area . With our work , we would like to push the community to think of ways that models can better merge language and vision modalites , instead of merely using one to supplement the other .

Challenge: Despite the success of language and vision models on visual question answering tasks, what these models are learning remains unknown because of coarse-grained datasets.
Approach: They propose to automatically inject one mistake to captions in the MS-COCO dataset as a foil word and three diagnostic tasks to study models' behaviors.
Outcome: Using the introduced dataset, they find that best performing models fail on the proposed tasks indicating their abilities to integrate two modalities.