There’s a Time and Place for Reasoning Beyond the Image (2022)

Abstract: Images are often more significant than only the pixels to human eyes, as we can infer, associate, and reason with contextual information from other sources to establish a more complete picture. For example, in Figure 1, we can find a way to identify the news articles related to the picture through segment-wise understandings of the signs, the buildings, the crowds, and more. This reasoning could provide the time and place the image was taken, which will help us in subsequent tasks, such as automatic storyline construction, correction of image source in intended effect photographs, and upper-stream processing such as image clustering for certain location or time. In this work, we formulate this problem and introduce TARA: a dataset with 16k images with their associated news, time, and location, automatically extracted from New York Times, and an additional 61k examples as distant supervision from WIT. On top of the extractions, we present a crowdsourced subset in which we believe it is possible to find the images’ spatio-temporal information for evaluation purpose. We show that there exists a 70% gap between a state-of-the-art joint model and human performance, which is slightly filled by our proposed model that uses segment-wise reasoning, motivating higher-level vision-language joint models that can conduct open-ended reasoning with world knowledge. The data and code are publicly available at https://github.com/zeyofu/TARA .

## introduction vision and language are two of most important information sources, and the fact that humans reason jointly with both sources has motivated artificial intelligence research to consider visually-grounded language understanding. most work in this area has focused on reasoning with local evidence @xcite @xcite , e.g. asking factoid questions such as the colors or shapes of objects and numbers of people, yet very few works @xcite encourage open-ended reasoning where a model needs to look beyond task inputs. however, humans can relate visual cues to corresponding contextual information that could be multi-modal, and draw on background knowledge when interpreting and grounding images. for example, as figure 1 shows, people that are familiar with the news can infer that the location is times square through the iconic screen panels, and further estimate the period of time by looking at the crowds and the signs. and, this can be done without explicitly including related news pieces as input. in fact, even though some people would not have the prior knowledge to identify the relevant events, it is likely that they would have good estimate of the location and time by interpreting textual evidence in the image, the language in the signs, entity names, building styles, and other details in the input image. in this work, we identify and formulate this problem, spatio-temporal grounding of images, a ਮੋ ਦੀ ਜੀ, ਸਾਡੀ ਵੈ ਕਸੀਨ ਿਵਦੇ ਸ਼ ਿਕਉ 1 ਭੇ ਜ ਿਦੱ ਤੀ? nsui punjab indian police costume a tuk-tuk or auto rickshaw in india location and time evidence modern street bike modern short sleeve clothing face covering ⋯ ocr tools scene text languages: punjabi and english mr. modi, why did you send our vaccine abroad? objects faces keyword: punjab narendra modi -prime minister of india time is may, 2021. ## dataset collection each example in tara includes a news image, along with its time, location, caption, and corresponding news background such as headline, abstract, and news type. these are included for training or analysis purposes, but the task is to guess the correct time and location as accurately as possible given only the image. in developing the dataset, our goal is to collect a large corpus of semantically rich images that human with world knowledge can correctly identify the time and location, using evidence from the image, background knowledge, and appealing to external knowledge (which we call "reasoning" here). we design the process of collecting and identifying the images so that it facilitates this type of reasoning, and then use crowd sourcing to label a random 20% of high-quality images for development and testing. figure 4 illustrates our data collection procedure. ## baselines we assess the quality of our dataset through human annotation, and evaluate on existing visual reasoning approaches. ## conclusion in this work, we introduce tara, a new dataset and task for spatio-temporal grounding of images that requires open-ended joint reasoning with world knowledge. tara provides a dataset of 16k highquality images from nyt and wikipedia-based supervision for additional 61k images. compared to previous visual-language understanding datasets, tara requires more complicated reasoning ability and existing state-of-the-art models such as clip are far from human levels, suggesting that our task remains a significant challenge with large room for improvement. we hope that tara will inspire future work on reasoning beyond image's local segments in vision-language understanding.