Geo-Aware Image Caption Generation (2020)

Abstract: Standard image caption generation systems produce generic descriptions of images and do not utilize any contextual information or world knowledge. In particular, they are unable to generate captions that contain references to the geographic context of an image, for example, the location where a photograph is taken or relevant geographic objects around an image location. In this paper, we develop a geo-aware image caption generation system, which incorporates geographic contextual information into a standard image captioning pipeline. We propose a way to build an image-specific representation of the geographic context and adapt the caption generation network to produce appropriate geographic names in the image descriptions. We evaluate our system on a novel captioning dataset that contains contextualized captions and geographic metadata and achieve substantial improvements in BLEU, ROUGE, METEOR and CIDEr scores. We also introduce a new metric to assess generated geographic references directly and empirically demonstrate our system’s ability to produce captions with relevant and factually accurate geographic referencing.

## introduction image caption generation is a popular task that aims at producing a natural language description of a given image. a standard neural image captioning system consists of two stages: an "encoder", a convolutional neural network that encodes the visual features of an image as a vector, and a "decoder", a language model that is initialized with this vector and generates a caption word by word. recent research @xcite @xcite has drawn attention to the fact that the standard approach is insufficient to imitate captions naturally produced by humans. people tend to describe images interpreting them based on context factors and world knowledge, while standard encoder-decoder captioning systems do not take any contextual or world knowledge into account. one of the aspects that are missing from standard caption generation systems is the ability to produce image descriptions influenced by the geographic context, i.e. geographic objects surrounding the image location. for example, consider the photograph in figure 1 : automatically generated: a park bench sitting in the middle of a park the person who took this photograph foot_0 captions it referring to the location of the photograph ("pitshanger park") and the most relevant parts of the surroundings ("ealing", "west london suburbs"). however, a neural caption generation system @xcite merely describes the objects in the image ("park bench", "park"). in this paper we present geo-aware image captioning, where geographic contextual information is incorporated into the generated captions. the contributions of this paper are as follows: experiments on the georic dataset demonstrate the ability of our geo-aware caption generation system to produce image descriptions that include meaningful and contextually relevant geographic information and show considerable improvements in several image captioning metrics (bleu, rouge, meteor, cider). ## the georic dataset a dataset for geo-aware image captioning has to contain not only images with captions but also the geographic information related to the image locations. since the generally used image captioning datasets (mscoco @xcite , flickr8k @xcite , flickr30k @xcite ) do not include any geographic metadata, we compiled our own dataset -the georic dataset (geo-aware rocky image captioning). we gathered data from geograph, an on-going project that aims to collect photographs of every square kilometer in great britain and ireland. the project's website foot_2 currently stores more than 6 million images with naturally produced captions. the website also provides rich metadata for every image including the latitude and longitude coordinates of the image location. our georic dataset consists of 29,038 images from the geograph project website, with captions and location coordinates. we selected captions that are exactly one sentence long (multi-sentence caption generation, although a promising research direction @xcite , is not addressed in this work) and include at least one spatial expression, such as "near", "north of", "across", etc. (in order to ensure that the captions contain enough geographic referencing). an example entry in the dataset is shown in ## geographic context we take a geographic context of an image to be a set of relevant geographic objects around the image location. this geographic context is used for compiling an image-specific vocabulary of geographic names as well as for complementing the image representation used by the text generation network (for which it needs to be additionally compressed into a single vector). in this section, we describe the process of constructing, encoding and compressing the geographic context. ## geo-aware caption generation our geo-aware caption generation system is based on the show, attend and tell caption generator @xcite , which is structured as an encoder-decoder pipeline with an added visual attention component. we introduce the geographic contextual information at both the encoder and the decoder stage. figure 2 shows the overall architecture of our system. concrete implementation details (vector sizes, learning rates, etc.) are provided in appendix a. image vector (51.22625, -0.51698) geographic database ## results and discussion in this section we first compare the performance of our geo-aware image captioning system with that of a non-geographic one, the show, attend and tell system, trained on the same georic dataset (which we will refer to as the "standard" system). second, we analyze the quality of the generated geographic references with a newly developed metric, which aims to directly assess the correctness of the spatial expressions usage instead of comparing the generated geographic references with the ground truth ones. ## related work most modern image captioning systems use the standard encoder-decoder approach with a cnn encoding the visual features of the image and an rnn, usually an lstm, decoding the image features into a textual description @xcite @xcite . much effort is directed at improving the components of the standard caption generation pipeline @xcite @xcite @xcite ; however, the standard approach itself has several shortcomings, such as producing captions that are too generic @xcite , inability to generate words that are not present in the training data @xcite and disregarding pragmatic factors and world knowledge @xcite . there have been a few attempts to create a context-aware, less generic caption generation system. @xcite and @xcite proposed generating image descriptions as templates with blank slots and filling them afterwards with named entities, including locations and dates. this template-based method might be problematic in case some of the generated blank slots cannot be filled based on the available data. closer to our work, @xcite used a more flexible approach 7 to enrich video descriptions with event and entity names that are extracted from related news articles. the authors also introduced a "knowledge gate vector", the purpose of which is to provide the description generation network with information about the types of events and entities mentioned in the article. similarly to @xcite , we use an external source (a geographic database) to create a representation of the context and adapt the text generator to produce out-of-vocabulary context-related words. our geographic context representation not only registers the presence of certain entity types but also incorporates various features of the relevant entities. to the best of our knowledge, none of the context-aware caption generation systems have had a specific focus on the geographic context of the image location. on the other hand, geographic information in particular has been used as an additional input to the systems developed for other tasks, such as image classification @xcite and creating distributional word embeddings @xcite . both @xcite and @xcite reported increased performance in the image classification task after providing their systems with geographic knowledge. however, the content of the geographic knowledge provided to their systems was very different. @xcite used only the normalized latitude and longitude coordinates of the image location. opposite to that, @xcite supplied to their system all the geological, ecological, sociological, demographic and other types of data that could be obtained from the external sources. our approach to constructing the geographic context strikes a balance between the two: we include explicit information about the most relevant geographic objects around the image location and ensure that this information is only what is most useful for the captioning process. cocos and callison-burch (2017) used geographic information for creating contextualized word embeddings, specifically information about the types of the objects around the word usage location. the authors assigned the same weight to the objects within the same radius from the location, however, arguably, such objects can still be more or less significant depending on their size and relative distance to the location. in our system, we use these characteristics of the objects to construct the maximally relevant geographic context. overall, our system is a novel combination of the two trends that have emerged in the recent research: the context-aware image caption generation and the usage of geographic information to enhance the performance of various nlp and vision and language models. ## conclusions in this paper, we have presented an approach to incorporate geographic contextual information into the caption generation pipeline. to the best of our knowledge, this is the first approach that focuses on generating geographically grounded image descriptions. in addition, we have introduced the georic dataset for image captioning, which includes contextualized captions and geographic metadata. our experiments on the georic dataset showed the effectiveness of our geo-aware captioning system and its advantage over a standard captioning network, with large improvements in several standardly used evaluation metrics, as well as a good performance on a metric designed specifically for geographic expressions. our system is thus able to produce contextualized captions that include correct geographic referencing without compromising the overall quality of the image description. in future work, we plan to refine the process of selecting the most appropriate geographic object in a given context and to experiment with including other types of relevant contextual information, as well as with other vision and language systems (such as visual question answering) that could benefit from geographic grounding.