G3: Geolocation via Guidebook Grounding (2022)

Abstract: We demonstrate how language can improve geolocation: the task of predicting the location where an image was taken. Here we study explicit knowledge from human-written guidebooks that describe the salient and class-discriminative visual features humans use for geolocation. We propose the task of Geolocation via Guidebook Grounding that uses a dataset of StreetView images from a diverse set of locations and an associated textual guidebook for GeoGuessr, a popular interactive geolocation game. Our approach predicts a country for each image by attending over the clues automatically extracted from the guidebook. Supervising attention with country-level pseudo labels achieves the best performance. Our approach substantially outperforms a state-of-the-art image-only geolocation method, with an improvement of over 5% in Top-1 accuracy. Our dataset and code can be found at https://github.com/g-luo/geolocation_via_guidebook_grounding.

## introduction image geolocation plays an important role in many applications, notably in fact checking and investigative journalism, as a means of verifying or debunking claims that are illustrated with images. for example, to verify the authenticity of video evidence of human rights abuses in cameroon, an investigator manually matched the depicted roof coloring, building architecture, and sign text to streetview imagery. 1 given the rising volume of online mis-and disinformation, it is increasingly important to develop accurate automated geolocation methods. beyond the recent importance of geolocation in fact-checking, it has also become a popular pastime in the form of online games like sweden often has white dashes on the sides of its roads. these dashes are different from the norwegian dashes as they are shorter than the space between the dashes. a key point to remember in geoguessr is that if you see a red house or several red houses then you are probably in a nordic country (sweden, norway, finland, iceland or denmark). ## geolocation via guidebook grounding while prior works @xcite @xcite propose datasets of paired images and locations, to the best of our knowledge, none have proposed datasets with images, locations, and relevant textual knowledge. to this end, we look to guidebooks for the interactive geolocation game geoguessr foot_3 as a source of knowledge for the specific domain of streetview images. for our task of geolocation via guidebook grounding we put together a new diverse corpus of streetview images and an associated text from a geoguessr guidebook (see examples in figure 2 ). guidebook text: geoguessr is a popular geolocation game where the user is placed into a navigable streetview scene and must predict either the gps coordinates or the country (depending on the gameplay mode). the game is so popular that people have built a community of online forums and guidebooks where they trade strategies for gameplay. to take advantage of this wealth of human knowledge, we mine text from a popular guide for playing the geoguessr game. 5 some of the key features of this guide are that it has a broad country coverage, is well structured, and has a more "formal" style. the guide is meant to teach novice players visual cues that discriminate various countries, for example the fact that dashed white lines on the edges of roads are quite common in the countries of denmark, norway, iceland and sweden. the guide is organized into sections for specific countries and specific visual cues, covering 102 countries and over 13 visual cue types (see figure 3 ). for our final knowledge base, we select sentences from the guide that mention at least one location. we use ner tags predicted by spacy @xcite and filter for sentences with an entity tagged "gpe", "loc", or "norp", resulting in 3,832 sentences, with an average length of 14 words. the clue sentences contain a total of 3,712 unique words and 3,182 unique lemmas. ## approach our multimodal approach, g 3 , improves upon traditional image-only methods for geolocation with the help of guidebook text. we concatenate the visual representation of an image-only geolocation model with a relevant textual clue representation derived from our guidebook, and use it in a linear country classification layer, see figure 6 . textual clue representation: given an input image to geolocate, we use a soft attention layer to compute attention scores over each sentence (clue) extracted from our guidebook to indicate its relevance. we opt to use the location-based attention mechanism of @xcite as other attention mechanisms would take significantly more memory and time to train for a guidebook the size of ours. our soft attention layer takes an image query d, encodes it with a frozen clip rn50x16 image encoder f clip , and passes it through a fully connected layer with a relu activation to obtain attention logits for each clue in our guidebook: we precompute clue representations by applying a frozen roberta base model @xcite to each clue in our guidebook. 6 we then take the weighted average of these clue representations to obtain ĝ, an image-specific summary textual clue representation: guiding attention with pseudo labels: given the large number of clues in our dataset, we use weak supervision to guide the attention mechanism during training. we create pseudo labels associating clues to images using country information. specifically, we geoparse the clues via country demonyms and lexical matching, 7 mapping the named entities (as predicted by spacy, see section 3) in the clues to country labels, and therefore associated images. for example, a clue that mentions "japanese" would be associated with images where japan is the ground truth label. on average each image is matched to 76 country-relevant clues (2% of all guidebook clues). we then supervise the attention mechanism over clues with these pseudo labels. we add a binary cross entropy loss on our attention logits, where the label is a one-hot vector for each clue in our guidebook, defined by whether the clue mentions the country of the input image. our final loss function is as follows: ## experiments baseline models: for our baseline models we use isn, a unimodal model consisting of a visual encoder and linear classification layer trained to predict a hierarchical cell on earth given an image @xcite , and clip, a multimodal model consisting of a visual and text encoder trained to maximize the cosine similarity of matched image-text pairs via contrastive learning @xcite . we adapt the isn resnet50 model pretrained on millions of flickr images @xcite by modifying the output size of the final classification layer from the number of cells on earth to the number of countries in our dataset and further fine-tune the model on our streetview images. while clip was not trained specifically for the geolocation task, @xcite demonstrate reasonable performance on a number of geolocation benchmarks such as countries211 @xcite and im2gps @xcite via nearest neighbors regression, zero-shot prediction, and/or linear probing. we adapt the clip rn50x16 model pretrained on a large-scale web image-text dataset @xcite by taking a frozen representation from its visual encoder and feeding it to a linear classification layer (denoted by 'clip linear probe'). we also include the nearest neighbor version (clip nearest neighbor). ## analysis here we discuss qualitative success and failure cases of g 3 and the top clues attended over as well as comparisons with baseline image-only methods. ## conclusion we presented geolocation via guidebook grounding, a new multimodal task that includes a geodiverse dataset of images from streetview and text from a guidebook for the geoguessr game. we demonstrated that adding clues from guidebooks via our approach, g 3 , substantially outperforms past state-of-the-art image-only geolocation models on our task, with an absolute improvement of 5% to reach 70% in top-1 country classification. at the same time, there is still significant room left for improvement, and we hope that other practitioners will find this task interesting and relevant. the unique feature of our dataset is that the guidebook contains references to many nuanced and detailed characteristics of the visual scenes, posing challenging grounding problems. from recognizing specific road markings and reasoning about the scene geography, to recognizing flags, languages and other symbols, there are many skills necessary to solve the task. the focused domain of street scenes enables us to study this problem, while still being conceptually rich and capturing a multitude of diverse geographic locations. acknowledgements. this work was supported in part by dod including darpa's semafor, ptg and/or lwll programs, as well as bair's industrial alliance programs.