G eo C hain: Multimodal Chain-of-Thought for Geographic Reasoning (2025)

Abstract: This paper introduces GeoChain, a large-scale benchmark for evaluating step-by-step geographic reasoning in multimodal large language models (MLLMs). Leveraging 1.46 million Mapillary street-level images, GeoChain pairs each image with a 21-step chain-of-thought (CoT) question sequence (over 30 million Q&A pairs). These sequences guide models from coarse attributes to fine-grained localization across four reasoning categories - visual, spatial, cultural, and precise geolocation - annotated by difficulty. Images are also enriched with semantic segmentation (150 classes) and a visual locatability score. Our benchmarking of frontier MLLMs on a diverse 2,088-image subset reveals consistent challenges: models frequently exhibit weaknesses in visual grounding, display erratic reasoning, and struggle to achieve accurate localization, especially as the reasoning complexity escalates. GeoChain offers a robust diagnostic methodology, critical for fostering significant advancements in complex geographic reasoning within MLLMs.

## introduction as large vision-language models (vlms) continue to make rapid progress on general visual question answering and captioning tasks @xcite @xcite , their capacity for structured geographic reasoning remains underexplored. the ability to infer a location from visual cues -such as terrain, signage, vehicles, or architecture -considered alongside spatial and cultural knowledge, is crucial for real-world applications like remote sensing, disaster response, and autonomous navigation. more broadly, geographic localization serves as a testbed * equal contribution. for grounded intelligence, requiring models to reason over subtle visual features, incorporate world knowledge, and disambiguate locations that may be visually similar. despite this, existing benchmarks rarely probe the kind of step-by-step reasoning that such tasks demand. we introduce geochain, a novel multimodal benchmark for evaluating structured geographic reasoning in large language models (mllms). as depicted in figure 1 , each geochain sample features a street-level image from the mapillary dataset @xcite paired with a 21-step chain-of-thought (cot) question sequence. these sequences progressively guide models from coarse inferences, such as hemisphere or continent, to finegrained predictions like city, latitude, and longitude. the complete geochain framework comprises 1.46 million images, each with this 21-question cot structure, yielding over 30 million question-answer pairs. questions span four core reasoning categories: visual cues, spatial localization, cultural inference, and precise geolocation, all annotated with difficulty levels for granular evaluation. this curriculum-style structure offers vital diagnostic insights into where and why models fail across reasoning stages, moving beyond sole reliance on final predictions. to facilitate focused evaluations, we curated geochain test-mini, a diverse and challenging subset. this curation process leverages a locatability score, adapted from georeasoner @xcite and computed using features from a pretrained maskformer model @xcite . this score quantifies the visual identifiability of a location from a single image, allowing us to stratify geochain test-mini into easy, medium, and hard tiers based on thresholds in the 0.12-0.6 range. the resulting geochain test-mini contains 2,088 carefully selected images, designed to offer a representative yet manageable scale for robust mllm assessment. our main contributions are as follows: ## geochain benchmark construction the geochain benchmark is constructed by augmenting the mapillary street-level sequences (msls) dataset @xcite . msls provides a diverse collection of geo-tagged streetlevel imagery (approximately 1.4 million images in its full extent, with a geographical distribution across numerous cities as illustrated in figure 2 ), crucial for developing and evaluating geographic localization models. however, to facilitate finegrained, step-by-step reasoning, we introduce several layers of annotation and metadata. our contributions enhance the msls dataset in three primary ways: semantic class labeling, locatability score computation, and the design of a structured chainof-thought question battery. these augmentations, followed by a careful test set curation process, collectively enable a more nuanced evaluation of multimodal models' geographic reasoning capabilities. ## analysis in this section, we evaluate the performance of frontier vision-language models: gpt-4.1, gpt-4.1-mini (openai et al., 2024), claude 3.7 sonnet (sonnet, 2025), gemini 2.5 flash (google, 2025a), gemini 2.5 pro (google, 2025b), qwen 2.5 vl @xcite , pixtral large (mistral ai, 2024), and gemma 3 @xcite on the geochain "test-mini" benchmark, focusing on their ability to reason accurately and consistently across a structured 21-step geographic reasoning chain. ## conclusion this paper introduced geochain, a large-scale, chain-of-thought benchmark designed to dissect multimodal geographic reasoning in mllms using street-level imagery and a 21-step diagnostic framework. our evaluations on the curated geochain test-mini subset reveal that even leading mllms exhibit significant deficiencies in visual grounding, reasoning consistency, and localization accuracy, particularly as task and visual complexity escalate. by enabling a granular, step-by-step analysis, geochain moves beyond simple end-task accuracy to pinpoint these critical failure modes, thereby providing an essential diagnostic resource and methodology. we anticipate that geochain will steer future research towards developing more robust, geographically aware, and reliable ai systems capable of nuanced real-world understanding.