MAPW ise: Evaluating Vision-Language Models for Advanced Map Queries (2025)

Abstract: Vision-language models (VLMs) excel at tasks requiring joint understanding of visual and linguistic information. A particularly promising yet under-explored application for these models lies in answering questions based on various kinds of maps. This study investigates the efficacy of VLMs in answering questions based on choropleth maps, which are widely used for data analysis and representation. To facilitate and encourage research in this area, we introduce a novel map-based question-answering benchmark, consisting of maps from three geographical regions (United States, India, China), each containing around 1000 questions. Our benchmark incorporates 43 diverse question templates, requiring nuanced understanding of relative spatial relationships, intricate map features, and complex reasoning. It also includes maps with discrete and continuous values, covering variations in color mapping, category ordering, and stylistic patterns, enabling a comprehensive analysis. We evaluated the performance of multiple VLMs on this benchmark, highlighting gaps in their abilities, and providing insights for improving such models. Our dataset, along with all necessary code scripts, is available at map-wise.github.io.

## introduction vision-language models (vlms) have demonstrated impressive capabilities in tasks requiring joint understanding of visual information and natural language. they have achieved significant success in areas like image generation @xcite , multimodal sentiment analysis @xcite and visual question answering (vqa) @xcite @xcite . however, when applied to map-based question answering, the reasoning abilities of these models remain largely unexplored @xcite . choropleth maps, which use varying shades or colors to represent geographical data, present a unique challenge @xcite . while humans can readily grasp the spatial patterns and information conveyed by these color variations, their interpretation poses a significant challenge for visual language models and other analytical tools. this difficulty arises from the inherent challenge of translating visual data represented by different colors or shades into simpler, tabular formats. this research addresses this gap by analyzing the performance of vlms in answering questions related to choropleth maps representing different geographical regions (example shown in figure 1 ). we aim to answer the following research questions: to address these research questions, we created a novel dataset, mapwise, specifically for mapbased vqa. this dataset comprises 1,000 questions for three geographical regions: the united states, india, and china. the questions were manually created based on 43 unique templates, designed to evaluate model capabilities across topics ranging from data extraction to complex reasoning. furthermore, the dataset includes various map representations, including maps with and without annotations, a diverse range of colormaps, and stylistic patterns like hatching, creating a robust benchmark. we have used this dataset for experimentation across various leading vlms and mmlms, using diverse prompting techniques to establish a viable baseline. our study also included an analysis of model performance on counterfactual maps. these maps featured imaginary state names, jumbled state names, and counterfactual statistics. our analysis aimed to not only understand how well the models relied on the provided map data but also to what extent they relied on their internal knowledge. our contributions are: ## the mapwise dataset this section details the creation process of the mapwise dataset, including data gathering, manual question creation, and dataset validation. ## experimental evaluation this section outlines our experimental setup: we selected a mix of closed-source and open-source vision-language models (vlms) and multimodal large language models (mllms) for a comprehensive analysis. these models were tested with various prompting techniques, and we developed an evaluation metric to assess different answer types. ## results and analysis mapwise: a challenging benchmark. the mapwise dataset presents a compelling benchmark for evaluating the reasoning abilities of current vision-language models (vlms). as shown in ## biases in model prediction this section analyzes the performance variations of models across different map and question variants. these observations are influenced by question type, but we highlight the most prominent insights. ## human evaluation and baseline we conducted a human evaluation of the mapwise dataset to establish a human baseline and benchmark model performance against human evaluators. the dataset presents significant challenges, requiring evaluators to identify subtle shades and patterns while demonstrating understanding of spatial geographical relationships. our evaluation encompassed 450 questions, comprising 150 unique questions uniformly sampled from each of three countries. for each country, the sample included approximately 75 maps and 40 templates. to ensure comprehensive coverage, we maintained an equal distribution across answer types and map categories, incorporating both continuous and discrete maps as well as relative region-type questions. this systematic approach was implemented consistently across all three countries to capture the full range of scenarios within the dataset. for validation, we employed a majority voting system among three independent annotators. as shown in ## experiments with counterfactual data we performed additional analysis to evaluate models which are trained extensively on large datasets, under conditions where their internal factual knowledge was limited. to carry out our analysis, we created three types of counterfactual data that forced the models to rely exclusively on the provided maps. figure 3 shows such counterfactual maps. for the counterfactual dataset generation, we first uniformly sampled a subset of 240 unique questions from usa dataset, spreading over 90 maps and 26 templates. we also ensured approximately equal distribution of each answer type. using the sampled dataset as a representative sample (consisting of original names and values), we applied the following modifications to create our counterfactual dataset: imaginary names. states were assigned imaginary names, generated by gpt-4. (e.g., alabama was renamed aquilis, arkansas became davina, etc.) the first two letters of the imaginary names were used as state codes for the annotated maps. ## conclusion and future work this paper introduces mapwise, a new large-scale dataset tailored for understanding choropleth maps in three diverse countries: the united states, china, and india. looking ahead, there are many promising areas for further research based on what we found and from the existing studies. future studies could broaden the scope of datasets by including different types of maps. inspired by previous work @xcite , we could complement our dataset by exploring fictional maps or more detailed maps that include features such as rivers and roads. this expansion would help evaluate how well vlms generalize across diverse geographical contexts. further research is needed to identify and mitigate biases inherent in map interpretation. techniques like dataset perturbation, which introduces variations in map features and contexts, could provide deeper insights and help mitigate biases effectively. to improve how data is extracted, integrating external knowledge sources in future would be a promising strategy. models that use knowledge graphs, like rag networks filled with detailed information about state borders and regional relationships, could also improve how well vision language models (vlms) reason through map-based tasks. another future direction would be improving how vlms are trained to recognize colors more accurately and integrating additional datasets, training on auxiliary data such as charts, to improve their ability to interpret and process map-related information effectively. future work will also focus on expanding our dataset to overcome its size and diversity limitations. this includes automating question generation using llms and data tables, followed by rigorous human verification to maintain quality. we also plan to increase geographic diversity and incorporate a wider variety of map types beyond choropleth maps.