A lgo P uzzle VQA : Diagnosing Multimodal Reasoning Challenges of Language Models with Algorithmic Multimodal Puzzles (2025)

Abstract: This paper introduces the novel task of multimodal puzzle solving, framed within the context of visual question-answering. We present a new dataset, AlgoPuzzleVQA designed to challenge and evaluate the capabilities of multimodal language models in solving algorithmic puzzles that necessitate both visual understanding, language understanding, and complex algorithmic reasoning. We create the puzzles to encompass a diverse array of mathematical and algorithmic topics such as boolean logic, combinatorics, graph theory, optimization, search, etc., aiming to evaluate the gap between visual data interpretation and algorithmic problem-solving skills. The dataset is generated automatically from code authored by humans. All our puzzles have exact solutions that can be found from the algorithm without tedious human calculations. It ensures that our dataset can be scaled up arbitrarily in terms of reasoning complexity and dataset size. Our investigation reveals that multimodal language models such as GPT4V and Gemini exhibit limited performance in puzzle-solving tasks. We find that their performance is near random in a multi-choice question-answering setup for a significant number of puzzles. The findings emphasize the challenges of integrating visual, language, and algorithmic knowledge for solving complex reasoning problems.

## introduction puzzles have long been a source of fascination and intellectual challenge, serving both as entertaining pastimes and as means to advance mathematical and logical understanding. historically, puzzles have often emerged from complex problems, with their solutions contributing valuable insights to mathematical research @xcite @xcite * equal contribution. @xcite . in this paper, we explore the novel task of algorithmic visual puzzle solving. traditional visual question-answering (vqa) @xcite and visual reasoning (hudson and manning, 2019) datasets have mainly focused on combining language and vision through the lens of object detection, scene recognition, and spatial relationship, where the answer can be directly found in the image. another line of work has explored the application of external factual knowledge @xcite or commonsense world knowledge @xcite for vqa. recently, multimodal reasoning benchmarks such as scienceqa @xcite , and mmmu @xcite have been proposed that aim to evaluate the expert knowledge of llms through subject-specific questions spanning science, medicine, business, arts, etc. unlike previous research, our work integrates a new critical component -algorithmic reasoning, which demands the ability to apply logical rules to novel situations, rather than relying on pre-stored information or direct recall of facts. this distinction is vital as it shifts the focus from merely memorizing vast databases of knowledge to applying concrete reasoning steps for novel problem-solving. we thus propose to benchmark the problem-solving ability of ai models through visual puzzles. our puzzles are created to be selfcontained where minimal additional knowledge (beyond basic mathematics) is required to solve them. this helps us to disentangle the knowing part from the solving part, as the knowledge required to solve the problem is already provided as context. consider figure 1 as an example of some of our puzzles. we show the setting or the configuration of the puzzle/problem as an image, which we consider as the visual context. depending on the problem, the visual context may show different kinds of information such as the initial and final ## the proposed ontology our puzzles encompass a diverse range of visual, mathematical and algorithmic topics. we categorize the visual and algorithmic features present in each puzzle in ## from the ontology to creating algopuzzlevqa we create a total of 18 different puzzles for our dataset following various feature combinations from our ontology ( generating puzzle images and curating solutions. instances for each puzzle are created auto-matically from the human-written code of the particular puzzle which includes: creating language context. finally, we also carefully curate the language context (in english) for each puzzle using puzzle-specific templates. we make sure that the template describes all the necessary information and rules required for solving the puzzle. the templates were written by two authors of the paper and were verified by the other authors. extensibility. as mentioned earlier, the number of instances and the difficulty level of the puzzles can be scaled arbitrarily to create a large and challenging dataset. on the other hand, most existing benchmarks such as smart-101 @xcite rely on human annotation which is costly and difficult to scale. for now, we have created 100 instances for each puzzle. these instances are analogous to different test cases of the puzzle, i.e. they have different input combinations, initial and goal states, etc. reliably solving all the instances would require finding the exact algorithm to use and then applying it accurately. this is akin to how we verify the accuracy of a computer program aiming to solve a particular task through a broad range of test cases. ## conclusion in this study, we introduce algopuzzlevqa, a novel dataset comprising puzzles that demand multimodal reasoning over vision, language and algorithms. unlike existing multimodal reasoning datasets such as scienceqa or mmmu, the challenges in algopuzzlevqa do not hinge on possessing domain-specific knowledge for reasoning. instead, solving the problems in this dataset necessitates applying logical algorithmic steps to novel problem scenarios. to construct this dataset, we devised an ontology encompassing i) visual reasoning features such as colours, positions, shapes and sizes of puzzle components; and ii) algorithmic reasoning features such as arithmetic, combinatorics, optimization, etc. each puzzle in our dataset comprises multiple categories from the visual and algorithmic ontology. through rigorous experiments with llms such as gpt-4v, gemini-pro, claude 3, instructblip, and llava, we observed a consistent struggle in achieving satisfactory performance on algop-uzzlevqa, emphasizing the formidable hurdles in multimodal reasoning for algorithmic puzzlesolving. given that each puzzle is annotated with specific visual and algorithmic features based on our ontology, we draw insights into the precise skills lacking in llms to address particular problems. we also analyzed whether the models falter on this dataset due to deficiencies in the visual recognition setup or the algorithmic reasoning setup. while furnishing the models with accurate visual guidance of the puzzles enhances performance in some cases, achieving high-level performance in our proposed complex reasoning tasks remains an elusive goal.