MMC ode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems (2024)

Abstract: Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available.

## introduction programming is primarily aimed at fulfilling requirements, frequently entailing the translation of detailed and intricate specifications into executable code @xcite . in this endeavor, human developers regularly employ visual aids such as images and diagrams to facilitate effective communication and a better understanding of concepts @xcite . recently, automated code generation tools have attracted significant attention, largely attributing to the substantial advance in code large language models (code llms) @xcite @xcite @xcite . these models demonstrated unprecedentedly remarkable coding abilities, potentially assist to enhance productivity, reduce human error and democratize coding skills. nevertheless, these models are limited to processing text-only inputs, lacking the ability to interpret rich information presented through images. in a closely related development, the field has also observed the emergence of many powerful large multimodal models (lmms), marked by gpt-4v (openai, 2023b) and gemini @xcite , representing a significant step forward in bridging the modality of text and images. while there are multiple works evaluating these models in mathematical reasoning @xcite , perception and reasoning @xcite and instruction-following @xcite , there is a notable gap in evaluating lmms for code generation. to this end, we present mmcode, the first multimodal benchmark for rigorously evaluating the code generation ability of large multimodal models. it comprises 3,548 questions with 6,620 images collated from 10 programming-related websites encompassing a broad spectrum of subjects, extending from fundamental coding concepts to the application of code for solving mathematical problems. the generated code is rigorously checked by test cases. the overall framework is illustrated in figure 1 . our experiments revealed that current lmms struggle significantly to solve the tasks in mm-code. the most powerful lmms, gpt-4v and gemini, scored unsatisfactory pass rates as low as 19.4% and 5.0%, potentially due to the requirement of intense reasoning on the text descriptions and images. open-source lmms @xcite ) yield negligible pass rates because of their inability to understand the abstract meaning of the images. the findings reveal a significant deficiency in current lmms' ability to interpret and utilize multimodal information for code generation, highlighting an imperative need for further advancements in this area. we believe mmcode will serve as a pivotal benchmark for evaluating the forthcoming evolution of code lmms and inspire research in this area. ## mmcode in this section, we introduce the source and collection pipeline of mmcode. the collection pipeline comprises four stages: 1) raw data collection; 2) automatic filtering; 3) human filtering and 4) annotation. this pipeline to be introduced in the following sections guarantees the quality and diversity of the data collected for mmcode. ## data analysis in this section, we undertake a comprehensive exploration of mmcode, introducing its and statistical attributes to provide a nuanced understanding of mmcode. problem length. the diversity of data sources incorporated into mmcode results in significant variance in problem length, as can be seen in figure 2a . the mean length of the questions reaches 2,256 characters, with the 25th, 50th, and 75th percentile 5 model task type average linear tree graph 2d 3d chessboard map math patterns at 1,516, 2,127, and 2,791. this can be ascribed to the distinct style and difficulty of the questions presented in mmcode. certain questions articulate the instructions succinctly and directly, whereas others elaborate on the contextual background of the problem in detail. ## experiments in this section, we benchmark several language-only models and vision-language models with mmcode. a comparative analysis of the experimental results for these models is conducted, providing a thorough examination of their capabilities. ## conclusion in this paper, we present mmcode, the first multimodal coding dataset for evaluating algorithmic problem-solving skills in image-text interwoven contexts. we benchmarked a range of state-ofthe-art llms and lmms on mmcode and provide a detailed analysis. despite their advanced capabilities, these models demonstrate a significant challenge in leveraging visual contexts for code generation. we believe that mmcode will catalyze further research and innovation, paving the way for the creation of ai systems capable of handling sophisticated visual and textual reasoning in programming and beyond.