C ode V : Issue Resolving with Visual Data (2025)

Abstract: Large Language Models (LLMs) have advanced rapidly in recent years, with their applications in software engineering expanding to more complex repository-level tasks. GitHub issue resolving is a key challenge among these tasks. While recent approaches have made progress on this task, they focus on textual data within issues, neglecting visual data. However, this visual data is crucial for resolving issues as it conveys additional knowledge that text alone cannot. We propose CodeV, the first approach to leveraging visual data to enhance the issue-resolving capabilities of LLMs. CodeV resolves each issue by following a two-phase process: data processing and patch generation. To evaluate CodeV, we construct a benchmark for visual issue resolving, namely Visual SWE-bench. Through extensive experiments, we demonstrate the effectiveness of CodeV, as well as provide valuable insights into leveraging visual data to resolve GitHub issues.

## introduction large language models (llms) have advanced rapidly in recent years, with their applications in the field of software engineering becoming increasingly widespread @xcite @xcite . currently, llms' applications in software engineering have gradually expanded tasks at the code line and function level to more challenging repository-level tasks @xcite . within repository-level tasks, github issue resolving is a key challenge, where llms are tasked to resolve the issue based on the issue description and the defective codebase. @xcite . this task can accelerate program repair and is crucial for improving development efficiency. although recent approaches have made progress on this task, they focus exclusively on textual data figure 1 : an example of a visual github issue from plotly issue #1944. the visual data illustrates that the label parameters ("time" and "day") do not take effect. in github issues, neglecting visual data such as screenshots, diagrams, and videos @xcite @xcite . however, this visual data is crucial for resolving issues, as it conveys additional knowledge that text alone cannot, including actual results, expected results, and error messages. figure 1 shows a specific example where visual data illustrates the running result. moreover, we statistically analyze swe-bench @xcite , the most popular benchmark for issue resolving. the result shows that over 5% of github issues contain visual data, with even higher percentages in visualization libraries like seaborn foot_0 and matplotlib foot_1 , where they reach 45.5% and 27.2% respectively. this analysis further highlights the importance of resolving visual github issues. however, existing approaches struggle to resolve them effectively, as they overlook visual data, which calls for new solutions that leverage visual data. an intuitive approach is to extract visual data from the issue and include it as part of the prompt. while this approach seems to leverage visual data, it requires models with advanced multimodal and coding capabilities. currently, only the latest commercial models, @xcite and claude 3.5 sonnet @xcite , barely meet these requirements, but their capabilities remain highly limited. moreover, these models are less suitable for issue resolving due to high computational costs. based on our analysis, using these models within the popular swe-agent approach @xcite to run through all issues in swe-bench once is estimated to cost an average of over $4,700 @xcite , which imposes a significant financial burden on researchers. to address this, we propose codev, the first approach that leverages visual data to enhance the issue-resolving capabilities of llms at low cost. to resolve each issue, codev follows a two-phase process: data processing and patch generation. in the data processing phase, codev processes the visual data within the issue from both local and holistic perspectives. this phase produces fine-grained descriptions of the visual data and a structured summary of the entire issue. in the patch generation phase, codev leverages the processed information to assist llms in generating a patch to resolve the issue. to evaluate our approach, we construct a benchmark specifically designed for evaluating visual github issue resolving, called visual swe-bench. the benchmark comprises 133 task instances spanning 11 open-source github repositories, each of which has undergone rigorous selection. finally, we conduct a series of experiments to validate the effectiveness of our approach. experimental results demonstrate that codev achieves a round 63.13% relative improvement in the percentage of resolved instances on visual swe-bench compared to agentless. additionally, through case studies, we analyze the role of each component of codev, providing insights into leveraging visual data to resolve issues. overall, the contributions of this paper are as follows: ## approach figure 2 illustrates an overview of codev, which consists of two phases: data processing and patch generation. the first phase processes visual data and the second phase uses the processed information to assist llms in generating a patch. below is a detailed description of each phase. ## visual swe-bench benchmark in current benchmarks for the issue-resolving task, only the recently released swe-bench multimodal @xcite focuses on visual issues. however, as of writing, swe-bench multimodal foot_2 lacks evaluation fields, and its evaluation script has not been made public, making it unsuitable for evaluation. to evaluate codev, we construct a benchmark for resolving visual github issues, namely visual swe-bench. below, we detail our benchmark construction process and its key features. ## related works issue resolving approaches. to assist llms in resolving github issues, many approaches have already been proposed. retrieval augmented generation (rag) @xcite is a direct approach that resolves the issue by first extracting relevant code snippets from the repository and then using them to prompt llms to generate a patch. swe-agent @xcite meticulously designs an agent-computer interface (aci) that enables llm agents to interact with repository environments to solve software engineering tasks. autocoderover @xcite combines llms with code search, utilizes program structure, and conducts iterative searches for program improvement. coder @xcite ) is a multi-agent approach for issue-resolving tasks, adopting a multi-agent framework and pre-defined task graphs. agentless @xcite points out the limitations of using agents and proposes a simple two-phase process of localization and repair to solve software development problems. however, these existing approaches overlook visual data within issues. codev bridges this gap by processing visual data from both local and holistic perspectives, enhancing the capabilities of llms to resolve complex visual issues. ## conclusion we propose codev, an approach that leverages visual data to resolve issues automatically. it processes visual data and provides llms with valuable information that enhances their ability to resolve issues. to evaluate codev, we construct a benchmark for visual issue resolving, namely visual swe-bench. through extensive experiments, we demonstrate the effectiveness of codev and find that it maintains robust performance across vlms with varying model sizes. additionally, through case studies, we analyze the function of each component of codev, offering insights on leveraging visual data to resolve issues.