IW -Bench: Evaluating Large Multimodal Models for Converting Image-to-Web (2025)

Abstract: Recently, advancements in large multimodal models have led to significant strides in image comprehension capabilities. Despite these advancements, there is a lack of a robust benchmark specifically for assessing the image‐to‐web conversion proficiency of these large models. It is essential to ensure the integrity of the web elements generated, which comprise both visible and invisible categories. Previous evaluation methods (e.g., BLEU) are notably susceptible to significant alterations due to the presence of invisible elements. Furthermore, it is crucial to measure the layout information of web pages—i.e., the positional relationships between elements—which has been overlooked by prior work. To address these challenges, we have curated and aligned a benchmark of images and corresponding web codes (IW-bench). Specifically, we propose Element Accuracy, which tests the completeness of elements by parsing the Document Object Model (DOM) tree. We also introduce Layout Accuracy to analyze positional relationships by converting the DOM tree into a common subsequence. In addition, we design a five‐hop multimodal Chain‐of‐Thought prompting strategy for improved performance, consisting of: 1) SoM prompt injection, 2) inferring elements, 3) inferring layout, 4) inferring web code, and 5) reflection. Our benchmark comprises 1,200 image–code pairs with varying levels of difficulty. We have conducted extensive experiments on existing large multimodal models, providing insights into their performance and identifying areas for improvement in the image‐to‐web domain.

## introduction the development of large multimodal models has emerged as a new trend, starting with @xcite ). an increasing number of multimodal figure 1 : the process of image-to-web. we prompt a large multimodal model to generate web code based on the input image. finally, we need to compare whether the newly rendered result is consistent with the input image. constructing a benchmark for appraising imageto-web tasks is markedly more intricate than the conventional programming challenges @xcite . this heightens complexity stems from the multifaceted nature of web code, encompassing html, css, and javascript. it demands scrutiny not only of functional correctness but also of visual elements like layout, design, and user interaction. predominant evaluation frameworks for the image-to-code task @xcite often hinge on similarity metrics such as bleu. nevertheless, these metrics encounter substantive limitations in the context of web evaluation. the intricate web of javascript functionalities and css styling options means that disparate web codes can produce indistinguishable visual outcomes. consequently, such conventional metrics falter in precisely assessing the completeness of elements depicted in images and their layout intricacies. a further complication arises from non-visible elements, introducing a fragility in these evaluation methods. varied development ways can lead to substantial discrepancies in these non-visible components, even among web pages that appear visually identical. recent websight @xcite is finetuned on numerous image-code pairs, however it has not conducted performance evaluations. these challenges underscore the imperative for a more sophisticated and comprehensive benchmark. to address the challenges, we introduce iw-bench to assess the capabilities of large multimodal models in the image-to-web. firstly, we design a data construction pipeline, comprising 1200 entries of three difficulty levels: level i (simple), level ii (medium), and level iii @xcite . for metric innovation, we propose element accuracy and layout accuracy. specifically, to effectively measure the completeness of web elements, the element accuracy tests the completeness of the elements from six dimensions (tag, text content, attribute, style, javascript, children) by parsing the document object model (dom) tree. to analyze the relative positional relationships of elements, we first traverse the dom tree in the same manner (in-order traversal) to obtain a list of elements, then we calculate the overlap between the common sub-sequence and the ground truth for the layout accuracy. moreover, we establish a effective five-hop multimodal chain-of-thought method to enhance the model performance, including five specialized hops: som prompt injection, inferring elements, inferring layout, inferring web code, and reflection. to summarize: ## related work multimodal benchmarks for large multimodal models @xcite @xcite @xcite have assess the instruction-following and reasoning capabilities. as these foundation models become more relevant to real-world applications, unlike prior work, we propose iw-bench to benchmark their generation capabilities of the hot image-to-web area on a diverse set of visual contexts. to the best of our knowledge, websight (laurençon et al., 2024) is the only relevant benchmark. however, it is more like a fine-tuning dataset for large multimodal models rather than a benchmark for evaluation. the detailed comparison is in ## iw-bench overview: we design a pipeline for the construction of iw-bench in figure 3 . specifically, to obtain complex-level data, we crawl publicly available web pages and perform de-identification and simplification. for simple and medium level data, we prompt gpt-4 (openai, 2023a) to generate them. finally, we filter out some low-quality samples from the obtained image-code pairs. quantity of iw-bench is in ## five-hop multimodal chain-of-thought in this section, we design a five-hop multimodal chain-of-thought prompting for image-to-html task in figure 4 , which contains five hop: 1) som prompt injection. 2) inferring elements. 3) inferring layout. 4)inferring web code. 5) reflection. ## ablation in this section, we conduct the ablation on our five-hop multimodal chain-of-thought method, all experiments are conducted on gpt4v. ## visualization in this section, we present web page renderings after various reflections, highlighting progressive enhancements. in figure 7 , the original design features a prominent title, three subheadings-business planning, cards, consulting-and the corresponding text. the navigation bar includes home, blog, properties, about, github, and language options (en, es). the first reflection accurately captures the structure of navigation bar, though the text does not completely match. the alignment of the main title and subheadings requires adjustment. by the second reflection, text fidelity improves, aligning closer with the original content, and the navigation bar layout becomes more compact. the third reflection brings further precision to text and tightens the overall page layout, aligning visual elements more accurately and reducing layout errors. the use of color and contrast also more closely mirrors the original, enhancing visual consistency. ## conclusion in this paper, we introduce iw-bench. two evaluation metrics are proposed: element accuracy to assess the completeness of elements, and layout accuracy to evaluate the positional relationships of elements. furthermore, we outline a five-hop multimodal chain-of-thought method aimed at enhancing image-to-web conversion. we evaluate large multimodal models and provide an analysis of the results.