Can Multimodal LLM s See Materials Clearly? A Multimodal Benchmark on Materials Characterization (2025)

Abstract: Materials characterization is fundamental to acquiring materials information, revealing the processing-microstructure-property relationships that guide material design and optimization. While multimodal large language models (MLLMs) have recently shown promise in generative and predictive tasks within materials science, their capacity to understand real-world characterization imaging data remains underexplored. To bridge this gap, we present \textbf{MatCha} , the first benchmark for materials characterization image understanding, comprising 1,500 questions that demand expert-level domain expertise. MatCha encompasses four key stages of materials research comprising 21 distinct tasks, each designed to reflect authentic challenges faced by materials scientists. Our evaluation of state-of-the-art MLLMs on MatCha reveals a significant performance gap compared to human experts. These models exhibit degradation when addressing questions requiring higher-level expertise and sophisticated visual perception. Simple few-shot and chain-of-thought prompting struggle to alleviate these limitations. These findings highlight that existing MLLMs still exhibit limited adaptability to real-world materials characterization scenarios. We hope MatCha will facilitate future research in areas such as new material discovery and autonomous scientific agents. MatCha is available at https://github.com/FreedomIntelligence/MatCha.

## introduction materials characterization serves as a critical means of obtaining information about the physical world @xcite , offering insights that transcend the limitations of human sensory perception. it enables multi-scale analysis, providing rich data on material morphology, composition, and structure. this information, in turn, reveals the physical, chemical, and mechanical properties essential * corresponding author. for guiding new material design and optimization @xcite . for instance, scanning electron microscopy (sem) and transmission electron microscopy (tem) images are instrumental in determining the underlying mechanisms of steel bar fracture in buildings @xcite . despite its importance, interpreting diverse and complex imaging data generated by various characterization techniques remains a significant challenge that demands extensive domain expertise. typically, even experienced materials researchers invest considerable time analyzing these multifaceted results. this analytical burden becomes particularly acute when handling high-throughput data, where efficiency bottlenecks arise. while convolutional neural networks (cnns) have been utilized for various materials characterization tasks @xcite @xcite @xcite , providing valuable insights, their application faces notable limitations. predominantly, these cnnbased approaches are task-specific, exhibiting limited cross-task generalization. furthermore, prior work has largely focused on morphological perception, resulting in shallow image content understanding that often falls short of the nuanced interpretations required by materials scientists in real-world scenarios. these constraints highlight the need for more versatile and deeply understanding models. leveraging mllms @xcite @xcite @xcite for these challenges offers distinct advantages. mllms have demonstrated strong performance and generalization in both natural and domain-specific image understanding @xcite , and have spurred revolutionary changes in materials science, including property prediction @xcite @xcite , new material design @xcite @xcite , and autonomous scientific agents @xcite . these models show great potential to assist materials scientists with diverse materials characterization tasks via natural language interaction, thereby facilitating new material development and boosting scientific productivity. to realize this potential, mllms must first be capable of accurately interpreting diverse materials characterization images, recognizing fundamental visual content, and performing reasoning based on visual cues. however, existing evaluations of mllms on scientific imaging data primarily focus on biomedical domains @xcite @xcite , are confined to relatively simplistic figures and charts in limited scientific fields @xcite @xcite , or lacking sufficient depth to represent the complexity of authentic materials research scenarios @xcite @xcite . consequently, a comprehensive, expert knowledge-anchored multimodal benchmark specifically designed for materials science to rigorously assess current models is notably absent. this gap impedes the progress toward ai-assisted research and autonomous scientific discovery agents. to bridge this gap, we present matcha, a challenging multimodal benchmark for materials characterization imaging data understanding. the core strengths of matcha are threefold: (1) practical and realistic task design: the design philosophy of matcha originates from real-world scientific workflows. the tasks are derived directly from the research processes of materials scientists and are designed to reflect authentic challenges in practice. we first benchmark state-of-the-art mllms on matcha under the zero-shot setting, observing a substantial performance gap between models and human experts as well as noticeable performance degradation across different task stages due to limited generalization capability. next, we further investigate whether these performance gaps can be bridged by in-context learning or by guiding the model through a chain-of-thought (cot) process. the results show that while some models do benefit from these strategies, others exhibit unstable or even degraded performance, and a significant gap to human expert performance persists. in summary, our contributions are as follows: ## the matcha benchmark we introduce matcha, a comprehensive and challenging benchmark for advancing multimodal materials characterization visual analysis and understanding. matcha comprises 1,500 questions across 21 tasks, reflecting expert-level difficulty and grounded in real-world scientific scenarios faced by scientists. in the following sections, we elaborate on how we construct tasks ( §3.1), collect data ( §3.2), generate questions ( §3.3), and analyze its composition ( §3.4). the holistic construction pipeline is shown in fig. 1 . ## conclusion automated interpretation of complex materials characterization imaging data remains a significant bottleneck. in this work, we introduce matcha, the first benchmark designed to assess the capability of current mllms to understand materials characterization imagery. we construct a suite of realistic and scientifically meaningful tasks to evaluate state-ofthe-art mllms. our comprehensive evaluations pinpoint their inadequacy for generalization, deep domain knowledge, complex morphology perception, and nuanced materials analysis. matcha thus serves as a critical tool for diagnosing these core deficiencies, aiming to guide the development of mllms that can truly accelerate materials research and enable autonomous scientific discovery.