Stress-Testing Multimodal Foundation Models for Crystallographic Reasoning (2025)

Abstract: Evaluating foundation models for crystallographic reasoning requires benchmarks that isolate generalization behavior while enforcing physical constraints. This work introduces, xCrysAlloys, a multiscale multicrystal dataset with two physically grounded evaluation protocols to stress-test multimodal generative models. The Spatial-Exclusion benchmark withholds all supercells of a given radius from a diverse dataset, enabling controlled assessments of spatial interpolation and extrapolation. The Compositional-Exclusion benchmark omits all samples of a specific chemical composition, probing generalization across stoichiometries. Nine vision–language foundation models are prompted with crystallographic images and textual context to generate structural annotations. Responses are evaluated via (i) relative errors in lattice parameters and density, (ii) a physics-consistency index penalizing volumetric violations, and (iii) a hallucination score capturing geometric outliers and invalid space-group predictions. These benchmarks establish a reproducible, physically informed framework for assessing generalization, consistency, and reliability in large-scale multimodal models. Dataset and implementation are available at https://github.com/KurbanIntelligenceLab/StressTestingMMFMinCR.

## introduction crystalline solids underpin a wide range of modern technologies. their periodic atomic arrangements determine the band gaps of semiconductors, the ion-transport channels in battery electrodes, and the phonon spectra that govern thermal conductivity in microelectronics @xcite . even a single misassigned lattice parameter can cascade through simulation pipelines, distorting derived physical models and impeding materials discovery @xcite . structural resolution has traditionally re-lied on labor-intensive diffraction techniques or exhaustive structure enumeration followed by density functional theory (dft) relaxation @xcite . synthesis methods such as hydrothermal growth @xcite , chemical vapor deposition @xcite , and high-pressure processing @xcite further introduce domain-specific variability by accessing distinct thermodynamic regimes and defect topologies. recent progress in generative modeling, particularly autoregressive language models capable of emitting crystallographic information files @xcite , enables rapid lattice generation with chemically plausible compositions. however, existing materials databases-such as aflow @xcite , the materials project @xcite , and oqmd @xcite -remain predominantly unimodal and typically lack expertwritten, human-interpretable descriptions of crystal chemistry. this absence of multimodality impedes systematic evaluation of large vision-language models and language models in crystallographic reasoning. current scientific multimodal benchmarks are limited in scale, visually simplistic, and textually sparse, constraining analysis of factual accuracy, hallucination patterns, and compliance with physical laws. to overcome these limitations, xcrysalloys, a new multimodal dataset of crystalline alloy materials is presented, accompanied by two physically grounded benchmarking protocols. the spatialexclusion (se) benchmark withholds supercells of a specific radius from the set {r k } 10 @xmath0, enabling controlled evaluation of spatial interpolation (interior radii) and extrapolation (boundary radii). in parallel, the compositional-exclusion (ce) benchmark withholds all samples corresponding to a target chemical composition, assessing generalization across compositional space. state-of-theart foundation models are evaluated under both benchmarks by generating structural annotations from crystallographic images and textual prompts. model outputs are parsed into a structured ma-terial properties schema and assessed for geometric accuracy, consistency with physical constraints, and hallucination risk. these benchmarks provide a reproducible, domain-informed framework for measuring generalization and reliability in large-scale generative models, and contribute to emerging efforts to probe, refine, and safely deploy scientific knowledge at scale. the remainder of the manuscript is structured as follows. section 2 surveys the theoretical foundations and related literature. section 3 details the methodological framework. section 4 describes the dataset construction, evaluation metrics, and experimental procedures. section 5 presents the empirical findings. section 6 discusses the study's limitations, and section 7 concludes with final observations. ## experiments dataset. xcrysalloys, comprises ten crystalline compounds of technological relevance: ag, au, ch 3 nh 3 pbi 3 , fe 2 o 3 , mos 2 , pbs, sno 2 , srtio 3 , tio 2 , and zno. for each material, spherical nanoclusters were extracted at four target radii @xmath1.7, 0.8, 0.9, 1.0} nm (r7-r10), yielding a multiscale corpus of 3d atomic structures. each nanocluster was rendered in ten orientations-one canonical and nine using fibonaccisphere rotations-to ensure quasi-uniform coverage over @xcite . this process generated over 400 crystallographic images per material-derived from 4 radius levels and 10 orientations per structure (i.e., 4 × 10 = 40 images per material-radius combination)-paired with expert-curated annotations conforming to the material properties schema. full details on structure generation are provided in section 3.1. an overview is shown in figure 1 . ## limitations this study isolates two complementary generalization regimes-geometric interpolation/extrapolation and chemical extrapolation-using a curated dataset of ten crystalline materials across four radii. while representative, this selection captures only a limited region of compositional and structural diversity present in real-world materials. all models are evaluated in a zero-shot setting with default decoding configurations, without fine-tuning, retrieval augmentation, or domain adaptation, which may underrepresent their full capabilities. evaluation emphasizes first-order structural properties such as lattice constants, density, and stoichiometry, along with a single volumetric consistency index. higher-order descriptors-including phonon spectra, band topology, or symmetrypreserving deformations-are not considered. the analysis focuses on static prediction quality and does not measure model responsiveness to feedback, learning curves under domain supervision, or variance across decoding seeds. ## conclusion this work introduces xcrysalloys and its two complementary benchmarks-se and ce-that isolate geometric interpolation and chemical extrapolation in crystallographic prediction. the evaluations reveal that current vision-language foundation models struggle to internalize core physical principles, as evidenced by high relative errors, substantial degradation in transfer settings, and disrupted interproperty correlations. the prevalence of hallucinated outputs and violations of basic physical laws further underscores the limitations of purely datadriven training in scientific domains. to advance reliability and generalization, future models must incorporate explicit physical constraints, symmetry priors, and uncertainty-aware reasoning. the proposed benchmarks provide a reproducible and physically grounded testbed for evaluating model robustness in structured scientific settings. by bridging multimodal language understanding with domain-specific inductive biases, this work aims to foster the development of more trustworthy foundation models for materials science and beyond.