Two Steps from Hell: Compositionality on Chemical LM s (2025)

Abstract: This paper investigates compositionality in chemical language models (ChemLLMs). We introduce STEPS, a benchmark with compositional questions that reflect intricate chemical structures and reactions, to evaluate models’ understanding of chemical language. Our approach focuses on identifying and analyzing compositional patterns within chemical data, allowing us to evaluate how well existing LLMs can handle complex queries. Experiments with state-of-the-art ChemLLMs show significant performance drops in compositional tasks, highlighting the need for models that move beyond pattern recognition. By creating and sharing this benchmark, we aim to enhance the development of more capable chemical LLMs and provide a resource for future research on compositionality in chemical understanding.

## methodology existing chemical language models (chemlms) are primarily evaluated on narrow, single-domain tasks (e.g., reaction prediction or property estimation). however, real-world chemical reasoning often requires integrating knowledge across domains-for example, predicting the environmental impact of a reaction product demands understanding both reaction mechanisms and physicochemical properties. to bridge this gap, we propose steps (structured tasks for evaluating and promoting compositionality), a framework that trains and evaluates chemlms on compositional tasks where complex questions are solved by combining simpler subtasks. below, we formalize steps and its design principles. ## models and datasets dataset compilation to create a benchmark for compositional reasoning, we compile and modify existing chemical datasets: chebi-20, moleculeace, uspto-50k, and pubchem (see tab. ## evaluation results this section presents the results of our evaluation of various chemllms across a range of compositional reasoning tasks. the findings are summarized in ## conclusion the results of our study reveal that current chemical llms struggle with compositional reasoning, particularly in multi-step tasks and out-ofdistribution (ood) scenarios. while specialized models excel in specific tasks, general-purpose models demonstrate adaptability but falter when integrating information across multiple steps. key findings include significant performance drops in multi-step tasks, highlighting a reliance on surfacelevel pattern recognition rather than true compositional understanding. these insights highlight the need for better architectures and training to im-prove compositional reasoning, with a promising direction being the integration of external tools into llm prompts for hybrid reasoning.