PRESTO : Progressive Pretraining Enhances Synthetic Chemistry Outcomes (2024)

Abstract: Multimodal Large Language Models (MLLMs) have seen growing adoption across various scientific disciplines. These advancements encourage the investigation of molecule-text modeling within synthetic chemistry, a field dedicated to designing and conducting chemical reactions to synthesize new compounds with desired properties and applications. Current approaches, however, often neglect the critical role of multi-molecule graph interaction in understanding chemical reactions, leading to suboptimal performance in synthetic chemistry tasks. This study introduces PRESTO (Progressive Pretraining Enhances Synthetic Chemistry Outcomes), a new framework that bridges the molecule-text modality gap by integrating a comprehensive benchmark of pretraining strategies and dataset configurations. It progressively improves multimodal LLMs through cross-modal alignment and multi-graph understanding. Our extensive experiments demonstrate that PRESTO offers competitive results in downstream synthetic chemistry tasks. The code can be found at https://github.com/IDEA-XL/PRESTO.

## introduction multi-modal large language models (mllms) have achieved extensive success across diverse scientific domains, including medicine @xcite , material science @xcite , and biochemistry @xcite . motivated by these advances, moleculetext modeling emerges as a new research direction, aiming to bridge the modality gap between molecules and texts @xcite . these methods have shown promising results on molecule captioning, retrieval, and denovo molecule design @xcite @xcite @xcite . in this study, we explore molecule-text modeling within synthetic chemistry. synthetic chemistry involves designing and executing chemical reactions to create new compounds with specific properties and applications. it is a field of immense practical value and includes tasks like forward reaction and retrosynthesis prediction. prior molecule-text modeling works @xcite @xcite have explored synthetic chemistry tasks, but they mostly overlook the 2d molecular graph information. however, 2d molecular graph information is crucial for understanding molecular topologies and is essential for synthetic chemistry in prior graphbased retrosynthesis studies @xcite . on the other hand, while pioneering works @xcite @xcite have enabled text llms to perceive 2d molecular graphs, these methods struggle to process multiple 2d molecular graphs in chemical reactions, resulting in limited performance on tasks such as forward reaction prediction and reagent recommendation. this limitation stems from their inadequate exploration and analysis of multi-modal pretraining strategies @xcite and dataset configuration @xcite , which do not fully support the comprehension of multiple graphs: to bridge this research gap, we first present a comprehensive benchmark and the corresponding analysis for pretraining strategies and dataset configurations for synthetic chemistry. while several prior benchmarks @xcite overlap with synthetic chemistry, they, unfortunately, encompass a limited subset of synthetic chemistry tasks, often mishandle dataset splitting, and sometimes include potential data leakage. we prevent this by cleaning the data meticulously and generating challenging test sets with scaffold splitting. our analysis shows that progressive multimodal domain pretraining significantly enhances reaction condition prediction accuracy. further, we find that increasing the granularity of molecular representation and using interleaved molecule-text data with name-conversion datasets during pretraining improve downstream task performance by better leveraging domain knowledge. building on the insights from our benchmark, we propose progressive pretraining enhances synthetic chemistry outcomes (presto), a specialized framework tailored for synthetic chemistry tasks. presto enables a mllm to process and understand interleaved molecular graph-text inputs, enhancing the model's understanding of the principles of chemical reaction by effectively utilizing mutual interactions between molecule-molecule and molecule-text pairs in context. to achieve this, presto designs a pretraining strategy and a pretraining dataset curated for multi-graph understanding. specifically, presto improves the llm's performance on synthetic chemistry in two stages progressively: (1) in the first training stage, presto cultivates the mllm's ability of cross-modal alignment; (2) in the second stage, presto focuses on multi-graph understanding, and injects domain knowledge of synthetic chemistry into the llm. further, to support effective pretraining, we construct a dataset comprising ∼3 million samples of synthetic procedure descriptions and molecule name conversions. through extensive experiments, we demonstrate that presto can effectively prepare a multi-modal llm for 10198 downstream tasks of synthetic chemistry. ## related works deep learning for synthetic chemistry. synthetic chemistry, a fundamental problem in chemistry, has seen significant advances through deep learning models that assist in various reactionrelated tasks using descriptor-based @xcite , graph-based @xcite , and sequencebased approaches @xcite . recent works @xcite @xcite also adapt language models for tasks such as forward reaction prediction @xcite , retrosynthesis @xcite , and reaction type classification @xcite , demonstrating high accuracy. although these models specialize in specific synthetic chemistry tasks, their pretraining on domain-specific data limits their ability to generalize and adapt to other synthetic tasks. to address this issue, multitask methods @xcite have been explored and demonstrate strong capabilities across domains. however, they are constrained by using only molecular sequences as input, overlooking the potential of textual information to assist in modeling. in contrast, our approach integrates reaction-related textual information with molecular modeling, enabling a flexible adaptation to various downstream tasks. molecule & text modeling (mtm). the integration of biomolecular modeling with natural language leverages rich textual data sources to enhance understanding and facilitate downstream text-related molecular tasks @xcite @xcite @xcite . various approaches have been proposed to learn effective representations of molecules, including 1d sequences @xcite @xcite @xcite , 2d graphs @xcite @xcite , 3d conformations @xcite and a combination of them @xcite . cross-modalities modeling includes contrastive learning over molecules and text @xcite @xcite or unified alignment of the two modalities through language modeling @xcite @xcite . prior works have primarily focused on individual molecule understanding or molecule-text retrieval, while our research expands to model multiple molecules and contextual text, thereby facilitating tasks relevant to chemical reactions. ## analyzing pre-training strategy and dataset configuration in this section, we conduct experiments to evaluate the impact of different pretraining strategies and dataset configurations on downstream tasks. experimental setting. we use the gin @xcite pretrained by moleculestm @xcite as the default graph encoder f m and a two- ## conclusion and future work this study explores integrating multimodal llms into synthetic chemistry tasks to overcome the molecule-text modality gap. we highlight the importance of multi-graph datasets and progressive pretraining methods, showing significant improvements in reaction predictions and synthetic chem-istry tasks. as a result, we introduce presto, which outperforms baseline llms. meanwhile, current multimodal molecule models are limited to generating only 1d sequences. as a potential direction, we envision developing models capable of producing comprehensive molecular representations (i.e., 2d, 3d). future research could also expand the diversity of datasets to include more molecular structures and improve the llm's capability for dialogue. we aim to advance the fields of synthetic chemistry and compound discovery, ultimately creating a more powerful and versatile assistant for chemists.