M ol TC : Towards Molecular Relational Modeling In Language Models (2024)

Abstract: Molecular Relational Learning (MRL), aiming to understand interactions between molecular pairs, plays a pivotal role in advancing biochemical research. Recently, the adoption of large language models (LLMs), known for their vast knowledge repositories and advanced logical inference capabilities, has emerged as a promising way for efficient and effective MRL. Despite their potential, these methods predominantly rely on textual data, thus not fully harnessing the wealth of structural information inherent in molecular graphs. Moreover, the absence of a unified framework exacerbates the issue of insufficient data exploitation, as it hinders the sharing of interaction mechanism learned across various datasets. To address these challenges, this work proposes a novel LLM-based multi-modal framework for molecular interaction modeling following Chain-of-Thought (CoT) theory, termed MolTC, which effectively integrate graphical information of two molecules in pair. To train this integrated framework efficiently, we introduce a *multi-hierarchical CoT theory* to refine its training paradigm, and conduct a comprehensive *Molecular Interactive Instructions* dataset for the development of biochemical LLMs involving MRL.Our experiments,conducted across various datasets involving over 4,000,000 molecular pairs, exhibit the superiority of our method over current GNN and LLM-based baselines. Code is available at https://github.com/MangoKiller/MolTC.

## introduction molecular relational learning (mrl) @xcite , aiming to understand interactions between molecular pairs, has gained significant interest due to its wide range of applications @xcite . for example, drug-drug interactions (ddis) are critical in pharmacology and drug development @xcite , while solute-solvent interactions (ssis) are fundamental in solution chemistry and the design of chemical processes @xcite . however, the exhaustive experimental validation of these interactions is notoriously time-consuming and costly. in response, adopting large language models (llms) @xcite , known for their vast knowledge repositories and advanced logical inference capabilities, has emerged as an efficient and effective alternative for mrl @xcite . despite their promise, a primary concern of current llm-based paradigm is the insufficient data exploitation. specifically, they predominantly rely on the textual data such as smiles (simplified molecular input line entry system) and property descriptions, thus not fully harnessing the wealth of structural information inherent in molecular graphs @xcite , as indicated in figure 1 @xcite . current studies have indicated that it is challenging for llms to fully understand the complex graphs based solely on textual data, hence, it's crucial to explicitly model these structures given their significance in mrl @xcite . compounding this concern is the absence of a unified framework for llm-based mrl @xcite . concretely, this absence impedes the sharing and integration of interaction mechanisms learned across various datasets, leading to a fragmentation in collective insights. especially, it poses a catastrophic challenge for tasks with a limited number of labeled pairs @xcite , where llms often struggle with due to the high risk of overfitting, as illustrated in figure 1 @xcite . worse still, such limited datasets are prevalent in mrl since the experimental acquisition is often constrained by high costs @xcite . the first molecule @xmath0, its smiles @xmath1, while the second molecule @xmath2, its smiles @xmath3. what is the solvation gibbs free energy of this molecule pairs? if the first is the solute and the second is the solvent, the gibbs free energy of this pair is ## methodology in this section, we detail our moltc, which harnesses the power of llms for comprehending molecular interactions. we begin with the introduction of model framework in section 2.1. taking a step further, the training paradigm guided by the principle of multi-hierarchical cot is outlined in section 2.2. moreover, the dynamic parameter sharing strategy tailored for moltc and our developed datasets, mot-instructions, are elaborated in section 2.3 and 2.4, respectively. ## experiment in this section, we aim to answer the following research questions: ## discussion potential data leakage risk. llm might have memorized the prediction target during pretraining using a vast amount of text corpus. hence, we acknowledge that llm-based methods for biochemistry tasks such as molt5 @xcite , molca @xcite and our moltc may face potential data leakage risks due to the memory during the pre-training process of their base llms. however, we claim that ablation experiments can mitigate the influence of data leakage, demonstrating the effectiveness of methods despite using potentially contaminated llms. for instance, in our ablation experiments, the backbone llms are fed with smiles strings and textual descriptions of molecules solely, and their performance is much worse than that of original methods. this stark contrast leads to two analytical observations: in summary, while these ablation experiments can not completely eliminate the impact of data leakage, they do mitigate it. furthermore, it is worth noting that while the risk of data leakage exists, it does not deter researchers from continuously exploring and optimizing llm-based frameworks for biochemical tasks. in the future, by constructing new biochemical datasets through biochemical experiments, which do not exist in the llm's training data, researchers can eliminate the data leakage issue in these frameworks thoroughly. ## conclusion this work focuses on molecule rationale learning, which plays a pivotal role in predicting molecular interactions. specifically, we introduce a novel, unified llm-based framework for predicting molecular interactive properties, termed moltc. to efficiently train it, we propose a multi-tiered cot principle to guide the training paradigm. experiments conducted across twelve varied datasets demonstrate the superiority of our method over the current gnn and llm-based baselines. this breakthrough sets a new standard for integrating multimodal data in llm-based mrl.