M ol CA : Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter (2023)

Abstract: Language Models (LMs) have demonstrated impressive molecule understanding ability on various 1D text-related tasks. However, they inherently lack 2D graph perception — a critical ability of human professionals in comprehending molecules’ topological structures. To bridge this gap, we propose MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter. MolCA enables an LM (i.e., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector. Specifically, the cross-modal projector is implemented as a Q-Former to connect a graph encoder’s representation space and an LM’s text space. Further, MolCA employs a uni-modal adapter (i.e., LoRA) for the LM’s efficient adaptation to downstream tasks. Unlike previous studies that couple an LM with a graph encoder via cross-modal contrastive learning, MolCA retains the LM’s ability of open-ended text generation and augments it with 2D graph information. To showcase its effectiveness, we extensively benchmark MolCA on tasks of molecule captioning, IUPAC name prediction, and molecule-text retrieval, on which MolCA significantly outperforms the baselines.

## introduction language models (lms) have demonstrated significant achievements across various domains @xcite . notably, the wealth of biochemical literature in lms' pretraining data has enabled lms to obtain a high-level understanding of biochemical concepts and molecule properties. this can be reflected by their promising performances in biochemical and medical questionanswering benchmarks @xcite . therefore, it becomes increasingly urgent to incorporate these lms to augment research in chemistry and biology. for this purpose, we aim to utilize lms for molecule understanding. as shown in figure 1a , most existing lms @xcite @xcite represent molecules by their 1d simplified molecular input line entry system (smiles) strings @xcite and process them in a manner similar to texts. while convenient, treating molecules as strings overlooks the molecules' 2d graph representations, which are crucial to human professionals in comprehending the molecule structures @xcite . to combat that, recent works @xcite represent molecules as graphs and use a graph neural network (gnn; @xcite as the molecular graph encoder. the graph encoder is trained jointly with an lm through cross-modal contrastive learning @xcite , as illustrated in figure 1b . however, the application scope of cross-modal contrastive learning is limited @xcite : it is suitable for retrieval tasks, but is insufficient for open-ended molecule-to-text generation tasks, such as molecule captioning @xcite ) and molecule's iupac name prediction @xcite . this is because molecule-to-text generation is a conditional generation task @xcite . it requires the lm to understand 2d graphs as the generation conditions, which contrastive learning cannot achieve. @xcite attempt to directly input 2d graphs' representations into lms, however showing limited improvement. language model graph encoder cross-modal projector model architecture downstream task cross-modal retrieval graph encoder cross-modal projector -graph encoder cross-modal projector generation tasks: molecule caption, iupac name prediction stage pretrain stage 1 pretrain stage 2 fine-tune stage language model figure 2: molca's three-stage training pipeline. as figure 2 illustrates, molca uses a three-stage training pipeline to integrate its components. the two pretrain stages aim to develop the cross-modal alignment ability of the cross-modal projector. in pretrain stage 1, the projector and the encoder are trained to extract the molecule features that are the most relevant to the text. this stage endows the resulting model with powerful molecule-text retrieval ability. in pretrain stage 2, the cross-modal projector is connected to a frozen lm and trained for molecule captioning. this task forces the crossmodal projector to produce soft prompts that the lm can understand. in the final stage, molca is fine-tuned for downstream generation tasks. our contributions can be summarized as follows: ## model architecture here we introduce three key components of molca's architecture: 1) a graph encoder for 2d structure understanding, 2) an lm for text generation, and 3) a cross-modal projector to connect the graph encoder and the lm. we describe the uni-modal adapter in section 3.3. ## training pipeline this section delves into the details of molca's three-stage training pipeline (cf. figure 2 ). the two pretrain stages leverage a dataset of moleculetext pairs @xmath0, y 1 ), (g 2 , y 2 ), ...} to train the cross-modal projector and the graph encoder. the goal of pretraining is to translate 2d molecular graphs into soft prompts that a frozen lm can understand. the fine-tune stage focuses on efficient adaptation to downstream generation tasks. ## related works here we briefly review the molecule-related literature. we discuss molca's relations to visionlanguage pretraining methods in appendix a. molecule understanding via 1d language modeling. due to the extensive biochemical literature in their training corpus, some open-domain lms @xcite @xcite have obtained a high-level understanding of molecular and chemical concepts. this is demonstrated through their promising performances in text-related biochemical and medical question-answering benchmarks @xcite . among these lms, galactica @xcite shows competitive performances for using a corpus that is primarily composed of scientific literature. focusing on the chemistry domain, kv-plm @xcite ) models molecules by applying masked language modeling loss on 1d smiles. @xcite propose to predict the chemistry experiment actions by reading chemical reaction equations. molt5 @xcite presents several t5-based (raffel et al., 2020) lms for smiles-to-text and textto-smiles translations. further, @xcite propose to fine-tune t5 for chemical reaction prediction and retrosynthesis tasks. molca is different from these methods that exclusively utilize 1d smiles to represent molecules. instead, molca aims to enable lms to perceive molecules' 2d graph representations. molecule-text contrastive learning. driven by the demand of a molecule-text retrieval system, text2mol @xcite employs crossmodal contrastive learning to train a molecular graph encoder of @xcite and a text encoder of sci-bert @xcite . subsequent works @xcite @xcite have proposed enhancements, including the addition of inter-modal contrastive learning loss @xcite and applying the model for text-based molecule editing @xcite . however, cross-modal contrastive learning is unsuitable for open-ended conditional generation task @xcite , because of its focus on learning a similarity function. to resolve the problem, we propose molca to enable the lm's understanding of 2d molecular graphs, facilitating molca's capability of open-ended moleculeto-text generation. ## conclusion and future works in this work, we propose molca, a novel molecular language modeling method. molca aims to enable lms to perceive 2d graphs for molecule-totext generation. for this purpose, molca features a cross-modal projector to map representations of 2d graphs into the text space of lms. it also employs a uni-modal adapter for efficient downstream adaptation. molca achieves state-of-the-art performances on molecule captioning and molecule-text retrieval benchmarks. looking forward, we are interested in exploring lms for 3d molecular modeling and drug discovery tasks.