Moleco: Molecular Contrastive Learning with Chemical Language Models for Molecular Property Prediction (2024)

Abstract: Pre-trained chemical language models (CLMs) excel in the field of molecular property prediction, utilizing string-based molecular descriptors such as SMILES for learning universal representations. However, such string-based descriptors implicitly contain limited structural information, which is closely associated with molecular property prediction. In this work, we introduce Moleco, a novel contrastive learning framework to enhance the understanding of molecular structures within CLMs. Based on the similarity of fingerprint vectors among different molecules, we train CLMs to distinguish structurally similar and dissimilar molecules in a contrastive manner. Experimental results demonstrate that Moleco significantly improves the molecular property prediction performance of CLMs, outperforming state-of-the-art models. Moreover, our in-depth analysis with diverse Moleco variants verifies that fingerprint vectors are highly effective features in improving CLMs’ understanding of the structural information of molecules.

## introduction in drug discovery and materials science, applying deep neural networks to molecular property prediction has brought increasing attention @xcite . these networks can predict molecular properties with a significantly reduced cost compared with traditional methods like wet lab experiments. moreover, combined with transfer learning approaches with large-scale pre-training, deep neural networks have shown their versatility and generalization capacity, allowing for applying a single model across various tasks. this also reduces the need for task-specific modeling, leading to high usability and practicality. figure 1 : illustration of moleco. we extract and construct a set of similar molecules by measuring the cosine similarity between fingerprint vectors of different molecules. we subsequently maximize the agreement between pairs of structurally similar molecules, while minimizing that of the other molecules in a batch. recently, inspired by the success of the pretrained language models @xcite , chemical language models (clms) have been introduced and shown their excellence in predicting molecular properties @xcite @xcite . these clms, typically employing transformer architectures @xcite , are trained on large-scale string-based molecular descriptors to learn universal molecular representations. string-based molecular descriptors, such as simplified molecular-input line-entry system (smiles) @xcite , compactly represent molecules in a text format, providing benefits in handling large-scale molecule data. moreover, the employed transformer architectures have shown high efficiency and parallelizability in processing large-scale molecular data. despite the efficiency of string-based molecular descriptors, they contain limited structural information of molecules in an implicit manner, which is critical in predicting molecular properties @xcite . for example, smiles involves redundant, invalid descriptors of molecular structures and needs to be interpreted to uncover its structural information. moreover, typical pre-training approaches for clms, namely masked language modeling @xcite , do not explicitly train models to capture such structural information. thus, current clms suffer from capturing the relationships between molecular structures and properties @xcite . in this work, we introduce moleco (molecular contrastive learning with chemical language models), a novel contrastive learning framework to enhance the understanding of clms on the structural information of molecules. moleco leverages contrastive learning among different molecules based on a structural similarity calculated using fingerprint embeddings @xcite , which contain substructure information of molecules. specifically, based on the fingerprint embeddings, moleco identifies and utilizes top-k structurally similar molecules as positive samples, while using the others in batch as negative samples. this contrastive learning approach enriches models' representation to better reflect the relationships between different molecules in molecular substructures. furthermore, moleco additionally trains clms with a prediction of structural embeddings in a multitask learning manner, as direct guidance of structure information of molecules. we evaluate moleco on various tasks from moleculenet benchmarks @xcite , including eight classification and four regression tasks. our extensive experiments verify that, although fingerprints are highly simplified and straightforward methods to embed structural information of molecules, moleco significantly improves the molecular property prediction of clms. notably, moleco achieves performance improvements of 1.8% and 7.3% on average in molecular property classification and regression tasks, respectively, compared with state-of-the-art models. moreover, our in-depth analysis demonstrates that the proposed fingerprint-based similarity effectively identifies structurally similar molecules, leading to the improvements in clms' understanding of structural properties of molecules. our main contributions are as follows: ## methodology in this section, we propose moleco, which trains clms to explicitly learn structural similarities of different molecules in a contrastive manner. specifically, we first obtain structural similarities using fingerprint embeddings of molecules, and subsequently train models to contrastively learn the similarities, as illustrated in figure 1 . in addition, we introduce an auxiliary training objective that directly predicts molecular embeddings, to further enhance the structural understanding of clms. after moleco training, we fine-tune the models on downstream tasks to predict molecular properties. ## conclusion we have introduced moleco, a novel contrastive learning framework to enhance the structural understanding of clms to improve molecular property prediction. we have trained clms to contrast structurally similar and dissimilar molecules, which are identified by using the fingerprint vectors of molecules. we have observed that moleco outperforms state-of-the-art models on diverse molecular property prediction benchmarks. furthermore, our in-depth analysis has confirmed that moleco effectively improves the structural understanding of clms, leading to significant performance improvements. particularly, fingerprints, which are highly simplified embedding methods, have most effectively improved the molecular property prediction of clms among diverse design choices. we plan to investigate the applicability of moleco on multi-modal transformers and generative clms.