M ol TRES : Improving Chemical Language Representation Learning for Molecular Property Prediction (2024)

Abstract: Chemical representation learning has gained increasing interest due to the limited availability of supervised data in fields such as drug and materials design. This interest particularly extends to chemical language representation learning, which involves pre-training Transformers on SMILES sequences - textual descriptors of molecules. Despite its success in molecular property prediction, current practices often lead to overfitting and limited scalability due to early convergence. In this paper, we introduce a novel chemical language representation learning framework, called MolTRES, to address these issues. MolTRES incorporates generator-discriminator training, allowing the model to learn from more challenging examples that require structural understanding. In addition, we enrich molecular representations by transferring knowledge from scientific literature by integrating external materials embedding. Experimental results show that our model outperforms existing state-of-the-art models on popular molecular property prediction tasks.

## introduction deep neural networks (dnns) have emerged as a compelling, computationally efficient approach for predicting molecular properties, with significant implications in material engineering and drug discovery. by training dnns on molecule data to predict the properties in a supervised manner or to reconstruct molecules in an unsupervised manner, these networks can significantly reduce the costs of traditional methods, which typically require chemical experts and wet-lab experiments. moreover, dnn-based molecular prediction has gained increasing popularity due to the generalization capacity of dnns. this allows for the application of a single (pre-)trained model across various tasks, reducing the need for task-specific modeling. figure 1 : existing pre-training methods for chemical language representation learning already converge at their early stage without seeing the entire data. consequently, molformer @xcite , a state-of-theart chemical language representation learning method, exhibits limited scalability in terms of data size. inspired by recent advances in pre-trained language models in the field of natural language processing (nlp), several chemical language representation learning methods based on transformers @xcite have been proposed. these methods typically employ self-supervised tasks on smiles (simplified molecular-input line entry system) sequences of molecules, analogous to the masked language modeling (mlm) commonly used in bert @xcite . since modern transformers are designed to scale to massive nlp corpora @xcite , they offer practical advantages in terms of efficiency and throughput. this enables the models to leverage massive amounts of smiles sequences to learn universal representations for molecules, leading to performance improvements in a wide range of molecular property prediction tasks @xcite . however, as these models typically follow settings designed for natural language modeling, the optimal pre-training settings for chemical language representation learning remain underexplored. through extensive investigation into the pretraining of smiles transformers, we have discovered that the current pre-training task, mlm on smiles sequences using a random masking strategy, is not effective for learning informative molecular representations. we have empirically observed that this task can be easily solved using surface patterns, leading to overfitting and limited scalability, as shown in figure 1 . this may be attributed to two inherent properties of smiles. first, existing large-scale molecule datasets exhibit unbalanced atom distributions @xcite . for example, in zinc @xcite , a representative dataset containing billions of molecules, carbon (c), nitrogen (n), and oxygen (o) comprise 95% of the tokens in total smiles sequences. second, the smiles grammar contains many superficial patterns, such as numbers representing ring structures that always appear twice. these patterns allow the model to predict original tokens by using simple rules, without learning the underlying chemical information. furthermore, unlike natural language, which is fundamentally grounded in concepts and possesses general expressivity across various problem-solving scenarios, smiles is designed solely to express molecular structure and does not directly represent molecular properties. thus, the current pre-training task likely provides a limited notion of molecular properties. in this paper, we propose a novel framework for pre-training smiles transformers, called moltres (molecular transformer with enhanced self-supervised learning), to address the aforementioned issues. our framework focuses on two key objectives: (1) increasing the difficulty of the pre-training task, and (2) incorporating external knowledge related to molecular properties into model representations. to achieve these goals, we first present a novel self-supervised learning pipeline, coined dynamol, based on generatordiscriminator training @xcite . this method trains a model to distinguish real smiles tokens from synthetically generated replacements, jointly used with substructure-level masking. it fa-cilitates to significantly increase the masking ratio for more challenging training examples that require an understanding of molecular structure, while minimizing discrepancy caused by mask tokens. in addition, we enhance model representations by integrating mat2vec word representations @xcite trained on massive scientific literature. this integration helps to directly embody molecular properties in the learned representations. to demonstrate the effectiveness of moltres, we conduct extensive experiments and ablation studies on diverse molecular property prediction tasks. we evaluate moltres on eight classification and four regression tasks from moleculenet @xcite , covering quantum mechanical, physical, biophysical, and physiological properties of chemicals. our results indicate that moltres outperforms state-of-the-art baselines across most tasks, including 1d sequence-, 2d graph-, and 3d geometry-based chemical models. furthermore, we observe that moltres outperforms state-ofthe-art models on seven polymer property prediction tasks, showing its generalizability to different chemical tasks. further analysis shows that moltres significantly improves the capabilities of chemical language representation learning by addressing the limitations of existing approaches. our contributions are summarized as follows: ## related work in recent years, representation learning has prevailed in numerous applications in natural language processing @xcite and computer vision @xcite . this trend has triggered many studies in chemical representation learning. the approaches in this field can be classified into three categories based on molecular descriptors used for pre-training: chemical language representation learning, chemical graph representation learning, and multi-modal chemical representation learning. ## moltres: molecular transformer with enhanced self-supervised learning in this section, we detail our framework, moltres, which is illustrated in figure 2 . we propose a novel pre-training task, called dynamol, which incorporates generator-discriminator training with substructure masking to increase the difficulty of a pre-training task. in addition, we integrate word representations that have been trained on scientific literature to enrich information directly related to molecular properties in the representations. ## conclusion in this work, we have proposed a novel chemical language representation learning framework, moltres. we have identified critical, previously unaddressed issues in existing methods for chemical language representation learning, specifically early convergence and overfitting, and presented two methods, generator-discriminator training with substructure masking and knowledge transfer from scientific literature based on mat2vec. our experimental results validate the superiority of our framework over existing chemical models across a wide range of molecular property prediction tasks, showing that moltres uniquely enables a robust training of chemical language models and provides substantial improvements in performances.