Lost in Translation: Chemical Language Models and the Misunderstanding of Molecule Structures (2024)

Abstract: The recent integration of chemistry with natural language processing (NLP) has advanced drug discovery. Molecule representation in language models (LMs) is crucial in enhancing chemical understanding. We propose Augmented Molecular Retrieval (AMORE), a flexible zero-shot framework for assessment of Chemistry LMs of different natures: trained solely on molecules for chemical tasks and on a combined corpus of natural language texts and string-based structures. The framework relies on molecule augmentations that preserve an underlying chemical, such as kekulization and cycle replacements. We evaluate encoder-only and generative LMs by calculating a metric based on the similarity score between distributed representations of molecules and their augmentations. Our experiments on ChEBI-20 and QM9 benchmarks show that these models exhibit significantly lower scores than graph-based molecular models trained without language modeling objectives. Additionally, our results on the molecule captioning task for cross-domain models, MolT5 and Text+Chem T5, demonstrate that the lower the representation-based evaluation metrics, the lower the classical text generation metrics like ROUGE and METEOR.

## introduction drawing inspiration from the progress of transformer-like architectures in nlp @xcite , the drug discovery community has embraced state-of-the-art molecular-generation methodologies. this includes leveraging lmbased approaches such as chemberta, t5chem, chemformer, and bartsmiles (chithrananda * these authors contributed equally to this work. the order of author names was randomly determined. @xcite @xcite . smiles (simplified molecular input line entry system) @xcite , a commonly employed molecular representation, enables the use of molecules in a string-based format. these single-domain models are usually pre-trained on large smiles datasets like zinc-15 @xcite , then fine-tuned for downstream tasks like reaction and property prediction on datasets like uspto @xcite and moleculenet @xcite . recently, lms like molt5 @xcite and text+chem t5 @xcite have been introduced to integrate chemical and linguistic knowledge. these models were pretrained on chemical and textual data, e.g., the large c4 @xcite and zinc-15, and finetuned on cross-domain tasks like molecule captioning. however, the evaluation of downstream tasks does not directly assess knowledge of chemistry, such as understanding various representations of the same molecular chemical structure. in this work, we seek to answer the following research question (rq): do chemical language models (chemlms) learn patterns and relationships within symbolic representations of molecular structures during pretraining, enabling them to differentiate molecular structures based on learned patterns? our main contribution is the novel unified approach (fig. 1 ) to verify if chemlms have effectively grasped the fundamental rules on the construction of molecular representations, such as smiles. our hypothesis posits that augmentation should not significantly alter the similarity score between distributed representations of molecules and their augmented versions. to address the rq, we conduct experiments using bert-based @xcite , gpt-based @xcite , and t5-based (raffel et al., 2020) models. [ch3][c]1[ch][ch][ch][c]2[c]([c](=[o])[oh])[ch][c]([o-])[ch][c]12 @xmath0 figure 1 : our evaluation involves generating augmented molecules for all molecules in a dataset using one of four augmentation procedures. by encoding original molecules and augmented smiles representations and calculating distances between embeddings, the study determines model performance based on top-1 accuracy, where the correct augmented smiles must be retrieved at the top rank. our work is designed to provide insights into the extent to which chemlms can discern molecular structures and the impact of augmentation on their performance. to summarize, our main contributions are as follows: ## ♡amore: augmented molecular retrieval in this section, we introduce amore, a flexible embedding-based evaluation framework for language model analysis in the chemistry domain. ## models for our experiments, we adopted various transformer-based @xcite molecular representation models, including encoder-only, encoder-decoder, and decoder-only architectures. all models are publicly available at huggingface. ## discussion in this paper, we release a general framework for analyzing knowledge awareness of modern lms in the chemical domain. while we rely on l2 distance as a similarity distance throughout all our experiments, an arbitrary embedding similarity measure can be employed. similarly, possible augmentation types are not limited to the ones considered in our research and can be extended. this flexibility might open new avenues for interpretation and analysis for lms in the chemical domain. our experiments have shed light on the research question formulated in the introduction and revealed a few critical limitations of the existing lms in chemistry-related tasks. first, the embedding space of chemical lms is not robust even to simple augmentations of smiles strings known as identity transformations of molecules in chemistry. while robustness to these augmentations can vary across different model layers, no intermediate layer would be stable to smiles augmentations. second, the performance of chemical lms at downstream tasks, such as molecule captioning, can be significantly limited when being passed an out-of-distribution (ood) input. these two findings demonstrate that the existing chemical lms have problems with distinguishing the same molecules in different representations during nlp-inspired pre-training procedures. they overfit on a specific format of input molecular string representations rather than truly gain an understanding of molecules. finally, cross-modal chemical lms tend to be more robust to ood input samples, highlighting the importance of further developing multimodal models for chemistry and nlp. meanwhile, metrics for the isomers dataset are lower and show minimal differences across models, likely attributed to the dataset's structure comprising isomeric aromatic compounds with identical molecular formulas and atom counts. the key idea is that chemical models must accurately translate augmented smiles into molecular structures. without fully understanding smiles syntax and distinguishing same-structure smiles, chemlms remain vulnerable to real-world data perturbations. this analysis aims to inform revisions to the established pipeline for learning chemical representations from nlp. ## conclusion in this paper, we introduce amore, a novel method (fig. 1 ) based on embedding distance and smiles augmentation to explore and evaluate the trustworthiness of language model's representations of a chemical substance and its ability to recognize molecule structures in smiles string representations. by using this method, we assessed the most popular chemical lms for several benchmarks (chebi-20, qm9, and moleculenet). we propose to use an isomeric subset of the qm9 dataset, which is novel to this task. though the first attempts to study the impact of chemical augmentations on text+chem t5 and molt5 for molecule captioning exist, this is limited to cross-domain generative architectures requiring nlp tokens, constraining the number of suitable models for evaluation. the key novelty of our paper lies in the proposed probing scheme. it is the first application of computation of distances between embeddings for benchmarking chemical llms. as a result, our amore framework drastically extends this scope for evaluating and comparing models in domain-specific diverse architectures, including encoder-only versus generative models, as well as uni-modal lms (with molecule atom tokens only) versus cross-modal models (atom + nlp tokens). it is important to emphasize that our method exploits unique specifics of the chemical domain. in contrast with typical nlp tasks, our augmentations lead to the creation of total synonyms of a molecule, which are absent in general words of natural language. our framework opens avenues for future research, ranging from understanding the functionality of molecule smiles representations in lms to addressing weaknesses in chemical tasks and enhancing efficiency.