T ext2 M ol: Cross-Modal Molecule Retrieval with Natural Language Queries (2021)

Abstract: We propose a new task, Text2Mol, to retrieve molecules using natural language descriptions as queries. Natural language and molecules encode information in very different ways, which leads to the exciting but challenging problem of integrating these two very different modalities. Although some work has been done on text-based retrieval and structure-based retrieval, this new task requires integrating molecules and natural language more directly. Moreover, this can be viewed as an especially challenging cross-lingual retrieval problem by considering the molecules as a language with a very unique grammar. We construct a paired dataset of molecules and their corresponding text descriptions, which we use to learn an aligned common semantic embedding space for retrieval. We extend this to create a cross-modal attention-based model for explainability and reranking by interpreting the attentions as association rules. We also employ an ensemble approach to integrate our different architectures, which significantly improves results from 0.372 to 0.499 MRR. This new multimodal approach opens a new perspective on solving problems in chemistry literature understanding and molecular machine learning.

## introduction discovering new properties and applications of different molecules is critical for accelerating discovery in medicine and science. existing databases contain tens of millions of molecules; pubchem @xcite alone has 110 million compounds. many information retrieval (ir) tools for chemistry rely on queries based on natural language descriptions of the molecules and existing chemical reactions. hundreds of millions of possible molecules cannot all possibly undergo laboratory 1. h 2 o h 2 o 2 c 2 h 6 o query figure 1 : given a natural language description of water, we want to rank the corresponding molecule h 2 o first among all the possible molecules. much current work focuses on images and language @xcite ), but it is beneficial for the community to consider modalities beyond traditional ones, increasing their work's impact and efficacy. for example, integrating nlp and ## task definition to push the boundaries of multimodal models, we present a new ir task: text2mol. given a text query and list of molecules without any reference textual information (represented, for example, as smiles strings, graphs, or other equivalent representations) retrieve the molecule corresponding to the query. figure 1 shows an example of this task. from a text description of a molecule, the model must incorporate the information in the description into a semantic representation which can be used to directly retrieve the molecule. this requires the integration of two very different types of information: the structured knowledge represented by text and the chemical properties present in molecular graphs. we assume there is only one correct (relevant) molecule for each description, so we consider two measures for this task: hits@1 and mean reciprocal rank (mrr). ## conclusions and future work in this work, we present text2mol: a novel and challenging cross-modal information retrieval task to retrieve molecules using natural language descriptions. to tackle this problem, we apply contrastive representation learning to a bertbased text encoder and both mlp and gcn-based molecule encoders. we show that these models are complementary and that an ensemble approach combines them very effectively. we also show that the ensemble approach is effective for combining identically trained neural networks (with different parameter initialization), and we consider attentionbased association rules. improved encoder architectures will likely yield improvements, and further investigation of how model architectural choices affect these rules and their interactions for reranking may be interesting as well. in the future, we plan to further improve results by integrating external knowledge as constraints. it should also be noted that this task is possible in the reverse direction, from molecules to descriptions. this has many possible applications, such as finding relevant descriptions for newly discovered molecules.