Predictive Chemistry Augmented with Text Retrieval (2023)

Abstract: This paper focuses on using natural language descriptions to enhance predictive models in the chemistry field. Conventionally, chemoinformatics models are trained with extensive structured data manually extracted from the literature. In this paper, we introduce TextReact, a novel method that directly augments predictive chemistry with texts retrieved from the literature. TextReact retrieves text descriptions relevant for a given chemical reaction, and then aligns them with the molecular representation of the reaction. This alignment is enhanced via an auxiliary masked LM objective incorporated in the predictor training. We empirically validate the framework on two chemistry tasks: reaction condition recommendation and one-step retrosynthesis. By leveraging text retrieval, TextReact significantly outperforms state-of-the-art chemoinformatics models trained solely on molecular data.

## introduction in this paper, we propose a method for leveraging automatically retrieved textual knowledge to improve the predictive capacity of chemistry models. these chemoinformatics models are utilized for a wide range of tasks, from analyzing properties of individual molecules to capturing their interactions in chemical reactions @xcite @xcite . however, standard approaches make these predictions operating solely on molecular encodings. despite significant advances in neural molecular representations, the accuracy of these models still offers room for improvement. we hypothesize that their performance can be further enhanced using relevant information from the scientific literature. for instance, consider the task of finding a catalyst for the chemical reaction shown in figure 1 . this prediction proves to be challenging when considering the reaction components alone @xcite . at the same time, scientific literature provides several cues about potential catalysts (see highlighted excerpts). while on their own these paragraphs might not provide a comprehensive answer, they can guide a molecular predictor. this intuition motivates our approach to aggregating textual sources with molecular representations when reasoning about chemistry tasks. specifically, we aim to align the representations of the reaction and its corresponding text description, enabling the model to operate in the combined space. this design not only enables the model to retrieve readily available information from the literature but also enhances its generalization capacity for new chemical contexts. we propose textreact, a novel predictive chemistry framework augmented with text retrieval. textreact comprises two modules -a smilesto-text retriever foot_0 that maps an input reaction to corresponding text descriptions, and a text-augmented predictor that fuses the input reaction with the retrieved texts. the model learns the relation between a chemical reaction and text via an auxiliary masked lm objective incorporated in the predictor training. furthermore, to improve generalization to unseen reaction classes, we simulate novel inputs by eliminating from the training data the closest textual descriptions for given reactions. in addition to condition recommendation, textreact can be readily applied to other chemistry tasks. in our experiments, we also consider one-step retrosynthesis @xcite , the task of predicting reactants used to synthesize a target molecule (see figure 2 ). by leveraging relevant text, textreact achieves substantial performance improvement compared to the state-of-theart chemoinformatics models trained on reaction data alone. for instance, for condition recommen- @xmath0: reaction output: reaction condition nh2 n o s o o n h o s o n h n h n + ## related work multimodal retrieval this field studies retrieval algorithms when the query and target are in different modalities, as exemplified by image-text retrieval @xcite @xcite @xcite . previous research learns multimodal embeddings in image and text using techniques ranging from kernel methods to more expressive neural networks. this line also extends to other modalities such as video @xcite and audio @xcite . our retrieval method closely relates to clip @xcite , a multimodal model that leverages contrastive learning to align images with their corresponding natural language descriptions. moving to the chemistry domain, @xcite proposed text2mol to retrieve molecules using natural language queries. our smiles-totext retriever operates in the opposite direction, i.e., we use the aligned embedding spaces to retrieve a text description given a reaction query. we then use the retrieved text to enhance chemistry prediction. retrieval-augmented nlp there have been several studies on augmenting nlp models with retrieval. the idea is to retrieve relevant documents from a corpus and use them as additional context to the model. earlier works on open-domain question answering adopted standalone retrievers to identify supporting paragraphs for the given questions @xcite @xcite . @xcite jointly trained an end-to-end retriever with a language model, and @xcite extended the idea to general sequence-to-sequence generation. our work implements a similar idea of retrieval augmentation. however, we focus on tasks in the chemistry domain (see figures 1 and 2 ), with the goal of enhancing chemistry prediction models with natural language descriptions retrieved from the literature. ## experimental setup text corpus we construct a text corpus from uspto patent data processed by @xcite . each paragraph in the corpus provides a description of the synthesis procedure of a chemical reaction. the corpus consists of 2.9 million paragraphs, with an average length of 190 tokens. 4 while the original dataset includes structured reaction data extracted from each paragraph, including reactants, products, and reaction conditions, our model does not rely on this extracted data but instead learns directly from the unlabeled text. ## conclusion this paper presents textreact, a novel method that augments predictive chemistry with text retrieval. we employ information retrieval techniques to identify relevant text descriptions for a given chemistry input from an unlabeled corpus, and supply the retrieved text as additional evidence for chemistry prediction. in two chemistry tasks, textreact demonstrates strong performance when retrieving from the full corpus, and maintains a significant improvement when retrieving from a harder corpus that excludes the gold texts. our results highlight the promising potential of incorporating text retrieval methods in the field of chemistry. as chemically similar reactions have similar conditions and outcomes, effectively retrieving and grounding textual knowledge from patents and publications into the chemistry space can significantly enhance the predictive power of computational models.