L + M -24: Building a Dataset for L anguage+ M olecules @ ACL 2024 (2024)

Abstract: Language-molecule models have emerged as an exciting direction for molecular discovery and understanding. However, training these models is challenging due to the scarcity of molecule-language pair datasets. At this point, datasets have been released which are 1) small and scraped from existing databases, 2) large but noisy and constructed by performing entity linking on the scientific literature, and 3) built by converting property prediction datasets to natural language using templates. In this document, we detail the L+M-24 dataset, which has been created for the Language + Molecules Workshop shared task at ACL 2024. In particular, L+M-24 is designed to focus on three key benefits of natural language in molecule design: compositionality, functionality, and abstraction

## introduction the world faces an enormous number of problems in the coming decades on scales of complexity never-before-seen, in areas such as climate change, healthcare, and pandemics. to address these issues, we need to discover inventive scientific solutions which are scalable, flexible, and inexpensive. broadly speaking, many of these problems will require molecular solutions from the chemistry domain, such as developing new drugs (e.g. kinase inhibitors (ferguson and gray, 2018)), materials (e.g. organic photovoltaics @xcite ), and chemical processes @xcite . these solutions exist in extremely large search spaces, which makes ai tools a necessity. language-molecule models have emerged as an exciting direction for molecular discovery and understanding @xcite @xcite @xcite @xcite @xcite @xcite . however, training these models is challenging due to the scarcity of molecule-language pair datasets. at this point, datasets have been released which are 1) small and scraped from existing databases @xcite @xcite , 2) large but noisy and constructed by performing entity linking on the scientific literature @xcite , and 3) templatebased built on prediction datasets @xcite . approaches utilizing pseudo-data have also been attempted @xcite . these approaches have helped remedy the problem of data scarcity in this domain; however, these approaches frequently ignore key benefits of natural language: 1) compositionality, 2) abstraction, and 3) functionality @xcite . to this end, for the language + molecules workshop at acl 2024, we release l+m-24, which we construct to test these three goals, particularly compositionality, using recently released data sources @xcite @xcite . l+m-24 is divided into four categories with important applications in the small-molecule domain: 1) biomedical, 2) light and electricity, 3) human interaction and organoleptics, and 4) agriculture and industry. improving understanding of these applications can have important implications in problems such as drug discovery, climate issues, more efficient and green industrial processes, and improved food production. ## task formulation the dataset is primarily intended for language↔molecule translation, which consists of two tasks: generating 1) a caption given a molecule and 2) a molecule given a description. ## data sources we constructed our dataset using three different databases. we will first describe the process we used to extract information from each, followed by our overall strategy for adding hierarchy into the dataset. we want to deeply thank the authors of these resources for making them publicly available for the community. ## evaluation metrics overall, we adopt the evaluation metrics proposed by @xcite . however, we include invalid molecules in the calculations of fts metrics (setting the score to zero for invalid molecules). we also add a uniqueness metric to the generated molecules for held-out combinations of properties @xcite . further, we also look at property-specific precision, recall, and f-1 scores. these scores are calculated by matching tokenized names in the generated captions. these scores are ## benchmarks molt5 models @xcite were finetuned for 20 epochs on the "split_train" data split and evaluated on the "split_valid", both of which are available online. huggingface's transformers @xcite was used for finetuning with a learning rate of 2e-5 and weight decay of 0.01. a batch size of 128 was used for small and base models, and a batch size of 48 for large models. further, meditron-7b @xcite was finetuned for 5 epochs with a context length of 930, 2e-6 learning rate, and batch size of 8/16 (molecule/caption generation). models are released online. results for captioning are reported in tables 2, 3 and 4. tables 5, and 6 shows results for molecule generation. overall, the dataset proves to be fairly challenging for these naively finetuned models. on captioning, meditron-7b achieves a maximum overall f-1 score of 16.81 for property identification ( ## future directions overall, this dataset proves to be quite challenging. we find that some specific properties in particular are challenging for the model. this may be because the model understands these properties, but is unwilling to use them in its descriptions due to the training procedure. this limitation may be addressed with more sophisticated decoding algorithms or by better finetuning methods. future work will also likely benefit from incorporating other modalities, such as proteins, to provide better understanding to the model for some property types. notably, certain properties display what may be emergent behavior; scaling training data or model size may yield non-linear improvements. in this dataset, we focus on composition, abstraction, and function. future work may also wish to integrate other recent trends: instruction-following and dialogue @xcite @xcite @xcite , tool use @xcite , additional molecule representations (e.g., 3d @xcite ), additional modalities @xcite , or molecule editing @xcite . further, we note the need for improved evaluation metrics, especially in the case of molecule generation for function where there may be many possible outputs. specific methods for improving compositionality may be another fruitful avenue for research @xcite . it may also be interesting to use molecule-language instruction-following models within larger search frameworks, such as chemreasoner @xcite . ## conclusion in this manuscript, we describe the process for creating the l+m-24 dataset. l+m-24 is designed to focus on three key benefits of natural language in molecule design: compositionality, functionality, and abstraction. it is the featured shared task at the first language + molecules workshop at acl 2024.