Enhancing Cross Text-Molecule Learning by Self-Augmentation (2024)

Abstract: The development of Large Language Models (LLMs) has greatly advanced the field of drug discovery, with the belief that natural language can enhance human control over molecule design. However, the scarcity of high-quality labeled data remains a challenge for cross text-molecule learning. Existing datasets are limited due to the difficulty of collecting precise molecule-description pairs. Although recent efforts have utilized pseudo data generated by LLMs for augmentation, the lack of specialized chemistry knowledge of LLMs and the absence of an effective high quality data selector may introduce noise into the annotations, compromising the models’ robustness. To address these challenges, this paper introduces a novel framework that interweaves model fine-tuning and data augmentation to overcome the scarcity of high-quality data. The proposed approach involves an iterative procedure where the model plays dual roles in annotating unlabeled data and sampling a subset of high-quality data until convergence is achieved, enhancing the model’s understanding and adaptability. Additionally, a new dataset called SAPubChem-41 is presented, which comprises meticulously curated high-quality parallel molecule-description pairs designed specifically for fine-tuning purposes. This research provides an important contribution to the field by addressing the need for high-quality datasets and presenting an effective framework for cross text-molecule learning.

## introduction the emergence of large language models (llms) has significantly propelled the development of drug discovery @xcite . recent progress in language models shed light on drug discovery, with the vision that humans can possess a higher-level control over molecule design facilitated by natural * corresponding author. language. @xcite introduce two new tasks: molecule captioning (mol2cap) and text-based molecule generation (cap2mol), and present molt5 @xcite based on the t5 @xcite architecture to translate between molecule and text. subsequently, diverse endeavors have been undertaken to tackle these challenges. text+chem t5 @xcite and biot5 @xcite are also t5-like models, which incorporate multi-task and multidomain pretraining process and enable the bidirectional generation between different modalities in a single model. some other methods employ the generative pretrained transformer (gpt) architecture. molxpt @xcite proposes to wrap molecules in the sentences to make the pertaining corpus. more recently, molca @xcite considers the 2d graph information for molecules and designs a projector to connect molecular content to language models. these studies mark the research efforts in the realm of cross text-molecule learning. despite rapid development in cross textmolecule modalities, there is still demand for highquality labeled data in this field. such data of superior quality is indispensable to capacitate models for downstream tasks. however, the available datasets relevant to the tasks are limited, a consequence of the arduous nature of collecting precise and enlightening parallel molecule-description pairs. this scarcity of annotated data hinders the model's ability to learn between the modalities of the molecule and the text. as shown in figure 1 , recent efforts by @xcite address the scarcity challenge by utilizing pseudo data generated by llms as augmentation. the problem is that the absence of specialized knowledge in chemistry may introduce noise into the data annotated by general large language models, thereby compromising the robustness of models fine-tuned on such datasets. in spite of these commendable advance-ments, the accessibility of high-quality datasets for fine-tuning remains notably deficient. to address these challenges, this paper introduces a novel framework that interweaves model fine-tuning and data augmentation (see fig. 1 , aiming to rectify the high-quality data scarcity observed in the realm of cross text-molecule learning. our proposed approach integrates the model into the process by assigning it dual roles: annotating unlabelled data and sampling a subset of high-quality data. this iterative procedure continues until convergence is achieved, thereby refining the model's understanding and adaptability. additionally, we present a new, larger dataset sapubchem-41. this meticulously curated dataset comprises a wealth of high-quality parallel molecule-description pairs meticulously designed for the specific purpose of fine-tuning. the model optimized using sapubchem-41 has been evaluated across a total of 13 molecule-text benchmarks, exhibiting superior performance in comparison to the model optimized using the original dataset, chebi-20. our key contributions can be summarized as: ## method in this section, we describe a computational framework for jointly optimizing model performance and dataset quality. ## experiments in order to validate the efficacy of utilizing our framework, we have undertaken a series of comprehensive experiments. ## conclusion in this paper, we have explored and grappled with the challenge of a deficiency in the accessibility of high-quality datasets for model fine-tuning in the field of cross text-molecule learning, by developing a novel framework. this innovative framework efficiently combines model fine-tuning with data augmentation, integrating the model in such a way that it expands its roles to include annotating unlabeled data and sampling a subset of high-quality data. the paper further introduces a high-quality dataset, the sapubchem-41, expressly designed for fine-tuning purposes. the experiments' results were impressive as the model optimized using sapubchem-41 consistently outperformed the model optimized using the original dataset, chebi-20. going forward, this paper's findings present crucial steps toward addressing high-quality data scarcity for fine-tuning and ensuring further advancement in the exciting field of cross text-molecule learning.