Enhanced B io T 5+ for Molecule-Text Translation: A Three-Stage Approach with Data Distillation, Diverse Training, and Voting Ensemble (2024)

Abstract: This paper presents our enhanced BioT5+ method for the Language + Molecules shared task at the ACL 2024 Workshop. The task involves “translating” between molecules and natural language, including molecule captioning and text-based molecule generation using the L+M-24 dataset. Our method consists of three stages. In the first stage, we distill data from various models. In the second stage, combined with extra version of the provided dataset, we train diverse models for subsequent voting ensemble.We also adopt Transductive Ensemble Learning (TEL) to enhance these base models. Lastly, all models are integrated using a voting ensemble method. Experimental results demonstrate that BioT5+ achieves superior performance on L+M-24 dataset. On the final leaderboard, our method (team name: qizhipei ) ranks first in the text-based molecule generation task and second in the molecule captioning task, highlighting its efficacy and robustness in translating between molecules and natural language. The pre-trained BioT5+ models are available at https://github.com/QizhiPei/BioT5 .

## introduction with the development of large language models (llms) @xcite @xcite , the integration of molecules with natural language has garnered increasing attention in recent research efforts @xcite @xcite @xcite @xcite @xcite . notably, two critical generative tasks have emerged: molecule captioning (i.e., mol2text) and text-based molecule generation (i.e., text2mol) @xcite . these tasks are pivotal for biologists and chemists, as they facilitate the interpretation and creation of molecular structures through natural language descriptions. to leverage the advantages of natural language for molecular design and understanding @xcite @xcite , language + molecules workshop at acl 2024 has been organized. a shared molecule-text translation task and the corresponding paired dataset are presented to accelerate research in this field. ## methodology in this section, we give a detailed introduction to our three-stage methodology. notations. we use selfies @xcite as the sequence representation of the molecule. compared to smiles, selfies is a more robust molecular representation, which is beneficial for molecule generation tasks such as text2mol, as it ensures the generation of 100% valid molecules. the smiles in the l+m-24 dataset are converted to corresponding selfies using selfies toolkit foot_0 . let m and t denote molecular selfies and text descriptions, respectively, and m and t denote the corresponding collection of all sequences. let @xmath0, t i )} our goal is to develop a mol2text translation model f : @xmath1, which generates a caption from t for a given molecule from m , and a reverse text2mol translation model g : @xmath2. in this paper, all models follow the t5 @xcite -large architecture. our method consists of the following three stages: stage-1: data distillation. first, we train a mol2text translation model f 0 and a text2mol translation model g 0 on b. then, we use f 0 and g 0 to build the synthetic dataset: to further improve the diversity of the distilled data, we also use the officially provided meditron-7b @xcite mol2text model foot_1 f med to build another synthetic dataset: ## experiments in this section, we present our main results for the mol2text and text2mol tasks. baseline results on the validation set are derived from @xcite , and test set results are sourced from the official leaderboard. an ablation study is also conducted in section 4 to demonstrate the efficacy of our methodology. ## ablation study to validate the effectiveness of our tel training and voting ensemble, we conduct an ablation study for the mol2text task on the validation set of l+m-24. the results, shown in ## conclusion in this paper, we introduce our enhanced biot5+ model for the shared task of the language + molecules workshop at acl 2024. we adopt a three-stage approach: data distillation, diverse training, and voting ensemble. our method effectively leverages diverse datasets and advanced ensemble techniques to enhance model performance in both molecule captioning and text-based molecule generation tasks. experimental results show that our approach achieves superior performance across various evaluation metrics, highlighting the potential of our enhanced biot5+ model for integrating molecules and text.