Knowlab’s Submission to L + M Shared Task: All you need is continued pretraining of chemistry texts even for molecule captioning (2024)

Abstract: This paper presents our submission to the L+M-24 shared task, focused on translating molecular structures into natural language descriptions, known as the molecule captioning task. We selected a small language model (SLM), Phi-3-mini-4k, to evaluate the impact of continued pretraining and instruction tuning for domain-specific chemical knowledge. The Phi-3 model was continued pretrained with 90M chemistry textbooks and abstracts, followed by instruction tuning on 150K question answering sets of SMILES and general chemistry knowledge. Despite the continued pretraining phase not including direct exposure to SMILES representations, it significantly enhanced the Phi-3 model’s performance, a 300% increase for the BLEU scores, in the molecule captioning task. The code and model are released at https://github.com/bluesky333/Phi3KnowChem to facilitate research in chemical small language modeling.

## introduction the intersection of natural language processing (nlp) and chemistry began with drug discovery and biochemistry but recently moved to the other fields of chemistry such as electrochemistry for battery and rheology for chemical property prediction @xcite @xcite . with the recent advancement of large language models (llms), the language model for chemistry domain knowledge started to cover molecule representation such as the simplified molecular-input line-entry system (smiles) and 3d structure of molecules @xcite @xcite . as the research in llms has been facilitated by the benchmark datasets for evaluating the model's understanding of domain knowledge, there has been a pressing need for a benchmark specifically for molecule and language models @xcite . to address this gap, the l+m-24 shared task was introduced as one of the first competitions focused on translating between language and molecule representations @xcite . this task involves generating captions based on input molecules represented in smiles, pushing the boundaries of molecule captioning by leveraging language models. the task covers four key applications within chemical knowledge: biochemistry, electrochemistry, organoleptics, and agricultural chemistry. progress in these specific is essential for building foundational models applicable to small molecule applications. traditionally, models designed for such tasks require extensive domain-specific pretraining and fine-tuning with molecule representation to understand and generate chemistry-related text effectively @xcite @xcite . this process is often resource-intensive and requires large, specialized datasets. in this work, we explore the efficacy of continued pretraining and instruction tuning on a small language model (slm), specifically the 3.8b parameter model, phi-3-mini-4k, for the molecule captioning task. our approach involves two primary stages: the resulting models underwent fine-tuning with the shared task's training data for 1 epoch, and our best-performing model surpassed the performance of the molt5-base model, which was trained with 100 million smiles strings @xcite . the contributions of this paper are as follows: our study demonstrates that continued pretraining using a chemical text corpus significantly enhances performance in molecule captioning tasks, even without direct exposure to molecular representations. we saw an almost 300% increase for bleu scores, about a 67% increase for the rouge score, and a 19% increase for meteor with the continued pretraining. ## conclusion in this paper, we have presented our approach and findings from experimenting with the phi-3-mini-4k model on the molecule captioning task as part of the l+m-24 shared task. our experiment focused on evaluating the efficacy of continued pretraining and instruction tuning for enhancing the model's domain-specific chemical knowledge and its ability to generate accurate molecular descriptions. the results demonstrated that both continued pretraining and instruction tuning play critical roles in enhancing the performance of the phi-3 model on molecule captioning tasks. continued pretraining with a chemistry-specific corpus provides a substantial boost by enriching the model's knowledge base, while instruction tuning with targeted question-answer pairs refines its ability to handle specific queries related to chemical structures. the addition of beam search decoding, though providing marginal gains, contributes to producing higherquality and more accurate descriptions. these findings highlight the importance of domain-specific training and advanced decoding strategies in improving the capabilities of language models for specialized tasks like molecule captioning. in fact, on the leaderboard, our phi-3-knowchem outperformed the molt5-base model. nevertheless, while these findings provide valuable insights, further in-depth analysis is warranted to explore the nuances of model performance in the chemical domain fully. the exploration of other tasks such as chemical property prediction can contribute to more accurate and comprehensive assessments of lm performance in real-world chemical applications.