M ol2 L ang- VLM : Vision- and Text-Guided Generative Pre-trained Language Models for Advancing Molecule Captioning through Multimodal Fusion (2024)

Abstract: This paper introduces Mol2Lang-VLM, an enhanced method for refining generative pre-trained language models for molecule captioning using multimodal features to achieve more accurate caption generation. Our approach leverages the encoder and decoder blocks of the Transformer-based architecture by introducing third sub-layers into both. Specifically, we insert sub-layers in the encoder to fuse features from SELFIES strings and molecular images, while the decoder fuses features from SMILES strings and their corresponding descriptions. Moreover, cross multi-head attention is employed instead of common multi-head attention to enable the decoder to attend to the encoder’s output, thereby integrating the encoded contextual information for better and more accurate caption generation. Performance evaluation on the CheBI-20 and L+M-24 benchmark datasets demonstrates Mol2Lang-VLM’s superiority, achieving higher accuracy and quality in caption generation compared to existing methods. Our code and pre-processed data are available at https://github.com/nhattruongpham/mol-lang-bridge/tree/mol2lang/.

## introduction in the field of cheminformatics, molecule captioning plays a crucial role in helping researchers by automatically generating captions for molecular structures. the accuracy and quality of these captions are vital as they directly impact the understanding of chemical information and scientific discoveries. traditional techniques primarily rely on unimodal data, often focusing only on textual representations like smiles (simplified molecular input line entry system) @xcite strings or selfies (self-referencing embedded strings) @xcite strings. although these methods have shown satisfactory results, their dependence on a single modality limits the richness and accuracy of the generated captions. the rise of multimodal data, which uses information from different sources, presents an opportunity for significant advancements in molecule captioning. multimodal approaches integrate various forms of molecule, enabling a more comprehensive understanding of molecular characteristics. however, effectively utilizing multimodal data in generative models is challenging and requires advanced techniques to integrate and improve the models effectively. in this paper, we introduce an enhanced methodology, named mol2lang-vlm, to improve generative models in molecule captioning by utilizing multimodal features. our approach integrates selfies strings and high-level features from molecular images in the encoder, while incorporating smiles features and corresponding descriptions in the decoder. this multimodal integration allows the model to have a deeper understanding of chemical structures within the generative model, which is further refined during the decoder stage. ## results and discussion we also evaluate our proposed method, along with all three baseline models and biot5, on the chebi-20 dataset. moreover, we conduct an ablation analysis to evaluate mol2lang-vlm with and without employing the forget gate. ## conclusion this paper introduced mol2lang-vlm, a visionand text-guided generative pre-trained language model designed to enhance molecule captioning performance through multimodal fusion. our proposed approach achieved comparative results in terms of bleu, rouge, meteor, and text2mol metrics, demonstrating its effectiveness in generating accurate and meaningful captions for molecular structures. the findings highlight the potential of mol2lang-vlm in advancing molecule captioning tasks. future research can explore alternative fusion methods, fine-tuning strategies, and the generalization of the model to other tasks. additionally, integrating mol2lang-vlm with downstream applications and enhancing interpretability can further enhance its practical utility in the field of cheminformatics. sity (skku) and the bk21 four project. this work was supported by the korea bio data station (k-bds) with computing resources including technical support.