ALM ol: Aligned Language-Molecule Translation LLM s through Offline Preference Contrastive Optimisation (2024)

Abstract: The field of chemistry and Artificial Intelligence (AI) intersection is an area of active research that aims to accelerate scientific discovery. The integration of large language models (LLMs) with scientific modalities has shown significant promise in this endeavour. However, challenges persist in effectively addressing training efficacy and the out-of-distribution problem, particularly as existing approaches rely on larger models and datasets. In this context, we focus on machine language-molecule translation and deploy a novel training approach called contrastive preference optimisation, which avoids generating translations that are merely adequate but not perfect. To ensure generalisability and mitigate memorisation effects, we conduct experiments using only 10% of the data. Our results demonstrate that our models achieve up to a 32% improvement compared to counterpart models. Finally, we introduce a fine-grained, domain-agnostic evaluation method to assess hallucination in LLMs and promote responsible use.

## introduction the world is facing unprecedented complexity in the form of global challenges such as climate change, healthcare, and pandemics. innovative scientific solutions are urgently needed to address these challenges. chemistry has been at the forefront of developing such solutions, pioneering new drugs @xcite , creating advanced materials @xcite , or enhancing chemical processes @xcite . however, these frontiers are vast and require the involvement of artificial intelligence (ai) technology to navigate them effectively. large language models (llms) have shown promising potential for accelerating scientific discovery across various domains, including chemistry, biology, and materials science @xcite . existing work has applied successful paradigms from natural language processing (nlp) and multimodal representation learning to the chemistry domain. one common approach involves converting the inherent three-dimensional structures of molecules into smiles, which provide a mapping to symbolic character-level representations. subsequently, researchers have explored learning language-molecule representations either in separate yet coordinated spaces @xcite @xcite , in a joint space @xcite , or through hybrid approaches @xcite . in light of the recent significant advancements in the field, none of the above approaches effectively tackle the inherent challenges in training such models. instead, they rely on sparse or noisy synthetic data, often necessitating exponentially more data than is typically used in nlp tasks @xcite . however, training on larger models and datasets does not necessarily guarantee higher performance. a successful paradigm that augments the capabilities of llms across multiple nlp tasks is reinforcement learning with human feedback (rlhf) @xcite . although initially challenged by issues of slowness and instability, recent research has addressed many of these challenges by shifting towards closed-form losses that operate directly on offline preference data @xcite . rlhf has demonstrated superior performance compared to standard minimising crossentropy optimisation approaches. in this context, we address challenges related to effectively training robust language models when integrated with scientific modalities. we deploy a novel way of training llms for language-molecule translation that avoids generating translations that are only adequate but not perfect, called contrastive preference optimisation (cto) @xcite . cto is based on offline preferences instead of su-pervised fine-tuning, mimicking reference translations. to ensure that our models can effectively generalise instead of memorising patterns, we conduct experiments using only 10% of the l+m-24 dataset @xcite . our contributions have as follows: our empirical results demonstrate that our models consistently outperform the leading baseline, meditron, which is trained on the entire dataset, even in agnostic cross-modal scenarios ( § 4.4). ## background reinforcement learning with human feedback (rlhf) optimisation @xcite operates with a triple dataset @xmath0, y w , y l }, where y w and y l represent preferred and dis-preferred outputs, corresponding to input x, such that y w ≻ y l for x. the probability of y w over y l in pairwise comparisons is typically computed using the bradley-terry model @xcite : rlhf can present challenges due to its inherent slowness and instability, especially in distributed settings @xcite . recent work has shifted towards closed-form losses to align llms with human preferences. here, we experiment with contrastive preference optimisation that adopts a closed-form loss for rlhf. ## conclusion this work address training efficacy and the outof-distribution problem for automatic languagemolecule translation. we train models using only 10% of available data and deploying contrastive preference optimisation which avoids generating translations that are merely adequate but not perfect. we achieve significant improvement in performance when compared with models trained on extensive in and out-of-the-distribution data. finally, we propose a fine-grained, domain-agnostic evaluation method to assess hallucination in llms. our models show superior factual consistency for caption generation and character-level unigram overlaps for molecule generation.