M olecule QA : A Dataset to Evaluate Factual Accuracy in Molecular Comprehension (2024)

Abstract: Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information. Traditional evaluations fail to assess a model’s factual correctness. To rectify this absence, we present MoleculeQA, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative corpus. MoleculeQA is not only the first benchmark to evaluate molecular factual correctness but also the largest molecular QA dataset. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific aspects and pinpoints crucial factors for molecular modeling. Furthermore, we employ MoleculeQA in reinforcement learning to mitigate model hallucinations, thereby enhancing the factual correctness of generated information.

## introduction large language models (llms) have practical applications in molecule research @xcite , specifically in the use of cross-modal models @xcite . these models bridge the gap between molecular structures and natural language (bran and schwaller, 2023), helping experts understand the properties, potential applications, and acquisition methods of specific compounds, improving understanding of the complex molecular architecture and reducing experimental failure rates. ## analysis we propose the following research questions (rqs) for the molecular domain to guide our analysis: ## conclusion and future work in conclusion, this paper addresses the absence of evaluation for factual correctness in large language models (llms) within the molecular domain. by organizing molecular descriptions into a taxonomy and constructing qa pairs through human and llm efforts, we introduce moleculeqa, a novel dataset for molecular factual question answering. our evaluation reveals shortcomings of existing models, emphasizing critical factors for molecular comprehension and providing guidance for molecular llms' development. we also make preliminary attempt to alleviate the hallucinations in molecular llms based on moleculeqa. looking forward, we propose three future directions: (1) design a powerful molecular model based on our analysis. (2) investigate more and better methods to apply moleculeqa for the optimization of molecular llms. (3) incorporate additional data sources to enrich moleculeqa's comprehensiveness.