CheMM-R1

CheMM-R1 is a chemistry-specific multimodal large language model (MLLM) for molecular structure recognition and spectral elucidation. It is built on top of Qwen2.5-VL-3B-Instruct and trained with CheMMGRPO — a domain-adapted Group Relative Policy Optimisation pipeline that combines a chemistry cold-start SFT stage with reinforcement learning driven by chemistry-specific reward functions.

The model is introduced in the paper "CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models".

• Code & benchmark: https://github.com/liting980713/CheMM-R1

Capabilities

CheMM-R1 is designed for the following tasks:

• SmilesQA — predict the SMILES string of a molecule from its 2D structure image.
• IupacQA — predict the IUPAC name of a molecule from its 2D structure image.
• MwQA — predict the molecular weight of a molecule from its 2D structure image.
• SpectraQA — predict the SMILES of a molecule from its spectral images (IR, ¹H-NMR, ¹³C-NMR, positive-ion MS, negative-ion MS).

The model produces explicit step-by-step reasoning in <think> tags, intermediate SMILES in <smiles> tags, and the final answer in <answer> tags.

Method

Base model

• Architecture: Qwen2_5_VLForConditionalGeneration (Qwen2.5-VL-3B-Instruct)
• Precision: bfloat16
• Context length: up to 128K tokens, trained with max sequence length 20,480 (cold-start) and output length 5,120 (RL)

Training pipeline (CheMMGRPO)

1. Cold start (SFT) — the base model is fine-tuned on 40,000 chemistry QA instances from CheMM-Bench with multimodal Chain-of-Thought reasoning distilled from Gemini-2.5-Pro, to inject organic chemistry knowledge and reasoning patterns.
2. Reinforcement learning (GRPO) — the cold-started model is further optimised with GRPO using four chemistry-specific reward functions:
   • smiles_acc — chemical validity of generated SMILES
   • atom_acc, func_group_acc — structural accuracy
   • output_format — format compliance with <think>/<smiles>/<answer> tags
   • answer_acc — factual correctness via Levenshtein-based fuzzy match for textual answers (SMILES, IUPAC) and ±0.05 g/mol tolerance for molecular weights

Key hyperparameters

• Framework: MS-SWIFT
• Cold start: lr 1e-4, Adam (β1=0.9, β2=0.95, ε=1e-8), batch size 64, max model length 20,480, max image pixels 262,144
• RL (GRPO): 4 rollouts per question, sampling temperature 1.0, lr 1e-4, KL coefficient β=0.001, batch size 48, max output length 5,120
• Hardware: 8× H200 (141 GB) GPUs, ~12 hours total training

Dataset: CheMM-Bench

CheMM-R1 is trained and evaluated on CheMM-Bench, a multimodal chemistry reasoning benchmark with 48,500 long Chain-of-Thought reasoning steps across four tasks:

• 26,500 molecules for structure recognition (SmilesQA / IupacQA / MwQA)
• 22,000 molecules for structure elucidation (SpectraQA) with five spectral image types (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS)

Molecules are derived from the Alberts spectroscopic dataset (USPTO reaction database). SMILES → 2D structure conversion is performed with RDKit; IUPAC names are sourced from PubChem.

Results

Averaged accuracy / F1 on CheMM-Bench (CheMM-R1 vs. strongest baselines):

Model	SR Avg ACC	SR Avg F1	SpectraQA ACC	SpectraQA F1	Overall ACC	Overall F1
GPT-o3	5.78	10.94	1.50	2.96	3.34	6.46
Gemini-2.5-Pro	16.13	27.78	1.80	3.54	7.95	14.72
Claude-Sonnet-4	1.99	3.91	1.60	3.15	1.77	3.48
Grok-4	2.79	5.43	4.05	7.78	3.51	6.78
Gemini-2.5-Flash	8.58	15.80	1.10	2.18	4.31	8.26
CheMM-R1 (3B)	23.73	38.35	36.32	53.28	30.92	47.23

Tanimoto@1.0 structural match on SmilesQA / SpectraQA:

Model	SmilesQA	SpectraQA
Gemini-2.5-Pro	44.52	11.46
Grok-4	2.18	18.28
ChemVLM-8B	31.99	—
CheMM-R1 (3B)	60.00	56.57

See the paper for full tables including BLEU-1 and Levenshtein-distance similarity and ablations on cold-start vs. GRPO vs. CheMMGRPO.

Usage

from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from PIL import Image
import torch

model_id = "zzha6204/CheMM-R1"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

image = Image.open("molecule.png")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is the SMILES representation of this molecule?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

output_ids = model.generate(**inputs, max_new_tokens=2048)
response = processor.batch_decode(
    output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
)[0]
print(response)

For SpectraQA, pass all available spectral images (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) as a multi-image message.

Output format

CheMM-R1 is trained to produce:

<think> step-by-step chemical reasoning </think>
<smiles> intermediate SMILES </smiles>
<answer> final answer </answer>

Downstream parsers should extract the content of <answer> as the final prediction.

Intended use and limitations

• Intended for research on multimodal chemistry reasoning: molecular structure recognition and spectral elucidation of small organic molecules.
• Molecules are drawn from the USPTO-derived Alberts dataset; out-of-distribution performance on larger natural products, organometallics, or experimentally noisy real-world spectra is not guaranteed.
• Outputs — including SMILES, IUPAC names, molecular weights, and reasoning traces — may be incorrect and must not be used for safety-critical decisions in chemistry or medicinal research without expert verification.
• The model is derived from Qwen2.5-VL-3B-Instruct and inherits its license and any biases of the base model and distilled reasoning data (Gemini-2.5-Pro).

Citation

If you use CheMM-R1 or CheMM-Bench in your research, please cite:

@inproceedings{huang2025chemmr1,
  title     = {CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models},
  author    = {Huang, Liting and Zhang, Zhihao and Wang, Shoujin},
  year      = {2025}
}

Acknowledgements

• Base model: Qwen2.5-VL-3B-Instruct
• Training framework: MS-SWIFT
• Spectroscopic data: Alberts et al. (USPTO-derived), PubChem, RDKit