manifesta/adaption_scientific_chart_qa_17k

Ground Truth: Scientific Chart QA 17k (Gemma 3 27B LoRA)

A PEFT LoRA adapter trained on 17,070 scientific chart questions, where 14.5% of the questions cannot be answered from the figure and the correct response is to say so.

It did not beat its base model. The head-to-head result is 50 wins to 50, a null result. This card explains why, using the weights themselves rather than a guess. The short version: the adapter never touched the vision tower, and a chart QA task lives in the vision tower.

Publishing a null result with the diagnosis attached is more useful than publishing nothing, so the weights, the dataset, the training state and the failure analysis are all here.

Contents

• TL;DR
• Quickstart
• What is actually in these weights
• Evaluation
• Why the null result
• The cross-run finding
• Training
• Intended use and limits
• Dataset
• Live interface
• Related models
• License
• Citation

TL;DR

• What it is: a rank-8 LoRA adapter on google/gemma-3-27b-it, 8,380,416 trainable parameters, 33.5 MB. Text-side attention only.
• What it was for: answering questions about scientific figures, including the ones that cannot be answered, where "cannot be determined" is the correct answer.
• The number: 50 wins for the adapted model, 50 for the base. No improvement. Held-out eval loss did fall, 1.699 to 1.427, which turned out not to matter.
• The reason: the adapter has zero vision tensors. All 27 layers of the SigLIP vision tower and the multimodal projector are named in exclude_modules and were frozen. Only q_proj and v_proj in the 62 language-model decoder layers were trained.
• Base model pointer, corrected: adapter_config.json was exported declaring togethercomputer/gemma-3-27b-it-VLM, which does not exist. I changed it to google/gemma-3-27b-it on 2026-08-13 and kept the original string verbatim in the same file. See Quickstart.
• Built by: MANIFESTA (Aivaras Navardauskas) for the Adaption AutoScientist Challenge, Data Visualization.

Quickstart

The base model string was wrong. I fixed it, and the original is on the record

The export pipeline wrote this into adapter_config.json, and it shipped that way from 2026-08-06 to 2026-08-13:

"base_model_name_or_path": "togethercomputer/gemma-3-27b-it-VLM"

That repository does not exist. It returns HTTP 401 to an anonymous client, which is what the Hub returns for a repo that is not there, and togethercomputer publishes no gemma-3 repo under any name. google/gemma-3-27b-it-VLM, which the original auto-generated card advertised, does not exist either. Nobody typed that string by hand. The pipeline invented it, and I did not catch it before publishing.

The consequence was real: anything resolving the base from the config, including AutoPeftModel...from_pretrained on the adapter alone, could not find a base to load.

Corrected 2026-08-13. The file now reads:

"base_model_name_or_path": "google/gemma-3-27b-it",
"original_base_model_name_or_path": "togethercomputer/gemma-3-27b-it-VLM"

No weight changed and nothing else in the config changed. The original exported value stays in the same file so the artifact remains auditable: you can see what shipped first, what it was changed to, and decide for yourself whether the change was right.

Check both in one line each:

curl -s -o /dev/null -w "%{http_code}
" https://huggingface.co/api/models/google/gemma-3-27b-it              # 200
curl -s -o /dev/null -w "%{http_code}
" https://huggingface.co/api/models/togethercomputer/gemma-3-27b-it-VLM # 401

The real base is google/gemma-3-27b-it. The config.json shipped in this repo matches it exactly: 62 decoder layers, hidden size 5376, 32 attention heads, 16 key-value heads, head dim 128, vocab 262,208, and a 27-layer SigLIP tower at 896 px with patch size 14. The LoRA shapes agree too, q_proj projects to 4096 (32 x 128) and v_proj to 2048 (16 x 128).

Working load snippet

Name the base explicitly. Note that this is a vision language model, so use the image-text-to-text auto class, not AutoModelForCausalLM.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

BASE = "google/gemma-3-27b-it"        # gated, request access on the Hub first
ADAPTER = "manifesta/adaption_scientific_chart_qa_17k"

processor = AutoProcessor.from_pretrained(BASE)
base = AutoModelForImageTextToText.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "https://upload.wikimedia.org/wikipedia/commons/a/a4/Line_chart_example.png"},
        {"type": "text", "text": "What is the value of the blue series at x = 4? If the figure does not show it, say 'cannot be determined'."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Naming the base explicitly is still the snippet I would use, because google/gemma-3-27b-it is gated and you need to accept the licence on the Hub before anything can pull it.

Merging is optional and only touches the language model:

model = model.merge_and_unload()

What is actually in these weights

Read straight out of adapter_model.safetensors and adapter_config.json. You can reproduce every row of this table from the repo.

Property	Value
PEFT type	LoRA, task_type: CAUSAL_LM
Rank r	8
lora_alpha	8 (so the scaling factor alpha/r is exactly 1.0)
lora_dropout	0.0
bias	none
use_rslora / use_dora	false / false
Target modules	q_proj, v_proj
Tensors in the file	248
Decoder layers touched	62 of 62 (every layer, both modules, no gaps)
Vision tensors	0
Trainable parameters	8,380,416 (float32, 33.5 MB)
Share of the 27B base	roughly 0.031%
exclude_modules	334 entries, all of them vision

Every tensor key in the file starts with base_model.model.model.language_model.layers.. Not one starts with vision_tower, multi_modal_projector, or anything else on the image path.

The exclusion is explicit rather than accidental. exclude_modules contains 334 module names and every single one is part of the image pathway:

• all 27 SigLIP encoder layers, each with self_attn.{q,k,v,out}_proj, mlp.fc1, mlp.fc2, both layer norms and the activation
• model.vision_tower.embeddings.position_embedding
• model.vision_tower, model.vision_tower.encoder, model.vision_tower.encoder.layers
• model.multi_modal_projector.avg_pool

So the vision tower and the projector that feeds image tokens into the language model were both frozen by configuration, before a single gradient was computed.

Evaluation

Judged by Adaption's AutoScientist evaluator on a held-out split the adapter never saw, against the same base model it was trained from. Head-to-head, blind.

Metric	Base	Adapted	Change
Win rate	50	50	0
Held-out eval loss (first eval)	1.6992
Held-out eval loss (final)		1.4272	-0.272
Training loss	1.871 (step 1)	1.420 (step 34)	-0.451

Eval loss across the five checkpoints: 1.6992, 1.5371, 1.4683, 1.4370, 1.4272. Monotonic, textbook, and it bought nothing.

A 50 to 50 split is a tie, and a tie against your own base model is a null result. The adapter did not make Gemma 3 27B better at reading scientific figures. It also did not make it worse. Loss went down the whole way, which is exactly the trap: a clean loss curve tells you the model learned the answer format in the training data, not that it learned to see.

Why the null result

The task and the trained parameters do not overlap.

Answering a question about a scientific figure means doing four things: locating an axis and reading its scale, matching a plotted series to a legend entry, interpolating a value at a point, and deciding whether the thing being asked about is present in the figure at all. All four are perception. All four depend on the SigLIP tower and on the multimodal projector that turns 256 image tokens into something the decoder can attend over.

Neither of those was trained. What was trained is q_proj and v_proj inside the text decoder, which can change how the model phrases an answer, how it structures a derivation, and how readily it says "cannot be determined". It cannot change what the model sees.

So the honest reading is that this run had a ceiling built into it from the start. The dataset was not the bottleneck, and neither, mostly, was the step count. The trainable surface was.

A secondary factor worth naming: the run took 34 optimizer steps. With 16,684 rows in the training split and batch_size: "max", that is roughly 491 rows per optimizer step. Adding data widened the batch instead of adding steps, which is the subject of the next section.

What would need to change for a real result: put LoRA on the vision tower and the multimodal projector, or at minimum on multi_modal_projector, and force more optimizer steps by capping batch size. Until the image path is trainable, more chart data is not going to move the number.

The cross-run finding

Three completed AutoScientist runs, three different base models. Win rate tracked how weak the base already was on the domain, and did not track dataset size.

Run	Base	Params	Rows ingested	Steps	Rows per step	Win rate (adapted vs base)
Chart QA, first build	Qwen3.5 9B	9B	6,976	21	332	51 vs 49
Chart QA 17k (this model)	Gemma 3 27B	27B	17,070	34	502	50 vs 50
Verified math and code	Gemma 4 31B	31B	17,586	59	298	46 vs 54

Two things fall out of that table.

The bigger the base, the smaller the win. 9B scored 51, 27B scored 50, 31B scored 46. A stronger base has less room above it, and a rank-8 adapter on two projections is not enough to add capability the base does not already have. It can only redistribute what is there.

Scaling the corpus 2.4x did nothing. The chart corpus went from 6,976 rows to 17,070 rows, and the win rate went from 51 to 50. Because batch_size is set to max, the extra rows were absorbed into wider batches: step count only went 21 to 34 while rows per step went 332 to 502. More data per step is not more learning, it is a smoother gradient estimate for the same small number of updates.

Caveat, stated plainly: this is three data points from three different base models, three different corpora and three different domains. Base size, base family and task difficulty all move together here. It is not a controlled experiment and it should not be read as one. It is a pattern worth testing properly, which would mean holding the corpus fixed and varying only the base.

Training

Trained with Adaption AutoScientist, SFT with LoRA, on Adaptive Data output.

Hyperparameter	Value
Base model	google/gemma-3-27b-it, declared correctly in adapter_config.json since 2026-08-13. The original export value togethercomputer/gemma-3-27b-it-VLM is preserved in the same file as original_base_model_name_or_path
Method	SFT, LoRA (PEFT), chat format
LoRA rank / alpha / dropout	8 / 8 / 0.0
Trainable modules	q_proj, v_proj
Learning rate	1e-4
Scheduler	cosine, 0.5 cycles, warmup ratio 0.1, min LR ratio 0.1
Weight decay	0
Max gradient norm	2
Epochs	1
Batch size	max (per-device train batch size 1 with accumulation)
Optimizer steps	34
Evaluations	5, every 6 steps
Train on inputs	false
Precision	bfloat16 base, float32 adapter weights
Total FLOPs	1.389e18
Adaption job ID	de678a1e-4280-4a33-954e-8096e018f987
Training experiment ID	d2500abf-4185-4437-bc6d-3a1f85e8e02d
Adaption dataset ID	3f347c8b-5724-4f96-9417-251623d8aaa5

Numerical stability

This run was clean. Gradient norm peaked at 0.72 against a clipping threshold of 2, and not one of the 34 steps was clipped. Range across the run was 0.243 to 0.720.

That matters because it rules out the easy explanation. Nothing here blew up. The optimizer did what it was told and the loss came down smoothly. The null result is not a broken run, it is a run that trained the wrong parameters. Compare with the math and code adapter, where 29 of 59 steps exceeded the clip threshold and the peak norm was 677.

(The Adaption run dashboard reports the peak as 0.79. The value above, 0.72, is the maximum in trainer_state.json in this repo, which is the number you can check yourself.)

Intended use and limits

Use it for

• Reproducing and inspecting this null result. That is the honest primary use.
• Studying what a language-side-only LoRA does to a vision language model's answer behaviour, particularly refusal phrasing.
• A starting point for a run that actually adapts the vision tower.
• Research on chart question answering and on unanswerable-question handling.

Do not use it for

• Anything that assumes it reads figures better than stock Gemma 3 27B. It does not, and the evaluation says so.
• Extracting numbers from figures in medical, financial, safety or regulatory work. A misread axis is a wrong number delivered confidently.
• Presenting model-read values from a chart as measured data.
• Any use that violates the Gemma Terms of Use or the Gemma Prohibited Use Policy, which apply through the base model.

Technical limits

• Vision tower and multimodal projector are frozen and untrained. This is the headline limitation.
• Rank 8 on two projections, roughly 0.031% of base parameters, one epoch, 34 optimizer steps. A very light touch.
• English only.
• The declared base string was wrong until 2026-08-13. It resolves now, and the original is recorded in the config.
• Adapter weights are float32 while the base is bfloat16. PEFT casts on load, but expect the memory footprint of the adapter to be double what a bf16 export would be.
• One judge, one held-out split. Reported as-is, no reruns, no best-of selection.

Dataset

Trained on Adaption dataset 3f347c8b-5724-4f96-9417-251623d8aaa5, built from manifesta/scientific-chart-qa-17k (also on Kaggle).

• 17,070 rows ingested, 16,684 rows in the training split after adaptation
• Domain mix: data-analysis-visualization 87%, science 8%, academic-education 3%, medical 1%, technology 1%
• 14.5% of questions are unanswerable by design, where the correct output is a refusal
• 58.2% are exactly gradable against a known numeric answer
• 6,426 real figures (CC-BY 6,220, CC-BY-SA 164, CC0 42) plus 10,644 synthetic figures

Adaptive Data lifted the corpus quality before training:

Adaptive Data metric	Before	After
Quality score	6.0	7.1 (+18.3%)
Grade	C	B
Percentile	8.2	9.3

Worth separating two numbers that are easy to conflate: +18.3% is the improvement in the data, not in the model. The data got better. The model did not. Those are different measurements and only one of them is the headline.

Live interface

Ground Truth runs at manifestavisual.adaptionlabs.app. Drop a figure in, ask a question, and use the "Try to trick it" button to ask for a series that was never plotted.

The green "Correctly refused" card above is the behaviour the dataset was built around: asked about a series called Zephyr-net that does not appear in the grouped bar chart, the answer is "cannot be determined" rather than a confident invention.

Related models

• manifesta/scientific-chart-qa-lora-qwen3.5-9b, the same task on a 9B base with the first 6,976-row build. 51 vs 49, 21 optimizer steps. Its LoRA landed on only 8 of 32 decoder layers (indices 3, 7, 11, 15, 19, 23, 27, 31) and, like this one, on zero vision tensors. Its adapter_config.json carried the same export defect, togethercomputer/Qwen3.5-9B, and was corrected to Qwen/Qwen3.5-9B on the same day with the original preserved the same way.
• manifesta/adaption_verified_math_code_instruct, the math and code run on Gemma 4 31B. 46 vs 54, a regression, with a very different failure mode.
• manifesta/brandvoice-marketing-model, the Part 1 marketing adapter that did work, 56% win rate on Llama 3.3 70B.

Two chart runs, two model families, and in both cases the vision tower was never adapted. That is the finding this repo exists to record.

Everything behind these weights is public

The dataset this adapter was trained on, the scripts that built it, and a verifier that rechecks every number claimed here against the live artifacts:

https://github.com/A1VARA5/scientific-chart-qa-17k

git clone https://github.com/A1VARA5/scientific-chart-qa-17k
cd scientific-chart-qa-17k
python verify.py

Standard library only, no install step and no account. 16 checks, and the same 16 run on a daily schedule in GitHub Actions, so the badge above goes red if any claim on this card stops being true. The tensor facts in this card are among the checks: layer coverage is re-derived from the published adapter_model.safetensors by name, not copied from the config.

Dataset: https://huggingface.co/datasets/manifesta/scientific-chart-qa-17k

License

• Adapter weights: released under the Gemma Terms of Use. Derivatives of Gemma inherit Gemma's terms, so the adapter travels with the same conditions as the base.
• Base model: google/gemma-3-27b-it is gated. Accept the licence and request access on the Hub before loading. The Gemma Prohibited Use Policy applies.
• Training dataset: manifesta/scientific-chart-qa-17k. Source figures carry their own licences (CC-BY, CC-BY-SA, CC0) and are attributed per row in the dataset.

Citation

@misc{groundtruth_chartqa_2026,
  title  = {Ground Truth: a scientific chart QA adapter for Gemma 3 27B, and why it produced a null result},
  author = {Navardauskas, Aivaras},
  year   = {2026},
  note   = {MANIFESTA. Adapted with Adaptive Data by Adaption, AutoScientist Challenge, Data Visualization. Win rate 50 vs 50. Vision tower frozen.},
  howpublished = {\url{https://huggingface.co/manifesta/adaption_scientific_chart_qa_17k}}
}

Built with Adaptive Data by Adaption. Platform documentation at docs.adaptionlabs.ai. Dataset, adapter and analysis by MANIFESTA (Aivaras Navardauskas) for the AutoScientist Challenge.