rodriguescarson/polychart-shown-is-not-supported-lora

PolyChart — Shown Is Not Supported (LoRA)

A LoRA adapter for Llama-3.3-70B-Instruct that reads charts under a measure-then-claim contract: a claim is made only when the underlying values support it, and the distance between what a chart shows and what its data supports is reported as a graded quantity rather than a yes/no label.

Trained on Adaption's AutoScientist for the AutoScientist Challenge 2026.

Scores

	Base	This adapter
On-dataset win rate	15	85
Held-out category win rate	22	78

Read 78, not 85. The two measure different things and this is the most important caveat on this model. The on-dataset figure is computed on the training distribution; the held-out figure is computed on data the model never saw. In the study behind this adapter they sometimes moved in opposite directions — one matched pair went 87.2 → 81.0 on-dataset while going 66.3 → 77.3 held-out. Optimising the displayed number can select the weaker model.

What is novel here: severity as a continuous target

Every released misleading-chart dataset we know of, including Misviz (ACL 2026), asks a yes/no question: is this chart misleading? This model is trained on Tufte's Lie Factor as a continuous, signed target — −1.0 where an inverted axis reverses a real trend, 0.017 where an expanded axis flattens one, 4.69 where a truncated baseline invents one. A binary label collapses all three into the same bit.

Training composition is failure-weighted. Scoring the base model cell-by-cell over question-type × distortion located the hard cells — continuous severity estimation, precise value extraction under a distorted axis, and mechanism naming — and the 2,129-row dataset was weighted toward them (severity 27%), with already-solved cells removed.

That targeting is measurable: the uniform version of this dataset on a 109B base (Llama-4-Scout) reached 76 held-out; this failure-weighted version on a smaller 70B base reached 78. Capacity sets the ceiling; composition decides where in that ceiling you land.

Training

Base	Llama-3.3-70B-Instruct
Method	LoRA, r=64, alpha=128
Target modules	all linear
Epochs	3
Rows	2,129, failure-weighted
Eval loss	0.944 → 0.861 (no overfit)

Every row passed an independent verifier sharing no code with the generator. It caught nine silent failures in our own pipeline before shipping, four of them inside its own scaffolding.

A retraction

An earlier version of this card claimed "curation beats scale," on the basis that expanding a seed from 1,882 to 21,882 rows dropped the held-out category from 76 to 72. That claim is withdrawn.

It rested on a single unpaired comparison. Repeating the test properly — paired within seed, across 13 seeds and 176 fine-tuning jobs — reversed the conclusion: augmentation improved the held-out score on 13 of 13 seeds, mean +14 points, two-sided sign test p = 0.0002, and the platform's advertised "+16 points from crossing 20,000 datapoints" reproduced at +17.4.

Held-out difficulty spans 44.5 to 85.9 across those seeds, roughly three times the effect being measured, so any cross-seed comparison mostly reports which seeds landed in which arm. That is precisely the error the original claim made.

What survives is narrower and is stated above: curation decides where within a base's ceiling you land, and expansion helps most where the seed under-covers its category. The two are not competing prescriptions. A model whose thesis is shown is not supported owes that test to its own claims first, which is why the retraction is recorded here rather than quietly deleted.

Intended use and limits

Chart-interpretation QA and misleading-encoding detection. Not a general-purpose assistant.

• One task family, one platform, one base model.
• Held-out evaluations fire on roughly a third of jobs and cannot be triggered on demand, so several seeds contribute one draw per arm against a metric with sd 3–5 noise. Direction is reported rather than a pooled effect size.
• The evaluation domain is assigned by the platform, not chosen. Across six purpose-built datasets, five were scored against general and none against its own topic — so a held-out number is interpretable only alongside the domain it was computed against.
• Inherits the biases and failure modes of Llama-3.3-70B-Instruct.

Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "meta-llama/Llama-3.3-70B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "rodriguescarson/polychart-shown-is-not-supported-lora")
tok = AutoTokenizer.from_pretrained(base)

Links

• Client + platform findings: https://github.com/rodriguescarson/adaption-autoscientist-api
• Live Lie Factor demo: https://lie-factor.carsonrodrigues.com
• Datasets: https://huggingface.co/rodriguescarson

License

Llama 3.3 Community License, inherited from the base model.