flame-27m-grounded

A 27.1M-parameter English decoder trained to answer only from the context it is given — built as the language half of a small vision-language model, where the context will be visual tokens instead of text. Fine-tuned from flame-27m-instruct on grounded question-answering built from Visual Genome scene graphs.

This is not a replacement for flame-27m-instruct. It answers far more briefly (mean 4.9 words vs 11.9) and is tuned to refuse rather than guess. For open-ended chat, use the instruct model.

• Architecture: Llama-style decoder — hidden 512, 8 layers, 8 heads / 2 KV heads (GQA), intermediate 1280, RoPE (θ=1e6), context 2048, vocab 12000 (English BPE). Identical to Qwen3ForCausalLM, so vLLM / SGLang / TRL work directly.
• Training: flame-27m-instruct → 1 epoch SFT on 297k conversations: 65 % grounded QA from Visual Genome scene graphs rendered as text, 35 % chat replay to preserve general ability.

What it is for

A decoder with an encoder bolted on will hallucinate whatever the decoder is willing to assert. The instruct model claimed an absent object was present 96 % of the time, which makes it unusable as a VLM decoder however good the encoder is. This model was trained to fix three specific behaviours, measured on a held-out split of 11,780 questions over 2,032 unseen images:

behaviour	flame-27m-instruct	flame-27m-grounded
overall grounded accuracy	25.06 %	79.26 %
POPE object-hallucination rate	96.06 %	21.60 %
says "no" about an absent object	3.94 %	78.40 %
says "I can't tell" when unanswerable	0.09 %	74.98 %
denies a false premise	0.92 %	86.11 %
over-refusal on answerable questions	2.33 %	3.24 %
format adherence	51.27 %	94.50 %

It did not learn to dodge: the yes-ratio is 50.6 % and over-refusal only 3.24 %, so it discriminates rather than adopting a blanket "no" or refusing everything.

General ability is unchanged. Average over hellaswag / arc_easy / arc_challenge / piqa / winogrande / openbookqa / mmlu: 36.00 vs the instruct model's 36.06 — every task within ±1.3. IFEval is slightly higher (instruction-level 0.48 vs 0.24). Note that a 27M model is near chance on most of these, so this rules out a regression rather than demonstrating general strength.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ifx-pse-sys-ml/flame-27m-grounded", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("ifx-pse-sys-ml/flame-27m-grounded")

scene = "The image shows a fence, a blue bus, some trees and a sidewalk."
msgs = [{"role": "user", "content": f"{scene}\nIs there a person in this picture?\nAnswer yes or no."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=32, do_sample=False,
                     repetition_penalty=1.3, no_repeat_ngram_size=3, penalize_prompt=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))   # -> "No"

penalize_prompt=False matters for grounded use. A prompt-scoped repetition penalty punishes the model for copying the answer out of the context, which is exactly what a grounded decoder must do — it cost 32 points of accuracy in an earlier sweep (27 % vs 59 %). This model's generate() accepts penalize_prompt, no_repeat_ngram_size and repetition_penalty.

Reproducing the comparison

The numbers in the tables above come from a decoding sweep, not a single hand-picked setting. Comparing instruction-tuned models on free-form prompts goes wrong in two ways: decoding becomes a confound if one model runs at its best settings and another at its worst, and "better" has no meaning without an objective. So every model is swept over the same configurations and reported at the one that scores highest for it, against:

composite = accuracy + format_adherence - repetition_rate

Subtracting repetition stops a configuration winning by burying a correct answer inside hundreds of looping words — the characteristic failure of small models at low temperature.

This script is self-contained and reproduces the measurement:

"""Three-way generation comparison with a swept decoding configuration."""
import re
from dataclasses import dataclass, field
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODELS = ["ifx-pse-sys-ml/flame-27m-grounded",
          "ifx-pse-sys-ml/flame-27m-instruct",
          "HuggingFaceTB/SmolLM-135M-Instruct"]

@dataclass(frozen=True)
class Prompt:
    text: str
    gold: tuple = ()        # any of these counts as correct
    constraint: str = ""    # word1 | num | short | sent1 | sent2 | n2 | n3

PROMPTS = [
    Prompt("Write a short caption for a picture of a mountain lake at sunset.", (), "short"),
    Prompt("List three things you would expect to see in a photo of a kitchen.", (), "n3"),
    Prompt("In one sentence, explain what is usually happening at a busy street market.", (), "sent1"),
    Prompt("Describe what a golden retriever puppy playing in a park might look like."),
    Prompt("Answer briefly: what color is a clear daytime sky?", ("blue",), "short"),
    Prompt("What is the capital of France? Answer in one word.", ("paris",), "word1"),
    Prompt("How many days are there in a week? Answer with a number.", ("7", "seven"), "num"),
    Prompt("Explain in one sentence why ice floats on water.", (), "sent1"),
    Prompt("List three primary colors.", ("red", "blue", "yellow"), "n3"),
    Prompt("What is 15 plus 27? Answer with a number only.", ("42",), "num"),
    Prompt("Name two animals that can fly.", (), "n2"),
    Prompt("In one sentence, what is photosynthesis?", (), "sent1"),
    Prompt("Write a two-sentence bedtime story about a lost kitten.", (), "sent2"),
    Prompt("Which is larger, the Sun or the Earth? Answer in one word.", ("sun",), "word1"),
    Prompt("Give one reason why people wear sunscreen."),
]

CONFIGS = {
    "greedy":              {"do_sample": False, "repetition_penalty": 1.00, "no_repeat_ngram_size": 0},
    "greedy+rp1.1":        {"do_sample": False, "repetition_penalty": 1.10, "no_repeat_ngram_size": 0},
    "greedy+rp1.3":        {"do_sample": False, "repetition_penalty": 1.30, "no_repeat_ngram_size": 0},
    "greedy+nrng3":        {"do_sample": False, "repetition_penalty": 1.00, "no_repeat_ngram_size": 3},
    "greedy+rp1.3+nrng3":  {"do_sample": False, "repetition_penalty": 1.30, "no_repeat_ngram_size": 3},
    "greedy+rp1.15+nrng4": {"do_sample": False, "repetition_penalty": 1.15, "no_repeat_ngram_size": 4},
    "T0.7+p0.9":           {"do_sample": True, "temperature": 0.7, "top_p": 0.9,
                            "repetition_penalty": 1.00, "no_repeat_ngram_size": 0},
    "T0.7+p0.9+rp1.1":     {"do_sample": True, "temperature": 0.7, "top_p": 0.9,
                            "repetition_penalty": 1.10, "no_repeat_ngram_size": 0},
    "T0.3+p0.9":           {"do_sample": True, "temperature": 0.3, "top_p": 0.9,
                            "repetition_penalty": 1.00, "no_repeat_ngram_size": 0},
}

ENUMERATED = re.compile(r"(?m)^\s*(?:[-*\u2022]|\d[.)])|\b\d[.)]")

def repetition_rate(text, n=4):
    """Share of n-grams that are duplicates. 0.0 = none, ->1.0 = degenerate loop."""
    w = text.lower().split()
    if len(w) < 2 * n:
        return 0.0
    grams = [tuple(w[i:i + n]) for i in range(len(w) - n + 1)]
    return 1.0 - len(set(grams)) / len(grams)

def obeys_constraint(text, constraint):
    """Did the reply respect the constraint the prompt stated?"""
    if not constraint:
        return True
    w = text.split()
    sents = [s for s in re.split(r"[.!?]+", text) if s.strip()]
    return {
        "word1": len(w) <= 3,
        "num":   bool(re.search(r"\b\d+\b", text)) and len(w) <= 6,
        "short": len(w) <= 25,
        "sent1": len(sents) <= 1 and len(w) <= 45,
        "sent2": len(sents) <= 2 and len(w) <= 70,
        "n2":    len(ENUMERATED.findall(text)) >= 2,
        "n3":    len(ENUMERATED.findall(text)) >= 3,
    }[constraint]

def contains_gold(text, gold, window=30):
    """Is any accepted answer in the opening `window` words? Lenient on lists by design."""
    head = " ".join(text.lower().split()[:window])
    return any(g in head for g in gold)

@dataclass
class Score:
    accuracy: float
    format_adherence: float
    repetition: float
    mean_words: float
    texts: list = field(default_factory=list)

    @property
    def composite(self):
        return self.accuracy + self.format_adherence - self.repetition

def score(texts, prompts=PROMPTS):
    graded = [(t, p) for t, p in zip(texts, prompts) if p.gold]
    return Score(
        accuracy=sum(contains_gold(t, p.gold) for t, p in graded) / max(len(graded), 1),
        format_adherence=sum(obeys_constraint(t, p.constraint)
                             for t, p in zip(texts, prompts)) / len(prompts),
        repetition=sum(repetition_rate(t) for t in texts) / len(texts),
        mean_words=sum(len(t.split()) for t in texts) / len(texts),
        texts=list(texts),
    )

def make_generator(model_id, device="cuda", max_new_tokens=512, seed=0):
    # float32, deliberately: see "Load in float32" below -- bfloat16 at from_pretrained time
    # changes greedy output on this model.
    model = AutoModelForCausalLM.from_pretrained(
        model_id, dtype=torch.float32, trust_remote_code=True).to(device).eval()
    tok = AutoTokenizer.from_pretrained(model_id)
    # penalize_prompt=False keeps the repetition guards off the prompt, so the model is never
    # punished for copying an answer out of its context. Only flame accepts it.
    extra = {"penalize_prompt": False} if "flame" in model_id else {}

    def generate(prompt, cfg):
        ids = tok.apply_chat_template([{"role": "user", "content": prompt}],
                                      add_generation_prompt=True, return_tensors="pt").to(device)
        torch.manual_seed(seed)                 # fixed, so sampling configs are comparable
        with torch.no_grad():
            out = model.generate(ids, max_new_tokens=max_new_tokens,
                                 pad_token_id=tok.eos_token_id, **extra, **cfg)
        return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).strip()

    return generate

for model_id in MODELS:
    generate = make_generator(model_id)
    scores = {name: score([generate(p.text, cfg) for p in PROMPTS])
              for name, cfg in CONFIGS.items()}
    best = max(scores, key=lambda n: scores[n].composite)
    s = scores[best]
    print(f"{model_id:<38} best={best:<20} composite={s.composite:+.3f} "
          f"acc={s.accuracy:.3f} fmt={s.format_adherence:.3f} rep={s.repetition:.3f}")

What the sweep found

• Decoding is not a lever for this model. Accuracy is identical across all six greedy configurations and format adherence identical across all nine; only repetition moves, and any n-gram block drives it to zero. The best-scoring configuration beats the documented repetition_penalty 1.3 + no_repeat_ngram_size 3 by 0.001 — noise.
• Sampling costs accuracy here (0.833 → 0.500 at T=0.7) while it helps larger baselines that loop under greedy decoding. Use greedy for short grounded answers.
• format_adherence does not discriminate between configurations — it is a property of the model, not the decoding.
• contains_gold is lenient on lists: any gold word in the opening window counts, so a partially wrong list ("red, yellow, orange" for the primary colours) scores full marks. Read accuracy as "did it get near the answer".
• Six verifiable prompts cannot separate models reliably — one answer is worth 0.167. This is qualitative evidence about style and control, not a benchmark.

Load in float32

Load this model in float32, not bfloat16. It is 27M parameters — 108 MB — so there is nothing to save, and from_pretrained(dtype=torch.bfloat16) measurably changes greedy output:

load path	"Answer briefly: what color is a clear daytime sky?"
dtype=torch.float32	"The sky is blue."
dtype=torch.bfloat16	"The sky is clear."

Casting after an fp32 load (.to(torch.bfloat16)) is fine — it is constructing the model in bfloat16 that hurts, which points at the RoPE inverse frequencies: with rope_theta=1e6, computing them in bfloat16 loses enough precision to flip a greedy token. Every number in this card was measured in float32.

Limitations

• Terse by design. Mean output 4.9 words. Caption object recall is lower than the instruct model's (0.370 vs 0.594) — it names fewer objects because it says less.
• Held-out template gap. 82.97 % on training-style phrasings vs 72.52 % on unseen ones.
• Weakest skills: spatial relations (57.8 %), captioning (65.2 %), left/right (71.1 %).
• Trained on text, not pixels. Grounding behaviour was measured on rendered text scenes. Only an actual VLM integration can confirm it transfers to visual tokens.
• Ceiling. Across ten checkpoints of a longer training run, 10.6 % of eval items were wrong in every one — further supervised training does not reach them.

Decontamination

Splits are disjoint by image. Every COCO image (51,498) and every GQA val/test/testdev image (15,545) was removed before splitting, so POPE, VQAv2 and the COCO caption benchmarks remain clean for downstream evaluation.