pihull/pllum-12b-structured-output-lora

PLLuM-12B structured-output LoRA (merged) — v2

A LoRA fine-tune of CYFRAGOVPL/PLLuM-12B-instruct-2512, merged into the base weights. It targets reliable JSON-Schema-conditioned structured output, mainly in Polish, and keeps tool calling.

v2 replaces v1. v1 was trained on PLLuM-12B-nc-instruct-2412. Its preprocessing never rendered the output schema or the tool definitions into the prompt, and it regressed on structured-output tasks. v2 fixes this, uses a rebalanced data mix and starts from the 2512 instruct model.

License: apache-2.0 (inherited from the base model).

Evaluation

The benchmark is a Polish structured-output suite with 9,150 records in 41 task types: extraction, conversion, normalisation, multi-hop joins, adversarial injections and schema stress tests.

• Each prompt contains the task, the source data and an inline JSON Schema.
• A prediction counts as correct only if it is schema-valid (Draft 2020-12 with format checks) and passes per-field policies.
• Both models were run with vLLM in bf16, greedy decoding and the model's own chat template.

model	policy correct	strict JSON	schema valid
PLLuM-12B-instruct-2512 (base)	59.8%	62.3%	93.9%
this model	75.8%	98.1%	97.2%

Paired comparison: 1,562 records are fixed and 95 are broken.

• Largest gains: null for absent fields, hierarchy lookups, unit conversion/rounding, anyOf schemas, cardinality/ordering, relative dates, ignoring schemas injected into the data.
• Still weak: JSON→CSV with Polish number formatting, and deep multi-hop joins with computation.
• No benchmark data was used for training or checkpoint selection. There is zero 12-gram overlap between training prompts and benchmark source texts.

Held-out dev sets (used for checkpoint selection):

model	synthetic PL tasks, exact match	ScrapeGraph test split, leaf accuracy
base	56.4%	28.7%
this model	95.9%	42.7%

Tool calling was not re-evaluated on BFCL for v2.

Training

Data: 48.5k examples. The schema or tool definitions are always visible in the prompt.

• 32k synthetic Polish schema-conditioned tasks. They are procedurally generated and every gold answer is validated against its schema. They cover:
  • typed fields, nulls and optional keys
  • verbatim key names and enums
  • Polish numerals, dates and inflection normalised to the nominative
  • tables, CSV, XML/HTML and JSON reshaping
  • logs, escaping and unions
  • filtering, aggregation and ordering
  • multi-hop joins
  • injected-instruction robustness
• 8k ScrapeGraphAI-100K extraction examples (EN + machine-translated PL); only non-empty, schema-valid answers were kept.
• 2.5k Hermes json-mode examples (with the schema).
• 6k Glaive / xLAM tool-call examples (with tool definitions).

Format:

• The PLLuM-2512 chat template exactly: <s>[INST]{system}\n\n{user}[/INST]{answer}</s>.
• Answers are compact JSON with no code fences.
• Tool calls use <tool_call>{"name":...,"arguments":...}</tool_call>. Tool results go in the next [INST] block as <tool_response>.
• Loss is on assistant tokens only.

LoRA and compute:

• r=16, alpha=32, dropout 0.05 on all attention and MLP projections.
• LR 1e-4 cosine, effective batch 64, 2 epochs.
• 8×H100, about 66 minutes.

Usage

Load it like any causal LM with transformers or vllm, using the built-in chat template. For structured output, put the task, the data and the JSON Schema in the user message. For example:

<task instruction>

Dane źródłowe:
<data>

Zwróć wynik wyłącznie jako JSON zgodny z tym schematem:
<schema>

For tool use, list the tools in the system message inside <tools>...</tools> and ask for <tool_call> blocks.