maxdemarzi/black-swan-lora

black-swan PyRel LoRA

black_swan's sft18-qualified adapter for Qwen/Qwen2.5-Coder-1.5B-Instruct. It writes PyRel — a declarative Python query DSL — from a natural-language question and a schema.

71 MB. Serves Event Horizon, which runs Swan, PyRel and DuckDB as WebAssembly in the browser and needs generation from somewhere.

Scores

Executed, not string-matched: every prediction is run and its result compared with a verified reference, because PyRel has several correct spellings of most questions.

	TPC-DS, 22 questions	Spider, 217 held-out
sft18 (this)	15/22 — 68.2%	169/217 — 77.9%
sft17-e2	13/22	168/217
sft15	12/22	175/217 — 80.6%

The two benchmarks disagree, and which one matters depends on your schema. Spider's are 2–7 concepts and ~279 characters; TPC-DS is 29 concepts and 31,208. This adapter is shipped for the wide case, where it leads by three questions — and it is the only one here that writes a working join against 29 concepts (2/6, against 0/6 for sft15). On small schemas sft15 is better by six.

Two other differences that do not show in the totals. It emits far fewer invalid programs — 1 against sft15's 5 on TPC-DS, 7 against 18 on Spider — so failures arrive as wrong answers rather than dead ends. And it acts on error corrections: feeding swan's translated error back through answer_question's retry loop repairs an invalid program, where sft15 ignored the same correction.

Serve it with greedy decoding

The adapter was trained and evaluated under plain greedy decoding, and several servers apply a repetition penalty and top-k sampling even at temperature 0. That mismatch measured as one wrong answer in five, and it reads as the model being weak rather than as a serving setting. Use do_sample=False, or pin top_k 1, top_p 1.0, repeat_penalty 1.0.

merge_and_unload() gives byte-identical output and is ~18% faster at long prompts, so merge at load time rather than serving the adapter unmerged.

Quantization is not free on a wide schema: q4_K_M costs ~5% relative on Spider and BIRD but 23% on TPC-DS. Prefer f16 where the memory allows.

Prompt shape

It expects its own system prompt and a Schema: / Question: user turn, both in tools/local_generator.py (SYSTEM_V5, build_user_turn) and asserted byte-identical against the training split (train_spider_bird_v5.jsonl) by tools/verify_local_generator.py. A differently-worded system prompt is off-distribution and costs accuracy quietly.

Schemas render as Concept: prop, prop with foreign keys as Child.member -> Parent.

Known weakness

Joins on wide schemas, at 2/6. The traversal syntax is usually right and the property names are invented — sold_price for price, Demographics.county for Customer.address.county. Swan's error names the owning concept but not the path to it; swan#178 proposes adding that, which this model is well placed to use since it acts on corrections.