JacLLM-SFT-Ornith-9B-v1.2

Instruction-tuned code model for the Jac programming language, built on top of ornith-ai/Ornith-1.5-9B (a Qwen3.5-based text-only backbone).

This model stacks CPT + SFT on top of Ornith. The single merged LoRA carries both stages of adaptation, so the released weights already encode the CPT knowledge from JacLLM-CPT-Ornith-9B-v1.2. Use this variant for chat-style Jac tasks: Python→Jac translation, code completion, walker/OSP generation, docstring Q&A, etc.

Training

• Base: ornith-ai/Ornith-1.5-9B unwrapped to its text-only Qwen3.5 backbone
• Continued from: the CPT LoRA checkpoint (the SFT trainer continues the same LoRA weights, so this release contains CPT + SFT combined)
• LoRA: r=64, α=32, rsLoRA, all attention + MLP projections (q,k,v,o,gate,up,down), merged into base weights for release
• Optimizer: AdamW 8-bit, lr=2e-4 linear-decay, weight decay 1e-3, 1 epoch
• Mixing: sequential (each SFT task consumed in full, in listed order, file order preserved — no shuffle)
• Task order: code_completion → py2jac → js2jac → osp → farm → code_gen → scaffold2impl → qa
• Data sources: internal task-specific corpora covering code completion, Python-to-Jac conversion, JS-to-Jac conversion, object-spatial programming (walkers, nodes, edges), farm/pipeline patterns, code generation from natural-language specs, scaffold-to-implementation, and Q&A over Jac source.
• Toolchain: Unsloth 2026.8.19, Transformers 5.5.0, jac 0.36.0

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "jaseci/JacLLM-SFT-Ornith-9B-v1.2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [
    {"role": "system", "content": "You are an expert AI assistant specializing in the jac programming language."},
    {"role": "user",   "content": "Write a Jac function `add(a: int, b: int) -> int` that returns the sum, plus a `with entry` that prints add(2, 3)."},
]
inputs = tok.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,      # <-- see chat-template section
    return_tensors="pt",
).to(model.device)

out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0, inputs.shape[-1]:], skip_special_tokens=True))

Sample expected output:

def add(a: int, b: int) -> int {
    return a + b;
}

with entry {
    print(add(2, 3));
}

Chat template

The tokenizer ships with Ornith's native chat_template.jinja, patched so that enable_thinking defaults to False when unspecified. Unless a caller explicitly passes enable_thinking=True, the empty <think>\n\n</think>\n\n block is emitted before the assistant response — matching training. This model was not trained on non-empty reasoning traces, so enable_thinking=True will produce weird output.

Downstream tools (llama.cpp, vLLM, Unsloth Studio, TGI, …) that render the chat template with default arguments will now behave correctly out of the box.

Evaluation

Internal SFT eval (jac 0.36.0, our own held-out sets):

Task	Records	jac check Pass rate
code_completion	1,722	92.6%
py2jac	93	92.5%
js2jac	600	85.3%
code_gen	460	61.7%
osp	206	98.1%
Weighted overall	3,081	87.0%

External function-eval v1 dev split (an internal function-completion benchmark; 184 problems whose reference solutions pass under jac 0.36.0):

Metric	Value
AC (all hidden tests pass)	37.5%
Testcase pass rate (partial credit)	64.6%
Fair pass@1 (excluding jac-toolchain-side failures)	43.9%
LLM-as-judge relaxed pass@1	80.4%

Failure taxonomy on this eval attributes ~1/4 of failures to jac-side compiler bugs / native-pathway limits (E5043 Store/Load, E5090 kwargs unpacking, E5092 list lowering, E5020 LLVM IR) rather than the model.

Docstring underspecification. A large portion of the strict-AC failures are cases where the model produces a semantically defensible implementation of what the docstring literally says, but the hidden tests check specific undocumented magic strings, edge-case constants, or key-name choices that no reasonable reader could infer from the docstring alone. To quantify this, we scored every failing sample with an LLM-as-judge (Claude Opus 5) reading only the docstring + prefix + model output + reference; the judge marks samples where the algorithmic intent is correct as "reasonable-given-docstring". 80/116 failures (69%) came back as reasonable, pushing the effective "docstring-intent correct" rate to ~80%. The remaining 36 failures are genuine model bugs — off-by-one index math, wrong formula despite specific docstring, missing return path on a conditional branch, incorrect string constants, etc. — those are the training targets for a debugging dataset.

Known limitations

• Trained at r=64 LoRA and merged into a 4-bit-quantised base for release. Minor rounding differences vs. an adapter-loaded inference path are expected (bitsandbytes warns about this). For strict reproducibility, we can also publish a bf16 merged variant.
• No reasoning-mode training — always call with enable_thinking=False (the default after our template patch).
• Optimised for Jac code generation. Not aligned for open-ended chat, math, or safety-sensitive tasks.
• The internal eval uses in-house held-out sets, not a public benchmark; absolute numbers should not be compared to external code-eval leaderboards without normalisation.

Attribution

Base model © the ornith-ai team. CPT + SFT training and packaging by Jaseci Labs.