rahatneuron/lfm25-1.2b-toolace-control-s7

LFM2.5-1.2B ToolACE LoRA (control, seed-7 split)

LoRA adapter on LiquidAI/LFM2.5-1.2B-Instruct for single-turn tool calling, trained on a ToolACE-only split (seed 7: 10,246 train / 500 held-out). On that held-out ToolACE set, strict argument accuracy moved from 62.8% → 72.8% (+10.0 pp, paired bootstrap 95% CI [+6.6, +13.6]). A local BFCL alternative-aware metric moved from 87.8% → 90.4% (+2.6 pp, [+0.2, +5.0]). These are experimental scores, not official BFCL leaderboard numbers. Code and full writeup: Rahat-Kabir/toolcall-distill.

Try the demo: toolcall-distill-demo shows base vs. this LoRA side by side on 6 held-out examples (recorded outputs).

This card is for rahatneuron/lfm25-1.2b-toolace-control-s7 only. A separate earlier adapter (lfm25-1.2b-toolace-lora, original 200-row split, 63%→71%) is a different training run; do not mix those numbers with the table below.

How to use

Matches the project eval loader (PeftModel on the LiquidAI base, native chat template with tools=). Needs a recent transformers (chat-template tools= + return_dict) and peft (adapter was saved with peft 0.21.x; 0.18+ loads with a harmless unknown-key warning).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "LiquidAI/LFM2.5-1.2B-Instruct"
adapter_id = "rahatneuron/lfm25-1.2b-toolace-control-s7"

# CUDA: dtype=torch.bfloat16, device_map="auto"
# CPU / no bf16: dtype=torch.float32, no device_map, then .to("cpu")
tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(
    base_id, dtype=torch.float32, low_cpu_mem_usage=True,
)
model = model.to("cpu")
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
messages = [{"role": "user", "content": "What is the weather in Dhaka?"}]

inputs = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to("cpu")  # use model.device when on CUDA

with torch.inference_mode():
    generated = model.generate(**inputs, max_new_tokens=128, do_sample=False)

new_tokens = generated[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=False))

Verified on CPU (float32): the snippet above printed exactly:

<|tool_call_start|>[get_weather(city='Dhaka')]<|tool_call_end|><|im_end|>

LFM emits pythonic tool calls inside markers (single or double quotes are both normal). Pass tool schemas through apply_chat_template(..., tools=...). Greedy decoding with ≤256 new tokens matches the reported eval.

Training details

Item	Value
Base	LiquidAI/LFM2.5-1.2B-Instruct (Unsloth training path; saved adapter_config.json records unsloth/LFM2.5-1.2B-Instruct)
Data	Team-ACE/ToolACE, converted; seed-7 re-split 10,246 train rows (ToolACE-only; no Hermes/Glaive mix)
Supervision	First assistant turn only; packing off; left-truncate at 2048 tokens
LoRA	r=16, α=16, dropout=0; targets q_proj,k_proj,v_proj,out_proj,in_proj,w1,w2,w3
Optim	lr 2e-4, batch 2 × grad-accum 4 (effective 8), max_steps 1200 (~0.94 epoch at ~1280 steps/epoch)
Hardware	Hugging Face Jobs a10g-small, ~28 minutes wall time
Trainer	Unsloth + TRL SFT (scripts/train_toolace_sft.py)

Evaluation

Local metrics from the project scorer (exact argument match after light normalization; ordered tool names/count). Not official BFCL AST/execution scores.

Split / metric	Base	This adapter	Δ (95% CI)
ToolACE-500 strict arguments	62.8%	72.8%	+10.0 pp [+6.6, +13.6]
BFCL-500 alternative-aware arguments	87.8%	90.4%	+2.6 pp [+0.2, +5.0]
BFCL-500 strict selected-answer arguments	59.6%	54.2%	−5.4 pp (strict undercounts valid BFCL alternatives; see writeup)

CIs: 5,000 paired bootstrap resamples over examples. Alternative-aware BFCL accepts any listed nonempty gold alternative and treats a source "" as “omission allowed.”

Limitations

• Strict exact-match scoring on a held-out ToolACE-style distribution; English tool schemas from ToolACE/BFCL converters.
• Single training seed; no multi-seed variance.
• Eval covers first tool-call turns with tools present, not multi-turn completion or “when not to call a tool.”
• The published 500-row / BFCL / external-model runs used a render path that added one extra BOS token; base and adapter were paired. The current run_eval_job.py no longer adds it.
• A mixed Hermes/Glaive run did not beat this ToolACE-only control on the same splits.

License

This adapter is a derivative of LiquidAI LFM2.5 under the LFM Open License v1.0. Upload a copy of that license with the weights (see LICENSE alongside this card).

From the project’s THIRD_PARTY.md (quotes from the official license):

• Commercial use is conditioned on not exceeding the “Threshold” of “annual revenue of 10 million United States dollars ($10,000,000) or more”; Commercial Use by a Legal Entity that “exceeds the Threshold is not licensed” under the agreement.
• Redistribution: “You must give any other recipients of the Work or Derivative Works a copy of this License”; “You must cause any modified files to carry prominent notices stating that You changed the files”; retain copyright/patent/trademark/attribution notices from the Work.
• NOTICE: “If the Work includes a 'NOTICE' text file as part of its distribution, then any Derivative Works that You distribute” must include a readable copy of those attribution notices. The base model tree inspected for this card had a LICENSE and no NOTICE file.

This repository’s training/eval code is separately MIT; it does not override LFM terms for the weights.

Citation

@misc{kabir2026toolcalldistill,
  author       = {Rahat Kabir},
  title        = {toolcall-distill: Fine-tuning small LLMs for reliable tool calling},
  year         = {2026},
  howpublished = {\url{https://github.com/Rahat-Kabir/toolcall-distill}}
}

Also cite ToolACE: arXiv:2409.00920.