Instructions to use iromu/LFM2.5-230M-tools-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use iromu/LFM2.5-230M-tools-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="iromu/LFM2.5-230M-tools-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("iromu/LFM2.5-230M-tools-FP8")
model = AutoModelForCausalLM.from_pretrained("iromu/LFM2.5-230M-tools-FP8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use iromu/LFM2.5-230M-tools-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "iromu/LFM2.5-230M-tools-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "iromu/LFM2.5-230M-tools-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/iromu/LFM2.5-230M-tools-FP8

SGLang

How to use iromu/LFM2.5-230M-tools-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "iromu/LFM2.5-230M-tools-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "iromu/LFM2.5-230M-tools-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "iromu/LFM2.5-230M-tools-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "iromu/LFM2.5-230M-tools-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use iromu/LFM2.5-230M-tools-FP8 with Docker Model Runner:
```
docker model run hf.co/iromu/LFM2.5-230M-tools-FP8
```

LFM2.5-230M-tools FP8

The LFM2.5-230M tool-calling fine-tune in FP8 format, for GB10-class serving. The model was supervised-fine-tuned with LoRA so it emits tool calls in the LFM2.5 wire format, then quantized to FP8 with NVIDIA ModelOpt. The full BF16 model and its validation table live at iromu/LFM2.5-230M-tools.

This is the smallest model in the family line, and the one whose tool calls are not OpenAI-shaped. Read the Tool-call format section before using it in a runtime that expects tool_calls in a JSON envelope.

Base model

This checkpoint is the FP8 quantization of:

iromu/LFM2.5-230M-tools

— itself a LoRA fine-tune of LiquidAI/LFM2.5-230M-Base.

Tool-call format

The chat template is the model's contract, and it is not the OpenAI one. A call renders as a bracketed list between two markers:

<|tool_call_start|>[search_files(path="src/", query="load"), read_file(path="src/app.py")]<|tool_call_end|>

Argument values render in a Python-like form: strings single-quoted, mappings as JSON.
A tool's arguments must be a mapping, not a JSON-encoded string; the template calls items() on them.
The assistant span is the trainable one, and the schema list is rendered into the system turn rather than into a separate tool block.

Quantization

The checkpoint is FP8, W8A8: FP8 (E4M3) weights and FP8 activations. It was produced with NVIDIA ModelOpt (FP8_PER_CHANNEL_PER_TOKEN_CFG) and exported with export_hf_checkpoint, which writes the FP8 weights, the per-channel weight scales, and the hf_quant_config.json a serve reads. Linear weights are stored as FP8; lm_head and model.embed_tokens stay bf16.

The build uses per-channel weight scales and per-token dynamic activation scaling, so it writes no static input_scale. That is what the vLLM modelopt FP8 path expects. The static FP8_DEFAULT_CFG writes no input_scale either, and vLLM then reads that scale off an uninitialized sentinel and emits degenerate output on every row — so the per-channel/per-token config is the one the serve actually loads.

Training

Supervised fine-tuning with LoRA, run with NeMo-RL, assistant-turn-masked cross-entropy over the tool-calling logs.

LoRA: dim 32, alpha 32, dropout 0.05, target modules *.proj, *.w1, *.w2, *.w3 (the attention and feed-forward blocks)
Max sequence length: 4096
Precision: bfloat16
Optimizer: AdamW, lr 5e-5, weight decay 0.01
Schedule: 15-step linear warmup, then cosine decay to 1e-6
Global batch size: 64 (micro batch 2)
Training steps: 336 (validation every 84)
Loss masked to assistant turns via the chat template's generation keyword

Dataset

Training used the sft_tools split of the r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation dataset (OpenAI messages format; assistant turns carry tool_calls). The rows are the same corpus the Qwen tool-callers trained on, rendered through this family's template.

Intended use

Structured tool / function calling in a very small footprint (230M parameters)
Agent-style multi-step interactions where the runtime understands the LFM2.5 format
Low-memory serving with vLLM (FP8)

It is not intended to be a general-purpose assistant, and it is not a drop-in for a runtime that assumes the OpenAI tool-call envelope.

Sibling artifacts

BF16 model + validation table: iromu/LFM2.5-230M-tools
GGUF quants: iromu/LFM2.5-230M-tools-GGUF
NVFP4 checkpoint: iromu/LFM2.5-230M-tools-NVFP4

Usage

Serve the checkpoint with vLLM:

vllm serve iromu/LFM2.5-230M-tools-FP8 --quantization modelopt

Or with the vLLM Python API:

from vllm import LLM

llm = LLM(
    model="iromu/LFM2.5-230M-tools-FP8",
    quantization="modelopt",
)

Validation

Model	Quant	Cap	n	Tool call emitted	Names match	Exact args match	Δ exact vs BASE
LFM2.5-230M-tools	BF16	384 tok	272	68/272 (25.0%)	39/272 (14.3%)	36/272 (13.2%)	(predecessor)
LFM2.5-230M-tools	FP8	384 tok	272	69/272 (25.4%)	37/272 (13.6%)	27/272 (9.9%)	−3.3pp

Measured on the sft_tools held-out validation split (272 gradable rows), greedy decoding, under the declared LFM2.5 tool-call grammar. The FP8 row is served through vLLM (--quantization modelopt); the BF16 row is the HF/transformers baseline. Throughput is single-stream greedy decode, not serving throughput.

The absolute numbers are low because this is a 230M-parameter model trained on a non-OpenAI wire format, not because of the quantization: the FP8 row tracks the BF16 row within a few points on every metric.