Instructions to use rish-huggingface/alfred-4b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rish-huggingface/alfred-4b-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="rish-huggingface/alfred-4b-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("rish-huggingface/alfred-4b-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use rish-huggingface/alfred-4b-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/rish-huggingface/alfred-4b-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use rish-huggingface/alfred-4b-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rish-huggingface/alfred-4b-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rish-huggingface/alfred-4b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/rish-huggingface/alfred-4b-GGUF:Q4_K_M

SGLang

How to use rish-huggingface/alfred-4b-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "rish-huggingface/alfred-4b-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rish-huggingface/alfred-4b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "rish-huggingface/alfred-4b-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rish-huggingface/alfred-4b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use rish-huggingface/alfred-4b-GGUF with Ollama:
```
ollama run hf.co/rish-huggingface/alfred-4b-GGUF:Q4_K_M
```
Unsloth Desktop

How to use rish-huggingface/alfred-4b-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "rish-huggingface/alfred-4b-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use rish-huggingface/alfred-4b-GGUF with Docker Model Runner:
```
docker model run hf.co/rish-huggingface/alfred-4b-GGUF:Q4_K_M
```

Lemonade

How to use rish-huggingface/alfred-4b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull rish-huggingface/alfred-4b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.alfred-4b-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use rish-huggingface/alfred-4b-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default rish-huggingface/alfred-4b-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use rish-huggingface/alfred-4b-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rish-huggingface/alfred-4b-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "rish-huggingface/alfred-4b-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Access Alfred 4B for non-commercial evaluation

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Free for non-commercial evaluation under the Alfred 4B Evaluation License (LICENSE) and NVIDIA's license (LICENSE-NVIDIA).

Alfred 4B

A 4B model that outperforms NVIDIA's 120B Nemotron 3 Super at knowing when to act, when to ask, and when to say it can't: 0.70 vs 0.60 macro F1 on NVIDIA's When2Call benchmark.

A 4B-parameter assistant model trained for judgment in tool use: call a tool when everything it needs was said, ask when a required detail is missing, look things up instead of guessing, and say "I can't" when no tool fits. Fine-tuned from NVIDIA Nemotron 3 Nano 4B with teacher distillation and reinforcement learning with verifiable rewards (GRPO) in Alfred, a sandboxed environment that simulates a person's email, calendar and contacts, plus twelve procedurally generated apps.

Small enough to run on a laptop: a 4-bit GGUF is in alfred-4b-GGUF. Shared for non-commercial evaluation only (see License); access is granted automatically once you accept the terms.

What it does differently

Two real episodes from the evaluation, on apps it never trained on. Same request, same tools, same simulated user; "untrained" is NVIDIA Nemotron 3 Nano 4B, the model Alfred 4B was trained from.

"Which contractors have I saved?" The apps on offer handle bikes, tutoring and membership passes; none has contractors.

Untrained 4B, 3 of 3 tries: opens the tutoring app and answers with the user's tutors ("Based on your saved tutors, you have 3 contractors saved: Amélie D. ...").
Alfred 4B, 3 of 3 tries: "I don't have access to contractor information or saved contractor details. The available tools only relate to bike reservations, tutoring, and membership passes."

"Set up a session with my tutor Yusuf A. on 4/2 at 1:30pm." The booking tool takes the tutor's id, not a name.

Untrained 4B, 2 of 3 tries: books with tutor_id: "Yusuf A." (once also guessing the year 2024; it's January 2027), and the user rejects the draft.
Alfred 4B, 3 of 3 tries: looks up the user's tutors, finds Yusuf A. is T-475, and books T-475 on 2027-04-02 at 13:30, approved the first time.

Full transcripts are in the GitHub release (runs r4-base-newapps-* and r4-rl4-newapps-*).

Results

Our own runs, reasoning off. Everything below recomputes from the evidence in the GitHub repo (python scripts/verify_results.py), and the full write-up is RESULTS.md.

Measure	Nemotron 3 Nano 4B (base)	Alfred 4B	Nemotron 3 Super 120B
When2Call macro F1	0.61	0.70	0.60
When2Call: asked for missing information	59%	65%	57%
When2Call: said "I can't" when no tool fits	32%	50%	30%
When2Call: called a tool when it should	94%	96%	96%
BFCL multi-turn: asked for a missing detail	23%	28%
BFCL multi-turn: said a tool was missing	20%	18%
Apps never trained on, clean success (tools)	68%	99%
Made-up tool arguments on new tasks	23%	4%

On When2Call's 300 items, against Nemotron 3 Super on the same items and the same judge (paired bootstrap, 95%): macro F1 +0.10 [+0.06, +0.15], asking +8 points [+1, +15], saying "I can't" +20 points [+12, +29]. The base model and Super were run with their own settings (Super with reasoning on); Alfred 4B with reasoning off.

Use

Turn reasoning off, as in training and every evaluation: pass chat_template_kwargs: {"enable_thinking": false}. NVIDIA's recommended sampling for tool calling is temperature 0.6, top-p 0.95.

With vLLM, using NVIDIA's recommended flags for Nemotron 3 Nano (the reasoning parser plugin is nano_v3_reasoning_parser.py from the base model's repository):

vllm serve rish-huggingface/alfred-4b --trust-remote-code --mamba_ssm_cache_dtype float32 \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder \
    --reasoning-parser-plugin nano_v3_reasoning_parser.py --reasoning-parser nano_v3

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
reply = client.chat.completions.create(
    model="rish-huggingface/alfred-4b", messages=[{"role": "user", "content": "Find me a pharmacy."}], tools=[...],
    temperature=0.6, extra_body={"top_p": 0.95, "chat_template_kwargs": {"enable_thinking": False}})

With llama.cpp: llama-server -m alfred-4b-Q4_K_M.gguf --jinja, and the same chat_template_kwargs in each request. The 4-bit GGUF scores within the margin of the full model on the same tasks: 100% vs 99% clean success on trained tasks with tools, 97% vs 98% with plain replies, and 97% vs 100% on apps never trained on.

Training

Base: nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16, LoRA rank 32 on the attention and MLP projections, merged.
Distillation: 3,900 conversations across 20 task types, in a tool interface and a plain-text one: clean successes of Nemotron 3 Nano 30B as a teacher (71%), topped up with scripted correct episodes.
RL: GRPO, 8 attempts per task, rewards computed by code from what happened (no judge model): a task done earns reward; made-up values, unneeded or missing questions, unrequested actions, leaks and acting on prompt injections cost it. 60 steps; the best checkpoint by validation reward was kept.
Data: synthetic. Email text was written by Nemotron 3 Super with real emails from the Enron corpus as style references only. No data from When2Call, BFCL or AgentDojo, or their tool sets, was used in training.

Limitations

Asked for a two-part answer (a meeting's time and place), it can answer from a search result and never open the record that has the second part.
It recovers poorly after a user rejects a draft, sometimes repeating the same search.
Turning a weekday into a date ("Friday") is unreliable.
It declines and then offers alternatives, which some benchmarks count as asking.
English only; evaluated on synthetic tasks and three public benchmarks. It is not a substitute for a human approving actions: in Alfred, every action is a draft the user approves.

License

Evaluation only. The weights are licensed under the Alfred 4B Evaluation License (LICENSE in this repository): you may download and run them privately to evaluate the model and reproduce its results, non-commercially, and publish your evaluation results with credit. You may not use them commercially, make derivatives (fine-tuning, merging, distilling, training on outputs), or redistribute, publish or host them or any derivative. As a derivative of NVIDIA Nemotron 3 Nano 4B they are also governed by the NVIDIA Nemotron Open Model License (LICENSE-NVIDIA). NVIDIA and Nemotron are trademarks of NVIDIA Corporation. The training and evaluation code is separate, under AGPL-3.0, at github.com/Rishabh-Fuke/alfred.