roshangrewal/gemma4-e4b-toolcall-v02-gguf

Gemma 4 E4B — Tool-Calling v0.2 (GGUF)

GGUF quantized version of roshangrewal/gemma4-e4b-toolcall-v02 for use with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runners.

94.4% tool-calling accuracy on 1000 diverse queries. 4B parameters.

Quick Start

Ollama (one command)

ollama run hf.co/roshangrewal/gemma4-e4b-toolcall-v02-gguf

LM Studio

1. Open LM Studio
2. Search for roshangrewal/gemma4-e4b-toolcall-v02-gguf in the model browser
3. Download and start chatting

Note: This model retains full conversational abilities of Gemma 4. Tool-calling was added as an additional capability — it can still chat, reason, and answer questions normally. It simply also knows when and how to call tools.

Available Quantizations

File	Quant	Size	Use Case
gemma4-toolcall-v02-Q8_0.gguf	Q8_0	~8 GB	Best quality, needs 10GB+ RAM

⚠️ About the Prompt Format

Gemma 4 uses a custom token format internally that looks different from standard chat templates. When using GGUF directly (without HuggingFace's apply_chat_template), you need to construct prompts in this native format:

<|turn>system                          ← system turn start
You are a helpful assistant.           ← system message
<|tool>declaration:get_weather{...}<tool|>  ← tool definition
<turn|>                                ← system turn end
<|turn>user                            ← user turn start
What's the weather in Mumbai?          ← user message
<turn|>                                ← user turn end
<|turn>model                           ← model turn start (model generates from here)

Tool definitions use special quote tokens <|"|> instead of normal quotes:

declaration:function_name{description:<|"|>Some description<|"|>,parameters:{...}}

Model output will be in this format:

<|tool_call>call:get_weather{city:<|"|>Mumbai<|"|>}<tool_call|>

This is equivalent to the JSON: {"name": "get_weather", "arguments": {"city": "Mumbai"}}

Tip: If this format looks complex, consider using the full HuggingFace model instead — it has a processor.apply_chat_template() that handles all formatting automatically from standard JSON.

Usage with llama.cpp

# Download
huggingface-cli download roshangrewal/gemma4-e4b-toolcall-v02-gguf gemma4-toolcall-v02-Q8_0.gguf --local-dir .

# Run inference
./llama-cli -m gemma4-toolcall-v02-Q8_0.gguf \
  -p "<|turn>system\nYou are a helpful assistant.<|tool>declaration:get_weather{description:<|\"|>Get weather for a city<|\"|>,parameters:{properties:{city:{type:<|\"|>STRING<|\"|>}},required:[<|\"|>city<|\"|>],type:<|\"|>OBJECT<|\"|>}}<tool|><turn|>\n<|turn>user\nWhat's the weather in Mumbai?<turn|>\n<|turn>model\n" \
  -n 100

Usage with Ollama

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./gemma4-toolcall-v02-Q8_0.gguf

TEMPLATE """<|turn>system
{{ .System }}<turn|>
<|turn>user
{{ .Prompt }}<turn|>
<|turn>model
"""

PARAMETER temperature 0
PARAMETER stop "<turn|>"
PARAMETER stop "<eos>"

SYSTEM "You are a helpful assistant with access to tools."
EOF

# Create and run
ollama create gemma4-toolcall -f Modelfile
ollama run gemma4-toolcall "What's the weather in Delhi?"

Usage with Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(model_path="gemma4-toolcall-v02-Q8_0.gguf", n_ctx=4096)

# Build prompt in Gemma 4 native format
# Tool definition with special <|"|> quote tokens
prompt = '''<|turn>system
You are a helpful assistant.<|tool>declaration:get_weather{description:<|"|>Get current weather<|"|>,parameters:{properties:{city:{type:<|"|>STRING<|"|>}},required:[<|"|>city<|"|>],type:<|"|>OBJECT<|"|>}}<tool|><turn|>
<|turn>user
What's the weather in Mumbai?<turn|>
<|turn>model
'''

output = llm(prompt, max_tokens=100, stop=["<turn|>", "<eos>"])
print(output["choices"][0]["text"])
# Output: <|tool_call>call:get_weather{city:<|"|>Mumbai<|"|>}<tool_call|>
# Which means: calling get_weather with city="Mumbai"

Parsing the output

import re

response = output["choices"][0]["text"]

# Extract function name and parameters
match = re.search(r"call:(\w+)\{(.+?)\}", response, re.DOTALL)
if match:
    function_name = match.group(1)  # "get_weather"
    params_raw = match.group(2)
    
    # Parse params: key:<|"|>value<|"|> format
    params = {}
    for key, value in re.findall(r'(\w+):<\|"\|>(.*?)<\|"\|>', params_raw):
        params[key] = value
    # Also parse numeric params: key:number
    for key, value in re.findall(r'(\w+):(\d+(?:\.\d+)?)', params_raw):
        if key not in params:
            params[key] = float(value) if '.' in value else int(value)
    
    print(f"Function: {function_name}")  # get_weather
    print(f"Params: {params}")           # {"city": "Mumbai"}
else:
    # Model chose not to call a tool — responded with text directly
    print(f"Response: {response}")

Usage with LM Studio

1. Download gemma4-toolcall-v02-Q8_0.gguf
2. Open LM Studio → Load Model → Select the GGUF file
3. Chat with tool definitions in system prompt

Model Details

• Base model: google/gemma-4-E4B-it (4B parameters)
• Training: QLoRA SFT on 78K (~85 hours) tool-calling examples via Unsloth
• Accuracy: 94.4% on 1000 diverse queries (simple, ambiguous, many-tools, no-tool, complex params)
• Output format: Gemma 4 native tool-call tokens

BFCL Leaderboard (Official)

Category	Accuracy
Multiple	95.0%
Parallel	90.0%
Simple Python	88.5%
Parallel Multiple	86.0%
Live Simple	79.8%
Non-Live Average	86.5%

Performance by Category

Category	Accuracy
Simple (clear intent)	100%
Complex params	100%
Many tools (12+)	93%
Ambiguous (similar tools)	91.5%
No-tool (respond directly)	87.5%
OVERALL	94.4%

Related Models

• Full weights (fp16): roshangrewal/gemma4-e4b-toolcall-v02 — use this if you want automatic prompt formatting
• LoRA adapter: roshangrewal/gemma4-e4b-toolcall-v02-lora
• Base model: google/gemma-4-E4B-it