Instructions to use ibm-granite/granite-4.2-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-4.2-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-4.2-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-4.2-8b")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-4.2-8b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-4.2-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-4.2-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.2-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-4.2-8b

SGLang

How to use ibm-granite/granite-4.2-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-4.2-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.2-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-4.2-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.2-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-4.2-8b with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-4.2-8b
```

Reasoning support not detected by llama.cpp

by squid2 - opened 6 days ago

Discussion

squid2

6 days ago

•

edited 6 days ago

Thanks for making and sharing this model. For things I've tried, it seems substantially better than the similarly sized Gemma 4 E4B and not too far from Gemma 4 12B. However, I'm having issues getting this model to reason.

Running this GGUF with llama.cpp (version 0.4.0 from Homebrew on Mac OS), the model works but reasoning is turned off with no option to enable it showing in the web UI. Even when I try to force it with --reasoning on as a command line argument (which normally shouldn't be needed), it still doesn't reason. It seems llama.cpp isn't detecting the chat template's support for reasoning/thinking.

Here's the command I use to launch llama.cpp:

llama serve -hf ibm-granite/granite-4.2-8b-GGUF:Q4_K_M --temp 1.0 --top-p 0.95 -c 32768 --reasoning on --cors-origins localhost

squid2

6 days ago

•

edited 6 days ago

I just tried in llama-cli instead of the server's web UI, and there reasoning worked. I wonder what's going on; I'll debug further.

llama cli -hf ibm-granite/granite-4.2-8b-GGUF:Q4_K_M --temp 1.0 --top-p 0.95 -c 32768

squid2

6 days ago

•

edited 6 days ago

I just confirmed that reasoning works correctly with the server when I make requests to its API with my own agent/client software. So it seems the bug is confined to the llama.cpp web UI not detecting the model's reasoning support. The web UI's reasoning support detection code is here: https://github.com/ggml-org/llama.cpp/blob/790cf51aabd61763486050dec7451d9147cb7c61/tools/ui/src/lib/utils/chat-template-thinking-detector.ts#L46

gabegoodhart

IBM Granite org 4 days ago

Thanks for reporting this! When I run the model through the webui in router mode, I do see the thinking selector as an option and the thinking does properly display when I chat with the model. I'm running on a build from source on a linux box:

$ llama --version
version: 0.3.0-dev (build 1, commit 3173a56)
built with GNU 13.3.0 for Linux aarch64

What version of llama.cpp are you running? I'm definitely curious to track down if there's something in the chat template that's not being identified correctly as a thinking template.

squid2

4 days ago

•

edited 4 days ago

Thanks for looking into this Gabe!

I'm running Llama.cpp 0.4.1 now (previously 0.4.0). I did some more experimentation and found some bizarre behaviour that seems to be a bug in llama.cpp's web UI rather than any issue with the Granite model or chat template.

When I visit the llama.cpp Web UI at http://127.0.0.1:8080/ in my normal browser session, it doesn't show reasoning support but if I visit it at http://localhost:8080/ it shows reasoning, and if I visit either in private browsing it shows reasoning support. It seems some data in my browser local storage is causing a bug breaking reasoning in the Web UI. I want to figure out what the bug is, but in any case, it appears unrelated to this model and instead something weird in my setup.

gabegoodhart

IBM Granite org 4 days ago

Oooh, fun, this is why I stay away from webdev! Thanks for digging further.

gabegoodhart changed discussion status to closed 4 days ago

planetf1

IBM Granite org 4 days ago

I tried on macOS 27 using llama version: 0.4.1 (build 10964, commit b29c606e2) - installed from homebrew

Using 'llama serve -hf ibm-granite/granite-4.2-8b-GGUF:Q4_K_M --temp 1.0 --top-p 0.95 -c 32768 --reasoning on --cors-origins localhost' reasoning output from the model appeared fine in the UI, however there was no selection option available for choosing the reasoning level - I was only able to turn off via the serve options, or adding per-request custom json (as expected)

However if instead I start llama via 'llama serve' (which enables router mode), I DO then see the reasoning dropdown, similar to that showed above. I tried off and high (didn't test other values) which worked as you'd expect.

I believe this is expected (current) behaviour from the llama ui depending on the mode in which the server is started

squid2

4 days ago

•

edited 4 days ago

Yeah, it seems there's a regression in llama.cpp where when starting in non-router mode, it no longer gives you the menu to choose reasoning level or turn off reasoning.

EDIT: Confirmed, this is a known regression that has been present for two weeks now and hasn't been fixed: https://github.com/ggml-org/llama.cpp/issues/27981

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment