mrutkows/granite-4.2-3b-q4-mlx

Granite 4.2 models (MLX)

Granite 4.2 language models are a family of state-of-the-art open foundation models featuring dense decoder-only architectures. They natively support multilingual capabilities, a wide range of coding tasks, retrieval-augmented generation (RAG), tool usage, structured JSON output, and extended reasoning via a built-in thinking mode.

All models are publicly released under the Apache 2.0 license, allowing free use for both research and commercial purposes. The data curation and training processes were specifically designed for enterprise scenarios and customization, incorporating governance, risk, and compliance (GRC) evaluations alongside IBM's standard data clearance and document quality review procedures.

This repository contains MLX variants of an IBM Granite base model, converted using mlx-lm for native inference on Apple Silicon (M-series) hardware.

Please reference the base model's full model card here: https://huggingface.co/ibm-granite/granite-4.2-3b

Requirements

• macOS with Apple Silicon (M1 / M2 / M3 / M4 or later)
• Python ≥ 3.9

Installation

mlx-lm is a Python package that provides fast LLM inference and fine-tuning on Apple Silicon using the MLX framework.

Option 1 — install into your environment (recommended for repeated use):

pip install mlx-lm

Option 2 — run ephemerally via uvx (no persistent mlx-lm install required):

Prerequisite: uvx is part of uv, a fast Python package manager. Install it first by following the uv installation guide or with:

pip install uv

# mlx-lm is downloaded into a temporary environment on first use
uvx --with "mlx[cpu]" mlx_lm generate \
  --model mrutkows/granite-4.2-3b-q4-mlx \
  --prompt "Why is the sky blue according to science?" \
  --temp 0.7 \
  --top-p 0.9

Available MLX Variants

Each quantization variant is published as its own Hugging Face repository:

Variant	Repo name suffix	Description	Recommended for
bf16	-bf16-mlx	Full-precision BFloat16	Highest quality; requires ≥ 16 GB unified memory
q8	-q8-mlx	8-bit quantization (group-size 64)	High quality with ~50 % memory reduction vs bf16
q4	-q4-mlx	4-bit quantization (group-size 64)	Best efficiency; suitable for 8 GB unified memory

For example, the variants for granite-4.2-3b are published at:

mrutkows/granite-4.2-3b-bf16-mlx
mrutkows/granite-4.2-3b-q8-mlx
mrutkows/granite-4.2-3b-q4-mlx

Running

Run directly from a variant repository (mlx-lm will download weights on first use):

mlx_lm generate \
  --model mrutkows/granite-4.2-3b-q4-mlx \
  --prompt "Why is the sky blue according to science?" \
  --temp 0.7 \
  --top-p 0.9

To use a different quantization, substitute the variant suffix, for example -q8-mlx or -bf16-mlx.

You can also clone the repo and run from a local path:

git lfs install
git clone https://huggingface.co/mrutkows/granite-4.2-3b-q4-mlx

mlx_lm generate \
  --model ./granite-4.2-3b-q4-mlx \
  --prompt "Why is the sky blue according to science?" \
  --temp 0.7 \
  --top-p 0.9

Recommended Generation Parameters

This repository includes a generation_config.json copied from the base model. It records the recommended temperature and top_p values, but mlx_lm generate does not read them automatically — only eos_token_id is consumed by mlx-lm at inference time. Pass the values explicitly on every invocation using the flags below.

The Granite 4.2 base model recommends:

Parameter	Recommended value	CLI flag
Temperature	0.7	--temp 0.7
Top-p	0.9	--top-p 0.9

Without these flags, mlx_lm generate defaults to --temp 0.0 (greedy / deterministic decoding) and --top-p 1.0.

Example with the recommended values applied:

mlx_lm generate \
  --model mrutkows/granite-4.2-3b-q4-mlx \
  --prompt "Explain retrieval-augmented generation in simple terms." \
  --temp 0.7 \
  --top-p 0.9

In Python, pass them directly to generate():

from mlx_lm import load, generate

model, tokenizer = load("mrutkows/granite-4.2-3b-q4-mlx")

response = generate(
    model,
    tokenizer,
    prompt="Explain retrieval-augmented generation in simple terms.",
    temp=0.7,
    top_p=0.9,
    max_tokens=256,
)
print(response)

Thinking Mode

Granite 4.2 supports built-in chain-of-thought reasoning. When thinking is active the model produces its reasoning inside a <think>…</think> block before its final answer.

The chat template exposes two parameters:

Parameter	Values	Effect
enable_thinking	true (default) / false	Enables or disables the <think> block entirely.
reasoning_effort	"low" / "high" (default)	"low" signals a faster, lighter reasoning pass; omit or use "high" for full depth.

CLI — enable thinking with --prefill-response:

The recommended way to invoke thinking from mlx_lm generate is --prefill-response "<think>". This causes the tool to open the assistant turn with <think> as the first generated token, so the full <think>…</think> block appears in the output:

mlx_lm generate \
  --model mrutkows/granite-4.2-3b-q4-mlx \
  --prompt "What is the difference between a mutex and a semaphore?" \
  --prefill-response "<think>" \
  --max-tokens 1024

CLI — disable thinking for direct answers:

Pass enable_thinking=false via --chat-template-config to skip the reasoning block entirely:

mlx_lm generate \
  --model mrutkows/granite-4.2-3b-q4-mlx \
  --prompt "Summarise this paragraph in one sentence." \
  --chat-template-config '{"enable_thinking": false}'

Python API

from mlx_lm import load, generate

model, tokenizer = load(
    "mrutkows/granite-4.2-3b-q4-mlx"
)

# Standard generation
response = generate(
    model,
    tokenizer,
    prompt="What is Retrieval-Augmented Generation?",
    temp=0.7,
    top_p=0.9,
    max_tokens=256,
)
print(response)

Thinking mode — apply the chat template with enable_thinking=True, then generate:

from mlx_lm import load, generate

model, tokenizer = load(
    "mrutkows/granite-4.2-3b-q4-mlx"
)

messages = [{"role": "user", "content": "What is the difference between a mutex and a semaphore?"}]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
    # reasoning_effort="low"  # uncomment for a faster, lighter reasoning pass
)

response = generate(model, tokenizer, prompt=prompt, temp=0.7, top_p=0.9, max_tokens=1024)
print(response)

Supported Languages

Supported Languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. Users may finetune Granite 4.2 models for languages beyond these languages.

Intended Use

This model is designed to handle general instruction-following tasks and can be integrated into AI assistants across various domains, including business applications.

Capabilities

• Summarization
• Text classification
• Text extraction
• Question-answering
• Retrieval Augmented Generation (RAG)
• Code related tasks
• Function-calling tasks
• Multilingual dialog use cases
• Fill-In-the-Middle (FIM) code completions
• Thinking / extended reasoning (enable_thinking, reasoning_effort)

---

Learn more

• Developers: Granite Team, IBM
• Website: Granite Docs
• GitHub Repository: ibm-granite/granite-language-models
• Hugging Face: Granite 4.2 Language Models
• MLX Examples: ml-explore/mlx-examples
• License: Apache 2.0