Instructions to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN
# Run inference directly in the terminal:
llama cli -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN
# Run inference directly in the terminal:
llama cli -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN
# Run inference directly in the terminal:
./llama-cli -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN
# Run inference directly in the terminal:
./build/bin/llama-cli -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Use Docker

docker model run hf.co/pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

LM Studio
Jan

vLLM

How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Ollama
How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with Ollama:
```
ollama run hf.co/pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN
```

Unsloth Studio

How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN to start chatting

How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with Docker Model Runner:
```
docker model run hf.co/pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN
```

Lemonade

How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Run and chat with the model

lemonade run user.NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Run Hermes

hermes

Atomic Chat

NVIDIA-Nemotron-3.5-Lightning-30B-A3B — ROCmFP4-STRIX_LEAN

TL;DR

NVIDIA Nemotron 3.5 Lightning 30B-A3B (hybrid Mamba-2 + MoE + Attention) quantized to ROCmFP4-STRIX_LEAN (software FP4, GGUF type 106) for AMD Strix Halo (gfx1151). Runs at 84.3 tok/s generation on a Radeon 8060S iGPU — the fastest of any model tested on this hardware. 15.7 GiB footprint.

⚠️ This GGUF is for the ROCmFPX fork of llama.cpp on AMD RDNA 3.5 only. It will not load in stock llama.cpp.

⚠️ Critical warnings — read before use

Type 106 is invalid for stock llama.cpp. Q4_0_ROCMFP4_STRIX_LEAN (type 106) is a custom type from the charlie12345/ROCmFPX fork. Loading this file with a stock build will fail.
ROCmFPX fork required. Build the fork for gfx1151 (or use the matching Docker toolbox).
FP4 is software-dequantized on RDNA 3.5. gfx1151 has no native FP4 hardware; the kernel dequantizes FP4→FP16 in software. The speedup vs. Q4_K_M comes from the smaller memory footprint and the fork's optimized ROCm dequant path, not from hardware FP4.
Unified memory required. On Strix Halo the VRAM partition is tiny (512 MB). Run with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 and HSA_OVERRIDE_GFX_VERSION=11.5.1.
-fit off is mandatory for this architecture. llama-server with default -fit on deadlocks during "fitting params to device memory" on the 128-expert MoE. Always pass -fit off.

Benchmarks

Hardware: AMD RYZEN AI MAX+ 395, Radeon 8060S (gfx1151, 128 GB unified memory), ROCm 7.2.4. Method: llama-bench -ngl 999 -fa on -p 512 -n 128.

Format	tg128 (tok/s)	pp512 (tok/s)	Size
ROCmFP4-STRIX_LEAN	84.28	1051	15.72 GiB
Q4_K_M (same base)	63.64	813	22.82 GiB

ROCmFP4-STRIX_LEAN is +32% faster at generation and 31% smaller than Q4_K_M. Note: Q4_K_M falls back on 134/401 tensors (the Mamba/SSM tensors are not compatible with Q4_K_M block sizes); ROCmFP4 quantizes them natively (1/401 fallback), which is a large part of why it wins here.

Compared to other Strix Halo ROCmFP4 models: this is the fastest yet (+19% vs grug-35b-v2 at 70.92, +26% vs Ornith-1.0-35B at 66.68).

Quantization details

Type: Q4_0_ROCMFP4_STRIX_LEAN (type 106) — fork-specific.
Preset: STRIX_LEAN (uses q4_0_rocmfp4_fast dequant).
Bitrate: 4.28 BPW.
imatrix: bartowski calibration (185 entries covering the quantizable tensors).
Tensors: 401 (237 f32 — norms/SSM-state/bias; 164 ROCmFP4 — weight matrices).
SHA256: 91316f85d52d378ff963c42f293660f2ca094f8bce0b08ea481ff620bc5056a7

Architecture: nemotron_h_moe

Nemotron-H is a hybrid SSM-Transformer. Each of the 52 layers is one of:

Mamba-2 (SSM): linear-time recurrent state-space layer. ssm.conv_kernel=4, ssm.state_size=128, 64 heads. Handles long-range context with fixed KV cost.
MoE: 128 routed experts (6 active per token) + 1 shared expert. expert_feed_forward_length=1856, expert_shared_feed_forward_length=3712.
Multi-head attention: 32 heads, 2 KV heads (GQA), head_dim=128, RoPE θ=10000. Inserted periodically for global mixing.

Stats: 31.58 B total parameters, ~3.5 B active per token, 131072 vocab, 1M context length.

Note: the ROCmFPX fork's converter does not emit the MTP/nextn tensors, so this build is a plain (non-speculative) model. The base model's MTP layer is not used here.

imatrix

The importance matrix from bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF (185 entries, 822 chunks). Paths in the GGUF metadata have been neutralized to basenames during sanitization.

Usage

docker run --rm -d --name nemotron \
  --device /dev/kfd --device /dev/dri --group-add video --group-add render \
  -e HSA_OVERRIDE_GFX_VERSION=11.5.1 -e GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
  -p 1234:1234 -v /path/to/models:/models \
  <rocmfpx-fork-image> llama-server \
    -m /models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN.gguf \
    -ngl 999 -c 32768 -fa on --jinja --host 0.0.0.0 --port 1234 -fit off

The model has a chat template with a thinking/reasoning mode (the <think> block surfaces in reasoning_content on the OpenAI-compatible endpoint).

Attribution

Base model: NVIDIA, nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (OpenMDW-1.1).
Converter & quantization: charlie12345/ROCmFPX fork (MIT).
imatrix: bartowski.
Toolbox: kyuz0 llama.cpp-amd-gpu.
Quantized by: pugant.

License

OpenMDW License Agreement, version 1.1 (OpenMDW-1.1) — inherited from the upstream NVIDIA model. See LICENSE.

Acknowledgements

Thanks to NVIDIA for releasing Nemotron under a permissive license, to the ROCmFPX fork authors for the FP4 quantization work, and to bartowski for the calibration imatrix.

Limitations

FP4 dequantization is software-only on gfx1151; no native FP4 hardware is used.
MTP/speculative decoding is not enabled in this build.
Validated on Strix Halo (gfx1151) only; behavior on other RDNA versions is untested.

Disclaimer

Not affiliated with or endorsed by NVIDIA, AMD, ggml-org, bartowski, kyuz0, or the ROCmFPX maintainers. Provided "as is", without warranty.

Downloads last month: -

GGUF

Model size

32B params

Architecture

nemotron_h_moe

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for pugant/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN

Base model

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Quantized

(62)

this model