Instructions to use inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP

SGLang

How to use inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP
```

Nemotron-3.5-Lightning-1.4B-A0.1B-MTP

This is a tiny version of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 created for testing and development. MTP (Multi-Token Prediction) tensors are included in the checkpoint for testing MTP quantization pipelines.

Model Details

Base Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
Architecture: nemotron_h (hybrid Mamba + MoE)
Total Parameters: 1.4B (main: 1.0B + MTP layers: 0.4B)
Activated Parameters: ~0.1B (MoE with 2/16 active experts per token)
Weight dtype: bfloat16 (unquantized float; base model is NVFP4)

Configuration Changes

Parameter	Original	Tiny
`num_hidden_layers`	52	12
`hidden_size`	2688	2048
`n_routed_experts`	128	16
`num_attention_heads`	32	16
`mamba_num_heads`	64	32
`intermediate_size`	1856	1024
`moe_intermediate_size`	1856	1024
`moe_shared_expert_intermediate_size`	3712	2048
`vocab_size`	131072	131072 (unchanged)
`num_nextn_predict_layers`	1	1 (unchanged)
`mtp_layers_block_type`	`["full_attention", "moe"]`	`["full_attention", "moe"]` (unchanged)

MTP Layer Details

The checkpoint includes 69 MTP tensors (~405M params) in a separate shard model_mtp.safetensors. These are registered in model.safetensors.index.json under the mtp.* prefix. The MTP block mirrors the base model's mtp_layers_block_type: ["full_attention", "moe"] with 16 routed experts.

Note: The HuggingFace NemotronHForCausalLM class silently ignores mtp.* keys at load time (_keys_to_ignore_on_load_unexpected = [r"mtp.*"]), so MTP tensors must be loaded separately for quantization testing.

Checkpoint Structure

The checkpoint is sharded into two files:

model.safetensors — main model weights (247 tensors, ~4.4GB)
model_mtp.safetensors — MTP layer weights (69 tensors, ~0.4GB)
model.safetensors.index.json — unified weight map covering both shards

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

# Main model loads without MTP (HF class ignores mtp.* keys)
model = AutoModelForCausalLM.from_pretrained("inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Validation Output

Loading weights: 100%|██████████| 97/97
Success: 1.0000896453857422 <= 10.0

Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly.

Notes

This model is NOT intended for inference. It is a synthetic tiny model for testing MTP quantization pipelines in llm-compressor.
Weights are random bfloat16 floats, fine-tuned on a toy copypasta dataset to verify the training loop works.
The base model is NVFP4 quantized; this tiny model uses full float weights.
MTP tensors are present in the checkpoint but ignored by the standard HF loader — this is intentional, matching the behavior of the full-size model.

Creation Process

Created using the llm-compressor create-tiny-model Claude skill. The architecture was validated by instantiating on meta device, fine-tuned on a toy dataset to perplexity ~1.0, and MTP tensors were added as synthetic random bfloat16 weights matching the mtp_layers_block_type configuration.

Downloads last month: -

Safetensors

Model size

1B params

Tensor type

F32

Model tree for inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP

Base model

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

Finetuned

(1)

this model

Collection including inference-optimization/Nemotron-3.5-Lightning-1.4B-A0.1B-MTP

Tiny Models

Collection

Tiny models used for testing • 21 items • Updated about 13 hours ago • 3