patrickbdevaney/LLaDA2.2-mini-orin-nano

LLaDA2.2-mini for Jetson Orin Nano (packed, mixed 2/3/4-bit)

inclusionAI/LLaDA2.2-mini (16B MoE diffusion LLM, 1.4B active) quantized and packed for the pure-CUDA engine patrickbdevaney/llada2-orin-cuda, which serves it OpenAI-compatibly on a Jetson Orin Nano Super 8GB with 32K context.

These files are not a transformers checkpoint: they are the engine's tile format and need that repo to run.

What is in here

• experts_L*.bin: routed experts, GPTQ (group 128) in a Hadamard-rotated basis, per-expert bit widths chosen against a 32K-context memory budget (mostly 2-bit, some 3- and 4-bit; experts.json lists them).
• nonexpert_L*.bin: attention, shared expert and the dense layer-0 MLP, 8-bit GPTQ.
• lm_head.bin (8-bit) and lm_head_split.bin (the same codes split into high/low nibble planes, used by the engine's screened readout); embed.bin (fp16 embeddings, memory-mapped on the host); small.bin (norms, router, rotation signs); engine.idx (the index the engine loads).
• Tokenizer, chat template and tool_declaration_ts.py from the base model, unchanged, so this directory is also the server's --model directory.

KV cache at runtime: int4 (keys per channel per 32-token block, values per token).

Use

git clone https://github.com/patrickbdevaney/llada2-orin-cuda && cd llada2-orin-cuda/engine && make
huggingface-cli download patrickbdevaney/LLaDA2.2-mini-orin-nano --local-dir ~/models/llada2mini-orin
cd ../server && python3 server.py --model ~/models/llada2mini-orin --engine ~/models/llada2mini-orin

On the Orin Nano Super with pinned clocks: ~25 ms per 32-token denoising step at 1K context, ~180 tok/s on math/code/tool-call tasks at the server's default decode settings, ~150 tok/s at the model card's.

Quality

Measured against the BF16 model on held-out text, the quantization costs mostly in the routed experts (see QUANT_PLAN.md in the code repo). On the engine's scored test (96 generated math problems, 12 code tasks checked against tests, 8 tool calls) it scores 113-116 / 116. The misses have correct arithmetic and a repeated final digit; occasional repetition artifacts like this are the main visible cost of the 2-bit experts.

License

Apache-2.0, following the base model. Credit for the model: inclusionAI, LLaDA2.2.