usermma/Atomight-V2.2-UltraThink-0.5B-NCNN-test

Atomight-V2.2-UltraThink-0.5B — NCNN Conversion

This repository contains the NCNN-format conversion of the onnx-community/Atomight-V2.2-UltraThink-0.5B-ONNX model_int8.onnx model (a 0.5B-parameter INT8-quantized transformer).

Repository Layout

.
├── cpu/
│   ├── atomight_cpu.param    # NCNN network definition (fp32 storage)
│   └── atomight_cpu.bin      # NCNN weight blob (fp32)
├── gpu/
│   ├── atomight_gpu.param    # NCNN network definition (fp16-targeted)
│   └── atomight_gpu.bin      # NCNN weight blob (fp16-targeted)
├── source/
│   └── model_int8.onnx       # The original ONNX source model
└── README.md

Conversion Pipeline

The conversion was performed with the following pipeline:

1. NCNN repository cloned with git clone --depth 1 --recursive https://github.com/Tencent/ncnn.git
2. onnx2ncnn built from tools/onnx/onnx2ncnn.cpp against protobuf 3.21.12
3. ncnnoptimize built from tools/ncnnoptimize.cpp
4. ONNX → NCNN conversion via:

   onnx2ncnn model_int8.onnx atomight.param atomight.bin
   

Important Caveats

Unsupported Layer Types

onnx2ncnn is the legacy NCNN converter. It does not support several ONNX operators that are heavily used in modern transformer architectures. The following ONNX ops had no direct NCNN equivalent and were emitted as untranslated layer types in the .param file (NCNN runtime will treat them as custom layers):

ONNX Op	Count	Notes
MatMulInteger	169	INT8 quantized matmul (custom layer required)
DynamicQuantizeLinear	97	Dynamic per-token quantization (custom layer required)
Cast	376	int ↔ fp casts (need custom Cast layer)
Shape	369	Dynamic shape inference
Where	98	Conditional masking
ConstantOfShape	51	Constant tensor generation
Expand	50	Tensor broadcasting
Equal	50	Comparison for attention masks
IsNaN	24	NaN check (used in RMSNorm)
Range, Gather, And, LessOrEqual, DequantizeLinear	9	Misc

The .param/.bin files produced are valid NCNN-format files, but the model will not load in a vanilla NCNN runtime without registering custom layer types for the operations listed above.

ncnnoptimize Segfault

ncnnoptimize segfaults when processing this model because the unsupported layer types (e.g. MatMulInteger, DynamicQuantizeLinear) leave dangling blob references in the graph that the optimizer cannot resolve. As a result, the fp16 weight-storage optimization that would normally differentiate the GPU variant could not be applied. Both cpu/ and gpu/ directories contain the same .param/.bin artifacts; the gpu/ naming is preserved for the folder structure requested by the user.

Recommended Path: PNNX

For modern transformer models, the NCNN project recommends PNNX (PyTorch Neural Network eXchange), which has far better op coverage for transformer subgraphs (RMSNorm, RoPE, gated attention, etc.). PNNX requires a TorchScript .pt input, so the recommended workflow is:

# 1. Load the original PyTorch checkpoint (or onnx → torch via onnx2torch)
# 2. Script/trace to TorchScript: model.pt
# 3. Run PNNX:
pnnx model.pt inputshape="[[1,128]]" fp16=1
#    → produces model.ncnn.param + model.ncnn.bin (fp16 for GPU)
# 4. For CPU variant, re-run without fp16=1

PNNX was not used in this conversion because installing PyTorch + PNNX in the build environment exceeded available disk space.

Source Model

• Upstream repo: onnx-community/Atomight-V2.2-UltraThink-0.5B-ONNX
• Source file: onnx/model_int8.onnx (632 MB, INT8 quantized)
• Model architecture: 24-layer transformer (Qwen-style), hidden size 896, 14 attention heads, 2 KV heads (GQA), 49 MatMul ops per layer
• Original opset: 14 (with INT8 quantization ops)

Reproduction

To reproduce this conversion:

git clone --depth 1 --recursive https://github.com/Tencent/ncnn.git
cd ncnn && mkdir build && cd build
cmake -DNCNN_BUILD_TOOLS=ON -DNCNN_VULKAN=OFF ..
make -j$(nproc) onnx2ncnn ncnnoptimize

# Convert
./tools/onnx/onnx2ncnn model_int8.onnx atomight.param atomight.bin

# Organize (ncnnoptimize will fail — see caveats above)
mkdir -p cpu gpu
cp atomight.param cpu/atomight_cpu.param
cp atomight.bin   cpu/atomight_cpu.bin
cp atomight.param gpu/atomight_gpu.param
cp atomight.bin   gpu/atomight_gpu.bin

License

The NCNN conversion artifacts in this repo inherit the license of the upstream model. NCNN itself is BSD-3-Clause. See the upstream onnx-community/Atomight-V2.2-UltraThink-0.5B-ONNX repo for model-specific license terms.