Ling-3.0-tiny-fp8-ONNX

Foundry Local CPU variant

The separate foundry-cpu/ variant includes a CPU-compatible GenAI decoder package preserving FP8 bytes and quantization semantics. Its block GEMMs run in FP32: this is not native FP8 acceleration. See setup, tests and limitations, especially the required client-side NFC normalization and short-input validation scope.

ONNX export of Ling-3.0-tiny-fp8 using standard ONNX operators, Opset 19 and IR 10. Only the three complete language-model graphs are distributed.

Graph interface

Graph	Purpose
onnx/forward.onnx	Complete forward pass without returning caches
onnx/prefill.onnx	Prompt forward pass, logits and initial mixed-layer states
onnx/decode.onnx	One-token step consuming and returning all mixed-layer states

Batch size is one; padding is unsupported. Supply INT64 input_ids and contiguous INT64 position_ids. Pass every prefill state to its mapped decode input and update all states after each step. Ordinary activations and MLA/conv caches use BF16; KDA matrix states use FP32. Logits are FP32. See model_interface.json for exact names.

The consumer supplies tokenization, generation and state orchestration. No ready-to-use ONNX Runtime GenAI or Foundry Local adapter is provided.

FP8 representation and limitations

E4M3FN weight bytes and UE8M0 scale bytes are preserved. UE8M0 is represented as UINT8 and decoded explicitly. Activations follow the inspected per-block amax/448 quantization rule and FP8 rounding. Block matrix products and accumulation are expressed in FP32, with BF16 activations at the graph boundaries.

This export does not implement a native FP8 GEMM kernel or establish native FP8 performance. All-zero activation blocks retain the source rule's NaN behavior; no padding or added epsilon policy is supplied. KDA uses a standard ONNX Loop.

Validation status

Export and static ONNX structural/type/shape checks have completed. Runtime execution, numerical alignment, quality and performance benchmarks have not been performed for these artifacts. The status is EXPORTED_NOT_VALIDATED.

Repository layout

Configuration and tokenizer resources are at the repository root. Graphs are under onnx/; their shared external data is onnx/weights/model.bin. Keep this layout when moving the model. model_interface.json describes names, dtypes, shapes and state mappings. Files absent from the upstream checkpoint are not synthesized.

conversion/ contains only this model's export code, a static format checker, dependencies and English reproduction instructions. The export report is conversion/export_report.json. Original source checkpoints and trusted identity manifests are separate reproduction inputs.

Static format check

python conversion/scripts/check_onnx_format.py --model-dir .
# Optional: also stream and verify all recorded file/weight hashes.
python conversion/scripts/check_onnx_format.py --model-dir . --verify-hashes

These commands do not run inference. For export commands and environment details, see conversion/README.md.

License

See the original license and/or LICENSE_NOTICE.md at the repository root for upstream declarations and source references.