LLaMA3.1-8B-Instruct-DFlash-Safetensors

Quantized DFlash draft checkpoint derived from z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat, prepared for SiMa.ai LLiMa compilation. This is a speculative-decoding draft, not a standalone text-generation model. Use it with the prequantized target simaai/Llama-3.1-8B-Instruct-GPTQ-Safetensors from the SiMa.ai Pre-Quantized Models collection.

Quantization

• All 36 draft linear layers, including context fusion fc, attention Q/K/V/output projections, and MLP gate/up/down projections: round-to-nearest (RTN), symmetric INT8 per output channel.
• Weight scales: min/max observation, one scale per output row; no grouping or activation quantization.
• Normalization parameters and any biases: retained in BF16.
• Calibration dataset: none. Quantization observes weights only and requires no forward pass.
• Storage: compressed-tensors pack-quantized Safetensors. The quantization configuration and exact layer targets are in config.json.

Target embeddings and the target output head are not included or quantized by this recipe. The runtime/compiler must supply them from the matching target checkpoint.

Reproduce

Use Python 3.12 and the dependencies recorded in versions.txt. The original run used development builds; their source repositories and commit revisions are recorded in dependency_sources.json. These build versions may not be available from PyPI.

Download the original BF16 draft from z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat, then run the included script on CPU:

python quantize.py \
  --model-path /path/to/LLaMA3.1-8B-Instruct-DFlash-UltraChat \
  --output-dir /path/to/LLaMA3.1-8B-Instruct-DFlash-UltraChat-RTN-INT8-Safetensors \
  --threads 8

The output directory must not already exist. The script reads config.json and the single model.safetensors source file directly, so custom Transformers modeling code and a GPU are not required. It preserves the source configuration and adds compressed-tensors metadata.

Included: model.safetensors, config.json, quantize.py, quantize.py.sha256, versions.txt, and dependency_sources.json.

Validation and limitations

Export checks passed for all 36 INT8 linear weights, per-channel scale shapes, and finite positive scales. LLiMa loader validation passed for every quantized layer: unpacked INT8 weights and scales matched the preceding export exactly. Original configuration fields were preserved, and retained BF16 tensors matched the source byte-for-byte.

LLiMa compilation, draft acceptance rate, output quality, and runtime performance have not been validated for this quantized draft. It uses the same INT8 per-channel policy as the intended compiler flow; bit-for-bit equivalence to compiler-side quantization has not been established.

The source checkpoint's immutable Hugging Face revision was not recorded for this run.