wstewart3/Inkling-Small-W4A8-G128

This model is currently untested. However, this will hopefully be a "BETTER" version than NVFP4 for Hopper cards. You get the reduced weights, while maintaining the FP8 Tensor core use.

Inkling-Small: GPTQ W4A8-FP8 creation and INT4 export recovery

This guide records the workflow used to quantize thinkingmachines/Inkling-Small, resolve loading and memory problems, and recover its oversized export into physically packed INT4 storage. It accompanies quantize_inkling_w4afp8.py and recover_inkling_w4afp8.py (also called recover.py during testing).

Status: GPTQ completed successfully. The recovery dry run recognized the expected export damage and estimated 131.86 GiB (141.59 GB) of output tensor payload. That estimate is not proof of completed recovery or successful inference. The recovered checkpoint still needs validation with the intended vLLM installation.

1. What is being created?

Component	Setting
Source model	thinkingmachines/Inkling-Small
Decoder layers	42: two dense, then 40 sparse MoE
Routed experts per sparse layer	256
Shared experts per sparse layer	2
Hidden / dense MLP / expert width	4096 / 16384 / 2048
Weight quantization	Symmetric INT4, groups of 128
Input activation quantization	Dynamic FP8, token strategy
LLM Compressor preset	W4AFP8
Calibration	GPTQ, 512 sequences of 2048 tokens
Final storage format	Compressed Tensors pack-quantized

W4A8-FP8 means four-bit integer weights with eight-bit floating-point input activations. It does not mean FP4 weights. Some parameters remain BF16, including embeddings and retained multimodal/shared-expert parameters in this export. KV-cache precision is an independent inference setting.

2. Files and environments

File	Purpose
quantize_inkling_w4afp8.py	Model loading, CPU expert linearization, sequential GPTQ, initial export
recover_inkling_w4afp8.py	Restore projection/scale layout and pack existing INT4 codes
requirements-repack.txt	NumPy dependency for CPU-only recovery
repack_report.json	Generated recovery details and verification records

Keep quantization, recovery, and inference environments separate when practical. Recovery does not need CUDA, PyTorch, Transformers, LLM Compressor, or the safetensors Python package.

Recorded quantization versions

The supplied source bundles identified Transformers 5.14.1, LLM Compressor 0.13.0, and Compressed Tensors 0.18.0. The save-order bundle also identified PyTorch 2.13.0 and Hugging Face Hub 1.30.0. However, the exported configuration reported Compressed Tensors 0.18.1.a20260903. Preserve that discrepancy in provenance; these records are not a fully verified environment lockfile.

The quantizer imports torch, transformers, accelerate through the loading stack, datasets, llmcompressor, and compressed_tensors. Use the working CUDA/platform-specific environment and capture it rather than blindly upgrading the stack:

python -m pip freeze > quantization-environment.txt
python quantize_inkling_w4afp8.py --help

For recovery, Python 3.10 or newer is sufficient:

python -m venv .venv-recovery
source .venv-recovery/bin/activate
python -m pip install 'numpy>=1.24'
python recover_inkling_w4afp8.py --self-test

3. Hardware and job allocation

The quantization run used a GH200 system with roughly 95 GiB of CUDA-visible memory and over 500 GiB of host-reported memory. The GH200 480GB product label does not mean 480 GB of CUDA VRAM is available to this process.

Slurm limits matter independently of free -h. One failed job reported OUT_OF_MEMORY for its step with 542.55G MaxRSS against a 543.93G allocation; the parent allocation later reported TIMEOUT. A process can therefore be killed despite a misleading host-wide free-memory snapshot.

Example allocation for the site's grace partition; the requested duration must be allowed by local policy:

srun -p grace \
  --gres=gpu:1 \
  --constraint='NVIDIA-GH200-480GB' \
  --mem=0 \
  --time=24:00:00 \
  --job-name=inkling-w4afp8 \
  --pty bash

The two-hour default was insufficient. Allow loading, linearization, calibration, and saving to finish; 24 hours here is a scheduling example, not a runtime guarantee.

The successful troubleshooting path used CPU/disk weight storage and NUMA memory binding. Inspect topology before copying the binding:

numactl --hardware
nvidia-smi

On the observed GH200 node, NUMA node 0 was host memory and node 1 corresponded to GPU-associated memory. numactl --membind=0 kept ordinary host allocations on node 0. Do not assume this node numbering on another machine.

Plan disk capacity for the source model cache, offload files, initial quantized export, and recovered output to coexist. The recovered payload alone is about 132 GiB; extra space is needed for metadata and filesystem overhead. Do not delete the source cache or active offload files during quantization.

4. Create the initial GPTQ checkpoint

Run inside the GPU allocation with the working quantization environment active. This command matches the accompanying script's current arguments:

set -o pipefail
export TOKENIZERS_PARALLELISM=false

numactl --membind=0 python -u quantize_inkling_w4afp8.py \
  --model-id thinkingmachines/Inkling-Small \
  --output-dir ./Inkling-Small-W4A8-G128 \
  --offload-dir ./Inkling-Small-offload \
  --cpu-memory 110GiB \
  --calibration-dataset Salesforce/wikitext \
  --calibration-config wikitext-2-raw-v1 \
  --calibration-samples 512 \
  --max-seq-length 2048 \
  --gptq-dampening-frac 0.01 \
  --torch-threads 8 \
  --omp-threads 8 \
  2>&1 | tee quantization.log

110GiB is the script's conservative CPU weight-placement budget, not a maximum process RSS. Linearization, calibration Hessians, activations, and file caching add memory beyond it. The thread counts are starting settings, not dedicated CPU reservations. Use a fresh output directory. For reproducibility, add --revision with the actual source commit used and retain that commit with the environment record.

Compatibility corrections built into the script

Architecture conversion: the checkpoint's legacy fields mean:

Legacy field	Current field	Value
dense_intermediate_size	intermediate_size	16384
intermediate_size	moe_intermediate_size	2048
dense_mlp_idx=2	First two mlp_layer_types entries are dense	2

The observed conversion left moe_intermediate_size=3072. The script patches it to 2048 in memory and checks the layer pattern. It retains ignore_mismatched_sizes=False. Ignoring a shape mismatch would permit invalid reinitialized weights.

Tokenizer and data: the tokenizer may expose neither a padding token nor EOS. The script falls back to the model configuration's EOS ID for calibration bookkeeping without adding vocabulary entries. The current loader packs text into complete 2048-token blocks, so no padding is actually needed. It uses Salesforce/wikitext to avoid the invalid legacy hf://datasets/wikitext... URI, and passes processor=tokenizer to avoid constructing a multimodal processor for text calibration.

MoE conversion: fused expert parameters must be converted to calibration-friendly linear projections. The absence of the word Inkling in LLM Compressor sources did not imply missing support: its generic detector handled these experts. The current script loads using CPU/disk storage, calls set_onload_device(..., cpu), checks storage destinations, and explicitly linearizes on CPU. Sequential calibration then uses the accelerator.

Do not restore the earlier combined load/linearization context or GPU storage budget: those attempts exhausted CUDA memory before the first expert layer was converted. The current --gpu-memory option is ignored and --preload-linearize-moe is rejected. Leave sequential targets at the script default unless investigating a specific pipeline issue.

Progress and failure interpretation

free -h
df -h .
nvidia-smi
sacct -j "$SLURM_JOB_ID" --units=G \
  --format=JobID%20,State%20,ReqMem,AllocTRES%80,MaxRSS,ExitCode

GPTQ log lines such as METRIC - error 3005769.50 are quantization loss metrics, not Python exceptions. Their raw magnitude alone is not a model-quality verdict. NaNs, failed decompositions, tracebacks, or scheduler OOM events require investigation. Quality must be measured after inference works.

gc.collect() can collect unreachable Python objects and torch.cuda.empty_cache() can release unused cached CUDA allocations. Neither frees live model tensors or resolves a host-memory allocation limit. Allocator tuning cannot create capacity for live tensors; it was not the solution to the observed CPU/disk placement problem.

5. Why the initial export was approximately 253 GiB

The initial 14-shard export reported:

Stored dtype	Payload
I8	242.10 GiB
BF16	10.81 GiB
F32	Approximately zero at displayed precision

Its quantization configuration declared four-bit weights but used format: "int-quantized". Each logical four-bit code occupied an eight-bit integer element. Physical packing can approximately halve this I8 portion while retaining scales and BF16 parameters:

242.10 / 2 + 10.81 ≈ 131.86 GiB

Changing only the JSON format or casting I8 to I32 would not pack the data. Eight four-bit codes must be placed into each 32-bit storage word, with corresponding packed tensor names and logical shapes.

6. Why this checkpoint also needed recovery

The export had 13 scale names present in two adjacent shards with different values. Both copies had shape [2048,32], while the corresponding legacy fused I8 weight had shape [2048,8192]. An ordinary G128 interpretation would require scales [2048,64].

These were two required projection halves, not redundant copies. Selecting only the index-designated copy loses the other half. The recovery was derived from the supplied installed source:

1. Linearized expert modules register up_proj before gate_proj.
2. State-dict traversal and sharding preserve that insertion order.
3. Legacy weight conversion runs independently within each output shard.
4. Inkling's conversion patterns also match projection scale metadata.
5. When a gate/up pair straddles a shard boundary, each surviving half undergoes its own interleave transform.

For this exact export, earlier shard = up scale; later shard = gate scale. Each half must have its internal interleaving undone independently. Simply concatenating the saved scale halves is wrong.

The full fused weights also need their projection layout restored: weight columns and scale groups were interleaved at different granularities. Recovery restores separate gate/up projections, dense MLP row ordering, retained shared-expert ordering, and canonical Transformers names (model.language_model.* rather than legacy model.llm.*).

This is a narrowly validated recovery profile for the supplied export. Do not bypass a shape or duplicate-count error, change group size to 256 to make shapes fit, average scales, or discard nonidentical copies. Fresh exports from different library versions may need a different path.

7. Recover and physically pack INT4

Use the CPU-only recovery environment. The output directory must not exist, even as an empty directory, and must be separate from the source tree.

Metadata dry run

python recover_inkling_w4afp8.py \
  --source-dir ./Inkling-Small-W4A8-G128 \
  --output-dir ./Inkling-Small-W4A8-G128-v2 \
  --dry-run

The observed dry run reported:

Check	Result
Dense gate/up pairs	2
Routed expert gate/up pairs	10,240
Unquantized shared gate/up pairs	40
Boundary scale pairs	13
Recovered tensor entries	62,506
INT4 weight tensors after splitting	30,936
Recovered unpacked payload	252.91 GiB
Expected packed payload	131.86 GiB / 141.59 GB
Output shards with default target	67

A dry run checks metadata, not tensor values. The increased shard count is expected: recovery defaults to a 2 GiB payload target and restores separate projections.

Actual conversion

set -o pipefail
python -u recover_inkling_w4afp8.py \
  --source-dir ./Inkling-Small-W4A8-G128 \
  --output-dir ./Inkling-Small-W4A8-G128-v2 \
  --max-shard-size 2GiB \
  --chunk-size 8MiB \
  2>&1 | tee recovery.log

The command reads the source without modifying it. It performs no GPTQ recalibration. It restores tensor ordering and then packs the existing signed INT4 codes:

• Validate each code is in [-8,7].
• Encode each nibble as code + 8.
• Pack eight consecutive codes along the last dimension into a little-endian I32 word, first code in the least-significant nibble.
• Write .weight_packed and I64 .weight_shape; preserve recovered scale values and retained tensors.
• Set the top-level and group compression formats to pack-quantized and remap names in quantization metadata.

This is Compressed Tensors storage packing, not a Marlin-specific kernel rearrangement. Dynamic FP8 activation settings remain in the configuration.

Every weight chunk receives range and packing round-trip checks. Output tensors are reread and SHA256-checked before final configuration/index publication. An extra --check-values pass is optional and repeats input scanning already performed during conversion.

The script leaves REPACK_INCOMPLETE until completion. If it fails, keep the diagnostic output and retry into a fresh directory after addressing the error; resume is not implemented. It intentionally omits unrelated directories and the legacy recipe file. Temporary memory is bounded by chunks and transformations, rather than loading the entire model.

8. Verify the result

After the script explicitly reports success:

test ! -e ./Inkling-Small-W4A8-G128-v2/REPACK_INCOMPLETE
du -sh ./Inkling-Small-W4A8-G128-v2

Inspect repack_report.json, config.json, and model.safetensors.index.json. Expect pack-quantized, I32 packed weights, I64 shape tensors, retained BF16 tensors, and the 13 recovered scale-pair assignments. Filesystem usage may differ slightly from tensor payload.

The recovery self-tests and source-based ordering checks validate synthetic cases. They do not establish model accuracy or compatibility with an inference loader. Retain the initial quantized checkpoint until both loading and quality tests pass.

9. vLLM smoke test

Use an inference environment that supports Inkling and the relevant Compressed Tensors W4A8-FP8 implementation. Compatibility with the recovered canonical, linearized tensor names has not yet been established. Record pip freeze in that environment too.

For the same single ~95 GiB GH200, the ~132 GiB model requires offloading. This is an initial compatibility test, with an estimated 80 GiB CPU offload budget:

set -o pipefail
vllm serve ./Inkling-Small-W4A8-G128-v2 \
  --served-model-name inkling-test \
  --host 127.0.0.1 --port 8000 \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --cpu-offload-gb 80 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 2048 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 2048 \
  --language-model-only \
  --enforce-eager \
  2>&1 | tee vllm-smoke-test.log

These options are described in the vLLM serving reference. Eager execution disables compilation/CUDA graphs; text-only mode disables multimodal inputs. CPU offloading support for this particular quantized loader is part of the test. Check the installed version's help if a flag is unavailable.

After server startup, from the same compute node:

curl --fail-with-body http://127.0.0.1:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"inkling-test","prompt":"The capital of France is","temperature":0,"max_tokens":32}'

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"inkling-test","messages":[{"role":"user","content":"What is 17 multiplied by 23? Explain briefly."}],"temperature":0,"max_tokens":128}'

The completion test avoids chat-template dependency. Check coherent output, correct termination, and absence of missing-weight warnings. Then compare a fixed set of arithmetic, coding, summarization, and instruction-following prompts against a trusted baseline with matching sampling settings. Short smoke tests do not replace a quality evaluation.

Unsupported architecture, missing tensor names, scale shape errors, or unsupported quantization require loader investigation. Do not blindly append RoutedExperts to metadata or rerun GPTQ to address such failures. A missing chat template is a separate tokenizer/configuration issue.

10. Preserve provenance

Keep the source revision, quantization script, recovery script/version, environment records, calibration settings and seed, quantization log, recovery report/log, and inference validation results together. Also retain the installed export/save-order source bundles that established the boundary-scale recovery.

The recovery fixes storage representation and the identified export layout damage. It does not improve the original GPTQ approximation, quantify its accuracy loss, or prove multimodal behavior.