Inkling-4.5B-A4.2B-small

This is a small version of thinkingmachines/Inkling created for testing and development. It keeps every hyperparameter of the base model identical (hidden sizes, head counts, MoE topology, vocabulary, MTP settings, vision/audio towers) and only reduces the number of decoder layers, the number of routed experts, and correspondingly the per-layer type lists.

Model Details

• Base Model: thinkingmachines/Inkling
• Architecture: InklingForConditionalGeneration (multimodal, image-text-to-text; text tower inkling_text)
• Total Parameters: 4.53B
• Activated Parameters: 4.21B (per-token text pathway: total minus vision/audio towers minus the non-activated fraction of routed experts, 6 of 8 per MoE layer)

Configuration Changes

Only the three allowed change classes were applied; everything else is identical to the base:

Parameter	Base	Small
num_hidden_layers	66	3
layer_types	55x hybrid_sliding + 11x hybrid (5 sliding + 1 hybrid, repeating)	hybrid_sliding, hybrid_sliding, hybrid
mlp_layer_types	dense, dense, then 64x sparse	dense, sparse, sparse
local_layer_ids	55 sliding layers	[0, 1]
n_routed_experts	256	8

The 3 layers are the minimal set covering every distinct (attention, mlp) combination that occurs in the base (dense MLP only ever occurs with hybrid_sliding attention in the base, so a 4th layer is not needed). All other values are unchanged from the base, including hidden_size 6144, intermediate_size 24576, moe_intermediate_size 3072, num_experts_per_tok (topk) 6, n_shared_experts 2, 64/16 SWA heads and 64/8 full-attention heads (head_dim 128, d_rel 16), sliding_window_size 512, vocab_size 201024 (unpadded_vocab_size 200058), untied embed/unembed, num_nextn_predict_layers/num_mtp_layers 8, and the full vision (hmlp, 4 layers) and audio (dMel) towers.

Checkpoint Structure

Single-file bf16 model.safetensors (73 tensors) using the original base checkpoint naming (model.llm.embed/unembed.weight, model.llm.layers.N.attn.wq_du/wk_dv/wv_dv/wr_du/wo_ud.weight, fused mlp.w13_dn.weight / mlp.w2_md.weight dense MLPs, stacked mlp.experts.w13_weight / w2_weight + mlp.shared_experts.shared_w13_weight / shared_w2_weight MoEs, model.visual.layers.linear_N.weight, model.audio.encoder.weight). Tensor names and per-layer key sets match the base model.safetensors.index.json exactly, modulo layer count (validated programmatically: layer-0 key set == base dense layer 0, layer-1/2 key set == base MoE layer 5; only the base's model.mtp.* tensors are absent, see Notes). The checkpoint loads through AutoModelForImageTextToText.from_pretrained with the same weight-conversion path as the base repo, and all conversions round-trip bit-exactly.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained("inference-optimization/Inkling-4.5B-A4.2B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/Inkling-4.5B-A4.2B-small")

input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. Loaded the base AutoConfig and applied only the allowed changes (3 layers, 8 experts, consistent per-layer lists).
2. Built the model with skip_weights_download, re-initialized weights (init_weights() + NaN fixup), cast to bf16, and saved.
3. Converted the saved checkpoint from transformers-internal naming to the base checkpoint naming (reversing the inkling_mm_model weight-conversion mapping: attention/MLP renames, de-interleaving of fused w13/w13_dn tensors, expert stacking).
4. Fine-tuned on the skill's text-only copypasta dataset (loaded with AutoModelForImageTextToText, text-only forward) in bf16 until target perplexity was reached (lr 2e-4, ~300 steps).
5. Re-converted the fine-tuned checkpoint to base naming and validated structure + round-trip against the base index.

Validation

• Perplexity on the skill's validation text: 1.64 (target <= 10)
• Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.

Notes

• Backbone-only checkpoint: num_nextn_predict_layers/num_mtp_layers is kept at 8 exactly as in the base, but no MTP weights are included (transformers does not materialize them on default init/save; model.mtp.* tensors of the base are intentionally absent). This matches the convention of e.g. inference-optimization/GLM-5.2-0.8B-A0.8B. This single small model covers both the Inkling-0.6B-A0.6B and Inkling-0.92B-MTP collection entries (backbone-only per the collection refresh decision).
• Size: ~4.5B total parameters is the forced minimum for this architecture under the "keep all dims/hyperparameters" constraint: the untied embed + unembed alone are 2x 1.24B (vocab 201024 x hidden 6144), the minimal 3-layer set adds ~1.97B, and the unchanged vision/audio towers ~0.09B. The model has strictly fewer parameters than the ~952B base.
• The tokenizer ships without eos/pad tokens in the base repo (chat-template style); eos_token/pad_token (<|content_model_end_sampling|>, id 200006, matching the model config's eos_token_id) were set in tokenizer_config.json for training/generation convenience.
• The sconv (_keep_in_fp32_modules_strict) modules were cast to bf16 together with the whole model so the checkpoint is uniformly bf16.
• No quantization_config in the base; nothing stripped. bf16 throughout.