Inkling-Dense-54M-MTP

A dense Inkling fixture for testing the model.mtp.* checkpoint layout in llm-compressor PR #3225. Created with the repository's create-tiny-model skill. The backbone is trained on its eight toy texts; the MTP layer is randomly initialized and untrained. This fixture is useful for checkpoint-preservation, quantization/export, and speculative-serving regression tests. It is not a general-purpose language model or a draft-acceptance benchmark.

Model details

• Architecture: InklingForConditionalGeneration.
• Parameters: 54,136,067, including one MTP layer and reduced audio/vision components.
• BF16 weights: 108,279,142 bytes, approximately 103.3 MiB.
• Two dense backbone layers, one sliding and one full attention layer; no routed-expert tensors.
• One sliding-attention MTP layer, saved as 20 model.mtp.layers.0.* tensors.
• Original tokenizer and 201,024-entry padded vocabulary retained.

Setting	Original Inkling	This fixture
Text hidden size	6,144	128
Backbone layers	66	2
MTP layers	8	1
Attention heads	64	4
KV heads	8	2
Dense intermediate size	24,576	512
Head dimension	128	128
Context limit	1,048,576	512

Audio and vision projection dimensions match the reduced text hidden size. Text-only serving is tested; multimodal inference is not. Both Transformers' dense-layer fields and vLLM's legacy dense_mlp_idx / dense_intermediate_size fields are retained in the config.

Creation and provenance

1. Start from the corrected tiny Inkling config and compare against original Inkling. No original model weights are copied.
2. Initialize the reduced dense backbone with seed 0. Use the tokenizer and processor from Inkling-0.92B-MTP.
3. Train using the skill's COPYPASTAS dataset, learning rate 0.001, sequence length 128, and seed 0. Early stopping finishes at 200 steps. Evaluation perplexity is 1.0279 on the same eight training texts, not a held-out benchmark. The skill's separate sample check gives 1.0051.
4. Initialize one MTP layer with seed 0 and use Transformers' reversed weight conversions to serialize it. All 62 final checkpoint keys occur in the original Inkling checkpoint index. All tensors are finite.

Build provenance and scoped validation results are included in fixture_provenance.json. The model was created for llm-compressor PR #3225.

Usage

import torch
from transformers import AutoTokenizer, InklingForConditionalGeneration

path = "soyrsoyr/Inkling-Dense-54M-MTP"
model = InklingForConditionalGeneration.from_pretrained(
    path, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager"
)
tokenizer = AutoTokenizer.from_pretrained(path)
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=16)
print(tokenizer.decode(output[0]))

Serving and tested scope

Prepublication validation with load_context(InklingForConditionalGeneration, load_mtp=True) detected and attached MTP, quantized its layers with FP8_DYNAMIC, and exported seven MTP weight-scale tensors under the native model.mtp.* checkpoint names. A separate control loaded MTP through the same path, kept it in BF16, and successfully served the FP8 backbone with vLLM MTP drafting.

BF16 MTP serving also passed for the source fixture. An additional unloaded-copy check used load_mtp=False and oneshot(..., output_dir=...), verified exact equality of all 20 copied MTP tensors, and passed vLLM generation with a positive MTP draft-token counter. These are separate local validation runs, not a claim that fully quantized Inkling MTP serving works.

Quantizing and exporting the MTP layers also succeeds, but serving those quantized MTP layers fails in the tested vLLM implementation: Inkling's MTP decoder is constructed with quant_config=None and rejects weight_scale tensors. Keep MTP in BF16 for this serving test.

from vllm import LLM, SamplingParams

llm = LLM(
    model="soyrsoyr/Inkling-Dense-54M-MTP",
    dtype="bfloat16",
    max_model_len=256,
    max_num_seqs=2,
    gpu_memory_utilization=0.25,
    enforce_eager=True,
    enable_prefix_caching=False,
    disable_log_stats=False,
    limit_mm_per_prompt={"image": 0, "audio": 0, "video": 0},
    speculative_config={"method": "mtp", "num_speculative_tokens": 1},
)
outputs = llm.generate(["The capital of France is"], SamplingParams(temperature=0, max_tokens=16))
print(outputs[0].outputs[0].text)

Tested on one H100 with Transformers 5.17.0 and torch 2.13.0+cu130. The WDC vLLM installation reports distribution metadata 0.30.0 but runtime version 0.26.0; this is validation of that installed build, not a clean-wheel compatibility claim. Multimodal inference and multi-GPU serving are not covered. The dense design intentionally avoids the separate sparse Inkling expert-export issue.