Kimi-K2.5-2layer-BF16

A 2-layer BF16 slice of moonshotai/Kimi-K2.5, built as a GPU CI fixture for SkyRL. This is not a usable language model — two layers of a 61-layer network produce incoherent text. It exists so tests can exercise the real architecture, with real (truncated) weights, at a size that fits a single node.

Contents

• language_model.model.layers.0 — dense MLP layer (first_k_dense_replace=1)
• language_model.model.layers.1 — MoE layer with all 384 routed experts + 1 shared expert
• language_model.model.embed_tokens, language_model.lm_head, language_model.model.norm
• vision_tower.* and mm_projector.*, unchanged

text_config.num_hidden_layers is set to 2; everything else in the config is untouched.

BF16 conversion

Only the routed-expert weights are quantized in the source release (its ignore list covers attention, shared experts, dense MLP, lm_head and the vision tower). Those tensors were dequantized from compressed-tensors pack-quantized INT4 with the same arithmetic SkyRL uses (examples/train/megatron/dequantize_compressed_tensors_int4.py: nibble = q + 8, per-group q * weight_scale, group_size=32), and quantization_config was dropped from the config.

Every dequantized tensor was verified to be a fixed point of the fake-quant STE (scale_divisor=7.0, q_min=-7): re-quantizing reproduces the stored scales and codes bit-for-bit.

Total: 40.8 GB, ~20B parameters, 33 GB of which are the routed experts.