tiny-random/hy4-preview

This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from tencent/Hy4-preview.

Structural comparison with tencent/Hy4-preview:

Feature	Source	Tiny
Architecture / input	HYV4; text	Same
Vocabulary / tied embeddings	120,832 / false	Same
Backbone layers / hidden size	78 / 6,144	10 / 16
MLP layout	1 dense layer, then MoE	Same
Routed experts / top-k / shared experts	256 / 8 / 1	32 / 8 / 1
Dense / expert intermediate size	18,432 / 2,048	128 / 128
Attention Q heads; non-RoPE / RoPE / V dimensions	64; 192 / 64 / 256	8; 32 / 64 / 32
Q / KV LoRA rank	2,048 / 512	64 / 512
Indexer heads / dimension / selected tokens	32 / 128 / 2,048	Same
Full indexer layers	0, 1, 5, 9, ...	0, 1, 5, 9; remaining layers reuse the preceding indexer
Independent Hyper-Connections	4 streams; gated pre/post branches	Same
MLA gating / learned attention sinks / SwiGLU clamp	Elementwise / enabled / 10	Same
MTP	1 plain DSA/MoE decoder; shared embedding/head	Same
Checkpoint precision	BF16; FP32 iHC/main attention sinks and router correction bias	Same

Tokenizer, chat template and generation assets are copied unchanged from the pinned source. The registered Transformers indexed_attention config tag represents the source DSA layers.

File path	Size
model.safetensors	21.8MB

Example usage:

• SGLang (✅ verified)

  • Run on four one-GPU nodes with NODE_RANK=0..3, the same DIST_INIT_ADDR and an available local NCCL_PORT. Enables TP=4, NEXTN/MTP and CUDA Graph.
  • Verified on four NVIDIA H20 GPUs for English, Chinese and long DSA prompts; see Test environment for dependency versions.

model_id=tiny-random/hy4-preview
node_rank=${NODE_RANK:-0}
dist_init_addr=${DIST_INIT_ADDR}
nccl_port=${NCCL_PORT}
export SGLANG_DG_CACHE_DIR=/tmp/hy4-deepgemm-$node_rank
sglang serve \
  --model-path $model_id \
  --tp-size 4 \
  --nnodes 4 \
  --node-rank $node_rank \
  --dist-init-addr $dist_init_addr \
  --nccl-port $nccl_port \
  --fp8-gemm-backend triton \
  --moe-runner-backend triton \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --context-length 4096 \
  --max-total-tokens 4096 \
  --chunked-prefill-size 512 \
  --mem-fraction-static 0.3 \
  --cuda-graph-config '{"decode": {"bs": [1, 2, 4]}}' \
  --disable-custom-all-reduce

• Transformers (✅ verified)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tiny-random/hy4-preview"
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager",
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello"}], tokenize=True,
    add_generation_prompt=True, return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Codes to create this repo (✅ verified):

Click to expand

import json
import re
from copy import deepcopy
from pathlib import Path
from shutil import copyfile

import torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import save_file
from transformers import AutoConfig, AutoModelForCausalLM, set_seed

source_model_id = "tencent/Hy4-preview"
save_folder = "/tmp/tiny-random/hy4-preview"
source_revision = '705d81ee51566a186d645b74c974d642ef2828fe'
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())

raw.update(
    hidden_size=16, num_hidden_layers=10, num_attention_heads=8,
    intermediate_size=128, moe_intermediate_size=128, n_routed_experts=32,
    q_lora_rank=64, qk_nope_head_dim=32, v_head_dim=32, qk_head_dim=96,
    layer_types=["indexed_attention"] * 10,
)
raw["indexer_types"] = raw["indexer_types"][:10]
raw["mlp_layer_types"] = raw["mlp_layer_types"][:10]
(folder / "config.json").write_text(json.dumps(raw, indent=2))
config = AutoConfig.from_pretrained(folder)
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, attn_implementation="eager")
backbone = model.model
last = backbone.layers[-1]
model.model.mtp_layers = torch.nn.ModuleList([torch.nn.ModuleDict({
    "enorm": deepcopy(backbone.norm), "hnorm": deepcopy(backbone.norm),
    "eh_proj": torch.nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False, dtype=torch.bfloat16),
    "input_layernorm": deepcopy(last.input_layernorm), "self_attn": deepcopy(last.self_attn),
    "post_attention_layernorm": deepcopy(last.post_attention_layernorm), "mlp": deepcopy(last.mlp),
    "final_layernorm": deepcopy(backbone.norm),
})])
model.model.mtp_layers.apply(model._init_weights)

def public_name(name):
    for native, source in (("attn_hc", "hc_attn_layer"), ("ffn_hc", "hc_mlp_layer")):
        name = re.sub(r"\." + native + r"\.(fn|base|scale)$", r"." + source + r".hc_pre.hc_\1", name)
    name = name.replace(".self_attn.gate_proj", ".self_attn.linear_gate")
    name = name.replace(".self_attn.sinks", ".self_attn.learnable_sink_param")
    for suffix in ("fn", "base", "scale"):
        name = name.replace(".hc_head.hc_" + suffix, ".hc_head.hc_head_" + suffix)
    return name

state = {}
for name, value in model.state_dict().items():
    name = public_name(name)
    full_precision = name.endswith((".hc_fn", ".hc_base", ".hc_scale", ".hc_head_fn", ".hc_head_base", ".hc_head_scale"))
    full_precision |= name.endswith((".learnable_sink_param", ".e_score_correction_bias")) and not name.startswith("model.mtp_layers.")
    state[name] = value.detach().to(torch.float32 if full_precision else torch.bfloat16).cpu().contiguous()

save_file(state, folder / "model.safetensors", metadata={"format": "pt"})

assets = {"tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "LICENSE"}
for name in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
    copyfile(hf_hub_download(source_model_id, name, revision=source_revision), folder / name)

Printing the model:

Click to expand

HYV4ForCausalLM(
  (model): HYV4Model(
    (embed_tokens): Embedding(120832, 16, padding_idx=120002)
    (layers): ModuleList(
      (0): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=16, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=16, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=16, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=16, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=16, out_features=256, bias=False)
        )
        (mlp): HYV4MLP(
          (gate_proj): Linear(in_features=16, out_features=128, bias=False)
          (up_proj): Linear(in_features=16, out_features=128, bias=False)
          (down_proj): Linear(in_features=128, out_features=16, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (1): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=16, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=16, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=16, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=16, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=16, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=16, out_features=128, bias=False)
            (up_proj): Linear(in_features=16, out_features=128, bias=False)
            (down_proj): Linear(in_features=128, out_features=16, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (2-4): 3 x HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=16, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=16, bias=False)
          (gate_proj): Linear(in_features=16, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=16, out_features=128, bias=False)
            (up_proj): Linear(in_features=16, out_features=128, bias=False)
            (down_proj): Linear(in_features=128, out_features=16, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (5): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=16, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=16, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=16, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=16, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=16, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=16, out_features=128, bias=False)
            (up_proj): Linear(in_features=16, out_features=128, bias=False)
            (down_proj): Linear(in_features=128, out_features=16, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (6-8): 3 x HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=16, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=16, bias=False)
          (gate_proj): Linear(in_features=16, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=16, out_features=128, bias=False)
            (up_proj): Linear(in_features=16, out_features=128, bias=False)
            (down_proj): Linear(in_features=128, out_features=16, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (9): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=16, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=16, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=16, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=16, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=16, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=16, out_features=128, bias=False)
            (up_proj): Linear(in_features=16, out_features=128, bias=False)
            (down_proj): Linear(in_features=128, out_features=16, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
    )
    (norm): HYV4RMSNorm((16,), eps=1e-05)
    (rotary_emb): HYV4RotaryEmbedding()
    (hc_head): HYV4HyperHead(
      (input_norm): HYV4UnweightedRMSNorm()
    )
    (mtp_layers): ModuleList(
      (0): ModuleDict(
        (enorm): HYV4RMSNorm((16,), eps=1e-05)
        (hnorm): HYV4RMSNorm((16,), eps=1e-05)
        (eh_proj): Linear(in_features=32, out_features=16, bias=False)
        (input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=16, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=16, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=16, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=16, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=16, out_features=256, bias=False)
        )
        (post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=16, out_features=128, bias=False)
            (up_proj): Linear(in_features=16, out_features=128, bias=False)
            (down_proj): Linear(in_features=128, out_features=16, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (final_layernorm): HYV4RMSNorm((16,), eps=1e-05)
      )
    )
  )
  (lm_head): Linear(in_features=16, out_features=120832, bias=False)
)

Test environment:

• torch: 2.14.1
• transformers: 5.19.0
• huggingface_hub: 1.33.0
• safetensors: 0.8.0
• accelerate: 1.15.0
• sglang: 0.5.23.dev384+g9ab74b851
• flashinfer-python: 0.7.2
• kernels: 0.17.0
• Python: 3.12.13
• CUDA: 13.0

Change log:

• 2026-10-23: Initial version.

This repo is generated by GPT-6.1-Sol