yujiepan/spark-x2.6-tiny-random

This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from XHToken/Spark-X2.6-4B.

Structural comparison with XHToken/Spark-X2.6-4B:

Feature	Source	Tiny
Vocabulary / tied embeddings	131,072 / yes	same
Layers	36	8; two 3-sliding + 1-full cycles
Hidden size / head dimension	2,560 / 256	16 / 32
Query / KV heads	16 / 4	1 / 1
MLP width	10,240	64
Sliding window	512	same
Per-head output gate	sigmoid	same
Stored precision	BF16	BF16

The source implementation invokes eager attention directly. The tiny definition retains that source behavior rather than falsely advertising an optimized attention backend.

File path	Size
model.safetensors	4.3MB

Example usage:

• Transformers (✅ verified)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tiny-random/spark-x2.6"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
# The pinned Spark implementation calls eager_attention_forward directly;
# it exposes no SDPA/FlashAttention dispatch path to select instead.
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype, trust_remote_code=True).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
inputs = tokenizer("Hello", return_tensors="pt").to(device)
output = model.generate(**inputs, do_sample=False, min_new_tokens=10, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Codes to create this repo (✅ verified):

Click to expand

import json
from pathlib import Path
from shutil import copyfile

import torch
from huggingface_hub import hf_hub_download, list_repo_files
from transformers import AutoConfig, AutoModelForCausalLM, set_seed

source_model_id = "XHToken/Spark-X2.6-4B"
save_folder = "/tmp/tiny-random/spark-x26"
source_revision = "66e3ae7bee54e0a523d692cb8ef841f35409807b"
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
raw.update(
    hidden_size=16, head_dim=32, num_attention_heads=1, num_key_value_heads=1,
    intermediate_size=64, num_hidden_layers=8,
    layer_types=(["sliding_attention"] * 3 + ["full_attention"]) * 2,
)
(folder / "config.json").write_text(json.dumps(raw, indent=2), encoding="utf-8")
for filename in ("configuration_spark.py", "modeling_spark.py"):
    source = Path(hf_hub_download(source_model_id, filename, revision=source_revision)).read_text(encoding="utf-8")
    if filename == "modeling_spark.py":
        # Spark's pinned remote implementation targets Transformers 4.57.
        # These are API compatibility adaptations for Transformers 5.19;
        # the attention, state-dict names, and tied-embedding semantics stay unchanged.
        source = source.replace(
            '_tied_weights_keys = ["lm_head.weight"]',
            '_tied_weights_keys = {"lm_head.weight": "model.embedding.weight"}',
        )
        source = source.replace(
            '"input_embeds": inputs_embeds,\n                "attention_mask"',
            '"inputs_embeds": inputs_embeds,\n                "attention_mask"',
        )
        source = source.replace(
            '                "cache_position": cache_position,\n                "past_key_values"',
            '                "past_key_values"',
        )
    (folder / filename).write_text(source, encoding="utf-8")
config = AutoConfig.from_pretrained(folder, trust_remote_code=True)
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, trust_remote_code=True)
with torch.no_grad():
    for name, parameter in sorted(model.named_parameters()):
        if name.endswith("weight") and "norm" in name:
            parameter.fill_(1)
        else:
            torch.nn.init.normal_(parameter, mean=0, std=0.02)
model.save_pretrained(folder, safe_serialization=True)
assets = {"chat_template.jinja", "generation_config.json", "merges.txt", "special_tokens_map.json", "tokenizer.json", "tokenizer_config.json", "vocab.json"}
for filename in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
    copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), folder / filename)

Printing the model:

Click to expand

Spark2_5ForCausalLM(
  (model): Spark2_5Model(
    (embedding): Embedding(131072, 16, padding_idx=2)
    (layers): ModuleList(
      (0-7): 8 x Spark2_5DecoderLayer(
        (self_attn): Spark2_5Attention(
          (q_k_v_proj): Linear(in_features=16, out_features=96, bias=False)
          (g_proj): Linear(in_features=16, out_features=1, bias=False)
          (out_proj): Linear(in_features=32, out_features=16, bias=False)
        )
        (mlp): Spark2_5MLP(
          (gate_proj): Linear(in_features=16, out_features=64, bias=False)
          (up_proj): Linear(in_features=16, out_features=64, bias=False)
          (down_proj): Linear(in_features=64, out_features=16, bias=False)
          (act_fn): GELUActivation()
        )
        (input_layernorm): Spark2_5RMSNorm((16,), eps=1e-06)
        (post_attention_layernorm): Spark2_5RMSNorm((16,), eps=1e-06)
      )
    )
    (norm): Spark2_5RMSNorm((16,), eps=1e-06)
  )
  (lm_head): Linear(in_features=16, out_features=131072, bias=False)
)

Test environment:

• huggingface_hub: 2.3.0
• safetensors: 0.8.0
• tokenizers: 0.23.3
• torch: 2.13.0
• transformers: 5.19.0

Change log:

• 2026-10-23: Initial version.

This repo is generated by GPT-6.1-Sol