DSV2-Lite-IndexCache-Sparse-1250

A modified derivative of DeepSeek-V2-Lite, exported as BF16 safetensors from sparse training step 1250. The FSS layer pattern repeats nine times. Each full layer computes routes reused by its two shared layers. The causal key selection budget is 2048 tokens.

This new run completed on October 16, 2026. It starts from the corresponding warmup step 400 and follows the archived DSA recipe, including indexer_clip_alone=False, global batch size 256, sequence length 65536, constant learning rate 4e-5, seed 64, and 320,000 consumed training samples. It is distinct from earlier runs using indexer_clip_alone=True. Details are in provenance.json; matching the recipe does not establish bitwise reproduction of an earlier experiment.

Loading

Install requirements.txt (Transformers 4.57.3). A single GPU needs space for approximately 31.5 GB of BF16 weights plus inference working memory.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "fxmeng/DSV2-Lite-IndexCache-Sparse-1250"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda:0"
).eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda:0")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False,
                            pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The byte-identical modeling_dots_dsa.py is shared with the DSA, IndexCache, and LatentIndex warmup models. Configurations and weights select the method. attention_mode="sparse" is the default; checkpoint_stage records provenance. Use left padding for batched generation. No Megatron environment or custom CUDA extension is required.

Validation and scope

• All tensor names and shapes match; strict loading reports no missing, unexpected, or mismatched parameters.
• Every exported tensor is finite; the safetensors index and file sizes agree.
• Real-model greedy generation, cached versus uncached short-prompt logits, and a 2057-token sparse forward (exceeding top-k 2048) passed.
• The shared implementation passed 20 unit tests and 4 subtests.

validation.json contains the measured results. The BF16 maximum absolute cache logit difference was 0.1875, below the existing test tolerance of 0.5. This is not a claim of exact logit or generation equality for arbitrary prompts. GPU inference tests used CPU offload of decoder weights to fit shared GPU memory, without changing model forward logic.

The portable reference runtime scores selectors and applies masked SDPA in query chunks. It still performs dense work and does not provide optimized sparse CUDA serving. Testing does not establish 64K accuracy/performance or downstream benchmark quality, nor bitwise equality to Megatron training kernels. The inherited tokenizer length metadata and RoPE limits are not tested context length guarantees. The warmup KL training objective is not implemented here.

License

Base weights and tokenizer originate from DeepSeek-V2-Lite, copyright DeepSeek. The included DeepSeek License Agreement v1.0 governs the model weights. This derivative adds sparse indexers and a shared HF implementation.

Part of the LatentIndex collection.