CL-BLD-1B

CL-BLD-1B is a KDA hybrid language model released with Competing Learning in Hybrid Architectures.

Recurrent mixers and softmax attention can compete over retrieval. When recurrent layers acquire retrieval earlier, attention receives less learning signal to develop the same ability.

Better Layer Division (BLD) gives attention an earlier opportunity to learn retrieval by freezing entire recurrent blocks during early pretraining. Their mixer, MLP, and normalization are held fixed, while attention blocks, embeddings, and the final normalization keep training. All blocks are later unfrozen for continued pretraining, maintaining the division of labor between attention and recurrence.

This is a base language model. It was pretrained on DCLM and is not instruction tuned.

Model details

Field	Value
Total unique parameters, including embeddings	1,014,512,856
Shared input embedding / LM-head parameters	235,008,000 (counted once)
Non-embedding parameters	779,504,856
Published size label	1B
Hidden layers	24
Hidden width	1536
Vocabulary size	153000
Attention layer indices (zero based)	[0, 4, 8, 12, 16, 20]
Block pattern	Attention → Recurrent → Recurrent → Recurrent
Position encoding	NoPE
Training corpus	DCLM
Checkpoint	Final pretrained weights
Total tokens implied by recorded update counts	17,600,348,160

The published names use the established rounded total-parameter labels. Exact counts are given above. The checkpoint is copied byte for byte from the original safetensors file, retaining BF16 weights and FP32 auxiliary parameters.

Installation and loading

Use Linux, an NVIDIA GPU with BF16 support, and a CUDA-enabled environment. The validated environment and installation commands are recorded in ENVIRONMENT.md. This custom architecture requires the included Python files and trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "xingyushenai/CL-BLD-1B"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda").eval()

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Provenance and validation

provenance.json records checkpoint provenance, exact parameter counts, training updates, and the weight SHA-256. validation.json records release smoke tests; these tests are not a benchmark rerun. Input embeddings and the output head share one parameter and must not be counted twice.

For the method, training setup, and evaluation instructions, see the Competing Learning repository.