salyamq/smqBERT-kk-small

SmqBERT 100M Kazakh

SmqBERT is a 100.7M-parameter encoder-only masked language model for Kazakh. It was pretrained on salyamq/kk-corpus-v1 with an 8,192-token training sequence length and a hybrid local/global attention pattern.

The model is intended for masked-token prediction and as a pretrained encoder for downstream Kazakh NLP tasks. It is not a chat model or an autoregressive text-generation model.

Quick start

SmqBERT uses custom Transformers code, so trust_remote_code=True is required.

pip install -U torch transformers sentencepiece loguru

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline

model_id = "salyamq/smqBERT-kk-small"

if torch.cuda.is_available():
    major, _ = torch.cuda.get_device_capability()
    dtype = torch.bfloat16 if major >= 8 else torch.float16
    device = 0
else:
    dtype = torch.float32
    device = -1

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
)

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer,
    device=device,
)

predictions = fill_mask(
    "Қазақстанның астанасы — [MASK].",
    top_k=5,
)

for prediction in predictions:
    print(f"{prediction['token_str']:<20} {prediction['score']:.2%}")

Example top prediction:

Астана               53%+

Exact scores can vary slightly with device and numerical precision.

Model architecture

SmqBERT is an encoder-only masked language model with 100,724,224 trainable parameters (approximately 100.7M).

The model uses a 32,000-token SentencePiece vocabulary, 768-dimensional hidden states, and supports sequences up to 8,192 tokens. The encoder consists of 16 Transformer layers with 12 attention heads per layer. Each head has a dimension of 64.

The attention pattern alternates between local sliding-window attention and global attention. Eleven layers use sliding-window attention with a window size of 128 tokens, while five layers use global attention. Rotary position embeddings are used with separate configurations for local and global attention, with RoPE theta values of 10,000 and 160,000 respectively. Query and key normalization is applied in every attention layer.

Each Transformer layer follows a pre-normalized design:

• RMSNorm
• multi-head self-attention
• residual connection
• RMSNorm
• SwiGLU feed-forward network
• residual connection

The SwiGLU feed-forward network projects the 768-dimensional hidden state to two 1,024-dimensional branches, applies the SiLU activation to the gate branch, multiplies the branches element-wise, and projects the result back to 768 dimensions.

The masked-language-modeling head applies a dense projection, SiLU activation, RMSNorm, and a final projection to the 32,000-token vocabulary. The input embeddings and output decoder share the same weights. During training, sparse prediction is used, so the language-modeling head is evaluated only on masked token positions.

Training

The model was pretrained for one epoch on salyamq/kk-corpus-v1. The model was trained in bfloat16 precision with gradient checkpointing enabled.

The training optimizer was a combined Muon and AdamW optimizer:

• Muon was used for the Transformer hidden-state matrix parameters.
• AdamW was used for token embeddings, the masked-language-modeling head, normalization parameters, biases, and other non-Muon parameters.
• Muon learning rate: 0.02
• AdamW learning rate: 0.0003
• Betas: (0.9, 0.95)
• Epsilon: 1e-10
• Weight decay: 0.01
• Muon momentum: 0.95
• Muon update: Newton-Schulz orthogonalization with 5 iterations
• Nesterov momentum was enabled

The masked-language-modeling objective used a masking probability of 22% with the standard 80/10/10 masking strategy.

The 16-layer attention schedule contains 11 sliding-window layers and 5 global-attention layers:

[sliding, sliding, global] × 5 → sliding

Component	Value
Parameters	100,724,224
Layers	16
Hidden size	768
Attention heads	12
Head dimension	64
SwiGLU intermediate size	1,024
Vocabulary size	32,000
Maximum context	8,192 tokens
Local attention window	128
Global RoPE theta	160,000
Local RoPE theta	10,000
Normalization	RMSNorm
Weight tying	Input embeddings ↔ MLM decoder
Checkpoint precision	bfloat16

Context length and document packing

The model was trained on fixed-length 8,192-token packed sequences without padding. Multiple documents can be packed into one training sequence. Attention is isolated at document boundaries, and RoPE positions restart for every document, preventing unrelated packed documents from attending to each other.

Therefore, the supported maximum context length is 8,192 tokens, but this does not mean that every individual training document was exactly 8,192 tokens long. Long-context quality should be measured separately for each downstream use case.

Training overview

• Objective: masked language modeling (MLM)
• Training data: salyamq/kk-corpus-v1
• Sequence length: 8,192 packed tokens
• Precision: bfloat16
• Optimizer: Muon for hidden matrix parameters; AdamW for embeddings, norms, biases, and the MLM head
• Attention: padding-free variable-length attention with document isolation
• Position encoding: RoPE with separate theta values for local and global attention layers
• Efficiency: sparse MLM prediction during pretraining and tied input/output token embeddings

Training-step counts and downstream benchmark results are not included in this release, so no unreported evaluation scores should be inferred from this card.

Tokenizer

SmqBERT uses a 32,000-piece SentencePiece tokenizer. The primary special tokens are:

Token	ID
[UNK]	0
[PAD]	1
[BOS]	2
[EOS]	3
[CLS]	4
[SEP]	5
[MASK]	6

For masked-token prediction, place the literal [MASK] token in the input text.

Performance notes

• The model works without optional fused kernels by using PyTorch SDPA.
• On supported NVIDIA GPUs, FlashAttention can substantially improve long-context speed and memory usage.
• flash-attn is optional and is not required for ordinary CPU/GPU inference.
• Processing the full 8K context with the unfused fallback may be considerably slower and require more memory.

Optional FlashAttention installation:

pip install flash-attn --no-build-isolation

Intended uses

Suitable uses include:

• Kazakh masked-token prediction;
• contextual embeddings and feature extraction;
• fine-tuning for classification, token classification, retrieval, and other encoder-based Kazakh NLP tasks;
• continued pretraining on domain-specific Kazakh text.

Limitations

• SmqBERT is specialized for Kazakh; quality can degrade for other languages.
• It may reproduce biases, errors, or harmful associations present in its pretraining corpus.
• MLM predictions are contextual token completions, not verified factual answers.
• The model is not designed for dialogue or free-form text generation.
• No task-specific benchmark suite is reported in this release.
• Applications in high-impact domains should include task-specific evaluation and human oversight.

Loading custom code safely

trust_remote_code=True executes the modeling files stored in this repository. For reproducible or security-sensitive deployments, review the code and pin a specific Hub revision:

revision = "YOUR_COMMIT_HASH"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    revision=revision,
    trust_remote_code=True,
)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    revision=revision,
    trust_remote_code=True,
)

Citation

If you use this model, please cite:

@misc{tokhtobayev2026smqbert,
  author       = {Tokhtobayev, Salim},
  title        = {smqBERT-kk-small},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/salyamq/smqBERT-kk-small}}
}

## License

Apache 2.0