CRIA-LM-75M

Cria (noun): a baby llama, alpaca, vicuña, or guanaco. It is pronounced ˈkrē-ə.

~ Mirriam Webster

---

---

cRia-LM-75M is a 75.7M-parameter base language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters providing additional capacity on the second traversal.

The model was trained for approximately 10B tokens using standard cross-entropy loss combined with logit-level knowledge distillation from HuggingFaceTB/SmolLM2-360M. The Transformer layer architecture follows HuggingFaceTB/SmolLM2-135M.

This is a base model. It has not been instruction tuned and does not use a chat template. Layer architecture is based upon HuggingFaceTB/SmolLM2-135M with QK Norm added for training stability.

Model Details

Specification	Value
Parameters	75.7M
Architecture	Relaxed Recursive Transformer
Unique Transformer layers	13
Effective depth	24
Prelude layers	1
Shared recurrent layers	11
Recurrent passes	2
Coda layers	1
Hidden size	576
MLP intermediate size	1,536
Attention heads	9 query heads
KV heads	3
Attention type	Grouped-query attention
Head dimension	64
MLP	SwiGLU
Normalization	RMSNorm
Attention normalization	QK-Norm
Position encoding	RoPE
RoPE theta	100,000
Context length	2,048 tokens
Vocabulary size	49,152
Tokenizer	SmolLM2
Token embedding	Tied, factorized
Embedding rank	210
Recurrent LoRA rank	172
KV cache	Not implemented
Model type	Base causal language model

Architecture

cRia-LM-75M uses 13 unique Transformer layers arranged as:

1 prelude + (11 shared layers x 2 recurrent passes) + 1 coda

This gives an effective computational depth of 24 Transformer layers while storing the main parameters for only 13 unique layers.

The 11-layer recurrent block is shared between both passes. On the second pass, the recurrent linear projections receive rank-172 LoRA updates. These pass-specific parameters allow the two traversals to specialize while retaining the parameter efficiency of a shared block.

Each Transformer layer uses the same core architectural design as SmolLM2-135M:

• hidden size of 576
• 9 query heads and 3 KV heads with grouped-query attention
• head dimension of 64
• SwiGLU feed-forward network with intermediate size 1,536
• RMSNorm
• rotary position embeddings
• QK-Norm

Factorized Tied Embedding

The input embedding and language-model readout are tied through a rank-210 factorization:

49,152 x 210
210 x 576

The same factors are used for token lookup and output projection. This substantially reduces the parameter cost of the 49,152-token vocabulary while retaining the full vocabulary at the output.

Training

cRia-LM-75M was trained using knowledge distillation from the base HuggingFaceTB/SmolLM2-360M model.

Training uses a combination of:

• token-level logit knowledge distillation
• standard next-token cross-entropy
• dynamic CE/KD scale balancing

The training run processed approximately 10 billion tokens at a sequence length of 2,048 tokens.

Training Data

The primary training mixture used the following sampling weights:

Data source	Sampling weight
FineWeb-Edu	48%
DCLM-Edu	32%
Cosmopedia-v2	12%
FineMath-4+	5%
StarCoder Python	3%

The mixture combines educational web text, general text, synthetic educational material, mathematics, and Python code.

Optimization

Setting	Value
Training tokens	~10B
Sequence length	2,048
Objective	Dynamic CE + logit KD
Teacher	HuggingFaceTB/SmolLM2-360M
Optimizers	Muon + AdamW
LR schedule	Warmup-stable-decay
Warmup	1% of training
Decay start	80% of training
Final decay portion	20%

Muon is used for matrix parameters, with AdamW used for the remaining parameter groups.

Evaluation

Evaluation was performed zero-shot using lm-evaluation-harness in bfloat16 precision.

Benchmark	Metric	Score
HellaSwag	acc_norm	33.68
ARC-Easy	acc_norm	46.76
ARC-Challenge	acc_norm	23.46
PIQA	acc_norm	62.51
OpenBookQA	acc_norm	33.20
CommonsenseQA	acc	19.57
ArithMark-3	acc_norm	35.50

ArithMark-3 uses the length-normalized continuation log-likelihood over the 1,000-example evaluation set.

Open SLM Leaderboard Intelligence Index: 16.24

Benchmark results should be interpreted in the context of the model's size and base-model status. Scores may vary slightly with evaluation harness version, precision, and batching configuration.

Usage

The model uses a custom Transformers architecture, so trust_remote_code=True is required.

The tokenizer is compatible with SmolLM2.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sz14/cRia-LM-75M"
tokenizer_id = "HuggingFaceTB/SmolLM2-135M"

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
).to(device).eval()

prompt = "In mathematics, a vector is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Generation Performance

The current implementation does not provide a KV cache. Autoregressive generation therefore recomputes the active prefix for every generated token.

This has little effect on parameter count or language-model evaluation, but generation latency scales less favorably with sequence length than it would for a similarly sized model with cached keys and values.

Intended Use

cRia-LM-75M is intended primarily for:

• research on recursive parameter sharing
• experiments with compact language models
• further pretraining and domain adaptation
• supervised fine-tuning
• small language-model backbones
• architecture and knowledge-distillation research

Because this is a base model, prompts are treated as ordinary text continuation rather than instructions.

Limitations

cRia-LM-75M is a small base language model and should not be expected to match substantially larger pretrained models.

Known limitations include:

• weak multi-step reasoning
• limited factual knowledge
• potential factual errors and hallucinations
• no instruction-following training
• no chat template
• English-focused training
• a 2,048-token context window
• no KV cache in the current implementation

The model should not be treated as a reliable source of factual information or used without additional validation in high-stakes applications.

Architecture and Training Lineage

The Transformer layer design follows HuggingFaceTB/SmolLM2-135M, released by Hugging Face under the Apache 2.0 license.

HuggingFaceTB/SmolLM2-360M was used as the teacher during logit knowledge distillation.

The recursive parameter-sharing approach is based on:

Bae et al., "Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA," arXiv:2410.20672.

License

cRia-LM-75M is released under the Apache License 2.0.