CRIA-LM-75M-Instruct

Cria (noun): a baby llama, alpaca, vicuña, or guanaco. Pronounced ˈkrē-ə.

~ Merriam-Webster

---

---

cRia-LM-75M-Instruct is a 75.7M-parameter instruction-tuned language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters on the second traversal.

The model starts from cRia-LM-75M, then adds supervised instruction tuning and preference optimization. It ships with a native chat template using <|im_start|> and <|im_end|> markers.

Model Details

Specification	Value
Parameters	75.7M
Architecture	Relaxed Recursive Transformer
Unique Transformer layers	13
Effective depth	24
Prelude layers	1
Shared recurrent layers	11
Recurrent passes	2
Coda layers	1
Hidden size	576
MLP intermediate size	1,536
Attention heads	9 query heads
KV heads	3
Attention type	Grouped-query attention
Head dimension	64
MLP	SwiGLU
Normalization	RMSNorm
Attention normalization	QK-Norm
Position encoding	RoPE
RoPE theta	100,000
Context length	4,096 tokens
Vocabulary size	49,152
Tokenizer	SmolLM2 BPE tokenizer, per-digit tokenized
Token embedding	Tied, factorized
Embedding rank	210
Recurrent LoRA rank	172
Residual gains	35 learned scalars
Chat format	`<
Model type	Instruction-tuned causal language model

Architecture

cRia-LM-75M-Instruct keeps the base model's 13 unique Transformer layers:

1 prelude + (11 shared layers x 2 recurrent passes) + 1 coda

This gives an effective depth of 24 Transformer layers while storing the main parameters for 13 unique layers. The recurrent block shares its base weights across both passes. Rank-172 LoRA updates provide separate capacity on the second pass.

The full architecture and base-model history are documented on the cRia-LM-75M model card.

Training

Post-training starts from cRia-LM-75M. Base-model training details are kept on that model card.

Supervised fine-tuning

The first part uses the full training split of HuggingFaceTB/smol-smoltalk. Only assistant turns contribute to the language-model loss.

Setting	Value
Epochs	2
Maximum sequence length	4,096 tokens
Effective batch size	64 examples
Schedule	Cosine decay with 10% warmup
Muon peak LR	0.006
AdamW peak LR	1e-3
Recurrent fast-group peak LR	2e-3
Residual-gain peak LR	1e-4
Weight decay	0.01
Gradient clipping	1.0
Seed	42

HuggingFaceTB/SmolLM2-360M-Instruct supplies same-tokenizer KL targets during the first SFT epoch. The teacher temperature is 1.2. A lagged CE-to-KL ratio sets the distillation scale after 50 calibration steps, ramps it over 250 steps, and caps it at 0.5 so ground-truth cross-entropy remains the main objective.

Muon updates the ordinary two-dimensional Transformer matrices. AdamW handles embeddings, norms, recurrent LoRA and QK-Norm parameters, and the residual-gain scalars.

Direct Preference Optimization

The SFT checkpoint is then trained on the train_prefs split of HuggingFaceH4/ultrafeedback_binarized.

Setting	Value
Epochs	2
Maximum sequence length	1,024 tokens
Maximum prompt length	512 tokens
Starting per-device batch size	64 pairs
Gradient accumulation	2
Optimizer	Fused AdamW
Learning rate	1e-6
DPO beta	0.5
Schedule	Cosine decay with 10% warmup
Weight decay	0.01
Gradient clipping	1.0
Seed	42

Reference log probabilities are precomputed from the SFT checkpoint. Training uses BF16 computation with TF32 matrix multiplication enabled.

Evaluation

The tables below report cRia-LM-75M-Instruct at revision 57c83387f67b8c808b9f4f18c1c0047b72f66e24.

All runs used complete evaluation sets with no sample limits. Model tensors and log-softmax calculations used FP32, CUDA matrix multiplication used TF32, and the shared evaluation cap was 2,048 tokens. SmolLM and SmolLM2 were evaluated under the same local protocol. These numbers were measured for this card and were not copied from their model cards.

Instruction benchmarks

IFEval and BBH use each checkpoint's native chat template. BBH is zero-shot. MT-Bench uses the official 80-question, two-turn set with single-answer grading by gpt-5.6-luna.

Benchmark	Metric	cRia-LM-75M-Instruct	SmolLM-135M-Instruct	SmolLM2-135M-Instruct
IFEval	strict prompt/instruction average	26.93	12.81	28.31
IFEval	prompt-level strict accuracy	20.89	7.39	21.26
IFEval	instruction-level strict accuracy	32.97	18.23	35.37
BBH	zero-shot exact match	24.51	19.15	25.69
MT-Bench	overall score, /10	1.76	1.60	2.01
MT-Bench	first-turn score, /10	2.19	1.91	2.63
MT-Bench	second-turn score, /10	1.33	1.29	1.40

Basic text benchmarks

HellaSwag, ARC-Easy, ARC-Challenge, and PIQA use zero-shot normalized continuation likelihood with each model's native chat template. ArithMark-3 uses its raw-text continuation protocol at a 1,024-token context cap.

Benchmark	Metric	cRia-LM-75M-Instruct	SmolLM-135M-Instruct	SmolLM2-135M-Instruct
HellaSwag	acc_norm	32.07	38.55	40.34
ARC-Easy	acc_norm	37.75	42.05	45.96
ARC-Challenge	acc_norm	25.17	26.19	28.84
ARC average	mean acc_norm	31.46	34.12	37.40
PIQA	acc_norm	59.68	64.80	67.30
ArithMark-3	acc_norm	35.20	36.90	40.00

These scores compare checkpoints under one fixed local setup. cRia has 75.7M parameters, while both comparison models have about 135M. Prompt format, harness version, precision, and shot count can move small-model scores by several points.

Usage

The model ships with a custom Transformers implementation, so trust_remote_code=True is required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sz14/cRia-LM-75M-Instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device).eval()

messages = [
    {"role": "system", "content": "Answer clearly and concisely."},
    {"role": "user", "content": "Explain why the sky is blue."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
    return_dict=True,
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
    )

new_tokens = output[0, inputs.input_ids.shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The chat template inserts a default system message when one is not supplied. Pass a system message explicitly when an application needs consistent behavior.

Intended Use

cRia-LM-75M-Instruct is intended primarily for:

• research on compact instruction-tuned models
• experiments with recursive parameter sharing
• local prototypes with limited memory
• further supervised fine-tuning and preference optimization
• testing small-model chat and tool pipelines

Limitations

cRia-LM-75M-Instruct is a very small language model. Its responses can be short, repetitive, incorrect, or poorly formatted.

Known limitations include:

• weak multi-step reasoning
• limited factual knowledge
• low instruction-following reliability on complex prompts
• substantial quality loss across multi-turn conversations
• potential factual errors and hallucinations
• English-focused training
• a 4,096-token maximum context window

The model should not be treated as a reliable source of factual information or used without additional validation in high-stakes applications.

Architecture and Training Lineage

The Transformer layer design follows HuggingFaceTB/SmolLM2-135M, released by Hugging Face under the Apache 2.0 license.

The recursive parameter-sharing approach is based on:

Bae et al., "Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA," arXiv:2410.20672.

The SFT distillation scale follows the pooled, lagged balancing idea used in cross-tokenizer distillation work:

Sreenivas et al., "X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation," arXiv:2605.21699.

License

cRia-LM-75M-Instruct is released under the Apache License 2.0.