rostlabs/rost-1b-base

rost is a bilingual Romanian/English language model trained from scratch, with a purpose-built Romanian tokenizer rather than one inherited from an English model. This repository holds the pretrained base model.

Model overview

parameters	1.384B total (24 layers, 1,536 hidden, 12 heads)
context length	4,096 tokens
vocabulary	32,768, bilingual -- rostlabs/rost-tok-bilingual
position encoding	RoPE, theta 100,000
attention	sliding-window pattern SSSL; 18 of 24 layers see a quarter context
precision on disk	float32 safetensors. Load as bfloat16 -- that is what it trained in
stage	base, checkpoint step 011136
languages	Romanian (primary), English

Not a Llama or Mistral derivative: the architecture is its own, so the modelling code ships in this repository and trust_remote_code=True is required.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "rostlabs/rost-1b-base", trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()
tokenizer = AutoTokenizer.from_pretrained("rostlabs/rost-1b-base")

messages = [{"role": "user", "content": "Care este capitala Romaniei?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(inputs["input_ids"].to("cuda"), max_new_tokens=128)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The weights are stored in bfloat16, the precision the model was trained in and the precision the forward pass casts to regardless. They were published as float32 at first, because release verification compares logits exactly and that is only meaningful in float32 — but that made the files twice the size for no information: 47% of the parameters were already bfloat16-exact, and under bfloat16 the two files produce bitwise identical logits. Loading in float32 is still possible and shifts the top-8 probabilities by up to 1.7e-03, a mode the model was never trained in.

Measured on a single RTX 5070 (12 GB, SDPA path): ~91 tokens/second, ~3.3 GB resident.

Recommended sampling parameters

generation_config.json carries these, so model.generate() uses them without being asked:

{
  "do_sample": true,
  "temperature": 0.8,
  "top_k": 50,
  "repetition_penalty": 1.1,
  "eos_token_id": [32759, 32763]
}

The repetition penalty is not decoration. Measured over 198 generations of 400 tokens, on prompts chosen to provoke the failure:

decoding	replies that looped	worst repeated 6-gram
greedy	50%	44x
temperature 0.2, no penalty	44%	44x
temperature 0.6, no penalty	22%	10x
temperature 0.6, penalty 1.1	0%	2x

A penalty of 1.1 also raises the share of replies that end by emitting their stop token, rather than running out of budget, from 50% to 83%. Stronger settings suppress repetition further and cost accuracy: at 1.3 the model stopped looping and began inventing etymologies, so 1.1 is the mildest setting that works and that is why it is the default.

Before this revision the repository set no eos_token_id in any file, so generate() had no stop condition and ran to max_new_tokens on every call.

Running it with llama.cpp

No GGUF files are published for this base checkpoint; the quantisations at rostlabs/rost-1b-instruct-GGUF are of the instruct model. The same conversion produces them from these weights.

Either way the file needs a llama.cpp with this architecture compiled in, because llama.cpp compiles architectures in rather than loading them dynamically. A fork with it applied is at rostlabs/llama.cpp, whose master is upstream master plus one commit.

Not available

Ollama and LM Studio. Both bundle their own llama.cpp, so neither will read those files until this architecture is merged upstream. That is prepared but not submitted.

Tool and function calling. The instruct checkpoint has had conversational fine-tuning only. No agent fine-tuning has been applied, so the tool tokens are not reachable under any prompt framing. Do not build an agent on this checkpoint yet.

Training

tokens	11.68B, single pass -- no data was repeated
optimizer steps	11,136 at a 1,048,576-token batch
hardware	8x H100 80GB, ~3.6 hours
schedule	warmup-stable-decay, decay over the final 30%
precision	bfloat16 with FP8 matmuls

Trained in two phases, which is the substance of the recipe rather than a detail:

	Romanian	English (ClimbMix)	DQA	code
phase 1, steps 0-7,795	30%	60%	5%	5%
phase 2, steps 7,795-11,136	55%	35%	10%	--

The second phase raises Romanian while the learning rate decays, so the model finishes on a Romanian-heavy diet. A d6 rehearsal measured Romanian improving roughly 8x faster than English across that phase.

Data

source	licence
Romanian: FineWeb2-ro, educational score >= 3, diacritic-normalised	ODC-BY
English: Nemotron ClimbMix	CC-BY-NC-4.0
High-quality QA: Nemotron-CC-v2.1 HQ-DQA	gated
Code: Nemotron-CC-Code-v1 (phase 1 only)	gated
Chat fine-tuning: OpenLLM-Ro sets	CC-BY-NC-4.0

Evaluation

OpenLLM-Ro suite, base checkpoint, 400 rows per task, zero-shot, scored by likelihood over the options:

task	accuracy	normalised	chance
ro_hellaswag	31.25	38.50	25.0
ro_truthfulqa (MC1)	21.25	34.00	14.3
ro_arc_challenge	24.50	30.25	28.6
ro_mmlu	28.50	29.75	25.0
ro_winogrande	52.25	52.25	50.0
mean (normalised)		36.95	28.6

Read these against the chance column, not on their own. The model is clearly above chance on sentence completion and truthfulness, and close to chance on ro_arc_challenge and ro_winogrande. At 400 rows a task, differences under about 5 points are inside the noise.

These are not comparable to the OpenLLM-Ro leaderboard. Published figures there average each task over several few-shot settings; these are zero-shot, which understates them. For scale rather than ranking: RoLlama2-7b-Base reports a 42.05 four-task accuracy average against this model's 34.12 -- from a model 5x larger trained on far more data.

English suite, base checkpoint, EleutherAI lm-evaluation-harness 0.4.12, zero-shot except where marked, full test sets, scored by likelihood over the options -- the standard method for base models, so these are comparable to published tables produced at the same settings:

task	accuracy	normalised	chance
sciq	77.8	73.3	25.0
hellaswag	37.7	46.5	25.0
piqa	61.6	61.6	50.0
boolq	58.1	--	~50
arc_easy	43.0	41.8	25.0
lambada_openai	34.5	--	--
winogrande	52.5	--	50.0
arc_challenge	23.9	27.9	25.0
openbookqa	16.6	26.0	25.0
mmlu (5-shot)	26.7	--	25.0

The same reading rule applies: judge against the chance column. Science QA, sentence completion and physical common sense are clearly learned; winogrande, arc_challenge, openbookqa and mmlu are at or near chance; and lambada is the weakest skill, consistent with English being the minority share of an 11.68B-token budget. For scale rather than ranking: on the completion tasks this sits between Pythia-410M and Pythia-1B -- models that saw roughly 40x more English text.

Also measured: CORE metric 0.2450 on the base model (0.1608 at step 2,000), and 0.3021 validation bits-per-byte for the instruct checkpoint against 0.5113 for the base model it started from.

Intended use

Research on Romanian language modelling, Romanian text generation and completion, and as a base for further fine-tuning. It is small enough to run on a consumer GPU, which is the point.

Out of scope: anything requiring factual reliability, agent or tool use, long-context work beyond 4,096 tokens, commercial deployment (see the licence), and any decision affecting a person's rights, health, safety or finances.

Limitations

• It confabulates confidently. Asked about Bucharest it correctly names the capital and then places it in the wrong county. Specifics need checking.
• It repeats. Restating a sentence with the clauses swapped is a common failure, inherited from the base model and only partly removed by fine-tuning. Decode with the shipped repetition_penalty of 1.1; with penalties off and a low temperature it will repeat one sentence until it runs out of tokens.
• Reasoning is near chance. See ro_arc_challenge and ro_winogrande above.
• 4,096 tokens of context, well short of contemporary models.
• Domain skew. The Romanian pretraining data is roughly one third health content by character count, with history, geography, finance and education each near 10%, and entertainment, gaming and software each under 1%. Fluency is uneven accordingly.
• The Romanian corpus is internally duplicated, which inflates absolute Romanian bits-per-byte by around 0.09. Comparisons on fixed sets are unaffected.
• No safety tuning of any kind has been applied.

Licence

CC-BY-NC-4.0, non-commercial. Both halves of the training data carry non-commercial terms -- ClimbMix is CC-BY-NC-4.0 ("for research and development only") and the OpenLLM-Ro fine-tuning sets are CC-BY-NC-4.0 -- and the model inherits them. The ClimbMix mirror used is tagged MIT, which does not override the upstream terms. The Romanian data is ODC-BY and requires attribution to FineWeb2.

Citation

@misc{rost2026,
  title  = {rost: a bilingual Romanian-English language model trained from scratch},
  author = {Iancu, Stefan},
  year   = {2026},
  url    = {https://huggingface.co/rostlabs/rost-1b-base}
}

The original training checkpoint (model_*.pt, meta_*.json) ships alongside the safetensors, for use with the training code.