Boris-2-Preview-2

The second preview of Boris-2, a 144M-parameter base language model from OpenCerebral, trained from scratch on a single NVIDIA V100. This preview is the main run's 100B-token checkpoint with a short learning-rate decay (2.5B tokens) applied, the same decay length as Boris-2-Preview-1. The full run continues to 200B tokens.

This is a base model. It continues text; it does not follow instructions or chat.

Results

Zero-shot, on the float32 weights in this repository: lm-evaluation-harness 0.4.12 for the first four, the leaderboard's own bencharithmark-3.py (unmodified, default settings) for ArithMark-3.

Benchmark	Metric	Preview-1	Preview-2
HellaSwag	acc_norm	45.23	46.60
ARC-Easy	acc_norm	58.38	59.13
ARC-Challenge	acc_norm	29.95	30.20
PIQA	acc_norm	69.21	70.84
ArithMark-3	acc_norm	37.60	42.20
Open SLM Index		27.91	30.58

The Index is the Open SLM Leaderboard formula: (N(HellaSwag,25) + N(mean(ARC-E, ARC-C),25) + N(PIQA,50) + 0.65 * N(ArithMark-3,25)) / 3.65, with N(v,c) = 100(v-c)/(100-c).

Single-checkpoint scores on these benchmarks move by roughly plus or minus 0.8 Index between neighbouring checkpoints. No checkpoint or seed was selected on these scores: the decay start (100B) and length (2.5B tokens) were fixed in advance, and this is the one checkpoint at its end. The decay data mix was chosen from 200M-token pilot decays on an earlier checkpoint, scored on these benchmarks plus SVAMP and ASDiv as held-out word-problem checks.

The decay moved the scores from HellaSwag 40.75, ARC-Easy 56.19, ARC-Challenge 29.18 and PIQA 69.37 at the 100B checkpoint to the figures above. ArithMark-3 rose 4.6 points over Preview-1, from the grade-school word-problem data added to the decay. That gain is specific to ArithMark's skills so far: on SVAMP (35.3) and ASDiv (39.3), scored as four-way multiple choice, Preview-2 is level with the pilot baseline (36.0 and 38.5).

Model

Field	Value
Parameters	143.7M
Transformer blocks	111.9M
Token embedding (tied with output)	16.8M
N-gram table	15.0M
Layers	31
Hidden size	512
Attention	8 heads of 64, full multi-head
MLP	SwiGLU, inner 1664
Context	2048 tokens
Position encoding	RoPE, theta 10,000
Vocabulary	32,768 (byte-level BPE, trained for this model)

Two components differ from a standard decoder:

• Canon layers. Each block applies a causal 3-tap depthwise convolution to the fused query/key/value projection, so every position mixes in the two previous positions before attention.
• N-gram table. An exact lookup on the last two and three token ids (210,938 bigrams and 23,437 trigrams, no hashing) retrieves a learned rank-64 vector that is added to the residual stream before block 2. It is a gather, not a matrix multiply.

Training packed several documents per sequence with attention, Canon and the n-gram lookup all masked at document boundaries. The modeling code reproduces this: a new document starts after every end-of-text token (eos_resets_context in the config).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "opencerebral/Boris-2-Preview-2"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

x = tok("The water cycle begins when", return_tensors="pt")
y = model.generate(**x, max_new_tokens=40, do_sample=False)
print(tok.decode(y[0], skip_special_tokens=True))

The custom code is plain PyTorch with no extra dependencies. It has no KV cache, so generation recomputes the prefix every step; at this size that is fast enough for sampling and evaluation. Run in float32 or bfloat16. The residual stream is kept in float32 internally because one channel reaches values near the float16 limit.

The converted model was checked against the training code on the same weights: logits match to within 4e-5 in float32, and lm-evaluation-harness reproduces the training evaluator's benchmark scores to within one item.

Training

Field	Value
Tokens	102.5B (100.0B stable + 2.5B decay)
Hardware	1x NVIDIA V100 32GB
Precision	float16 mixed precision, float32 residual stream
Batch	524,288 tokens per step, sequence length 2048
Optimizer	Muon (matrices, lr 0.0025) and AdamW (embedding, n-gram table, norms, Canon; lr 0.003)
Weight decay	0.1 on matrices, embedding and n-gram table, and on Canon from 41B tokens; none on norm gains
Schedule	warmup, constant learning rate, then 1-sqrt decay to zero
Throughput	about 59,500 tokens per second, using hand-written Volta attention and fused kernels

Main run data, by tokens: up to 55B tokens, about 89% ClimbMix and 11% FineMath; from 55B tokens, about 90% ClimbMix, 10% UltraData-Math (L3 synthetic QA) and 0.15% Orca-Math.

Decay data, by tokens:

Source	Share
ClimbMix	68.35%
Cosmopedia WikiHow-style tutorials	12%
UltraData-Math L3	10%
MetaMathQA, GSM8K-derived rows only (training split)	5%
DCLM-Baseline	3%
Calc-ape210k and Calc-mawps, training splits only	1.5%
Orca-Math	0.15%

Contamination

No benchmark training, validation or test split was trained on. MetaMathQA's rows are rewrites of the GSM8K training set; GSM8K is not one of the evaluated benchmarks. ASDiv and SVAMP, used only as held-out checks during development, were not trained on.

• Every source except ClimbMix was filtered against every evaluation item (HellaSwag, ARC, PIQA, ArithMark-3) with a 13-gram overlap check. ClimbMix, the bulk of the data, has not been filtered.
• About two thirds of HellaSwag is built from WikiHow articles, and the Cosmopedia tutorials were generated from WikiHow titles. As for Preview-1, the validation items were split by whether their WikiHow title is also one of the 177,353 tutorial titles, and the 100B checkpoint was compared with this one:

HellaSwag items	n	100B	Preview-2	Gain
WikiHow, title in the tutorial data	2,966	39.92	47.13	+7.22 ± 1.13
WikiHow, title not in the tutorial data	3,833	39.53	46.57	+7.04 ± 1.07
ActivityNet (no WikiHow link)	3,243	42.95	46.13	+3.18 ± 1.09

The two WikiHow groups gained the same amount (0.2 points apart, well within noise). The gain is a model getting better at how-to text in general, not at the specific topics in the tutorial data.

Limitations

A 144M-parameter base model trained on 102.5B tokens. It makes factual and arithmetic mistakes, loses track of longer arguments, and repeats itself under greedy decoding. It has no safety tuning. It is a research preview, not a product.

Sample outputs

Greedy decoding, float32, unedited, cut for length. The same twelve prompts as Preview-1; all twelve are in samples.json.

Prompt	Completion
To solve 47 + 68, first	add 47 and 68, then add 68, what is the sum? Answer: x=47+68 = 105
A prime number is	a natural number greater than 1 that has no positive divisors other than 1 and itself.
Photosynthesis is the process by which	plants and other organisms use light energy to convert carbon dioxide and water into sugar and oxygen.
How to boil an egg: First,	you need to crack the egg into a bowl. Then, you need to put the egg in a pot of boiling water. ...
To tie a shoelace, start by	threading the yarn through the eye of the needle. ...
The water cycle begins when	the sun heats the earth's surface. The water vapor in the air condenses into liquid water. ...
If a train travels 60 miles in 2 hours, its speed is	? Answer: x=60/2 = 30
Sam had 24 apples and gave away 9, so Sam now has	24 - 9 = 15 apples. #### 15 The answer is: 15
The capital of France is	Paris, and the capital of the United States is Washington. The capital of the United Kingdom is London. ...
Gravity is the force that	pulls objects towards the center of the Earth. It is a fundamental concept in physics and is described by Newton's law of universal gravitation.
The area of a rectangle is found by	multiplying its length by its width.
Plants need sunlight because	they need it to grow. Sunlight is the energy source for photosynthesis, which is the process by which plants convert light energy into chemical energy.

Compared with Preview-1, the train question and France's capital now come out right, and the last prompt no longer loops. 47 + 68 is still wrong (105), the egg is cracked before boiling, and the shoelace prompt drifts into sewing. The word-problem data shows in the answer format (#### 15, The answer is:).