plasmova/nova-v3

Nova v3

Nova v3 is a decoder-only causal language model exported from the native Nova v3 checkpoint. This repository includes the model weights, tokenizer, and the custom Transformers configuration and modeling code needed to load it.

Model details

• Vocabulary size: 32,768
• Hidden size: 640
• Transformer layers: 26
• Attention heads: 10 query heads and 5 key/value heads
• Maximum context: 2,048 tokens
• Checkpoint step: 3,500

The exported model supports key/value (KV) caching during generation. The cache is enabled by default and can be explicitly controlled with use_cache=True or use_cache=False.

Install

Install PyTorch and Transformers, then load the model with its tokenizer:

pip install torch transformers

The custom modeling code runs on Transformers 4.57 and on 5.x. On 5.x it writes to the key/value cache through the Cache API, and it still accepts the legacy tuple cache of 4.x.

Inference with a system prompt

The model uses custom code in this repository. Review that code before enabling trust_remote_code.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "plasmova/nova-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

system_prompt = "You are Nova, a helpful assistant."
user_prompt = "Explain why the sky appears blue in one sentence."
prompt = (
    f"<|system|>{system_prompt}<|/system|>"
    f"<|user|>{user_prompt}<|/user|><|assistant|>"
)
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=128,
    eos_token_id=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|/assistant|>")],
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The system_prompt is optional and can be customized. The example runs on CPU by default. Nova supports up to 2,048 tokens of context; for GPU inference, move the model and inputs to the GPU and select a supported dtype.

Evaluation

Zero-shot evaluation with lm-evaluation-harness 0.4.13 used the first 500 examples of each task on CPU. These are limited samples, not full validation-set results.

Benchmark	Samples	Accuracy	Normalized accuracy
HellaSwag	500	37.4%	43.6%
ARC Easy	500	51.0%	48.2%
ARC Challenge	500	21.8%	25.0%
WinoGrande	500	52.6%	52.2%
OpenBookQA	500	18.8%	30.0%

A €0-budget result against SmolLM2-135M

At an author-reported €0 training spend, Nova v3 (148.8M parameters) showed a narrow WinoGrande edge over SmolLM2-135M in a matched, zero-shot lm-evaluation-harness 0.4.13 run. It was within one point on both tasks:

Benchmark	Nova v3	SmolLM2-135M	Nova delta
HellaSwag (acc_norm)	43.6%	44.2%	−0.6 pp
WinoGrande (acc_norm)	52.2%	50.6%	+1.6 pp

Each score uses 500 examples. acc_norm follows lm-evaluation-harness's normalized multiple-choice scoring, which divides each candidate log-likelihood by the character length of its choice string. Both WinoGrande margins are smaller than the approximate 2.2-point standard errors, so treat them as directional early signals rather than statistically established wins. For additional context, the SmolLM2-135M model card reports 42.1% on HellaSwag and 51.3% on WinoGrande using its own evaluation setup; those published scores are not directly comparable to this matched run.

Per-run scores and standard errors are available in benchmark_comparison.json.

GSM8K: tied score on a 100-question sample

In a matched run on the first 100 GSM8K test questions, Nova v3 and SmolLM2-135M-Instruct each answered 4/100 correctly (4.0% exact match). Both used five examples in context, deterministic generation, CPU inference, and lm-evaluation-harness 0.4.13 with the same plain GSM8K prompt format. The approximate standard error is 2.0 percentage points for each score. This is a small sample: it shows equal results in this run, not that the models are equivalent overall.

Details and per-run metrics are available in gsm8k_comparison.json.

License and training data

The model weights and accompanying custom model code are released under the Apache License 2.0. See LICENSE.

The v3 training code specifies a pretraining mixture of FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath, Wikipedia, and CodeParrot, followed by SmolTalk fine-tuning. Dataset contents retain their respective upstream terms; see the linked dataset cards for details.