Veyra2-Apricot-50M-Base

Veyra2-Apricot-50M-Base is a 49.3M-parameter Llama-like causal language model trained from scratch on approximately 20B tokens.

It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation. If you are looking for a base model that is better at prompt continuation you should use veyra-ai/Veyra-30M-Base instead.

Model Details

Parameters: 49,303,040
Architecture: Llama-like causal LM
Layers: 16
Hidden size: 512
Attention heads: 8
KV heads: 2
Head dim: 64
Intermediate size: 1408
Vocabulary size: 8192
Context length used in training: 1024
Activation: SwiGLU / SiLU
Normalization: RMSNorm
Attention: GQA with QKV-Norm
Positional encoding: RoPE
Weight tying: tied input embeddings and LM head
Training tokens: approximately 19.99B
Training precision: bfloat16
Optimizer: Muon

Tokenizer

Special tokens:

• <|bos|>: 0
• <|eos|>: 1
• <|pad|>: 2
• <|unk|>: 3
• <|im_start|>: 4
• <|im_end|>: 5

Training Data

The model was trained on a 20B-token pretraining mixture.

• FineWeb-Edu sample-10BT: 45%
• Cosmopedia-v2: 30%
• DCLM-Edu: 15%
• FineMath finemath-3plus: 10%

Training Summary

• Final step: 9532
• Tokens seen: 19,990,052,864
• Tokens per step: 2,097,152
• Global batch size: 2048
• Sequence length: 1024
• Last train loss: 2.5062
• Final eval loss: 2.4978
• Final eval perplexity: 12.16

Evaluation

SciCloze Eval

• SciCloze-900 accuracy: 47.56%
• SciCloze-900 correct: 428 / 900

SciCloze-900 repository: veyra-ai/SciCloze-900

SciCloze-900 subject breakdown:

• Biology: 53.67%
• Physics: 45.67%
• Chemistry: 43.33%

SciCloze-900 empirical difficulty breakdown:

• Easy: 94.41%
• Medium: 47.71%
• Hard: 7.81%

Other Evals

Zero-shot results:

Open SLM Bench verified:

• Average: 38.81%
• PIQA acc_norm: 62.13%
• ARC-Easy acc_norm: 42.47%
• ARC-Challenge acc_norm: 23.29%
• HellaSwag acc_norm: 31.28%
• ArithMark-2.0 acc: 28.96%

Additional local evaluations:

• Winogrande acc: 50.12%
• OpenBookQA acc_norm: 28.80%
• BoolQ acc: 60.67%
• SciQ accuracy: 72.70%
• SciQ normalized accuracy: 64.20%

Usage

This model uses custom Transformers remote code because it includes QKV-Norm and a custom architecture definition.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "veyra-ai/Veyra2-Apricot-50M-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="cuda",
)

prompt = "In the 19th century"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
        repetition_penalty=1.1,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Notes on Generation

Veyra2-Apricot-50M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.

Intended Use

This model is intended for:

• small language model research
• continued pretraining
• benchmarking
• science-oriented cloze and multiple-choice evaluation
• experimentation with compact causal LMs

Limitations

• Not instruction tuned
• Not RLHF tuned
• Not safe for factual or high-stakes use without additional validation
• Can hallucinate names, citations, species, references, and technical claims
• Open-ended text may drift off-topic
• Context length during training was 1024 tokens

Citation

If you use this model, please cite the model repository:

veyra-ai/Veyra2-Apricot-50M-Base