QED-Base-v3
QED-Base-v3 is a ~111M parameter causal language model pretrained from scratch by Qarvexium. It is a base model — it has not been instruction-tuned or aligned for chat, and is designed to continue text rather than follow instructions or hold a conversation.
Model Details
- Developed by: Qarvexium
- Model type: Decoder-only causal language base model
- Language: English
- License: MIT
- Tokenizer: QED-B3 tokenizer (SentencePiece, 56,000 vocabulary)
Architecture
| Component | Value |
|---|---|
| Tokenizer | QED-B3 tokenizer |
| Vocabulary size | 56,000 |
| Model type | Decoder-only Transformer |
| Parameters | ~111M |
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 |
| KV heads | 4 |
| Attention | GQA |
| Intermediate FFN size | 1,792 |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Position encoding | RoPE |
| Context length | 2,048 |
| RoPE theta | 10,000 |
Weight-tied embeddings/LM head.
Uses
Direct Use
As a base model, QED-Base-v3 is intended for:
- Text completion / continuation
- Research on small-scale language model pretraining
- Experimenting with the QED architecture
- Studying tokenizer and language-model behavior
- A starting checkpoint for further fine-tuning
- Instruction tuning and downstream model development
Out-of-Scope Use
This model has not been instruction-tuned, RLHF'd, or safety-aligned.
It should not be deployed directly as a chat or assistant model, or in applications requiring reliable instruction-following or content moderation, without additional fine-tuning and evaluation.
Bias, Risks, and Limitations
QED-Base-v3 is an experimental base language model and may produce incorrect, nonsensical, repetitive, biased, or otherwise undesirable text.
Because it is a base model, it does not have built-in instruction-following or refusal behavior.
Its relatively small parameter count also means that its factual knowledge, reasoning ability, and generalization capabilities are limited compared with substantially larger language models.
Outputs should be evaluated and filtered before use in user-facing applications.
How to Get Started
The repository includes a lightweight inference implementation in infer.py.
from infer import load_model, load_tokenizer, run
model = load_model("QED-Base-v3.pt")
tokenizer = load_tokenizer("qed-b3-tok.model")
text = run(
"Once upon a time",
model,
tokenizer,
max_new_tokens=100
)
print(text)
For generation, the included inference implementation supports temperature, top-k, top-p, repetition penalty, and seeded generation.
Tokenizer
QED-Base-v3 introduces a new tokenizer trained specifically for this model generation, rather than recycling the tokenizer used by QED-Base-v1.
The tokenizer uses a 56,000-token vocabulary and is provided in the repository as:
qed-b3-tok.model
QED Family
QED-Base-v3 is part of the QED family of language models.
| Model | Description |
|---|---|
| QED-Base-v1 | First-generation QED base model |
| QED-Base-v2 | Second-generation QED base model |
| QED-Base-v3 | Third-generation QED base model with a new tokenizer |
| QED-B1/B2-Instruction | Instruction-tuned QED variants |
QED-Base-v3 is intended to serve as a foundation for future QED experiments and fine-tuned models.
License
MIT License