my0919175/Sovythos-303M-Base

🦋 SOVYTHOS-V2

Sovythos Egyptian Reasoning & Code Base Model

A production-grade, Llama-compatible decoder-only Transformer trained entirely from scratch for Arabic, English, and Code.

---

📋 Table of Contents

• Overview
• Model Specifications
• Architecture Details
• Tokenizer
• Training
• Usage
• GGUF Inference
• Intended Uses
• Limitations
• Roadmap
• Citation
• License

---

🦋 Overview

SOVYTHOS-V2 is the second-generation foundation model of the Sovythos AI Project — a fully open-source, decoder-only Transformer trained from randomly initialized weights on a multilingual corpus spanning Modern Standard Arabic, Egyptian Arabic, English, and programming languages.

This release represents a significant architectural leap from the original 66M Base model, scaling to ~300 million parameters while adopting a Llama-compatible design (RMSNorm, RoPE, GQA, SwiGLU) to ensure seamless deployment across the modern inference ecosystem, including llama.cpp, llama-cpp-python, and Hugging Face transformers.

SOVYTHOS-V2 is released as a Base (Pretrained) model. It has not undergone instruction tuning or RLHF and is intended for researchers, developers, and educators who require a sovythos, transparent foundation for downstream training.

Key Design Principle: Every component — from the tokenizer to the tensor naming convention — is engineered for direct compatibility with the Llama/GGUF toolchain. No conversion hacks required.

---

📊 Model Specifications

Property	Value
Model	SOVYTHOS-V2
Parameters	~300 Million
Architecture	Decoder-only Transformer (Llama-style)
Hidden Size	1,024
Layers	24
Attention Heads	16 (Query) / 4 (Key-Value)
Head Dimension	64
Context Length	2,048 tokens
RoPE Theta	10,000
Vocabulary Size	32,000
Framework	PyTorch 2.x
Weight Format	Safetensors + GGUF (F16 / F32)
Training	From Scratch
Release	Base

---

🏗️ Architecture Details

SOVYTHOS-V2 implements a modern, efficient decoder stack aligned with the Llama architecture specification for frictionless ecosystem integration.

Normalization

• RMSNorm applied pre-attention and post-attention (no bias, no learned affine shift)

Position Encoding

• RoPE (Rotary Position Embedding) with rotate_half (NEOX-style) implementation
• Base frequency θ = 10,000
• Precomputed cosine/sine caches up to max sequence length

Attention Mechanism

• Grouped Query Attention (GQA) with 16 query heads and 4 key-value heads
• Reduces KV-cache memory footprint by 4× during inference compared to standard MHA
• Compatible with memory-efficient attention kernels (scaled_dot_product_attention)

Feed-Forward Network

• SwiGLU activation with gated projection
• Hidden dimension rounded to nearest multiple of 256 for optimal GPU kernel utilization
• gate_proj, up_proj, down_proj naming matches Llama/GGUF conventions exactly

Weight Initialization

• Gaussian initialization (σ = 0.02) for embeddings and linear layers
• Scaled down-projection and output-projection initialization by 1/√(2×layers) to stabilize deep training

Training Optimizations

• torch.compile graph mode (when available)
• TF32 matmul acceleration on Ampere+
• Automatic Mixed Precision (AMP) with fp16 forward pass
• Gradient Checkpointing for long-context training on consumer GPUs
• Gradient Accumulation with effective batch scaling
• Cosine LR Decay with linear warmup
• Gradient Clipping (max norm = 1.0)

---

🔤 Tokenizer

SOVYTHOS-V2 ships with a custom-trained Byte-Pair Encoding (BPE) tokenizer optimized for Arabic morphology, English, and code syntax.

Design Choices

Feature	Implementation
Pre-tokenization	GPT-4 / cl100k-style regex split (Unicode-aware)
Numerical Splitting	Digits grouped by 3 (e.g., 1,234,567 → 1 234 567)
Fallback	ByteLevel encoding for lossless coverage of any Unicode symbol
Normalization	NFC Unicode normalization (unifies Arabic hamza & diacritic forms)
Special Tokens	<|endoftext|>, <|pad|>, <|user|>, <|assistant|>, <|system|>, <|code|>, <|/code|>, <|think|>, <|/think|>
Vocab Size	32,000
Min Frequency	2

Coverage Efficiency (Tokens per Character)

Lower is better. Measured on a held-out sample of the training corpus:

Domain	Tokens / Char	Notes
Arabic (MSA + Egyptian)	~0.35	Efficient subword decomposition for Semitic morphology
English	~0.28	Competitive with modern English-centric tokenizers
Code	~0.42	Strong performance on Python / JavaScript identifiers

The tokenizer is saved as a standard tokenizer.json (Hugging Face tokenizers format) and its merges/vocab are directly compatible with GGUF gpt2 tokenizer model type.

---

🏋️ Training

SOVYTHOS-V2 was trained entirely from scratch. No pretrained weights from Llama, GPT, or any other foundation model were used as initialization.

Data Pipeline

• Corpus: Multilingual web text, technical documentation, mathematical content, and code repositories
• Languages: Modern Standard Arabic, Egyptian Arabic, English, and multilingual programming languages
• Cleaning: Duplicate-line removal, short-line filtering (< 2 chars), NFC normalization, and aggressive deduplication of repeated contiguous lines
• Format: Binary tokenized memmap for high-throughout training with np.uint16 encoding

Training Recipe

Hyperparameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.95, ε=1e-8)
Peak Learning Rate	3e-4
Minimum Learning Rate	3e-5
Warmup Steps	1,000
LR Schedule	Cosine decay
Weight Decay	0.1 (2D+ tensors only)
Max Gradient Norm	1.0
Precision	FP16 (AMP) with FP32 master weights
Batch Size	4 (per device)
Gradient Accumulation	8 steps
Effective Batch Size	32
Block Size	512 tokens

Checkpointing & Resumption

• Checkpoints saved in Safetensors format (secure, fast, language-agnostic)
• Automatic resumption from ckpt_dir/best/ including step count and best validation loss
• Best checkpoint selection based on validation loss

Export

• Native GGUF export after training (or from existing checkpoints via --export_gguf_only)
• Supported quantizations: F32, F16 (post-hoc Q4/Q8 via llama.cpp tools)
• Tensor name map strictly follows Llama GGUF specification (blk.{i}.attn_q.weight, blk.{i}.ffn_gate.weight, etc.)

---

💻 Usage

Loading with PyTorch (Native)

from model import Model, SovythosConfig

# Load from safetensors checkpoint
model = Model.from_pretrained("sovythos_ckpt/best", device="cuda")

# Generate
text = "The capital of Egypt is"
ids = tokenizer.encode(text).ids
input_ids = torch.tensor([ids], device="cuda")

output = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    eos_token_id=tokenizer.token_to_id("<|endoftext|>")
)
print(tokenizer.decode(output[0].tolist()))

Using the Tokenizer

from tokenizers import Tokenizer

tok = Tokenizer.from_file("sovythos_tokenizer.json")
encoded = tok.encode("إزيك يا كبير؟")
print(encoded.ids)  # [ ... ]
print(tok.decode(encoded.ids))  # "إزيك يا كبير؟"

---

🚀 GGUF Inference

Because SOVYTHOS-V2 uses Llama-compatible tensor naming and a GPT-2-format tokenizer, the exported GGUF file runs out-of-the-box with llama.cpp and llama-cpp-python.

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="sovythos-v2.f16.gguf",
    n_ctx=2048,
    n_gpu_layers=-1,  # Offload all layers to GPU
    verbose=False
)

output = llm(
    "The capital of Egypt is",
    max_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    stop=["<|endoftext|>"]
)
print(output["choices"][0]["text"])

llama.cpp CLI

./main -m sovythos-v2.f16.gguf        -n 50        --temp 0.8        --top-k 40        --top-p 0.9        -p "The capital of Egypt is"

No conversion scripts required. The GGUF file is produced directly by the training framework with correct llama arch metadata.

---

🎯 Intended Uses

SOVYTHOS-V2 is designed as a research and development foundation for:

• Continued pretraining on domain-specific corpora
• Supervised fine-tuning (SFT) for chat or task-specific behavior
• Instruction tuning and alignment experiments
• Educational exploration of modern decoder-only architectures
• Arabic NLP research (MSA, Egyptian dialect, code-switching)
• Low-resource and Sovythos AI development

Not intended for: Direct deployment as a production chatbot without additional safety fine-tuning and evaluation.

---

⚠️ Limitations

As a Base (pretrained) model, SOVYTHOS-V2 exhibits expected limitations:

• No instruction following without fine-tuning
• Hallucination of factual information (names, dates, technical details)
• Repetition in long-form generation
• Incomplete reasoning on complex multi-step problems
• Limited context coherence beyond a few hundred tokens in zero-shot settings
• Code generation produces syntactically valid structures but may contain logic errors

These limitations are standard for base models of this scale and will be addressed in future Instruct/Chat releases.

---

🛣️ Roadmap

Milestone	Status
✅ SOVYTHOS-V2 Base	Released
🟡 SOVYTHOS-V2-Instruct	In Development
⬜ SOVYTHOS Chat	Planned
⬜ Larger Parameter Models (1B+)	Planned
⬜ Long Context Extension (8K+)	Planned
⬜ Advanced Reasoning Tuning	Planned
⬜ Improved Egyptian Dialect Understanding	Planned

---

🌌 Sovythos AI Project

SOVYTHOS-V2 is the flagship foundation model of the Sovythos AI Project, an open ecosystem dedicated to building transparent, multilingual language models with native Arabic support and modern reasoning capabilities.

The project prioritizes:

• Sovereignty: Full training pipeline ownership (data → tokenizer → model → export)
• Transparency: Open weights, open code, open training logs
• Accessibility: GGUF compatibility for edge and consumer hardware
• Community: Research-first releases with clear limitation disclosures

---

📖 Citation

If you use SOVYTHOS-V2 in your research, please cite:

@misc{sovythosv2,
  title={SOVYTHOS-V2: Sovythos Egyptian Reasoning and Code Base Model},
  author={Mahmoud Yasser},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/...}}
}

---

👨‍💻 Creator

Mahmoud Yasser

Founder of the Sovythos AI Project

---

📄 License

Apache License 2.0

---

🦋 Welcome to the next chapter of the Sovythos ecosystem.

SOVYTHOS-V2 is a production-ready foundation for researchers, developers, and builders who believe in open, Sovythos AI.

Future releases will introduce instruction tuning, long-context support, and larger-scale reasoning models.

⭐ If you find this project useful, please consider following the repository and sharing your experiments.