nepetai/ivis-400m-gpu

Ivis-400M-Nepetai

Ivis Nepetai هو نموذج لغوي هجين يجمع ثلاث معماريات في كتلة واحدة: Transformer (GQA + RoPE) + Mamba SSM + Mixture of Experts.

A hybrid language model combining three architectures in a single stack: Grouped-Query Attention Transformer, Mamba State-Space layers, and sparse MoE.

Model details | تفاصيل النموذج

Property	Value
Total parameters	450,993,024 (~451M)
Active parameters	~250M (MoE: 4 experts × 2 active)
Layers	10
Hidden size	1,152
Attention	GQA — 18 query heads / 6 KV heads, head_dim 64, RoPE θ=10K
Mamba SSM	layers 3 & 7 · d_state=16 · d_conv=4 · expand=2
MoE	layers 6–9 · 4 experts × 2 active · hidden 2,304
Context length	256 tokens
Vocabulary	65,536 (SentencePiece BPE)
Languages	العربية + English

What is it for? | ما تخصصه

• توليد نصوص ثنائية اللغة (عربي/إنجليزي)
• التحقق العملي من استقرار المعمارية الهجينة (Transformer+Mamba+MoE) أثناء التدريب
• نموذج بحثي تجريبي (research preview)

Training status | حالة التدريب

Trained on Kaggle Tesla P100-16GB (PyTorch 2.2.0+cu118), Adafactor lr=3e-4. This is an experimental pilot checkpoint from a short validation run (40 steps) — not a fully-trained production model. Loss decreased from ~2540 → ~57 early in the run.

تم التدريب على GPU مجاني (Kaggle P100) لعدة خطوات تحقق فقط — هذا ليس نموذجاً مدرّباً بالكامل.

Files | الملفات

File	Description
ivis-400m-pytorch.pt	Full PyTorch checkpoint (state_dict + model_info)
ivis-400m.gguf	GGUF container (f16, 910MB, all 135 tensors)
config.json	Architecture hyperparameters
tokenizer.model / tokenizer.vocab	SentencePiece BPE tokenizer (AR+EN)
infer_numpy.py	Local inference engine — pure numpy, no PyTorch needed
convert_to_gguf.py	Torch-free .pt → GGUF converter

Run locally | التشغيل المحلي

pip install gguf sentencepiece numpy   # no torch required!

python infer_numpy.py --gguf ivis-400m.gguf \
  --tokenizer tokenizer.model \
  --prompt "البرمجة هي" --tokens 40

# measured on CPU: d_model=1152 layers=10 heads=18/6 moe=4@L6+
# generation speed ≈ 7 tokens/s (numpy, single thread)

## Related | نماذج مرتبطة

- [`nepetai/ivis-50m-pilot`](https://huggingface.co/nepetai/ivis-50m-pilot) — same architecture family at 55M scale, **fully validated end-to-end** with measured training curve (loss 9.20→0.92 over 300 steps), GGUF conversion and a pure-numpy inference engine that runs without PyTorch.

## Citation

```bibtex
@misc{ivis-nepetai-400m,
  title={Ivis-400M-Nepetai: Hybrid Transformer-Mamba-MoE Language Model},
  author={Nepetai Arabic},
  year={2026},
  url={https://huggingface.co/nepetai/ivis-400m-gpu}
}