ArmanNN
A hybrid language model combining Causal Attention, Selective SSM (parallel scan), Sparse Mixture-of-Experts, with learned fusion gates and path routers.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("mhd-rahman/ArmanNN-Base", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mhd-rahman/ArmanNN-Base")
inputs = tokenizer("The future of AI is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Architecture
- Parameters: 553,942,528
- Layers: 12
- d_model: 1024
- Heads: 16
- Experts: 4 (top-2)
- Max seq length: 1024
- Trained steps: 76,000
Training Data
Mixed pretraining corpus:
- FineWeb-Edu (62.5%)
- Code - StarCoder (12.5%)
- Wikipedia (8.3%)
- OpenWebMath (16.7%)
- Downloads last month
- -