Introduction

PAKNeura-V3 is a state-of-the-art Mixture of Experts (MoE) language model with 671B total parameters (37B active), designed to deliver high computational efficiency with superior reasoning capabilities. As Pakistan's first open-weight AI initiative, PAKNeura-V3 represents a significant milestone in democratizing access to cutting-edge language model technology for the Pakistani community and beyond.

The model is built upon three core technical foundations:

Mixture of Experts Architecture: Leveraging 256 routed experts with 8 active experts per token, enabling efficient computation while maintaining broad knowledge coverage.
Extended Context Understanding: Native support for up to 128K tokens of context, facilitating complex long-form reasoning and document understanding.
Multilingual Capability: Optimized for English, Urdu, and Pakistani regional languages, with a vocabulary size of 129,280 tokens.

Model Architecture

PAKNeura-V3 employs a Transformer-based Mixture of Experts (MoE) architecture with the following detailed configuration:

Hyperparameter	Value
Total Parameters	671B
Active Parameters	37B
Number of Layers	61
Hidden Dimension	7,168
Attention Heads	128
Key-Value Heads	128
Intermediate Size (FFN)	18,432
MoE Intermediate Size	2,048
Number of Routed Experts	256
Number of Shared Experts	1
Top-K Experts per Token	8
Expert Group Count	8
Top-K Groups	4
KV LoRA Rank	512
Q LoRA Rank	1,536
Vocabulary Size	129,280
Max Position Embeddings	163,840
Context Length	128K tokens
Norm Type	RMSNorm (eps: 1e-6)
Activation Function	SiLU
Weight Precision	bfloat16 / FP8
RoPE Theta	10,000

Attention Mechanism

PAKNeura-V3 utilizes Multi-head Latent Attention (MLA) with KV compression for efficient inference. The model supports both MHA and MQA modes, enabling flexible deployment across different hardware configurations.

Quantization

The model supports FP8 quantization (E4M3 format) with dynamic activation scaling, enabling reduced memory footprint and faster inference on compatible hardware.

Benchmarks

PAKNeura-V3 demonstrates competitive performance across a comprehensive suite of academic benchmarks covering reasoning, coding, knowledge, and language understanding tasks.

Category	Benchmark	Score
Knowledge	MMLU-Pro	85.0
Reasoning	GPQA Diamond	82.4
Reasoning	AIME 2025	93.1
Mathematics	HMMT Feb 2025	92.5
Mathematics	HMMT Nov 2025	90.2
Mathematics	IMO AnswerBench	78.3
Coding	LiveCodeBench	83.3
Coding	Codeforces	2386
Coding	SWE-bench Verified	73.1
Coding	SWE-bench Multilingual	70.2
Search	BrowseComp	67.6
Search	BrowseCompZh	65.0
Language	HLE	25.1

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "pakneura/PAKNeura-V3"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Pakistan's AI future is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Deployment Recommendations

Full Precision: 4x NVIDIA A100 (80GB) minimum
FP8 Quantized: 2x NVIDIA A100 (80GB)
Consumer GPU: 4-bit quantization on RTX 4090 (dual)
Sampling: temperature = 1.0, top_p = 0.95

Chat Template

PAKNeura-V3 supports the user, assistant, and developer roles. Example usage:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("pakneura/PAKNeura-V3")

messages = [
    {"role": "user", "content": "What is the capital of Pakistan?"},
    {"role": "assistant", "content": "The capital of Pakistan is Islamabad.", "reasoning_content": "Islamabad is the capital since 1963..."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

Weights Status

Component	Status
Configuration	✅ Complete
Tokenizer	✅ Complete
Model Weights	⏳ In Progress
Inference Code	⏳ In Progress

About PAKNeura

PAKNeura is a Pakistani AI organization dedicated to developing open-weight language models and advancing AI research in Pakistan. Our mission is to make cutting-edge AI technology accessible to the Pakistani community.

Website: www.pakneura.com

License

Citation

If you use PAKNeura-V3 in your research, please cite:

@misc{pakneura2025pakneurav3,
    title={PAKNeura-V3: Pakistan's First Open-Weight Language Model},
    author={PAKNeura AI},
    year={2025},
    publisher={Hugging Face},
    howpublished={https://huggingface.co/pakneura/PAKNeura-V3}
}

Contact

Website: www.pakneura.com
Hugging Face: pakneura
Discussions: PAKNeura-V3 Discussions

Made with ❤️ in Pakistan

Downloads last month: 8