siddharth797/fenrir-MoE-v1-L

Fenrir‑MoE‑v1‑L (large)

A large Mixture-of-Experts (MoE) language model following the DeepSeek‑MoE design. It uses fine‑grained expert segmentation and shared experts for improved capacity while keeping per‑token FLOPs manageable.

Model Details

• Architecture: Transformer + MoE (shared + routed experts)
• Hidden size: d_model = 768
• Number of layers: 10
• Attention heads: 16
• Total routed experts: 96
• Shared experts: 1
• Top‑K routing: 2
• Expert intermediate dimension: 128 (fine‑grained)
• Shared expert intermediate dimension: 256
• Vocabulary size: 50257 (GPT‑2 tokenizer)
• Context length: 256 tokens
• Total parameters: ≈ 353 million (active per token ≈ 112 million)

This model offers a good trade‑off between capacity and inference cost.

How to Use

Make sure you have transformers installed:

pip install transformers torch


from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("siddharth797/fenrir-MoE-v1")
model = AutoModel.from_pretrained(
    "siddharth797/fenrir-MoE-v1-L",
    trust_remote_code=True
).to("cuda")


## Text Generation
### The repository includes a generation.py helper. Use it as follows:

from generation import generate_response, format_alpaca_prompt
formatted_prompt = format_alpaca_prompt(case["instruction"], case["input"])

response = generate_response(
    prompt_text=formatted_prompt,
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=96,
    temperature=0.7
)





@article{deepseekmoe2024,
  title={DeepSeek-MoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models},
  author={DeepSeek-AI},
  year={2024}
}