siddharth797/fenrir-MoE-v1

Fenrir‑MoE‑v1 (small)

A compact Mixture-of-Experts (MoE) language model based on the DeepSeek‑MoE architecture, fully compatible with the Hugging Face transformers library.

Model Details

• Architecture: Transformer + MoE (shared + routed experts)
• Hidden size: d_model = 512
• Number of layers: 6
• Attention heads: 8
• Total routed experts: 16
• Shared experts: 2
• Top‑K routing: 4
• Expert intermediate dimension: 512
• Vocabulary size: 50257 (GPT‑2 tokenizer)
• Context length: 256 tokens
• Total parameters: ≈ 112 million (active per token ≈ 86 million)

This model is intended for research, fine‑tuning, or lightweight inference on CPU/GPU.

How to Use

Make sure you have transformers installed:

pip install transformers torch


from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("siddharth797/fenrir-MoE-v1")
model = AutoModel.from_pretrained(
    "siddharth797/fenrir-MoE-v1",
    trust_remote_code=True
).to("cuda")



## Text Generation
### The repository includes a generation.py helper. Use it as follows:

from generation import generate_response, format_alpaca_prompt
formatted_prompt = format_alpaca_prompt(case["instruction"], case["input"])

response = generate_response(
    prompt_text=formatted_prompt,
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=96,
    temperature=0.7
)





@article{deepseekmoe2024,
  title={DeepSeek-MoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models},
  author={DeepSeek-AI},
  year={2024}
}