lathatechie99
/

gqa-moe-topk

Model card Files Files and versions

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

GQA-MoE Transformer

This model uses Grouped Query Attention (GQA) with Mixture of Experts.

Architecture

Query heads: 4
KV heads: 2
Experts: 4
Top-K: 2

Key Features

Custom GQA implementation (not nn.MultiheadAttention)
Top-K routing for experts
Encoder-Decoder architecture for summarization

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support