File size: 630 Bytes
2f9e2c6 06bc59c 2f9e2c6 06bc59c af9abca 6be2da5 70481ea |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
---
license: apache-2.0
language:
- en
---
## StripedHyena-Hessian-7B (SH-7B)
### Model Architecture
StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged in [Hyena](https://arxiv.org/abs/2302.10866) blocks, different from traditional decoder-only Transformers.
- Costant memory decoding in Hyena blocks via representation of convolutions as state-space models (modal or canonical form), or as truncated filters.
- Lower latency to preprocess long prompts.
- Improvements to training and inference compute-optimal scaling laws, compared to Transformers.
|