chore: add more information
Browse files
README.md
CHANGED
|
@@ -9,7 +9,9 @@ language:
|
|
| 9 |
|
| 10 |
### Model Architecture
|
| 11 |
|
| 12 |
-
The architecture of StripedHyena-Hessian-7B is
|
| 13 |
-
|
| 14 |
-
StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged in [Hyena](https://arxiv.org/abs/2302.10866) blocks.
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
### Model Architecture
|
| 11 |
|
| 12 |
+
The architecture of StripedHyena-Hessian-7B is different from traditional decoder-only Transformers.
|
|
|
|
|
|
|
| 13 |
|
| 14 |
+
StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged in [Hyena](https://arxiv.org/abs/2302.10866) blocks.
|
| 15 |
+
- Costant memory decoding by representation of convolutions as state-space models (modal or canonical form), or as truncated filters.
|
| 16 |
+
- Lower latency to preprocess long prompts.
|
| 17 |
+
- Improvements to training and inference compute-optimal scaling laws, compared to Transformers.
|