puneeshkhanna
commited on
Commit
•
d5488b9
1
Parent(s):
b61e7ea
Update README.md
Browse files
README.md
CHANGED
@@ -21,7 +21,7 @@ Falcon3-7B-Base supports 4 languages (english, french, spanish, portuguese) and
|
|
21 |
- Architecture
|
22 |
- Transformer based causal decoder only architecture
|
23 |
- 28 decoder blocks
|
24 |
-
- Grouped query attention (GQA) for faster inference: 12 query heads and 4
|
25 |
- Wider head dimension: 256
|
26 |
- High RoPE value to support long context understanding: 1000042
|
27 |
- 32k context length
|
|
|
21 |
- Architecture
|
22 |
- Transformer based causal decoder only architecture
|
23 |
- 28 decoder blocks
|
24 |
+
- Grouped query attention (GQA) for faster inference: 12 query heads and 4 key value heads
|
25 |
- Wider head dimension: 256
|
26 |
- High RoPE value to support long context understanding: 1000042
|
27 |
- 32k context length
|