Text Generation
Transformers
Safetensors
English
falcon_mamba
Eval Results
Inference Endpoints
IChahed commited on
Commit
4141e0d
1 Parent(s): 8c8f700

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -189,7 +189,7 @@ The model training took roughly two months.
189
 
190
  ## Benchmarks
191
 
192
- We evaluate our model on all benchmarks of the leaderboard's version 2 using the `lm-evaluation-harness` package, and we evaluate it on the benchmarks of version 1 using `lighteval`.
193
 
194
 
195
  | `model name` |`IFEval`| `BBH` |`MATH LvL5`| `GPQA`| `MUSR`|`MMLU-PRO`|`Average`|
@@ -217,7 +217,7 @@ We evaluate our model on all benchmarks of the leaderboard's version 2 using the
217
  | `TRI-ML/mamba-7b-rw` | 46.48 | 80.24 | 57.72 | 76.40 | - | 4.70 | - |
218
  |***Hybrid SSM-attention models***| | | | | | | |
219
  | `recurrentgemma-9b` |52.00 | 80.40 | 60.50 | 73.60 | 38.60 | 42.60 | 57.95 |
220
- | `Zyphra/Zamba-7B-v1` | 46.48 | 80.24 | 57.72 | 76.40 | - | 30.78 | - |
221
  |***Transformer models*** | | | | | | | |
222
  | `Falcon2-11B` | 59.73 | 82.91 | 58.37 | 78.30 | 52.56 | 53.83 | **64.28** |
223
  | `Meta-Llama-3-8B` | 60.24 | 82.23 | 66.70 | 78.45 | 42.93 | 45.19 | 62.62 |
 
189
 
190
  ## Benchmarks
191
 
192
+ We evaluate our model on all benchmarks of the leaderboard's version 2 using the `lm-evaluation-harness` package, and we evaluate it on the benchmarks of version 1 using `lighteval`. The reported evaluation results on the leaderboard version 2 are normalized following HuggingFace score normalization.
193
 
194
 
195
  | `model name` |`IFEval`| `BBH` |`MATH LvL5`| `GPQA`| `MUSR`|`MMLU-PRO`|`Average`|
 
217
  | `TRI-ML/mamba-7b-rw` | 46.48 | 80.24 | 57.72 | 76.40 | - | 4.70 | - |
218
  |***Hybrid SSM-attention models***| | | | | | | |
219
  | `recurrentgemma-9b` |52.00 | 80.40 | 60.50 | 73.60 | 38.60 | 42.60 | 57.95 |
220
+ | `Zyphra/Zamba-7B-v1` | 56.14 | 82.23 | 58.11 | 79.87 | 36.23 | 30.78 | 57.23 |
221
  |***Transformer models*** | | | | | | | |
222
  | `Falcon2-11B` | 59.73 | 82.91 | 58.37 | 78.30 | 52.56 | 53.83 | **64.28** |
223
  | `Meta-Llama-3-8B` | 60.24 | 82.23 | 66.70 | 78.45 | 42.93 | 45.19 | 62.62 |