Update README.md
Browse files
README.md
CHANGED
|
@@ -18,7 +18,7 @@ language:
|
|
| 18 |
|
| 19 |
|
| 20 |
|
| 21 |
-
Calibrated with
|
| 22 |
|
| 23 |
|
| 24 |
# Inference with vLLM
|
|
@@ -219,7 +219,7 @@ We rely on [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-h
|
|
| 219 |
| Benchmark | | | |
|
| 220 |
|----------------------------------|------------------------|-----------------------------|---------------------------------|
|
| 221 |
| | google/gemma-3-12b-it | pytorch/gemma-3-12b-it-INT4 | pytorch/gemma-3-12b-it-AWQ-INT4 |
|
| 222 |
-
|
|
| 223 |
|
| 224 |
|
| 225 |
<details>
|
|
|
|
| 18 |
|
| 19 |
|
| 20 |
|
| 21 |
+
Calibrated with 30 samples of `mmlu_philosophy`, got eval accuracy of 76.86, while gemma-3-12b-it-INT4 is 75.56, and bfloat16 baseline is 79.10
|
| 22 |
|
| 23 |
|
| 24 |
# Inference with vLLM
|
|
|
|
| 219 |
| Benchmark | | | |
|
| 220 |
|----------------------------------|------------------------|-----------------------------|---------------------------------|
|
| 221 |
| | google/gemma-3-12b-it | pytorch/gemma-3-12b-it-INT4 | pytorch/gemma-3-12b-it-AWQ-INT4 |
|
| 222 |
+
| philosophy | 79.10 | 75.56 | 76.85 |
|
| 223 |
|
| 224 |
|
| 225 |
<details>
|