Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: nvidia-open-model-license
|
| 4 |
+
license_link: >-
|
| 5 |
+
https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
|
| 6 |
+
inference: false
|
| 7 |
+
fine-tuning: false
|
| 8 |
+
tags:
|
| 9 |
+
- fp8
|
| 10 |
+
- vllm
|
| 11 |
+
base_model: mgoin/Nemotron-4-340B-Instruct-hf
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
## Nemotron-4-340B-Instruct-hf-FP8
|
| 15 |
+
|
| 16 |
+
Converted and quantized checkpoint of [nvidia/Nemotron-4-340B-Instruct](https://huggingface.co/nvidia/Nemotron-4-340B-Instruct). Specifically it was produced from the [v1.0 .nemo checkpoint on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/nemotron-4-340b-instruct/files).
|
| 17 |
+
|
| 18 |
+
This runs in vLLM with this PR: https://github.com/vllm-project/vllm/pull/6611
|
| 19 |
+
|
| 20 |
+
### Evaluations
|
| 21 |
+
|
| 22 |
+
All the below evaluations were run with `lm-eval==0.4.3` on 8xA100 GPUs.
|
| 23 |
+
|
| 24 |
+
```
|
| 25 |
+
lm_eval --model vllm --model_args pretrained=/home/mgoin/code/Nemotron-4-340B-Instruct-hf-FP8,tensor_parallel_size=8,distributed_executor_backend="ray",gpu_memory_utilization=0.6,enforce_eager=True --tasks mmlu --num_fewshot 0 --batch_size 4
|
| 26 |
+
vllm (pretrained=/home/mgoin/code/Nemotron-4-340B-Instruct-hf-FP8,tensor_parallel_size=8,distributed_executor_backend=ray,gpu_memory_utilization=0.6,enforce_eager=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 4
|
| 27 |
+
| Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr|
|
| 28 |
+
|---------------------------------------|-------|------|-----:|------|---|-----:|---|-----:|
|
| 29 |
+
|mmlu |N/A |none | 0|acc |↑ |0.7905|± |0.0033|
|
| 30 |
+
| - abstract_algebra | 0|none | 0|acc |↑ |0.5100|± |0.0502|
|
| 31 |
+
| - anatomy | 0|none | 0|acc |↑ |0.8074|± |0.0341|
|
| 32 |
+
| - astronomy | 0|none | 0|acc |↑ |0.8816|± |0.0263|
|
| 33 |
+
| - business_ethics | 0|none | 0|acc |↑ |0.7300|± |0.0446|
|
| 34 |
+
| - clinical_knowledge | 0|none | 0|acc |↑ |0.8453|± |0.0223|
|
| 35 |
+
| - college_biology | 0|none | 0|acc |↑ |0.9236|± |0.0222|
|
| 36 |
+
| - college_chemistry | 0|none | 0|acc |↑ |0.5300|± |0.0502|
|
| 37 |
+
| - college_computer_science | 0|none | 0|acc |↑ |0.7400|± |0.0441|
|
| 38 |
+
| - college_mathematics | 0|none | 0|acc |↑ |0.4500|± |0.0500|
|
| 39 |
+
| - college_medicine | 0|none | 0|acc |↑ |0.7630|± |0.0324|
|
| 40 |
+
| - college_physics | 0|none | 0|acc |↑ |0.5490|± |0.0495|
|
| 41 |
+
| - computer_security | 0|none | 0|acc |↑ |0.8300|± |0.0378|
|
| 42 |
+
| - conceptual_physics | 0|none | 0|acc |↑ |0.8043|± |0.0259|
|
| 43 |
+
| - econometrics | 0|none | 0|acc |↑ |0.7105|± |0.0427|
|
| 44 |
+
| - electrical_engineering | 0|none | 0|acc |↑ |0.7448|± |0.0363|
|
| 45 |
+
| - elementary_mathematics | 0|none | 0|acc |↑ |0.6667|± |0.0243|
|
| 46 |
+
| - formal_logic | 0|none | 0|acc |↑ |0.5952|± |0.0439|
|
| 47 |
+
| - global_facts | 0|none | 0|acc |↑ |0.4900|± |0.0502|
|
| 48 |
+
| - high_school_biology | 0|none | 0|acc |↑ |0.9097|± |0.0163|
|
| 49 |
+
| - high_school_chemistry | 0|none | 0|acc |↑ |0.7143|± |0.0318|
|
| 50 |
+
| - high_school_computer_science | 0|none | 0|acc |↑ |0.9100|± |0.0288|
|
| 51 |
+
| - high_school_european_history | 0|none | 0|acc |↑ |0.8788|± |0.0255|
|
| 52 |
+
| - high_school_geography | 0|none | 0|acc |↑ |0.9242|± |0.0189|
|
| 53 |
+
| - high_school_government_and_politics| 0|none | 0|acc |↑ |0.9845|± |0.0089|
|
| 54 |
+
| - high_school_macroeconomics | 0|none | 0|acc |↑ |0.8333|± |0.0189|
|
| 55 |
+
| - high_school_mathematics | 0|none | 0|acc |↑ |0.4630|± |0.0304|
|
| 56 |
+
| - high_school_microeconomics | 0|none | 0|acc |↑ |0.8824|± |0.0209|
|
| 57 |
+
| - high_school_physics | 0|none | 0|acc |↑ |0.6159|± |0.0397|
|
| 58 |
+
| - high_school_psychology | 0|none | 0|acc |↑ |0.9394|± |0.0102|
|
| 59 |
+
| - high_school_statistics | 0|none | 0|acc |↑ |0.7639|± |0.0290|
|
| 60 |
+
| - high_school_us_history | 0|none | 0|acc |↑ |0.9412|± |0.0165|
|
| 61 |
+
| - high_school_world_history | 0|none | 0|acc |↑ |0.9409|± |0.0153|
|
| 62 |
+
| - human_aging | 0|none | 0|acc |↑ |0.8072|± |0.0265|
|
| 63 |
+
| - human_sexuality | 0|none | 0|acc |↑ |0.8855|± |0.0279|
|
| 64 |
+
| - humanities |N/A |none | 0|acc |↑ |0.7594|± |0.0060|
|
| 65 |
+
| - international_law | 0|none | 0|acc |↑ |0.9091|± |0.0262|
|
| 66 |
+
| - jurisprudence | 0|none | 0|acc |↑ |0.8704|± |0.0325|
|
| 67 |
+
| - logical_fallacies | 0|none | 0|acc |↑ |0.8528|± |0.0278|
|
| 68 |
+
| - machine_learning | 0|none | 0|acc |↑ |0.6786|± |0.0443|
|
| 69 |
+
| - management | 0|none | 0|acc |↑ |0.8641|± |0.0339|
|
| 70 |
+
| - marketing | 0|none | 0|acc |↑ |0.9359|± |0.0160|
|
| 71 |
+
| - medical_genetics | 0|none | 0|acc |↑ |0.8400|± |0.0368|
|
| 72 |
+
| - miscellaneous | 0|none | 0|acc |↑ |0.9221|± |0.0096|
|
| 73 |
+
| - moral_disputes | 0|none | 0|acc |↑ |0.8382|± |0.0198|
|
| 74 |
+
| - moral_scenarios | 0|none | 0|acc |↑ |0.6168|± |0.0163|
|
| 75 |
+
| - nutrition | 0|none | 0|acc |↑ |0.8791|± |0.0187|
|
| 76 |
+
| - other |N/A |none | 0|acc |↑ |0.8214|± |0.0065|
|
| 77 |
+
| - philosophy | 0|none | 0|acc |↑ |0.8521|± |0.0202|
|
| 78 |
+
| - prehistory | 0|none | 0|acc |↑ |0.8796|± |0.0181|
|
| 79 |
+
| - professional_accounting | 0|none | 0|acc |↑ |0.6383|± |0.0287|
|
| 80 |
+
| - professional_law | 0|none | 0|acc |↑ |0.6838|± |0.0119|
|
| 81 |
+
| - professional_medicine | 0|none | 0|acc |↑ |0.8824|± |0.0196|
|
| 82 |
+
| - professional_psychology | 0|none | 0|acc |↑ |0.8611|± |0.0140|
|
| 83 |
+
| - public_relations | 0|none | 0|acc |↑ |0.8000|± |0.0383|
|
| 84 |
+
| - security_studies | 0|none | 0|acc |↑ |0.8204|± |0.0246|
|
| 85 |
+
| - social_sciences |N/A |none | 0|acc |↑ |0.8811|± |0.0057|
|
| 86 |
+
| - sociology | 0|none | 0|acc |↑ |0.9055|± |0.0207|
|
| 87 |
+
| - stem |N/A |none | 0|acc |↑ |0.7180|± |0.0076|
|
| 88 |
+
| - us_foreign_policy | 0|none | 0|acc |↑ |0.9600|± |0.0197|
|
| 89 |
+
| - virology | 0|none | 0|acc |↑ |0.5482|± |0.0387|
|
| 90 |
+
| - world_religions | 0|none | 0|acc |↑ |0.9006|± |0.0229|
|
| 91 |
+
|
| 92 |
+
| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
|
| 93 |
+
|------------------|-------|------|-----:|------|---|-----:|---|-----:|
|
| 94 |
+
|mmlu |N/A |none | 0|acc |↑ |0.7905|± |0.0033|
|
| 95 |
+
| - humanities |N/A |none | 0|acc |↑ |0.7594|± |0.0060|
|
| 96 |
+
| - other |N/A |none | 0|acc |↑ |0.8214|± |0.0065|
|
| 97 |
+
| - social_sciences|N/A |none | 0|acc |↑ |0.8811|± |0.0057|
|
| 98 |
+
| - stem |N/A |none | 0|acc |↑ |0.7180|± |0.0076|
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
The [original paper](https://d1qx31qr3h6wln.cloudfront.net/publications/Nemotron_4_340B_8T_0.pdf) evals for reference:
|
| 102 |
+
|
| 103 |
+
|