NeMo
PyTorch
nemotron
srvm commited on
Commit
019a9d0
1 Parent(s): 66505a7

Add evaluation preview

Browse files
Files changed (1) hide show
  1. README.md +23 -0
README.md CHANGED
@@ -53,6 +53,29 @@ print(output_text)
53
 
54
  Minitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
55
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  ## Citation
57
 
58
  If you find our work helpful, please consider citing our paper:
 
53
 
54
  Minitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
55
 
56
+ ## Evaluation Results
57
+
58
+ *5-shot performance.* Language Understanding evaluated using [Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300):
59
+
60
+ | Average |
61
+ | :---- |
62
+ | 63.8 |
63
+
64
+ *Zero-shot performance.* Evaluated using select datasets from the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) with additions:
65
+
66
+ HellaSwag | Winogrande | GSM8K| ARC-C | XLSum |
67
+ | :------------- | :------------- | :------------- | :------------- | :------------- |
68
+ | 80.7 | 79.0 | 51.3 | 52.6 | 31.2
69
+
70
+
71
+ *Code generation performance*. Evaluated using [HumanEval](https://github.com/openai/human-eval):
72
+
73
+ | p@1, 0-Shot |
74
+ | :------------- |
75
+ | 31.6 |
76
+
77
+ Please refer to our [paper](https://arxiv.org/abs/2407.14679) for the full set of results.
78
+
79
  ## Citation
80
 
81
  If you find our work helpful, please consider citing our paper: