Update README.md

README.md

@@ -10,4 +10,28 @@ smoothing_strength=0.8
 
 I will verify the validity of the model and update the readme as soon as possible.
 
-edit: The performance in my ERP test was comparable to Mistral-Nemo-Instruct-2407-GPTQ-INT8, which I consider a successful quantization.
+edit: The performance in my ERP test was comparable to Mistral-Nemo-Instruct-2407-GPTQ-INT8, which I consider a successful quantization.
+
+
+lm_eval --model vllm   --model_args pretrained="/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token",add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0   --tasks gsm8k   --num_fewshot 5   --limit 250   --batch_size 1
+vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token,add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: 1
+|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
+|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
+|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.784|±  |0.0261|
+|     |       |strict-match    |     5|exact_match|↑  |0.768|±  |0.0268|
+
+
+
+lm_eval --model vllm \
+l_args >   --model_args pretrained="/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token",add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0 \
+ks hellaswag \
+ >   --tasks hellaswag \
+>    --limit 150 \
+>   --num_fewshot 10 \
+--batch_size 1
+
+vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token,add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0), gen_kwargs: (None), limit: 150.0, num_fewshot: 10, batch_size: 1
+|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
+|---------|------:|------|-----:|--------|---|-----:|---|-----:|
+|hellaswag|      1|none  |    10|acc     |↑  |0.5800|±  |0.0404|
+|         |       |none  |    10|acc_norm|↑  |0.7533|±  |0.0353|