Hugging Face's logo

noneUsername
/
saiga_nemo_12b-W8A8-Dynamic-Per-Token-better

Safetensors
mistral
8-bit precision
Model card Files Community
noneUsername commited on
Commit
d373faa
·
verified ·
1 Parent(s): 173101b

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +78 -0
README.md ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model:
3
+ - IlyaGusev/saiga_nemo_12b
4
+ ---
5
+ vllm (pretrained=/root/autodl-tmp/saiga_nemo_12b,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
6
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
7
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
8
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.808|± |0.0250|
9
+ | | |strict-match | 5|exact_match|↑ |0.760|± |0.0271|
10
+
11
+ vllm (pretrained=/root/autodl-tmp/output88,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
12
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
13
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
14
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.784|± |0.0261|
15
+ | | |strict-match | 5|exact_match|↑ |0.748|± |0.0275|
16
+
17
+ vllm (pretrained=/root/autodl-tmp/output8765,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
18
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
19
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
20
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.792|± |0.0257|
21
+ | | |strict-match | 5|exact_match|↑ |0.760|± |0.0271|
22
+
23
+ vllm (pretrained=/root/autodl-tmp/output875,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
24
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
25
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
26
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.792|± |0.0257|
27
+ | | |strict-match | 5|exact_match|↑ |0.768|± |0.0268|
28
+
29
+ vllm (pretrained=/root/autodl-tmp/output873,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
30
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
31
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
32
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.780|± |0.0263|
33
+ | | |strict-match | 5|exact_match|↑ |0.736|± |0.0279|
34
+
35
+ vllm (pretrained=/root/autodl-tmp/output868,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
36
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
37
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
38
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.772|± |0.0266|
39
+ | | |strict-match | 5|exact_match|↑ |0.728|± |0.0282|
40
+
41
+
42
+
43
+
44
+ vllm (pretrained=/root/autodl-tmp/saiga_nemo_12b,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
45
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
46
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
47
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.796|± |0.0180|
48
+ | | |strict-match | 5|exact_match|↑ |0.734|± |0.0198|
49
+
50
+ vllm (pretrained=/root/autodl-tmp/output88,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
51
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
52
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
53
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.776|± |0.0187|
54
+ | | |strict-match | 5|exact_match|↑ |0.722|± |0.0201|
55
+
56
+ vllm (pretrained=/root/autodl-tmp/output8765,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
57
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
58
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
59
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.790|± |0.0182|
60
+ | | |strict-match | 5|exact_match|↑ |0.732|± |0.0198|
61
+
62
+ vllm (pretrained=/root/autodl-tmp/output875,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
63
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
64
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
65
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.778|± |0.0186|
66
+ | | |strict-match | 5|exact_match|↑ |0.732|± |0.0198|
67
+
68
+ vllm (pretrained=/root/autodl-tmp/output873,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
69
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
70
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
71
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.774|± |0.0187|
72
+ | | |strict-match | 5|exact_match|↑ |0.722|± |0.0201|
73
+
74
+ vllm (pretrained=/root/autodl-tmp/output868,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
75
+ |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr|
76
+ |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
77
+ |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.772|± |0.0188|
78
+ | | |strict-match | 5|exact_match|↑ |0.722|± |0.0201|