Update README.md
Browse files
README.md
CHANGED
@@ -31,3 +31,23 @@ res = pipe(
|
|
31 |
print(res[0]["generated_text"])
|
32 |
```
|
33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
31 |
print(res[0]["generated_text"])
|
32 |
```
|
33 |
|
34 |
+
## Quick eval
|
35 |
+
|
36 |
+
Quick eval for: `pszemraj/t5e-mini-nl24-flan`
|
37 |
+
|
38 |
+
|
39 |
+
hf (pretrained=pszemraj/t5e-mini-nl24-flan,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|
40 |
+
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|
41 |
+
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
|
42 |
+
|boolq | 2|none | 0|acc |↑ |0.4541|± |0.0087|
|
43 |
+
|openbookqa | 1|none | 0|acc |↑ |0.1300|± |0.0151|
|
44 |
+
| | |none | 0|acc_norm |↑ |0.2700|± |0.0199|
|
45 |
+
|piqa | 1|none | 0|acc |↑ |0.6159|± |0.0113|
|
46 |
+
| | |none | 0|acc_norm |↑ |0.6077|± |0.0114|
|
47 |
+
|social_iqa | 0|none | 0|acc |↑ |0.3705|± |0.0109|
|
48 |
+
|tinyArc | 0|none | 25|acc_norm |↑ |0.2913|± | N/A|
|
49 |
+
|tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.0269|± | N/A|
|
50 |
+
| | |strict-match | 5|exact_match|↑ |0.0055|± | N/A|
|
51 |
+
|tinyHellaswag| 0|none | 10|acc_norm |↑ |0.3538|± | N/A|
|
52 |
+
|tinyMMLU | 0|none | 0|acc_norm |↑ |0.2551|± | N/A|
|
53 |
+
|winogrande | 1|none | 0|acc |↑ |0.5217|± |0.0140|
|