pszemraj commited on
Commit
9a6f73e
1 Parent(s): 16fb8da

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +20 -0
README.md CHANGED
@@ -31,3 +31,23 @@ res = pipe(
31
  print(res[0]["generated_text"])
32
  ```
33
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
  print(res[0]["generated_text"])
32
  ```
33
 
34
+ ## Quick eval
35
+
36
+ Quick eval for: `pszemraj/t5e-mini-nl24-flan`
37
+
38
+
39
+ hf (pretrained=pszemraj/t5e-mini-nl24-flan,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
40
+ | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
41
+ |-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
42
+ |boolq | 2|none | 0|acc |↑ |0.4541|± |0.0087|
43
+ |openbookqa | 1|none | 0|acc |↑ |0.1300|± |0.0151|
44
+ | | |none | 0|acc_norm |↑ |0.2700|± |0.0199|
45
+ |piqa | 1|none | 0|acc |↑ |0.6159|± |0.0113|
46
+ | | |none | 0|acc_norm |↑ |0.6077|± |0.0114|
47
+ |social_iqa | 0|none | 0|acc |↑ |0.3705|± |0.0109|
48
+ |tinyArc | 0|none | 25|acc_norm |↑ |0.2913|± | N/A|
49
+ |tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.0269|± | N/A|
50
+ | | |strict-match | 5|exact_match|↑ |0.0055|± | N/A|
51
+ |tinyHellaswag| 0|none | 10|acc_norm |↑ |0.3538|± | N/A|
52
+ |tinyMMLU | 0|none | 0|acc_norm |↑ |0.2551|± | N/A|
53
+ |winogrande | 1|none | 0|acc |↑ |0.5217|± |0.0140|