File size: 3,529 Bytes
10b2964 16fb8da 10b2964 9a6f73e 34dcd7e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
---
license: apache-2.0
base_model: google/t5-efficient-mini-nl24
datasets:
- Open-Orca/FLAN
language:
- en
library_name: transformers
pipeline_tag: text2text-generation
---
# t5e-mini-nl24-flan
25k steps on FLAN as an initial test/validation that [code](https://github.com/pszemraj/nanoT5/tree/flan-dataset) works. Not practically useful.
```py
from transformers import pipeline
pipe = pipeline(
"text2text-generation",
model="pszemraj/t5e-mini-nl24-flan",
)
res = pipe(
"true or false: water is wet.",
top_k=4,
penalty_alpha=0.6,
max_new_tokens=128,
)
print(res[0]["generated_text"])
```
## Quick eval
Quick eval for: `pszemraj/t5e-mini-nl24-flan`
hf (pretrained=pszemraj/t5e-mini-nl24-flan,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
|boolq | 2|none | 0|acc |↑ |0.4541|± |0.0087|
|openbookqa | 1|none | 0|acc |↑ |0.1300|± |0.0151|
| | |none | 0|acc_norm |↑ |0.2700|± |0.0199|
|piqa | 1|none | 0|acc |↑ |0.6159|± |0.0113|
| | |none | 0|acc_norm |↑ |0.6077|± |0.0114|
|social_iqa | 0|none | 0|acc |↑ |0.3705|± |0.0109|
|tinyArc | 0|none | 25|acc_norm |↑ |0.2913|± | N/A|
|tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.0269|± | N/A|
| | |strict-match | 5|exact_match|↑ |0.0055|± | N/A|
|tinyHellaswag| 0|none | 10|acc_norm |↑ |0.3538|± | N/A|
|tinyMMLU | 0|none | 0|acc_norm |↑ |0.2551|± | N/A|
|winogrande | 1|none | 0|acc |↑ |0.5217|± |0.0140|
<details>
<summary>base model evals: click to expand</summary>
Quick eval for: `google/t5-efficient-mini-nl24`
hf (pretrained=google/t5-efficient-mini-nl24,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|-------------|------:|----------------|-----:|-----------|---|-----:|---|------|
|boolq | 2|none | 0|acc |↑ |0.3783|± |0.0085|
|openbookqa | 1|none | 0|acc |↑ |0.1280|± |0.0150|
| | |none | 0|acc_norm |↑ |0.2660|± |0.0198|
|piqa | 1|none | 0|acc |↑ |0.5473|± |0.0116|
| | |none | 0|acc_norm |↑ |0.5267|± |0.0116|
|social_iqa | 0|none | 0|acc |↑ |0.3536|± |0.0108|
|tinyArc | 0|none | 25|acc_norm |↑ |0.3101|± | N/A|
|tinyGSM8k | 0|flexible-extract| 5|exact_match|↑ |0.0145|± | N/A|
| | |strict-match | 5|exact_match|↑ |0.0055|± | N/A|
|tinyHellaswag| 0|none | 10|acc_norm |↑ |0.2616|± | N/A|
|tinyMMLU | 0|none | 0|acc_norm |↑ |0.2839|± | N/A|
|winogrande | 1|none | 0|acc |↑ |0.4996|± |0.0141|
</details> |