|
--- |
|
language: |
|
- en |
|
license: apache-2.0 |
|
library_name: transformers |
|
tags: |
|
- transformers |
|
datasets: |
|
- mwitiderrick/OpenPlatypus |
|
base_model: vihangd/shearedplats-2.7b-v2 |
|
inference: true |
|
model_type: llama |
|
prompt_template: '### Instruction:\n |
|
|
|
{prompt} |
|
|
|
### Response: |
|
|
|
' |
|
created_by: mwitiderrick |
|
pipeline_tag: text-generation |
|
model-index: |
|
- name: mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
results: |
|
- task: |
|
type: text-generation |
|
dataset: |
|
name: hellaswag |
|
type: hellaswag |
|
metrics: |
|
- type: hellaswag (0-Shot) |
|
value: 0.5283 |
|
name: hellaswag(0-Shot) |
|
- task: |
|
type: text-generation |
|
dataset: |
|
name: winogrande |
|
type: winogrande |
|
metrics: |
|
- type: winogrande (0-Shot) |
|
value: 0.6464 |
|
name: winogrande(0-Shot) |
|
- task: |
|
type: text-generation |
|
dataset: |
|
name: arc_challenge |
|
type: arc_challenge |
|
metrics: |
|
- type: arc_challenge (0-Shot) |
|
value: 0.3652 |
|
name: arc_challenge(0-Shot) |
|
source: |
|
url: https://huggingface.co/mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
name: shearedplats-2.7b-v2-instruct-v0.1 model card |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: AI2 Reasoning Challenge (25-Shot) |
|
type: ai2_arc |
|
config: ARC-Challenge |
|
split: test |
|
args: |
|
num_few_shot: 25 |
|
metrics: |
|
- type: acc_norm |
|
value: 40.19 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: HellaSwag (10-Shot) |
|
type: hellaswag |
|
split: validation |
|
args: |
|
num_few_shot: 10 |
|
metrics: |
|
- type: acc_norm |
|
value: 70.08 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU (5-Shot) |
|
type: cais/mmlu |
|
config: all |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 28.12 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: TruthfulQA (0-shot) |
|
type: truthful_qa |
|
config: multiple_choice |
|
split: validation |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: mc2 |
|
value: 41.23 |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: Winogrande (5-shot) |
|
type: winogrande |
|
config: winogrande_xl |
|
split: validation |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 65.04 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GSM8k (5-shot) |
|
type: gsm8k |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 2.12 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1 |
|
name: Open LLM Leaderboard |
|
--- |
|
# ShearedPlats-7b Instruct |
|
|
|
This is an [ShearedPlats-7b model](https://huggingface.co/vihangd/shearedplats-2.7b-v2) that has been fine-tuned on 2 epochs of the |
|
[Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus) dataset. |
|
|
|
The modified version of the dataset can be found [here](mwitiderrick/Open-Platypus) |
|
## Prompt Template |
|
``` |
|
### Instruction: |
|
|
|
{query} |
|
|
|
### Response: |
|
<Leave new line for model to respond> |
|
``` |
|
## Usage |
|
```python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM,pipeline |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1") |
|
model = AutoModelForCausalLM.from_pretrained("mwitiderrick/shearedplats-2.7b-v2-instruct-v0.1") |
|
query = "Provide step-by-step instructions for making a sweet chicken bugger" |
|
text_gen = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=350) |
|
output = text_gen(f"### Instruction:\n{query}\n### Response:\n") |
|
print(output[0]['generated_text']) |
|
""" |
|
### Instruction: |
|
Provide step-by-step instructions for making a sweet chicken bugger |
|
### Response: |
|
Step 1: Prepare the ingredients |
|
|
|
You will need a mixture of ground chicken, breadcrumbs, butter, Worcestershire sauce, garlic powder, onion powder, salt, and pepper. |
|
|
|
Step 2: Form the bugger |
|
|
|
Take a piece of chicken breast meat and use a sharp knife to cut it into small cubes. Place the cubes in a bowl and add the remaining ingredients: breadcrumbs, butter, Worcestershire sauce, garlic powder, onion powder, salt, and pepper. Mix the ingredients together until they are well combined. |
|
|
|
Step 3: Shape the bugger |
|
|
|
Take a piece of the bugger mixture and form it into a ball. Place the ball on a plate or in a bag and refrigerate it for 30 minutes. |
|
|
|
Step 4: Cook the bugger |
|
|
|
Heat a grill pan or grill to medium-high heat. Take the bugger out of the refrigerator and place it on the grill. Cook the bugger for 5-7 minutes on each side, or until it is cooked through. |
|
|
|
Step 5: Serve and enjoy! |
|
|
|
Once the bugger is cooked, serve it hot and enjoy! |
|
|
|
Note: You can also use a sweet chicken bugger mix to make sweet chicken buggers. Simply follow the instructions above, but use the sweet chicken bugger mix instead of the ground chicken. |
|
|
|
Enjoy your sweet chicken buggers! |
|
""" |
|
``` |
|
## Evals |
|
``` |
|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |
|
|---------|-------|------|-----:|--------|-----:|---|-----:| |
|
|hellaswag|Yaml |none | 0|acc |0.5283|± |0.0050| |
|
| | |none | 0|acc_norm|0.7068|± |0.0045| |
|
|
|
|
|
| Groups |Version|Filter|n-shot| Metric | Value | |Stderr| |
|
|----------|-------|------|-----:|-----------|------:|---|-----:| |
|
|truthfulqa|N/A |none | 0|acc | 0.3411|± |0.0016| |
|
| | |none | 0|bleu_max |19.4174|± |0.6888| |
|
| | |none | 0|bleu_acc | 0.3378|± |0.0166| |
|
| | |none | 0|bleu_diff |-4.4165|± |0.6611| |
|
| | |none | 0|rouge1_max |43.6923|± |0.8239| |
|
| | |none | 0|rouge1_acc | 0.3305|± |0.0165| |
|
| | |none | 0|rouge1_diff|-6.4023|± |0.7680| |
|
| | |none | 0|rouge2_max |28.4074|± |0.8883| |
|
| | |none | 0|rouge2_acc | 0.2827|± |0.0158| |
|
| | |none | 0|rouge2_diff|-6.7716|± |0.8844| |
|
| | |none | 0|rougeL_max |40.2657|± |0.8218| |
|
| | |none | 0|rougeL_acc | 0.3023|± |0.0161| |
|
| | |none | 0|rougeL_diff|-6.5447|± |0.7706| |
|
|
|
|----------|-------|------|-----:|------|-----:|---|-----:| |
|
|winogrande|Yaml |none | 0|acc |0.6464|± |0.0134| |
|
|
|
|-------------|-------|------|-----:|--------|-----:|---|-----:| |
|
|arc_challenge|Yaml |none | 0|acc |0.3652|± |0.0141| |
|
| | |none | 0|acc_norm|0.3908|± |0.0143| |
|
``` |
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_mwitiderrick__shearedplats-2.7b-v2-instruct-v0.1) |
|
|
|
| Metric |Value| |
|
|---------------------------------|----:| |
|
|Avg. |41.13| |
|
|AI2 Reasoning Challenge (25-Shot)|40.19| |
|
|HellaSwag (10-Shot) |70.08| |
|
|MMLU (5-Shot) |28.12| |
|
|TruthfulQA (0-shot) |41.23| |
|
|Winogrande (5-shot) |65.04| |
|
|GSM8k (5-shot) | 2.12| |
|
|
|
|
