File size: 11,028 Bytes
9af2118 8205857 e68d435 8205857 e68d435 8205857 e68d435 8205857 e68d435 8205857 e68d435 8205857 e68d435 8205857 e68d435 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 |
---
license: apache-2.0
datasets:
- ai2_arc
- allenai/ultrafeedback_binarized_cleaned
- argilla/distilabel-intel-orca-dpo-pairs
- jondurbin/airoboros-3.2
- codeparrot/apps
- facebook/belebele
- bluemoon-fandom-1-1-rp-cleaned
- boolq
- camel-ai/biology
- camel-ai/chemistry
- camel-ai/math
- camel-ai/physics
- jondurbin/contextual-dpo-v0.1
- jondurbin/gutenberg-dpo-v0.1
- jondurbin/py-dpo-v0.1
- jondurbin/truthy-dpo-v0.1
- LDJnr/Capybara
- jondurbin/cinematika-v0.1
- WizardLM/WizardLM_evol_instruct_70k
- glaiveai/glaive-function-calling-v2
- jondurbin/gutenberg-dpo-v0.1
- grimulkan/LimaRP-augmented
- lmsys/lmsys-chat-1m
- ParisNeo/lollms_aware_dataset
- TIGER-Lab/MathInstruct
- Muennighoff/natural-instructions
- openbookqa
- kingbri/PIPPA-shareGPT
- piqa
- Vezora/Tested-22k-Python-Alpaca
- ropes
- cakiki/rosetta-code
- Open-Orca/SlimOrca
- b-mc2/sql-create-context
- squad_v2
- mattpscott/airoboros-summarization
- migtissera/Synthia-v1.3
- unalignment/toxic-dpo-v0.2
- WhiteRabbitNeo/WRN-Chapter-1
- WhiteRabbitNeo/WRN-Chapter-2
- winogrande
---
# A bagel, with everything (except DPO)
## Overview
This is the pre-DPO version of the mistral-7b model fine-tuned with https://github.com/jondurbin/bagel
The DPO counterpart can be found here: https://huggingface.co/jondurbin/bagel-dpo-7b-v0.4
This model is likely better for roleplay usage.
### Data sources
There are many data sources used in the bagel models. See https://github.com/jondurbin/bagel for more information.
__*Only train splits are used, and a decontamination by cosine similarity is performed at the end as a sanity check against common benchmarks. If you don't know the difference between train and test, please learn.*__
<details>
<summary>SFT data sources</summary>
- [ai2_arc](https://huggingface.co/datasets/ai2_arc)
- Abstraction and reasoning dataset, useful in measuring "intelligence" to a certain extent.
- [airoboros](https://huggingface.co/datasets/unalignment/spicy-3.1)
- Variety of categories of synthetic instructions generated by gpt-4.
- [apps](https://huggingface.co/datasets/codeparrot/apps)
- Python coding dataset with 10k problems.
- [belebele](https://huggingface.co/datasets/facebook/belebele)
- Multi-lingual reading comprehension dataset.
- [bluemoon](https://huggingface.co/datasets/Squish42/bluemoon-fandom-1-1-rp-cleaned)
- Roleplay data scraped from Bluemoon, then cleaned and formatted as ShareGPT.
- [boolq](https://huggingface.co/datasets/boolq)
- Corpus of yes/no questions (which can be surprisingly difficult for AI to answer apparently?)
- [camel-ai biology](https://huggingface.co/datasets/camel-ai/biology)
- GPT-4 generated biology instructions.
- [camel-ai chemistry](https://huggingface.co/datasets/camel-ai/chemistry)
- GPT-4 generated chemistryinstructions.
- [camel-ai math](https://huggingface.co/datasets/camel-ai/math)
- GPT-4 generated math instructions.
- [camel-ai physics](https://huggingface.co/datasets/camel-ai/physics)
- GPT-4 generated physics instructions.
- [capybara](https://huggingface.co/datasets/LDJnr/Capybara)
- Multi-turn dataset used to create the capybara models.
- [cinematika](https://huggingface.co/datasets/jondurbin/cinematika-v0.1) (instruction and plain text)
- RP-style data synthesized from movie scripts so the model isn't quite as boring as it otherwise would be.
- [emobank](https://github.com/JULIELab/EmoBank)
- Emotion annotations using the Valence-Arousal-Domninance scheme.
- [evol-instruct](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_70k)
- WizardLM's evol instruct 70k dataset.
- [glaive-function-calling-v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2)
- GlaiveAI function calling dataset.
- [gutenberg](https://www.gutenberg.org/) (plain text)
- Books/plain text, again to make the model less boring, only a handful of examples supported by [chapterize](https://github.com/JonathanReeve/chapterize)
- [limarp-augmented](https://huggingface.co/datasets/grimulkan/LimaRP-augmented)
- Augmented and further modified version of [LimaRP](https://huggingface.co/datasets/lemonilia/LimaRP)
- [lmsys_chat_1m](https://huggingface.co/datasets/lmsys/lmsys-chat-1m) (only gpt-4 items, also used for DPO)
- Chats collected by the lmsys chat arena, containing a wide variety of chats with various models.
- [lollms](https://huggingface.co/datasets/ParisNeo/lollms_aware_dataset)
- LoLLMs question answering dataset by ParisNeo, with helpful question answer pairs for using LoLLMs.
- [mathinstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct)
- Composite dataset with a variety of math-related tasks and problem/question formats.
- [natural_instructions](https://huggingface.co/datasets/Muennighoff/natural-instructions)
- Millions of instructions from 1600+ task categories (sampled down substantially, stratified by task type)
- [openbookqa](https://huggingface.co/datasets/openbookqa)
- Question answering dataset.
- [pippa](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT)
- Deduped version of [PIPPA](https://huggingface.co/datasets/PygmalionAI/PIPPA) in ShareGPT format.
- [piqa](https://huggingface.co/datasets/piqa)
- Phyiscal interaction question answering.
- [python_alpaca](https://huggingface.co/datasets/Vezora/Tested-22k-Python-Alpaca)
- Python instruction response pairs, validated as functional.
- [ropes](https://huggingface.co/datasets/ropes)
- Reasoning Over PAragraph Effects in Situations - enhances ability to apply knowledge from a passage of text to a new situation.
- [rosetta_code](https://huggingface.co/datasets/cakiki/rosetta-code)
- Code problems and solutions in a variety of programming languages taken from rosettacode.org.
- [slimorca](https://huggingface.co/datasets/Open-Orca/SlimOrca)
- Collection of ~500k gpt-4 verified chats from OpenOrca.
- [sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context)
- SQL-targeted dataset, combining WikiSQL and Spider.
- [squad_v2](https://huggingface.co/datasets/squad_v2)
- Contextual question answering (RAG).
- [airoboros-summarization](https://huggingface.co/datasets/mattpscott/airoboros-summarization)
- Combination of various summarization datasets, formatted into the airoboros context-obedient format.
- [synthia](https://huggingface.co/datasets/migtissera/Synthia-v1.3)
- GPT-4 generated data using advanced prompting from Migel Tissera.
- whiterabbitneo [chapter 1](https://huggingface.co/datasets/WhiteRabbitNeo/WRN-Chapter-1) and [chapter 2](https://huggingface.co/datasets/WhiteRabbitNeo/WRN-Chapter-2)
- Offensive cybersecurity dataset by WhiteRabbitNeo/Migel Tissera
- [winogrande](https://huggingface.co/datasets/winogrande)
- Fill in the blank style prompts.
</details>
<details>
<summary>DPO data sources</summary>
- [airoboros 3.2](https://huggingface.co/datasets/jondurbin/airoboros-3.2) vs [airoboros m2.0](https://huggingface.co/datasets/jondurbin/airoboros-gpt4-m2.0)
- The creative/writing tasks from airoboros-2.2.1 were re-generated using gpt4-0314 and a custom prompt to get longer, more creative, less clichè responses for airoboros 3.1, so we can use the shorter/boring version as the "rejected" value and the rerolled response as "chosen"
- [contextual-dpo](https://huggingface.co/datasets/jondurbin/contextual-dpo-v0.1)
- Contextual prompt/response dataset using the airoboros context-obedient question answering format.
- [helpsteer](https://huggingface.co/datasets/nvidia/HelpSteer)
- Really neat dataset provided by the folks at NVidia with human annotation across a variety of metrics. Only items with the highest "correctness" value were used for DPO here, with the highest scoring output as "chosen" and random lower scoring value as "rejected"
- [distilabel_orca_dpo_pairs](https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs)
- Another interesting dataset, originally by Intel, enhanced by argilla with [distilabel](https://github.com/argilla-io/distilabel) which provides various DPO pairs generated from prompts included in the SlimOrca dataset.
- [gutenberg-dpo](https://huggingface.co/datasets/jondurbin/gutenberg-dpo-v0.1)
- DPO pairs meant to increase the models novel writing abilities, using public domain books from https://gutenberg.org/
- [py-dpo](https://huggingface.co/datasets/jondurbin/py-dpo-v0.1)
- Python DPO dataset (based on the SFT python_alpaca dataset above)
- [toxic-dpo](https://huggingface.co/datasets/unalignment/toxic-dpo-v0.2)
- __*highly toxic and potentially illegal content!*__ De-censorship, for academic and lawful purposes only, of course. Generated by llama-2-70b via prompt engineering.
- [truthy](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1)
- DPO pairs meant to increase truthfulness of the model, e.g. common misconceptions, differentiate between AI assistants and roleplayed human in terms of corporeal awareness/locality/etc.
- [ultrafeedback](https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned)
- One of the bits of magic behind the Zephyr model. Only the items with a chosen score of 8 or higher were included.
</details>
## Prompt formatting
In sticking with the theme of the bagel, I didn't want to use a single prompt format, so I used 4 - vicuna, llama-2, alpaca, and chat-ml.
I also didn't want to randomly select a single prompt format for each item (hoping each instruction would generalize more when used in a variety of prompt formats), so each instruction is converted into every prompt format (with 0.75 probability).
This means each epoch of our fine-tune is the equivalent of 3 epochs.
<details>
<summary>Llama-2 chat (recommended)</summary>
```
[INST] <<SYS>>
{system}
<</SYS>>
{instruction} [/INST]
```
</details>
<details>
<summary>Alpaca (sort of)</summary>
The only caveat here for alpaca format is that most of the datasets didn't have a separate `"input"` value, so there is no `### Input:` block - any additional input should just be in the instruction section.
```
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{system prompt, if provided}
{instruction}
### Response:
```
The main difference here is that because of the dataset formatting and variety of data sources, it would have been much to tedious to add an `### Input:` block, so the inputs are just in the instruction section.
<details>
<summary>Vicuna</summary>
```
{system prompt, if provided, randomly defaulting to "A chat between a user and an unbiased, uncensored assistant."}
USER: {instruction}
ASSISTANT:
```
</details>
<details>
<summary>ChatML</summary>
```text
{bos}<|im_start|>{role}
{text}
<|im_end|>{eos}
```
</details> |