|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- ai2_arc |
|
- allenai/ultrafeedback_binarized_cleaned |
|
- argilla/distilabel-intel-orca-dpo-pairs |
|
- jondurbin/airoboros-3.2 |
|
- codeparrot/apps |
|
- facebook/belebele |
|
- bluemoon-fandom-1-1-rp-cleaned |
|
- boolq |
|
- camel-ai/biology |
|
- camel-ai/chemistry |
|
- camel-ai/math |
|
- camel-ai/physics |
|
- jondurbin/contextual-dpo-v0.1 |
|
- jondurbin/gutenberg-dpo-v0.1 |
|
- jondurbin/py-dpo-v0.1 |
|
- jondurbin/truthy-dpo-v0.1 |
|
- LDJnr/Capybara |
|
- jondurbin/cinematika-v0.1 |
|
- WizardLM/WizardLM_evol_instruct_70k |
|
- glaiveai/glaive-function-calling-v2 |
|
- jondurbin/gutenberg-dpo-v0.1 |
|
- grimulkan/LimaRP-augmented |
|
- lmsys/lmsys-chat-1m |
|
- ParisNeo/lollms_aware_dataset |
|
- TIGER-Lab/MathInstruct |
|
- Muennighoff/natural-instructions |
|
- openbookqa |
|
- kingbri/PIPPA-shareGPT |
|
- piqa |
|
- Vezora/Tested-22k-Python-Alpaca |
|
- ropes |
|
- cakiki/rosetta-code |
|
- Open-Orca/SlimOrca |
|
- b-mc2/sql-create-context |
|
- squad_v2 |
|
- mattpscott/airoboros-summarization |
|
- migtissera/Synthia-v1.3 |
|
- unalignment/toxic-dpo-v0.2 |
|
- WhiteRabbitNeo/WRN-Chapter-1 |
|
- WhiteRabbitNeo/WRN-Chapter-2 |
|
- winogrande |
|
--- |
|
|
|
# A bagel, with everything (except DPO) |
|
|
|
 |
|
|
|
## Overview |
|
|
|
This is the pre-DPO version of the mistral-7b model fine-tuned with https://github.com/jondurbin/bagel |
|
|
|
The DPO counterpart can be found here: https://huggingface.co/jondurbin/bagel-dpo-7b-v0.4 |
|
|
|
This model is likely better for roleplay usage. |
|
|
|
### Data sources |
|
|
|
There are many data sources used in the bagel models. See https://github.com/jondurbin/bagel for more information. |
|
|
|
__*Only train splits are used, and a decontamination by cosine similarity is performed at the end as a sanity check against common benchmarks. If you don't know the difference between train and test, please learn.*__ |
|
|
|
<details> |
|
<summary>SFT data sources</summary> |
|
|
|
- [ai2_arc](https://huggingface.co/datasets/ai2_arc) |
|
- Abstraction and reasoning dataset, useful in measuring "intelligence" to a certain extent. |
|
- [airoboros](https://huggingface.co/datasets/unalignment/spicy-3.1) |
|
- Variety of categories of synthetic instructions generated by gpt-4. |
|
- [apps](https://huggingface.co/datasets/codeparrot/apps) |
|
- Python coding dataset with 10k problems. |
|
- [belebele](https://huggingface.co/datasets/facebook/belebele) |
|
- Multi-lingual reading comprehension dataset. |
|
- [bluemoon](https://huggingface.co/datasets/Squish42/bluemoon-fandom-1-1-rp-cleaned) |
|
- Roleplay data scraped from Bluemoon, then cleaned and formatted as ShareGPT. |
|
- [boolq](https://huggingface.co/datasets/boolq) |
|
- Corpus of yes/no questions (which can be surprisingly difficult for AI to answer apparently?) |
|
- [camel-ai biology](https://huggingface.co/datasets/camel-ai/biology) |
|
- GPT-4 generated biology instructions. |
|
- [camel-ai chemistry](https://huggingface.co/datasets/camel-ai/chemistry) |
|
- GPT-4 generated chemistryinstructions. |
|
- [camel-ai math](https://huggingface.co/datasets/camel-ai/math) |
|
- GPT-4 generated math instructions. |
|
- [camel-ai physics](https://huggingface.co/datasets/camel-ai/physics) |
|
- GPT-4 generated physics instructions. |
|
- [capybara](https://huggingface.co/datasets/LDJnr/Capybara) |
|
- Multi-turn dataset used to create the capybara models. |
|
- [cinematika](https://huggingface.co/datasets/jondurbin/cinematika-v0.1) (instruction and plain text) |
|
- RP-style data synthesized from movie scripts so the model isn't quite as boring as it otherwise would be. |
|
- [emobank](https://github.com/JULIELab/EmoBank) |
|
- Emotion annotations using the Valence-Arousal-Domninance scheme. |
|
- [evol-instruct](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_70k) |
|
- WizardLM's evol instruct 70k dataset. |
|
- [glaive-function-calling-v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) |
|
- GlaiveAI function calling dataset. |
|
- [gutenberg](https://www.gutenberg.org/) (plain text) |
|
- Books/plain text, again to make the model less boring, only a handful of examples supported by [chapterize](https://github.com/JonathanReeve/chapterize) |
|
- [limarp-augmented](https://huggingface.co/datasets/grimulkan/LimaRP-augmented) |
|
- Augmented and further modified version of [LimaRP](https://huggingface.co/datasets/lemonilia/LimaRP) |
|
- [lmsys_chat_1m](https://huggingface.co/datasets/lmsys/lmsys-chat-1m) (only gpt-4 items, also used for DPO) |
|
- Chats collected by the lmsys chat arena, containing a wide variety of chats with various models. |
|
- [lollms](https://huggingface.co/datasets/ParisNeo/lollms_aware_dataset) |
|
- LoLLMs question answering dataset by ParisNeo, with helpful question answer pairs for using LoLLMs. |
|
- [mathinstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct) |
|
- Composite dataset with a variety of math-related tasks and problem/question formats. |
|
- [natural_instructions](https://huggingface.co/datasets/Muennighoff/natural-instructions) |
|
- Millions of instructions from 1600+ task categories (sampled down substantially, stratified by task type) |
|
- [openbookqa](https://huggingface.co/datasets/openbookqa) |
|
- Question answering dataset. |
|
- [pippa](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT) |
|
- Deduped version of [PIPPA](https://huggingface.co/datasets/PygmalionAI/PIPPA) in ShareGPT format. |
|
- [piqa](https://huggingface.co/datasets/piqa) |
|
- Phyiscal interaction question answering. |
|
- [python_alpaca](https://huggingface.co/datasets/Vezora/Tested-22k-Python-Alpaca) |
|
- Python instruction response pairs, validated as functional. |
|
- [ropes](https://huggingface.co/datasets/ropes) |
|
- Reasoning Over PAragraph Effects in Situations - enhances ability to apply knowledge from a passage of text to a new situation. |
|
- [rosetta_code](https://huggingface.co/datasets/cakiki/rosetta-code) |
|
- Code problems and solutions in a variety of programming languages taken from rosettacode.org. |
|
- [slimorca](https://huggingface.co/datasets/Open-Orca/SlimOrca) |
|
- Collection of ~500k gpt-4 verified chats from OpenOrca. |
|
- [sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) |
|
- SQL-targeted dataset, combining WikiSQL and Spider. |
|
- [squad_v2](https://huggingface.co/datasets/squad_v2) |
|
- Contextual question answering (RAG). |
|
- [airoboros-summarization](https://huggingface.co/datasets/mattpscott/airoboros-summarization) |
|
- Combination of various summarization datasets, formatted into the airoboros context-obedient format. |
|
- [synthia](https://huggingface.co/datasets/migtissera/Synthia-v1.3) |
|
- GPT-4 generated data using advanced prompting from Migel Tissera. |
|
- whiterabbitneo [chapter 1](https://huggingface.co/datasets/WhiteRabbitNeo/WRN-Chapter-1) and [chapter 2](https://huggingface.co/datasets/WhiteRabbitNeo/WRN-Chapter-2) |
|
- Offensive cybersecurity dataset by WhiteRabbitNeo/Migel Tissera |
|
- [winogrande](https://huggingface.co/datasets/winogrande) |
|
- Fill in the blank style prompts. |
|
</details> |
|
|
|
<details> |
|
<summary>DPO data sources</summary> |
|
|
|
- [airoboros 3.2](https://huggingface.co/datasets/jondurbin/airoboros-3.2) vs [airoboros m2.0](https://huggingface.co/datasets/jondurbin/airoboros-gpt4-m2.0) |
|
- The creative/writing tasks from airoboros-2.2.1 were re-generated using gpt4-0314 and a custom prompt to get longer, more creative, less clichè responses for airoboros 3.1, so we can use the shorter/boring version as the "rejected" value and the rerolled response as "chosen" |
|
- [contextual-dpo](https://huggingface.co/datasets/jondurbin/contextual-dpo-v0.1) |
|
- Contextual prompt/response dataset using the airoboros context-obedient question answering format. |
|
- [helpsteer](https://huggingface.co/datasets/nvidia/HelpSteer) |
|
- Really neat dataset provided by the folks at NVidia with human annotation across a variety of metrics. Only items with the highest "correctness" value were used for DPO here, with the highest scoring output as "chosen" and random lower scoring value as "rejected" |
|
- [distilabel_orca_dpo_pairs](https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs) |
|
- Another interesting dataset, originally by Intel, enhanced by argilla with [distilabel](https://github.com/argilla-io/distilabel) which provides various DPO pairs generated from prompts included in the SlimOrca dataset. |
|
- [gutenberg-dpo](https://huggingface.co/datasets/jondurbin/gutenberg-dpo-v0.1) |
|
- DPO pairs meant to increase the models novel writing abilities, using public domain books from https://gutenberg.org/ |
|
- [py-dpo](https://huggingface.co/datasets/jondurbin/py-dpo-v0.1) |
|
- Python DPO dataset (based on the SFT python_alpaca dataset above) |
|
- [toxic-dpo](https://huggingface.co/datasets/unalignment/toxic-dpo-v0.2) |
|
- __*highly toxic and potentially illegal content!*__ De-censorship, for academic and lawful purposes only, of course. Generated by llama-2-70b via prompt engineering. |
|
- [truthy](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1) |
|
- DPO pairs meant to increase truthfulness of the model, e.g. common misconceptions, differentiate between AI assistants and roleplayed human in terms of corporeal awareness/locality/etc. |
|
- [ultrafeedback](https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned) |
|
- One of the bits of magic behind the Zephyr model. Only the items with a chosen score of 8 or higher were included. |
|
</details> |
|
|
|
## Prompt formatting |
|
|
|
In sticking with the theme of the bagel, I didn't want to use a single prompt format, so I used 4 - vicuna, llama-2, alpaca, and chat-ml. |
|
I also didn't want to randomly select a single prompt format for each item (hoping each instruction would generalize more when used in a variety of prompt formats), so each instruction is converted into every prompt format (with 0.75 probability). |
|
|
|
This means each epoch of our fine-tune is the equivalent of 3 epochs. |
|
|
|
<details> |
|
<summary>Llama-2 chat (recommended)</summary> |
|
|
|
``` |
|
[INST] <<SYS>> |
|
{system} |
|
<</SYS>> |
|
|
|
{instruction} [/INST] |
|
``` |
|
</details> |
|
|
|
<details> |
|
<summary>Alpaca (sort of)</summary> |
|
|
|
The only caveat here for alpaca format is that most of the datasets didn't have a separate `"input"` value, so there is no `### Input:` block - any additional input should just be in the instruction section. |
|
|
|
``` |
|
Below is an instruction that describes a task. Write a response that appropriately completes the request. |
|
|
|
### Instruction: |
|
{system prompt, if provided} |
|
{instruction} |
|
|
|
### Response: |
|
``` |
|
|
|
The main difference here is that because of the dataset formatting and variety of data sources, it would have been much to tedious to add an `### Input:` block, so the inputs are just in the instruction section. |
|
|
|
<details> |
|
<summary>Vicuna</summary> |
|
|
|
``` |
|
{system prompt, if provided, randomly defaulting to "A chat between a user and an unbiased, uncensored assistant."} |
|
USER: {instruction} |
|
ASSISTANT: |
|
``` |
|
</details> |
|
|
|
<details> |
|
<summary>ChatML</summary> |
|
|
|
```text |
|
{bos}<|im_start|>{role} |
|
{text} |
|
<|im_end|>{eos} |
|
``` |
|
</details> |
