--- license: apache-2.0 datasets: - ai2_arc - allenai/ultrafeedback_binarized_cleaned - argilla/distilabel-intel-orca-dpo-pairs - jondurbin/airoboros-3.2 - codeparrot/apps - facebook/belebele - bluemoon-fandom-1-1-rp-cleaned - boolq - camel-ai/biology - camel-ai/chemistry - camel-ai/math - camel-ai/physics - jondurbin/contextual-dpo-v0.1 - jondurbin/gutenberg-dpo-v0.1 - jondurbin/py-dpo-v0.1 - jondurbin/truthy-dpo-v0.1 - LDJnr/Capybara - jondurbin/cinematika-v0.1 - WizardLM/WizardLM_evol_instruct_70k - glaiveai/glaive-function-calling-v2 - jondurbin/gutenberg-dpo-v0.1 - grimulkan/LimaRP-augmented - lmsys/lmsys-chat-1m - ParisNeo/lollms_aware_dataset - TIGER-Lab/MathInstruct - Muennighoff/natural-instructions - openbookqa - kingbri/PIPPA-shareGPT - piqa - Vezora/Tested-22k-Python-Alpaca - ropes - cakiki/rosetta-code - Open-Orca/SlimOrca - b-mc2/sql-create-context - squad_v2 - mattpscott/airoboros-summarization - migtissera/Synthia-v1.3 - unalignment/toxic-dpo-v0.2 - WhiteRabbitNeo/WRN-Chapter-1 - WhiteRabbitNeo/WRN-Chapter-2 - winogrande --- # A bagel, with everything (except DPO) ![bagel](bagel.png) ## Overview This is the pre-DPO version of the mistral-7b model fine-tuned with https://github.com/jondurbin/bagel The DPO counterpart can be found here: https://huggingface.co/jondurbin/bagel-dpo-7b-v0.4 This model is likely better for roleplay usage. ### Data sources There are many data sources used in the bagel models. See https://github.com/jondurbin/bagel for more information. __*Only train splits are used, and a decontamination by cosine similarity is performed at the end as a sanity check against common benchmarks. If you don't know the difference between train and test, please learn.*__
SFT data sources - [ai2_arc](https://huggingface.co/datasets/ai2_arc) - Abstraction and reasoning dataset, useful in measuring "intelligence" to a certain extent. - [airoboros](https://huggingface.co/datasets/unalignment/spicy-3.1) - Variety of categories of synthetic instructions generated by gpt-4. - [apps](https://huggingface.co/datasets/codeparrot/apps) - Python coding dataset with 10k problems. - [belebele](https://huggingface.co/datasets/facebook/belebele) - Multi-lingual reading comprehension dataset. - [bluemoon](https://huggingface.co/datasets/Squish42/bluemoon-fandom-1-1-rp-cleaned) - Roleplay data scraped from Bluemoon, then cleaned and formatted as ShareGPT. - [boolq](https://huggingface.co/datasets/boolq) - Corpus of yes/no questions (which can be surprisingly difficult for AI to answer apparently?) - [camel-ai biology](https://huggingface.co/datasets/camel-ai/biology) - GPT-4 generated biology instructions. - [camel-ai chemistry](https://huggingface.co/datasets/camel-ai/chemistry) - GPT-4 generated chemistryinstructions. - [camel-ai math](https://huggingface.co/datasets/camel-ai/math) - GPT-4 generated math instructions. - [camel-ai physics](https://huggingface.co/datasets/camel-ai/physics) - GPT-4 generated physics instructions. - [capybara](https://huggingface.co/datasets/LDJnr/Capybara) - Multi-turn dataset used to create the capybara models. - [cinematika](https://huggingface.co/datasets/jondurbin/cinematika-v0.1) (instruction and plain text) - RP-style data synthesized from movie scripts so the model isn't quite as boring as it otherwise would be. - [emobank](https://github.com/JULIELab/EmoBank) - Emotion annotations using the Valence-Arousal-Domninance scheme. - [evol-instruct](https://huggingface.co/datasets/WizardLM/WizardLM_evol_instruct_70k) - WizardLM's evol instruct 70k dataset. - [glaive-function-calling-v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) - GlaiveAI function calling dataset. - [gutenberg](https://www.gutenberg.org/) (plain text) - Books/plain text, again to make the model less boring, only a handful of examples supported by [chapterize](https://github.com/JonathanReeve/chapterize) - [limarp-augmented](https://huggingface.co/datasets/grimulkan/LimaRP-augmented) - Augmented and further modified version of [LimaRP](https://huggingface.co/datasets/lemonilia/LimaRP) - [lmsys_chat_1m](https://huggingface.co/datasets/lmsys/lmsys-chat-1m) (only gpt-4 items, also used for DPO) - Chats collected by the lmsys chat arena, containing a wide variety of chats with various models. - [lollms](https://huggingface.co/datasets/ParisNeo/lollms_aware_dataset) - LoLLMs question answering dataset by ParisNeo, with helpful question answer pairs for using LoLLMs. - [mathinstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct) - Composite dataset with a variety of math-related tasks and problem/question formats. - [natural_instructions](https://huggingface.co/datasets/Muennighoff/natural-instructions) - Millions of instructions from 1600+ task categories (sampled down substantially, stratified by task type) - [openbookqa](https://huggingface.co/datasets/openbookqa) - Question answering dataset. - [pippa](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT) - Deduped version of [PIPPA](https://huggingface.co/datasets/PygmalionAI/PIPPA) in ShareGPT format. - [piqa](https://huggingface.co/datasets/piqa) - Phyiscal interaction question answering. - [python_alpaca](https://huggingface.co/datasets/Vezora/Tested-22k-Python-Alpaca) - Python instruction response pairs, validated as functional. - [ropes](https://huggingface.co/datasets/ropes) - Reasoning Over PAragraph Effects in Situations - enhances ability to apply knowledge from a passage of text to a new situation. - [rosetta_code](https://huggingface.co/datasets/cakiki/rosetta-code) - Code problems and solutions in a variety of programming languages taken from rosettacode.org. - [slimorca](https://huggingface.co/datasets/Open-Orca/SlimOrca) - Collection of ~500k gpt-4 verified chats from OpenOrca. - [sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) - SQL-targeted dataset, combining WikiSQL and Spider. - [squad_v2](https://huggingface.co/datasets/squad_v2) - Contextual question answering (RAG). - [airoboros-summarization](https://huggingface.co/datasets/mattpscott/airoboros-summarization) - Combination of various summarization datasets, formatted into the airoboros context-obedient format. - [synthia](https://huggingface.co/datasets/migtissera/Synthia-v1.3) - GPT-4 generated data using advanced prompting from Migel Tissera. - whiterabbitneo [chapter 1](https://huggingface.co/datasets/WhiteRabbitNeo/WRN-Chapter-1) and [chapter 2](https://huggingface.co/datasets/WhiteRabbitNeo/WRN-Chapter-2) - Offensive cybersecurity dataset by WhiteRabbitNeo/Migel Tissera - [winogrande](https://huggingface.co/datasets/winogrande) - Fill in the blank style prompts.
DPO data sources - [airoboros 3.2](https://huggingface.co/datasets/jondurbin/airoboros-3.2) vs [airoboros m2.0](https://huggingface.co/datasets/jondurbin/airoboros-gpt4-m2.0) - The creative/writing tasks from airoboros-2.2.1 were re-generated using gpt4-0314 and a custom prompt to get longer, more creative, less clichè responses for airoboros 3.1, so we can use the shorter/boring version as the "rejected" value and the rerolled response as "chosen" - [contextual-dpo](https://huggingface.co/datasets/jondurbin/contextual-dpo-v0.1) - Contextual prompt/response dataset using the airoboros context-obedient question answering format. - [helpsteer](https://huggingface.co/datasets/nvidia/HelpSteer) - Really neat dataset provided by the folks at NVidia with human annotation across a variety of metrics. Only items with the highest "correctness" value were used for DPO here, with the highest scoring output as "chosen" and random lower scoring value as "rejected" - [distilabel_orca_dpo_pairs](https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs) - Another interesting dataset, originally by Intel, enhanced by argilla with [distilabel](https://github.com/argilla-io/distilabel) which provides various DPO pairs generated from prompts included in the SlimOrca dataset. - [gutenberg-dpo](https://huggingface.co/datasets/jondurbin/gutenberg-dpo-v0.1) - DPO pairs meant to increase the models novel writing abilities, using public domain books from https://gutenberg.org/ - [py-dpo](https://huggingface.co/datasets/jondurbin/py-dpo-v0.1) - Python DPO dataset (based on the SFT python_alpaca dataset above) - [toxic-dpo](https://huggingface.co/datasets/unalignment/toxic-dpo-v0.2) - __*highly toxic and potentially illegal content!*__ De-censorship, for academic and lawful purposes only, of course. Generated by llama-2-70b via prompt engineering. - [truthy](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1) - DPO pairs meant to increase truthfulness of the model, e.g. common misconceptions, differentiate between AI assistants and roleplayed human in terms of corporeal awareness/locality/etc. - [ultrafeedback](https://huggingface.co/datasets/allenai/ultrafeedback_binarized_cleaned) - One of the bits of magic behind the Zephyr model. Only the items with a chosen score of 8 or higher were included.
## Prompt formatting In sticking with the theme of the bagel, I didn't want to use a single prompt format, so I used 4 - vicuna, llama-2, alpaca, and chat-ml. I also didn't want to randomly select a single prompt format for each item (hoping each instruction would generalize more when used in a variety of prompt formats), so each instruction is converted into every prompt format (with 0.75 probability). This means each epoch of our fine-tune is the equivalent of 3 epochs.
Llama-2 chat (recommended) ``` [INST] <> {system} <> {instruction} [/INST] ```
Alpaca (sort of) The only caveat here for alpaca format is that most of the datasets didn't have a separate `"input"` value, so there is no `### Input:` block - any additional input should just be in the instruction section. ``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: {system prompt, if provided} {instruction} ### Response: ``` The main difference here is that because of the dataset formatting and variety of data sources, it would have been much to tedious to add an `### Input:` block, so the inputs are just in the instruction section.
Vicuna ``` {system prompt, if provided, randomly defaulting to "A chat between a user and an unbiased, uncensored assistant."} USER: {instruction} ASSISTANT: ```
ChatML ```text {bos}<|im_start|>{role} {text} <|im_end|>{eos} ```