README.md

metadata

license: apache-2.0
datasets:
  - ai2_arc
  - allenai/ultrafeedback_binarized_cleaned
  - argilla/distilabel-intel-orca-dpo-pairs
  - jondurbin/airoboros-3.2
  - codeparrot/apps
  - facebook/belebele
  - bluemoon-fandom-1-1-rp-cleaned
  - boolq
  - camel-ai/biology
  - camel-ai/chemistry
  - camel-ai/math
  - camel-ai/physics
  - jondurbin/contextual-dpo-v0.1
  - jondurbin/gutenberg-dpo-v0.1
  - jondurbin/py-dpo-v0.1
  - jondurbin/truthy-dpo-v0.1
  - LDJnr/Capybara
  - jondurbin/cinematika-v0.1
  - WizardLM/WizardLM_evol_instruct_70k
  - glaiveai/glaive-function-calling-v2
  - jondurbin/gutenberg-dpo-v0.1
  - grimulkan/LimaRP-augmented
  - lmsys/lmsys-chat-1m
  - ParisNeo/lollms_aware_dataset
  - TIGER-Lab/MathInstruct
  - Muennighoff/natural-instructions
  - openbookqa
  - kingbri/PIPPA-shareGPT
  - piqa
  - Vezora/Tested-22k-Python-Alpaca
  - ropes
  - cakiki/rosetta-code
  - Open-Orca/SlimOrca
  - b-mc2/sql-create-context
  - squad_v2
  - mattpscott/airoboros-summarization
  - migtissera/Synthia-v1.3
  - unalignment/toxic-dpo-v0.2
  - WhiteRabbitNeo/WRN-Chapter-1
  - WhiteRabbitNeo/WRN-Chapter-2
  - winogrande

A bagel, with everything (except DPO)

Overview

This is the pre-DPO version of the mistral-7b model fine-tuned with https://github.com/jondurbin/bagel

The DPO counterpart can be found here: https://huggingface.co/jondurbin/bagel-dpo-7b-v0.4

This model is likely better for roleplay usage.

Data sources

There are many data sources used in the bagel models. See https://github.com/jondurbin/bagel for more information.

Only train splits are used, and a decontamination by cosine similarity is performed at the end as a sanity check against common benchmarks. If you don't know the difference between train and test, please learn.

SFT data sources

• ai2_arc
  • Abstraction and reasoning dataset, useful in measuring "intelligence" to a certain extent.
• airoboros
  • Variety of categories of synthetic instructions generated by gpt-4.
• apps
  • Python coding dataset with 10k problems.
• belebele
  • Multi-lingual reading comprehension dataset.
• bluemoon
  • Roleplay data scraped from Bluemoon, then cleaned and formatted as ShareGPT.
• boolq
  • Corpus of yes/no questions (which can be surprisingly difficult for AI to answer apparently?)
• camel-ai biology
  • GPT-4 generated biology instructions.
• camel-ai chemistry
  • GPT-4 generated chemistryinstructions.
• camel-ai math
  • GPT-4 generated math instructions.
• camel-ai physics
  • GPT-4 generated physics instructions.
• capybara
  • Multi-turn dataset used to create the capybara models.
• cinematika (instruction and plain text)
  • RP-style data synthesized from movie scripts so the model isn't quite as boring as it otherwise would be.
• emobank
  • Emotion annotations using the Valence-Arousal-Domninance scheme.
• evol-instruct
  • WizardLM's evol instruct 70k dataset.
• glaive-function-calling-v2
  • GlaiveAI function calling dataset.
• gutenberg (plain text)
  • Books/plain text, again to make the model less boring, only a handful of examples supported by chapterize
• limarp-augmented
  • Augmented and further modified version of LimaRP
• lmsys_chat_1m (only gpt-4 items, also used for DPO)
  • Chats collected by the lmsys chat arena, containing a wide variety of chats with various models.
• lollms
  • LoLLMs question answering dataset by ParisNeo, with helpful question answer pairs for using LoLLMs.
• mathinstruct
  • Composite dataset with a variety of math-related tasks and problem/question formats.
• natural_instructions
  • Millions of instructions from 1600+ task categories (sampled down substantially, stratified by task type)
• openbookqa
  • Question answering dataset.
• pippa
  • Deduped version of PIPPA in ShareGPT format.
• piqa
  • Phyiscal interaction question answering.
• python_alpaca
  • Python instruction response pairs, validated as functional.
• ropes
  • Reasoning Over PAragraph Effects in Situations - enhances ability to apply knowledge from a passage of text to a new situation.
• rosetta_code
  • Code problems and solutions in a variety of programming languages taken from rosettacode.org.
• slimorca
  • Collection of ~500k gpt-4 verified chats from OpenOrca.
• sql-create-context
  • SQL-targeted dataset, combining WikiSQL and Spider.
• squad_v2
  • Contextual question answering (RAG).
• airoboros-summarization
  • Combination of various summarization datasets, formatted into the airoboros context-obedient format.
• synthia
  • GPT-4 generated data using advanced prompting from Migel Tissera.
• whiterabbitneo chapter 1 and chapter 2
  • Offensive cybersecurity dataset by WhiteRabbitNeo/Migel Tissera
• winogrande
  • Fill in the blank style prompts.

DPO data sources

• airoboros 3.2 vs airoboros m2.0
  • The creative/writing tasks from airoboros-2.2.1 were re-generated using gpt4-0314 and a custom prompt to get longer, more creative, less clichè responses for airoboros 3.1, so we can use the shorter/boring version as the "rejected" value and the rerolled response as "chosen"
• contextual-dpo
  • Contextual prompt/response dataset using the airoboros context-obedient question answering format.
• helpsteer
  • Really neat dataset provided by the folks at NVidia with human annotation across a variety of metrics. Only items with the highest "correctness" value were used for DPO here, with the highest scoring output as "chosen" and random lower scoring value as "rejected"
• distilabel_orca_dpo_pairs
  • Another interesting dataset, originally by Intel, enhanced by argilla with distilabel which provides various DPO pairs generated from prompts included in the SlimOrca dataset.
• gutenberg-dpo
  • DPO pairs meant to increase the models novel writing abilities, using public domain books from https://gutenberg.org/
• py-dpo
  • Python DPO dataset (based on the SFT python_alpaca dataset above)
• toxic-dpo
  • highly toxic and potentially illegal content! De-censorship, for academic and lawful purposes only, of course. Generated by llama-2-70b via prompt engineering.
• truthy
  • DPO pairs meant to increase truthfulness of the model, e.g. common misconceptions, differentiate between AI assistants and roleplayed human in terms of corporeal awareness/locality/etc.
• ultrafeedback
  • One of the bits of magic behind the Zephyr model. Only the items with a chosen score of 8 or higher were included.

Prompt formatting

In sticking with the theme of the bagel, I didn't want to use a single prompt format, so I used 4 - vicuna, llama-2, alpaca, and chat-ml. I also didn't want to randomly select a single prompt format for each item (hoping each instruction would generalize more when used in a variety of prompt formats), so each instruction is converted into every prompt format (with 0.75 probability).

This means each epoch of our fine-tune is the equivalent of 3 epochs.

Llama-2 chat (recommended)

[INST] <<SYS>>
{system}
<</SYS>>

{instruction} [/INST]

Alpaca (sort of)

The only caveat here for alpaca format is that most of the datasets didn't have a separate "input" value, so there is no ### Input: block - any additional input should just be in the instruction section.

Below is an instruction that describes a task.  Write a response that appropriately completes the request.

### Instruction:
{system prompt, if provided}
{instruction}

### Response:

The main difference here is that because of the dataset formatting and variety of data sources, it would have been much to tedious to add an ### Input: block, so the inputs are just in the instruction section.

Vicuna

{system prompt, if provided, randomly defaulting to "A chat between a user and an unbiased, uncensored assistant."}
USER: {instruction}
ASSISTANT: 

ChatML

{bos}<|im_start|>{role}
{text}
<|im_end|>{eos}