Text Generation
Transformers
Safetensors
llama
conversational
Inference Endpoints
text-generation-inference
jondurbin commited on
Commit
3e3e512
1 Parent(s): 51e4993

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +196 -0
README.md CHANGED
@@ -1,3 +1,199 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ datasets:
4
+ - ai2_arc
5
+ - jondurbin/airoboros-3.2
6
+ - codeparrot/apps
7
+ - facebook/belebele
8
+ - boolq
9
+ - jondurbin/cinematika-v0.1
10
+ - drop
11
+ - lmsys/lmsys-chat-1m
12
+ - TIGER-Lab/MathInstruct
13
+ - cais/mmlu
14
+ - Muennighoff/natural-instructions
15
+ - openbookqa
16
+ - piqa
17
+ - Vezora/Tested-22k-Python-Alpaca
18
+ - cakiki/rosetta-code
19
+ - Open-Orca/SlimOrca
20
+ - spider
21
+ - squad_v2
22
+ - migtissera/Synthia-v1.3
23
+ - datasets/winogrande
24
+ - nvidia/HelpSteer
25
+ - Intel/orca_dpo_pairs
26
+ - unalignment/toxic-dpo-v0.1
27
+ - jondurbin/truthy-dpo-v0.1
28
+ - allenai/ultrafeedback_binarized_cleaned
29
+ - Squish42/bluemoon-fandom-1-1-rp-cleaned
30
+ - LDJnr/Capybara
31
+ - JULIELab/EmoBank
32
+ - kingbri/PIPPA-shareGPT
33
  ---
34
+
35
+ # A bagel, with everything
36
+
37
+ ![bagel](bagel.png)
38
+
39
+ ## Overview
40
+
41
+ __*This model is basically unusable; sadly tinyllama is not a useful base model*__
42
+
43
+ An experimental fine-tune of [tinyllama](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) using [bagel](https://github.com/jondurbin/bagel)
44
+
45
+ ### Data sources
46
+
47
+ *Yes, you will see benchmark names in the list, but this only uses the train splits, and a decontamination by cosine similarity is performed at the end as a sanity check*
48
+
49
+ - [ai2_arc](https://huggingface.co/datasets/ai2_arc)
50
+ - Abstraction and reasoning dataset, useful in measuring "intelligence" to a certain extent.
51
+ - [airoboros](https://huggingface.co/datasets/unalignment/spicy-3.1)
52
+ - Variety of categories of synthetic instructions generated by gpt-4.
53
+ - [apps](https://huggingface.co/datasets/codeparrot/apps)
54
+ - Python coding dataset with 10k problems.
55
+ - [belebele](https://huggingface.co/datasets/facebook/belebele)
56
+ - Multi-lingual reading comprehension dataset.
57
+ - [bluemoon](https://huggingface.co/datasets/Squish42/bluemoon-fandom-1-1-rp-cleaned)
58
+ - Roleplay data scraped from Bluemoon, then cleaned and formatted as ShareGPT.
59
+ - [boolq](https://huggingface.co/datasets/boolq)
60
+ - Corpus of yes/no questions (which can be surprisingly difficult for AI to answer apparently?)
61
+ - [capybara](https://huggingface.co/datasets/LDJnr/Capybara)
62
+ - Multi-turn dataset used to create the capybara models.
63
+ - [cinematika](https://huggingface.co/datasets/jondurbin/cinematika-v0.1) (instruction and plain text)
64
+ - RP-style data synthesized from movie scripts so the model isn't quite as boring as it otherwise would be.
65
+ - [drop](https://huggingface.co/datasets/drop)
66
+ - More reading comprehension.
67
+ - [emobank](https://github.com/JULIELab/EmoBank)
68
+ - Emotion annotations using the Valence-Arousal-Domninance scheme.
69
+ - [gutenberg](https://www.gutenberg.org/) (plain text)
70
+ - Books/plain text, again to make the model less boring, only a handful of examples supported by [chapterize](https://github.com/JonathanReeve/chapterize)
71
+ - [lmsys_chat_1m](https://huggingface.co/datasets/lmsys/lmsys-chat-1m) (only gpt-4 items, also used for DPO)
72
+ - Chats collected by the lmsys chat arena, containing a wide variety of chats with various models.
73
+ - [mathinstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct)
74
+ - Composite dataset with a variety of math-related tasks and problem/question formats.
75
+ - [mmlu](https://huggingface.co/datasets/cais/mmlu)
76
+ - Massive Multitask Language Understanding - a wide variety of questions about various subject matters.
77
+ - [natural_instructions](https://huggingface.co/datasets/Muennighoff/natural-instructions)
78
+ - Millions of instructions from 1600+ task categories (sampled down substantially, stratified by task type)
79
+ - [openbookqa](https://huggingface.co/datasets/openbookqa)
80
+ - Question answering dataset.
81
+ - [pippa](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT)
82
+ - Deduped version of [PIPPA](https://huggingface.co/datasets/PygmalionAI/PIPPA) in ShareGPT format.
83
+ - [piqa](https://huggingface.co/datasets/piqa)
84
+ - Phyiscal interaction question answering.
85
+ - [python_alpaca](https://huggingface.co/datasets/Vezora/Tested-22k-Python-Alpaca)
86
+ - Python instruction response pairs, validated as functional.
87
+ - [rosetta_code](https://huggingface.co/datasets/cakiki/rosetta-code)
88
+ - Code problems and solutions in a variety of programming languages taken from rosettacode.org.
89
+ - [slimorca](https://huggingface.co/datasets/Open-Orca/SlimOrca)
90
+ - Collection of ~500k gpt-4 verified chats from OpenOrca.
91
+ - [spider](https://huggingface.co/datasets/spider)
92
+ - SQL-targeted dataset.
93
+ - [squad_v2](https://huggingface.co/datasets/squad_v2)
94
+ - Contextual question answering (RAG).
95
+ - [synthia](https://huggingface.co/datasets/migtissera/Synthia-v1.3)
96
+ - GPT-4 generated data using advanced prompting from Migel Tissera.
97
+ - [winogrande](https://huggingface.co/datasets/winogrande)
98
+ - Fill in the blank style prompts.
99
+
100
+ Only the train splits were used (if a split was provided), and an additional pass of decontamination is performed using approximate nearest neighbor search (via faiss).
101
+
102
+ ## Prompt formatting
103
+
104
+ In sticking with the theme of the bagel, I didn't want to use a single prompt format, so I used 4 - vicuna, llama-2, alpaca, and chat-ml (sorta).
105
+ I also didn't want to randomly select a single prompt format for each item (hoping each instruction would generalize more when used in a variety of prompt formats), so each instruction is actually converted into every prompt format.
106
+
107
+ This means each epoch of our fine-tune is really basically 4 epochs. So, for the fine-tunes, I would recommend only doing 1 epoch (or 0.75 epochs). I am testing with a single epoch using a relatively low learning rate.
108
+
109
+ ### Alpaca (sort of)
110
+
111
+ ```
112
+ Below is an instruction that describes a task. Write a response that appropriately completes the request.
113
+
114
+ ### Instruction:
115
+ {system prompt, if provided}
116
+ {instruction}
117
+
118
+ ### Response:
119
+ ```
120
+
121
+ The main difference here is that because of the dataset formatting and variety of data sources, it would have been much to tedious to add an `### Input:` block, so the inputs are just in the instruction section.
122
+
123
+ ### Vicuna
124
+
125
+ ```
126
+ {system prompt, if provided, randomly defaulting to "A chat between a user and an unbiased, uncensored assistant."}
127
+ USER: {instruction}
128
+ ASSISTANT:
129
+ ```
130
+
131
+ ### ChatML (sort of)
132
+
133
+ I don't really understand the point of having special tokens for `<|im_start|>` and `<|im_end|>`, because in practice they just act as BOS and EOS tokens (but, please correct me if I'm wrong).
134
+
135
+ So, instead of:
136
+ ```text
137
+ {bos}<|im_start|>{role}
138
+ {text}
139
+ <|im_end|>{eos}
140
+ ```
141
+
142
+ I just changed it to:
143
+ ```text
144
+ {bos}{role}
145
+ {text}
146
+ {eos}
147
+ ```
148
+
149
+ If you *really* want to use `<|im_start|>` and `<|im_end|>`, just update your `tokenizer_config.json` to use `<|im_start|>` instead of `<s>` and `<|im_end|>` instead of `</s>` and when tokenizing. And if you still don't like what I've done to this chat-ml-ish format, feel free to cry into your pillow or fork the code and do a new fine-tune.
150
+
151
+ ### Llama-2 chat
152
+
153
+ ```
154
+ [INST] <<SYS>>
155
+ {system}
156
+ <</SYS>>
157
+
158
+ {instruction} [/INST]
159
+ ```
160
+
161
+ ### Default via chat template
162
+
163
+ The model's `tokenizer_config.json` includes the default chat template (llama-2), so you can simply use the `apply_chat_template` method to build the full prompt.
164
+
165
+ ```
166
+ import transformers
167
+ tokenizer = transformers.AutoTokenizer.from_pretrained('jondurbin/bagel-8x7b-v0.2')
168
+ chat = [
169
+ {"role": "system", "content": "You are Bob, a friendly AI assistant."},
170
+ {"role": "user", "content": "Hello, how are you?"},
171
+ {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
172
+ {"role": "user", "content": "I'd like to show off how chat templating works!"},
173
+ ]
174
+ print(tokenizer.apply_chat_template(chat, tokenize=False))
175
+ ```
176
+
177
+ ### Contribute
178
+
179
+ If you're interested in new functionality/datasets, take a look at [bagel repo](https://github.com/jondurbin/bagel) and either make a PR or open an issue with details.
180
+
181
+ To help me with the fine-tuning costs (which are extremely expensive for these large combined datasets):
182
+
183
+ - https://bmc.link/jondurbin
184
+ - ETH 0xce914eAFC2fe52FdceE59565Dd92c06f776fcb11
185
+ - BTC bc1qdwuth4vlg8x37ggntlxu5cjfwgmdy5zaa7pswf
186
+
187
+ ### Licence and usage restrictions
188
+
189
+ The base model is tinyllama, which is licensed as apache-2.0 - no issues there.
190
+
191
+ The fine-tuning data, however, includes several datasets that have data generated at least in part by OpenAI's gpt-4.
192
+
193
+ I am not a lawyer, so I can't help determine if this is actually commercially viable, but some questions that often come up are:
194
+
195
+ - Does the OpenAI ToS apply only to the user who created the dataset initially, and not subsequent models?
196
+ - If the dataset was released under a permissive license, but actually includes OpenAI generated data, does that ToS supersede the license?
197
+ - Does the dataset fall completely under fair use anyways, since the model isn't really capable of reproducing the entire training set verbatim?
198
+
199
+ Use your best judgement and seek legal advice if you are concerned about the terms. In any case, by using this model, you agree to completely indemnify me.