Constitutional AI with Open LLMs
•
2
Lewis Tunstall PRO
lewtun
AI & ML interests
LLMs, LLMs, LLMs
Articles
Organizations
Posts
2
Post
Introducing Zephyr 141B-A35B 🪁:
HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: mistral-community/Mixtral-8x22B-v0.1
Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:
🧑🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by @JW17 and @nlee-208 and @j6mes and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO.
🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from @LDJnr LDJnr/Capybara and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.
What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.
Kudos to @alvarobartt @JW17 and @nlee-208 for this very nice and fast-paced collab!
For more details on the paper and dataset, checkout our collection: HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: mistral-community/Mixtral-8x22B-v0.1
Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:
🧑🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by @JW17 and @nlee-208 and @j6mes and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO.
🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from @LDJnr LDJnr/Capybara and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.
What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.
Kudos to @alvarobartt @JW17 and @nlee-208 for this very nice and fast-paced collab!
For more details on the paper and dataset, checkout our collection: HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
Post
Can we align code generation models to be good at chat without compromising their base capabilities 🤔?
This was the question the H4 team asked itself when BigCode released StarCoder2 a bit over a week ago. We knew that code models like deepseek-ai/deepseek-coder-6.7b-instruct and m-a-p/OpenCodeInterpreter-DS-33B get impressive scores on code benchmarks like HumanEval, but they tend to score poorly on chat benchmarks like MT Bench and IFEval. We also knew that the Zephyr recipe we applied to Mistral 7B produced a strong chat model, so we wondered -- could be tweaked to produce a strong coding assistant?
It turns out the answer is yes and I'm happy to share StarChat2, a DPO fine-tune of StarCoder2 15B that scores highly on both HumanEval and MT Bench / IFEval 🌟!
The most interesting lesson for me was that you get better models by blending in more code/math data than chat during the SFT step - in terms of tokens, we found a ratio of 3:1 worked best.
Anyway, here's a demo of the model, along with all the code and datasets we used to train it:
* Demo: HuggingFaceH4/starchat2-playground
* Collection: HuggingFaceH4/starchat2-15b-65f068417b330fafad751fce
* Recipe: https://github.com/huggingface/alignment-handbook
Hope it's useful to others!
This was the question the H4 team asked itself when BigCode released StarCoder2 a bit over a week ago. We knew that code models like deepseek-ai/deepseek-coder-6.7b-instruct and m-a-p/OpenCodeInterpreter-DS-33B get impressive scores on code benchmarks like HumanEval, but they tend to score poorly on chat benchmarks like MT Bench and IFEval. We also knew that the Zephyr recipe we applied to Mistral 7B produced a strong chat model, so we wondered -- could be tweaked to produce a strong coding assistant?
It turns out the answer is yes and I'm happy to share StarChat2, a DPO fine-tune of StarCoder2 15B that scores highly on both HumanEval and MT Bench / IFEval 🌟!
The most interesting lesson for me was that you get better models by blending in more code/math data than chat during the SFT step - in terms of tokens, we found a ratio of 3:1 worked best.
Anyway, here's a demo of the model, along with all the code and datasets we used to train it:
* Demo: HuggingFaceH4/starchat2-playground
* Collection: HuggingFaceH4/starchat2-15b-65f068417b330fafad751fce
* Recipe: https://github.com/huggingface/alignment-handbook
Hope it's useful to others!
Collections
4
spaces
20
models
213
lewtun/kto-aligned-model-lora
Updated
lewtun/gemma-7b-dpo-full-openhermes-mix1-beta-0.05
Text Generation
•
Updated
•
8
lewtun/gemma-7b-dpo-full-openhermes-mix1-beta-0.4
Text Generation
•
Updated
•
8
lewtun/gemma-7b-dpo-full-openhermes-mix1-beta-0.01
Text Generation
•
Updated
•
8
lewtun/gemma-7b-dpo-full-openhermes-mix1-beta-0.2
Text Generation
•
Updated
•
8
lewtun/gemma-7b-dpo-full-openhermes-mix1-beta-0.1
Text Generation
•
Updated
•
8
lewtun/gemma-7b-dpo-full-mix1-beta-0.05-epoch-3
Text Generation
•
Updated
•
8
•
1
lewtun/gemma-7b-dpo-full-mix1-beta-0.05-epoch-2
Text Generation
•
Updated
•
8
lewtun/gemma-7b-sft-full-openhermes-v0
Text Generation
•
Updated
•
8
lewtun/gemma-7b-dpo-full-mix2-beta-0.1
Text Generation
•
Updated
•
9
datasets
45
lewtun/sharegpt_prompts_annotated
Viewer
•
Updated
•
1
lewtun/custom-splits-test
Viewer
•
Updated
lewtun/cherry_picked_completions
Viewer
•
Updated
lewtun/splits-test
Viewer
•
Updated
•
2
lewtun/helpful-anthropic-raw
Viewer
•
Updated
•
2
lewtun/instruct-foo
Viewer
•
Updated
•
2
lewtun/test-model-outputs
Viewer
•
Updated
•
2
lewtun/rlhf-interface-dataset
Viewer
•
Updated
•
2
lewtun/alps
Viewer
•
Updated
•
2
lewtun/hamburgers
Viewer
•
Updated
•
2