|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- tatsu-lab/alpaca |
|
--- |
|
|
|
## 🍮 🦙 Flan-Alpaca: Instruction Tuning from Humans and Machines |
|
|
|
Our [repository](https://github.com/declare-lab/flan-alpaca) contains code for extending the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) |
|
synthetic instruction tuning to existing instruction-tuned models such as [Flan-T5](https://arxiv.org/abs/2210.11416). |
|
The pretrained models and demos are available on HuggingFace 🤗 : |
|
|
|
| Model | Parameters | Training GPUs | |
|
|---------------------------------------------------------------------------|------------|-----------------| |
|
| [Flan-Alpaca-Base](https://huggingface.co/declare-lab/flan-alpaca-base) | 220M | 1x A6000 | |
|
| [Flan-Alpaca-Large](https://huggingface.co/declare-lab/flan-alpaca-large) | 770M | 1x A6000 | |
|
| [Flan-Alpaca-XL](https://huggingface.co/declare-lab/flan-alpaca-xl) | 3B | 1x A6000 | |
|
| [Flan-Alpaca-XXL](https://huggingface.co/declare-lab/flan-alpaca-xxl) | 11B | 4x A6000 (FSDP) | |
|
|
|
### Why? |
|
|
|
[Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) represents an exciting new direction |
|
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily. |
|
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data. |
|
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model. |
|
However, the original implementation is less accessible due to licensing constraints of the |
|
underlying [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) model. |
|
Furthermore, users have noted [potential noise](https://github.com/tloen/alpaca-lora/issues/65) in the synthetic |
|
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but |
|
less diverse) instructions such as [Flan-T5](https://arxiv.org/abs/2210.11416). |
|
|
|
### Usage |
|
|
|
``` |
|
from transformers import pipeline |
|
|
|
prompt = "Write an email about an alpaca that likes flan" |
|
model = pipeline(model="declare-lab/flan-alpaca-xl") |
|
model(prompt, max_length=128, do_sample=True) |
|
|
|
# Dear AlpacaFriend, |
|
# My name is Alpaca and I'm 10 years old. |
|
# I'm excited to announce that I'm a big fan of flan! |
|
# We like to eat it as a snack and I believe that it can help with our overall growth. |
|
# I'd love to hear your feedback on this idea. |
|
# Have a great day! |
|
# Best, AL Paca |
|
``` |
