We provide a low-rank adapter for an instruction-tuned 6.7B-parameter GPT3-style language model.
Write a list of things I need to do to plan a watch party for Game 4 of the Knicks-Cavs playoff series.
Corresponding response from the model:
1. Find out where the game will be played and purchase tickets. 2. Decide on food, drinks, decorations, and entertainment. 3. Plan the route to get there. 4. Arrange transportation. 5. Organize games and activities for guests. 6. Make sure everyone has their tickets. 7. Set up the venue. 8. Prepare for the event. 9. Clean up after the event. 10. Celebrate with friends and family.
Setting up the model using
from peft import PeftModel from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("cerebras/Cerebras-GPT-6.7B") tokenizer.pad_token_id = 0 model = AutoModelForCausalLM.from_pretrained( "cerebras/Cerebras-GPT-6.7B", load_in_8bit=True, device_map="auto", ) model = PeftModel.from_pretrained( model, "lucasmccabe-lmi/GPT-6.7B-CleanedAlpaca-1" )
For instructions that do not require extraneous inputs, the recommended prompt is:
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: < -- instruction goes here --- > ### Response:
For instructions that do require extraneous inputs, the recommended prompt is:
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: < -- instruction goes here -- > ### Input: < -- extraneous input goes here -- > ### Response:
Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt.
This model was instruction-tuned on a cleaned version of the Stanford Alpaca dataset, consisting of 52k post-processed instruction-input-output triplets derived from OpenAI's
Batch size: 128
Cutoff length: 512
Learning rate: 2e-5
LoRA r: 4
LoRA alpha: 16
LoRA dropout: 0.05
LoRA target modules:
License: The instruction-tuning data is subject to the Creative Commons 4.0 license.
- Repository: cerebras/Cerebras-GPT-6.7B
- Paper: arxiv:2304.03208
- License: The base model is subject to the Apache 2.0 license.
- Model type: Transformer-based Language Model
We used LMI's internal
liger library, which is built on
PyTorch and the excellent Hugging Face stack (
We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.
- Downloads last month