Hugging Face's logo

lucasmccabe-lmi
/
GPT-6.7B-CleanedAlpaca-1

Text Generation PEFT
English alpaca gpt lora llm
Model card Files Community
2
Edit model card

Model Card

We provide a low-rank adapter for an instruction-tuned 6.7B-parameter GPT3-style language model.

Example instruction:

Write a list of things I need to do to plan a watch party for Game 4 of the Knicks-Cavs playoff series.

Corresponding response from the model:

1. Find out where the game will be played and purchase tickets.
2. Decide on food, drinks, decorations, and entertainment.
3. Plan the route to get there.
4. Arrange transportation.
5. Organize games and activities for guests.
6. Make sure everyone has their tickets.
7. Set up the venue.
8. Prepare for the event.
9. Clean up after the event.
10. Celebrate with friends and family.

Quickstart

Setting up the model using transformers and peft:

from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("cerebras/Cerebras-GPT-6.7B")
tokenizer.pad_token_id = 0

model = AutoModelForCausalLM.from_pretrained(
    "cerebras/Cerebras-GPT-6.7B",
    load_in_8bit=True,
    device_map="auto",
)
model = PeftModel.from_pretrained(
  model,
  "lucasmccabe-lmi/GPT-6.7B-CleanedAlpaca-1"
)

Prompting

For instructions that do not require extraneous inputs, the recommended prompt is:

Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here --- >

### Response:

For instructions that do require extraneous inputs, the recommended prompt is:

Below is an instruction that describes a task. Write a response that appropriately completes the request.


### Instruction:

< -- instruction goes here -- >

### Input:

< -- extraneous input goes here -- >

### Response:

Since the model performs causal language modeling, the model's response to the prompt is the text completing the sequence beginning with the prompt.

Instruction-Tuning

This model was instruction-tuned on a cleaned version of the Stanford Alpaca dataset, consisting of 52k post-processed instruction-input-output triplets derived from OpenAI's text-davinci-003.

  • Epochs: 3

  • Batch size: 128

  • Cutoff length: 512

  • Learning rate: 2e-5

  • LoRA r: 4

  • LoRA alpha: 16

  • LoRA dropout: 0.05

  • LoRA target modules: c_attn

  • Dataset: yahma/alpaca-cleaned

  • License: The instruction-tuning data is subject to the Creative Commons 4.0 license.

Base Model

This model was instruction-tuned from a 6.7B variant from the Cerebras-GPT family. These models were pre-trained to the "Chinchilla-optimal" 20*6.7B tokens from EleutherAI/The Pile.

Software

We used LMI's internal liger library, which is built on PyTorch and the excellent Hugging Face stack (transformers, accelerate, etc.).

Licensing Information

We release this adapter under the Creative Commons NonCommercial (CC BY-NC 4.0) license.

Author

Downloads last month
1

Datasets used to train lucasmccabe-lmi/GPT-6.7B-CleanedAlpaca-1