Update README.md

README.md

@@ -13,6 +13,288 @@ pipeline_tag: text-generation
 
 This instruction model was built via parameter-efficient QLoRA finetuning of [llama-2-70b](https://huggingface.co/meta-llama/Llama-2-70b-hf) on the first 25k rows of [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin) (an open-source implementation of [Microsoft's Orca](https://www.microsoft.com/en-us/research/publication/orca-progressive-learning-from-complex-explanation-traces-of-gpt-4/)). Finetuning was executed on a single H100 (80 GB PCIe) for roughly 17 hours on the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
 
+* Model license: Llama 2 Community License Agreement
+* Basic usage: [notebook](assets/basic_inference_llama_2_70b_dolphin.ipynb)
+* Finetuning code: [notebook](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/sft-llama-2-70b-dolphin-peft.ipynb)
+* Loss curves: [plot](https://huggingface.co/dfurman/llama-2-70b-dolphin-peft#finetuning-description)
+* Runtime stats: [table](https://huggingface.co/dfurman/llama-2-70b-dolphin-peft#runtime-tests)
+
+### Example prompts and responses
+
+Example 1: 
+
+**User**:
+> You are a helpful assistant. Write me a numbered list of things to do in New York City.\n
+
+**llama-2-70b-dolphin-peft**:
+> 1. Visit the Statue of Liberty: Take a ferry to Liberty Island and see the iconic symbol of freedom and democracy.
+> 2. Explore Central Park: Enjoy the lush greenery, take a stroll, or rent a bike to explore the park's many attractions, such as the Central Park Zoo and Strawberry Fields.
+> 3. Visit the Empire State Building: Take an elevator to the observation deck for breathtaking views of the city.
+> 4. Walk the Brooklyn Bridge: Cross the historic bridge and enjoy the stunning views of the Manhattan skyline.
+> 5. Experience Times Square: Take in the bright lights, billboards, and bustling energy of this iconic intersection.
+> 6. Visit the Metropolitan Museum of Art: Explore the vast collection of art and artifacts from around the world.
+> 7. Take a food tour: Sample the diverse cuisine of New York City, from pizza to bagels to ethnic specialties.
+> 8. Attend a Broadway show: Enjoy a world-class performance in the heart of the Theater District.
+
+<br>
+
+Example 2: 
+
+**User**:
+> You are a helpful assistant. Daniel is in need of a haircut. His barber works Mondays, Wednesdays, and Fridays. So, Daniel went in for a haircut on Sunday. Does this make logical sense? Respond only with a yes or no answer in as few words as possible.\n
+
+**llama-2-70b-dolphin-peft**:
+
+> No.
+> 
+> Explanation: Daniel's barber is not available on Sundays, so he cannot get a haircut on that day. He should go on a day when the barber is working, such as Monday, Wednesday, or Friday.
+> 
+> So, the answer is no, it does not make logical sense for Daniel to go for a haircut on Sunday. He should go on a day when the barber is available.
+
+<br>
+
+Example 3: 
+
+**User**:
+
+> You are a helpful assistant. Write a short email inviting my friends to a dinner party on Friday. Respond succinctly.\n
+
+**llama-2-70b-dolphin-peft**:
+
+> Subject: Dinner Party Invitation - Friday, 7 PM
+> 
+> Dear Friends,
+> 
+> I hope this email finds you well! I'm excited to invite you to a dinner party at my place this Friday, at 7 PM. We'll have a delicious meal, great company, and lots of fun. Please let me know if you can make it, so I can plan accordingly.
+> 
+> Looking forward to seeing you all!
+> 
+> Kind regards,
+> Your Name
+
+> P.S. Please let me know if you have any dietary restrictions or allergies.
+
+<br>
+
+Example 4: 
+
+**User**:
+
+> You are a helpful assistant. Tell me a recipe for vegan banana bread.\n
+
+**llama-2-70b-dolphin-peft**:
+
+> Sure! Here's a delicious and easy vegan banana bread recipe:
+> 
+> Ingredients:
+> - 2 cups all-purpose flour
+> - 1/2 cup sugar
+> - 1/2 cup vegan butter (such as Earth Balance)
+> - 1/2 cup vegan milk (such as almond milk)
+> - 1/2 cup unsweetened applesauce
+> - 1/2 cup mashed ripe bananas (about 2 medium bananas)
+> - 1 teaspoon baking soda
+> - 1/2 teaspoon salt
+> - 1/2 teaspoon ground cinnamon
+> - 1/2 teaspoon ground nutmeg
+> - 1/2 teaspoon ground cloves
+> - 1/2 cup chopped walnuts (optional)
+> 
+> Instructions:
+> 1. Preheat the oven to 350°F (175°C). Grease a 9x5-inch loaf pan with vegan butter or cooking spray.
+> 2. In a large bowl, mix together the flour, sugar, vegan butter, vegan milk, applesauce, bananas, baking soda, salt, cinnamon, nutmeg, and cloves. Stir until well combined.
+> 3. Fold in the chopped walnuts, if using.
+> 4. Pour the batter into the prepared loaf pan.
+> 5. Bake for 50-60 minutes, or until a toothpick inserted into the center of the bread comes out clean.
+> 6. Let the bread cool in the pan for 10 minutes before transferring it to a wire rack to cool completely.
+> 7. Slice and enjoy!
+> 
+> Note: You can also add chocolate chips, dried fruit, or other mix-ins to the batter for extra flavor and texture. Enjoy your vegan banana bread!
+
+## Model Description
+
+The architecture is a modification of a standard decoder-only transformer.
+
+The llama-2 models have been modified from a standard transformer in the following ways:
+* It uses [grouped-query attention](https://arxiv.org/pdf/2305.13245.pdf) (GQA), a generalization of multi-query attention which uses an intermediate number of key-value heads.
+* It uses the [SwiGLU activation function](https://arxiv.org/abs/2002.05202)
+* It uses [rotary positional embeddings](https://arxiv.org/abs/2104.09864) (RoPE)
+
+| Hyperparameter | Value |
+|----------------|-------|
+| n_parameters | 70B |
+| tokens | 2.0T |
+| vocab size | 32000 |
+| sequence length | 4096 |
+
+## Finetuning Description
+
+This model was trained on a single H100 (80 GB PCIe) for about 17 hours using the [Lambda Labs](https://cloud.lambdalabs.com/instances) platform.
+
+![loss curves](https://raw.githubusercontent.com/daniel-furman/sft-demos/main/assets/jul_24_23_1_14_00_log_loss_curves_llama-2-70b-dolphin.png)
+
+The above loss curve was generated from the run's private wandb.ai log.  
+
+## PreTraining Data
+
+For more details on the pretraining process, see [Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf).
+
+The data was tokenized using the [Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf) tokenizer.
+
+## Limitations and Biases
+
+_The following language is modified from [EleutherAI's GPT-NeoX-20B](https://huggingface.co/EleutherAI/gpt-neox-20b)_
+
+This model can produce factually incorrect output, and should not be relied on to produce factually accurate information.
+This model was trained on various public datasets.
+While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+
+## How to Use
+
+Basic usage: [notebook](assets/basic_inference_llama_2_70b_dolphin.ipynb)
+
+Install and import the package dependencies:  
+
+```python
+!pip install -q -U huggingface_hub peft transformers torch accelerate
+```
+
+```python
+import torch
+from peft import PeftModel, PeftConfig
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+```
+
+Sign into a HF account with access to Llama-2:
+
+```python
+from huggingface_hub import notebook_login
+notebook_login()
+```
+
+Basic model loading:
+
+```python
+peft_model_id = "dfurman/llama-2-70b-dolphin-peft"
+config = PeftConfig.from_pretrained(peft_model_id)
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    config.base_model_name_or_path,
+    quantization_config=bnb_config,
+    use_auth_token=True,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
+tokenizer.pad_token = tokenizer.eos_token
+
+# Load the Lora model
+model = PeftModel.from_pretrained(model, peft_model_id)
+```
+
+Once loaded, the model and tokenizer can be used with the following code:
+
+```python
+def llama_generate(
+    model: AutoModelForCausalLM,
+    tokenizer: AutoTokenizer,
+    prompt: str,
+    max_new_tokens: int = 128,
+    temperature: int = 1.0,
+) -> str:
+    """
+    Initialize the pipeline
+    Uses Hugging Face GenerationConfig defaults
+        https://huggingface.co/docs/transformers/v4.29.1/en/main_classes/text_generation#transformers.GenerationConfig
+    Args:
+        model (transformers.AutoModelForCausalLM): Falcon model for text generation
+        tokenizer (transformers.AutoTokenizer): Tokenizer for model
+        prompt (str): Prompt for text generation
+        max_new_tokens (int, optional): Max new tokens after the prompt to generate. Defaults to 128.
+        temperature (float, optional): The value used to modulate the next token probabilities.
+            Defaults to 1.0
+    """
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+    inputs = tokenizer(
+        [prompt],
+        return_tensors="pt",
+        return_token_type_ids=False,
+    ).to(
+        device
+    )  # tokenize inputs, load on device
+
+    # when running Torch modules in lower precision, it is best practice to use the torch.autocast context manager.
+    with torch.autocast("cuda", dtype=torch.bfloat16):
+        response = model.generate(
+            **inputs,
+            max_new_tokens=max_new_tokens,
+            temperature=temperature,
+            return_dict_in_generate=True,
+            eos_token_id=tokenizer.eos_token_id,
+            pad_token_id=tokenizer.pad_token_id,
+        )
+
+    decoded_output = tokenizer.decode(
+        response["sequences"][0],
+        skip_special_tokens=True,
+    )  # grab output in natural language
+
+    return decoded_output[len(prompt) :]  # remove prompt from output
+```
+
+We can now generate text! For example:
+
+```python
+prompt = "You are a helpful assistant. Tell me a recipe for vegan banana bread.\n"
+
+response = llama_generate(
+    model,
+    tokenizer,
+    prompt,
+    max_new_tokens=500,
+    temperature=0.92,
+)
+
+print(response)
+```
+
+
+### Runtime tests
+
+| runtime / 50 tokens (sec) | GPU             | attn | torch dtype | VRAM (GB) |
+|:-----------------------------:|:----------------------:|:---------------------:|:-------------:|:-----------------------:|
+| 4.50                        | 1x H100 (80 GB PCIe)  | torch               | nf4    | 39                    |
+
+The above runtime stats were generated from this [notebook](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/llama-2/dolphin/postprocessing-llama-2-70b-dolphin-peft.ipynb). 
+
+## Acknowledgements
+
+This model was finetuned by Daniel Furman on July 23, 2023 and is intended primarily for research purposes.
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.
+
+## Meta citation for llama-2 blog
+
+```
+@online{Meta2023Introducing,
+    author    = {Meta AI},
+    title     = {Meta and Microsoft Introduce the Next Generation of Llama},
+    year      = {2023},
+    url       = {https://about.fb.com/news/2023/07/llama-2/},
+    note      = {Accessed: 2023-07-24},
+    urldate   = {2023-07-24}
+}
+```
+
 ---
 
 ### Framework versions