Update README.md

README.md

@@ -23,7 +23,45 @@ This model is a fine-tuned version of [google/gemma-7b](https://huggingface.co/g
 The model was fine-tuned with the following [script using Lora (no, qlora)](https://huggingface.co/philschmid/gemma-7b-dolly-chatml/blob/main/trl-lora.py). I also included a [inference script](https://huggingface.co/philschmid/gemma-7b-dolly-chatml/blob/main/inference.py) to make sure it works since there were some issues with Gemma. Results of the inference test are
 
 ```bash
+prompt:
+ What is the capital of Germany? Explain why thats the case and if it was different in the past?
+response:
+ Berlin is the capital of Germany.  It was the capital of Prussia until 1918, when the monarchy was abolished.  It was also the capital of the Weimar Republic.  It was the capital of the Third Reich until 1945, when it was liberated by the allies.  It has been the capital of the Federal Republic of Germany since 1949.  It is the largest city in the country with a population of 3.6 million people.  It is also the seat of the government and parliament.
 
+prompt:
+ In a town, 60% of the population are adults. Among the adults, 30% have a pet dog and 40% have a pet cat. What percentage of the total population has a pet dog?
+response:
+ 60% of the total population have a pet dog. The calculation is 30% of adults multiplied by 60% of the total population. 30% of adults is 18% of the total population and 18% multiplied by 60% is 10.8% or 60% of the total population.
+```
+
+### Run inference
+
+```python
+import torch
+from peft import AutoPeftModelForCausalLM
+from transformers import  AutoTokenizer, pipeline
+
+peft_model_id = "philschmid/gemma-7b-dolly-chatml"
+
+# Load Model with PEFT adapter
+tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
+model = AutoPeftModelForCausalLM.from_pretrained(peft_model_id, device_map="auto", torch_dtype=torch.float16)
+pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
+eos_token = tokenizer("<|im_end|>",add_special_tokens=False)["input_ids"][0]
+print(f"eos_token: {eos_token}")
+
+# run inference
+messages = [
+    {
+        "role": "user",
+        "content": "What is the capital of Germany? Explain why thats the case and if it was different in the past?"
+    }
+]
+
+prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+outputs = pipe(prompt, max_new_tokens=1024, do_sample=True, temperature=0.7, top_k=50, top_p=0.95, eos_token_id=eos_token)
+print(outputs[0]['generated_text'][len(prompt):])
+#  Berlin is the capital of Germany.  It was the capital of Prussia until 1918, when the monarchy was abolished.  It was also the capital of the Weimar Republic.  It was the capital of the Third Reich until 1945, when it was liberated by the allies.  It has been the capital of the Federal Republic of Germany since 1949.  It is the largest city in the country with a population of 3.6 million people.  It is also the seat of the government and parliament.
 ```