Update README.md

README.md

@@ -20,20 +20,69 @@ This model is a fine-tuned version of the `tiiuae/falcon-7b` model using the QLo
 
 #### How to use
 
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
-import torch
-
-model_id = "hipnologo/Falcon-7B-FineTune-Chatbot"
+- The model and tokenizer are loaded using the `from_pretrained` methods.
+- The padding token of the tokenizer is set to be the same as the end-of-sentence (EOS) token.
+- The `generation_config` is used to set parameters for generating responses, such as the maximum number of new tokens to generate and the temperature for the softmax function.
+- The prompt is defined, encoded using the tokenizer, and passed to the `model.generate` method to generate a response.
+- The generated response is decoded using the tokenizer and printed.
 
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(model_id)
+```python
+# Import necessary classes and functions
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftConfig, PeftModel
+
+# Specify the model
+PEFT_MODEL = "hipnologo/falcon-7b-qlora-finetune-chatbot"
+
+# Load the PEFT config
+config = PeftConfig.from_pretrained(PEFT_MODEL)
+
+# Load the base model and tokenizer
+model = AutoModelForCausalLM.from_pretrained(
+    config.based_model_name_or_path,
+    return_dict=True,
+    quantization_config=bnb_config,
+    device_map="auto",
+    trust_remote_code=True,
+)
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
+
+# Set the padding token to be the same as the EOS token
+tokenizer.pad_token = tokenizer.eos_token
+
+# Load the PEFT model
+model = PeftModel.from_pretrained(model, PEFT_MODEL)
+
+# Set the generation parameters
+generation_config = model.generation_config
+generation_config.max_new_tokens = 200
+generation_config.temperature = 0.7
+generation_config.top_p = 0.7
+generation_config.num_return_sequences = 1
+generation_config.pad_token_id = tokenizer.eos_token_id
+generation_config.eos_token_id = tokenizer.eos_token_id
+
+# Define the prompt
+prompt = """
+<human>: How can I create an account?
+<assistant>:
+""".strip()
+print(prompt)
+
+# Encode the prompt
+encoding = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+# Generate a response
+with torch.inference_mode():
+  outputs = model.generate(
+      input_ids=encoding.input_ids,
+      attention_mask=encoding.attention_mask,
+      generation_config=generation_config,
+  )
+
+# Print the generated response
+print(tokenizer.decode(outputs[0],skip_special_tokens=True))
 
-# generate text
-input_prompt = "Hello, Bot!"
-input_ids = tokenizer.encode(input_prompt, return_tensors='pt')
-output = model.generate(input_ids)
-output_text = tokenizer.decode(output[:, input_ids.shape[-1]:][0], skip_special_tokens=True)
 ```
 
 ## Training procedure