Update README.md

README.md

@@ -14,6 +14,39 @@ language:
 - All prompts are formatted to conversations pairs of [USER] and [Yaya]. For example:
 
 [User]: Compose a long Gulf War story set in Istanbul, where a content chair encounters Elizabeth Barrett Browning, inspired by the works of Henrik Ibsen.
+
 [Yaya]: As an old and content chair sat in the corner of a bustling coffee house in Istanbul, he witnessed the city's frenzied energy. It was the early 1990s, and the Gulf War had just begun.
 
-There was great political strife in the air, and tensions were high. People rushed about, frantically discussing the war and its potential impact on their lives. [...]
+There was great political strife in the air, and tensions were high. People rushed about, frantically discussing the war and its potential impact on their lives. [...]
+
+- Load LoRA weights with PEFT model
+```
+        from transformers import  GPTJForCausalLM,AutoTokenizer, GenerationConfig
+        from peft import PeftModel
+
+        lora_weights = 'kietbs/pygmalion_6B_yaya'  # Please download the weight, and change this path accordingly
+        load_in_8bit = True
+        model = GPTJForCausalLM.from_pretrained(pretrain_name, load_in_8bit=load_in_8bit, device_map='auto', torch_dtype=torch.float16)
+        model = PeftModel.from_pretrained(model,lora_weights,torch_dtype=torch.float16,device_map={'':0})
+        model = torch.compile(model)
+
+        GenerationConfig(
+                temperature=0.1,
+                top_p=0.75,
+                top_k=40,
+                num_beams=4
+            )
+
+    text = '[User]: What's the best food in Hanoi?''
+    input_ids = st.session_state.tokenizer(text, return_tensors='pt')['input_ids'].to('cuda')
+    with torch.no_grad():
+        output = st.session_state['model'].generate(input_ids=input_ids, generation_config=st.session_state.gen_config,return_dict_in_generate=True, output_scores=True,max_new_tokens=256)
+        s = output.sequences[0]
+        output = st.session_state.tokenizer.decode(s)
+        print('Raw:',output)
+        
+```
+
+Output:
+[User]: What's the best food in Hanoi?
+[Yaya]: The best food in Hanoi can vary depending on what you're looking for. Some of the most popular dishes include pho, banh mi, banh xeo, and bún chả.<|endoftext|>