Update README.md

README.md

@@ -1,3 +1,65 @@
 ---
-license: mit
+license: afl-3.0
+language:
+- en
+widget:
+- text: "<human>What's my name?<bot>"
+  example_title: "Who am I?"
+- text: "<human>How to make a campfire<bot>"
+  example_title: "Tutorial"
 ---
+
+# Supervised Finetuning demonstration
+
+Models are finetuned on generated conversation curated from the [Open Assistant](https://github.com/LAION-AI/Open-Assistant).
+
+
+# Mixing reward model with sampling
+
+We can use reward model to rank the best answer using this example code:
+
+```
+import torch
+from transformers import AutoModelForSequenceClassification
+from transformers import AutoTokenizer, AutoModelForCausalLM
+tokenizer = AutoTokenizer.from_pretrained("theblackcat102/galactica-1.3b-v2")
+model = AutoModelForCausalLM.from_pretrained("theblackcat102/galactica-1.3b-v2").eval().half().cuda()
+
+reward_name = "OpenAssistant/reward-model-deberta-v3-large"
+rank_model, rank_tokenizer = AutoModelForSequenceClassification.from_pretrained(reward_name), AutoTokenizer.from_pretrained(reward_name)
+rank_model = rank_model.eval().half().cuda()
+
+questions = ["<question>How do I make a resume?<answer>"]
+for question in questions:
+    inputs = tokenizer(question, return_tensors="pt", padding=True).to(0)
+    if 'token_type_ids' in inputs:
+        inputs.pop('token_type_ids')
+    outputs = model.generate(**inputs, do_sample=True,
+        top_k=60,
+        max_length=220,
+        num_return_sequences=80, 
+        early_stopping=True
+    )
+    print(question)
+
+    results = []
+    for i, beam_output in enumerate(outputs):
+        output = tokenizer.decode(beam_output, truncate_before_pattern=[r"\n\n^#", "^'''", "\n\n\n"])
+        question, answer = output.split('<answer>', maxsplit=1)
+        answer = answer.split('<question>')[0].replace('<|endoftext|>', '').lstrip().split('<answer>')[0]
+        rank_inputs = rank_tokenizer(question, answer, return_tensors="pt", padding=True, max_length=512, truncation=True).to(1)
+        score = rank_model(**rank_inputs).logits[0].cpu().detach()
+        results.append((answer, score, output))
+    full_results[question] = results
+    sorted_result = sorted(results, key=lambda x:x[1], reverse=True)
+    total_scores += sorted_result[0][1].item()
+    print('score',sorted_result[0][1].item())
+    print('-----Best rank-----')
+    print(sorted_result[0][0])
+    print('-------------------')
+```
+
+
+Checkout weights and biases [report](https://api.wandb.ai/report/theblackcat102/8yg0c0r2) for training detail. 
+
+Thanks to [BASIC lab](https://basiclab.lab.nycu.edu.tw/Yummy/index.html#) for compute resource. BASIC Lab is an academic research lab which focuses in multi-modality learning and data mining domain.