francislabounty commited on
Commit
b04d0e7
1 Parent(s): 5af3814

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +52 -0
README.md CHANGED
@@ -1,3 +1,55 @@
1
  ---
2
  license: mit
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
  ---
4
+ LoRA weights only and trained for research - nothing from the foundation model. Trained using Open-Assistant's dataset. Shout-out to Open-Assistant and LAION for giving us early research access to the dataset.
5
+
6
+ Sample usage
7
+ ```python
8
+ import torch
9
+ import os
10
+ import transformers
11
+ from peft import PeftModel
12
+ from transformers import LlamaTokenizer, LlamaForCausalLM
13
+
14
+ model_path = "decapoda-research/llama-13b-hf"
15
+ peft_path = 'serpdotai/llama-oasst-lora-13B'
16
+ tokenizer_path = 'decapoda-research/llama-13b-hf'
17
+
18
+ model = LlamaForCausalLM.from_pretrained(model_path, load_in_8bit=True, device_map="auto") # or something like {"": 0}
19
+ model = PeftModel.from_pretrained(model, peft_path, torch_dtype=torch.float16, device_map="auto") # or something like {"": 0}
20
+ tokenizer = LlamaTokenizer.from_pretrained(tokenizer_path)
21
+
22
+ batch = tokenizer("\n\nUser: Are you sentient?\n\nAssistant:", return_tensors="pt")
23
+
24
+ with torch.no_grad():
25
+ out = model.generate(
26
+ input_ids=batch["input_ids"].cuda(),
27
+ attention_mask=batch["attention_mask"].cuda(),
28
+ max_length=100,
29
+ do_sample=True,
30
+ top_k=50,
31
+ top_p=1.0,
32
+ temperature=1.0
33
+ )
34
+ print(tokenizer.decode(out[0]))
35
+ ```
36
+ The model will continue the conversation between the user and itself. If you want to use as a chatbot you can alter the generate method to include stop sequences for 'User:' and 'Assistant:' or strip off anything past the assistant's original response before returning.
37
+
38
+ Trained for 4 epochs with a sequence length of 2048 on 8 A6000s with an effective batch size of 120.
39
+
40
+ Training settings:
41
+ ```json
42
+ lr: 2.0e-04
43
+ lr_scheduler_type: linear
44
+ warmup_ratio: 0.06
45
+ weight_decay: 0.1
46
+ optimizer: adamw_torch
47
+ LoRA config:
48
+
49
+ target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
50
+ r: 64
51
+ lora_alpha: 32
52
+ lora_dropout: 0.05
53
+ bias: "none"
54
+ task_type: "CAUSAL_LM"
55
+ ```