seonglae commited on
Commit
0445c45
1 Parent(s): c44b096

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +43 -0
README.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ inference: false
3
+ license: other
4
+ tags:
5
+ - llama-2
6
+ - llama2
7
+ - gptq
8
+ - auto-gptq
9
+ - 7b
10
+ - llama
11
+ - 4bit
12
+ ---
13
+
14
+ # Get Started
15
+ This model should use [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) so you need to use `auto-gptq`
16
+
17
+ ```py
18
+ from transformers import AutoTokenizer, pipeline, LlamaForCausalLM, LlamaTokenizer
19
+ from auto_gptq import AutoGPTQForCausalLM
20
+
21
+ model_id = 'seonglae/llama-2-7b-chat-hf-gptq'
22
+ tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
23
+ model = AutoGPTQForCausalLM.from_quantized(
24
+ model_id,
25
+ model_basename=model_basename,
26
+ trust_remote_code=True,
27
+ device='cuda:0',
28
+ use_triton=False,
29
+ use_safetensors=True,
30
+ )
31
+
32
+ pipe = pipeline(
33
+ "text-generation",
34
+ model=model,
35
+ tokenizer=tokenizer,
36
+ temperature=0.5,
37
+ top_p=0.95,
38
+ max_new_tokens=100,
39
+ repetition_penalty=1.15,
40
+ )
41
+ prompt = "USER: Are you AI?\nASSISTANT:"
42
+ pipe(prompt)
43
+ ```