Provide usage example

README.md

@@ -1,4 +1,19 @@
 ---
+tags:
+- finetuned
+- quantized
+- 4-bit
+- AWQ
+- transformers
+- pytorch
+- mistral
+- text-generation
+- conversational
+- license:apache-2.0
+- autotrain_compatible
+- endpoints_compatible
+- text-generation-inference
+- region:us
 base_model: senseable/WestLake-7B-v2
 license: apache-2.0
 language:
@@ -37,6 +52,52 @@ This repo contains AWQ model files for [Common Sense's WestLake 7B v2](https://h
 
 These files were quantised using hardware kindly provided by [SolidRusT Networks](https://solidrust.net/).
 
+## How to use
+
+### Install the necessary packages
+
+```bash
+pip install --upgrade autoawq autoawq-kernels
+```
+
+### Example Python code
+
+```bash
+from awq import AutoAWQForCausalLM
+from transformers import AutoTokenizer, TextStreamer
+
+quant_path = "/srv/home/shaun/repos/samantha-1.1-westlake-7b-laser-AWQ"
+
+# Load model
+model = AutoAWQForCausalLM.from_quantized(quant_path,
+                                          fuse_layers=True)
+tokenizer = AutoTokenizer.from_pretrained(quant_path,
+                                          trust_remote_code=True)
+streamer = TextStreamer(tokenizer,
+                        skip_prompt=True,
+                        skip_special_tokens=True)
+
+# Convert prompt to tokens
+prompt_template = """\
+<|system|>
+</s>
+<|user|>
+{prompt}</s>
+<|assistant|>"""
+
+prompt = "You're standing on the surface of the Earth. "\
+        "You walk one mile south, one mile west and one mile north. "\
+        "You end up exactly where you started. Where are you?"
+
+tokens = tokenizer(prompt_template.format(prompt=prompt),
+                  return_tensors='pt').input_ids.cuda()
+
+# Generate output
+generation_output = model.generate(tokens,
+                                  streamer=streamer,
+                                  max_new_tokens=512)
+```
+
 ### About AWQ
 
 AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.