Create README.md

README.md

@@ -0,0 +1,27 @@
+4bit AWQ Quantized Version of [parlance-labs/hc-mistral-alpaca-merged](https://huggingface.co/parlance-labs/hc-mistral-alpaca-merged)
+
+I used AutoAWQ
+
+```python
+from awq import AutoAWQForCausalLM
+from transformers import AutoTokenizer
+
+# setup
+quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
+quant_path="hc-mistral-alpaca-merged-awq"
+model_path="parlance-labs/hc-mistral-alpaca-merged"
+model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+
+# quantize and save model
+model.quantize(tokenizer, quant_config=quant_config)
+model.save_quantized(quant_path)
+tokenizer.save_pretrained(quant_path)
+```
+
+After you save the model you can upload it to the hub
+
+```bash
+cd hc-mistral-alpaca-merged-awq
+huggingface-cli upload parlance-labs/hc-mistral-alpaca-merged-awq .
+```