Update README.md

README.md

@@ -30,8 +30,15 @@ The prompt format is the same as the [Zephyr](https://huggingface.co/HuggingFace
 ```
 
 
-The quantized model can be utilized through the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/chat_templating) functionality as follows:
+# Using the model with Huggingface
 
+First you need to install the dependencies
+
+```
+pip install autoawq transformers
+```
+
+The quantized model can be utilized through the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/chat_templating) functionality as follows:
 
 
 ```python
@@ -78,6 +85,41 @@ outputs = model.generate(input_prompt, max_new_tokens=256, do_sample=True)
 print(tokenizer.batch_decode(outputs)[0])
 ```
 
+# Using the model with vLLM
+
+Install vLLM
+
+```
+pip install vllm
+```
+
+Then use from python API:
+
+```python
+from vllm import LLM, SamplingParams
+from transformers import AutoTokenizer
+
+
+tokenizer = AutoTokenizer.from_pretrained(
+    "ilsp/Meltemi-7B-Instruct-v1-AWQ",
+    trust_remote_code=False
+)
+
+prompts = ["Πες μου αν έχεις συνείδηση."]
+prompts = [tokenizer.apply_chat_template(p, add_generation_prompt=True, tokenize=False) for p in prompts]
+
+sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=256)
+llm = LLM(model="ilsp/Meltemi-7B-Instruct-v1-AWQ", tokenizer="ilsp/Meltemi-7B-Instruct-v1-AWQ", quantization="awq")
+
+outputs = llm.generate(prompts, sampling_params)
+
+for output in outputs:
+    prompt = output.prompt
+    generated_text = output.outputs[0].text
+    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
+```
+
+
 # Ethical Considerations
 
 This model has not been aligned with human preferences, and therefore might generate misleading, harmful, and toxic content.