Update README.md

README.md

@@ -31,4 +31,50 @@ roadGPT will process your query, referencing the relevant IRC guidelines to prov
 ### Deployment:
 roadGPT is proudly hosted on Huggingface, making it accessible to professionals, students, and road safety advocates worldwide. Our user-friendly interface ensures that you can get the answers you need with just a few clicks, regardless of your technical expertise.
 
+### Inference
+Quickly get inference running with the following required installation:
+```bash
+pip install ai2-olmo
+```
+Now, proceed as usual with HuggingFace:
+```python
+import os
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    BitsAndBytesConfig,
+    pipeline,
+)
+from transformers import BitsAndBytesConfig
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype="float16",
+    bnb_4bit_use_double_quant=False,
+)
+
+model_name = "rukaiyah-indika-ai/rv-chatbot-2"
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    quantization_config=bnb_config
+)
+    
+def generate_response(prompt):
+    inst = "You are a very helpful assistant providing solutions to road-related queries. Ensure you provide correct and relevant answers according to the IRC guidelines. If you don't know the answer to a question, please don't share false information."
+    prompt = "What are potholes, and how do they appear? "
+    pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, temperature=0.2, max_new_tokens=256)
+    ranked_results = pipe(f"<s>[INST] {inst}{prompt} [/INST]")
+
+    for result in ranked_results:
+        response = result['generated_text']
+        response = response.split("[/INST]", 1)[-1]
+        response = response.replace("<s>", "")
+        response = response.replace("</s>", "")
+        return response
+
+>> ' Potholes are bowl-shaped cavities in pavements caused by localized disintegration. They vary in size: small (25mm deep, 200mm wide), medium (25-50mm deep, up to 500mm wide), and large (exceeding 50mm deep, more than 500mm wide). Factors like water infiltration, traffic stress, and weathering lead to their development by breaking down pavement layers.'
+
+
 Dive into the world of road engineering with roadGPT at your side. Whether you're drafting a project, studying for your next exam, or simply curious about road regulations, roadGPT is here to guide you according to the esteemed IRC guidelines. Start your query now and experience the future of road-related assistance.