File size: 1,552 Bytes
97a762e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
---
license: other
language:
- en
pipeline_tag: text-generation
inference: false
tags:
- transformers
- gguf
- imatrix
- SOLAR-10.7B-Instruct-v1.0
---
Quantizations of https://huggingface.co/upstage/SOLAR-10.7B-Instruct-v1.0
# From original readme
# **Usage Instructions**
This model has been fine-tuned primarily for single-turn conversation, making it less suitable for multi-turn conversations such as chat.
### **Version**
Make sure you have the correct version of the transformers library installed:
```sh
pip install transformers==4.35.2
```
### **Loading the Model**
Use the following Python code to load the model:
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Upstage/SOLAR-10.7B-Instruct-v1.0")
model = AutoModelForCausalLM.from_pretrained(
"Upstage/SOLAR-10.7B-Instruct-v1.0",
device_map="auto",
torch_dtype=torch.float16,
)
```
### **Conducting Single-Turn Conversation**
```python
conversation = [ {'role': 'user', 'content': 'Hello?'} ]
prompt = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, use_cache=True, max_length=4096)
output_text = tokenizer.decode(outputs[0])
print(output_text)
```
Below is an example of the output.
```
<s> ### User:
Hello?
### Assistant:
Hello, how can I assist you today? Please feel free to ask any questions or request help with a specific task.</s>
``` |