teknium

/

OpenHermes-2.5-Mistral-7B

like 783

Hello there, first of all thank you for this great work.
I have an issue when I try to run the model

import transformers
model = transformers.AutoModelForCausalLM.from_pretrained('teknium/OpenHermes-2.5-Mistral-7B', load_in_8bit=True,  device_map='auto')
tokenizer = transformers.AutoTokenizer.from_pretrained('teknium/OpenHermes-2.5-Mistral-7B', add_eos_token=True)

messages = [
    {"role": "system", "content": "You are Hermes 2."},
    {"role": "user", "content": "Hello, what are your limitations ?"}
]
gen_input = tokenizer.apply_chat_template(messages, return_tensors="pt")
gen_output = model.generate(**gen_input)
tokenizer.decode(gen_output[0], skip_special_tokens=True)

I get the error :
```python

TypeError Traceback (most recent call last)
Cell In [100], line 7
2 messages = [
3 {"role": "system", "content": "You are Hermes 2."},
4 {"role": "user", "content": "Hello, what are your limitations ?"}
5 ]
6 gen_input = tokenizer.apply_chat_template(messages, return_tensors="pt")
----> 7 gen_output = model.generate(**gen_input)
8 tokenizer.decode(gen_output[0], skip_special_tokens=True)

TypeError: transformers.generation.utils.GenerationMixin.generate() argument after ** must be a mapping, not Tensor

Please, is there something I'm doing wrong ? I am using (transformers==4.36.0.dev0)