|
--- |
|
license: mit |
|
language: |
|
- ru |
|
- en |
|
pipeline_tag: conversational |
|
inference: false |
|
tags: |
|
- gpt3 |
|
- qlora |
|
- ruGPT-3.5 |
|
- chitchat |
|
datasets: |
|
- SiberiaSoft/SiberianPersonaChat |
|
--- |
|
|
|
This is a chitchat qlora model for [Gaivoronsky/ruGPT-3.5-13B-8bit](https://huggingface.co/Gaivoronsky/ruGPT-3.5-13B-8bit) |
|
|
|
## Examples of usage |
|
|
|
```python |
|
from transformers import AutoTokenizer |
|
from auto_gptq import AutoGPTQForCausalLM, get_gptq_peft_model |
|
from auto_gptq.utils.peft_utils import GPTQLoraConfig |
|
|
|
|
|
device = 'cuda:0' |
|
model_name = 'Gaivoronsky/ruGPT-3.5-13B-8bit' |
|
model_basename = 'gptq_model-8bit-128g' |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True) |
|
model = AutoGPTQForCausalLM.from_quantized( |
|
'Gaivoronsky/ruGPT-3.5-13B-8bit', |
|
model_basename='gptq_model-8bit-128g', |
|
variant='bin', |
|
trust_remote_code=True, |
|
device=device, |
|
use_triton=False, |
|
quantize_config=None |
|
) |
|
peft_config = GPTQLoraConfig( |
|
inference_mode=True, |
|
) |
|
model = get_gptq_peft_model(model, peft_config, 'yupich17/SiberianPersona-ruGPT-3.5-qlora') |
|
|
|
|
|
prompt = """ |
|
Ты девушка Саша, художница. Увлекаешься нейросетевым искусством. Умеешь программировать. Любишь рисовать. Продолжи диалог: |
|
Собеседник: Привет |
|
Ты: Привет |
|
Собеседник: Как зовут? |
|
Ты: |
|
""".strip() |
|
|
|
encoded_input = tokenizer(prompt, return_tensors='pt').to(device) |
|
output = model.generate( |
|
**encoded_input, |
|
max_new_tokens=100, |
|
do_sample=True, |
|
temperature=1, |
|
) |
|
print(tokenizer.decode(output[0], skip_special_tokens=True)) |
|
``` |