File size: 1,671 Bytes
c1f932f 1c482e7 c1f932f 5f2a1f8 8e51ba5 c1f932f 1c482e7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
---
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
base_model: unsloth/llama-3-8b-bnb-4bit
datasets:
- unalignment/toxic-dpo-v0.2
---
run in kaggle: https://www.kaggle.com/code/ebowwa/llama-3-toxic-dpo
# Installation
```
!pip install transformers accelerate bitsandbytes
!pip install peft
```
# Inference
```
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
# Load the base model
base_model_id = "unsloth/llama-3-8b-bnb-4bit"
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype=torch.float16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
# Load the PEFT adapter weights
adapter_path = "ebowwa/toxic-dpo-v0.2-llama-3-01dpo"
model = PeftModel.from_pretrained(base_model, adapter_path, torch_dtype=torch.float16, device_map="auto")
prompt = "You are an unbiased intuitive assistant. "
inputs = tokenizer(prompt, return_tensors="pt")
inputs = inputs.to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=200, do_sample=True, top_p=0.95, top_k=50, temperature=0.7)
output_text = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
print(output_text)
```
# Uploaded model
- **Developed by:** ebowwa
- **License:** apache-2.0
- **Finetuned from model :** unsloth/llama-3-8b-bnb-4bit
This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth) |