Instructions to use mrmsoares/poti-wp-llama3-8b-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mrmsoares/poti-wp-llama3-8b-lora with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("unsloth/llama-3-8b-bnb-4bit")
model = PeftModel.from_pretrained(base_model, "mrmsoares/poti-wp-llama3-8b-lora")

Transformers

How to use mrmsoares/poti-wp-llama3-8b-lora with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mrmsoares/poti-wp-llama3-8b-lora")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mrmsoares/poti-wp-llama3-8b-lora")
model = AutoModelForCausalLM.from_pretrained("mrmsoares/poti-wp-llama3-8b-lora", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mrmsoares/poti-wp-llama3-8b-lora with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mrmsoares/poti-wp-llama3-8b-lora"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mrmsoares/poti-wp-llama3-8b-lora",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/mrmsoares/poti-wp-llama3-8b-lora

SGLang

How to use mrmsoares/poti-wp-llama3-8b-lora with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mrmsoares/poti-wp-llama3-8b-lora" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mrmsoares/poti-wp-llama3-8b-lora",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mrmsoares/poti-wp-llama3-8b-lora" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mrmsoares/poti-wp-llama3-8b-lora",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use mrmsoares/poti-wp-llama3-8b-lora with Docker Model Runner:
```
docker model run hf.co/mrmsoares/poti-wp-llama3-8b-lora
```

See axolotl config

axolotl version: 0.13.0.dev0


# ============================================
# 🔥 POTI-WP - MODO ESTÁVEL MAS FORTE 🔥
# ============================================

base_model: unsloth/llama-3-8b-bnb-4bit
model_type: LlamaForCausalLM
load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: mrmsoares/poti-wp-instruct
    type: alpaca
    shards: 1

dataset_prepared_path: last_run_prepared
val_set_size: 0.02
output_dir: ./outputs/poti-wp-lora-chernobyl

adapter: qlora

sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false

lora_r: 64
lora_alpha: 128
lora_dropout: 0.05
lora_target_linear: true

# Desabilita os kernels otimizados que estão dando problema
lora_mlp_kernel: false
lora_qkv_kernel: false
lora_o_kernel: false

micro_batch_size: 2
gradient_accumulation_steps: 8

num_epochs: 4
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 2e-4
train_on_inputs: false
group_by_length: false

bf16: auto
fp16: false
tf32: true

gradient_checkpointing: true

flash_attention: true

torch_compile: false

dataloader_num_workers: 2
dataloader_pin_memory: true

logging_steps: 1
warmup_steps: 10
evals_per_epoch: 2
saves_per_epoch: 1

weight_decay: 0.01
seed: 42
neftune_noise_alpha: 5

outputs/poti-wp-lora-chernobyl

This model is a fine-tuned version of unsloth/llama-3-8b-bnb-4bit on the mrmsoares/poti-wp-instruct dataset. It achieves the following results on the evaluation set:

Loss: 12.6679
Ppl: 317384.6004
Memory/max Active (gib): 12.45
Memory/max Allocated (gib): 12.45
Memory/device Reserved (gib): 16.26

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 10
training_steps: 16

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	12.6570	313965.2955	11.18	11.18	11.46
0.5264	0.4324	2	12.6579	314226.4989	12.45	12.45	16.26
0.4617	0.8649	4	12.6410	308956.2535	12.45	12.45	16.26
0.3077	1.2162	6	12.6273	304764.1832	12.45	12.45	16.26
0.206	1.6486	8	12.6059	298315.9366	12.45	12.45	16.26
0.2138	2.0	10	12.5923	294282.2715	12.45	12.45	16.26
0.1438	2.4324	12	12.6695	317914.7353	12.45	12.45	16.26
0.1409	2.8649	14	12.6859	323160.2306	12.45	12.45	16.26
0.1006	3.2162	16	12.6679	317384.6004	12.45	12.45	16.26

Framework versions

PEFT 0.18.1.dev0
Transformers 4.57.1
Pytorch 2.8.0+cu128
Datasets 4.4.2
Tokenizers 0.22.1

Downloads last month: 4

Model tree for mrmsoares/poti-wp-llama3-8b-lora

Base model

meta-llama/Meta-Llama-3-8B

Quantized

unsloth/llama-3-8b-bnb-4bit

Adapter

(316)

this model