File size: 1,573 Bytes
c784878 0478b53 c784878 c79e2d2 0478b53 c784878 0478b53 c784878 081dcf7 c784878 0478b53 c784878 0478b53 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
---
base_model: unsloth/Meta-Llama-3.1-8B
language:
- en
license: llama3.1
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
datasets:
- mpasila/Discord-short-sharegpt
---
Trained on Discord chatlogs from this [dataset](https://huggingface.co/datasets/mpasila/Discord-short-sharegpt).
Uses Llama 3.1 formatting.
LoRA: [mpasila/Llama-3.1-Discord-Short-LoRA-8B](https://huggingface.co/mpasila/Llama-3.1-Discord-Short-LoRA-8B)
Trained with regular LoRA (not quantized/QLoRA) and LoRA rank was 128 and Alpha set to 32. Trained for 1 epoch using A40 for about 5,5 hours.
```python
args = UnslothTrainingArguments(
per_device_train_batch_size = 1,
gradient_accumulation_steps = 8,
warmup_ratio = 0.1,
num_train_epochs = 1,
learning_rate = 5e-5,
embedding_learning_rate = 5e-6,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.00,
lr_scheduler_type = "cosine",
seed = 3407,
output_dir = "outputs",
),
```
# Uploaded model
- **Developed by:** mpasila
- **License:** Llama 3.1 Community License Agreement
- **Finetuned from model :** unsloth/Meta-Llama-3.1-8B
This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth) |