File size: 1,363 Bytes
41d0385 5ae0dd3 41d0385 9735123 41d0385 9735123 41d0385 9735123 41d0385 70c6292 41d0385 a7d464c 41d0385 6e821fc 41d0385 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
---
license: other
library_name: peft
tags:
- llama2
---
# Model Card for PIPPA ShareGPT Subset Lora 7b
It is an experimental Lora focused on Roleplay that uses a subset of [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT)
## Usage
Custom
```
SYSTEM: Do thing
USER: {prompt}
CHARACTER:
```
## Bias, Risks, and Limitations
This Lora is not intended for supplying factual information or advice in any form
## Training Details
### Training Data
1k of conversation from [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT)
### Training Procedure
The version of this Lora uploaded on this repository was trained using a 8x RTX A6000 cluster in 8-bit with regular LoRA adapters and 32-bit AdamW optimizer.
#### Training Hyperparameters
Training using a fork of [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) with two paths [Patch 1](https://github.com/bdashore3/axolotl/commit/995557bdf3c6c8b3e839b224ef9513fc2b097f30) [Patch 2](https://github.com/bdashore3/axolotl/commit/8970280de2ea01e41c044406051922715f4086cb)
- load_in_8bit: true
- lora_r: 16
- lora_alpha: 16
- lora_dropout: 0.01
- gradient_accumulation_steps: 6
- micro_batch_size: 4
- num_epochs: 3
- learning_rate: 0.000065
## Environmental Impact
Finetuning this model on 8xNVIDIA A6000 48GB in parallel takes about 30 minutes (7B) |