|
--- |
|
license: agpl-3.0 |
|
library_name: peft |
|
tags: |
|
- not-for-all-audiences |
|
--- |
|
|
|
# Pippa-13b-qlora |
|
|
|
This is a repository of my Llama-2-13b Qlora checkpoints of the [PIPPA-13b-ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-ShareGPT) dataset. |
|
|
|
You can read more about the dataset on its relevant page. It's a ShareGPT reformat of the [PIPPA dataset](https://huggingface.co/datasets/PygmalionAI/PIPPA) by PygmalionAI. The reformat was done to allow for axolotl compatability. |
|
|
|
### Architecture |
|
|
|
- **Model Architecture**: Llama-2-13b |
|
- **Training Algorithm**: QLora |
|
- **Dataset Used**: PIPPA-ShareGPT (pippa_sharegpt_trimmed.jsonl) |
|
|
|
### Training Details |
|
|
|
- **Dataset**: [PIPPA-ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-ShareGPT) |
|
- **Datset type**: ShareGPT |
|
- **Training Parameters**: [See Here](https://gist.github.com/bdashore3/55ae04892f31609f2c3779c4a8a55408) |
|
- **Training Environment**: Axolotl |
|
- **sequence_len**: 4096 |
|
|
|
## Instruct Format |
|
|
|
ShareGPT gets converted to vicuna format. The dataset uses modified roles of `USER` and `CHARACTER` instead of `USER` and `ASSISTANT`. |
|
|
|
``` |
|
SYSTEM: Enter roleplay mode... |
|
USER: {prompt} |
|
CHARACTER: |
|
``` |
|
|
|
## Notes |
|
|
|
This Qlora was produced as an experiment to see how the public version of PIPPA can affect a model. As a result, I have no idea if this lora is of great quality or absolute garbage. |
|
|
|
## Acknowledgments |
|
|
|
Thanks to: |
|
- PygmalionAI: The creators of the PIPPA dataset |
|
- Axolotl: Finetuning suite |
|
|
|
## Donate? |
|
All my infrastructure and cloud expenses are paid out of pocket. If you'd like to donate, you can do so here: [https://ko-fi.com/kingbri](https://ko-fi.com/kingbri) |
|
|
|
You should not feel obligated to donate, but if you do, I'd appreciate it. |
|
|
|
## Axolotl stuff |
|
|
|
## Training procedure |
|
|
|
|
|
The following `bitsandbytes` quantization config was used during training: |
|
- quant_method: bitsandbytes |
|
- load_in_8bit: False |
|
- load_in_4bit: True |
|
- llm_int8_threshold: 6.0 |
|
- llm_int8_skip_modules: None |
|
- llm_int8_enable_fp32_cpu_offload: False |
|
- llm_int8_has_fp16_weight: False |
|
- bnb_4bit_quant_type: nf4 |
|
- bnb_4bit_use_double_quant: True |
|
- bnb_4bit_compute_dtype: bfloat16 |
|
### Framework versions |
|
|
|
|
|
- PEFT 0.6.0.dev0 |
|
|
