File size: 2,675 Bytes
1266055 49be3f0 1266055 4f1065d 1266055 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
license: other
tags:
- llama-2
---
# Model Card: Pygmalion LRP Grad L2 7B
This model uses [Pygmalion 2 7B](https://huggingface.co/PygmalionAI/pygmalion-2-7b) as a base and merged with LimaRP v1(52%) Lora customized with Metharme format
This merge of Lora with Model was done with this [script](https://github.com/zarakiquemparte/zaraki-tools/blob/main/apply-lora-weight-ltl.py)
- Credits to [Suikamelon](https://huggingface.co/lemonilia) for the LimaRP dataset
- Credits to [Pygmalion AI](https://huggingface.co/PygmalionAI) for the base model
## Weights of Lora merge:
```
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.5,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
```
## Prompting
The model has been trained on prompts using three different roles, which are denoted by the following tokens: `<|system|>`, `<|user|>` and `<|model|>`.
The `<|system|>` prompt can be used to inject out-of-channel information behind the scenes, while the `<|user|>` prompt should be used to indicate user input.
The `<|model|>` token should then be used to indicate that the model should generate a response. These tokens can happen multiple times and be chained up to
form a conversation history.
### Prompting example
The system prompt has been designed to allow the model to "enter" various modes and dictate the reply length. Here's an example:
```
<|system|>Enter RP mode. Pretend to be {{char}} whose persona follows:
{{persona}}
You shall reply to the user while staying in character, and generate long responses.
```
## Bias, Risks, and Limitations
The intended use-case for this model is fictional writing for entertainment purposes. Any other sort of usage is out of scope.
As such, it was **not** fine-tuned to be safe and harmless: the base model _and_ this fine-tune have been trained on data known to contain profanity and texts that
are lewd or otherwise offensive. It may produce socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.
Outputs might often be factually wrong or misleading.
## Training Details
This model use LimaRP by [Suikamelon](https://huggingface.co/lemonilia) dataset converted to metharme prompt format trained using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) and the lora merge was applied in the tool mentioned above
## Training Hyperparameters
```
load_in_8bit: true
adapter: lora
lora_r: 8
lora_alpha: 16
lora_dropout: 0.01
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 3
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.000065
bf16: true
tf32: true
```
## Environmental Impact
Finetuning the LimaRP Lora on 1 x NVIDIA L40 takes about 1h45m
|