I messed up on this model and forgot to add special tokens for <|im_start|> & <|im_end|> - but it still seems to work fine. If you're having any issues let me know in the Discussions tab and I can retrain it.
This model is usable, but you can get better results with the fixed latest model! Check out the latest model instead of using this one!
A tiny 1B model that refuses basically anything you ask it! Trained on the refusal dataset. Prompt format is ChatML.
Training results:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.4352 | 0.0580 | 1 | 2.4462 |
| 1.5742 | 0.5217 | 9 | 1.4303 |
| 1.5084 | 1.0435 | 18 | 1.3672 |
| 1.0814 | 1.5217 | 27 | 1.3483 |
| 1.1024 | 2.0435 | 36 | 1.3204 |
| 0.6554 | 2.5217 | 45 | 1.4286 |
| 0.6163 | 3.0435 | 54 | 1.4375 |
| 0.5058 | 3.5072 | 63 | 1.4908 |
Training hyperparemeters:
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 4
Base model: https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
- Downloads last month
- 6
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for mrfakename/refusal-old
Finetuned
this model