---
license: other
base_model: microsoft/Orca-2-13b
tags:
- generated_from_trainer
model-index:
- name: Orca-2-13B-Pygmalion-LoRA
  results: []
datasets:
- PygmalionAI/PIPPA
language:
- en
---

++ This model's response was too short, so I re-trained it, check this out: https://huggingface.co/ricecake/Orca-2-13B-Pyg-and-Bluemoon

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
# Orca-2-13B-Pygmalion-LoRA

This LoRA adapter is a fine-tuned version of [microsoft/Orca-2-13b](https://huggingface.co/microsoft/Orca-2-13b) on the [PygmalionAI/PIPPA](https://huggingface.co/datasets/PygmalionAI/PIPPA) dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9190

## Model description

More information needed


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 2
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| No log        | 0.0   | 1     | 3.2585          |
| 1.9811        | 0.05  | 536   | 2.0113          |
| 1.9507        | 0.1   | 1072  | 1.9877          |
| 1.9576        | 0.15  | 1608  | 1.9766          |
| 1.9308        | 0.2   | 2144  | 1.9671          |
| 1.9193        | 0.25  | 2680  | 1.9597          |
| 1.8522        | 0.3   | 3216  | 1.9530          |
| 1.895         | 0.35  | 3752  | 1.9483          |
| 1.869         | 0.4   | 4288  | 1.9432          |
| 1.8664        | 0.45  | 4824  | 1.9383          |
| 1.8661        | 0.5   | 5360  | 1.9347          |
| 1.8576        | 0.55  | 5896  | 1.9337          |
| 1.8573        | 0.6   | 6432  | 1.9286          |
| 1.8665        | 0.65  | 6968  | 1.9280          |
| 1.8429        | 0.7   | 7504  | 1.9243          |
| 1.8621        | 0.75  | 8040  | 1.9221          |
| 1.8074        | 0.8   | 8576  | 1.9209          |
| 1.8199        | 0.85  | 9112  | 1.9202          |
| 1.8733        | 0.9   | 9648  | 1.9193          |
| 1.8387        | 0.95  | 10184 | 1.9190          |


### Framework versions

- Transformers 4.35.1
- Pytorch 2.1.0+cu121
- Datasets 2.14.7
- Tokenizers 0.14.1