openhermes-mistral-gptq-dpo

This model is a fine-tuned version of TheBloke/OpenHermes-2-Mistral-7B-GPTQ on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8689
• Rewards/chosen: -1.0765
• Rewards/rejected: -0.9664
• Rewards/accuracies: 0.4800
• Rewards/margins: -0.1101
• Logps/rejected: -139.8855
• Logps/chosen: -135.5758
• Logits/rejected: -2.2110
• Logits/chosen: -2.2313

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• training_steps: 50

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6842	0.01	10	0.7768	-0.4985	-0.4411	0.5750	-0.0574	-134.6327	-129.7961	-2.2478	-2.2667
0.8539	0.02	20	2.0287	-9.4492	-8.7234	0.4800	-0.7258	-217.4559	-219.3030	-1.5781	-1.5859
0.9433	0.03	30	1.6915	-7.3191	-6.8845	0.5	-0.4346	-199.0667	-198.0014	-1.6865	-1.6973
0.6295	0.04	40	0.9729	-1.6986	-1.5214	0.5200	-0.1772	-145.4362	-141.7968	-2.1859	-2.2057
0.8681	0.05	50	0.8689	-1.0765	-0.9664	0.4800	-0.1101	-139.8855	-135.5758	-2.2110	-2.2313

Framework versions

• Transformers 4.34.1
• Pytorch 2.1.0+cu121
• Datasets 2.14.6
• Tokenizers 0.14.1