mistral-llm-recipes-en-ja-continuous-pretrained-v1-dev-finetune-docs-dpo-lora-debug

This model is a fine-tuned version of skim-wmt24/mistral-llm-recipes-en-ja-continuous-pretrained-v1-dev-finetune-docs-hf on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0000
• Rewards/chosen: 2.2801
• Rewards/rejected: -39.1533
• Rewards/accuracies: 1.0
• Rewards/margins: 41.4334
• Logps/rejected: -445.0770
• Logps/chosen: -359.7921
• Logits/rejected: -1.3840
• Logits/chosen: -0.5887

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.0	0.2107	100	0.0000	2.3145	-32.8393	1.0	35.1539	-381.9371	-359.4480	-1.3607	-0.5569
0.0	0.4215	200	0.0000	2.3217	-36.0030	1.0	38.3247	-413.5737	-359.3759	-1.3774	-0.5735
0.0	0.6322	300	0.0000	2.2793	-38.2847	1.0	40.5641	-436.3912	-359.8001	-1.3815	-0.5881
0.0	0.8430	400	0.0000	2.2847	-38.6597	1.0	40.9444	-440.1409	-359.7462	-1.3831	-0.5885
0.0	1.0537	500	0.0000	2.2864	-38.7575	1.0	41.0438	-441.1187	-359.7298	-1.3832	-0.5886
0.0	1.2645	600	0.0000	2.2828	-38.9276	1.0	41.2104	-442.8197	-359.7650	-1.3838	-0.5891
0.0	1.4752	700	0.0000	2.2807	-38.9749	1.0	41.2556	-443.2929	-359.7865	-1.3838	-0.5890
0.0	1.6860	800	0.0000	2.2817	-39.1524	1.0	41.4341	-445.0674	-359.7761	-1.3833	-0.5885
0.0	1.8967	900	0.0000	2.2829	-39.1605	1.0	41.4434	-445.1483	-359.7638	-1.3833	-0.5887
0.0	2.1075	1000	0.0000	2.2801	-39.1533	1.0	41.4334	-445.0770	-359.7921	-1.3840	-0.5887

Framework versions

• PEFT 0.11.1
• Transformers 4.41.2
• Pytorch 2.3.0a0+ebedce2
• Datasets 2.20.0
• Tokenizers 0.19.1