deberta-v3-large-survey-fluency-rater-half-gpt4

This model is a fine-tuned version of microsoft/deberta-v3-large on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4343
• Krippendorff: 0.9058
• Spearman: 0.9547
• Absolute Agreement: 0.8474
• Agreement Within One: 0.9471

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1000
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Krippendorff	Spearman	Absolute Agreement	Agreement Within One
No log	1.0	52	1.9138	-0.3957	nan	0.0833	0.7778
No log	2.0	104	1.9217	-0.3957	nan	0.0833	0.7778
No log	3.0	156	1.9585	-0.3957	nan	0.0833	0.7778
No log	4.0	208	2.3977	-0.3957	nan	0.0833	0.7778
No log	5.0	260	2.8616	-0.3957	nan	0.0833	0.7778
No log	6.0	312	2.6726	-0.3957	nan	0.0833	0.7778
No log	7.0	364	2.7845	-0.3957	nan	0.0833	0.7778
No log	8.0	416	2.6765	-0.3957	nan	0.0833	0.7778
No log	9.0	468	3.4009	-0.3957	nan	0.0833	0.7778
1.2312	10.0	520	2.9620	-0.3823	-0.0500	0.0833	0.7778
1.2312	11.0	572	3.1665	-0.3823	-0.0500	0.0833	0.7778
1.2312	12.0	624	2.3547	-0.1219	-0.0203	0.1111	0.8056
1.2312	13.0	676	3.1385	-0.2794	-0.1482	0.1111	0.8056
1.2312	14.0	728	2.3297	-0.2464	-0.1777	0.2778	0.8056
1.2312	15.0	780	3.0634	-0.1121	0.0026	0.1389	0.8056
1.2312	16.0	832	3.3036	0.1503	0.0999	0.1389	0.8611
1.2312	17.0	884	3.1543	0.1079	0.0845	0.1389	0.8333
1.2312	18.0	936	3.4091	-0.0607	0.0699	0.25	0.8056
1.2312	19.0	988	3.2090	0.2886	0.1196	0.1389	0.8611
0.67	20.0	1040	3.6203	0.2271	0.2339	0.25	0.8333

Framework versions

• Transformers 4.26.0
• Pytorch 1.13.1
• Datasets 2.10.1
• Tokenizers 0.12.1