deberta-v3-large-survey-fluency-rater-gpt4

This model is a fine-tuned version of microsoft/deberta-v3-large on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.9754
• Krippendorff: -0.0424
• Spearman: 0.1432
• Absolute Agreement: 0.6897
• Agreement Within One: 0.7676

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1000
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Krippendorff	Spearman	Absolute Agreement	Agreement Within One
No log	1.0	50	1.9660	-0.7952	nan	0.0278	1.0
No log	2.0	100	1.9587	-0.7952	nan	0.0278	1.0
No log	3.0	150	1.9548	-0.7155	-0.3236	0.0278	0.9306
No log	4.0	200	2.0047	-0.7155	-0.3236	0.0278	0.9306
No log	5.0	250	2.2550	-0.3248	-0.3996	0.125	0.7361
No log	6.0	300	2.4123	-0.1335	-0.1046	0.25	0.7361
No log	7.0	350	2.4828	-0.1883	-0.2324	0.2083	0.7361
No log	8.0	400	2.4175	-0.3139	-0.4086	0.125	0.7361
No log	9.0	450	2.6742	-0.1335	-0.1046	0.25	0.7361
1.2917	10.0	500	2.8155	-0.2880	nan	0.2917	0.7222
1.2917	11.0	550	3.0146	-0.2880	nan	0.2917	0.7222
1.2917	12.0	600	3.0548	-0.1729	-0.1993	0.2222	0.7361
1.2917	13.0	650	3.3351	-0.2880	nan	0.2917	0.7222
1.2917	14.0	700	3.0936	-0.1883	-0.2324	0.2083	0.7361
1.2917	15.0	750	3.3908	-0.1335	-0.1046	0.25	0.7361
1.2917	16.0	800	2.3496	-0.375	-0.3916	0.0833	0.7778
1.2917	17.0	850	3.6256	-0.1729	-0.1993	0.2222	0.7361
1.2917	18.0	900	3.3170	-0.2037	-0.3121	0.1528	0.7361
1.2917	19.0	950	3.9354	-0.2369	-0.3285	0.1528	0.7361
0.7206	20.0	1000	3.6945	-0.1997	-0.2613	0.1944	0.7361

Framework versions

• Transformers 4.26.0
• Pytorch 1.13.1
• Datasets 2.10.1
• Tokenizers 0.12.1