deberta-v3-large-survey-related_passage_consistency-rater-gpt4

This model is a fine-tuned version of microsoft/deberta-v3-large on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.2162
• Krippendorff: 0.7977
• Spearman: 0.8143
• Absolute Agreement: 0.9548
• Agreement Within One: 0.9786

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1000
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Krippendorff	Spearman	Absolute Agreement	Agreement Within One
No log	1.0	50	2.0598	-0.7589	nan	0.0139	0.9861
No log	2.0	100	2.0443	-0.7589	nan	0.0139	0.9861
No log	3.0	150	2.0270	-0.7589	nan	0.0139	0.9861
No log	4.0	200	1.8726	-0.2279	nan	0.4167	0.8333
No log	5.0	250	2.4468	-0.2279	nan	0.4167	0.8333
No log	6.0	300	2.6542	-0.2279	nan	0.4167	0.8333
No log	7.0	350	2.9210	-0.2279	nan	0.4167	0.8333
No log	8.0	400	2.8895	-0.2279	nan	0.4167	0.8333
No log	9.0	450	3.0362	-0.2279	nan	0.4167	0.8333
0.9688	10.0	500	3.2257	-0.1554	-0.0528	0.4028	0.8333
0.9688	11.0	550	3.4561	-0.2279	nan	0.4167	0.8333
0.9688	12.0	600	3.6475	-0.2279	nan	0.4167	0.8333
0.9688	13.0	650	4.0143	-0.2279	nan	0.4167	0.8333
0.9688	14.0	700	3.9822	-0.2279	nan	0.4167	0.8333
0.9688	15.0	750	3.7748	-0.2279	nan	0.4167	0.8333
0.9688	16.0	800	4.2108	-0.2279	nan	0.4167	0.8333
0.9688	17.0	850	4.3713	-0.2279	nan	0.4167	0.8333
0.9688	18.0	900	4.5353	-0.2279	nan	0.4167	0.8333
0.9688	19.0	950	4.6441	-0.2279	nan	0.4167	0.8333
0.1831	20.0	1000	4.2640	-0.2279	nan	0.4167	0.8333

Framework versions

• Transformers 4.26.0
• Pytorch 1.13.1
• Datasets 2.10.1
• Tokenizers 0.12.1