distilbert-base-uncased-distilled-clinc

This model is a fine-tuned version of distilbert-base-uncased on the clinc_oos dataset. It achieves the following results on the evaluation set:

• Loss: 0.3215
• Accuracy: 0.9458

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 384
• eval_batch_size: 384
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 60

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	1.0	40	3.4217	0.1958
No log	2.0	80	2.9574	0.6029
No log	3.0	120	2.4741	0.7294
No log	4.0	160	2.0307	0.7926
No log	5.0	200	1.6384	0.8339
No log	6.0	240	1.3096	0.8658
No log	7.0	280	1.0421	0.8974
2.2117	8.0	320	0.8340	0.9126
2.2117	9.0	360	0.6804	0.9216
2.2117	10.0	400	0.5784	0.93
2.2117	11.0	440	0.5109	0.9348
2.2117	12.0	480	0.4665	0.9387
2.2117	13.0	520	0.4352	0.9387
2.2117	14.0	560	0.4149	0.9406
2.2117	15.0	600	0.3973	0.9439
0.4743	16.0	640	0.3867	0.9429
0.4743	17.0	680	0.3793	0.9426
0.4743	18.0	720	0.3732	0.9435
0.4743	19.0	760	0.3649	0.9445
0.4743	20.0	800	0.3583	0.9455
0.4743	21.0	840	0.3577	0.9448
0.4743	22.0	880	0.3520	0.9432
0.4743	23.0	920	0.3488	0.9465
0.2577	24.0	960	0.3470	0.9458
0.2577	25.0	1000	0.3434	0.9471
0.2577	26.0	1040	0.3427	0.9465
0.2577	27.0	1080	0.3407	0.9452
0.2577	28.0	1120	0.3389	0.9461
0.2577	29.0	1160	0.3377	0.9465
0.2577	30.0	1200	0.3380	0.9458
0.2577	31.0	1240	0.3338	0.9461
0.219	32.0	1280	0.3348	0.9465
0.219	33.0	1320	0.3334	0.9461
0.219	34.0	1360	0.3314	0.9468
0.219	35.0	1400	0.3290	0.9481
0.219	36.0	1440	0.3292	0.9477
0.219	37.0	1480	0.3296	0.9458
0.219	38.0	1520	0.3290	0.9461
0.219	39.0	1560	0.3267	0.9458
0.2039	40.0	1600	0.3281	0.9458
0.2039	41.0	1640	0.3272	0.9458
0.2039	42.0	1680	0.3245	0.9468
0.2039	43.0	1720	0.3260	0.9461
0.2039	44.0	1760	0.3244	0.9455
0.2039	45.0	1800	0.3245	0.9458
0.2039	46.0	1840	0.3243	0.9455
0.2039	47.0	1880	0.3235	0.9455
0.1965	48.0	1920	0.3228	0.9455
0.1965	49.0	1960	0.3232	0.9465
0.1965	50.0	2000	0.3228	0.9461
0.1965	51.0	2040	0.3232	0.9468
0.1965	52.0	2080	0.3220	0.9461
0.1965	53.0	2120	0.3211	0.9465
0.1965	54.0	2160	0.3217	0.9458
0.1965	55.0	2200	0.3219	0.9465
0.1929	56.0	2240	0.3216	0.9461
0.1929	57.0	2280	0.3218	0.9465
0.1929	58.0	2320	0.3212	0.9458
0.1929	59.0	2360	0.3214	0.9455
0.1929	60.0	2400	0.3215	0.9458

Framework versions

• Transformers 4.34.0
• Pytorch 2.0.1
• Datasets 2.14.5
• Tokenizers 0.14.1