muhtasham/bert-tiny-finetuned-finer-longer

bertiny-finetuned-finer-full

This model is a fine-tuned version of google/bert_uncased_L-2_H-128_A-2 on the 10% of finer-139 dataset for 40 epochs according to paper. It achieves the following results on the evaluation set:

• Loss: 0.0788
• Precision: 0.5554
• Recall: 0.5164
• F1: 0.5352
• Accuracy: 0.9887

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 40

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
0.0852	1.0	11255	0.0929	1.0	0.0001	0.0002	0.9843
0.08	2.0	22510	0.0840	0.4626	0.0730	0.1261	0.9851
0.0759	3.0	33765	0.0750	0.5113	0.2035	0.2912	0.9865
0.0569	4.0	45020	0.0673	0.4973	0.3281	0.3953	0.9872
0.0488	5.0	56275	0.0635	0.5289	0.3749	0.4388	0.9878
0.0422	6.0	67530	0.0606	0.5258	0.4068	0.4587	0.9880
0.0364	7.0	78785	0.0600	0.5588	0.4186	0.4787	0.9883
0.0307	8.0	90040	0.0589	0.5223	0.4916	0.5065	0.9883
0.0284	9.0	101295	0.0595	0.5588	0.4813	0.5171	0.9887
0.0255	10.0	112550	0.0597	0.5606	0.4944	0.5254	0.9888
0.0223	11.0	123805	0.0600	0.5533	0.4998	0.5252	0.9888
0.0228	12.0	135060	0.0608	0.5290	0.5228	0.5259	0.9885
0.0225	13.0	146315	0.0612	0.5480	0.5111	0.5289	0.9887
0.0204	14.0	157570	0.0634	0.5646	0.5120	0.5370	0.9890
0.0176	15.0	168825	0.0639	0.5611	0.5135	0.5363	0.9889
0.0167	16.0	180080	0.0647	0.5631	0.5120	0.5363	0.9888
0.0161	17.0	191335	0.0665	0.5607	0.5081	0.5331	0.9889
0.0145	18.0	202590	0.0673	0.5437	0.5280	0.5357	0.9887
0.0166	19.0	213845	0.0687	0.5722	0.5008	0.5341	0.9889
0.0155	20.0	225100	0.0685	0.5325	0.5337	0.5331	0.9885
0.0142	21.0	236355	0.0705	0.5626	0.5166	0.5386	0.9890
0.0127	22.0	247610	0.0694	0.5426	0.5358	0.5392	0.9887
0.0112	23.0	258865	0.0721	0.5591	0.5129	0.5351	0.9888
0.0123	24.0	270120	0.0733	0.5715	0.5081	0.5380	0.9889
0.0116	25.0	281375	0.0735	0.5621	0.5123	0.5361	0.9888
0.0112	26.0	292630	0.0739	0.5634	0.5181	0.5398	0.9889
0.0108	27.0	303885	0.0753	0.5548	0.5155	0.5344	0.9887
0.0125	28.0	315140	0.0746	0.5507	0.5221	0.5360	0.9886
0.0093	29.0	326395	0.0762	0.5602	0.5156	0.5370	0.9888
0.0094	30.0	337650	0.0762	0.5625	0.5157	0.5381	0.9889
0.0117	31.0	348905	0.0767	0.5519	0.5195	0.5352	0.9887
0.0091	32.0	360160	0.0772	0.5501	0.5198	0.5345	0.9887
0.0109	33.0	371415	0.0775	0.5635	0.5097	0.5353	0.9888
0.0094	34.0	382670	0.0776	0.5467	0.5216	0.5339	0.9887
0.009	35.0	393925	0.0782	0.5601	0.5139	0.5360	0.9889
0.0093	36.0	405180	0.0780	0.5568	0.5156	0.5354	0.9888
0.0087	37.0	416435	0.0783	0.5588	0.5143	0.5356	0.9888
0.009	38.0	427690	0.0785	0.5483	0.5178	0.5326	0.9887
0.0094	39.0	438945	0.0787	0.5541	0.5154	0.5340	0.9887
0.0088	40.0	450200	0.0788	0.5554	0.5164	0.5352	0.9887

Framework versions

• Transformers 4.20.1
• Pytorch 1.12.0+cu113
• Datasets 2.3.2
• Tokenizers 0.12.1