mobilebert_sa_pre-training-complete

This model is a fine-tuned version of google/mobilebert-uncased on the wikitext wikitext-103-raw-v1 dataset. It achieves the following results on the evaluation set:

• Loss: 1.3239
• Accuracy: 0.7162

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 10
• distributed_type: multi-GPU
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• training_steps: 300000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.6028	1.0	7145	1.4525	0.6935
1.5524	2.0	14290	1.4375	0.6993
1.5323	3.0	21435	1.4194	0.6993
1.5191	4.0	28580	1.4110	0.7027
1.5025	5.0	35725	1.4168	0.7014
1.4902	6.0	42870	1.3931	0.7012
1.4813	7.0	50015	1.3738	0.7057
1.4751	8.0	57160	1.4237	0.6996
1.4689	9.0	64305	1.3969	0.7047
1.4626	10.0	71450	1.3916	0.7068
1.4566	11.0	78595	1.3686	0.7072
1.451	12.0	85740	1.3811	0.7060
1.4478	13.0	92885	1.3598	0.7092
1.4441	14.0	100030	1.3790	0.7054
1.4379	15.0	107175	1.3794	0.7066
1.4353	16.0	114320	1.3609	0.7102
1.43	17.0	121465	1.3685	0.7083
1.4278	18.0	128610	1.3953	0.7036
1.4219	19.0	135755	1.3756	0.7085
1.4197	20.0	142900	1.3597	0.7090
1.4169	21.0	150045	1.3673	0.7061
1.4146	22.0	157190	1.3753	0.7073
1.4109	23.0	164335	1.3696	0.7082
1.4073	24.0	171480	1.3563	0.7092
1.4054	25.0	178625	1.3712	0.7103
1.402	26.0	185770	1.3528	0.7113
1.4001	27.0	192915	1.3367	0.7123
1.397	28.0	200060	1.3508	0.7118
1.3955	29.0	207205	1.3572	0.7117
1.3937	30.0	214350	1.3566	0.7095
1.3901	31.0	221495	1.3515	0.7117
1.3874	32.0	228640	1.3445	0.7118
1.386	33.0	235785	1.3611	0.7097
1.3833	34.0	242930	1.3502	0.7087
1.3822	35.0	250075	1.3657	0.7108
1.3797	36.0	257220	1.3576	0.7108
1.3793	37.0	264365	1.3472	0.7106
1.3763	38.0	271510	1.3323	0.7156
1.3762	39.0	278655	1.3325	0.7145
1.3748	40.0	285800	1.3243	0.7138
1.3733	41.0	292945	1.3218	0.7170
1.3722	41.99	300000	1.3074	0.7186

Framework versions

• Transformers 4.26.0
• Pytorch 1.14.0a0+410ce96
• Datasets 2.9.0
• Tokenizers 0.13.2