mBART-TextSimp-LT-BatchSize4-lr5e-5

This model is a fine-tuned version of facebook/mbart-large-50 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0720
• Rouge1: 0.7898
• Rouge2: 0.643
• Rougel: 0.783
• Sacrebleu: 57.6148
• Gen Len: 33.6014

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 8

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Sacrebleu	Gen Len
0.407	1.0	209	0.1938	0.6481	0.4813	0.6379	42.027	33.6014
0.8377	2.0	418	0.1076	0.6446	0.4765	0.632	40.6092	33.7852
0.0589	3.0	627	0.0561	0.7659	0.6056	0.7581	51.836	33.6014
0.0237	4.0	836	0.0551	0.7816	0.6292	0.774	54.6775	33.6014
0.009	5.0	1045	0.0598	0.78	0.628	0.7723	54.4212	33.6014
0.0059	6.0	1254	0.0648	0.7876	0.6424	0.7805	56.5662	33.6014
0.003	7.0	1463	0.0694	0.7883	0.6405	0.781	57.3259	33.6014
0.0013	8.0	1672	0.0720	0.7898	0.643	0.783	57.6148	33.6014

Framework versions

• Transformers 4.33.0
• Pytorch 2.1.2+cu121
• Datasets 2.14.4
• Tokenizers 0.13.3