text-correction-en-small

This model is a fine-tuned version of google/flan-t5-small on the small portion of this dataset.

Model description

This model is fine-tuned on the task of text correction: it fixes English text with the wrong grammar, punctuation and order of words. Significantly larger model that supports Russian language as well can be found here.

Training and evaluation data

Around 10% of the data from this dataset was used.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 1

Framework versions

• Transformers 4.41.0
• Pytorch 2.2.2
• Datasets 2.19.1
• Tokenizers 0.19.1