zibajoon/20231109_layoutlm2_5k_20ep_Doc_NA

results

This model is a fine-tuned version of microsoft/layoutlmv2-base-uncased on the None dataset. It achieves the following results on the evaluation set:

• Loss: nan

Model description

This DocVQA model, built on the Layout LM v2 framework, represents an initial step in a series of experimental models aimed at document visual question answering. It's the "medium" version in a planned series, trained on a mid-sized dataset of 5k samples (split between training and test) over 20 epochs. The training setup was modest, employing mixed precision (fp16), with manageable batch sizes and a focused approach to learning rate adjustment (warmup steps and weight decay). Notably, this model was trained without external reporting tools, emphasizing internal evaluation. As the first iteration in a progressive series that will later include medium (5k samples) and large (50k samples) models, this version serves as a foundational experiment, setting the stage for more extensive and complex models in the future.

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.99	62	5.4841
No log	2.0	125	4.6253
No log	2.99	187	4.3093
No log	4.0	250	4.0361
No log	4.99	312	3.6892
No log	6.0	375	3.3862
No log	6.99	437	3.0017
4.3469	8.0	500	nan
4.3469	8.99	562	nan
4.3469	10.0	625	nan
4.3469	10.99	687	nan
4.3469	12.0	750	nan
4.3469	12.99	812	nan
4.3469	14.0	875	nan
4.3469	14.99	937	nan
21709.916	16.0	1000	nan
21709.916	16.99	1062	nan
21709.916	18.0	1125	nan
21709.916	18.99	1187	nan
21709.916	19.84	1240	nan

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.1+cu118
• Datasets 2.10.1
• Tokenizers 0.14.1