microsoft

/

table-transformer-detection

like 213

Dear All,

After reading all the threads available in the internet I am using below script to fine tune table-transformer-detection
https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb

I have
Replace:

processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")

for:

processor = DetrImageProcessor()

also
Replace:

DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50",
revision="no_timm",
num_labels=len(id2label),
ignore_mismatched_sizes=True)
for:

TableTransformerForObjectDetection.from_pretrained(
"microsoft/table-transformer-structure-recognition",
ignore_mismatched_sizes=True,
)

After finetuning the model using Trainer(max_steps=3000, gradient_clip_val=0.1) I am getting very low accuracy below
IoU metric: bbox
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.334
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.539
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.356
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.334
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.223
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.468
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.487
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.487

My dataset size is:
Number of training examples: 159
Number of validation examples: 19

Any thoughts on this ?
P.S. I know about this project https://github.com/microsoft/table-transformer/ , I know how to finetune using this project , I also know about convert_table_transformer_original_pytorch_checkpoint_to_pytorch.py, present is transformers repo. But my question is why the above finetune is giving me such a low accuracy ? should I increase my dataset size or am I missing anything. I am using a proprietary dataset