File size: 5,600 Bytes
4cc5389 479ca8b 4cc5389 479ca8b 4cc5389 0ed06e2 48e5481 479ca8b 48e5481 479ca8b 4cc5389 479ca8b 4cc5389 479ca8b 4cc5389 5bce316 81a3d24 5bce316 4cc5389 479ca8b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 |
---
license: mit
base_model: microsoft/Florence-2-large-ft
tags:
- image-text-to-text
- generated_from_trainer
model-index:
- name: Florence-2-large-TableDetection
results: []
datasets:
- ucsahin/pubtables-detection-1500-samples
pipeline_tag: image-text-to-text
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# Florence-2-large-TableDetection
This model is a fine-tuned version of [microsoft/Florence-2-large-ft](https://huggingface.co/microsoft/Florence-2-large-ft) on [ucsahin/pubtables-detection-1500-samples](https://huggingface.co/datasets/ucsahin/pubtables-detection-1500-samples) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.7601
[microsoft/Florence-2-large-ft](https://huggingface.co/microsoft/Florence-2-large-ft) can detect various objects in zero-shot setting with the task prompt "\<OD\>". Please check [Florence-2-large sample inference](https://huggingface.co/microsoft/Florence-2-large/blob/main/sample_inference.ipynb) for how to use Florence-2 model in inference. However, the ft-base model is not able to detect tables on a given image.
The following Colab notebook showcases how you can finetune the model with your custom data to detect objects.
[Florence2-Object Detection-Finetuning-HF-Trainer.ipynb](https://colab.research.google.com/drive/1Y8GVjwzBIgfmfD3ZypDX5H1JA_VG0YDL?usp=sharing)
## Model description
- This model is a multimodal language model fine-tuned for the task of detecting tables in images given textual prompts. The model utilizes a combination of image and text inputs to predict bounding boxes around tables within the provided images.
- The primary purpose of this model is to assist in automating the process of table detection within images. It can be utilized in various applications such as document processing, data extraction, and image analysis, where identifying tables within images is essential.
## How to Get Started with the Model
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
In Transformers, you can load the model and inference as follows: (Note that ```trust_remote_code=True``` is needed to run the model. It will only download the external custom codes from the original [HuggingFaceM4/Florence-2-DocVQA](https://huggingface.co/HuggingFaceM4/Florence-2-DocVQA).)
```python
from transformers import AutoProcessor, AutoModelForCausalLM
import matplotlib.pyplot as plt
import matplotlib.patches as patches
model_id = "ucsahin/Florence-2-large-TableDetection"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, device_map="cuda") # load the model on GPU
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
def run_example(task_prompt, image, max_new_tokens=128):
prompt = task_prompt
inputs = processor(text=prompt, images=image, return_tensors="pt")
generated_ids = model.generate(
input_ids=inputs["input_ids"].cuda(),
pixel_values=inputs["pixel_values"].cuda(),
max_new_tokens=max_new_tokens,
early_stopping=False,
do_sample=False,
num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
parsed_answer = processor.post_process_generation(
generated_text,
task=task_prompt,
image_size=(image.width, image.height)
)
return parsed_answer
def plot_bbox(image, data):
# Create a figure and axes
fig, ax = plt.subplots()
# Display the image
ax.imshow(image)
# Plot each bounding box
for bbox, label in zip(data['bboxes'], data['labels']):
# Unpack the bounding box coordinates
x1, y1, x2, y2 = bbox
# Create a Rectangle patch
rect = patches.Rectangle((x1, y1), x2-x1, y2-y1, linewidth=1, edgecolor='r', facecolor='none')
# Add the rectangle to the Axes
ax.add_patch(rect)
# Annotate the label
plt.text(x1, y1, label, color='white', fontsize=8, bbox=dict(facecolor='red', alpha=0.5))
# Remove the axis ticks and labels
ax.axis('off')
# Show the plot
plt.show()
########### Inference
from datasets import load_dataset
dataset = load_dataset("ucsahin/pubtables-detection-1500-samples")
example_id = 5
image = dataset["train"][example_id]["image"]
parsed_answer = run_example("<OD>", image=image)
plot_bbox(image, parsed_answer["<OD>"])
```
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.3199 | 1.0 | 169 | 1.0372 |
| 0.7922 | 2.0 | 338 | 0.9169 |
| 0.6824 | 3.0 | 507 | 0.8411 |
| 0.6109 | 4.0 | 676 | 0.8168 |
| 0.5752 | 5.0 | 845 | 0.7915 |
| 0.5605 | 6.0 | 1014 | 0.7862 |
| 0.5291 | 7.0 | 1183 | 0.7740 |
| 0.517 | 8.0 | 1352 | 0.7683 |
| 0.5139 | 9.0 | 1521 | 0.7642 |
| 0.5005 | 10.0 | 1690 | 0.7601 |
### Framework versions
- Transformers 4.42.0.dev0
- Pytorch 2.3.0+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1 |