|
--- |
|
license: mit |
|
base_model: naver-clova-ix/donut-base |
|
library_name: transformers |
|
tags: |
|
- donut |
|
- parser |
|
- irs |
|
- tax |
|
- document AI |
|
- '1040' |
|
pipeline_tag: image-text-to-text |
|
--- |
|
|
|
# Donut - fine-tuned for US IRS Form 1040 (2023) data parsing and extraction |
|
This donut model has been fine-tuned to parse and extract data from IRS (US) tax form 1040 (year 2023). It performs OCR and returns extracted data in JSON format using zero shot prompt. |
|
|
|
|
|
## Model Details & Description |
|
The base model is ['naver-clova-ix/donut-base'][base], the model is finetuned for data parsing and extraction. The added_tokens.json file lists all the labels that can be extracted. |
|
|
|
For inference use image size width: 1536 px and height: 1536 px |
|
|
|
# How to Get Started with the Model |
|
Use the code below to get started with the model. |
|
|
|
```python |
|
from transformers import DonutProcessor, VisionEncoderDecoderModel |
|
from PIL import Image |
|
import torch |
|
import re |
|
|
|
model_name = 'hsarfraz/irs-tax-form-1040-2023-doc-parser' |
|
|
|
processor = DonutProcessor.from_pretrained(model_name) |
|
model = VisionEncoderDecoderModel.from_pretrained(model_name) |
|
|
|
device = "cuda" if torch.cuda.is_available() else "cpu" |
|
model.to(device) |
|
model.eval() |
|
|
|
image_name = 'replace with name of the form 1040 (2023) image file ' |
|
|
|
img = Image.open(image_name) |
|
new_width = 1536 |
|
new_height = 1536 |
|
|
|
# resize input image to finetuned images size |
|
img = img.resize((new_width, new_height), Image.LANCZOS) |
|
|
|
pixel_values = processor(img.convert("RGB"), return_tensors="pt").pixel_values |
|
pixel_values = pixel_values.to(device) |
|
|
|
# prompt |
|
task_prompt = "<s_cord-v2>" |
|
decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt")["input_ids"] |
|
decoder_input_ids = decoder_input_ids.to(device) |
|
|
|
outputs = model.generate(pixel_values,decoder_input_ids=decoder_input_ids, |
|
max_length=model.decoder.config.max_position_embeddings, |
|
early_stopping=True, |
|
pad_token_id=processor.tokenizer.pad_token_id, |
|
eos_token_id=processor.tokenizer.eos_token_id, |
|
use_cache=True, |
|
num_beams=1, |
|
bad_words_ids=[[processor.tokenizer.unk_token_id]], |
|
return_dict_in_generate=True, |
|
# output_scores=True, |
|
) |
|
|
|
|
|
sequence = processor.batch_decode(outputs.sequences)[0] |
|
sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "") |
|
sequence = re.sub(r"<.*?>", "", sequence, count=1).strip() # remove first task start token |
|
output_json = processor.token2json(sequence) |
|
|
|
print('----------------------------------') |
|
print('--- Parsed data in json format ---') |
|
print('----------------------------------') |
|
print(output_json) |
|
|
|
``` |
|
# FAKE Synthetic Form 1040 (2023) for illustration purposes only |
|
|
|
<div align="left"> |
|
<img width="800" alt="FAKE 1040 form for illustration purposes" src="https://huggingface.co/hsarfraz/irs-tax-form-1040-2023-doc-parser/resolve/main/fake_synthetic_form_1040_example.png"> |
|
</div> |
|
|
|
# Example of json output (based on FAKE 1040 form) |
|
|
|
```json |
|
{ |
|
"lbl_0_03": "Michael Evans", |
|
"lbl_0_04": "Caldwell", |
|
"lbl_0_05": "741-52-5353", |
|
"lbl_0_06": "None", |
|
"lbl_0_07": "None", |
|
"lbl_0_08": "None", |
|
"lbl_0_09": "289 Blackwell Land Suite 380 New Tiffany, NH 07548", |
|
"lbl_0_11": "East Amandaport", |
|
"lbl_0_12": "VI", |
|
"lbl_0_13": "47832", |
|
"lbl_0_14": "None", |
|
"lbl_0_15": "None", |
|
"lbl_0_16": "25677", |
|
"lbl_0_55": "385321.36", |
|
"lbl_0_56": "None", |
|
"lbl_0_57": "None", |
|
"lbl_0_58": "None", |
|
"lbl_0_59": "None", |
|
"lbl_0_60": "None", |
|
"lbl_0_61": "None", |
|
"lbl_0_62": "None", |
|
"lbl_0_63": "None", |
|
"lbl_0_67": "None", |
|
"lbl_0_68": "481161.23", |
|
"lbl_0_69": "None", |
|
"lbl_0_70": "None", |
|
"lbl_0_71": "None", |
|
"lbl_0_72": "749100.68", |
|
"lbl_0_73": "418381-6", |
|
"lbl_0_74": "None", |
|
"lbl_0_77": "755042.64", |
|
"lbl_0_78": "None", |
|
"lbl_0_79": "560928.32", |
|
"lbl_0_80": "493913.73", |
|
"lbl_0_81": "None", |
|
"lbl_0_82": "738597.72", |
|
"lbl_0_83": "34990.46" |
|
} |
|
``` |
|
|
|
|
|
|
|
[base]: https://huggingface.co/naver-clova-ix/donut-base |
|
|
