Hugging Face's logo

hsarfraz
/
irs-tax-form-1040-2023-doc-parser

Image-Text-to-Text
Transformers
Safetensors
vision-encoder-decoder
donut
parser
irs
tax
document AI
1040
Inference Endpoints
Model card Files Community
irs-tax-form-1040-2023-doc-parser
File size: 4,263 Bytes 
176ef1e
3e0a6ee
cd79eab
176ef1e
3e0a6ee
176ef1e
 
0e770bf
3e0a6ee
176ef1e
 
3e0a6ee
 
176ef1e
3e0a6ee
176ef1e
c269efd
 
 
2cdc2be
 
 
 
 
176ef1e
2cdc2be
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c269efd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
176ef1e
c269efd
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
 
---
license: mit
base_model: naver-clova-ix/donut-base
library_name: transformers
tags: ['donut','parser','irs','tax','document AI','1040']
---

# Donut - fine-tuned for US IRS Form 1040 (2023) data parsing and extraction
This donut model has been fine-tuned to parse and extract data from IRS (US) tax form 1040 (year 2023). It performs OCR and returns extracted data in JSON format using zero shot prompt. 


## Model Details & Description
The base model is ['naver-clova-ix/donut-base'][base], the model is finetuned for data parsing and extraction. The added_tokens.json file lists all the labels that can be extracted.   

For inference use image size width: 1536 px and height: 1536 px

# How to Get Started with the Model
Use the code below to get started with the model.

```python
from transformers import DonutProcessor, VisionEncoderDecoderModel
from PIL import Image
import torch
import re

model_name = 'hsarfraz/irs-tax-form-1040-2023-doc-parser'

processor = DonutProcessor.from_pretrained(model_name)
model = VisionEncoderDecoderModel.from_pretrained(model_name)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

image_name = 'replace with name of the form 1040 (2023) image file '

img = Image.open(image_name)
new_width = 1536
new_height = 1536

# resize input image to finetuned images size  
img = img.resize((new_width, new_height), Image.LANCZOS)   

pixel_values = processor(img.convert("RGB"), return_tensors="pt").pixel_values
pixel_values = pixel_values.to(device)

# prompt 
task_prompt = "<s_cord-v2>"
decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt")["input_ids"]
decoder_input_ids = decoder_input_ids.to(device)

outputs = model.generate(pixel_values,decoder_input_ids=decoder_input_ids,
                               max_length=model.decoder.config.max_position_embeddings,
                               early_stopping=True,
                               pad_token_id=processor.tokenizer.pad_token_id,
                               eos_token_id=processor.tokenizer.eos_token_id,
                               use_cache=True,
                               num_beams=1,
                               bad_words_ids=[[processor.tokenizer.unk_token_id]],
                               return_dict_in_generate=True,
                            #    output_scores=True,
                               )


sequence = processor.batch_decode(outputs.sequences)[0]
sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
output_json = processor.token2json(sequence)

print('----------------------------------')
print('--- Parsed data in json format ---')
print('----------------------------------')
print(output_json)

```
# FAKE Synthetic Form 1040 (2023) for illustration purposes only

![FAKE 1040 form for illustration purposes][fake_doc]

# Example of json output (based on FAKE 1040 form)

```json
{
    "lbl_0_03": "Michael Evans",
    "lbl_0_04": "Caldwell",
    "lbl_0_05": "741-52-5353",
    "lbl_0_06": "None",
    "lbl_0_07": "None",
    "lbl_0_08": "None",
    "lbl_0_09": "289 Blackwell Land Suite 380 New Tiffany, NH 07548",
    "lbl_0_11": "East Amandaport",
    "lbl_0_12": "VI",
    "lbl_0_13": "47832",
    "lbl_0_14": "None",
    "lbl_0_15": "None",
    "lbl_0_16": "25677",
    "lbl_0_55": "385321.36",
    "lbl_0_56": "None",
    "lbl_0_57": "None",
    "lbl_0_58": "None",
    "lbl_0_59": "None",
    "lbl_0_60": "None",
    "lbl_0_61": "None",
    "lbl_0_62": "None",
    "lbl_0_63": "None",
    "lbl_0_67": "None",
    "lbl_0_68": "481161.23",
    "lbl_0_69": "None",
    "lbl_0_70": "None",
    "lbl_0_71": "None",
    "lbl_0_72": "749100.68",
    "lbl_0_73": "418381-6",
    "lbl_0_74": "None",
    "lbl_0_77": "755042.64",
    "lbl_0_78": "None",
    "lbl_0_79": "560928.32",
    "lbl_0_80": "493913.73",
    "lbl_0_81": "None",
    "lbl_0_82": "738597.72",
    "lbl_0_83": "34990.46"
}
```



[base]: https://huggingface.co/naver-clova-ix/donut-base 
[fake_doc]: https://huggingface.co/hsarfraz/irs-tax-form-1040-2023-doc-parser/blob/main/fake_synthetic_form_1040_example.png