S ynth NID : Synthetic Data to Improve End-to-end B angla Document Key Information Extraction (2023)

Abstract: End-to-end Document Key Information Extraction models require a lot of compute and labeled data to perform well on real datasets. This is particularly challenging for low-resource languages like Bangla where domain-specific multimodal document datasets are scarcely available. In this paper, we have introduced SynthNID, a system to generate domain-specific document image data for training OCR-less end-to-end Key Information Extraction systems. We show the generated data improves the performance of the extraction model on real datasets and the system is easily extendable to generate other types of scanned documents for a wide range of document understanding tasks. The code for generating synthetic data is available at https://github.com/dv66/synthnid

## introduction document key information extraction (kie) is a very crucial task to extract structured or semistructured information from printed documents and images @xcite @xcite . numerous user-facing applications nowadays require scanning and extracting information as key-value pairs from raw images of invoices, receipts, id cards etc. previously, these types of information extraction systems required a mandatory ocr engine and rule-based approaches in the pipeline. however ocr-based systems become error-prone easily because of the lack of contextual understanding @xcite . also, handpicked rule-based extraction systems cannot handle all possible transformations and variations that can be found in scanned documents. recently, a surge of pre-trained models has been witnessed in the area of document understanding. these models overcome the problems of ocr-powered key information extraction sys-* equal contribution tems by completely removing the need to apply character-level recognition and information aggregation. these models are essentially vision transformers which are pre-trained on huge datasets of scanned documents across multiple languages @xcite . these pre-trained models achieve state-of-the-art in various document understanding tasks on the particular languages they were pre-trained on. it is possible to fine-tune these models for downstream extraction tasks in other languages with sub-optimal performances. in the case of low-resource languages like bangla, it is a severe issue because there are almost no datasets for the document key information extraction task in bangla. although a number of works are found in the literature regarding bangla ocr and/or text detection systems @xcite @xcite , none of them perform end-to-end key information retrieval which is essential to retrieve necessary fields from the scanned document. collecting annotated data for document key information extraction is also quite expensive and time-consuming. to reduce labelling effort and cost of labelling, synthetic data is often used alongside real data to train models for various document understanding tasks @xcite . unfortunately, general-purpose synthetic image generators do not focus on key information extraction only but on general-purpose document understanding tasks. most of the state-of-the-art generators support only english or rich-resource corpus thus lowresource languages like bangla are completely ignored. also, the lack of availability of high-quality bangla corpus for key information extraction tasks is another reason for the absence of end-to-end models in this area. the end-to-end key information extraction model requires huge datasets with millions of samples @xcite and hundreds of gpu hours which also contributes to this issue. one option is to fine-tune the pre-trained multi-lingual end-to-end models on the target language e.g. bangla to perform key information extraction on real documents like the national id card of bangladesh, license plates etc. this approach is particularly useful when the whole document image is used as an information source and the target output is a structured data format like json. because then an extra field extraction or linking stage is no longer required to add to the extraction pipeline. in this work, we propose a system synthnid, to generate domain-specific synthetic data which improves the performance of end-to-end document key information extraction tasks when fine-tuned alongside real data. our primary focus for this work was to extract key values from the national id card of bangladesh which contains a mixture of english and bangla text in the document but using our approach a wide range of scanned documents can also be generated for key information extraction tasks. we demonstrate the effectiveness of the generated data by fine-tuning end-to-end key information extraction models. our synthetic data increases the model's performance in extracting key-value pairs from real datasets. ## related work document key information extraction is a widely studied task in the literature. most popular models incorporate the output of an ocr engine and learn to parse them from scanned documents @xcite . @xcite and @xcite have applied document key information extraction to various real-world applications. most of these approaches introduce a learning framework where the text is detected separately using an off-the-shelf ocr engine and a sequence model takes the input from the previous stage considering the text content and locality of the information. despite the convenience of end-to-end models for kie tasks, only a few are available in the literature like ocr-free document understanding transformer @xcite . this model takes the whole document image as an input and applies a visual attention mechanism to learn the output sequence which is essentially a key-value structure like json. however, the endto-end model variants are only pre-trained in chinese, japanese, korean and english. although there are a number of works present in literature regarding bangla ocr and a few in document understanding, almost none of them address end-to-end key information extraction on bangla scanned documents or multilingual scanned documents where bangla is present. bbocr is a scalable document ocr that employs a bangla text recognition model using synthetic datasets @xcite . badlad is a large multidomain bangla document layout analysis dataset which contains more than 33k manually labeled documents from a wide range of sources including books, magazines, newspapers etc. @xcite . @xcite improves document layout analysis performance leveraging mask r-cnn architectures. they show competitive results in segmenting bangla documents. most of the existing works in bangla document understanding have tackled problems like text extraction or layout analysis by segmenting the image components, whereas none of them considered extracting key information in a structured format which can be easily used by independent user-facing applications. in this effort, we have addressed this gap in the existing literature and aimed to solve the issue by new approaches to generate bangla synthetic documents for domain-specific kie tasks. ## experiments for our end-to-end model, we used the ocr-free document transformer @xcite which is the current state-of-the-art in kie. the model outperforms various other models like layoutlm @xcite , layoutlmv2 @xcite , lay-outxlm @xcite , spade @xcite , wyvern @xcite in document key information extraction tasks. the model is essentially a vision transformer @xcite which is pre-trained on a huge dataset containing real and synthetic data 13m in total. the real dataset iit-cdip @xcite ) contains around 11m samples of complex scanned english ## ethical considerations while developing our system we prioritized end users' privacy protection. the app was developed and used to collect data inside the organization. informed consent was obtained from the app users and stringent data anonymization measures were applied while using real data for testing the models' performances. while generating the synthetic data, every field is generated in a completely random strategy. our ethical framework was aimed to develop high-quality bangla kie models while protecting user privacy, maintaining transparency, and ensuring responsible data handling thus strengthening our commitment to conduct ethical ai research. ## conclusion in this paper, we have introduced a scheme to generate high-quality domain-specific synthetic data for the key information extraction task on bangla scanned documents. we have shown the synthetic data generated using our approach enhances the performance of end-to-end kie models. in future, we will investigate the areas where effective labelling strategies can be employed to learn good models with a low amount of data using active learning techniques.