Language, OCR , Form Independent ( LOFI ) pipeline for Industrial Document Information Extraction (2024)

Abstract: This paper presents LOFI (Language, OCR, Form Independent), a pipeline for Document Information Extraction (DIE) in Low-Resource Language (LRL) business documents. LOFI pipeline solves language, Optical Character Recognition (OCR), and form dependencies through flexible model architecture, a token-level box split algorithm, and the SPADE decoder. Experiments on Korean and Japanese documents demonstrate high performance in Semantic Entity Recognition (SER) task without additional pre-training. The pipeline’s effectiveness is validated through real-world applications in insurance and tax-free declaration services, advancing DIE capabilities for diverse languages and document types in industrial settings.

## introduction many industries handle complex documents known as visually rich documents (vrds), containing text, tables, and figures. in real-world industry scenarios involving vrds, we should consider a process of semantic entity recognition (ser) @xcite to automate workflows. for example, in insurance claims processing, patient information and diagnostic details need to be extracted from medical reports submitted by customers. additionally, in accounting and tax filing processes, purchase information should be extracted from receipts or other tax documents. to address the automation demands of the industry, we face three main challenges: however, related research has not comprehensively addressed these three issues together. we have focused on considering these three challenges collectively in order to meet the automation demands of the industry. language independence: there's a lack of publicly available datasets and models that work with lrl, languages that are less used compared to english and chinese, such as korean and japanese. most vrd datasets, such as ephoie, funsd, and cord @xcite @xcite are primarily in english or chinese, and most open models (layoutlm, layoutlmv2, layoutlmv3, bros, geolayoutlm) @xcite @xcite are trained with open datasets @xcite . as multilingual models like lay-outxlm @xcite and lilt @xcite exist, we choose lilt for our base model due to its flexibility across different languages. ocr independence: models like layoutlm, layoutlmv2, layoutlmv3, lilt, bros, and geolayoutlm use word-level or segment-level bounding boxes to encode spatial information of text. however, languages with linguistic features differing from english face challenges in extracting such bounding boxes. for instance, japanese lacks spaces between words @xcite , and korean employs particles @xcite , resulting in single bounding boxes containing multiple words with distinct semantic meanings. consequently, the complexity of bounding boxes varies across languages. therefore, a framework capable of performing ser independent of any ocr engine result used is essential. see figure 5 in appendix for an example of tokenlevel box split algorithm. form independence: to create the model's input format from document images, the ocr engine results need to be arranged in an appropriate reading order. however, documents that occur in real industries are mostly photos, faxes, scanned copies, etc., which frequently have distortions or rotations @xcite . for documents with these characteristics or complex forms, it is difficult to determine the appropriate reading order @xcite . in this paper, we present a practical die pipeline for ser tasks, lofi (language, ocr, form independent extraction) pipeline. our experiments on korean medical bills and japanese receipts demonstrate its effectiveness, achieving entity-level f1 scores of 95.64% and 94.60%, respectively. our main contributions are: ## related works in this section, we show related works on language, ocr, and form methodologies on document information extraction (die) on semantic entity recognition (ser) tasks. ## lofi pipeline in this section, we present the methodology of solving language, ocr, form dependency issues, and our lofi (language, ocr, form independent) pipeline, a die pipeline for ser tasks, as shown in figure 1 . to outline lofi pipeline process: ## experiment setting to assess our pipeline's performance, particularly the model, we conduct experiments on two types of low-resource language (lrl) business documents and two open datasets as shown in ## experiment results we use the entity-level f1 score as the measure standard @xcite for both experiments. ## conclusions and future work in this paper, we propose lofi, a die pipeline for ser tasks in low-resource language (lrl) business documents. the lofi pipeline extracts text and bounding boxes from image documents via ocr, preprocesses them using a token-level box split logic, and performs ser fine-tuning without pre-training by replacing the plm. it achieves language independence through plm replacement, ocr independence via token-level box split logic, and form independence by extracting information despite image rotation or distortion. demonstrated on korean and japanese datasets, we anticipate its applicability to other lrl business documents. future research will focus on data augmentation, efficient annotation, and improved decoder architectures to handle document challenges to enhance ai capabilities for diverse business scenarios and document types.