Attention-Based Graph Neural Network with Global Context Awareness for Document Understanding (2020)

Abstract: Information extraction from documents such as receipts or invoices is a fundamental and crucial step for office automation. Many approaches focus on extracting entities and relationships from plain texts, however, when it comes to document images, such demand becomes quite challenging since visual and layout information are also of great significance to help tackle this problem. In this work, we propose the attention-based graph neural network to combine textual and visual information from document images. Moreover, the global node is introduced in our graph construction algorithm which is used as a virtual hub to collect the information from all the nodes and edges to help improve the performance. Extensive experiments on real-world datasets show that our method outperforms baseline methods by significant margins.

## introduction information extraction @xcite @xcite is a widely studied task of retrieving structured information from texts and many inspiring achievements have been made in this field. however, most of these works are generally focusing on extracting entities and relationships from plain texts which are not appropriate to apply directly on document understanding. document understanding is the process of automatically recognizing and extracting key texts from scanned unstructured documents and saving them as structured data. document understanding was already introduced in a competition of icdar 2019, where the goal was to detect texts in documents and extract key texts from receipts and invoices. in this work, we focus on document understanding which is mainly about key information extraction from scanned unstructured documents. the following paragraphs summarize the challenges of the task and the contributions of our work. ## related works several rule-based document understanding systems were proposed in @xcite @xcite . @xcite presented a case for the importance of rule-based approaches to industry practitioners. smartfix by @xcite employs specific configuration rules designed for each template. the study by @xcite offers a template matching based algorithm to solve the document understanding problem and plenty of templates have to be constructed and maintained to deal with different situations. proceedings of the 19th china national conference on computational linguistics, pages 853-862, hainan, china, october 30 -novermber 1, 2020. (c) technical committee on computational linguistics, chinese information processing society of china c c l 2 0 2 0 computational linguistics however, rule-based methods rely heavily on the predefined templates or rules and are not scalable and flexible for most document understanding problems since documents in real life have no fixed layout. furthermore, updating the templates or rules requires a lot of effort. a recent study by @xcite proposed convolutional universal text information extractor (cutie). cutie treats the document understanding task as an image semantic segmentation task. it applies convolutional neural networks on gridded texts where texts are semantical embeddings. however, this work only uses text-level features and doesn't involve image-level features. inspired by bert @xcite , @xcite proposed layoutlm method. it applies bert architecture for the pre-training of text and layout. although layoutlm uses image features in the pre-training stage and it performs well on several downstream tasks, the potential relationship between two text segments hasn't been taken into consideration. in addition, sufficient data and time are required to pre-train the model inefficiently. since graph neural networks @xcite @xcite have shown great success in unstructured data tasks, more and more research works are focusing on using gnn to tackle the document understanding problem. liu et al. (2019) presented a gcn-based method for information extraction from document images. it is a work attempting to extract key information with customized graph convolution model. however, prior knowledge and extensive human efforts are needed to predefine task-specific node and edge representations. one study by @xcite explores the feature fusion of textual and visual embeddings by gnn. this work differs from ours because it still treats the document understanding task as the sequence tagging problem and uses a bi-directional lstm model to extract entities which has already been proved to have limited ability to learn the relationship among distant words. ## proposed method this section demonstrates the architecture of our proposed model. to extract textual context, our model first encodes each text segment in the document by pre-trained bert model as its corresponding text embeddding. then using multiple layers of cnn to get its image embedding. the combination of these two types of embeddings will generate unique global node representation and various local node representations. these node representations contain both visual context and textual context and will be used as node input to the graph attention network. our model transforms the document understanding task into a node classification problem by taking both local context and global context into account. ## experiments we use pytorch framework to implement our experiments on a gtx 1080ti gpu and apply our model for information extraction from two real-world datasets. ## conclusions and future works this paper studies the problem of document understanding. in this work, we present a novel method that takes global context into account to refine the graph architecture on the complex documents. the explanatory experiments suggest that our proposed model is capable of extracting more information from documents to learn a more expressive representation through attention-based graph convolutions. we hope that our research will serve as a base for future studies on document understanding. furthermore, we intend to extend our model to other document related tasks, such as document classification or document clustering.