Doc- GCN : Heterogeneous Graph Convolutional Networks for Document Layout Analysis (2022)

Abstract: Recognizing the layout of unstructured digital documents is crucial when parsing the documents into the structured, machine-readable format for downstream applications. Recent studies in Document Layout Analysis usually rely on visual cues to understand documents while ignoring other information, such as contextual information or the relationships between document layout components, which are vital to boost better layout analysis performance. Our Doc-GCN presents an effective way to harmonize and integrate heterogeneous aspects for Document Layout Analysis. We construct different graphs to capture the four main features aspects of document layout components, including syntactic, semantic, density, and appearance features. Then, we apply graph convolutional networks to enhance each aspect of features and apply the node-level pooling for integration. Finally, we concatenate features of all aspects and feed them into the 2-layer MLPs for document layout component classification. Our Doc-GCN achieves state-of-the-art results on three widely used DLA datasets: PubLayNet, FUNSD, and DocBank. The code will be released at https://github.com/adlnlp/doc_gcn

## introduction digital documents (incl. scanned document images and pdf files) are popular and convenient for storing written textual information, so almost 2.5 trillion documents worldwide are available in the digital format @xcite . however, it is challenging to automatically recognize the layout and components of these unstructured digital documents and extract meaningful information using this format. for example, the financial office would require the scanned document image after their client signed. it is then crucial to recognize and extract the form component, such as the form title, person name, and the date the document is signed. this task is widely called document layout analysis (dla). the dla task aims at understanding the documents from either 1) the physical analysis by detecting the document structure and the boundaries of each layout region or 2) the logical analysis by categorizing the detected layout components (segments) into the predefined document element classes, such as title, date, author, and @xcite . in this research, we focus on the logical dla task to classify the different layout components of pdf documents by understanding the relationships between components. traditional deep learning-based dla approaches mainly focus on processing visual features of layout components @xcite @xcite using cnn-based models. some recent studies started to use texts to solve the problem with the support of semantic information for each layout component @xcite . however, applying visual and textual features is not enough to analyze the characteristics and relations of document layout components in order to classify them. in this paper, we try to fill this gap by defining and proposing: 1) layout components characteristic representation and 2) relation representation between components. the first question would be 'what would be the best aspects to represent the characteristics of different document layout components?'. the text density/sparsity in each document component is a valuable feature. for example, a paragraph is more dense and usually contains more texts than a table. moreover, syntactic information can be a key characteristic. it is obvious that a title mainly consists of noun phrases only, whereas a paragraph contains more sentences with the noun and verb phrases. hence, in this research, apart from the common visual features and semantic text features used by previous works, we propose the four major aspects, including text density, components appearance, syntactic and semantic information of textual contents, of each component in order to let the model conduct more comprehensive learning of the documents properties. another question would be 'how to represent the relation among document layout components?' most dla studies did not apply multiple aspects of features. even if applied @xcite , those features are integrated via simple concatenations and do not consider the influences of relationships between layout components on the classification performances. the characteristics of each component/segment are not enough to analyze the whole document layout and its corresponding components. assume a 'text' component is detected based on its characteristics, but it is more accurate to classify it as the 'figure caption' if it is right above or below a figure. thus, capturing and encoding the relationships between layout components is crucial for better layout analysis. in this paper, we propose a novel heterogeneous graph convolutional network (gcn)-based dla model, doc-gcn, on a document page level, taking the document layout component (segment) as nodes in the graph and encoding the relative positional and structural relations between layout components. we first construct six different graphs, each encoding one aspect of features among the layout components' syntactic, semantic, text density, and visual features. the syntactic and density aspects have two graph variants based on the different node embedding initialization methods. we use the gcn to update the node embedding by learning from its intimate neighbors, and the nodelevel pooling is then applied to integrate the graph variants. we concatenate the updated features of each aspect, getting the final layout component representation for the final classification of layout component types. in summary, the contributions of our work are as follows: ## doc-gcn we propose a graph-based network doc-gcn to encode and integrate the different aspects of document layout components. for each pdf page with n document layout components, we construct six different graphs: and g semc , capturing the features of all the document layout components in the page from four aspects: density, appearance, syntactic and semantic. each graph ## conclusion 10 we successfully handled the importance of the dla task, document component/segment classification. it focused on extracting important information (i.e., title, author, date, form answer) from the digital documents, including scanned document images and pdf files. we propose a heterogeneous graph-based dla model, doc-gcn, which integrates text density, appearance, and syntactic and semantic properties of the document layout components to generate comprehensive representations of documents. the graph structure also enables the model to capture and learn the relationships between the layout components when making predictions. our model outperforms all baselines on three publicly available dla datasets. we strongly hope that doc-gcn will motivate and provide insights into the future integration of different modalities and aspects for the logical document layout analysis task.