Entity Relation Extraction as Dependency Parsing in Visually Rich Documents (2021)

Abstract: Previous works on key information extraction from visually rich documents (VRDs) mainly focus on labeling the text within each bounding box (i.e.,semantic entity), while the relations in-between are largely unexplored. In this paper, we adapt the popular dependency parsing model, the biaffine parser, to this entity relation extraction task. Being different from the original dependency parsing model which recognizes dependency relations between words, we identify relations between groups of words with layout information instead. We have compared different representations of the semantic entity, different VRD encoders, and different relation decoders. For the model training, we explore multi-task learning to combine entity labeling and relation extraction tasks; and for the evaluation, we conduct experiments on different datasets with filtering and augmentation. The results demonstrate that our proposed model achieves 65.96% F1 score on the FUNSD dataset. As for the real-world application, our model has been applied to the in-house customs data, achieving reliable performance in the production setting.

## introduction in real-life scenarios, there are many types of visually rich documents (vrds), such as invoices, questionnaire forms, declaration materials and so on. these documents contain abundant layout information which helps us to understand the content while texts alone are not enough. in recent years, many works focus on how to extract key information from vrds based on the results of ocr (optical character recognition), which recognizes bounding boxes and texts within the boxes @xcite . each bounding box contains 1) a group of words that belong together from a semantic and spatial standpoint and 2) visual features such as layout, tabular structure and font size of the boxes in the document. we call such bounding boxes and texts within the boxes semantic entities foot_0 , and each entity contains the word group and layout coordinates 2 . key information extraction (kie) is such a task to analyze visually rich documents, which usually contains two steps, entity labeling and entity relation extraction. similar to named entity recognition (ner) and relation extraction in the traditional natural language processing (nlp), entity labeling aims to assign predefined labels to the semantic entities in vrds (g. jaume and thiran, 2019; @xcite , and entity relation extraction 3 predicts relations between these semantic entities. compared with ner and relation extraction in the traditional nlp, kie from vrds is a more challenging task. first, a normal (named-) entity in plain text does not contain layout information as those semantic entities in vrds. second, normal relation extraction predicts the relation between two given mentions, while relation extraction in vrds needs to predict the relation between any two semantic entities in the document. as figure 1 (left) illustrates, the entity labeling task is to tag "533" with the label "answer", "registration no." with label "question". however, which question can be answered by "533" remains unknown without entity relation extraction. compared with labeling, the entity relation extraction task is less explored, but its benefits at least include: 1) pro-figure 1 : left part shows one visually rich document from the funsd data. group of words with one bounding box means one semantic entity and we number each entity as b i . different colors of boxes mean their different entity labels as legends under the example list. relation links between semantic entities always point from key entities to value entities. we convert entity relations in the vrd to one tree and add the pseudo root node following the similar setting in dependency tree, and then link zero-head entities to the pseudo node as the pseudo links shown in the right part. as another similar task to entity relation extraction in vrds, dependency parsing aims to find out syntactic relations between words of an input sentence, which has been widely studied for decades. both of these two tasks capture pairwise relationship between basic units of the input data. we adapt the popular biaffine dependency parser which utilizes the biaffine attention to compute scores between words @xcite into the entity relation extraction task due to their similarity. since visual features play an important role in the vrds, we introduce abundant layout information into different layers of the model to enhance the original text-only biaffine model: apart from the above, inspired by the joint pos tagging and dependency parsing model @xcite , we propose the multi-task learning for both entity labeling and relation extraction to further improve the performance. abundant detailed experiments are conducted to verify our approach of applying the biaffine dependency parser to semantic entity relation extraction task in vrds. our proposed relation extraction model achieves 65.96% f1 score on the funsd dataset, demonstrating the effectiveness of our model. as for the real-world application scenario, our model has also been applied to the in-house customs data, achieving reliable performance in the production setting. the contributions of this paper are as follows: ## related work visually rich document understanding includes many tasks, such as layout recognization @xcite , table detection and recognition @xcite and key information extraction @xcite @xcite @xcite . our paper focuses on the key information extraction task which contains two subtasks, entity labeling and relation extraction. the former subtask tags entities with predefined labels, such as task 3 on the sroie data released by @xcite , while the latter discovers relations between entities, such as subtask c(3) on the funsd data (g. jaume and thiran, 2019). to encode the semantic entity in vrds, @xcite and @xcite replace bilstm (bi-directional long short-term memory) used by @xcite with bert @xcite or roberta @xcite . @xcite propose layoutlm, which adds the 2-d position embedding into language model based on bert and pretrain their language model on large-scale scanned document images with more visually-related loss function. experiments verify that encoding the word group and layout coordinates at the same time is more effective for vrd understanding. layoutlmv2 additionally introduces visual embedding into input layer and integrates a spatial-aware self-attention mechanism into the transformer architecture @xcite . and layoutlmv2 performs better than layoutlm in downstream vrd understanding tasks. while encoding vrds, previous works take entity labeling task as sequence labeling and reimplement the named entity recognition (ner) framework @xcite but ignore layout information. then, many works introduce a gcnbased module to encode layout information and combine textual and visual information together @xcite @xcite . in the gcn module, @xcite ; @xcite take layout features between entity b i and b j as edge embedding to up-date entity representation while @xcite prune irrelevant nodes in graph according to same x-axis or y-axis coordinates to get the adjacency matrix. to predict relations between entities, g. jaume and thiran (2019) provide one simple approach which concatenates the representations of two entities and use multi-layer perceptron (mlp) to obtain the relation score between entities. @xcite also use the mlp scorer but take gnn as document encoder instead of bert and perform better. in the field of dependency parsing, @xcite propose the biaffine attention mechanism to compute scores between words, and achieve better performance than the mlp mechanism used by @xcite . as the biaffine attention is widely used in other tasks like ner @xcite and semantic role labeling @xcite , we propose to use it for the entity relation extraction task in this work. ## entity relation extraction as dependency parsing both semantic entity relation extraction and dependency parsing tasks aim to decide whether there exists relation between two entities/words and assume that links always point from key/head unit to value/modifier unit shown in figure 1 . therefore, we can draw lessons from the dependency parsing exploration as it has been studied for several decades and achieved great progress. biaffine parser, a strong model in dependency parsing, achieves competitive performance and is widely used in different scenes and tasks. this section introduces how to apply the biaffine parser to our relation extraction task according to their similarity and difference. ## conclusion this paper focuses on the largely-unexplored entity relation extraction task in vrds. we take advantages of previous works in semantic entity labeling and dependency parsing and propose our relation extraction model sera. our improved entity relation extraction model achieves 64.60% f1 score on the funsd data, outperforming previous baseline by a large margin; and we employ two simple but effective training strategies to further improve the performance to 65.96%, i.e., multi-task learning with entity labeling and data augmentation. in addition, we verify the effectiveness of our model on the real-world customs data with different layouts in the production setting. in the future, we plan to incorporate more visual features into the relation extraction model and also extend it into more domains and business scenarios.