T ree F orm: End-to-end Annotation and Evaluation for Form Document Parsing (2024)

Abstract: Visually Rich Form Understanding (VRFU) poses a complex research problemdue to the documents’ highly structured nature and yet highly variable style and content. Current annotation schemes decompose form understanding and omit key hierarchical structure, making development and evaluation of end-to-end models difficult. In this paper, we propose a novel F1 metric to evaluate form parsers and describe a new content-agnostic, tree-based annotation scheme for VRFU: TreeForm. We provide methods to convert previous annotation schemes into TreeForm structures and evaluate TreeForm predictions using a modified version of the normalized tree-edit distance. We present initial baselines for our end-to-end performance metric and the TreeForm edit distance, averaged over the FUNSD and XFUND datasets, of 61.5 and 26.4 respectively. We hope that TreeForm encourages deeper research in annotating, modeling, and evaluating the complexities of form-like documents.

## introduction visually rich document understanding (vrdu) has been a growing field in multimodal ai research. vrdu takes document images as input and applies tasks such as document classification @xcite @xcite , information extraction @xcite , visual question-answering (vqa) @xcite @xcite , inter alia. consequently, state-of-the-art vrdu models rely on image-to-text understanding, such as optical character recognition (ocr), as well as multimodal models that can exploit textual, visual, and spatial features of documents. vrfu is a subset of vrdu that focuses on forms, i.e., documents that contain a collection of hierarchical key-value pairs (in various styles) regarding specific entities. forms are highly structured documents, and so can be directly parsed into figure 1 : excerpt of a funsd form. headers are marked in burgundy, questions are marked green, and answers are marked in blue. entity links provided by the funsd annotation schemes are marked in orange. links in pink were created for treeform. a structured object. parsing receipts, which share a similar though smaller and simpler structure, have been the focus of much document parsing research @xcite @xcite . in this work, we focus on the broader field of form parsing. the funsd dataset @xcite , and its multilingual counter-part xfund @xcite , are standard vrfu datasets used in the literature. these datasets split form parsing into entity labeling and entity linking tasks. unfortunately, this annotation scheme is unable to fully express the structure of a form. moreover, to the best of our knowledge, no holistic approach exists for evaluating complete funsd-type predictions. few other form parsing datasets exist, such as funsd+ @xcite and the national archives forms (naf) dataset @xcite . we do not address these directly but many of our contributions can be extended to these annotation schemes. in this paper, we tackle the problem of complete form parsing through a tree-based approach that 1 successfully captures the hierarchical structure of forms which is often missed in modern annotation schemes. our contributions are summarized below: ## limitations of form parsing datasets most vrfu research employs funsd-type annotations @xcite . this annotation scheme contains two components: first, a list of semantic entities defined by a group of tokens and a semantic label (header, question, or answer). 2 second, a list of directed links between entities to denote form structure. an example of an funsd annotation is given in app. c. while the funsd (and xfund) project was a fundamental step in vrfu, its approach contains underlying issues (vu and nguyen, 2020). other non-funsd schematic vrfu datasets exist @xcite . however, they are not widely used and are not without similar issues. task decomposition. the funsd annotation scheme proposes that vrfu should be decomposed into semantic entity labeling and entity link-dataset lang. scheme |train| |test| funsd en funsd 149 50 xfund multi funsd 1043 350 funsd+ en funsd+ 1023 116 naf en naf 741 63 inconsistent annotations. the funsd annotations contain several inconsistencies in both their entity labeling and entity linking components. inconsistencies arise in hierarchical forms, handwritten input, and missing entity links. while many of these inconsistencies were improved in (vu and nguyen, 2020), some issues still persist ## an end-to-end evaluation of funsd in this section, we propose a new f1 metric to evaluate funsd-type predictions on their combined entity labeling and entity linking performance. our metric, inspired by the labeled attachment score (las) @xcite , is based on constructing a tree of the funsd annotations. each node in the tree represents a semantic funsd entity, and each edge in the tree is a link between entities, labeled with the entity label of the child node. entities with no incoming links have an incoming edge from a dummy root node, labeled with the entity's label. an example of this structure is given in figure 2b . let t be the set of predicted edges and g be the set of ground truth edges. we can then respectively define the precision and recall as: the above f1 metric does not consider nodealignment (i.e., word grouping) in its evaluation. indeed, it assumes that the nodes of the predicted and ground truth trees are aligned. node alignment can be done through a greedy algorithm. for each predicted entity, we compute the normalized levenshtein distance @xcite to each true entity. 3 we then greedily select alignments that meet a certain threshold (edit distance less than 0.4), until no possible alignments remain. we define the node-alignment accuracy (naa) metric as the mean normalized levenshtein distance between each pair of aligned nodes. foot_3 ## a tree-based annotation scheme in this section we present a novel tree-based form annotation scheme, treeform. unlike our f1 and naa metrics, treeform does not assume the funsd task decomposition. instead, treeform provides a fully structured object that enables complete representation of a form. a treeform node is represented by a json object (i.e., a dictionary) whose key and value respectively contain the node's (textual) value and a dictionary where each key is either another node or an edge label that leads to a group of child nodes. therefore, a treeform tree is represented using a single json object which efficiently contains all form information necessary for any down-stream vrfu task. @xcite also parses complete forms into json objects, however, their approach requires multiple objects per form and so annotations are less readable. treeform considers forms to be comprised of three key structures: headers, question-answer pairs, and tables; each of these structures is represented in a tree structure which is detailed in app. b. an example of a treeform annotation is given in figure 2c . ## experiments to evaluate treeform as well as our end-to-end funsd metrics, we fine-tune layoutxlm @xcite and donut @xcite on each of the funsd and xfund datasets. 6 we chose these two models as they represent the two cur-rent approaches to form parsing: pipelined foot_6 (lay-outxlm) and end-to-end (donut). we describe pre-and post-processing steps required for applying the funsd and treeform evaluations to each model in app. d. importantly, we performed postprocessing steps to the output of layoutxlm to enable evaluation using our metrics. the results are given in for all languages, our end-to-end f1 metric is lower than one or both of the standard labeling and linking f1 scores. this indicates that while past work may have high labeling (or linking) f1 scores, they may not have full form understanding capabilities. this is further exacerbated as our f1 metric requires the naa score to provide a holistic overview. as expected, layoutxlm outperforms donut with respect to naa as it has access to ocr tokens. however, donut still outperforms layoutxlm on f1 for some languages, suggesting it is better at understanding hierarchical structure. we consider the median ganted score for evaluating treeform predictions. donut outperformed layoutxlm in all languages except for chinese. this is expected as donut was designed with document parsing in mind and so could be directly fine-tuned on treeform data whereas layoutxlm was pipelined using the funsd annotation scheme (see app. d). as ganted has not yet been widely used in the literature, it is difficult to assess the effectiveness of these systems. nevertheless, dessurt @xcite reported a ganted score of 23.4 on the funsd dataset which aligns with the scores in ## conclusion in this paper, we described the challenges of current form understanding and the limitations of the current datasets and their annotation schemes. we introduced a novel f1 metric that can be applied to current annotation schemes and further proposed a new tree-based annotation scheme, treeform, that enables complete form parsing. we applied state-of-the-art models (layoutxlm and donut) to provide the first treeform baselines. we envision future work to create a new treeform dataset, that contains correct and consistent annotations.