Q uery F orm: A Simple Zero-shot Form Entity Query Framework (2023)

Abstract: Zero-shot transfer learning for document understanding is a crucial yet under-investigated scenario to help reduce the high cost involved in annotating document entities. We present a novel query-based framework, QueryForm, that extracts entity values from form-like documents in a zero-shot fashion. QueryForm contains a dual prompting mechanism that composes both the document schema and a specific entity type into a query, which is used to prompt a Transformer model to perform a single entity extraction task. Furthermore, we propose to leverage large-scale query-entity pairs generated from form-like webpages with weak HTML annotations to pre-train QueryForm. By unifying pre-training and fine-tuning into the same query-based framework, QueryForm enables models to learn from structured documents containing various entities and layouts, leading to better generalization to target document types without the need for target-specific training data. QueryForm sets new state-of-the-art average F1 score on both the XFUND (+4.6% 10.1%) and the Payment (+3.2% 9.5%) zero-shot benchmark, with a smaller model size and no additional image input.

## introduction form-like document understanding has become a booming research topic recently thanks to its many real-world applications in industry. form-like documents refer to documents with rich typesetting formats, such as invoices and receipts in everyday inventory workflow. automatically extracting and organizing structured information from form-like documents is a valuable yet challenging problem. recent methods @xcite @xcite often discuss the problem of form-like document understanding, e.g. document entity extraction (dee), in the supervised in the pre-training stage, we extract millions of schemas and entity-value pairs from publicly available webpages to generate a large amount of query-value pairs to teach the backbone model to make query-conditional prediction. during fine-tuning, we extract more accurate entity-value pairs from the available annotated document and directly learn schema information from data. finally, we evaluate the pretrained model on a different target document without training data. in this work, we propose a novel query-based framework, queryform, to learn transferable knowledge from different types of documents for zero-shot entity extraction on the target document type. the workflow of queryform is illustrated in figure 1 . ideally, we would like to prompt the model: this document has the following [schema], please extract its [entity] value, and model is able to accurately predict the corresponding word tokens belong to the queried entity. to this end, we encode both schema and entity information in our query, so that the model is no longer limited by a certain document type and a fixed set of entity types (or classes). moreover, our query-based design can even benefit further from large-scale datasets with diverse schemas and entity types. in order to feed this kind of composite query, we propose a dual prompting strategy to effectively prompt the backbone model, e.g., a pre-trained transformer, to make conditional prediction. as its name suggests, the dual prompting strategy consists of an e(ntity)-prompt and a s(chema)-prompt. depending on the annotations we have, we can either generate the prompts from semantic labels, or learn them directly from data. although similar concepts to dual prompting exist in the vision field @xcite to solve different problems, the main design in queryform is original in dee. we also propose a query-based pre-training method, queryweb, which leverages the highlyaccessible and inexhaustible resource -publicly available webpages. during the pre-training stage, the model learns to quickly adapt to various queries composed of different s-prompts and e-prompts generated from html source of webpages. after the decoupling of entity and schema that are tied to document types, the model can learn more transferable knowledge -leveraging the rich layout, scale and content information in webpages to make query-conditional predictions. in summary, our work makes the following contributions: ## methodology we propose queryform as a general query-based framework for solving the zero-shot dee problem. queryform consists of a novel dual prompting strategy and a specially-designed pre-training approach called queryweb. in figure 3 , the model is first pre-trained on a large-scale webpage dataset to learn to make conditional prediction under rela-tively noisy queries generated from combination of webpage domains (proxy of schema) and html tags (proxy of entity). then, the model is fine-tuned on form-like documents with a unified schema to learn more specialized knowledge, by learning schema information in s-prompt and further encode more accurate entity-level knowledge in e-prompt. finally, we test the model on the target document type in a zero-shot fashion (figure 1 ). ## conclusion this paper presents queryform, a novel framework to address the challenging zero-shot document entity extraction problem. the dual prompting design in queryform offers a refreshing view to unify the pre-training and fine-tuning objectives, allowing us to leverage large-scale form-like webpages with html tags as weak annotations. queryform sets new state-of-the-art results on multiple zero-shot dee benchmarks. we believe queryform serves as a flexible framework for document understanding tasks, and multiple interesting directions could be further explored within the framework, such as prompt design, richer pre-training sources, etc.