Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents (2023)

Abstract: We investigate semi-structured document classification in a zero-shot setting. Classification of semi-structured documents is more challenging than that of standard unstructured documents, as positional, layout, and style information play a vital role in interpreting such documents. The standard classification setting where categories are fixed during both training and testing falls short in dynamic environments where new classification categories could potentially emerge. We focus exclusively on the zero-shot learning setting where inference is done on new unseen classes. To address this task, we propose a matching-based approach that relies on a pairwise contrastive objective for both pretraining and fine-tuning. Our results show a significant boost in Macro F1 from the proposed pretraining step and comparable performance of the contrastive fine-tuning to a standard prediction objective in both supervised and unsupervised zero-shot settings.

## introduction textual information assumes many forms ranging from unstructured (e.g., text messages) to semistructured (e.g., forms, invoices, letters), all the way to fully structured (e.g., databases or spreadsheets). our focus in this work is classification of semi-structured documents. a semi-structured document consists of information that is organized using a regular visual layout, and includes tables, forms, multi-columns, (nested) bulleted lists, and that is either understandable only in the context of its visual layout or that requires substantial more work to understand without the visual layout. automatic processing of semi-structured documents comes with a unique set of challenges including a non-linear text flow @xcite , layout inconsistencies, and low-accuracy optical character recognition. prior work has shown that integrating the two-dimensional layout information of such documents is critical in models for analyzing such documents @xcite @xcite . due to these challenges, methods for unstructured document classification, such as static word vectors @xcite and standard pretrained language models @xcite @xcite perform poorly with semi-structured inputs as they model text in a one-dimensional space and ignore information about document layout and style @xcite . past work on semi-structured document classification @xcite @xcite @xcite has focused exclusively on the full-shot setting, where the target classes are fixed and identical across training and inference, neglecting the zero-shot setting @xcite , which requires generalization to unseen classes during inference. our work addresses zero-shot classification of semi-structured documents in english using the matching framework, which has been used for many tasks on unstructured text @xcite @xcite @xcite . under this framework, a matching (similarity) metric between documents and their assigned classes is maximized in a joint embedding space. we extend this matching framework with two enhancements. first, we use a pairwise contrastive objective (rethmeier and augenstein, 2020; @xcite that increases the similarity between documents and their ground-truth labels, and decreases it for incorrect pairs of documents and labels. we augment the textual representations of documents with layout features representing the positions of tokens on the page to capture the twodimensional nature of the documents. second, we propose an unsupervised contrastive pretraining procedure to warm up the representations of documents and classes. in summary, (i) we study zero-shot classification of semi-structured documents, which, to the best of our knowledge, has not been explored before. (ii) we use a pairwise contrastive objective to both pretrain and fine-tune a matching model for the task. this technique uses a layout-aware document encoder and a regular text encoder to maximize the similarity between documents and their ground-truth labels. (iii) using this contrastive objective, we propose an unsupervised pretraining step with pseudo-labels @xcite to initialize document and label encoders. the proposed pretraining step improves f1 scores by 9 and 19 points in supervised and unsupervised zero-shot settings respectively, compared to a setup without this pretraining. ## approach this section describes our proposed architecture ( § 2.1), pretrained model ( § 2.2), as well as the contrastive objective used for pretraining ( § 2.3) and fine-tuning ( § 2.4). ## conclusion this work explores zero-shot classification of semistructured documents. we proposed two contrastive techniques for pretraining and fine-tuning of a matching model. our fine-tuning objective showed comparable results to the standard crossentropy loss used widely in the literature and our contrastive pretraining significantly boosted zeroshot f 1 in supervised and unsupervised scenarios.