A Critical Analysis of Document Out-of-Distribution Detection (2023)

Abstract: Large-scale pre-training is widely used in recent document understanding tasks. During deployment, one may expect that models should trigger a conservative fallback policy when encountering out-of-distribution (OOD) samples, which highlights the importance of OOD detection. However, most existing OOD detection methods focus on single-modal inputs such as images or texts. While documents are multi-modal in nature, it is underexplored if and how multi-modal information in documents can be exploited for OOD detection. In this work, we first provide a systematic and in-depth analysis on OOD detection for document understanding models. We study the effects of model modality, pre-training, and fine-tuning across various types of OOD inputs. In particular, we find that spatial information is critical for document OOD detection. To better exploit spatial information, we propose a spatial-aware adapter, which serves as a parameter-efficient add-on module to adapt transformer-based language models to the document domain. Extensive experiments show that adding the spatial-aware adapter significantly improves the OOD detection performance compared to directly using the language model and achieves superior performance compared to competitive baselines.

## introduction the recent success of large-scale pre-training has propelled the widespread deployment of deep learning models in the document domain, where model predictions are used to help humans make decisions in various applications such as tax form processing and medical reports analysis. however, models are typically pre-trained on data collected from the web but deployed in an environment with distributional shifts @xcite . for instance, the outbreak of covid-19 has led to continually changing data distributions in machine-assisted medical document analysis systems @xcite . this motivates the need for reliable document understanding models against outof-distribution (ood) inputs. the goal of ood detection is to categorize indistribution (id) samples into one of the known categories and detect inputs that do not belong to any known classes at test time @xcite . a plethora of ood detection methods has been proposed for single-modal (image or text) inputs @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . recent works @xcite @xcite @xcite also demonstrate promising ood detection performance based on large-scale models pre-trained on text-image pairs, as pre-training enables models to learn powerful and transferable feature representations @xcite . however, it remains largely unexplored if existing findings in the ood detection literature for images or texts can be naturally extended to the document domain. multiple unique challenges exist for document ood detection. unlike natural images, texts, or image-text pairs, no captions can describe a document and images in documents rarely contain natural objects. moreover, the spatial relationship of text blocks further differentiates multimodal learning in documents from multimodal learning in the vision-language domain @xcite . in addition, while recent pre-training methods have demonstrated remarkable performance in downstream document understanding tasks @xcite @xcite @xcite @xcite , existing pre-training datasets for documents are limited and lack diversity. this is in sharp contrast to common pretraining datasets for natural images. it remains underexplored whether existing ood detection methods are reliable in the document domain and how pre-training impacts ood reliability. in this work, we first present a comprehensive study to better understand ood detection in the document domain through the following questions: ## experimental setup models. fig. 2a summarizes common structures for document pre-training and classification models 2 . while documents typically come in the form of images @xcite , an ocr system can be used to extract words and their coordinates from the input image. therefore, models can use singlemodal or multi-modal information. we categorize these models according to the input modalities into the following groups: (1) models using only visual features, (2) models using solely textual features, (3) models incorporating both visual and textual features, and (4) models integrating additional spatial (especially layout) information. further details can be found in appendix a. constructing id and ood datasets. we construct id datasets from rvl-cdip @xcite , where 12 out of 16 classes are selected as id classes. dataset details are in appendix a. we consider two ood scenarios: in-domain and outdomain, based on the content (e.g., words, background) and layout characteristics. we further support our domain selection using otdd @xcite , a flexible geometric method for comparing probability distributions, which enables us to compare any two datasets regardless of their label sets. we observe a clear gap between in-domain and out-domain data, which aligns with our data selection. further details can be found in appendix a.1. ## the importance of spatial-awareness in previous sections, we mainly focus on mainstream text-based and vision-based models for inand out-domain ood detection. next, we consider models tailored to document processing, which we refer to as spatial-aware models, such as lay-outlmv3 and udoc. given fine-tuned models, we compare the performance of logit-based and distance-based ood scores. ## conclusions in this work, we provide a comprehensive and indepth study on the impacts of pre-training, finetuning, model-modality, and ood scores on a broad variety of document ood detection tasks. we present novel insights on document ood detection, which are under-explored or in contrast with ood detection works based on vision-language models. in particular, we highlight that spatial information is critical for ood detection in documents. we further propose a spatial-aware adapter as an add-on module to transformer-based models. our module adapts pre-trained language models to the document domain. extensive experiments on a broad range of datasets verify the effectiveness of our design. we hope our work will inspire future research toward improving ood robustness for reliable document understanding.