A C RITICAL A NALYSIS OF O UT - OF -D ISTRIBUTION D ETECTION FOR D OCUMENT U NDERSTANDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Large-scale pretraining is widely used in recent document understanding models.
During deployment, one may expect that large-scale pretrained models should trigger a conservative fallback policy when encountering out-of-distribution (OOD) samples, which suggests the importance of OOD detection.
However, most existing OOD detection methods focus on single-modal inputs such as images or texts.
While documents are multi-modal in nature, it is underexplored if and how multi-modal information in documents can be exploited for OOD detection.
In this work, we first provide a systematic and in-depth analysis on OOD detection for document understanding models.
We study the effects of model modality, pretraining, and finetuning across various types of OOD inputs.
In particular, we find that spatial information is critical for document OOD detection.
To better exploit spatial information, we propose a simple yet effective spatial-aware adapter, which serves as an add-on module to adapt transformer-based language models to document domain.
Extensive experiments show that our method consistently improves ID accuracy and OOD detection performance compared to baselines.
We hope our findings can help inspire future works on understanding OOD robustness for documents.
I NTRODUCTION
The recent success of large-scale pretrained models has led to the widespread deployment of deep models in various applications.
In the document domain, model predictions are increasingly used to help humans make decisions in important applications ranging from tax form processing, machine learning assistant medical reports analysis, deep analyses from financial forms, etc .
However, in most cases, models are pretrained on collected data but are then deployed in an environment with a different distribution over the observed data (Cui et al., 2021).
For example, with the outbreak of COVID-19 (Velavan & Meyer, 2020), machineassisted medical document analysis systems have to face continually changing data distributions.
This motivates the need for reliable methods in the document domain to detect out-of-distribution (OOD) inputs.
The goal of OOD detection is to categorize in-distribution (ID) test samples into one of the known categories and detect instances that do not belong to any known classes (Huang & Li, 2021; Bendale & Boult, 2016).
Generally, a model is optimized on a particular task ( e.g ., image classification (Deng et al., 2009)), and a companion OOD detector is built as a safeguard for the classifier.
Recently, large-scale pretrained models have demonstrated promising results in multiple domains (Dosovitskiy et al., 2021;
Hendrycks et al., 2020) as pretraining enables models to learn powerful and transferable feature representations (Radford et al., 2021).
In particular, the models obtained by finetuning large-scale pretrained models are significantly better at OOD detection even with a simple distance metric (Lee et al., 2018;
Radford et al., 2021).
It is underexplored whether existing OOD detection methods that demonstrate success for images or text can be naturally extended to documents.
The main challenges posed in document OOD detection stem from the fact that document understanding is inherently multi-modal, thus, it is suboptimal to rely on a single modality.
The majority of recent OOD detection approaches focus on single-modal learning (Hsu et al., 2020;
Zhou et al., 2021; Xu et al., 2021a; Jin et al., 2022), and they are not compatible with document understanding tasks which require multi-modal learning.
The spatial relationship of text blocks in documents further differentiated the document multimodal learning from the multimodal learning in vision-language domain (Lu et al., 2019; Li et al., 2020).
In addition, recent document pretraining methods have demonstrated remarkable performance on various downstream document understanding tasks (Xu et al., 2020; 2021b;
Huang et al., 2022; Li et al., 2021; Cui et al., 2021; Hong et al., 2022; Gu et al., 2022; Wang et al., 2022).
However, existing pretraining datasets for documents are limited and lack diversity, in sharp contrast to common pre-trainining datasets for natural images.
Therefore, it is not obvious which OOD detection methods are reliable in the document domain and how pretraining impacts OOD robustness.
This paper investigates the OOD robustness in the document domain through the following questions: (1)
Are pretrained models robust to OOD examples?
Is further pretraining beneficial?
How do the pretraining data and tasks affect the performance?
(2) How does multimodality (textual, visual, and spatial) affect OOD robustness?
(3) Are existing OOD detection methods developed for natural images and texts transferrable to documents?
We present a large-scale evaluation of recent approaches.
We focus on models pretrained on different data types and evaluate them on a diverse range of document understanding benchmarks across visual, textual, and spatial modalities.
Our key contributions are summarized as follows:
• We show that pretraining datasets and tasks significantly impact OOD detection performance.
Through extensive pretraining and finetuning experiments, we find that higher finetuning performance on ID data does not usually translate to better performance on OOD data.
This observation emphasizes the importance of considering metrics beyond ID performance for measuring model reliability.
• We propose a spatial-aware adapter, which can serve as an add-on module to transformer-based models and learn the spatial-aware representation.
Our method can easily transfer the pretrained language models to the document domain.
Extensive experiments show that our method can consistently improve ID accuracy and OOD detection performance across a broad spectrum of datasets.
• We show that recent conclusions drawn from OOD detection methods are valid for images and texts but do not always transfer to documents.
For a wide range of document models, we observe that OOD samples are easier to identify in the feature space than in the logit space.
The rest of the paper is organized as follows.
Sec. 2 provides the preliminaries and related works.
In Sec. 3, we provide a comprehensive analysis of OOD robustness for document models and conclude in Sec. 4.
P RELIMINARIES AND R ELATED W ORKS 2. D OCUMENT M ODELS AND P RETRAINING
Large-scale pretrained models have attracted a lot of attention in the document domain.
In vision or natural language processing (NLP) tasks, pretraining has shown great success in producing generic representations that learn from large-scale unlabeled corpora (Devlin et al., 2018;
Lu et al., 2019; Su et al., 2019; He et al., 2020).
Document pretraining also seeks to find universal representations suitable for any downstream task.
However, the unique characteristics of document images distinguish document pretraining works from previous ones in vision or language domains.
For documents, the contents are spatially distributed, and visual and textual information co-occurs within the semantic regions.
In contrast, inputs in the language domain are pure texts, and inputs in the vision-language domain are image-text pairs.
Recent document pretraining models differ in architecture and objectives, as depicted in Fig.
LayoutLM
(Xu et al., 2020) extends BERT to learn contextualized word representations for document images through multi-task learning.
It takes a sequence of Optical Character Recognition (OCR) (Smith, 2007)
[Figure]
words and word bounding boxes as and
LayoutLMv2 (Xu et al., 2021b) improves on the prior work by including an image encoder in pretraining and training them jointly.
Like LayoutLMv2, DocFormer (Appalaraju et al., 2021) also adopts a CNN model to extract image grid features.
It fuses the spatial information as an inductive bias for the self-attention module.
The latest version, LayoutLMv3 (Huang et al., 2022), shares similar ideas as LayoutLMv2 and further enhances the visual and spatial characteristics by introducing two other tasks: masked image modeling and word-patch alignment.
Another line of works for document pretraining focuses on different granularities of document images and takes region-level text blocks as the basic input elements, such as SelfDoc (Li et al., 2021) and UDoc (Gu et al., 2021).
The pretraining tasks of SelfDoc and UDoc are based on feature space.
They adopt a crossmodal encoder to model the relationship between visual and textual features.
Instead of using the spatial information at the input layer, SelfDoc and UDoc encode the 2D spatial information with a linear mapping and fuse the position embeddings at the output layer of the image encoder and sentence encoder.
Despite the promising performance of those pretrained models on downstream applications, it remains largely underexplored whether recent document pretraining models are robust to various types of OOD data, the role of pretraining and finetuning, and the key factors for document OOD detection.
2. O UT - OF - DISTRIBUTION D ETECTION
Many OOD detection methods have been proposed for deep models, including generative model-based methods (Ge et al., 2017; Oza & Patel, 2019; Nalisnick et al., 2019;
Ren et al., 2019; Xiao et al., 2020; Morteza & Li, 2022), and discriminative-model based methods.
For the latter category, an OOD scoring function can be derived based on the softmax output or logit space (Liu et al., 2020; Hsu et al., 2020; Huang & Li, 2021;
Liang et al., 2018; Sun et al., 2021), gradient information (Huang et al., 2021), or the feature space (Sastry & Oore, 2020; Sehwag et al., 2021; Winkens et al., 2020; Sun et al., 2022) of a classifier.
Despite their impressive performance, most of the scores are developed for natural images and text inputs.
A recent work (Larson et al., 2022) studies OOD detection performance for documents, but only explores a limited number of models and OOD detection methods.
Furthermore, the relationship between pretraining, finetuning, and spatial information is underexplored.
In this work, we provide a finer-grained and comprehensive analysis and hope to shed light on the key factors of OOD robustness for documents.
Notations We denote the input and label space X in and Y in = { 1 , . . .
, K } , respectively.
Let D in = { ( x in i , y in i ) }
Ni =1 denote an ID dataset, where x ∈ X in is the input feature vector, and y in ∈ Y in denotes the semantic label for K -way classification.
Let D out =
( x out i , y out i ) }
Mi =1 denote an OOD test set where y out ∈ Y out , and Y out ∩ Y in = ∅ .
OOD detection can be formulated as a binary classification problem, which aims to distinguish between ID and OOD data.
We express the neural network model f :=
g ◦ h as a composition of a feature extractor
h : X in → R d and a classifier g :
R d → R K , which maps the feature embedding of an input to K real-valued numbers known as logits.
During inference time, OOD detection can be performed by exercising a thresholding mechanism G γ ( x ) = 1 { S ( x ) ≥ γ } where by convention samples with higher scores S ( x ) are classified as ID and vice versa.
The threshold γ is typically chosen so that a high fraction of ID data ( e.g. , 95%) is correctly classified.
We group OOD detection methods into two major categories: logit-based scores are derived from the logit layer of the model, while distance-based methods are directly based on the feature embedding layer, as shown in Fig.
We describe a few popular OOD detection methods for each category as follows.
• Logit-based: Maximum Softmax Probability (MSP) score (Hendrycks & Gimpel, 2017)
S MSP = max i ∈
[ K ] e f i ( x ) / (cid:80) Kj =
e f j ( x ) naturally arises as a classic baseline since logits can be converted to a categorical distribution p ( y | x ) ; Energy score (Liu et al., 2020): S Energy = log (cid:80) i ∈
[ K ] e f i ( x ) utilizes the Helmholtz free energy of the data and theoretically aligns with the logarithm of the ID density; MaxLogit score (Hendrycks et al., 2022): S Maxlogit =
max i ∈
[ K ] f i ( x ) removes the softmax function in MSP and demonstrates promising performance on large-scale natural image datasets recently.
• Distance-based : Distance-based methods directly leverage feature embeddings h based on the idea that OOD inputs are relatively far away from ID centroids or prototypes.
Depending on the distributional assumption of feature embeddings, methods can be characterized as 1) parametric methods such as Mahalanobis score (Lee et al., 2018; Sehwag et al., 2021) which assumes ID embeddings follow classconditional Gaussian distributions and use Mahalanobis distance from the ID centroid as the distance metric; 2) non-parametric methods such as KNN+ ( Sun et al., 2022) which uses cosine similarity as the distance metric.
Evaluation Metrics To evaluate OOD detection performance, we adopt two commonly used metrics (Hendrycks & Gimpel, 2017):
Area Under the Receiver Operating Characteristic (AUROC) and False
Positive Rate at 95% Recall (FPR95).
For ID test sets, we report Accuracy (Acc), F1 score, and Mean
Average Precision (mAP).
A NALYZING OOD R OBUSTNESS FOR D OCUMENT M ODELS
In this section, we consider the task of document classification, where models are expected to classify documents into categories such as scientific papers , resumes , etc .
However, it is underexplored whether models are robust to OOD samples at test time.
Most document classification datasets exist in the form of images (Harley et al., 2015).
Usually, the first step is to pass the image through an OCR system to obtain a set of text blocks along with their coordinates in the image.
Given the input image, extracted words, and coordinates, models can utilize single-modal or multi-modal information to classify the document.
Models Fig.
2 (a) shows common structures for document image pretraining and classification models 1 .
According to the input modalities, we categorize them into the following groups: (1) Vision-based : Since current document datasets exist as images, we can treat document classification as the standard image classification problem.
In our experiments, we consider ResNet-50 (He et al., 2016) and
ViT (Fort et al., 2021) as exemplar document image classification models.
As for pretrained weights, we consider two settings: pretrained on ImageNet (Deng et al., 2009) and further pretrained on IIT-CDIP (Lewis et al., 2006).
We adopt masked image modeling (MIM) for image pretraining with a mask ratio of 0.6.
Note that the document classification dataset we used in this paper, RVL-CDIP, is a subset of IIT-CDIP.
Hence, unless otherwise specified, the IIT-CDIP pretraining data used in this paper excludes RVL-CDIP.
[Figure] [Figure] (2) Text-based: Alternatively, we can define the classification as a text classification problem since documents typically contain words.
In our experiments, we consider RoBERTa (Liu et al., 2019) as the backbone and append a classifier for finetuning.
Since some documents such as scientific papers consist of sentences with more than 512 tokens, we also consider Longformer (Beltagy et al., 2020), which can handle a maximum of 4,096 input tokens.
Similar to the vision-based models, we further pretrain the language models with masked language modeling (MLM) on IIT-CDIP extracted text corpus.
(3) Text+Spatial: Layout information plays a crucial role in the document domain.
As shown in Fig.
3, a document is composed of words or images with some specific layouts.
To investigate the effect of layout information, we adopt LayoutLM as a baseline.
It is specifically designed for documents and trained on the full IIT-CDIP data.
Inspired by the promising OOD detection performance of spatial-aware models (Sec 3.3) and the recent advances in adapter-based transformers (Pfeiffer et al., 2020), we propose a new spatial-aware adaptor, a small learned module that can be inserted within a pretrained language model.
Besides the simplicity, our adapter is competitive for both ID classification and OOD detection (Sec 3.4).
(4) Visual+Textual+Spatial: Current state-of-the-art methods tailored to documents consider various input granularity and modality and utilize textual, visual, and spatial information for document tasks.
Despite the promising performance, such models are large in size and computationally heavy.
We select two representative models to evaluate upon: LayoutLMv3 and UDoc.
For a fair comparison, both models are pretrained on full IIT-CDIP.
Constructing ID and OOD Datasets We construct ID (Harley al., 2015).
Specifically, we specify 12 out of 16 classes as ID classes.
For OOD datasets, we consider two scenarios: (1) In-domain OOD : To determine the OOD categories, we extensively analyze the performance of recent document classification models.
Fig.
2(b) shows a detailed comparison of per-category accuracy on the
RVL-CDIP test set.
Naturally, for the classes the model performs poorly on, we may expect models to detect such inputs as OOD instead of assigning a specific ID class with low confidence.
We observe that thecategories letter , form , scientific report , presentation result in the worst performance across most of models with different modality, which we use as OOD categories and construct the OOD datasets accordingly.
The
ID dataset is constructed from the remaining 12 categories.
We refer to these OOD datasets as in-domain , as they are also constructed from RVL-CDIP.
(2) Out-domain OOD : In the open-world setting, test inputs can have significantly different color schemes and layouts compared to ID samples.
To mimic such scenarios, we use two public datasets as the out-domain
OOD test sets.
Specifically, NJU-Fudan Paper-Poster Dataset (Qiang et al., 2019) contains scientific posters in the digital PDF format, and we extract the document contents with 2 .
CORD (Park et al., 2019) is a receipt understanding dataset that contains significantly different inputs from that in RVL-CDIP.
As shown in Fig.
3, document images in CORD are receipt images without creases or warping, which requires the model to be capable of handling text information but also visual and spatial information.
In the following sections, we provide detailed analysis and share insights on various aspects of OOD detection performance for document understanding models under different OOD detection methods.
Further details on the setup are provided in Appendix A.
3. A RE PRETRAINED MODELS SUFFICIENT FOR OOD DETECTION ?
As shown in Sec. 2.1, most domain processing models deployed in the real world are pretrained on a largescale dataset.
Naturally, one may expect pre-trained models to be robust to OOD data when equipped with competitive OOD detection methods.
To better understand the role of pretraining, we first provide more nuanced discussions on the following questions: 1) Are models equally robust to in-domain and out-domain
OOD inputs?
2) How does model modality impact OOD detection performance?
We consider a wide range of models pretrained on pure-text/image data ( e.g ., ImageNet, Wikipdeia, etc ).
A detailed description of these models can be found in Appendix A.1.2.
During finetuning, we combine the pretrained model with a classifier and finetune on RVL-CDIP (ID).
For models before and after finetuning, we extract the final feature outputs as the feature embeddings and use the same KNN+ score (Sun et al., 2022) for OOD detection.
The results are shown in Figure 4.
We observe the following trends.
First, finetuning largely improves OOD detection performance for both in-domain and out-domain OOD data.
Pretrained models, despite the fact that they have “seen” a diverse collection of data during pretraining, do not yield sufficient OOD robustness.
The same trend holds broadly across models with different modalities.
Second, the improvement of finetuning is less significant for out-domain OOD data.
For example, the AUROC on
Receipt (out-domain OOD) for pretrained ViT model is 97.13, whereas finetuning only improves by 0.79%.
This suggests that pretrained models do have the potential to separate data from different domains due to the diversity of data used for pretraining, while it remains hard for pretrained models to perform finer-grained separation for in-domain OOD inputs.
Therefore, finetuning is beneficial for improving both types of OOD detection performance as a consequence of improved feature representation.
R e s N e t
0 S w i n B a s e
V i T B a s e
R o B E R T a B a s e
L o n g f
o
r
m e r B a s e
A U R O C ( % ) 48.
[Figure] [Figure]
[Figure]
PretrainedFinetuned
R e s N e t
0 S w i n B a s e
V i T B a s e
R o B E R T a B a s e
L o n g f
o
r
m e r B a s e 85.
96.01 Out-domain OOD (Sci.
Poster)
PretrainedFinetuned
R e s N e t
0 S w i n B a s e
V i T B a s e
R o B E R T a B a s e
L o n g f
o
r
m e r B a s e 97.
Out-domain OOD (Receipt)
PretrainedFinetuned
[Figure]
To make the analysis more thorough
[Figure]
we have two additional in-domain OOD settings: (1) select the classes the model performs well on, as in-domain OOD categories; (2) randomly select classes as OOD categories.
As shown in Appendix (Table 10 and Table 11), we can see that finetuning also improves both types of
OOD detection, which further reaffirm our conclusion.
We also visualize the optimal transport dataset distance (OTDD) (Alvarez-Melis & Fusi, 2020) between in-domain and out-domain OOD datasets in Appendix (Fig.
10(b)
and Fig. 10(c)).Please refer to the Appendix for more details. 3. T HE IMPACT OF PRETRAINING DATA ON ZERO - SHOT OOD DETECTION In the previous section, we analyze the impacts of finetuning for OOD detection where the pretraining dataset is fixed and unrelated to documents.
Next, we dive deeper and study the impacts of pretraining dataset on zero-shot OOD detection.
For each model, we adopt the same pretraining objective while adjusting the amount of pretraining data.
Specifically, we increase the data diversity by appending 10, 20, 40, and 100% of randomly sampled data from IIT-CDIP dataset (around 11M) and pretrain each model.
After pretraining, we measure OOD detection performance with KNN+ score based on feature embeddings.
For out-domain OOD data (Fig. 5, right), increasing the amount of pretraining data can significantly improve the zero-shot OOD detection performance (w.o. finetuning) for models across different modalities.
This further verifies our previous hypothesis that pretraining with diverse data is beneficial for coarse-grained OOD detection, such as inputs from different domains ( e.g ., color schemes).
On the other hand, for in-domain OOD inputs, even increasing the amount of pretraining data by over 40% provides negligible improvements (Fig. 5, left).
This also suggests the necessity of finetuning for improving in-domain OOD detection.
We further explore zero-shot OOD detection by removing the potential OOD categories from IIT-CDIP. In practice, we first adopt the LayoutLMv1 finetuned on RVL-CDIP as the classifier for predicting labels for all IIT-CDIP document images. Fig.
5(b) shows the distribution of the predicted classes on IIT-CDIP.
Next, we remove the “ OOD ” categories from the IIT-CDIP data and pretrain two models (RoBERTa and LayoutLMv1) with 10, 20, 40, and 100% of randomly sampled data from the filtered IIT-CDIP.
Fig. 5(c) shows our zeroshot OOD performance.
Note that we do not show 0% in Fig. 5(c) since we pretrain LayoutLMv1 from scratch.
For RoBERTa, we start from the public pretrained model and see a similar trend in Fig. 5(c) – the influence of pretraining for those well-pretrained language models is minor for in-domain OOD detection since there is a considerable gap between OCR words and pure-text data.
E.g ., words in a document are spatially arranged, while words in text corpus are arranged sequentially.
In contrast, pretraining data has a bigger impact on those models trained from scratch – the zero-shot performance of LayoutLMv1 increases when more pretraining data is added.
We provide more details in the Appendix (Table 4 and Table 5).
[Figure]
3. I NVESTIGATING SPATIAL - AWARE MODELS FOR OOD DETECTION
In previous sections, we mainly focus on mainstream text-based and vision-based models to analyze the effects of finetuning and pretraining on in- and out-domain OOD detection.
Next, we largely expand the scope of our study by incorporating models tailored to document processing, which we refer to as spatialaware models, such as layoutLM, LayoutLMv3, and UDoc.
Moreover, given finetuned models, we are able to compare the performance of logit-based scores and distance-based scores.
Some key comparisons are shown in Figure 6.
Please refer to the Appendix for full results.
[Figure]
Distance-based vs. logit-based score
We can see that simple KNN-based score outperforms logit-based scores for both in-domain and out-domain OOD data.
The trend holds consistently across models with different modalities.
Moreover, spatial-aware models demonstrate both stronger OOD detection performance for in and out-domain OOD.
For example, with the best scoring function (KNN+), compared to RoBERTa, LayoutLMv3 improves the average AUROC by 7 . 09 % for out-domain OOD and 7 . 54 %.
The significant improvement suggests the value of spatial and visual information in improving OOD robustness for document data.
Note that despite this paper mainly comparing the logit-based and distance-based scores, we need to be aware that Gradient-based score has also been proposed for OOD detection.
We also report the GradNorm (Huang et al., 2021)
OOD detection score in Appendix A.3 and achieves similar performance as logit-based scores.
3. T OWARDS SIMPLE AND EFFECTIVE SPATIAL - AWARE ADAPTORS
Spatial-aware models tailored for documents such as LayoutLM rely on spatial information and demonstrates superior OOD detection performance.
This brings us a question: given the abundance of well-pretrained large-scale language models on text data such as RoBERTa, is there a simple and effective method that allows us to exploit the pretrained language model to document inputs for effective OOD detection?
Next, we show that by enhancing transformer-based pretrained models with a spatial-aware adapter module, we can achieve good performance with minimal code edits.
Spatial-aware adapter Given a public pretrained RoBERTa model, depending on the position of the adaptor, we consider two architectures: 1) appending the adapter to the word embedding layer, denoted as SpatialRoBERTa (pre).
It requires pretraining and finetuning, as illustrated in the top row in Fig.
7; 2) appending the adapter to the final layer, denoted as Spatial-BoBERTa (post).
As the model can utilize pretrained textual encoder, it only requires finetuning (illustrated in the bottom row).
In the following, we only discuss SpatialRoBERTa (pre).
Full results for both Spatial-RoBERTa variants are in the Appendix.
We freeze the word embedding layer during pretraining for the following considerations: 1) word embeddings learned from large-scale corpus already cover most of those words from documents; 2) pretraining on documents without strong language dependency may not help improve word embeddings.
For example, in semi-structured documents ( e.g ., forms, receipts), language dependencies are not be as strong as rich-text documents ( e.g ., letters, resumes), which may degenerate the learned word representations.
In practice, each word has a normalized bounding box (x0, y0, x1, y1), where (x0, y0) / (x1, y1) corresponds to the position of the upper left / lower right in the bounding box.
Each coordinate is fed into an embedding layer and outputs a position embedding.
All position embeddings are added to the initial word embedding to form a new spatial-aware embedding.
Spatial-RoBERTa significantly outperforms RoBERTa To verify the effectiveness of Spatial-RoBERTa,
We compare OOD detection performance for pre-trained and fine-tuned models.
The results are shown in
Fig. 8.
Spatial-RoBERTa significantly improves the OOD detection performance, especially after finetuning.
Specifically, compared to RoBERTa, Spatial-RoBERTa improves AUROC by 9 . 20 % for in-domain OOD and 4 .
95 % for out-domain OOD data.
This further verifies the importance of spatial-awareness for OOD detection in the document domain.
[Figure]
Spatial-RoBERTa is for
ID classification and OOD detection Beyond
OOD detection performance, we also examine
ID classification accuracy and plot the two metrics for all the models with different modalities in Fig.
We find that there exists a positive correlation between ID accuracy and OOD detection performance (measured by AUROC) for both in-domain and out-domain OOD.
Moreover, spatial-aware models display superior ID accuracy and OOD robustness compared to text-only and vision-only models.
Finally, our SpatialRoBERTa provides a simple and effective solution that greatly improves upon RoBERTa and matches the performance of models with specific architecture such as LayoutLM.
Specifically,
Spatial-RoBERTa Large achieves 97.37 ID accuracy, which is even higher than LayoutLM (97.28) and UDoc (97.36).
Furthermore, since our Spatial-RoBERTa Base and Spatial-RoBERTa Large freeze the word embed- ding during pretraining, it can learn spatial-aware feature target document data while keeping the word embedding fixed, thus reduce the trainable model size and reduce the training cost.
[Table]
Accuracy (ID) AUROC (Sci. Poster) R o B E R T a - b a s e
L o n g f
o
r
m e r - b a s e R o B E R T a - b a s e ( + ) R e s N e t L o n g f o
r m e r - b a s e ( + ) V i T - b a s e V i T - B a s e ( + ) S w i n - b a s e
S p a t i a l - R o B E R
T a - b a s e
L a y o u t L M - b a s e U D
o c
S p a t i a l - R o B E R
T a - l a r g e
L a y o u t L M v 3 - b a s e
P e r c e n t a g e
ID Acc.
and Out-domain (Receipt) OOD AUROC
Accuracy (ID) AUROC (Recipt)
C ONCLUSION AND O UTLOOK
This paper presents a large-scale study of various methods for quantifying OOD robustness across different data modalities and models for document domains.
Our key novelties include a large-scale investigation of OOD robustness in the document domain and a simple yet powerful spatial-aware adapter for transformer-based language models.
We start from document classification and explore the pretrained models for document OOD robustness.
A variety of substantial experiments in different settings demonstrates that pretraining datasets and tasks greatly impact OOD detection performance.
Notably, OOD samples in the document domain are more accessible to identify in the feature space than in the logit space.
Investigations from various perspectives explain certain intriguing phenomena and inspire more research on evaluating
OOD robustness towards more reliable document understanding models.
A A PPENDIX A. D OCUMENT C LASSIFICATION A.1. D ATASETS
RVL-CDIP consists of 320K/40K/40K training/validation/testing images under 16 categories.
We selectcategories and treat them as ID (In-Domain) data.
We extract the text and layout information with Google
OCR engine 3 which provides both tokens and text blocks along with their corresponding bounding boxes.
Most recent models take the full IIT-CDIP as pretraining data and finetuning on RVL-CDIP.
However, it is not reasonable for the OOD setting since RVL-CDIP is a subset of IIT-CDIP.
To make OOD results more reliable, in our experiments, we exclude the RVL-CDIP from the IIT-CDIP during pretraining. We measure the distance between in-domain and out-domain datasets via OTDD 4 . We first visualize the OTDD distance between ID and the OOD data (in-domain and out-domain datasets in our main paper) in
Fig. 10(a). During analysis, we sample the maximum number of 1000 images from each data and calculate the distance between datasets. It can be seen that there is a clear gap between in-domain data and out-domain data.
To make the analysis more thorough, we have two additional in-domain OOD settings: (1) select the classes the model performs well as OOD data; (2) randomly select classes as OOD data.
Fig.
10(a)
and
Fig.
10(c) show the dataset distance.
As for the other two selection strategies, we can see that the domain gap is not as clear as the subset we selected for the main experiments.
Interestingly, for those rare-word documents, such as file-folder and advertisements, the dataset distances are larger than documents with rich words.
The background colors and layouts may yield a big distinction for those documents.
[Equation]
[Table]
[Table]
p
[Table]
a l T
[Table]
n
[Table]
p
[Table]
t
D
[Table]
t
[Table]
s e t D
[Table]
t
[Table]
[Table]
[Table]
[Table]
D )
[Table]
V
[Table]
D I P
[Table]
OO D
[Table]
L
[Table]
tt e r )
R V L
[Table]
C D I P ( OO D
[Table]
d w
[Table]
i
[Table]
n )
[Table]
L
[Table]
D
[Table]
P
[Table]
OO D
[Table]
r t i
[Table]
n t )
[Table]
L
[Table]
D
[Table]
P
[Table]
D
[Table]
o )
[Table]
i
[Table]
P
[Table]
t e r ( OO D )
[Table]
D
[Table]
RVL-CDIP (OOD, Memo)
Sci.
Poster (OOD)
CORD (OOD) 123.8 853.4 1014.8 6118.5 948.4 2745.6 10850.
10000 O p t i m a l T r a n s p o r t D a t a s e t
D i s t a n c e (c) OOD (Random selection). A.1. M ODELS All models reported in Fig. 2(b), except for UDoc, are initialized with model pretrained weights from Huggingface 5 and finetune on the full RVL-CDIP training set.
During finetuning, we train those models on
RVL-CDIP with the cross-entropy loss.
Models are optimized with Adam optimizer (Kingma & Ba, 2014) for 30 epochs with a batch size of 50 and a learning rate of 2 × 10 − 5 on 8 A100 GPUs.
We list the hyperparameters of models used in our paper as follows:
3 https://cloud.google.com/vision/docs/ocrhttps://github
[Figure]
[Figure] Language-only: (1) BERT and RoBERTa.
We adopt the RoBERTa Base (12 layer, 768 hidden size) and BERT Base (12 layer, 768 hidden size) as the backbone and set the maximum sequence length to 512.
For RoBERTa, the classifier is composed of two linear layers followed by a tanh activation function.
(2) Longformer.
We also adopts the Longformer Base (12 layer, 768 hidden size) as the backbone and sets the maximum sequence length to 4,096.
Vision-only: (1) ResNet50:
This model adopts the ResNet50 (pretrained on ImageNet-1k) as the backbone.
We finetune it at resolution 224 × 224.
(2) ViT:
This model adopts ViT Base (vit-base-patch16-224, pretrained on ImageNet-21k) as the visual backbone.
We finetune it at resolution 224x224.
(3) SwinB: This model adopt swin transformer (swin-base-patch4-window7-224-in22k, pretrained on ImageNet-21k) as backbone.
We finetune it at resolution 224 × 224.
Layout+Language: (1) LayoutLMv1: This model adopts the LayoutLM (layoutlm-base-uncased, 12 layer, 768 hidden size, pretrained on IIT-CDIP ) as the backbone.
We set the maximum sequence length to 512.
(2) Spatial-RoBERTa Base (Pre): This model is combines our spatial-aware adapter to the pretrained
RoBERTa Base model.
The adapter is applied to the word embedding layer.
During pretraining, we freeze the pretrained word embedding and optimize the spatial-aware adapter and transformers.
During finetuning, we optimize all the parameters.
(3) Spatial-RoBERTa Base (Post): Instead of inserting the spatial-aware adapter in the input layer, this model combines the spatial-aware adapter at the transformers’ output layer.
Vision+Language+Layout: (1) LaytouLMv3:
This model adopt the LayoutLMv3 (layoutlmv3-base,layer, 768 hidden size, pretrained on IIT-CDIP) as the backbone. 21)
UDoc: This model follows the design of UDoc.
The only difference is the sentence encoder, we adopt a smaller version of the pretrained sentence encoder (all-MiniLM-L6-v2, 6 layer, 384 hidden size) instead of the sentence encoder (bert-base-nli-meantokens, 12 layer, 768 hidden size) used in their paper. A. B EYOND D OCUMENT C LASSIFICATION
[Figure]
Going beyond document classification, we explore OOD detection for two entity-level tasks: document entity recognition and document object detection.
Basic units such as text, tables, and figures in the document are the objects that need to be detected and recognized.
Document entity recognition aims to predict the label for each semantic entity with given bounding boxes.
Document object detection is an object detection task for document images.
Specifically, we denote the input as x , the bounding box coordinates associated with object instances in the image as b ∈ R 4 , and use the model with parameters θ to model the bounding box regression p θ ( b | x ) and the label classification p θ ( y | x, b ) .
Given a test input ˆ x , the OOD detection scoring function for entity detection and recognition can be unified as S (ˆ x, ˆ b ) , where ˆ b denotes the object instance predicted by the object detector.
In particular, for document entity recognition, since the bounding boxes are provided, the OOD score can be simplified as S (ˆ x, ¯ b ) , where ¯ b is the given object instance.
A.2. D ATASETS
Document Entity Recognition
The original FUNSD (Jaume et al., 2019) dataset contains 149/50 training/testing images.
we treat entities with category other / header as the OOD entities.
After doing the split, if we treat other as OOD, we have a total number of 8,330/1,019 ID/OOD entities in total.
Otherwise, if we treat header as OOD, we have 8,981/368 ID/OOD entities in total.
Document Object Detection PubLayNet
(Zhong et al., 2019) contains 336K/11K training/validation images with 6 category labels ( text , title , list , figure , and table ).
The original IIIT-AR-13K (Mondal et al., 2020) contains ( table , figure , natural image , logo , and signature ).
In our paper, considering the overlap between
IIIT-AR-13K and PubLayNet, we select those images that contain Natural Image as the OOD test set.
After filtering, we have 2,880 OOD entities across 1,837 document images.
We consider three ID datasets in this experiment.
(1) PubLayNet:
This is the original PubLayNet dataset.
We treat all the entities in training/validation images as ID entities.
(2) Considering the domain shift between ID data (PubLayNet) and OOD data (IIIT-AR-13K).
We combine the PubLayNet training data with the images from IIIT-AR-13K with overlapping annotations ( table and figure ) and train the object detection model.
A.2. M ODELS
Document Entity Recognition Fig.
13 illustrates the entity recognition models used in this paper.
We consider the entity on regions instead of tokens since regions contain richer semantic information.
As for the pretrained model, we adopt UDoc (pretrained on IIT-CDIP) since it models the inputs at the region level.
Based on UDoc framework, we develop the following models.
Vision/Vision+Layout: (1) ResNet-50: This model is composed of the ResNet-50 from pretrained UDoc.
It adopts the RoI pooling followed by a classifier to extract the entity features.
(2) ResNet-50+Position: This model also adapts UDoc pretrained ResNet-50.
It further improves the RoI features to be spatial-aware by adding position embedding, where the position embeddings are mapped from bounding boxes via a linear mapping layer.
Language/Language+Layout: (1) Sentence BERT: This model adopts the language branch of UDoc and appends the classifier to the output of the sentence encoder.
(2) Sentence BERT+Position: This model is close to the above model but adds position embedding to the sentence embeddings.
Vision+Language+Layout: (1) ResNet-50+sentence BERT: This model follows the same framework as
UDoc, but replaces the sentence encoder in their original design with a more miniature sentence encoder (all-MiniLM-L6-v2).
(2) SwinT+Sentence BERT: This model replaces the ResNet-50 visual backbone with a pretrained Swin tiny model (swin-tiny-patch4-window7-224) adopted from the Huggingface.
All the models are finetuned with cross-entropy loss for 100 epochs with a learning rate of 10 − 5 and batch size of 8 on one A100 GPU.
Document Object Detection Two object detection models are considered in this paper: (1) Vanilla FasterRCNN: This model is the Faster-RCNN with ResNet-50 visual backbone.
(2) Faster-RCNN with VOS:
This model enhances above model with VOS 6 .
Following their paper, we use 1,000 samples for each ID class to estimate the class-conditional Gaussians.
We train detection models with the Detectron2 framework (Wu et al., 2019).
Models are trained for 180k iterations with a base learning rate of 0.01 and a batch size of 8.
Mean average precision (MAP) @ intersection over union (IOU) [0.50:0.95] of bounding boxes is used to measure the performance. [Figure] [Figure]
A.2. E XPERIMENTAL SETUP
For document entity recognition, we construct ID and OOD datasets from FUNSD.
Each semantic entity includes a list of words, a label, and a bounding box.
The standard label set for this dataset containscategories: question , answer , header , and other .
In this paper, we select the entity labeled as other or header category as OOD.
And the entities belonging to the other three categories are ID.
Instead of treating entity recognition as a named-entity recognition problem, we follow UDoc and solve this problem at the semantic region level.
We replace the sentence encoder in UDoc with a smaller sentence encoder (all-MiniLML6-v2 7 ) from Huggingface (Wolf et al., 2019).
We also have the following model variants to verify the effectiveness of each modality combination: textual-only, visual-only, textual+spatial, visual+spatial, and visual+textual+spatial.
For document object detection, we use PubLayNet as the ID dataset.
We construct the OOD dataset from
IIIT-AR-13K. Unlike PubLayNet, where the documents are scientific articles, IIIT-AR-13K is a dataset for graphical object detection in business documents ( e.g ., annual reports).
Hence, there exists an obvious domain gap between these two datasets.
We select natural images as the OOD entity and filter images that contain the OOD entity.
We first adopt the vanilla Faster RCNN with ResNet-50 backbone for document object detection as the baseline model.
We also enhance Faster RCNN with VOS (Du et al., 2022), a recent unknown-aware learning framework to improve OOD detection performance for natural images.
A.2. O BSERVATIONS
To identify the entity type, models should not only understand the words but also require spatial and visual reasoning ability.
We summarize our findings on document entity recognition in Fig.
14 (a) and describe them in more detail in Table 1.
We can see that models can better predict the entity type with the help of the spatial position and also achieves better OOD robustness.
Considering the weak language dependency between entities, it is not supervising that vision-based models achieve better performance than text-based models.
We can see that UDoc with ResNet-50 achieves the best performance on two OOD test sets, illustrating that visual information plays a major role in increasing the discrimination of entities with similar semantics.
We summarize our findings on document object detection in Fig.
14 (b) and describe them in more detail in Table 2.
We can see that the OOD detection performance is further improved by introducing document images from IIIT-AR-13K with the same ID annotations as training data.
In Fig.
15, we visualize some document entity recognition OOD detection results.
In Fig.
16, we visualize the prediction on sample OOD images, using object detection models trained without VOS (top) and with
VOS (bottom), respectively.
There is a clear difference between PubLayNet and IIIT-AR-13K – natural image annotations and entities rarely exist in PubLayNet.
We can see that vanilla Faster RCNN trained on
PubLayNet produces false positives when applied to the OOD document image from IIIT-AR-13K. After introducing the unknown-aware learning method optimized for both ID and OOD, as shown in Table 2, the
FPR95 reduces while preserving the mAP on the ID data.
This experiment indicates that bringing uncertainty estimation into the entity detection training procedure can improve the reliability of the document object detection system.
[Figure]
A. A DDITIONAL E XPERIMENTAL R ESULTS
• Table 1 corresponds to the results shown in Fig. 15 and Fig. 14(a). • Table 2 corresponds to the results shown in Fig.
16 and Fig.
14(b).
• Table 3 and Table 7 correspond to the results shown in Fig.
5(a).
• Table 4 and Table 5 correspond to the results shown in Fig. 5(c).
• Table 6 corresponds to the results shown in Fig. 8 and Fig. 9. • Table 9 and Table 8 correspond to the results shown in Fig. 4 and Fig.
9. • Table 10 and Table 11 correspond to the analysis in Sec.
3.1. • Table 12 corresponds to the results shown in Fig.
[Table] [Table] [Table]
R o BE
R T a B a s e
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
.09 KNN
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
R o BE
R T a B a s e
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
88.21 KNN70.
[Table]
14 KNN71.
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
12 KNN90.
[Table]
93.05 KNN91.
92.04 KNN91.
91.08 KNN92.
[Table]
L ayo u t L
M v1 B a s e
[Table] [Table]
L ayo u t L M v1 B a s e [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
t i a l - R o BE R T a
[Table] [Table]
i a l - R o BE
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Sp a t i a l - R o BE R T a
L a r g e ( P re )
[Table]
[Table]
[Table]
V i T B a s e
[Table]
[Table]
[Table] [Table]
PretrainonImageNet
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
PretrainonImageNet –
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
.22 KNN
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
97.92 KNN63.
97.67 KNN61.
97.24 KNN60.
96.82 KNN58.
96.39 KNN58.
96.03 PretrainonImageNet –
KNN98.
97.13 KNN98.
96.80 KNN98.
[Table]
[Table] [Table] [Table] [Table]
PretrainonImageNet
[Table]
PretrainonImageNet
[Table]
V i T B a s e
[Table]
PretrainonImageNet
[Table]
[Table]
L o n g f
o r
[Table] [Table] [Table] [Table]
PretrainonImageNet
[Table]
[Table]
[Table]
V
[Table]
PretrainonImageNet –
[Table]
[Table]