Layout-Aware Text Representations Harm Clustering Documents by Type (2020)

Abstract: Clustering documents by type—grouping invoices with invoices and articles with articles—is a desirable first step for organizing large collections of document scans. Humans approaching this task use both the semantics of the text and the document layout to assist in grouping like documents. LayoutLM (Xu et al., 2019), a layout-aware transformer built on top of BERT with state-of-the-art performance on document-type classification, could reasonably be expected to outperform regular BERT (Devlin et al., 2018) for document-type clustering. However, we find experimentally that BERT significantly outperforms LayoutLM on this task (p <0.001). We analyze clusters to show where layout awareness is an asset and where it is a liability.

## introduction organizations are inundated by paperwork, often in the form of pdfs. automated processing can help to organize and extract information from these documents, but the right process for a given document depends on its type: invoices are handled differently than contracts, for example. document classification by type enables such a system; however, it requires training data for all of the desired classes, and finding such data to fit a given business's needs is difficult. there is no one-size-fits-all ontology of document types. while some types, such as invoices, may be common across industries, others, such as loan applications or home-inspection reports, are domain-specific. users wishing to define their own classes will benefit from a system that enables them to group their own documents. to help with this, the present work addresses the task of clustering documents by type. humans grouping documents by type can use both the text and the appearance of documents. for example, we can distinguish a gas bill from an article at a glance, but we need to read at least a few words to determine whether a dense, two-column document is an article or a warranty. we therefore expect that a hybrid document representation that combines layout and text information should outperform a text-only representation when clustering documents by type. layoutlm @xcite is such a hybrid system and achieves state-of-theart performance for document-type classification, outperforming text-only baselines. we therefore hypothesized that layoutlm would also outperform these baselines for document-type clustering. sections 3 and 4 describe the systems we compared and the experiments we used to try to confirm this hypothesis. however, the main contribution of this work is experimental evidence of the opposite: layoutlm performed significantly worse than a simple bert baseline on this task (section 5). analysis of output clusters (section 5.1) helps to explain this unexpected result. document-type classification classification of documents by type has frequently been treated as an image classification problem. many works have used varying cnn architectures @xcite @xcite @xcite or other vision-based techniques @xcite . some works have combined vision and nlp for document-type classification, using ocr for text extraction. noce et al. (2016 ) assigned the most relevant words unique colors, then filled the bounding boxes of those words with the corresponding color, enabling the cnn processing the image to "see" the word. asim et al. (2019) provided the most important words as features to a cnn, later combining the output with an image stream that used an inceptionv3 cnn architecture. dauphinee et al. (2019) concatenated the output of a cnn image classifier with a multilayer perceptron bagof-words classifier, then fed the concatenation to a meta-classifier. ferrando et al. (2020) used an ensemble of a bert classifier and efficientnets cnns. audebert et al. (2020) concatenated image features (from a mobilenet v2 cnn) with text features (generated by passing fasttext embeddings for the text through a 1d cnn) to form the input to a multilayer perceptron. cosma et al. ( 2020 ) used text to help pretrain part of their classifier: they performed lda to determine documents' topics, then trained their cnn to try to predict those topics using only the document image. they ultimately used the cnn as part of a model to predict document type using the image only. all of these systems are supervised, whereas this work addresses unsupervised clustering. csurka et al. (2016) trained models on rvl-cdip, then used those models to generate representations for clustering other document-type datasets. abuelwafa et al. (2019) used unsupervised feature learning to improve their representations of document images for clustering. they applied transformations to document images to generate surrogate classes, then trained a cnn to classify them. they used that trained cnn to generate representations of document images for clustering. there is, to our knowledge, no previous work clustering rvl-cdip. ## systems we compare layoutlm and bert, as well as a tf-idf baseline (sklearn's foot_0 @xcite implementation with default hyperparameters). in each case, we use the specified system to generate one vector representation for each document image, then cluster using sklearn's k-means, with k set to the number of gold classes plus one. bert @xcite is a transformerbased bidirectional model that generates contextualized word embeddings for a sequence of words. the input to a bert model for the i-th token in the sequence is a sum of (a) its token embedding; (b) a position embedding for position i; and (c) a segment embedding indicating whether the token is in the first or second segment of the input sequence. layoutlm @xcite is a bert-like transformer model modified to generate layoutaware contextualized word embeddings. in place of bert's single positional embedding, layoutlm adds positional embeddings for the x-and ycoordinates of a bounding box around the token. the token's embedding thus incorporates its twodimensional location on the page and its size. this architecture achieves state-of-the-art performance for supervised classification by document type. both bert and layoutlm output a vector for each token in the input sequence plus the special [cls] token. however, k-means, like most clustering algorithms, requires a single vector representation of each example. classifiers use the [cls] embedding as a single-vector representation for the entire sequence. however, prior work @xcite has shown that, for bert without fine-tuning, this is not a good representation of the semantics of the entire sequence. other options include combining all of the vectors in the output sequence by either averaging or max pooling-set the i-th value in the output vector equal to the max i-th value over all of the sequence vectors. for bert, we use the average as our representation, since @xcite showed it captured semantic similarity better than the [cls] token. for layoutlm, we try all three methods. ## experiments we evaluate on rvl-cdip foot_1 @xcite , scanned tobacco-litigation documents from the illinois institute of technology complex document information processing (iit-cdip) collection, labeled with type, such as letter or invoice. the complete class list appears in we use layoutlm's publicly-released code and base model for experiments. 4 this model was pretrained on iit-cdip, excluding documents in rvl-cdip. for bert, we use the transformers package 5 with the bert-base-uncased model, pretrained on books and wikipedia. because layoutlm's masked language model pretrained on documents from the same domain, while bert's did not, the dataset could favor layoutlm. we calculate f 1 and adjusted rand index (ari) for each system, using manning et al. ( 2008 )'s definitions of true and false positives and negatives. we use sklearn @xcite 's implementation of ari. we report the mean over 5 runs and use a two-tailed t-test to determine whether systems differ significantly from the bert baseline. ## results results are shown in in contrast to prior work on bert, where the [cls] token was a worse representation than averaging (reimers and gurevych, 2019; wang and kuo, 2020), the best-performing layoutlm system used the [cls] token embedding. we suspect this is because averaging or max-pooling lay-outlm vectors blends together bounding box information for all tokens, erasing the benefits of a layout-sensitive transformer. in light of these results, we also tested [cls] token and max-pooling for bert on this task. consistent with prior work, averaging outperformed both; see all of these scores are low, especially in comparison to classification results. the comparison is misleading, of course, since classification requires training data, and clustering addresses the case where such data is not available. nevertheless, much improvement will be required before document-type clustering is useful for practical applications. ## conclusion a model can learn when to leverage each. thus, layoutlm performed quite well at classifying documents by type. but when clustering, there is no model to indicate how to weight features in determining document similarities. in this context, layout information significantly harms performance. future work should explore ways to incorporate benefits of layout information into a representation while limiting its harm, as well as how layout information affects tasks that fall between classification and clustering, such as semi-supervised learning. such questions must be answered for documenttype clustering to become practical.