Skim-Attention: Learning to Focus via Document Layout (2021)

Abstract: Transformer-based pre-training techniques of text and layout have proven effective in a number of document understanding tasks. Despite this success, multimodal pre-training models suffer from very high computational and memory costs. Motivated by human reading strategies, this paper presents Skim-Attention, a new attention mechanism that takes advantage of the structure of the document and its layout. Skim-Attention only attends to the 2-dimensional position of the words in a document. Our experiments show that Skim-Attention obtains a lower perplexity than prior works, while being more computationally efficient. Skim-Attention can be further combined with long-range Transformers to efficiently process long documents. We also show how Skim-Attention can be used off-the-shelf as a mask for any Pre-trained Language Model, allowing to improve their performance while restricting attention. Finally, we show the emergence of a document structure representation in Skim-Attention.

## introduction more and more companies have started automating their document processing workflows by leveraging artificial intelligence techniques. this had lead to the emergence of a dedicated research topic, document intelligence foot_0 (di), which encompasses the techniques used to read, interpret and extract information from business documents. such documents span multiple pages and contain rich multi-modal information that include both text and layout. earliest approaches to analyzing business documents rely on rule-based algorithms @xcite , but the success of deep learning has put computer vision and natural language processing (nlp) models at the heart of contemporary approaches @xcite . with the massive impact of large pre-trained transformer-based language models @xcite , di researchers have recently started leveraging transformers. at the core of the transformer architecture is self-attention, a powerful mechanism which contextualizes tokens with respect to the whole sequence. while being the key to the success of transformers, it is also its bottleneck: the time and memory requirements of self-attention grow quadratically with sequence length. as a consequence, only short sequences can be processed (512 tokens or 1,024 at most), making it impossible to capture long-term dependencies. this is an important issue for di since texts can be very dense and long in business documents. to allow efficient training on very long sequences, there has been growing interest in building model architectures that reduce the memory footprint and computational requirements of transformers @xcite @xcite . this plethora of long-range transformers lie in one specific research direction: capturing long-range dependencies by reducing the cost of self-attention. these transformer architectures all operate on serialized texts, i.e. one-dimensional sequences of words, completely disregarding the document layout. however, layout, i.e. the physical organization of a document's contents, carries useful information about the semantics of the text and has a significant impact on readers' understanding @xcite . thus, ignoring the document layout leads to a considerable loss of information. to address this issue, an orthogonal direction that has gained traction recently is based on integrating layout information into transformer-based language models. joint pre-training of text and layout has allowed models to reach state-of-the-art performance in several downstream tasks concerning layout-rich documents @xcite @xcite . despite their effec-tiveness, these approaches all "read" the contents token by token to compute attention. we claim that one does not need to have read each word in a document page to be able to understand a specific paragraph. thus, we argue that, to efficiently process long documents, it is a waste of effort and computation to contextualize a token with respect to the entire input sequence. to shift towards processing long documents with awareness of their structure, we propose to take into account layout in a more intuitive and efficient way. first, we present a quick cognitive experiment wherein we show that layout plays a fundamental role in humans' comprehension of documents. in light of this experiment, we claim that one can already gather a lot of information from the layout alone. as a consequence, we propose skim-attention, a new self-attention mechanism that is solely based on the 2-d position of tokens in the page, independently from their semantics. to exploit this mechanism, we introduce skimformer and skimmingmask, two frameworks for integrating skim-attention into transformer models. skimformer is an end-to-end transformer language model that replaces self-attention with skim-attention. based on the tokens' spatial locations, skimformer computes the skim-attention scores only once, before using them in each layer of a text-based transformer encoder. skimformer can also be adapted to long-range transformers to model longer documents. conversely, skimming-mask uses skim-attention as a mask to sparsify attention in any transformer language model. each token is restricted to its k most attended tokens, as indicated by skim-attention, which allows for a smaller context length. in summary, our main contributions are as follows: ## preliminary experiments: human evaluation how much does the document layout help in comprehending long textual contents? how faster is it for humans to find information in documents when layout is provided? to answer these questions, we conduct a simple cognitive experiment wherein we measure the amount of time needed for human annotators to retrieve information from both formatted and plain-text documents. half of the time, they are given access to the full layout, and the other half, to plain text only (i.e., no layout nor formatting). 2 average standard deviation formatted 6.05 1.73 plain-text 15.18 9.06 we believe that machines could benefit from the the document layout, just like humans, as a strategy to retrieve information faster while expending less effort. in particular, layout information could be of great help in reducing the cost of self-attention in transformer models. ## proposed approach using common sense, and in light of the cognitive experiment previously reported, it is clear that thelayout is of utmost importance for humans to understand long documents. we propose to take into account layout by introducing skim-attention, a selfattention module that computes attention solely based on spatial positions. to process long and layout-rich documents, we propose different ways of integrating this mechanism into transformer architectures. ## conclusion we present skim-attention, a new structure-aware attention mechanism. we conduct extensive experiments to show the effectiveness of skim-attention, both as an end-to-end model (skimformer) and as a mask for any language model (skim-attention). we hope this work will pave the way towards a new research direction for efficient attentions. for future works, we will investigate how to integrate image features, and explore tasks that require capturing longer-range dependencies.