Beyond Layout Embedding: Layout Attention with G aussian Biases for Structured Document Understanding (2023)

Abstract: Effectively encoding layout information is a central problem in structured document understanding. Most existing methods rely heavily on millions of trainable parameters to learn the layout features of each word from Cartesian coordinates. However, two unresolved questions remain: (1) Is the Cartesian coordinate system the optimal choice for layout modeling? (2) Are massive learnable parameters truly necessary for layout representation? In this paper, we address these questions by proposing Layout Attention with Gaussian Biases (LAGaBi): Firstly, we find that polar coordinates provide a superior choice over Cartesian coordinates as they offer a measurement of both distance and angle between word pairs, capturing relative positions more effectively. Furthermore, by feeding the distances and angles into 2-D Gaussian kernels, we model intuitive inductive layout biases, i.e., the words closer within a document should receive more attention, which will act as the attention biases to revise the textual attention distribution. LAGaBi is model-agnostic and language-independent, which can be applied to a range of transformer-based models, such as the text pre-training models from the BERT series and the LayoutLM series that incorporate visual features. Experimental results on three widely used benchmarks demonstrate that, despite reducing the number of layout parameters from millions to 48, LAGaBi achieves competitive or even superior performance.

## introduction structured document understanding (sdu) has gained significant research attention in the field of intelligent document processing @xcite @xcite . it focuses on extracting layout structures and contents from scanned or digital documents, leading to enhanced performance in several downstream tasks like form comprehension and receipt understanding. unlike conventional text understanding @xcite @xcite , sdu goes beyond comprehending serialized text and requires the ability to interpret documents with diverse layouts @xcite @xcite @xcite . documents with varying layouts often contain text fields positioned in different ways. to take advantage of existing pre-trained language models, early methods @xcite @xcite propose to directly add 2-d position embedding to the word embedding for each word as input for transformer. the position embedding encode the 2-d absolute coordinates (x 0 , y 0 , x 1 , y 1 ) of each word in the document through multi position encoding layers, where (x 0 , y 0 ) represents the upper left point and (x 1 , y 1 ) represents the lower right point of the bounding box for each word. some researches @xcite @xcite @xcite further proposed that absolute positions are inefficient for representing the spatial relationships between words. they employ relative positions between words to encode spatial relationships. for example, @xcite map the relative positions into embeddings, which are then multiplied with the semantic embedding of the word to calculate inter-word layout scores. this score is incorporated into the self-attention layers to combine semantic and layout features. despite the significant progress made, we argue that current methods, whether based on absolute or relative positioning, heavily rely on a large number of trainable parameters for position embeddings from cartesian coordinates, often comprising millions of parameters. this raises two unexplored questions: (1) is the cartesian coordinate system the optimal choice for layout modeling? (2) are massive learnable layout parameters truly necessary? concerning the former, various coordinate systems exist, including cartesian, polar, and spherical coordinates, yet previous research has solely focused on cartesian coordinates. regarding the latter, it is intuitively expected that words closer within a document should receive more attention. however, this simple inductive bias may not be effectively learned solely through gradient updates. in this paper, we present a unified investigation of the two aforementioned problems. regarding the choice of coordinate systems, we find that polar coordinates offer a more efficient representation than cartesian coordinates for expressing relative positions. by computing the differences in distance and angle between two words in polar space, polar coordinates outperform their cartesian counterparts by providing extra angle information. for layout learning, we discover that layout modeling can be achieved by a specific distribution: words closer in space receive higher layout scores, eliminating the need for extra position embeddings. combining these two choices, we propose lagabi (layout attention with gaussian biases). lagabi formulates pairwise spatial relationships between tokens using the distance and angle in the polar coordinate system. moreover, the distance and angle are fed into a 2-d gaussian distribution to output a layout score. we choose the gaussian distribution because it guarantees that the layout score decreases when either the distance or angle variables increase, making it the most commonly used distribution for this purpose. the layout score is then incorporated into the original self-attention as attention bias, resulting in a revised distribution that considers both text and layout features. we introduce trainable gaussian kernels to better align the semantic and layout scores at different scales, adding just 4× attention heads extra parameters. for instance, based on roberta @xcite with 12 attention heads, there are only 48 additional parameters that need to be learned for encoding layout features. extensive experiments demonstrate that lagabi achieves remarkable performance on diverse sdu benchmarks, including funsd @xcite , cord @xcite , and xfund @xcite , across both monolingual and multilingual scenarios. lagabi emerges as a versatile module that seamlessly integrates with transformerbased language models, such as bert (kenton and toutanova, 2019), roberta, and infoxlm @xcite , empowering them to effectively process structured documents and achieve significant performance gains of up to 27.01 points. additionally, lagabi can be incorporated into complex sdu models that leverage visual features, such as layoutlm @xcite , layoutlmv2 @xcite , and layoutlmv3 @xcite , leading to further performance enhancements and establishing new state-of-the-art results. most of the above approaches encode the 2-d absolute positions, ignoring the critical relative spatial relationships between words that are essential to textual semantic understanding. @xcite proposed to encode the spatial relationships as relative position embeddings, which are then multiplied with the semantic embedding of the token to calculate inter-word layout scores.this score is incorporated into the self-attention to combine semantic and layout features. geolay-outlm @xcite introduces geometric relations and brand-new geometric pre-training tasks in different levels for learning the geometric layout representation, whose geometric relations are largely dependent on some pre-defined manual rules. titl @xcite encodes the relative positions between words in a simpler manner, and it adopts linear layers to convert the 2-d discrete distance (implemented through bucketing) between words into attention biases motivated by t5 @xcite . such relative attention biases have also been employed by layoutlmv2 @xcite and layoutlmv3 @xcite , yielding notable results. different from them, we model the relative positions from a new perspective, i.e. introducing polar coordinates with distance and angle that are more efficient in representing relative spatial relationships. furthermore, we employ the extremely lightweight gaussian kernels to encode polar coordinates into attention biases, offering a more streamlined approach with fewer trainable parameters and aligning better with human intuition compared to linear layers. ## methodology structured documents typically contain both textual words and layouts, where textual words denote the main content of the document, while layouts record the organizational form of text. optical character recognition (ocr) as a classical technique for image document parsing can recognize the text as well as its locations. formally, given a document d, ocr identifies the text words @xmath0, where n is the number of words while c @xmath1, y 0 i , x 1 i , y 1 i ] represents the left-top and right-bottom coordinates of the bounding box that contains the i th word. an ideal model for document understanding should take both text words and their layout positions into consideration. the key to structured document comprehension, building upon the achievements of text understanding techniques such as transformer @xcite , lies in effectively representing document layouts while remaining compatible with language representation. therefore, there are two crucial problems that need to be solved: 1. how to model document layouts efficiently? 2. how to integrate the layouts into transformer to guide textual semantic understanding of the document content? in this paper, we introduce a novel layout modeling method for structured document understanding, namely layout attention with gaussian biases (lagabi). as shown in figure 1 , it mainly consists of two parts: modeling relative positions with polar coordinates and layout attention with gaussian biases. the former is responsible for modeling the 2-d relative positions between words by po-lar coordinates which are capable of representing inter-word spatial relationships; the latter focuses on transforming polar coordinates into attention biases, which will modify the original semantic query-key attentions between words into a more suitable distribution that accurately captures the underlying layout structure of documents. ## conclusion in this paper, we propose a model-agnostic and language-independent method that leverages layout attention with gaussian biases to encode the relative spatial positions for structured document understanding (sdu). specifically, we first model the inter-word spatial relationships using polar coordinates. then the query-key attention scores are revised by the gaussian biases that are related to their spatial relationships. our method can be applied to a series of transformer-based models with extremely few parameters, improving their performance for sdu tasks. experiments based on six transformer-based sdu models and three monolingual/multilingual benchmarks fully demonstrate the effectiveness of our proposal. this research provides new ideas for structured document understanding tasks, which are expected to promote the efficient development of document intelligence.