A Simple yet Effective Learnable Positional Encoding Method for Improving Document Transformer Model (2022)

Abstract: Positional encoding plays a key role in Transformer-based architecture, which is to indicate and embed token sequential order information. Understanding documents with unreliable reading order information is a real challenge for document Transformer models. This paper proposes a simple and effective positional encoding method, learnable sinusoidal positional encoding (LSPE), by building a learnable sinusoidal positional encoding feed-forward network. We apply LSPE to document Transformer models and pretrain them on document datasets. Then we finetune and evaluate the model performance on document understanding tasks in form, receipt, and invoice domains. Experimental results show our proposed method not only outperforms other baselines, but also demonstrates its robustness and stability on handling noisy data with incorrect order information.

## introduction document understanding (in some context known as document intelligence, document ai) aims to extract, recognize, and understand information from document images. the performance of document understanding model is largely benefited from recent development of large scale pre-training technique on cross-modality data and effective transformer architectures @xcite . document transformer model, e.g. layoutlm @xcite , is pretrained from visually-rich document data which consists of text, layout, and visual information based on transformer architecture @xcite . recently, @xcite ; @xcite ; @xcite ; @xcite propose various approaches to further improve the performance of transformer models on more challenging document understanding tasks. different from recurrent and convolutional based structures, transformer based model does not encode relative or absolute position information ex-plicitly since it is solely based on order-invariant attentional mechanism. in the original transformer architecture @xcite , both learnable vector embedding and sinusoidal function are introduced as positional encoding methods for capturing positional information from input tokens. in order to improve positional representation ability, @xcite ; @xcite ; @xcite ; @xcite introduce several relative position strategies into attention computation steps in the transformer. along with sequential reading order from text, visually-rich documents contain more spatial information of text blocks which poses a greater challenge to understand rich semantic and spatial relationship information at the same time. to obtain text blocks from document images, current off-the-shelf methods are borrowing results from existing optical character recognition (ocr) engine while the reading order of text blocks is mostly arranged by a heuristic manner, top-to-bottom and left-to-right @xcite . for documents with complex layout, such as forms, invoices, or receipts, the performance of reading order is not consistent which always leads to irrelevant or embarrassing predictions @xcite . moreover, existing document transformer models suffer from huge performance degradation on noisy data with unreliable reading order information @xcite . therefore positional encoding plays an essential role in document transformer models, which is to encode position embedding from data with inherent reading or spatial information. thus, it is crucial to improve the robustness and learnability of position encoding methods, and therefore boost the model performance on noisy data with unreliable order and spatial information. in this paper, we propose a learnable sinusoidal position encoding method, lsp e, by building a learnable fully connected feed-forward sinusoidal positional encoding network. and we apply it to represent multidimensional position information in the document transformer model. compared with current discrete embedding layer in the transformer model, our method is numeric continuous for position scales which could improve positional representation of relative positions or distances between spatial elements. our approach keeps the advantage of extrapolability from sinusoidal function which could extend to longer position than training cases. in addition, we build a learnable sinusoidal position network, which helps the pretrained language model to be easily adapted to various downstream tasks effectively. we pretrain layoutlm with various positional encoding methods and other baselines. then we evaluate and compare the models' performance on document understanding downstream tasks. experimental results show that our lsp e method significantly outperforms other baselines and recent document language models on funsd, sroie and our in-house invoice datasets. in addition, we evaluate the model robustness on noisy data by utilizing global and local shuffling augmentation strategies. our method shows stable performance than other positional encoding methods with unreliable reading order information. furthermore, we visualize and analyze similarity of positional representation of each method from 1d to 2d positional embedding of our pretrained models. in summary, our contributions could be highlighted as follows: 1) we propose a simple and effective learnable positional encoding method with better learnability and extrapolability. it can be applied to any transformer based models to help them better encode and understand positional information. 2) we pretrain document transformer models with lsp e and other methods, and evaluate model performance on document understanding tasks. experimental results show our proposed method outperforms other baselines and recent sota approaches on funsd, sroie, and a largescale invoice dataset.3) by the ablation study of employing global and local block shuffling augmentations, our method demonstrates optimal performance and robustness on noisy data with unreliable reading order information. finally, our pretrained models with implementation of position encoding code will be publicly available. 2 background positional encoding methods in transformer in the original proposal of transformer architecture @xcite , both learnable vector and sinusoidal function are introduced as positional encoding methods and perform nearly identically in their downstream tasks. although sinusoidal version with predefined wavelength has unique extrapolability which allows to encode longer sequential position than pre-training samples, it does not always perform well on downstream tasks @xcite , due to the lack of learnability and flexibility. in practical, most pretrained language models, (e.g. @xcite ), utilize learnable vector embedding @xcite as positional representation. recently, several approaches are proposed to enhance positional representation by adding relative position information into attention score computation stage to improve performance of transformer based models @xcite @xcite . by leveraging relative positional encoding and other advanced pretraining techniques, @xcite and @xcite achieve state-of-the-art performance on multiple nature language understanding tasks. @xcite explore the position encoding method in vision domain and propose a learnable fourier feature to enhance positional encoding in transformer. it outperforms other methods on both accuracy and convergence speed with vision transformer @xcite based model. since it is nontrivial to modify or replace backbone of model structure during fine-tuning stage, some research works propose auxiliary tasks @xcite or data augmentation approaches @xcite to lever-age absolute or relative position information without modifying model structure. ## methodology in this section, we formulate our positional encoding method lsp e and introduce its applications on document transformer based language models. in order to evaluate its robustness and stability on noisy data with unreliable order information, we introduce two augmentation strategies: global and local text-block shuffling during fine-tuning stage. ## experiments we apply four positional encoding methods (learnv ec, sine, lsp esc, lsp e) to a representative transformer based model: layoutlm without visual feature. we conduct pretraining and finetuning on these models to identify the affect of different positional encodings to the performance of transformers on document understanding tasks. ## position embedding similarity analysis figure 6 demonstrates the position-wise cosine similarity of r 2d representation of five specific points to the rest positions in our pretrained models and in the official layoutlm. sine captures close similar embeddings only, where its 2d similary map decays rapidly from central point and shows sharp edge on the border. the official lay-outlm model shows boarder vision horizontally with proper spatial correlation, but still fail to capture long-term relations. our lsp e shows higher wave frequency on both x-and y-axes which tend to capture the long distance signals with obvious periodic pattern. ## conclusions in this paper, we propose a simple but effective learnable positional encoding method lsp e to improve the positional representation in transformer based models. by building a sinusoidal position feed-forward network, our method has better learnability and extrapolability in position representation. experimental results on funsd, sroie and an in-house invoice datasets clearly show the effectiveness of our method on document understanding tasks. by leveraging global and local shuffling augmentation methods and removing order information from inputs, we demonstrate our method substantially outperforms other positional encoding methods on noisy data with unreliable reading order. for future research, we will employ and evaluate our method on other tasks or modalities such as vision transformer @xcite .