In this paper we introduce a novel approach to the problem of understanding documents where  the local semantics is influenced by non-trivial layout. Namely , we modify the Transformer  architecture in a way that allows it to use the graphical features defined by the layout , without the need to re-learn the language semantics from scratch , thanks to starting the training process from a model pretrained on classical language modeling tasks .
We introduce a new simple approach to the problem of understanding documents where non-trivial layout influences the local semantics . To this end , we modify the Transformer encoder architecture in a way that allows it to use layout features obtained from an OCR system , without the need to re-learn the language semantics from scratch . We augment the input of the model only with the coordinates of token bounding boxes, avoiding the use of raw images. This leads to a layout-aware language model which can be then fine-tuned on downstream tasks. The model is evaluated on an end-to-end information extraction task using four publicly available datasets: Kleister NDA, Kleister Charity, SROIE and CORD. We show that it achieves superior performance on datasets consisting of visually rich documents, at the same time outperforming the baseline RoBERTa on documents with flat layout (NDA F1 increase from 78.50 to 80.42). Our solution ranked 1st on the public leaderboard for the Key Information Extraction from the SROIE dataset, improving the SOTA F1-score from 97.81 to 98.17 .
R
In this paper we introduce a novel
We introduce a new simple
A
non-trivial layout influences
R
is influenced by non-trivial layout. Namely
. To this end
A
encoder
R
the graphical features defined by the layout
layout features obtained from an OCR system
R
, thanks to starting the training process from a model pretrained on classical language modeling tasks
. We augment the input of the model only with the coordinates of token bounding boxes, avoiding the use of raw images. This leads to a layout-aware language model which can be then fine-tuned on downstream tasks. The model is evaluated on an end-to-end information extraction task using four publicly available datasets: Kleister NDA, Kleister Charity, SROIE and CORD. We show that it achieves superior performance on datasets consisting of visually rich documents, at the same time outperforming the baseline RoBERTa on documents with flat layout (NDA F1 increase from 78.50 to 80.42). Our solution ranked 1st on the public leaderboard for the Key Information Extraction from the SROIE dataset, improving the SOTA F1-score from 97.81 to 98.17