Training L ayout LM from Scratch for Efficient Named-Entity Recognition in the Insurance Domain (2025)

Abstract: Generic pre-trained neural networks may struggle to produce good results in specialized domains like finance and insurance. This is due to a domain mismatch between training data and downstream tasks, as in-domain data are often scarce due to privacy constraints. In this work, we compare different pre-training strategies for LayoutLM. We show that using domain-relevant documents improves results on a named-entity recognition (NER) problem using a novel dataset of anonymized insurance-related financial documents called PAYSLIPS. Moreover, we show that we can achieve competitive results using a smaller and faster model.

## introduction modern natural language processing pipelines heavily rely on pre-trained neural networks, primarily language models @xcite @xcite and context-sensitive embeddings @xcite @xcite . the development of neural architectures based on the attention mechanism @xcite allows to efficiently pre-train them on gpu using large datasets @xcite : most recent networks can contain several hundreds of billions parameters (e.g., @xcite . despite their experimental success, commercial use of pre-trained neural networks can be limited for the following reasons. firstly, downstream tasks in information retrieval may require to continuously analyze large amounts of data, which prevents the use of the largest models due to inference time bottleneck. secondly, applications in specific fields such as financial, medical or insurance, can forbid the use of api-based models due to privacy concerns. thirdly and lastly, authors may at some point decide to not publicly share latest versions of their models, or to change the license to forbid commercial use. 1 as such, it is increasingly important to ensure replicability and robustness to changes in training data (including for domain transfer) not only for scientific reasons, but also to ensure widespread commercial deployment. in this work, we study layoutlm @xcite for named-entity recognition (ner) on financial documents from the insurance domain. our aim is to understand how such a model can be used in a constrained setting: can performance in downstream tasks be improved by pre-training on domain-specific documents, even when the amount of available data is limited? can inference time be improved while maintaining downstream performance? to address this, we pre-train several models from scratch using a smaller, but more relevant, set of publicly available documents. to evaluate these models, we build a novel dataset, payslips, that contains anonymized insurance pay statements with annotated financial information for ner, detailed in our contributions can be summarized as follows: our code and data are publicly available. foot_1 2 related work contextual embeddings. @xcite first proposed to pre-train a bidirectionnal lstm on large corpora to learn context-sensitive word embeddings that can be used to improve results on downstream tasks. the bert model @xcite instead uses a self-attentive network (i.e. the encoder part of a transformer) to take full advantage of gpu architectures. however, bert cannot be trained using the standard language modeling objective as it is not an autoregressive model. instead, the authors proposed a masked language modeling objective where the loss aims to increase a reconstruction term on a hidden part of the input. ## payslips dataset we build a novel dataset containing financial pay statements from the insurance domain which we call payslips. this dataset consists of a training set of 485 pages and a test set of 126 pages. the documents originate from data of disability insurance. in the event of a work-related accident, this insurance product compensates the insured person during their recovery period. to determine the indemnity amount, the insurer verifies salary information from each insured person's payslip. to speed up information processing, it is essential to build tools capable of automatically extracting the useful financial information. to this end, we worked with insurance professionals and identified nine specific fields, as detailed in payslips was annotated in-house by people familiar with the documents. then, samples have been validated by insurance specialists to ensure annotation quality. more details about the annotation process are given in appendix d. for privacy reasons, unnecessary or potentially identifying information was altered or deleted. moreover, images are not shared as they are not used by our layoutlm model, but could give visual cues about the entity emitting the files. ## experiments we tackle the ner problem using the standard bio-tagging approach @xcite , i.e. each token is tagged with either o (not in a mention), b-label (beginning of a mention) or i-label (inside of a mention), where label is any mention label allowed in the dataset. we can then trivially rebuild the full predicted mentions from the predicted bio tags. we fine-tune all models with a batch size of 16 for 10 epochs, with a fixed learning rate of 5 × 10 -5 . ## conclusion in this work, we pre-train from scratch a lay-outlm model using the docile dataset. importantly, we show that our model obtain better results on a novel domain-specific ner dataset. this shows that it is still possible to develop fast and state-of-the-art in-house models that allow commercial usage. we also release our novel payslips dataset that can be used to challenge document processing models in financial domains.