LOCR : Location-Guided Transformer for Optical Character Recognition (2024)

Abstract: Academic documents are packed with texts, equations, tables, and figures, requiring comprehensive understanding for accurate Optical Character Recognition (OCR). While end-to-end OCR methods offer improved accuracy over layout-based approaches, they often grapple with significant repetition issues, especially with complex layouts in Out-Of-Domain (OOD) documents.To tackle this issue, we propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on an original large-scale dataset comprising over 53M text-location pairs from 89K academic document pages, including bounding boxes for words, tables and mathematical symbols. LOCR adeptly handles various formatting elements and generates content in Markdown language. It outperforms all existing methods in our test set constructed from arXiv.LOCR also eliminates repetition in the arXiv dataset, and reduces repetition frequency in OOD documents, from 13.19% to 0.04% for natural science documents. Additionally, LOCR features an interactive OCR mode, facilitating the generation of complex documents through a few location prompts from human.

## introduction academic literature comprises a wealth of highquality content, yet much of it is provided in formats like pdf that are not readily for machine reading. particularly, most academic documents of the previous centuries are scanned version. digitizing academic documents are important for scientific research, literature retrieval, and large-language model training. however, academic document layout tends to be highly intricate, including text, equa-tions, images, tables, and annotations, posing challenges for obtaining accurate ocr results. one approach to document ocr is to first analyze the layout of the document and then extract the text content @xcite . while progress has been made in any of the two stages or handling specific types of elements, such as table detection and recognition @xcite , handwritten formula recognition @xcite and structured information extraction @xcite , it is very difficult for models to understand all the elements and connect the different chunks into a coherent sequence. recently, an end-to-end transformer structure, donut @xcite , was proposed for document understanding. it effectively addresses the complexity of combining multiple models and the issue of error propagation. without too many changes in the model, @xcite processes academic pdfs into markup language. however, these methods are prone to hallucination and repetitive loops. in fact, getting trapped in a repetitive loop is a common problem with transformer-based models sampling with greedy search decoding @xcite . it is challenging for a language model to accurately capture all the content of text-intensive documents without position perception. by visualizing the cross-attention during the prediction process of nougat (see appendix d), we found that the cross-attention cannot be focused on the correct position when the layout is complex. this indicates that the positional information influence the text decoding to a great extent. inspired by this, we introduce locr, which incorporates positional guidance for the model to focus on the correct word to address the issue of repetitive loop. the most significant feature that distinguishes our model from previous works is the incorporation of positional autoregression alongside text autoregression. locr simultaneously predicts the current token and the position of the next token, which is used to prompt the decoding of the next token. through this method, we not only combine positional information with text information but also avoid the tedious process and error accumulation in the two-stage ocr method. taking document images as input, our model outputs document content in markdown format, including special formats such as superscripts and subscripts. furthermore, we propose an importance decay strategy to intuitively penalize locations that have already been visited to avoid repetition. with the record of visited locations, we decrease the importance of these positions. the repetition behavior is eliminated in the arxiv test set, and decreases for out-of-domain documents. for documents with complex layouts, we also introduce an interactive ocr mode, allowing the model to continue to decode the text where the user has dragged a box. with these enhancement strategies, the generation ability of the model is significantly improved. additionally, we propose a data engine for constructing academic document ocr dataset with positional annotations. we collect a large-scale dataset of 89k academic document pages with 53m text-location pairs. to the best of our knowledge, it is the first dataset that includes a bounding box of each mathematical symbol in academic documents. in summary, the contributions of this paper are: ## dataset 3.1 data collection to the best of our knowledge, there is no paired dataset containing markup-formatted document contents along with corresponding bounding boxes (bbox) for each word and mathematical symbol. we proposed a data engine to collect such paired data. the process is shown in figure 2 . we get the tex source files of academic papers from arxiv. in the first step, we assign a unique rgb color identifier to each word and mathematical symbol automatically by using xcolor package in latex (see step1 ). in the second step, following the same pipeline as nougat (blecher et al., 2023), we compile latex files into pdf and markdown files respectively. since pdf is a rich text format that supports color changes, we obtain colorful pdf files. while markdown is a plain text format, the rgb identifiers are compiled into text forms (see step2). in the third step, we use the pymupdf package of python to parse the colorful pdf files and extract the pair of (color, bbox). at the same time, we parse the markdown file with regular expressions to get the paired (color, text) data. finally, we merge the two pairs of data by the key of rgb color to get paired (text, bbox) data (see step3). we collected academic papers released on arxiv from 2007 to 2023. during data processing, some articles failed the conversion due to user-defined macros or non-standardized formats. after all conversion and data cleaning, our dataset is composed of 88998 pages, which include, but are not limited to, the bounding box of plain text, greek letters, arithmetic symbols, superscripts, subscripts, and tabular symbols. examples of our dataset is available in appedix a1. latex \section{criticality in deep learning nets} \subsection{from feed-forward to…} we will focus now on a a feed-forward network, with two layers, @xmath0 and @xmath1 connected… latex \section{\textcolor[rgb]{180,000,000}{criticality}…} \subsection{\ textcolor[rgb]{180,000,050}{from}…} \textcolor[rgb]{180,000,100}{we} \textcolor[rgb]{180,000,105}{will}… @xmath2 \textcolor[rgb]{180,000,190}{and}… pdf 3 criticality in deep learning nets 3.1 from feed-forward to fully connected… w e will focus now on a feed-forward network, with two layers, 𝑎 𝑖 and 𝑏 𝑗 connected… markdown ## 3 textcolor[rgb]180,000,010criticality… ### textcolor[rgb]180,000,050from… textcolor[rgb]180,000,100we textcolor[rgb]180,000,105will @xmath3 textcolor[rgb]180,000,190and… ## discussion in our work, we introduce locr, which incorporates location guiding into the language model. our approach significantly mitigates the problem of repetitive loops encountered by transformer-based models. the interactive mode can be utilized to construct datasets for fine-tuning ocr models to specific domains and enhancing the generalization capability. we believe that locr can be applied to digitize documents from various fields with complex layouts, thereby assisting academic research, literature retrieval, and large language model training. ## limitations although the frequency of repetition has been significantly mitigated, it has not been entirely eradicated in out-of-domain documents. secondly, when parsing other types of documents beyond academic papers, some human interaction is needed. additionally, our model encounters difficulties when the initial word on a page is incomplete, leading to imperfect handling. we will continue our work to address these issues.