OLALA : Object-Level Active Learning for Efficient Document Layout Annotation (2022)

Abstract: Layout detection is an essential step for accurately extracting structured contents from historical documents. The intricate and varied layouts present in these document images make it expensive to label the numerous layout regions that can be densely arranged on each page. Current active learning methods typically rank and label samples at the image level, where the annotation budget is not optimally spent due to the overexposure of common objects per image. Inspired by recent progress in semi-supervised learning and self-training, we propose OLALA, an Object-Level Active Learning framework for efficient document layout Annotation. OLALA aims to optimize the annotation process by selectively annotating only the most ambiguous regions within an image, while using automatically generated labels for the rest. Central to OLALA is a perturbation-based scoring function that determines which objects require manual annotation. Extensive experiments show that OLALA can significantly boost model performance and improve annotation efficiency, facilitating the extraction of masses of structured text for downstream NLP applications.

## introduction when working with historical documents, social scientists have often used keyword methods that do not require the recognition of structured layouts (see e.g. @xcite for a review of the literature on historical newspapers). to apply neural nlp methods to these documents, it is essential to accurately detect the layouts and extract the structured content. for example, a historical newspaper scan contains a mixture of article regions, headlines, captions, advertisements, etc. commercial ocr software will typically read the multi-figure 1 : three exemplar document layouts from pub-laynet @xcite , hjdataset @xcite , and prima @xcite . there are numerous layout objects per page, and many of them are very similar. directly labeling them all will result in wasted labeling budget. deep learning-based approaches can be used for document layout analysis and content parsing @xcite @xcite . figure 1 illustrates that document layout object detection, like image object detection, requires identifying content regions and categories within images. a key distinction, however, is that it is common for dozens to hundreds of content regions to appear on a single page in documents, as opposed to only several objects per image in natural image datasets (e.g. 5 on average in the ms-coco dataset @xcite ). additionally, the region category distribution is often heavily imbalanced and requires more pages to be annotated to allow for reasonable exposure of uncommon categories (e.g. footnotes, watermarks, or mastheads). hence, the manual labeling process used on natural images to create high-quality labeled datasets can be prohibitively costly to replicate for documents of central interest to social scientists, who typically have heavily constrained annotation budgets. as a result, extracting structured text is often infeasible, limiting the application of modern nlp techniques in historical document applications. active learning (al) has been widely adopted in image object detection for optimizing labeling efficiency via prioritizing the most important samples to annotate @xcite @xcite . however, while the end goal is to annotate individual objects within an image, these al methods typically score and select samples at the image level, rather than at the object level. @xcite study an annotator-centered labeling cost estimation method and prioritize labeling for high-cost images. in the context of deep learning, image level scores are generated via aggregation of marginal scores for candidate boxes @xcite or applying query by committee @xcite to features maps @xcite . @xcite propose a pixel level scoring method using convolutional backbones and aggregate them to informativeness scores for image ranking. for category-imbalanced layouts, common in documents, such image level selection can suffer from the over-exposure of common objects. recent advances in semi-supervised learning (ssl) and self-training can boost model performance using unlabeled data @xcite . the self-supervised sample mining (ssm) algorithm @xcite proposes to stitch high-confidence patches from unlabeled data to labeled data to improve both labeling efficiency and model performance. it enables object-level prediction selection but requires objects to be sparsely distributed, making it inapplicable to our case where content is densely arranged. to address these challenges, we propose a novel al framework, olala, object-level active learning for efficient layout annotation. shown in figure 2 , critical objects, rather than images, are individually evaluated and selected for labeling. during the labeling process, olala trains a model to generate object predictions. within an image, only the most ambiguous predictions are chosen for human inspection and annotation, addressing the inefficient use of annotation budget on common objects or categories. central to this process is a semi-automatic prediction correction algorithm. inspired by previous endeavors of automated layout dataset generation @xcite @xcite , olala incorporates prior knowledge about layout structures to ensure the high quality of the created dataset. it can identify false-positives and false-negatives in the unselected model predictions, and correct them with minimal extra supervision. additionally, we design a novel object-level scoring function governing the region selection process. the perturbation-based scoring method evaluates consistency of both object position and category predictions between the original and perturbed inputs. compared to prior work, it is carefully designed for layout datasets with unique arrangement of content regions, and can identify errors of critical importance to layout analysis tasks. in other contexts where predictions on unlabeled images @xcite or weak labels @xcite are used to boost model performance, the predicted labels are discarded after model training. olala includes a rigorous process to validate the accuracy of predictions, meaning that the full labeled dataset -created by human and machine -can be released publicly and potentially used by social scientists for transfer learning on other applications. through extensive experiments, we study how the proposed approach can improve labeling efficiency in two different scenarios. we show that olala can create datasets with better trained model performance compared to image-level al baselines, for a given limited annotation budget. on the other side, as only part of an image requires annotation in our method, we demonstrate that our method can create datasets of the same size with far less human effort. to the best of our knowledge, this is the first al method dedicated to document layout analysis. olala was motivated by our need to automate the extraction of structured text from millions of historical documents, to enable modern nlp analyses on information trapped in hard copy. we are using it extensively on real world documents for this purpose, with the olala labeling interface described in the supplementary material. section 2 introduces the olala framework, and section 3 describes the perturbation based object scoring method. sections 4 and 5 demonstrate how olala can improve labeling efficiency and model performance under different scenarios. ## perturbation-based scoring function the scoring function f also plays an important role in the olala framework. it evaluates prediction ambiguity and determines which objects to select for labeling. we propose a perturbation scoring method based on both the bounding box and category predictions. inspired by the self-diversity idea in @xcite and @xcite , the proposed method hypothesizes that the adjacent image patches share similar features vectors, and the predicted object boxes and categories for them should be consistent. therefore, any large disagreement between the original and perturbed predictions indicates that the model is insufficiently trained for this type of input, or there are anomalies in the sample. both cases demand user attention. specifically, for each object prediction ŷj = ( bj , ĉj ) ∈ ŷi , we take the bounding box prediction @xmath0, y, w, h) and apply some small shifts to perturb the given box, where x, y are the coordinate of the top left corner, and w, h are the width and height of the box. the new boxes are created via horizontal and vertical translation by a ratio of α and β: p @xmath1, @xmath2, w, h), where p jk is the k-th perturbed box for box prediction bj , and a total of k perturbations will be generated. based on the image features within each p jk , the model generates new box and category predictions (q jk , v jk ). we then measure the disagreement between the original prediction ( bj , ĉj ) and the perturbed versions {(q jk , v jk )} k @xmath3, and use it as a criterion for selecting objects for labeling. in practice, we build this method upon a typical object detection architecture composed of two stages @xcite : 1) a region proposal network estimates possible bounding boxes, and 2) a region classification and improvement network (roiheads 3 ) predicts the category and modifies the box prediction based on the input proposals. we use the perturbed boxes {p jk } k @xmath4, and obtain the new box and class predictions {(q jk , v jk )} k @xmath5. for object regions of low confidence, the new predictions are unstable under such perturbation, and the predicted boxes and category distribution can change drastically from the original version. to this end, we formulate the position disagreement d p and the category 3 it's a module name in detectron2 @xcite . disagreement d c for the j-th object prediction as the proposed method can effectively identify false-positive object predictions. based on the selfdiversity assumption, incorrect category prediction ĉj will cause high d c because of the divergence of the new class prediction v jk for nearby patches. when the predicted box bj is wrong, the perturbed box p jk is less likely to be the appropriate proposal box. the generated predictions (q jk , v jk ) are unreliable, causing higher overall disagreement d. applicability to layout datasets compared to previous work, the perturbation-based scoring function aims to solve two challenges unique to layout analysis tasks. first, layout regions are boundary-sensitive: a small vertical shift of a text region box could cause the complete disappearance of a row of texts. however, existing methods designed for image-level selection usually focus on the categorical-rather than positionalinformation in outputs (i.e. @xcite considers the marginal score of the object category predictions and does not use the bounding boxes, and @xcite indirectly uses the positional information based on a pixel map for image-level aggregation). by contrast, our method identifies samples that lead to ambiguous boundary predictions via d p . moreover, document images usually contain numerous objects per page and content regions are densely arranged. hence, we cannot adapt the object-level scoring function in @xcite , which requires cropping an object, randomly pasting it to another image, and evaluating the consistency between the original and the newly detected boxes for this object. the random pasting will introduce non-existing structures (e.g. , overlaying a figure over tables or texts), and the calculated score cannot reliably assess the prediction. with olala, the original document structures are untouched. ## experimental setup objective several experiments are designed to study the validity of the proposed olala framework and evaluate how it can improve the efficiency of the labeling process. methods are considered better if they achieve similar accuracy while using less labeling budget m than their counterparts, or obtain higher accuracy given the same m. in the experiments, we measure object detection accuracy using mean average precision (ap) scores @xcite , and the labeling budget refers to the number of objects to label by default. datasets to validate our approach, we run simulations on three representative layout analysis datasets: publaynet @xcite , prima @xcite , and hj-dataset @xcite . publaynet is a large dataset of 360k images. the images and annotations are generated from noiseless digital pdf files of medical papers. as the original training set in publaynet is too large to conduct experiments efficiently, we use a downsampled version. prima is created by human annotators drawing bounding boxes for text regions in both scanned magazines and technical articles, resulting in greater heterogeneity in this dataset than in publaynet. hj-dataset contains layout annotation for 2k historical japanese documents. hjdataset was established using noisy image scans, and the creation method is a combination of rule-based layout parsing from images and human correction. labeling simulation when running simulations, we build two additional helper algorithms to imitate human labeling behavior. first, for the selected objects, the corresponding ground-truths is found via a best-matching algorithm. for each prediction, we calculate the iou with all ground-truth objects and choose the top one to substitute the prediction. duplicated ground-truths selected in an image will be removed by this process. in real-world labeling experiments, we also notice human annotators do not need to correct an object prediction if it is accurate (high iou with the ground-truth and category is the same). to best simulate this phenomena, if a selected prediction has an @xmath6.925 (determined empirically) with a ground-truth object of the same category, we do not substitute it with the ground-truth and only use a discounted budget η = 0.2. finally, to mimic annotators' search for false-negative regions, we compute the pairwise iou between the ground truth y i and the combined labeling objects ỹi . ground-truth objects whose maximum iou with predicted objects is less than ζ are chosen to add to ỹi , and the remaining budget is reduced accordingly. ζ is set to 0.05 in the following experiments to allow minor overlapping caused by noise in the predictions. ## conclusion the objective of this paper is to develop rigorous methods that can increase the efficiency of extracting structured texts -required for downstream nlp applications -from social science documents. we propose the object-level active learning annotation figure 4 : the created object sources (line plot) and dataset accuracy (bar plot) during the training process. the number of manually labeled objects (blue) decreases and directly used model predicted objects (orange) increases. as the model becomes more accurate, a higher portion of selected objects become accurate (green). results shown are averaged from the three olala methods in the publaynet experiments.