Lightweight Contextual Logical Structure Recovery (2022)

Abstract: Logical structure recovery in scientific articles associates text with a semantic section of the article. Although previous work has disregarded the surrounding context of a line, we model this important information by employing line-level attention on top of a transformer-based scientific document processing pipeline. With the addition of loss function engineering and data augmentation techniques with semi-supervised learning, our method improves classification performance by 10% compared to a recent state-of-the-art model. Our parsimonious, text-only method achieves a performance comparable to that of other works that use rich document features such as font and spatial position, using less data without sacrificing performance, resulting in a lightweight training pipeline.

## introduction logical structure recovery in scientific document processing (sdp) provides fundamental information about scientific documents. the logical structure of a document is "the hierarchy of logical labels that indicates the construction of the document" @xcite . recovering the logical structure gives insight into the structure of a long scientific document and aids further sdp tasks such as abstractive summarization, metadata extraction, and information extraction, etc. logical structure recovery classifies the lines of a scientific document into predefined semantic categories that represent its role in the document (cf. we answer this challenge by creating a parsimonious but robust model that operates on purely textual data without incorporating such features. instead, we rely on better context modeling of surrounding lines, identifying the continuity of logical structure of the document, and making use of abundant unlabeled data. first, we consider multiple lines of marginally breaked text as context (cross-line context) and use attention @xcite on top of transformer models @xcite to obtain context-sensitive sentence embeddings of lines. second, we employ semi-supervised learning @xcite over the abundance of unlabeled data to address the lack of labeled data in the recovery of logical structures. lastly, we employ elements of loss engineering from recent semi-supervised learning frameworks such as uda @xcite without the use of unlabeled data to increase performance under a supervised training regime to deploy a lightweight training pipeline. although only plain text is used for training, our model achieves results close to the current stateof-the-art (sota) compared to models based on rich text features. furthermore, we show that semisupervised learning helps improve sota for logical structure recovery by 10% on macro-f1. ## related work aside from the text of scientific papers, previous work extracts rich text information -such as font size, font style, paragraphing -as rich text information is a primary factor in discerning the logical structure of a document (rahman and finin, 2019 ). for example, sectlabel @xcite extracts rich text information from scientific documents using ocr, then subsequently applying conditional random fields (crf; @xcite to classify the extracted text into predetermined labels. @xcite extends this approach further, combining the usage of spatial measures, typesetting, and minimal text patterns with contextual meaning into a 2d crf model for classification. @xcite 's work involves using remnant visual cues extracted from text data including line breaks, indentation, and text alignment to augment logical structure extraction while using random forest as their primary model. other work focus on the usage of layout itself to discern such logical structures, utilizing deep object detection models such as r-cnn models @xcite @xcite to capture logical structures, taking "screenshots" of the pdf document as input. lay-outlm models @xcite @xcite combine object detection models with textual transformers @xcite along with positional embeddings of logical structures on the page to form multimodal models, while document image transformers (dit; li et al. 2022) use vision transformers (vit; dosovitskiy et al. 2021) as backbone models for further image-based detections of the logical structures. although rich text information is usually incorporated, there are models, such as the sciwing toolkit (ramesh kashyap and kan, 2020), for logical structure recovery that operate only on plain text. our work is in line with such lightweight textonly methods, which benefit from the simple and streamlined input without redundant metadata. in contrast to sciwing's simple text representation for each line, we aim to incorporate richer textual information from the cross-line context and make use of abundant unlabeled data available. ## contextual model construction we attempt the task of logical structure classification, as proposed by @xcite , and label each line in scientific papers to represent its logical structure. we address this task in a purely textual method, employing modern nlp model architectures and training techniques to achieve our goal of creating a more lightweight and streamlined approach. we consider this task as a line-based classification problem as we want to preserve the notion of margin breaks without having to include layout or spatial information. given a document d n of length n, we have the following: ## semi-supervised learning supervised learning can be used to produce accurate models when adequate labeled data are provided. while unlabeled data is easy to obtain, labeled data are scarce, particularly in the sdp domain. semi-supervised learning (ssl) methods address this problem using both labeled and unlabeled data, resulting in better performance compared to purely supervised means. ## analysis we analyze in detail both the architectural changes ( §6.1, 6.2) and training techniques ( §6.3, 6.4). we employ an iterative alteration of models in our experiments, starting with sciwing's sectlabel ## conclusion this paper shows that, with effective use of multiline context, the results of plain text logical structure recovery models are comparable with other models that use rich text information. we achieve this by employing transformers to produce highquality sentence embeddings, applying sliding window attention to consider cross-line context, and further optimizing by engineering loss functions such as employing training signal annealing, incorporating consistency loss, and/or training under a semi-supervised regime. further work on purely contextual models may extend to solving the class imbalance problem of logical structures, which is further amplified due to the usage of semi-supervised training. given the importance of neighboring context, one cannot simply rebalance the dataset. these issues require other methods to decrease such biases.