DNLP @ F in TOC ’20: Table of Contents Detection in Financial Documents (2020)

Abstract: Title Detection and Table of Contents Generation are important components in detecting document structure. In particular, these two elements serve to provide the skeleton of the document, providing users with an understanding of organization, as well as the relevance of information, and where to find information within the document. Here, we show that using tesseract with Levenstein distance, a feature set inspired by Alk et al., we were able to correctly classify the title to an F1 measure 0.73 and 0.87, and the table-of-contents to a harmonic mean of 0.36 and 0.39, in English and French respectively. Our methodology works with both PDF and scanned documents, giving it a wide range of applicability within the document engineering and storage domains.

## introduction in recent years, there has been increasing interest in applying natural language processing (nlp) techniques to financial documents. one of the main publishing formats for these documents is portable document format (pdf), which are usually available on stock exchange websites, company websites, or linked to as part of regulatory filings. generally, financial documents are professionally created (which reduces spelling and grammatical noise), are relatively commonly structured (from a reader's point of view), and provide very similar end-user information. there are, however, variations in the organization and naming conventions, which present challenges in extracting information from documents. the main focus of the fintoc task @xcite is to analyze the the structure of financial documents to detect elements such as headings, sub-headings, and titles. the task is split into two parts: title detection and table-of-contents generation, for both english and french documents. our team participated in both tasks, but found more success in the table-of-contents generation (toc), placing first in the standings. using the python implementation of tesseract, we identified the text regions, and then used levenstein distance to match the provided labels to the detected regions. we then selected features, inspired by the work of akl et al. to be used in classification for both the title detection and the table of contents generation @xcite . for the classification, we used logistic regression (lr), random forest (rf), and support vector machines (svm). the rest of the paper is organized as follows: we first begin with background and related works, then outline our method, dataset, and finally present our results, for each task, starting with title detection. ## background title detection and table-of-contents generation are highly related tasks, as they both depend on being able to successfully first detect document structure. a better computational understanding of the structure of annual financial reports will provide opportunities for a much more fine-grained analysis of the narratives, temporal comparison, and provide a basis of comparability between companies and sectors. the united states (u.s.) is often seen as a benchmark for financial reporting layout as the u.s. securities and exchange commission (sec) mandates a particular structure for regulatory filing. another aspect of the u.s. standard is that the sec publishes its regulatory filings in html format on its online repository, edgar; companies will then simultaneously make the same information available in pdf format, most often on its corporate website. @xcite this rigorous structure is not found in other countries, making multi-jurisdictional information extraction a significant challenge. to address this, el-haj et al. used 10,000 annual reports during the period of 2003-2014 in spanish and portuguese from the london stock exchange, and created a tool called frie-frse (corporate financial information environment -final report structure extractor) to detect the annual report structure @xcite . a variety of approaches have been taken in previous work to detect titles. gopinath et al. @xcite used two main approaches for extraction of title and prose: a domain-independent approach (di) and a domain-dependent approach (dd). the data is mutually exclusively grouped into two subgroups using k-means clustering. an overlap score is then generated for potential titles, and those which exceed the 75% overlap threshold are added as candidate titles. the approaches only differ in the feature selection in that a neural network is used to classify examples previously labelled during di. hercig and kral used the brainy maximum entropy classifier to detect titles, using 7 key features: character n-grams, binary features, first and last orto-characters, font size and type, as well as text length @xcite . at the 2019 fintoc workshop, daniel also proposed using various features such as visual characteristics, punctuation density and character n-grams, achieving an f1-measure score of 94.88% @xcite . the findse team used a supervised learning approach that used linguistic, semantic, and morphological features to classify a block of text as either title or non-title, resulting in an f1-measure of 97.01% @xcite . a table-of-contents is a document's roadmap; it shows the hierarchy and organization of a document, helps users quickly access relevant information according to their query, and provides additional information such as the length of the document. yet, while there is a commonly accepted idea of what information the table-of-contents should contain, there is no actual imposed or required standard that corporate entities must use. in fact, it can be seen that some companies provide a sparse table-of-contents with only the major sections included, whereas others provide much more fine-grained detail. therefore, being able to independently generate a table-of-contents can significantly enrich the search of electronic documents. tian and peng augmented their training and testing data provided in the task by adding additional fields to the data @xcite . from the augmented data, they retrieved unique tokens for training the word embedding. the vectors were then in attention based long-short term memory (lstm) and bidirectional lstm models, using 10-fold cross-validation. their method performed very well, and they ranked 1st in the 2019 competition. najah-imane et al. proposed a toc-generation pipeline based on binary classification and sequence label modelling @xcite . classification is done to separate titles from non-titles, and sequence labelling provides a hierarchical structure to the detected titles. @xcite introduced an aggregation-based method to enhance toc extraction using system submissions from the international conference on document analysis and recognition (icdar) book structure extraction competitions @xcite . their results show that using both methods together outperforms existing approaches using both the title-based and link-based evaluation measures on a dataset of more than 2,000 books. by efficiently combining the results of existing systems in an unsupervised way, they consistently beat the state-of-the-art in book structure extraction, with performance improvements that are statistically significant. @xcite used dnn-based approach which it scored a weighted f1 of 97.16% on the test data. they evaluated a svm classifier. it was trained on a combination of features, compiled from the existing features presented within the original csv file as well as additional features extracted from pre-processing work on the xml files. their second model is a bilstm-attention model relying on word embedding, to make use of the attention mechanism. the third model is a cnn classifier. the purpose of this method is to evaluate the combinatorics of characters at word level (as predictors) and how relevant they are for the task. ## methodology in the following sections we describe the methodology used to build the prediction models. the processing pipeline consists of two general steps: feature extraction and classification. ## conclusion in this paper, we presented our work as part of the 2020 fintoc task for title detection and table-ofcontents generation. we also demonstrated that using tesseract in conjunction with levenstein distance and a set of key features returned a harmonic mean of 0.34 and 0.37 for english and french, respectively, earning us 1st place for table-of-contents generation. our method also returned f1 measures of 0.59 and 0.64 for english and french, respectively. while previous researchers have used methods primarily focused on pdf documents, our approach used ocr scanning in order to include scanned documents in addition to pdfs. while document storage practices have evolved over time with pdf being the standard, this has not always been the case. many older documents were scanned and are archived in that format. employing an approach that considers this evolution, as we did, makes our method more widely applicable to electronic documents. in the future, we would like to apply deep learning methods to both title detection and table-of-contents generation, as this approach is currently not widely researched. another area of interest would be in embedding information pointers in documents to help better identify titles, sections, headers, and subheaders. finally, we would like to experiment using computer vision for title detection and use it to guide better generation of tables-of-contents.