Taxy.io@ F in TOC -2020: Multilingual Document Structure Extraction using Transfer Learning (2020)

Abstract: In this paper we describe our system submitted to the FinTOC-2020 shared task on financial doc- ument structure extraction. We propose a two-step approach to identify titles in financial docu- ments and to extract their table of contents (TOC). First, we identify text blocks as candidates for titles using unsupervised learning based on character-level information of each document. Then, we apply supervised learning on a self-constructed regression task to predict the depth of each text block in the document structure hierarchy using transfer learning combined with document features and layout features. It is noteworthy that our single multilingual model performs well on both tasks and on different languages, which indicates the usefulness of transfer learning for title detection and TOC generation. Moreover, our approach is independent of the presence of actual TOC pages in the documents. It is also one of the few submissions to the FinTOC-2020 shared task addressing both subtasks in both languages, English and French, with one single model.

## introduction while large amounts of documents are created and published in machine-readable file formats such as pdf, only a small fraction come with structural information, e.g. on their table of contents (toc). this is especially true in the financial domain where a table of contents, e.g. for annual reports or shareholder reports, would be particularly helpful. even though it is often regulated which content these documents must include @xcite , an automated analysis of their structure remains difficult for several reasons. one is that financial documents usually come with a complicated layout consisting of text blocks, tables and figures of various kinds. also, such documents typically come with a highly nested hierarchy of subsections. additionally, most of these documents are published as pdf, which is the defacto standard for the creation of electronic documents, even though this file format comes with several inherent drawbacks @xcite . pdf files contain little structural information about the contents, such as words, lines or paragraphs, which would be helpful for automatic structure analysis. moreover, there exist various different ways to generate a pdf, which further complicates the automatic analysis of the structure of such documents, especially in multicolumn settings. despite the importance and diverse use cases of automatic structure analysis of documents, there is only a small research stream focusing on this topic. the fintoc-2020 shared task about financial document structure extraction @xcite promotes research in this field by proposing two language tracks for english and french together with a benchmark data set of financial documents. the goal of the shared task is to design systems that solve the following two subtasks: in this paper we describe our solution to the shared task, a multilingual approach based on transfer learning that jointly solves both problems title detection and toc generation for both languages. figure 1 : the financial documents of the fintoc-2020 shared task vary greatly with respect to their number of pages, number of titles and the depth of their toc tree. on the entire set across both languages, the number of pages ranges from 3 to 285, the number of titles ranges from 20 to 1039 and the toc tree depth from 1 to 9. the paper is organized as follows. we first review previous work in section 2 and describe the data that was provided for the fintoc-2020 shared task in section 3. we then present our system in section 4 before we report and discuss our results in section 5. the paper ends with a conclusion in section 6. ## previous work the first approaches to toc generation are based on physical layout analysis, e.g. @xcite . here, physical entities of a document, such as pages, paragraphs, and figures are extracted and mapped onto a hierarchy of logical entities such as titles, authors and sections from which the toc is then derived. these approaches are often based on heuristic rules and grammars @xcite . as these predefined rules are mostly domain-specific, these approaches fail to generalize to a diverse set of documents from different domains @xcite . other approaches explicitly parse the title hierarchy from embedded toc pages @xcite . while these approaches generalize across different domains, they fail on documents without toc pages or whenever the toc page does not reflect the entire document structure, e.g. in cases of deeply nested subtitles not mentioned on the toc page @xcite . yet another set of approaches leverage learning-based methods to predict the toc, e.g. based on both layout and text features of the document. in (najah-imane et al., 2019), for example, titles are first detected using a convolutional neural network (cnn) @xcite ; then the corresponding depth of each title is predicted using a combination of a bidirectional long short term memory (bilstm) network and a conditional random field (crf) model, as suggested by @xcite for sequence tagging. the approaches from the fintoc-2019 challenge @xcite mostly fall into the latter two categories: explicit toc parsing and learning-based methods. ## data the training data of the fintoc-2020 shared task consists of 52 english and 47 french financial pdf documents. every document page comes along with a set of annotations, which include both the text and the depth for each title. the box plots in figure 1 show the heterogeneity of the financial documents from this data set. in the median, a financial document from this data set has 34 pages, 132 titles and a toc tree depth of 4. however, the number of pages ranges from 3 to 285, so some documents are very small while others are rather large. also, the number of titles varies from 20 to 1039 across the different documents. while some toc trees are flat with a depth of only 1, some are very nested with a depth of 9. this heterogeneity of the financial documents in the data set underlines their complex layout and hence the difficulty of the task to detect titles and extract the toc. ## system our system works directly on the entire content of a document and is trained for both english and french documents jointly. we apply a two-step approach, where we first detect candidate text blocks and predict their toc depth in a subsequent step as a regression task. the system is described in figure 2 . ## results and discussion in our experiments, we optimized our model for different hyperparameters using a 20% dev set and the evaluation metric from the fintoc-2020 shared task @xcite . more specifically, we evaluated 1) the impact of fine-tuning the pretrained layers of the bert model, 2) the impact of additionally incorporating document and layout features, and 3) the impact of different learning rates. figure 3 shows a positive impact of both fine-tuning the pretrained layers of the bert model (orange vs blue lines) and of additionally incorporating document and layout features (dashed vs solid lines). the inverted u-shape of the curve indicates that 5e-5 is a good choice for the learning rate. across all experiments we trained our models for 2 epochs using a fixed 0.25 dropout ratio in the dropout layer. we trained the model 3 times with different seeds and averaged the outcomes for the final depth prediction, which we rounded to get an integer depth. we discarded text blocks with depth 0 to only submit titles. however, our approach also comes with limitations. selecting fixed clustering hyperparameters assumes that all documents share a similar layout. while we optimized the text block detection hyperparameters for the given set of financial documents, this does not generalize well for other documents, which is mainly due to the limited size of the training set with only 99 documents. moreover, our text block detection is based on dbscan, which requires label encoding where categorical features, e.g. font type, are artificially transformed into numerical ones. using clustering algorithms such as @xcite , which handle numerical and categorical attributes directly, is promising future work. ## conclusion we proposed a multilingual two-step approach for both title detection and toc generation. first, we identified title candidates by clustering a document into text blocks using dbscan. then, we trained a neural network to predict the depth of each text block in the document structure hierarchy. the architecture of the network combines a pre-trained multilingual bert model with a carefully selected set of document and layout features. we have learned that fine-tuning bert and adding document and layout features improves the toc generation accuracy. the approach presented in this paper can be used for both english and french financial documents, even in cases where a toc page is not present.