The Financial Document Structure Extraction Shared task ( F in T oc 2020) (2020)

Abstract: This paper presents the FinTOC-2020 Shared Task on structure extraction from financial documents, its participants results and their findings. This shared task was organized as part of The 1st Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation (FNP-FNS 2020), held at The 28th International Conference on Computational Linguistics (COLING’2020). This shared task aimed to stimulate research in systems for extracting table-of-contents (TOC) from investment documents (such as financial prospectuses) by detecting the document titles and organizing them hierarchically into a TOC. For the second edition of this shared task, two subtasks were presented to the participants: one with English documents and the other one with French documents.

## introduction the use of pdf electronic documents is recurrent in the financial domain. they are used to share and broadcast information concerning investment strategies, policy and regulation. even with a great layout, long documents can be hard to navigate, hence, the presence of a table-of-contents (toc) can provide a valuable assistance for potential investors or regulators by increasing readability and facilitating navigation. in this shared task, we focus on extracting the toc of financial prospectuses. in these official documents, investment funds accurately depict their characteristics and investment modalities. depending on their country of origin, they might be edited with or without a toc, and they might follow a template as well. but even though their format is regulated, the choice of the text format, the layout, the graphics and tabular presentation of the data is in the hand of the editor. thus, the toc is of fundamental importance to tackle sophisticated nlp tasks such as information extraction or question answering on long documents. in this paper, we report the results and findings of the fintoc-2020 shared task. 1 the shared task was organized as part of the 1st joint workshop on financial narrative processing and multiling financial summarisation @xcite , to be held at the 28th international conference on computational linguistics (coling'2020). a total of 5 teams submitted runs and contributed 5 system description papers. all system description papers are included in the fnp-fns 2020 workshop proceedings and cited in this report. ## previous work on toc extraction there are mainly two concepts in the literature to approach toc extraction. the first one parses the hierarchical structure of sections and subsections from the toc pages embedded in the document. this area of research was mostly motivated by the inex @xcite and icdar competitions @xcite @xcite which aim at extracting the toc of old and lenghtly ocr-ised books. the documents we target in this shared task are very different: they contain graphical elements, and the text is not displayed to respect a linear reading direction but is optimized to condense information and catch the eye of the reader. apart from these competitions, we find the methods proposed by @xcite @xcite , also based on the parsing of the toc page. in the second category of approaches, we find algorithms that detect the titles of the document using learning methods based on layout and text features. the set of titles is then hierarchically ordered according to a predefined rule-based function @xcite @xcite . lately, we find systems that address the hierarchical ordering of the titles as a sequence labelling task, using neural networks models such as recurrent neural networks and lstm networks @xcite . ## task description as part of the fnp-fns workshop, we present a shared task on financial document structure extraction. participants to this shared task were given two sets of financial prospectuses with a wide variety of document structure and length. their systems had to automatically process the documents to extract their document structure, or toc. in fact, the two sets were specific to two different subtasks: for both sets, we observe that: for each subtask, all participating teams were provided with a training dataset which included the original pdfs alongside their corresponding json file representing the toc of the document. this json represented the toc by giving the titles, their pages, their depths and their ids, as shown in fig. 2 . a private test set was used to evaluate the tocs generated by the participants systems. as stated in section 2, most of the previous research on toc generation has focused on short papers such as research publications (arxiv database), or weakly graphical material such as digitalized books. however, the task of extracting the toc of commercial documents with a complex layout structure in the domain of finance is not much explored in the literature. ## shared task data in this section, we discuss the corpus of documents used for the toc extraction subtasks. ## participants and systems a total of 50 teams registered in the shared task all from different institutions. eventually, 5 teams participated and submitted a paper with the description of their method, see participating teams explored and implemented a wide variety of techniques and features. in this section, we give a brief description of each system. more details could be found in the description papers published in the proceedings of the fnp-fns 2020 workshop. amex-ai labs @xcite : this team participated in the english subtask only. several pre-processing steps, including headers and footers removal, are performed to segment the textual content of the documents into elements that are classified later as titles and non-titles. they separately trained two title detectors, one on an external dataset, and the other on the english dataset provided by the fintoc2020 shared task. then, they concatenated both models to form their final title detector. daniel @xcite : unlike the other teams, this team focused on removing tables to clean the textual content of the documents. they detected titles by looking for numbered lines, leveraging stylistic properties, and checking the existence of a line in the list of training titles. for the hierarchy part, they clustered the titles previously detected to infer their hierarchical level using stylistic features. it is interesting to see that this approach ranked well on the english dataset but not so well on the french dataset. dnlp @xcite : the dnlp team participated in both subtasks. they used tesseract an open-source ocr tool to extract the text regions. then, they defined a set of features to use with team affiliation tasks amex-ai labs @xcite american express ai labs, bangalore e daniel @xcite stih, sorbonne univeristy f and e dnlp @xcite dalhousie university f and e taxy.io @xcite taxy.io f and e uwb @xcite university of west bohemia f and e taxy.io @xcite ): the taxy.io team participated in both subtasks, with a multilingual pipeline. they used an unsupervised learning approach to tackle text block detection. they first run a dbscan clustering on pages characters to extract features and then run a second dbscan clustering to identify the text blocks. finally, they classify each text block, represented by features from the previous step plus text features extracted with a multilingual bert model. uwb @xcite : uwb team participated in both subtasks but contributed to the title detection part only. they state title detection as a binary classification on text segments, for which they use a maximum entropy classifier, on top of a diverse set of features including orthographic characters and character n-grams. ## results and discussion evaluation metric since both subtasks tackle the same problem but on different corpora, we used the same evaluation metric. for the toc generation part, we adapted the metrics proposed by the structure extraction competition (sec) held at icdar 2013 @xcite : we adapted the script, replaced the customized levenshtein distance specifically designed for sec by a standard levenshtein distance whose edit cost is 1 in all cases, and removed the constraint on first and last 5 characters. the final ranking is based on the harmonic mean between inex f1 score and inex level accuracy. in the calculation of the inex f1 score, correct entries in the predicted toc are those which match the title of an entry in the groundtruth toc and have the same page number as this entry. the inex level accuracy evaluates the hierarchy of the predicted toc. if we denote by e ok an entry in the predicted toc with a correct page number, and by e ok an entry in the predicted toc with a correct page number and a correct hierarchical level, then the inex level accuracy is: we also provided scores for the title detection part separately: we used the f1 score, and considered as correct entries the predicted entries which match the titles of groudtruth entries according to the standard levenshtein distance. for both parts, the threshold on the levenshtein score was set to 0.85 foot_1 . moreover, the inex scores and title f1 score are calculated for each document and then averaged over the documents of the private set to produce two performance figures per team submission: one for toc extraction, and another for title detection (td). baseline for comparison purposes, we implemented a simple baseline toc extractor consisting of: for english @xcite . toc extraction for pdf documents is a realistic problem in everyday applications which explain the interest from and participation of both public universities and profit organizations.