|
--- |
|
license: mit |
|
language: |
|
- en |
|
- fr |
|
- es |
|
--- |
|
|
|
# π₯ Classifiers of FinTOC 2022 Shared task winners (ISPRAS team) π₯ |
|
|
|
Classifiers of texual lines of English, French and Spanish financial prospects in PDF format for the [FinTOC 2022 Shared task](https://wp.lancs.ac.uk/cfie/fintoc2022/). |
|
|
|
## π€ Source code π€ |
|
|
|
Training scripts are available in the repository https://github.com/ispras/dedoc/ (see `scripts/fintoc2022` directory). |
|
|
|
## π€ Task description π€ |
|
|
|
Lines are classified in two stages: |
|
1. Binary classification title/not title (title detection task). |
|
2. Classification of title lines into title depth classes (TOC generation task). |
|
|
|
There are two types of classifiers according to the stage: |
|
1. For the first stage, **binary classifiers** are trained. They return `bool` values: `True` for title lines and `False` for non-title lines. |
|
2. For the second stage, **target classifiers** are trained. They return `int` title depth classes from 1 to 6. More important lines have a lesser depth. |
|
|
|
## π€ Results evaluation π€ |
|
|
|
The training dataset contains English, French, and Spanish documents, so three language categories are available ("en", "fr", "sp"). |
|
To obtain document lines, we use [dedoc](https://dedoc.readthedocs.io) library (`dedoc.readers.PdfTabbyReader`, `dedoc.readers.PdfTxtlayerReader`), so two reader categories are available ("tabby", "txt_layer"). |
|
|
|
To obtain FinTOC structure, we use our method described in [our article](https://aclanthology.org/2022.fnp-1.13.pdf) (winners of FinTOC 2022 Shared task!). |
|
The results of our method (3-fold cross-validation on the FinTOC 2022 training dataset) for different languages and readers are given in the table below (they slightly changed since the competition finished). |
|
As in the FinTOC 2022 Shared task, we use two metrics for results evaluation (metrics from the [article](https://aclanthology.org/2022.fnp-1.12.pdf)): |
|
**TD** - F1 measure for the title detection task, **TOC** - harmonic mean of Inex F1 score and Inex level accuracy for the TOC generation task. |
|
|
|
<table border="1" class="dataframe"> |
|
<thead> |
|
<tr style="text-align: left;"> |
|
<th></th> |
|
<th>TD 0</th> |
|
<th>TD 1</th> |
|
<th>TD 2</th> |
|
<th>TD mean</th> |
|
<th>TOC 0</th> |
|
<th>TOC 1</th> |
|
<th>TOC 2</th> |
|
<th>TOC mean</th> |
|
</tr> |
|
</thead> |
|
<tbody> |
|
<tr> |
|
<th>en_tabby</th> |
|
<td>0.811522</td> |
|
<td>0.833798</td> |
|
<td>0.864239</td> |
|
<td>0.836520</td> |
|
<td>56.5</td> |
|
<td>58.0</td> |
|
<td>64.9</td> |
|
<td>59.800000</td> |
|
</tr> |
|
<tr> |
|
<th>en_txt_layer</th> |
|
<td>0.821360</td> |
|
<td>0.853258</td> |
|
<td>0.833623</td> |
|
<td>0.836081</td> |
|
<td>57.8</td> |
|
<td>62.1</td> |
|
<td>57.8</td> |
|
<td>59.233333</td> |
|
</tr> |
|
<tr> |
|
<th>fr_tabby</th> |
|
<td>0.753409</td> |
|
<td>0.744232</td> |
|
<td>0.782169</td> |
|
<td>0.759937</td> |
|
<td>51.2</td> |
|
<td>47.9</td> |
|
<td>51.5</td> |
|
<td>50.200000</td> |
|
</tr> |
|
<tr> |
|
<th>fr_txt_layer</th> |
|
<td>0.740530</td> |
|
<td>0.794460</td> |
|
<td>0.766059</td> |
|
<td>0.767016</td> |
|
<td>45.6</td> |
|
<td>52.2</td> |
|
<td>50.1</td> |
|
<td>49.300000</td> |
|
</tr> |
|
<tr> |
|
<th>sp_tabby</th> |
|
<td>0.606718</td> |
|
<td>0.622839</td> |
|
<td>0.599094</td> |
|
<td>0.609550</td> |
|
<td>37.1</td> |
|
<td>43.6</td> |
|
<td>43.4</td> |
|
<td>41.366667</td> |
|
</tr> |
|
<tr> |
|
<th>sp_txt_layer</th> |
|
<td>0.629052</td> |
|
<td>0.667976</td> |
|
<td>0.446827</td> |
|
<td>0.581285</td> |
|
<td>46.4</td> |
|
<td>48.8</td> |
|
<td>30.7</td> |
|
<td>41.966667</td> |
|
</tr> |
|
</tbody> |
|
</table> |
|
|
|
## π€ See also π€ |
|
|
|
Please see our article [ISPRAS@FinTOC-2022 shared task: Two-stage TOC generation model](https://aclanthology.org/2022.fnp-1.13.pdf) |
|
to get more information about the FinTOC 2022 Shared task and our method of solving it. |
|
We will be grateful, if you cite our work (see citation in BibTeX format below). |
|
|
|
``` |
|
@inproceedings{bogatenkova-etal-2022-ispras, |
|
title = "{ISPRAS}@{F}in{TOC}-2022 Shared Task: Two-stage {TOC} Generation Model", |
|
author = "Bogatenkova, Anastasiia and |
|
Belyaeva, Oksana Vladimirovna and |
|
Perminov, Andrew Igorevich and |
|
Kozlov, Ilya Sergeevich", |
|
editor = "El-Haj, Mahmoud and |
|
Rayson, Paul and |
|
Zmandar, Nadhem", |
|
booktitle = "Proceedings of the 4th Financial Narrative Processing Workshop @LREC2022", |
|
month = jun, |
|
year = "2022", |
|
address = "Marseille, France", |
|
publisher = "European Language Resources Association", |
|
url = "https://aclanthology.org/2022.fnp-1.13", |
|
pages = "89--94" |
|
} |
|
``` |