nastyboget commited on
Commit
5b32a4a
β€’
1 Parent(s): 42f8ada

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +142 -0
README.md CHANGED
@@ -1,3 +1,145 @@
1
  ---
2
  license: mit
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ language:
4
+ - en
5
+ - fr
6
+ - es
7
  ---
8
+
9
+ # πŸ”₯ Classifiers of FinTOC 2022 Shared task winners (ISPRAS team) πŸ”₯
10
+
11
+ Classifiers of texual lines of English, French and Spanish financial prospects in PDF format for the [FinTOC 2022 Shared task](https://wp.lancs.ac.uk/cfie/fintoc2022/).
12
+
13
+ ## πŸ€— Source code πŸ€—
14
+
15
+ Training scripts are available in the repository https://github.com/ispras/dedoc/ (see `scripts/fintoc2022` directory).
16
+
17
+ ## πŸ€— Task description πŸ€—
18
+
19
+ Lines are classified in two stages:
20
+ 1. Binary classification title/not title (title detection task).
21
+ 2. Classification of title lines into title depth classes (TOC generation task).
22
+
23
+ There are two types of classifiers according to the stage:
24
+ 1. For the first stage, **binary classifiers** are trained. They return `bool` values: `True` for title lines and `False` for non-title lines.
25
+ 2. For the second stage, **target classifiers** are trained. They return `int` title depth classes from 1 to 6. More important lines have a lesser depth.
26
+
27
+ ## πŸ€— Results evaluation πŸ€—
28
+
29
+ The training dataset contains English, French, and Spanish documents, so three language categories are available ("en", "fr", "sp").
30
+ To obtain document lines, we use [dedoc](https://dedoc.readthedocs.io) library (`dedoc.readers.PdfTabbyReader`, `dedoc.readers.PdfTxtlayerReader`), so two reader categories are available ("tabby", "txt_layer").
31
+
32
+ To obtain FinTOC structure, we use our method described in [our article](https://aclanthology.org/2022.fnp-1.13.pdf) (winners of FinTOC 2022 Shared task!).
33
+ The results of our method (3-fold cross-validation on the FinTOC 2022 training dataset) for different languages and readers are given in the table below (they slightly changed since the competition finished).
34
+ As in the FinTOC 2022 Shared task, we use two metrics for results evaluation (metrics from the [article](https://aclanthology.org/2022.fnp-1.12.pdf)):
35
+ **TD** - F1 measure for the title detection task, **TOC** - harmonic mean of Inex F1 score and Inex level accuracy for the TOC generation task.
36
+
37
+ <table border="1" class="dataframe">
38
+ <thead>
39
+ <tr style="text-align: left;">
40
+ <th></th>
41
+ <th>TD 0</th>
42
+ <th>TD 1</th>
43
+ <th>TD 2</th>
44
+ <th>TD mean</th>
45
+ <th>TOC 0</th>
46
+ <th>TOC 1</th>
47
+ <th>TOC 2</th>
48
+ <th>TOC mean</th>
49
+ </tr>
50
+ </thead>
51
+ <tbody>
52
+ <tr>
53
+ <th>en_tabby</th>
54
+ <td>0.811522</td>
55
+ <td>0.833798</td>
56
+ <td>0.864239</td>
57
+ <td>0.836520</td>
58
+ <td>56.5</td>
59
+ <td>58.0</td>
60
+ <td>64.9</td>
61
+ <td>59.800000</td>
62
+ </tr>
63
+ <tr>
64
+ <th>en_txt_layer</th>
65
+ <td>0.821360</td>
66
+ <td>0.853258</td>
67
+ <td>0.833623</td>
68
+ <td>0.836081</td>
69
+ <td>57.8</td>
70
+ <td>62.1</td>
71
+ <td>57.8</td>
72
+ <td>59.233333</td>
73
+ </tr>
74
+ <tr>
75
+ <th>fr_tabby</th>
76
+ <td>0.753409</td>
77
+ <td>0.744232</td>
78
+ <td>0.782169</td>
79
+ <td>0.759937</td>
80
+ <td>51.2</td>
81
+ <td>47.9</td>
82
+ <td>51.5</td>
83
+ <td>50.200000</td>
84
+ </tr>
85
+ <tr>
86
+ <th>fr_txt_layer</th>
87
+ <td>0.740530</td>
88
+ <td>0.794460</td>
89
+ <td>0.766059</td>
90
+ <td>0.767016</td>
91
+ <td>45.6</td>
92
+ <td>52.2</td>
93
+ <td>50.1</td>
94
+ <td>49.300000</td>
95
+ </tr>
96
+ <tr>
97
+ <th>sp_tabby</th>
98
+ <td>0.606718</td>
99
+ <td>0.622839</td>
100
+ <td>0.599094</td>
101
+ <td>0.609550</td>
102
+ <td>37.1</td>
103
+ <td>43.6</td>
104
+ <td>43.4</td>
105
+ <td>41.366667</td>
106
+ </tr>
107
+ <tr>
108
+ <th>sp_txt_layer</th>
109
+ <td>0.629052</td>
110
+ <td>0.667976</td>
111
+ <td>0.446827</td>
112
+ <td>0.581285</td>
113
+ <td>46.4</td>
114
+ <td>48.8</td>
115
+ <td>30.7</td>
116
+ <td>41.966667</td>
117
+ </tr>
118
+ </tbody>
119
+ </table>
120
+
121
+ ## πŸ€— See also πŸ€—
122
+
123
+ Please see our article [ISPRAS@FinTOC-2022 shared task: Two-stage TOC generation model](https://aclanthology.org/2022.fnp-1.13.pdf)
124
+ to get more information about the FinTOC 2022 Shared task and our method of solving it.
125
+ We will be grateful, if you cite our work (see citation in BibTeX format below).
126
+
127
+ ```
128
+ @inproceedings{bogatenkova-etal-2022-ispras,
129
+ title = "{ISPRAS}@{F}in{TOC}-2022 Shared Task: Two-stage {TOC} Generation Model",
130
+ author = "Bogatenkova, Anastasiia and
131
+ Belyaeva, Oksana Vladimirovna and
132
+ Perminov, Andrew Igorevich and
133
+ Kozlov, Ilya Sergeevich",
134
+ editor = "El-Haj, Mahmoud and
135
+ Rayson, Paul and
136
+ Zmandar, Nadhem",
137
+ booktitle = "Proceedings of the 4th Financial Narrative Processing Workshop @LREC2022",
138
+ month = jun,
139
+ year = "2022",
140
+ address = "Marseille, France",
141
+ publisher = "European Language Resources Association",
142
+ url = "https://aclanthology.org/2022.fnp-1.13",
143
+ pages = "89--94"
144
+ }
145
+ ```