nlpaueb commited on
Commit
b6b9f45
1 Parent(s): a7d27ac

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +13 -1
README.md CHANGED
@@ -19,11 +19,14 @@ widget:
19
 
20
  <img align="center" src="https://i.ibb.co/0yz81K9/sec-bert-logo.png" alt="SEC-BERT" width="400"/>
21
 
 
 
22
  SEC-BERT is a family of BERT models for the financial domain, intended to assist financial NLP research and FinTech applications.
23
  SEC-BERT consists of the following models:
24
  * SEC-BERT-BASE (this model): Same architecture as BERT-BASE trained on financial documents.
25
  * [SEC-BERT-NUM](https://huggingface.co/nlpaueb/sec-bert-num): Same as SEC-BERT-BASE but we replace every number token with a [NUM] pseudo-token handling all numeric expressions in a uniform manner, disallowing their fragmentation
26
  * [SEC-BERT-SHAPE](https://huggingface.co/nlpaueb/sec-bert-shape): Same as SEC-BERT-BASE but we replace numbers with pseudo-tokens that represent the number’s shape, so numeric expressions (of known shapes) are no longer fragmented, e.g., '53.2' becomes '[XX.X]' and '40,200.5' becomes '[XX,XXX.X]'.
 
27
 
28
  ## Pre-training corpus
29
 
@@ -31,12 +34,15 @@ The model was pre-trained on 260,773 10-K filings from 1993-2019, publicly avail
31
 
32
  ## Pre-training details
33
 
 
 
34
  * We created a new vocabulary of 30k subwords by training a [BertWordPieceTokenizer](https://github.com/huggingface/tokenizers) from scratch on the pre-training corpus.
35
  * We trained BERT using the official code provided in [Google BERT's GitHub repository](https://github.com/google-research/bert)</a>.
36
  * We then used [Hugging Face](https://huggingface.co)'s [Transformers](https://github.com/huggingface/transformers) conversion script to convert the TF checkpoint in the desired format in order to be able to load the model in two lines of code for both PyTorch and TF2 users.
37
  * We release a model similar to the English BERT-BASE model (12-layer, 768-hidden, 12-heads, 110M parameters).
38
  * We chose to follow the same training set-up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e-4.
39
  * We were able to use a single Google Cloud TPU v3-8 provided for free from [TensorFlow Research Cloud (TRC)]((https://sites.research.google/trc), while also utilizing [GCP research credits](https://edu.google.com/programs/credits/research). Huge thanks to both Google programs for supporting us!
 
40
 
41
  ## Load Pretrained Model
42
 
@@ -182,10 +188,13 @@ model = AutoModel.from_pretrained("nlpaueb/sec-bert-base")
182
 
183
  ## Publication
184
 
185
- The model has been officially released with the following article:<br>
 
 
186
  **["FiNER: Financial Numeric Entity Recognition for XBRL Tagging"](https://arxiv.org/abs/2203.06482)**<br>
187
  Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion Androutsopoulos and George Paliouras<br>
188
  In the Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) (Long Papers), Dublin, Republic of Ireland, May 22 - 27, 2022
 
189
 
190
  ```
191
  @inproceedings{loukas-etal-2022-finer,
@@ -207,6 +216,8 @@ In the Proceedings of the 60th Annual Meeting of the Association for Computation
207
 
208
  ## About Us
209
 
 
 
210
  [AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr) develops algorithms, models, and systems that allow computers to process and generate natural language texts.
211
 
212
  The group's current research interests include:
@@ -218,5 +229,6 @@ text classification, including filtering spam and abusive content,
218
  machine learning in natural language processing, especially deep learning.
219
 
220
  The group is part of the Information Processing Laboratory of the Department of Informatics of the Athens University of Economics and Business.
 
221
 
222
  [Manos Fergadiotis](https://manosfer.github.io) on behalf of [AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr)
 
19
 
20
  <img align="center" src="https://i.ibb.co/0yz81K9/sec-bert-logo.png" alt="SEC-BERT" width="400"/>
21
 
22
+ <div style="text-align: justify">
23
+
24
  SEC-BERT is a family of BERT models for the financial domain, intended to assist financial NLP research and FinTech applications.
25
  SEC-BERT consists of the following models:
26
  * SEC-BERT-BASE (this model): Same architecture as BERT-BASE trained on financial documents.
27
  * [SEC-BERT-NUM](https://huggingface.co/nlpaueb/sec-bert-num): Same as SEC-BERT-BASE but we replace every number token with a [NUM] pseudo-token handling all numeric expressions in a uniform manner, disallowing their fragmentation
28
  * [SEC-BERT-SHAPE](https://huggingface.co/nlpaueb/sec-bert-shape): Same as SEC-BERT-BASE but we replace numbers with pseudo-tokens that represent the number’s shape, so numeric expressions (of known shapes) are no longer fragmented, e.g., '53.2' becomes '[XX.X]' and '40,200.5' becomes '[XX,XXX.X]'.
29
+ </div>
30
 
31
  ## Pre-training corpus
32
 
 
34
 
35
  ## Pre-training details
36
 
37
+ <div style="text-align: justify">
38
+
39
  * We created a new vocabulary of 30k subwords by training a [BertWordPieceTokenizer](https://github.com/huggingface/tokenizers) from scratch on the pre-training corpus.
40
  * We trained BERT using the official code provided in [Google BERT's GitHub repository](https://github.com/google-research/bert)</a>.
41
  * We then used [Hugging Face](https://huggingface.co)'s [Transformers](https://github.com/huggingface/transformers) conversion script to convert the TF checkpoint in the desired format in order to be able to load the model in two lines of code for both PyTorch and TF2 users.
42
  * We release a model similar to the English BERT-BASE model (12-layer, 768-hidden, 12-heads, 110M parameters).
43
  * We chose to follow the same training set-up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e-4.
44
  * We were able to use a single Google Cloud TPU v3-8 provided for free from [TensorFlow Research Cloud (TRC)]((https://sites.research.google/trc), while also utilizing [GCP research credits](https://edu.google.com/programs/credits/research). Huge thanks to both Google programs for supporting us!
45
+ </div>
46
 
47
  ## Load Pretrained Model
48
 
 
188
 
189
  ## Publication
190
 
191
+ <div style="text-align: justify">
192
+
193
+ If you use this model cite the following article:<br>
194
  **["FiNER: Financial Numeric Entity Recognition for XBRL Tagging"](https://arxiv.org/abs/2203.06482)**<br>
195
  Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion Androutsopoulos and George Paliouras<br>
196
  In the Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) (Long Papers), Dublin, Republic of Ireland, May 22 - 27, 2022
197
+ </div>
198
 
199
  ```
200
  @inproceedings{loukas-etal-2022-finer,
 
216
 
217
  ## About Us
218
 
219
+ <div style="text-align: justify">
220
+
221
  [AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr) develops algorithms, models, and systems that allow computers to process and generate natural language texts.
222
 
223
  The group's current research interests include:
 
229
  machine learning in natural language processing, especially deep learning.
230
 
231
  The group is part of the Information Processing Laboratory of the Department of Informatics of the Athens University of Economics and Business.
232
+ </div>
233
 
234
  [Manos Fergadiotis](https://manosfer.github.io) on behalf of [AUEB's Natural Language Processing Group](http://nlp.cs.aueb.gr)