---
language:
- en
---
# Introduction
This is the implementation of the BERT model using the LongNet structure (paper: https://arxiv.org/pdf/2307.02486.pdf). The model is pre-trained with 10-K/Q filings of US firms from 1994 to 2018.

# Training code
https://github.com/minhtriphan/LongFinBERT-base/tree/main

# Training configuration
* The model is trained with 4 epochs using the Masked Language Modeling (MLM) task;
* The masking probability is 15%;
* Details about the training configuration are given in the log file named `train_v1a_0803_1144_seed_1.log`;

# Instruction to load the pre-trained model
* Clone the git repo
```
git clone https://github.com/minhtriphan/LongFinBERT-base.git
cd LongBERT
```
or
```
!git clone https://github.com/minhtriphan/LongFinBERT-base.git
import sys
sys.path.append('/LongFinBERT-base')
```

* Load the pre-trained tokenizer, model configuration, and model weights
```
from model import LongBERT
from custom_config import LongBERTConfig
from tokenizer import LongBERTTokenizer

backbone = 'minhtriphan/LongFinBERT-base'

tokenizer = LongBERTTokenizer.from_pretrained(backbone)
config = LongBERTConfig.from_pretrained(backbone)
model = LongBERT.from_pretrained(backbone)
```

# Contact
For any comments, questions, or feedback, please get in touch with us via phanminhtri2611@gmail.com or triminh.phan@unisg.ch.

# Paper
(updating)