Paper: Long BERT for bankruptcy prediction (ICLR-2024)

Reviewer comment (Novelty):
The method proposed in the paper lacks novelty. As shown in Table 7, the risk prediction task can essentially be framed as a text classification problem (Risky or Not Risky). However, it's worth noting that similar research utilizing a combination of BERT and LSTM for long text classification was introduced four years ago [1].

Author response:
The research paper recommended DocBERT, it might lead to confusion if is not read carefully. Besides the name is DocBERT, the authors propose the distillation of BERT into an LSTM model, that means that the final model used for prediction is an LSTM model. As is well known, LSTM models were replaced by BERT mainly because is unable to process in parallel and also not good in very long sequences (Of course are better than the RNN). This was reflected in our results, more than 5 LSTM time sequences, the performance drops. DocBERT uses a dataset that according to the provided dataset table, the maximum average number of words per document is bellow 400. Considering only 1.2 tokens per word, we are dealing with a maximum number of 480 tokens, even less than one BERT segment. Our dataset in average has 2.8K tokens per document and 2.7K words per document (English). This is the reason that a pure LSTM or BERT network does not work for processing long text.

Reviewer comment (Experiments):
A notable shortcoming of the paper is the absence of comparative experiments. The author did not provide a comparison between their proposed method and established techniques. Furthermore, there was no evaluation against advanced language models such as GPT-4 and LLaMa, which are capable of handling a wide range of tasks.

Author response:
Also as explained in the paper, we compared with established techniques for bankruptcy prediction (that are mainly numbers) and the text itself is not better than that. Because of this reason our project main goal is to have an integrated model that is having text information, numerical information and  other sources of information.
- GPT models like Llama 2  and GPT-3 are broadly used nowadays, but the main problem of these family of models (generative) is the hallucination. We are using also these models to process the text, but the amount of dirty answer that we receive is above 30%. Commercial models like GPT-4 and even the best performed versions of GPT3.5 can be costly for some companies that needs to process millions of documents. For this reason, we are focused on ways to have lighter classification models like BERT-Based models, faster and cheaper to implement. Nevertheless, we are including in the paper the results of using Llama 2 for bankruptcy prediction and the current limitations.

Reviewer comment (Novelty):
The paper also suffers from limited references to recent research. In addition to sparse citations of web pages and open-source code, the paper only includes references to a single study from 2022 and one from 2023. This lack of engagement with the latest research trends and findings weakens the paper's overall scholarly depth.

Author response:
The references provided are the ones that are focused on BERT, lately most of the research done was following the wave of GPT-based models, but those are not the models that we are pursuing because of the reason explained before. There is not much work related to bankruptcy prediction using long text sequences (or in other domains), and this is why we considered important to share our results and contributions.

Reviewer comment (Novelty):
It is unclear what the main contribution is; is it the application of an arbitrary length BERT model to a specific dataset or is it the specific formulation of the arbitrary length BERT model.

Author response:
We are clarifying that besides the research object of bankruptcy prediction we are also contributing to the architecture for dealing with long text using BERT models that can run even in a laptop with 1 GPU. We propose to use BERT+LSTM architectures for not so big documents (<=5 segments) and use BERT+concatenation for bigger documents (more than 5).

Reviewer comment (Experiments):
Was there a specific reason for how you built your arbitrary length BERT model as opposed to different versions in your references? Does your version perform better for general text classification tasks or did you just find it to work better for the bankruptcy prediction task?

Author response:
With respect to the question 1, We were applying these architectures in other models where we need to feed by page (3 segments) and other long documents (8 segments). The accuracy for other classification tasks we had were around 97 % (not related to bankruptcy). We are planning to create another paper to focus on these different use cases that the architecture can allow us.

Reviewer comment (Reproducibility):
Can you explain how the different languages are processed by your model? Does the processing to tokens effectively remove the differences?

Author response:
With respect to the question 2, the model BERT itself despite the fact of handling multiple languages, those embeddings are not fully related between each other, it can be seemed like parallel models per language. We could easily prove by comparing the distance between the embeddings of one word in different languages. In the other hand the more complexity of the GPT models can handle this problem easier. You can provide a text with different languages and to expect the answer in a specific language. The main problem of this GPT for classification is that there are going to hallucinate and create answer that could be difficult to process unless you apply more layers of validation, and also the resources required are bigger. One of our open research questions is how to address BERT embeddings of different languages to work together.

Reviewer comment (Evaluation):
Is there a way to determine some feature importance from your model. Could there be standard phrases in the dataset suggesting imminent bankruptcy that suggest that it should be relatively easy for a machine learning model to predict a company going under?

Author response:
With respect to the question 3. Thanks for the contribution, we are starting to include Explainability and ranking the top tokens that can lead us to the bankruptcy prediction.

Reviewer comment (Experiments):
It was rather unclear how the training and test set were determined. Was a random selection of documents held out in the test set? Was the test set the same as that used by Mai et al.? Finally, where do the "Risky" vs "Non Risky" labels come from?

Author response:
The test set was determined randomly by document. As the model propose to use many segments for each document, we can not do randomization of the segments by default, we need to do it by document to avoid losing the integrity in the document, because it will allow us to integrate in the next phase in a LSTM or concatenation model. As explained in the paper, the weakness of the paper of Mai et al, as other similar research, is that they are using the risk section of the SEC fillings, where are implicitly mentioning the level of risk of the company and an analysis, in that case they do not need to process the entire document, just the paragraphs that contains that information. In our case the dataset does not contain an explicit section of the risk analysis, and this is why we need to feed the entire annexes in order to look for risk information.

Reviewer comment (Experiments):
The authors should have directly compared their performance with respect to Mai et al.'s work or even a basic approach, e.g., randomly sampling a single 512 token segment from each training document. This would have demonstrated the benefit of the summarization and integrated prediction approach and justified their usage.

Author response:
We thank the proposition of selecting randomly 512 tokens (510) of each document to use as a base line to compare our model.

Reviewer comment (Novelty):
The technical contribution may not be significant enough for publication at ICLR. While a lot of time, energy and technical expertise was required to generate the empirical findings, it is unclear how broadly this work can be applied.

Author response:
As mentioned in the paper, the prediction of the bankruptcy is one of the goals, the other objective is to provide with two architectural options (LSTM or concatenation) for dealing with long text prediction. In our research project we are already using these models to do predictions by documents (other labels) and by pages, that requires more than 510 tokens. In consequence, the architecture can be broadly applied to any domain that requires the use of BERT models for long text prediction.

Reviewer comment (Writing):
What does the following phrase on page 4 mean: "For this reason, the shuffling and selection of train, test and validation data are done before the pre-processing"?

Author response:
About the question of shuffling, is because we shuffle the data at document level because in the preprocessing stage, we already have the documents divided into segments. To avoid the corruption of the integrity of the documents itself, we do the shuffling before the pre-processing phase.

Reviewer comment (Experiments):
Is sequence/overlap important at all? What if one just randomly samples segments instead?

Author response:
As suggested by the other reviewed we are going to use random selection of segments to make a base line to compare with.

Reviewer comment (Experiments):
Can this approach be applied to other corpuses with large text documents?

Author response:
In our research project we are already using these models to do predictions by documents (other labels) and by pages, that requires more than 510 tokens. In consequence, the architecture can be broadly applied to any domain that requires the use of BERT models for long text prediction.

Reviewer comment (Experiments):
Relies solely on textual information, and as acknowledged by the authors, numerical data remains crucial for risk prediction.

Author response:
One of the main goals of this paper is to see how good a bankruptcy prediction can be done with a model using only textual information, and it was mentioned in the abstract and conclusions, this is part of a bigger project that is going to integrate textual information, numerical information and other sources of information. We are going to put more emphasis in this in the paper to make it clearer.

Reviewer comment (Experiments):
Should still compare to traditional default prediction models in finance, like FIM.

Author response:
We have compared the results of our model with other methods like Logistic Regression, Random Forest and LightBM, that are also working with a very similar data. We thank the reviewer for providing the info about the FIM models but, we could not find a specific finance paper about a method / model FIM for bankruptcy prediction. We thank in advance is the paper's name can be provided in order to improve our work.