From Numbers to Words: Multi-Modal Bankruptcy Prediction Using the ECL Dataset (2023)

Abstract: In this paper, we present ECL, a novel multimodal dataset containing the textual and numerical data from corporate 10K filings and associated binary bankruptcy labels. Furthermore, we develop and critically evaluate several classical and neural bankruptcy prediction models using this dataset. Our findings suggest that the information contained in each data modality is complementary for bankruptcy prediction. We also see that the binary bankruptcy prediction target does not enable our models to distinguish next year bankruptcy from an unhealthy financial situation resulting in bankruptcy in later years. Finally, we explore the use of LLMs in the context of our task. We show how GPT-based models can be used to extract meaningful summaries from the textual data but zero-shot bankruptcy prediction results are poor. All resources required to access and update the dataset or replicate our experiments are available on github.com/henriarnoUG/ECL.

## introduction bankruptcy has far-reaching consequences that extend beyond the business owners, affecting various stakeholders such as employees, suppliers and creditors. on an economy-wide scale, bankruptcy risk plays a structural role in propagating recession @xcite . predicting the occurrence and timing of this corporate event precisely is challenging, due to the external factors and complex financial dynamics at play. yet, certain warning signals, such as decreasing revenues and rising debt, can serve as an indication of imminent bankruptcy. therefore, several researchers have directed their efforts towards the development of sound bankruptcy prediction models in the past decades @xcite . increasingly advanced prediction models @xcite , combined with well-chosen, informative features @xcite , have led to increased predictive performance in the field. in this paper, we contribute to the literature in two ways. first, we present ecl, a new dataset that contains the textual and numerical data from corporate 10k filings (cf. section 2) and associated binary bankruptcy labels. it is a unique compilation of three existing data sources: the edgarcorpus @xcite , compustat foot_0 and the lopucki bankruptcy research database. 1 second, we present baseline bankruptcy prediction models on each data modality, as well as on the combination, and critically evaluate their performance. based on our findings, we identify and formulate interesting avenues for future research. in recent work @xcite , we argue that contributions in the field of bankruptcy prediction are difficult to compare since (1) the considered evaluation scenarios vary strongly (which is related to the temporal nature of the data and the class imbalance) (2) there is no consensus on key evaluation metrics and (3) there is a lack of benchmark datasets. we introduced a carefully designed evaluation strategy, applied to a textonly benchmark dataset. in this paper, we adopt this evaluation setup, report the suggested evaluation metrics for our baseline models on ecl, and share our code and dataset to encourage reproducible future research (see github.com/ henriarnoug/ecl). our findings suggest that the textual and numerical content from a 10k contain complementary information for bankruptcy prediction. in some cases, the management of a company explicitly state that they consider filing for bankruptcy, making the prediction task based on text trivial. if this is not mentioned, the accounting figures are more informative for bankruptcy prediction. furthermore, the results show that our models trained on binary labels cannot distinguish 10k records filed in the year before bankruptcy from those records filed by financially unhealthy companies that did not file for bankruptcy just yet. based on this finding we argue that modelling the financial health of a company with a more gradual label is an interesting direction for future research. finally, we explore the potential of llms in the context of our task. we show that gpt-generated summaries from the text contained in the 10k filings are useful for bankruptcy prediction. despite this promising result, we find that the zero-shot prediction results of gpt-3.5 are significantly worse than the results of a simple keyword-based tf-idf model. the structure of this paper is as follows. in section 2 we present our dataset and discuss the prediction task. section 3 contains an overview of our experimental setup. the results, along with an in-depth qualitative analysis, are presented in section 4. the potential of llms for our task is explored in section 5 while section 6 concludes. ## the ecl dataset large companies operating in the u.s. are required to submit a variety of filings with the securities and exchange commission (sec) throughout the year. potential investors and other stakeholders use these filings to gain insight into the financial performance, business operations, risks and other aspects of the company of interest. notably, the most widely consulted sec filing is the form 10k, which is reported annually and contains detailed information on a company's past fiscal year. a 10k filing is composed of 15 different items including a description of the business (item 1), the management discussion and analysis (item 7) and a section on executive compensation (item 11), among others. item 8 of a form 10k contains the consolidated financial statements such as the balance sheet, the income statement and the cashflow statement. we carefully compiled the edgar-compustat-lopucki dataset, further referred to as ecl, containing data in two modalities (i.e., textual and numerical) from such 10k records that companies filed with the sec in the past. we present the dataset in the context of our current work on bankruptcy prediction, but are convinced that the multi-modal dataset has other possible uses, in terms of analysis or predictive modelling of a companies' financial and business situation. ## experimental setup as discussed above, a 10k record consists of various items and contains different data modalities. first, we separately explore the predictive value of (1) the numerical financial data of the 10k's and ( 2 ) the text in the reports, specifically from item 7: the management discussion and analysis. afterwards, we build a predictive model that uses both data types jointly. in this section we cover the design of the models and briefly discuss the training details. ## bankruptcy predictions result and analysis in this section we motivate the choice of performance metrics, report the results of the models on the test set, trained with optimal hyperparameter values, and discuss our most interesting findings. ## the potential of llms for text-based bankruptcy prediction gpt prompting strategy recently, large language models (llms) have shown to be tremendously successful on a variety of tasks, including financial text classification @xcite and zero-shot text summarisation @xcite . in this section, we explore how such models can be used in the context of bankruptcy prediction. more specifically, we will use gpt-3.5 turbo @xcite with a context window of 16,000 tokens to (1) summarise the text from the md&a section of the 10k filings into a single paragraph and (2) for zero-shot bankruptcy prediction. due to the associated costs, we do not perform the gpt-based experiments on the entire dataset. instead, we sample a balanced train set and a random test set from ecl of 1000 instances each. using a single prompt (shown in figure 4 in the appendix), we ask the model to summarise the md&a, with a particular focus on the elements that are indicative for the financial health of the company, and to assign a score, ranging from 1 to 10, that indicates how likely it is that the company will file for bankruptcy in the next year. the extracted summaries are then used to re-train the tf-idf baseline and the roberta model, which is now able to use a compact version of the entire document instead of only the first 512 tokens. we use the same training details as before with two exceptions. since our sampled training set is balanced, we no longer use a weighted loss function and reduce the batch size to 16. for the zero-shot bankruptcy prediction task, we extract the scores that gpt-3.5 assigned to each document in the test set, foot_9 rank the test set accordingly and calculate the performance metrics. since many instances are assigned the same score, we repeat this process 50 times and randomly shuffle documents with the same score in the ranked test set, to quantify the level of variation in the metrics due to the discrete nature of the gpt-assigned scores. the results of the models trained on the summaries, the original models and gpt-3.5 zero-shot scores on the sampled test of 1000 instances are reported in ## conclusion in this paper, we present ecl, a multi-modal dataset of textual and numerical data from corporate 10k filings and associated binary bankruptcy labels. we also present several classical and neural bankruptcy prediction models and provide an in-depth qualitative analysis of the results. first, our findings highlight the complementarity of the information contained in both data modalities. in the text, the management of the company sometimes explicitly states that they consider filing for bankruptcy in the coming year, making the prediction task trivial for a keyword-based tf-idf model. if this is not mentioned, the financial numerical features are better predictors for bankruptcy. the best results are attained when we combine the predictions of the textual and numerical models in an ensemble. second, we argue that our models achieve acceptable prediction levels that may prove useful in actual applications such as the automated screening of companies' financial status, although there clearly is room for further research on ecl. we did observe that our models, trained on binary bankruptcy labels, cannot distinguish between 10k records filed in the year preceding bankruptcy and records filed by financially unhealthy companies that are close to bankruptcy but not within one year. this indicates that modelling the financial health of a company using more fine-grained prediction targets, is an interesting avenue for future research as well. finally, we study the potential of llms in the context of bankruptcy prediction. we observe that the zero-shot bankruptcy prediction results of the gpt-3.5 model are poor. nonetheless, owing to the large context window of the model, we demonstrate its value by using the llm to extract meaningful summaries of the text in the 10k's for the bankruptcy prediction task. you are a financial analyst, specialized in assessing companies' financial health and communicating with clients. system i have the management discussion and analysis from a company's 10k report, and would like to know the elements that could indicate its financial health. user show me the 10k report. i will summarize the management discussion and analysis section into a 20-line paragraph with a focus on the company's financial health. assistant thank you. after the summary, give a conclusion, starting with 'conclusion: ', where you assign a score from 1 to 10, indicating how likely it is that the company will file for bankruptcy in the next year, with 1 being 'next-year bankruptcy extremely unlikely' and 10 'next-year bankruptcy extremely likely'. the report was filed on + "date" + here it is: + "text"