Next-Year Bankruptcy Prediction from Textual Data: Benchmark and Baselines (2022)

Abstract: Models for bankruptcy prediction are useful in several real-world scenarios, and multiple research contributions have been devoted to the task, based on structured (numerical) as well as unstructured (textual) data. However, the lack of a common benchmark dataset and evaluation strategy impedes the objective comparison between models. This paper introduces such a benchmark for the unstructured data scenario, based on novel and established datasets, in order to stimulate further research into the task. We describe and evaluate several classical and neural baseline models, and discuss benefits and flaws of different strategies. In particular, we find that a lightweight bag-of-words model based on static in-domain word representations obtains surprisingly good results, especially when taking textual data from several years into account. These results are critically assessed, and discussed in light of particular aspects of the data and the task. All code to replicate the data and experimental results will be released.

## introduction since the seminal work of @xcite , bankruptcy prediction has received considerable attention by both academics and practitioners. a sound prediction model has numerous applications. for instance, successful quantitative methods can help professionals, such as creditors and investors, in managing financial risk @xcite . furthermore, as @xcite has shown that economy-wide levels of bankruptcy risk play a structural role in propagating recession, regulators can use bankruptcy prediction models to monitor the financial health of key economic actors and control systematic risk. a large number of bankruptcy prediction models have been proposed in literature, such as the models from @xcite , @xcite , odom * * corresponding author and @xcite , @xcite and @xcite . however, it appears difficult to compare these studies and objectively assess progress in the field. we have identified the following three aspects that make comparison difficult: (1) the temporal nature and typical class imbalance of the bankruptcy prediction task leads to strongly deviating evaluation scenarios, (2) there is little consensus on the key evaluation metrics, and (3) there is no standard benchmark dataset. these issues are further discussed in section 2.2. in order to overcome these problems, we have designed and described our experimental setup with reproducibility on a common benchmark in mind. to that end, scripts to reconstruct the benchmark and reproduce the presented results are available at https://github.com/henriarnoug/ bankruptcybenchmarkbaselines . note that this paper investigates the potential to predict bankruptcy from textual disclosures only. extending this benchmark to the hybrid case of combined textual and structured features will be part of our future work. the contributions of this paper are as follows: (1) we introduce a reproducible benchmark for textbased bankruptcy prediction, based on novel and established economic datasets, (2) classical as well as neural baseline prediction models are provided, including results on next-year bankruptcy prediction from multiple years of textual data, and (3) insights into the results are given along with pointers to potential next steps in bankruptcy prediction. ## related work after a general overview of research on bankruptcy prediction (section 2.1), we describe some key aspects that make contributions in literature hard to compare (section 2.2). three years prior to bankruptcy "we are highly leveraged and a substantial portion of our liquidity needs arise from debt service requirements and from funding our costs of operations and capital expenditures, including acquisitions... we entered into a new asset-based revolving credit facility (abl facility)... secured by substantially all of our assets..." one year prior to bankruptcy " ... we received a waiver of certain events of default under the tla arising from the inclusion of a going concern qualification from our registered public accounting firm, breach of the ebitda financial covenant, and cross-default arising from the default under our abl facility... in order to address our liquidity issues and provide for a restructuring of our indebtedness to improve our long-term capital structure, we have entered into a restructuring support agreement ... pursuant to a prepackaged plan of reorganization to be filed in a case commenced under chapter 11 of the united states bankruptcy code..." ## methodology in the next sections, we describe the data sources (section 3.1) and motivate our design choices for the benchmark (section 3.2), document preprocessing (section 3.3), and the selected evaluation metrics (section 3.4). ## results and discussion when taking a single year of history into account, the w2v model is superior in terms of auc, recall@100 and cap while the tf-idf model achieves the best results in terms of ap. for the 2019 test set, the tf-idf model contains a slightly higher proportion of positive samples in the first decile but the w2v model is superior from the second decile onwards. when taking three years of history into account, the w2v model achieves the best results for the auc and cap metrics while the tf-idf model performs better with respect to ap and recall@100. when looking at decile rank, the w2v models performs best, having ranked all bankrupt companies in the top 30% of the samples for the 2019 test set. for each model, auc and cap are better when taking three years of history into account compared to a single year of history. the same applies for the decile rank (except for the tf-idf model and the longformer model in the first decile). ap is generally worse when using a longer history, except for the binary model with test set 2020 and the longformer model with test set 2019. the recall@100 metric varies over the two setups. we observe that the binary models based on a mere 20 keywords perform surprisingly well, although not on par with the tf-idf and w2v models. note that the latter are based on many more features (in particular, hyperparameter tuning led for the tf-idf model to 25.000 (10.000) features for single (three) year history). the relatively good performance of the binary baseline suggests that the presence of few very informative words is a strong indicator for impending bankruptcy. as an illustration, we list the top 15 unigrams and bigrams selected by the single year binary model in furthermore, the longformer model performs significantly worse than the other models. since we do not finetune the generic pre-trained longformer model on the our end task, the resulting generic document representations appear unable to capture those features in the text that are important for bankruptcy prediction. the w2v model leads overall to the best results, in particular for auc (on which model selection was performed over the validation set) and cap, and better than the longformer over the entire line. even though it is based on the mean representation over all words, it appears the relevant information regarding bankruptcy prediction is still sufficiently present. as opposed to the longformer, the w2v document representations come from in-domain data (i.e., pretrained on 10-k reports). finally, we critically evaluate the observed performance improvements for the three-year w.r.t. single-year history setting. the binary and tf-idf models are by construction unable to distinguish the different years, but in principle the w2v and longformer models could learn to capture a deteriorating financial situation over three years of history. however, when evaluating our final w2v models on the test sets with only complete observations (i.e., discard test instances with missing reports), we get the following results. the single year of history auc is 0.93 (0.94) and the recall@100 is 0.48 (0.36) while the three year history auc is 0.93 (0.93) and recall@100 was 0.24 (0.28). these results imply that our models taking three years of history into account only lead to better performance metrics as they are able to generate meaningful predictions for companies with some missing reports. building more expressive models that can leverage the changes in the documents over the years present an interesting avenue for future research. ## conclusion and future work bankruptcy prediction models are valuable in many real-world applications and have received considerable research attention. however, assessing actual progress in the field is not obvious due to the lack of a common benchmark. in this work, we introduce such a benchmark for bankruptcy prediction using textual data along with several baseline models that demonstrate the predictive value of the textual data. we give a detailed discussion on our benchmark and evaluation design choices and share our code to reproduce the experiments. in future work, we will focus on more advanced models to take into account the temporal evolution of enterprises' financial situation and more advanced language representations (i.e., by finetuning transformer encoders). we also plan to extend the benchmark with structured financial data to build hybrid prediction models.