Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLM s (2025)

Abstract: Large Language Models (LLMs) have been shown to perform well for many downstream tasks. Transfer learning can enable LLMs to acquire skills that were not targeted during pre-training. In financial contexts, LLMs can sometimes beat well-established benchmarks. This paper investigates how well LLMs perform at forecasting corporate credit ratings. We show that while LLMs are very good at encoding textual information, traditional methods are still very competitive when it comes to encoding numeric and multimodal data. For our task, current LLMs perform worse than a more traditional XGBoost architecture that combines fundamental and macroeconomic data with high-density text-based embedding features. We investigate the degree to which the text encoding methodology affects performance and interpretability.

## introduction corporate credit ratings indicate a borrower's ability to service its debt obligations and are a forwardlooking measure of a company's health @xcite . a company's credit rating is significant since it affects the cost of raising capital, which in turn could finance future infrastructure to increase revenue or profitability. an optimistic rating can result in a virtuous cycle whereby it is easier to raise money and grow the business @xcite , and a pessimistic rating can result in a vicious cycle in which competition can grow faster due to cheaper debt obligations. knowing which cycle a company may enter can be advantageous to investors. many major funds are also not allowed to own sub-prime assets, which makes forecasting a drop in credit rating very important so that the fund has more time to divest from the asset, which could result in a higher close price. figure 1 : example of the best-performing feature -highdensity clustering @xcite . each dot represents a sentence, and the colored areas representing high-density regions of the embedding space. recently, there has been a surge of interest in text-based forecasting @xcite @xcite . one reason for this trend is the progress that has been made in text modelling in general @xcite . given that financial news is often first disseminated through written or spoken communications @xcite , rather than in numeric or tabular formats, there has been a hope that important information can be included in models sooner than was possible without using linguistic information. another reason is that language can provide relevant context and forward-looking information, whereas financial numeric reporting alone is inherently retrospective. contained within a company's filings, text can provide insights about the future strategic direction of the company as well as historical information. the majority of text-based forecasting research has been focused on short text sequences, drawing primarily from sources such as social media @xcite , news articles @xcite , and analyst recommendations @xcite . in contrast, many fundamental fi-nancial documents, such as company filings, earnings call transcripts, and patents, are very long. considering that the shorter texts often serve as summaries, reflections or commentaries on the detailed primary sources and that speed of information acquisition is essential in finance @xcite , there must be more focus on text-based forecasting for longer text sequences. this paper evaluates the most effective ways to model longer text sequences within a text-based forecasting task. linked to the recent progress of llms, there has been growing interest in applying these models in a variety of downstream applications @xcite . it has been shown that as generative llms scale, they acquire abilities that were not present in smaller llm variants @xcite , such as modular arithmetic @xcite , nlu @xcite , commonsense reasoning @xcite , fact-checking @xcite and so on. these abilities have been impressive, but it is best not to be over-optimistic. the lack of training data transparency associated with some of the best-performing llms means that we cannot be certain whether some of the performance gains are due to the memorisation of benchmarks being in the training datasets @xcite @xcite . generative llms also seem to have a mediocre understanding of concepts like negation and complex logical reasoning @xcite @xcite . these limitations in the capabilities of llms could prove to be very consequential in financial contexts. in this paper, we test generative llms on a complex linguistic task, which has never been fully solved by human experts: credit rating forecasting. we show that while llms encode text-based information very well, they are not good at incorporating numeric information, and underperform a boosting-tree baseline. the contributions of this paper are as follows: ## dataset in part due to the lack of large open-source or readily available datasets with temporal metadata, most of the financial text-based forecasting studies have either focused on expensive proprietary datasets, or datasets spanning 2-3 years @xcite , making results hard to replicate and potentially biased to a specific time. while temporal bias in language-based tasks is hard to avoid due to limited historical data @xcite , we aim to reduce this by using a dataset spanning 23 years, increasing the models' exposure to different economic contexts. the cost and lack of transparency of large datasets have hindered progress in the field and made it harder to build on promising work due to the difficulty of replicating results. as such, all data used in this paper is either open source or available with a wrds subscription to enable effective dataset reconstruction. the data used in this paper is from us-based companies. ## methodology we deploy two frameworks to test different architectural methodologies on this task. the same data are provided to each of the frameworks. the first framework is a feature-based discriminative approach that uses a more traditional boosting-tree model and tests the different ways to encode the textual data. the second uses generative llms and prompting to output one of a fixed list of labels through a greedy search algorithm (app. a). ## results the results from the xgboost baseline are outlined in combining features yields a performance boost, particularly when hec features are integrated with numerical data. in addition, the models pick up on different signals. the final row of ## interpretability one of the disadvantages of generative llms is that, to a large degree, they are black boxes. while some work has successfully used attention weights and internal model states to analyse generated prompts @xcite , this is not a mature research area. much of the mechanistic interpretability literature has focused on toy models @xcite , and while there has been some progress made on extracting features from larger models using sparse autoencoders @xcite , the field is still very far from completely solving interpretability within llms. in the absence of a complete solution, it is worth acknowledging the interpretable features that traditional methods use. regulation in major economies increasingly emphasizes explainability alongside performance (european commission, 2020; us congress, 2022; uk secretary of state for dcms, 2022). the xgboost-a hec framework not only achieves the best performance among the models in this paper but also enables users to interpret its decisions. this section provides an example of how we can use this framework to understand the reasons behind decisions. the most obvious advantage of a feature-based system is that important features can be identified. ## conclusion the paper shows that while llms are good at encoding textual data and inferring signals that traditional methods cannot pick up on when combined with numerical data in the prompt there is performance deterioration. the other advantage is that traditional methods offer increased interpretability and a better understanding of the mechanisms behind certain predictions. in addition, traditional approaches don't suffer to the same extent from complications associated with training data contamination and memorization @xcite since the models used for the traditional features are much smaller than the generative models and memorisation in llms exhibits scaling law behaviour. while it is not impossible that the text data used in this paper was included in the training of the studied llms, any potential influence would likely have a greater impact on generative models, thereby reinforcing the findings of this paper. there has been some work jointly encoding text using generative llms with time-series information @xcite , but more work needs to be done to determine the best methodology for combining long text sequences with numerical information while utilizing the benefits of generative llm natural language understanding. this paper shows that combining multimodal information within the prompt is not sufficient.