Fine-Tuning Large Language Models for Stock Return Prediction Using Newsflow (2024)

Abstract: Large language models (LLMs) and their fine-tuning techniques have demonstrated superior performance in various language understanding and generation tasks.This paper explores fine-tuning LLMs for predicting stock returns with financial newsflow.Return prediction is fundamental for subsequent tasks like portfolio construction and optimization in quantitative investing. We formulate the model to include a text representation and forecasting modules. We propose to compare the encoder-only and decoder-only LLMs, considering they generate text representations in distinct ways.The impact of these different representations on return forecasting remains an open question.Meanwhile, we compare two simple methods of integrating LLMs’ token-level representations into the forecasting module.The experiments on real investment universes reveal that:(1) aggregated representations from LLMs’ token-level embeddings generally produce return predictions that enhance the performance of long-only and long-short portfolios;(2) in the relatively large investment universe, the decoder LLMs-based prediction model leads to stronger portfolios, whereas in the small universes, there are no consistent winners;(3) return predictions derived from LLMs’ text representations are a strong signal for portfolio construction, outperforming conventional sentiment scores.These findings shed light on developing suitable LLM fine-tuning methods for return prediction-based portfolio construction.

## introduction quantitative investing relies on extracting quantitative features or signals from various data sources including market prices, economic indicators, financial text, etc., to build and optimize investment portfolios @xcite . in recent years, the use of text data for quantitative investing has grown significantly, thanks to the advancement of natural language processing (nlp) techniques @xcite @xcite . in particular, large language models (llms) have demonstrated superior performance on various language understanding and generation tasks @xcite @xcite @xcite , and the finetuning technique allows for adapting the pre-trained llms to fit investing-related applications @xcite . this paper foot_0 is focused on return prediction with financial news for stock portfolio construction. return forecasting is useful for picking stocks with profit potentials to include in portfolios. financial news reports on events and announcements related to companies, industries, the economy, etc., and shows notable predictive power for stock future performance in previous studies @xcite @xcite . the conventional way of applying financial news data to stock picking involves a multi-step extraction-and-validation process as illustrated in fig. 1 (a), i.e., formulating the numerical features (e.g., sentiments, popularity, etc.) with the expectation that these features have a predictive relationship with stock future performance (e.g., forward return, volatility, etc.) @xcite , developing the feature extraction process (e.g., train a financial sentiment classification model), and validating the predictive power of extracted features by statistical analysis or building forecasting models. this process might be timeconsuming and require additional data (e.g., labeled sentiment data) and continuous refinements. llms generate numerical representations (or embeddings) of text that capture semantic relations, and these representations can naturally serve as features for forecasting tasks. based on this intuition, this paper explores direct news-to-return prediction through fine-tuning llms. fig. 1 illustrates the difference between the conventional feature extraction-and-validation process and our llm-based news-to-return process. though some previous works attempted to use text embedding for forecasting @xcite @xcite , few works have explored the potential of fine-tuning llms for stock return forecasting with newsflow. moreover, this paper has the contribution as follows: ## related work numerous works have investigated using financial text data for forecasting tasks. @xcite extracted the sentiment score from financial newsflow, social media, and tweets for stock price predicting. @xcite explored learning numeric representations of financial news by attention mechanisms for modeling stock movements. @xcite studied combining sentiment and text representations for return prediction. the advent of llms and related techniques provides a new powerful way of using text data for forecasting tasks in quantitative investing @xcite . encoder-only models such as bert @xcite and de-berta @xcite , focus on learning contextual embeddings for input text. decoderonly models like gpt-3 @xcite and @xcite are trained to generate text by predicting the next token in a sequence. llms are pre-trained on vast amounts of text data to learn general language patterns. the prompt technique is to design specific inputs to guide the pre-trained llm to produce the desired output without modifying the llm's parameters @xcite @xcite . fine-tuning techniques adjust the pre-trained llm's parameters to adapt to specific tasks @xcite @xcite . in particular, parameter-efficient fine-tuning techniques have gained popularity @xcite @xcite . some recent works use llms as feature extractors to obtain predictive signals from text. @xcite explored the fine-tuning of pre-trained llms to provide more accurate financial sentiment analysis. instead of fine-tuning llms, @xcite extracted factors from the financial news and price history by prompts on generative llms. @xcite used chainof-thought prompts @xcite on generative llms to analyze financial statements. @xcite fine-tuned llms for generating text responses of prediction and explanations. unlike existing works that extract features from text using llms, this paper focuses on fine-tuning llms to directly model the relationship between financial news text and numerical return values. meanwhile, we evaluate the text representations from different types of llms to study their different effectiveness for the return forecasting task. ## experiments in this part, we present some main results, while further details and a qualitative interpretation of predictions are provided in the appendix. ## conclusion this paper focuses on return forecasting with financial newsflow for quantitative portfolio construction. unlike the conventional feature extractionand-validation workflow, this paper explores finetuning llms to directly model the relationship between news text and stock forward return. the experiment results reveal the key findings: (1) aggregated representations from llms' tokenlevel embeddings generally produce the return predictions that enhance the portfolio performance; several open questions remain for future research. for instance, it is unclear whether the underperformance of encoder-only deberta in the large universe is due to the model size or other factors, and why deberta has varying performance in different small universes. evaluating recently proposed large encoder-only llms @xcite would be an interesting follow-up.