Leveraging LLM -based sentiment analysis for portfolio optimization with proximal policy optimization (2025)

Abstract: Reinforcement learning (RL) offers adaptive solutions to portfolio optimization, yet standard methods such as proximal policy optimization (PPO) rely exclusively on historical price data and overlook the impact of investor sentiment. We introduce sentiment-augmented PPO (SAPPO), a reinforcement learning framework that incorporates real-time sentiment signals extracted from Refinitiv financial news. Daily sentiment scores are generated using LLaMA 3.3. SAPPO integrates these signals into the PPO advantage function via a sentiment-weighted term, enabling allocation strategies that respond to both price movements and market sentiment. Experiments on a three-asset portfolio demonstrate that SAPPO increases the Sharpe ratio from 1.55 to 1.90 and reduces drawdowns relative to PPO. The optimal configuration uses a sentiment influence parameter \lambda = 0.1 , as validated through ablation studies and statistically significant t -tests ( p < 0.001 ). These findings show that sentiment-aware reinforcement learning improves trading performance and offers a robust alternative to purely price-based strategies.

## introduction portfolio optimization is a fundamental problem in financial management that aims to allocate resources across various assets to maximize returns and minimize risk @xcite @xcite . traditional approaches, such as mean-variance optimization, rely primarily on historical data to estimate expected returns and asset covariances @xcite . these static techniques often struggle to dynamically adapt to rapidly evolving market conditions, reducing their effectiveness in volatile financial environments @xcite . the emergence of reinforcement learning, and particularly deep reinforcement learning, provides promising new solutions to dynamic asset allocation problems by enabling adaptive decisionmaking @xcite @xcite . rl agents learn optimal allocation strategies through continuous interaction with financial environments, adapting policies based on market feedback @xcite . drl extends these capabilities by employing deep neural networks to approximate complex value functions and policy decisions, effectively handling nonlinear and nonstationary market behaviors @xcite @xcite . prominent drl algorithms, including ppo and deep q-networks (dqn), offer robust frameworks suitable for continuous action spaces in financial portfolio management @xcite @xcite . although ppo effectively captures market dynamics based on historical price data, existing implementations generally overlook the critical influence of investor sentiment on asset prices. financial markets exhibit significant sensitivity to sentiment-driven investor behaviors, making sentiment analysis an important supplementary component for accurately predicting market movements @xcite @xcite . advances in nlp and llms, such as finbert @xcite -which is fine-tuned for financial text-and llama 3.3 @xcite )-a general-purpose model-have made it feasible to extract and interpret sentiment from financial news, analyst reports, and market commentary. integrating sentiment signals into quantitative strategies has been shown to enhance predictive accuracy, volatility forecasting, and overall trading performance @xcite @xcite . we extend the ppo framework by introducing sentiment-augmented sappo, a novel reinforce-ment learning model explicitly incorporating realtime market sentiment into portfolio optimization. sappo integrates daily sentiment scores extracted from financial news articles using the llama 3.3 model, a transformer-based architecture. this integration provides the ppo agent with additional contextual insights beyond purely historical prices, allowing for more informed and adaptive allocation decisions. we evaluate the performance of sappo relative to a baseline ppo model that relies exclusively on historical price information. our comparative analysis employs key financial performance metrics such as the sharpe ratio, annualized returns, and maximum drawdown, assessing whether sentimentaware reinforcement learning strategies offer tangible improvements over conventional rl techniques. experimental results demonstrate that incorporating sentiment analysis leads to significantly better risk-adjusted returns and reduced drawdowns. these findings contribute to the existing literature by showcasing how leveraging financial sentiment in reinforcement learning frameworks can substantially enhance the adaptability and robustness of portfolio optimization strategies in dynamic market environments. ## related work portfolio optimization techniques have significantly evolved since @xcite introduced mean-variance optimization. traditional methods estimate asset returns and covariances from historical financial data, which often limits their adaptability in volatile market conditions @xcite . the rigidity inherent in these static optimization frameworks has motivated researchers to explore more dynamic and adaptive strategies. reinforcement learning provides an alternative approach by enabling agents to adapt asset allocation decisions through continuous interaction with the market environment @xcite . deep reinforcement learning extends these capabilities further, using deep neural networks to effectively approximate complex, nonlinear market dynamics @xcite . prominent drl algorithms, including ppo and deep q-networks (dqn), have shown robust performance in continuous decision-making scenarios such as portfolio management @xcite @xcite . ppo has gained popularity within financial drl due to its stable and effective policy updates in continuous action spaces @xcite . ppo optimizes stochastic policies iteratively by maximizing a clipped surrogate objective function, ensuring incremental updates of policy parameters. the algorithm employs an advantage function to evaluate the effectiveness of actions relative to an estimated baseline value. this structure enables ppo to balance exploration and exploitation, facilitating efficient learning in dynamic market environments @xcite ). ppo's combination of stability and adaptability has made it a reliable baseline method for portfolio optimization research. despite the strengths of ppo and related drl methods, most current implementations rely exclusively on structured numerical inputs such as historical price and volume data @xcite . these numerical approaches typically neglect qualitative market factors like investor sentiment, which play a critical role in short-term asset price fluctuations and volatility @xcite @xcite . investor sentiment strongly influences market dynamics, and purely numerical drl models often fail to anticipate sentiment-driven market shifts, leading to suboptimal allocation decisions @xcite . recent advancements in nlp have improved sentiment extraction accuracy from textual financial data. transformer-based llms, notably fin-bert @xcite , which is fine-tuned for financial sentiment analysis, and llama 3.3 @xcite , a general-purpose language model, can be applied to distinguish neutral financial reporting from sentiment-rich market commentary. these domain-specific llms outperform generalpurpose nlp models by producing more accurate and context-aware sentiment signals tailored for financial forecasting @xcite @xcite . hybrid strategies integrating sentiment analysis with quantitative finance have demonstrated significant improvements in predictive accuracy, volatility forecasting, and overall risk-adjusted performance @xcite @xcite . @xcite notably demonstrated that social media-derived sentiment can accurately predict short-term market movements. recent literature continues to support hybrid models combining structured market data and sentiment signals, frequently outperforming strategies relying solely on historical prices @xcite @xcite . we directly build upon these insights by explicitly integrating financial news sentiment into ppo. the proposed sappo model leverages sentiment scores derived from financial news using llama 3.3. our approach systematically compares sappo against traditional ppo, quantifying the benefits of incorporating sentiment signals. the results provide practical insights into enhancing adaptive portfolio management strategies within dynamic market environments. ## methodology we represent the financial market state at time step n using an array s n . this array consists of current portfolio weights w n and current adjusted closing spot prices s n for multiple assets. this setup enables the agent to make portfolio decisions informed by both its existing portfolio allocation and current market conditions @xcite . the discrete index @xmath0, where t represents continuous time and ∆@xmath1. the agent also maintains a cash account to ensure feasible transactions. each trading day ends with the observation of adjusted closing prices. the agent then computes daily returns and selects new allocation weights. portfolio rebalancing occurs at the beginning of the next trading day. trades are executed using market orders priced at the volume-weighted average price (vwap) during the first ten minutes of the trading session. this vwap-based execution reduces volatility typically associated with raw market-opening prices. we denote the action a n as the change in portfolio holdings at day n, positive elements of a n indicate asset purchases, negative elements correspond to asset sales. a self-financing constraint ensures that the total trade value sums to zero, we subtract from the portfolio transaction costs equal to 0.05% of the total turnover to reflect realistic market frictions. the immediate reward received by the agent is the logarithmic return of the portfolio, providing a scale-invariant measure. the two return definitions approximate each other for small values and are numerically stable in reinforcement learning training. the state-action value function q(s n , a n ) and the value function v (s n ) represent the expected cumulative discounted future rewards, conditional on the current state and action, and are defined as follows the state-action value function estimates cumulative future rewards achievable by selecting an action a n given the current state s n , whereas the value function estimates the expected return from the current state s n under the current policy. actions follow a stochastic policy distribution π(a n |s n ), which transitions states according to the probability distribution p(s n+1 |s n , a n ) @xcite . the discount factor γ ∈ (0, 1] determines the trade-off between immediate and long-term rewards, with γ = 0.99 employed in our experiments to prioritize future returns significantly. drl uses deep neural networks to approximate both the state-action-value function q and policy π effectively @xcite . our implementation uses ppo, a drl algorithm designed explicitly for continuous action spaces. ppo dynamically learns optimal portfolio rebalancing strategies directly from market interactions. the ppo policy uses a multivariate gaussian distribution, with the self-financing constraint in eq. ( 2 ) ensuring all trades remain budget-neutral. the policy's mean and covariance parameters are learned by a deep neural network parameterized by θ. ## experiments and results we evaluate the performance of the trained drl agents using a realistic backtesting framework on out-of-sample market data. the models are bench-marked against traditional portfolio strategies, including buy-and-hold and equal-weighted portfolios. figure 1 presents a risk-return comparison of the sappo and ppo portfolios alongside major benchmark indices. figure 1 : risk-return scatter plot as of january 1, 2020, for sappo and ppo portfolios compared to nasdaq-100, dji, and s&p 500. sappo shows the highest sharpe ratio and return among all strategies, indicating superior risk-adjusted performance from sentiment integration. the reinforcement learning agent demonstrates strong performance across multiple evaluation metrics. the annualized return of the sappo portfolio reaches 30.2%, while the ppo portfolio achieves 26.5%. both portfolios outperform major benchmark indices, including the nasdaq-100 (20%), the s&p 500 (15%), and the dji (10%). the riskreturn scatter plot (figure 1 ) highlights sappo's superior positioning in terms of volatility-adjusted returns, followed by ppo. compared to traditional indices, sappo and ppo exhibit higher returns but at the cost of increased volatility, indicating their ability to exploit market inefficiencies more effectively. the sharpe ratio of sappo surpasses that of ppo and all benchmark indices, confirming its improved risk-adjusted performance and highlighting the effectiveness of sentiment-aware reinforcement learning in portfolio optimization @xcite . figure 2 reveals how the ppo agent adjusts asset weights over time. the model increases exposure to microsoft during high-volatility periods, capitalizing on its stability, while balancing google and meta allocations for diversification. this adaptive reallocation highlights the agent's ability to respond to market changes dynamically @xcite . figure 3 presents the 30-day rolling volatility comparison, showing that the sappo and ppo portfolios exhibit higher volatility than major benchmark indices such as the nasdaq-100, s&p 500, and dji. the sappo portfolio demonstrates the highest volatility for most of the observed period, indicating a more aggressive trading strategy that reacts dynamically to market fluctuations. the ppo portfolio follows a similar trend but with slightly lower volatility, suggesting a relatively more balanced risk exposure. both sappo and ppo portfolios experience pronounced volatility spikes, particularly around mid-2019, aligning with increased market uncertainty. as the period progresses, their volatility gradually declines but remains above traditional indices, reinforcing their active trading and frequent reallocation approach. the nasdaq-100, s&p 500, and dow jones exhibit more stable and lower volatility levels, consistent with their passive investment nature. these results confirm that sentiment-aware reinforcement learning strategies adapt quickly to market changes, capturing short-term trends efficiently. however, the higher volatility associated with sappo and ppo highlights the tradeoff between increased return potential and short-term risk exposure. figure 4 : correlation heatmap comparing ppo portfolio returns with those of major indices. moderate correlation values (e.g., 0.67 with dji) suggest that ppo develops relatively independent allocation strategies, enhancing diversification. the correlation heatmap (figure 4 ) shows that the ppo portfolio maintains a moderate level of independence from major indices, with correlations of 0.67 with the dji and 0.75 with the s&p 500. this diversification suggests that the ppo agent develops unique portfolio allocation strategies, reducing reliance on broader market movements @xcite . the second experiment introduces market sentiment analysis into the ppo framework, forming the sappo model. by incorporating sentiment data from refinitiv financial news sources, processed using llama 3.3 via hugging face transformers, the agent receives an additional market signal to guide allocation decisions. this enables sentimentdriven adjustments in response to market sentiment shifts. the cumulative return comparison (figure 5 ) highlights the performance improvement achieved by sappo over standard ppo. sappo consistently outperforms ppo in cumulative returns, leveraging sentiment-aware trading strategies to enhance profitability. by reacting to shifts in market sentiment, sappo is better equipped to capture momentum and avoid adverse market conditions. sappo also shows a slightly higher daily average turnover rate of 12% compared to ppo's 3.5%. this indicates that, on average, sappo adjusts 12% of the portfolio's total value through buying and selling activities each day. this elevated turnover reflects the model's increased sensitivity to sentiment changes, resulting in more active rebalancing in response to daily news signals. these results indicate that sentiment-aware reinforcement learning enhances portfolio management by integrating external market sentiment signals. the ability to react to news-driven market sentiment fluctuations provides an additional layer of adaptability beyond price-based decision-making. the findings highlight the potential of combining reinforcement learning with financial sentiment analysis for dynamic investment strategies. appendix a reports the statistical significance of sappo's performance improvement over ppo. ## impact sentiment-aware reinforcement learning offers a measurable performance edge in portfolio optimization. sappo outperforms vanilla ppo by integrating real-time financial news sentiment into a deep reinforcement learning framework. this enhancement leads to significantly higher sharpe ratios and lower drawdowns, as confirmed by statistical significance testing and ablation studies reported in appendix a. these results validate sentiment as a meaningful input signal in dynamic allocation tasks. the findings contribute to the broader field of financial reinforcement learning by showcasing the tangible value of sentiment-aware trading strategies. sappo enables agents to respond more effectively to market fluctuations, capturing momentum and mitigating downside risk during adverse conditions. institutional investors, hedge funds, and algorithmic trading firms can benefit from models that adapt allocations based on evolving sentiment rather than relying solely on historical price movements. our research emphasizes the growing relevance of multi-modal financial decision-making. the sappo framework integrates structured market data with unstructured textual information to inform portfolio policies more holistically. the use of llama 3.3 for domain-specific sentiment extraction exemplifies the expanding role of foundation models in financial analysis. this work lays a foundation for future sentiment-aware trading systems that combine natural language understanding with adaptive reinforcement learning techniques. ## limitations and future work we demonstrate the value of sentiment-aware reinforcement learning, but it leaves several directions open for future research. the sentiment layer uses only financial news from refinitiv, processed via llama 3.3. while this ensures domain-specific, high-quality signals, it excludes other sources such as social media, earnings calls, and analyst reports. incorporating diverse sentiment channels could improve robustness and capture complementary market signals. the portfolio scope focuses on three technology stocks-google, microsoft, and meta. this controlled setting helps isolate model behavior but limits generalizability. extending sappo to sectordiverse or large-cap portfolios would test its effectiveness under broader market conditions and enhance practical relevance. the evaluation relies on historical backtesting from 2013 to 2020. this setup omits real-time market execution, order slippage, liquidity constraints, and shocks beyond the test window. future work could implement paper trading or live simulations to assess deployment readiness under actual trading constraints. the model uses daily sentiment updates available only at market close, with decisions applied the next day. this design does not exploit intra-day news shifts or fast-moving sentiment. integrating real-time or high-frequency sentiment signals could increase responsiveness and improve intra-day trading strategies. future research that addresses these limitations will improve the generalization, scalability, and practical deployment of sentiment-aware reinforcement learning in modern financial markets. ## conclusion we extend ppo by introducing a sentiment-aware reinforcement learning model for portfolio optimization. the proposed sappo framework incorporates llm-based sentiment analysis to integrate real-time financial news into trading decisions. the sentiment-enhanced model consistently delivers superior risk-adjusted performance, achieving higher sharpe ratios, stronger annualized returns, and reduced drawdowns compared to the standard ppo baseline. sappo also outperforms benchmark indices such as the nasdaq-100, s&p 500, and dji, demonstrating the value of combining sentiment signals with reinforcement learning. investor sentiment serves as a critical complementary signal, enhancing adaptability in dynamic portfolio management. incorporating sentiment provides the agent with greater adaptability to shift-ing market conditions and offers a viable alternative to purely price-driven strategies. these findings highlight the practical and theoretical relevance of sentiment-aware reinforcement learning in financial decision-making. this work lays the groundwork for future research on multimodal trading systems that combine structured market data with unstructured textual information.