FLAG - TRADER : Fusion LLM -Agent with Gradient-based Reinforcement Learning for Financial Trading (2025)

Abstract: Large language models (LLMs) fine-tuned on multimodal financial data have demonstrated impressive reasoning capabilities in various financial tasks. However, they often struggle with multi-step, goal-oriented scenarios in interactive financial markets, such as trading, where complex agentic approaches are required to improve decision-making. To address this, we propose FLAG-Trader, a unified architecture integrating linguistic processing (via LLMs) with gradient-driven reinforcement learning (RL) policy optimization, in which a partially fine-tuned LLM acts as the policy network, leveraging pre-trained knowledge while adapting to the financial domain through parameter-efficient fine-tuning. Through policy gradient optimization driven by trading rewards, our framework not only enhances LLM performance in trading but also improves results on other financial-domain tasks. We present extensive empirical evidence to validate these enhancements.

## introduction algorithmic financial trading represents a critically complex decision-making domain that perpetually grapples with the intertwined challenges of synthesizing heterogeneous market signals and dynamically refining strategies @xcite @xcite . traditional reinforcement learning (rl) approaches, despite their theoretical grounding in markov decision processes (mdps), confront three fundamental limitations when deployed in financial markets. firstly, their inability to coherently model multimodal market states-spanning sequential price movements, quantitative technical indicators, and unstructured textual sentiments-compromises data integration @xcite . secondly, non-stationary data distributions inherent to financial systems systematically erode strategy generalizability across market regimes @xcite . thirdly, the heavy reliance on manually crafted technical indicators (e.g., macd, rsi) and complex feature engineering @xcite introduces subjective biases, leads to information loss, and reduces the robustness of real-time decision-making, especially in volatile market conditions. the emergence of large language models (llms) offer significant potential for financial decision-making by addressing key limitations of rl-based trading strategies. leveraging their transformer architecture, they serve as multimodal feature extractors, integrating time-series and textual data, capturing long-range dependencies, and generalizing across market regimes, while also extracting nuanced sentiment signals without relying on manually crafted features @xcite @xcite @xcite . nonetheless, adapting llms for trading presents key challenges. first, their deployment often relies on agentic frameworks @xcite @xcite , which incur high implementation and operational costs due to their complex architecture. second, llms are primarily trained for static text generation, making them ill-suited for sequential decisionmaking in trading. this prompts us to the following question: can we design a framework that seamlessly integrates llms' reasoning with rl's reward-driven optimization to tackle the challenges of financial sequential decisionmaking? to resolve these interconnected challenges, we propose flag-trader, a unified architecture integrating linguistic processing (via llms) with gradient-driven rl policy optimization, as shown in figure 1 . this framework advances two synergistic innovations: a parameter-efficient fine-tuning module that jointly encodes temporal market data the environment provides the current state s t . a prompt containing task details, the action space, and the current state is fed into the llm, which outputs a trading action a t . the action is executed in the environment, yielding a reward r(s t , a t ) and next state s t+1 . the log-likelihood log π θ (a t |lang(s t )) is then leveraged by a policy gradient method (e.g., ppo), with experience tuples stored in a replay buffer for iterative updates. and textual streams into unified state representations and a hybrid rl component that explicitly incorporates external environment reward gradients into policy updates, ensuring alignment with trading performance metrics. our contributions are summarized as follows. first, we propose the flag-trader framework, where a partially fine-tuned llm acts as the policy network, leveraging pre-trained knowledge while adapting to the financial domain through parameter-efficient fine-tuning. the model processes market data using a textual state representation, enabling it to interpret and respond to market conditions effectively. rather than fine-tuning the entire llm, only a subset of its parameters is updated, balancing domain adaptation and knowledge retention. this design allows flag-trader to make informed trading decisions while remaining computationally efficient and preserving the llm's general reasoning capabilities. second, we conduct extensive experiments to evaluate flag-trader across multiple financial trading tasks. our results demonstrate that flag-trader consistently outperforms both the buyand-hold strategy and llm-agentic baselines, particularly in terms of cumulative return and sharpe ratio, which we prioritize for financial performance assessment. notably, our approach enables a smallscale (135m parameter) open-source llm to surpass much larger proprietary models, highlighting the effectiveness of rl fine-tuning in optimizing llm-driven trading strategies. these findings underscore the potential of integrating llms with rl to enhance financial decision-making while maintaining computational efficiency. ## problem statement we define the financial decision-making process as a finite horizon partially observable markov decision process (pomdp) with time index {0, • • • , t }, represented by the tuple: @xmath0, a, t , r, γ), where each component is described in detail below. ## flag-trader to tackle the challenge of directly fine-tuning an llm for both alignment and decision-making, we introduce flag-trader, a fused llm-agent and rl framework for financial stock trading. in flag-trader, a partially fine-tuned llm serves as the policy network, leveraging its pretrained knowledge while adapting to the financial domain through parameter-efficient fine-tuning, as shown in figure 2 . the model processes financial information using a textual state representation, allowing it to interpret and respond to market conditions effectively. instead of fine-tuning the entire network, only a subset of the llm's parameters is trained, striking a balance between adaptation and knowledge retention. in the following, we will present the prompt input design and the detailed architecture of flag-trader. ## experiments this section describes the overall experimental design and environmental setup for comparing the performance of different trading agents under consistent conditions. policy_net outputs a t from action space {"buy," "sell," "hold"} based on ( 8 ); 8: execute action a t in the environment and observe reward r(s t , a t ) and transition to new state s t+1 ; 9: store experience tuple (s t , a t , r t , s t+1 ) in replay buffer b; 10: if t mod τ = 0 then 11: update policy head θ p according to ( 10 ); 12: update value head θ v according to ( 11 ); 13: update the trainable llm layers θ train according to ( 12 ). ## conclusion in this paper, we introduced flag-trader, a novel framework that integrates llms with rl for financial trading. in particular, flag-trader leverages llms as policy networks, allowing for natural language-driven decisionmaking while benefiting from reward-driven optimization through rl fine-tuning. our framework enables small-scale llms to surpass larger proprietary models by efficiently adapting to market conditions via a structured reinforcement learning approach. through extensive experiments across multiple stock trading scenarios, we demonstrated that flag-trader consistently outperforms baseline methods, including llm-agentic frameworks and conventional rl-based trading agents. these results highlight the potential of integrating llms with rl to achieve adaptability in financial decision-making.