F in M o E : A M o E -based Large C hinese Financial Language Model (2025)

Abstract: Large-scale language models have demonstrated remarkable success, achieving strong performance across a variety of general tasks. However, when applied to domain-specific fields, such as finance, these models face challenges due to the need for both specialized knowledge and robust general capabilities. In this paper, we introduce FinMoE, a MOE-based large-scale Chinese financial language model that bridges the gap between general language models and domain-specific requirements. FinMoE employs a dense MoE architecture, where all expert networks are simultaneously activated and dynamically combined to effectively integrate general linguistic understanding with domain-specific financial expertise. Experimental results demonstrate that FinMoE achieves state-of-the-art performance on both general-purpose and financial benchmarks at a comparable scale, validating its ability to balance domain specialization with general knowledge and reasoning.

## introduction in recent years, large-scale language models such as instructgpt, chatgpt, and gpt-4 @xcite @xcite have demonstrated impressive advancements in conversational and generative ai. these models excel in understanding complex language structures and engaging in natural, coherent interactions, pushing the boundaries of natural language processing (nlp) applications. their development has profoundly influenced industries, enabling innovations in areas such as virtual assistants and intelligent customer support systems. however, applying such models effectively to domain-specific contexts, such as finance, remains a challenging task due to the need for domain expertise and specialized data. a significant challenge in building a financial large language model lies in achieving strong fi-nancial task performance while maintaining robust general capabilities. general-purpose models often lack sufficient domain-specific knowledge to address the intricacies of financial problems effectively. at the same time, models tailored exclusively to finance risk losing their ability to perform well on tasks requiring broader reasoning and general linguistic understanding. therefore, there is a critical need for financial models that combine strong general knowledge with specialized financial capabilities to ensure they can handle complex real-world financial scenarios effectively. moreover, answering financial questions often requires integrating financial concepts, domainspecific methodologies, and general reasoning frameworks. financial tasks are not only about understanding specialized terminology but also about applying general reasoning skills, contextual awareness, and common-sense knowledge to solve problems. this interplay between domain-specific and general-purpose knowledge is essential for tasks such as financial analysis, decision support, and risk assessment, where precise and reliable insights are paramount. to address these challenges, we propose fin-moe, a large-scale chinese financial language model built on a dense mixture-of-experts (moe) architecture @xcite @xcite . unlike conventional models, finmoe employs a dense moe approach where all expert networks are activated simultaneously during each forward pass, and their outputs are dynamically combined through a weighted summation mechanism. this design ensures that fin-moe leverages both domain-specific and generalpurpose knowledge, effectively capturing the intricate relationships between financial concepts and broader reasoning capabilities. by combining financial expertise with robust general abilities, finmoe bridges the gap between general and financial domain requirements. built upon a dense moe architecture and carefully designed training strategies, finmoe achieves stateof-the-art performance on both general-purpose and financial-specific benchmarks at a comparable scale. this demonstrates its ability to effectively balance domain-specific knowledge and general reasoning capabilities, providing accurate and reliable insights for financial institutions, investors, and researchers. ## related work the mixture of experts (moe) architecture @xcite has a long and storied history in the field of deep learning, dating back to its introduction as a method to enhance predictive performance by combining multiple expert models. this architecture was initially conceived as a way to address the limitations of single-model approaches, which often struggle to capture the complexity and diversity of real-world data. at its core, the moe framework is built on the principle of specialization, where each expert network is designed to focus on specific aspects of the data or tasks at hand. this modular approach allows the model to leverage the strengths of multiple specialized networks, each contributing unique insights to the overall prediction process. ## model structure in this section, we describe the architecture of fin-moe to address the unique challenges of financial tasks. unlike traditional sparse moe approaches, finmoe adopts a dense moe structure, which activates all experts simultaneously during each forward pass and combines their outputs through a weighted summation. this design choice ensures that every expert contributes to the final prediction, leveraging the capabilities of different experts in general and financial fields. the dense moe model in finmoe can be formally described as: the gating network g, parameterized by θ, plays a crucial role in moe. it determines the contribution of each expert network to the final output through a softmax-based gating mechanism. given an input x, the gating network first generates a gating value g(x; θ) , which is then passed through a softmax function to produce the gating weights g(x; θ) i . these weights determine how much emphasis each expert network f i receives during the summation. the gating network consists of a linear layer and a softmax layer, making it computationally efficient and effective in dynamically adjusting the expert contributions based on the input. this dynamic gating mechanism enables finmoe to adaptively integrate the outputs of all experts, ensuring that the model can effectively capture the complex relationships present in financial and general data. ## conclusion in this paper, we introduced finmoe, a mixtureof-experts-based large chinese financial language model designed to address the limitations of general-purpose language models in financial tasks. finmoe effectively integrates domain-specific financial expertise with strong general knowledge, achieving state-of-the-art performance across both general and financial benchmarks at a comparable scale. its innovative architecture, comprehensive training techniques enable it to deliver accurate and scalable solutions for complex financial tasks. future work will focus on further enhancing fin-moe's adaptability to evolving financial contexts and expanding its applications to other real-world scenarios @xcite @xcite , solidifying its role as a powerful tool in advancing ai research and practice.