Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models (2025)

Abstract: As large language models (LLMs) increasingly permeate the financial sector, there is a pressing need for a standardized method to comprehensively assess their performance. Existing financial benchmarks often suffer from limited language and task coverage, low-quality datasets, and inadequate adaptability for LLM evaluation. To address these limitations, we introduce Golden Touchstone, a comprehensive bilingual benchmark for financial LLMs, encompassing eight core financial NLP tasks in both Chinese and English. Developed from extensive open-source data collection and industry-specific demands, this benchmark thoroughly assesses models’ language understanding and generation capabilities. Through comparative analysis of major models such as GPT-4o, Llama3, FinGPT, and FinMA, we reveal their strengths and limitations in processing complex financial information. Additionally, we open-source Touchstone-GPT, a financial LLM trained through continual pre-training and instruction tuning, which demonstrates strong performance on the bilingual benchmark but still has limitations in specific tasks. This research provides a practical evaluation tool for financial LLMs and guides future development and optimization.The source code for Golden Touchstone and model weight of Touchstone-GPT have been made publicly available at https://github.com/IDEA-FinAI/Golden-Touchstone .

## introduction the rapid development of both proprietary @xcite @xcite @xcite and opensource large language models (llms) @xcite @xcite @xcite @xcite @xcite has led to their increasing application in various fields, including finance @xcite , healthcare @xcite , and law @xcite . among these, the financial sector shown in figure .1 stands out as a critical area for llm application due to its rich textual information and high practical value. in recent years, a variety of advanced financial large language models (finllms) have emerged, capable of specialized tasks such as financial sentiment analysis, content summarization, stock movement prediction, and question answering @xcite @xcite @xcite . these models leverage unique frameworks and tuning methods to enhance their performance on domain-specific benchmarks, offering robust solutions for realworld financial applications. however, existing financial benchmarks often suffer from limited language and task coverage, low-quality datasets, and inadequate adaptability for llm evaluation, leading to poor evaluation results @xcite @xcite @xcite @xcite . to address these challenges, we propose golden touchstone, a comprehensive bilingual benchmark for financial llms, encompassing eight core financial nlp tasks in both chinese and english. golden touchstone provides high-quality datasets, task-aligned metrics, and instructional templates to guide llms in generating task-appropriate responses. we evaluated several state-of-the-art models, including gpt-4o, qwen-3, llama-3, fingpt, and finma, on this benchmark. results indicate that while these models perform well on tasks such as sentiment analysis and entity extraction, there is significant room for improvement in areas like midea group's subsidiary pilot program for a diversified employee stock ownership plan must align with the company's "smart home + intelligent manufacturing" dual-smart strategy or belong to a company-intended key cultivation and relatively independent industrial platform. ## conclusion in this study, we introduce the golden touchstone benchmark, the inaugural structured and comprehensive bilingual benchmark specifically designed for english-chinese financial nlp. this benchmark encompasses a wide array of financial nlp tasks, including natural language understanding (nlu) and natural language generation (nlg) across eight categories: sentiment analysis, classification, entity extraction, summarization, stock market prediction, question answering, relation extraction, and multiple choice. by leveraging existing high-quality open-source financial datasets, we curated representative datasets and selected appropriate evaluation metrics for each task category. utilizing these resources, we conducted extensive evaluations of current models such as gpt-4o and prominent open-source financial llms, including fingpt and finma, thereby establishing performance benchmarks for financial llms within bilingual contexts. moreover, we contributed to the community by open-sourcing touchstone-gpt, a robust financial llm that employs a two-stage training approach and has demonstrated superior input-based inference capabilities on the golden touchstone benchmark compared to gpt-4o. our open-source initiative provides a bilingual english-chinese evaluation framework aimed at fostering the sustainable development of llms in a multilingual financial environment.