Financial Named Entity Recognition: How Far Can LLM Go? (2025)

Abstract: The surge of large language models (LLMs) has revolutionized the extraction and analysis of crucial information from a growing volume of financial statements, announcements, and business news. Recognition for named entities to construct structured data poses a significant challenge in analyzing financial documents and is a foundational task for intelligent financial analytics. However, how effective are these generic LLMs and their performance under various prompts are yet need a better understanding. To fill in the blank, we present a systematic evaluation of state-of-the-art LLMs and prompting methods in the financial Named Entity Recognition (NER) problem. Specifically, our experimental results highlight their strengths and limitations, identify five representative failure types, and provide insights into their potential and challenges for domain-specific tasks.

## introduction as an increasing amount of information is contained within documents and text available online, utilizing a series of natural language processing (nlp) techniques to automate the process of extracting meaningful information from unstructured text has become a critical task, especially in the financial domain @xcite . among all, named entity recognition (ner) serves as a foundational first step in identifying key entities, such as persons, organizations, and locations, enabling the construction of knowledge graphs and other applications. with the surge of large language models (llms), llms have demonstrated transformative capabilities in generative tasks, leveraging reinforcement learning from human feedback (rlhf) @xcite . llms achieve remarkable performance across a wide range of nlp tasks with minimal adaptation @xcite . however, their ability to perform domain-specific tasks, such as ner in the financial domain, remains less explored. for instance, in the sentence "johnson brothers rethink plan for st. paul waterfront shepard road development.", a generic ner model might incorrectly classify the company "johnson brothers"as a person. this understanding is critical, as it could influence numerous applications in finance. in this paper, we aim to evaluate the capabilities of state-of-the-art llms in performing ner tasks within the financial domain, their response to various prompt types, and their limitations in this context. to achieve this, we conduct a systematic analysis and present experimental results, comparing the effectiveness of leading llms with recent fine-tuned approaches. specifically, we evaluate three advanced llms with different parameter sizes, gpt-4o (openai, 2024), llama-3.1 @xcite , and gemini-1.5 @xcite -under three distinct prompting techniques: direct prompting, in-context learning, and chain-ofthought (cot) prompting. we perform our study by investigating the following two research questions (rqs): to sum up, the main contributions of this paper are as follows: ## study setup to understand current llms' capabilities in handling financial ner problems, we choose three state-of-the-art llms, each with three popular prompting strategies. we further select two representative transformer-based models and fine-tune them on financial data for comparison. ## discussion the findings of our study highlight several potential directions for improving the performance of llms on financial ner tasks: tuning llms for the financial domain. a significant proportion of the observed failure cases involve domain-specific proper nouns. fine-tuning llms with financial data could enhance their ability to accurately recognize such entities. implementing self-correction strategies. our analysis in rq2 identifies common mistakes made by llms in the finer-ord task. developing selfverification prompting strategies could allow llms to recognize and address these errors, thereby reducing recurrent failures. ## conclusion this study presents the first systematic evaluations of generic llms in the finer-ord task under different prompt designs, compared to state-of-theart fine-tuned transformer-based models. through comprehensive experiments with llms and their related lightweight versions, we demonstrate the capabilities and limitations of generic llms in handling domain-specific tasks. our findings categorize five representative types of failures, along with their underlying causes. we release artifcats for future research 1 .