The State of the Art of Large Language Models on Chartered Financial Analyst Exams (2024)

Abstract: The Chartered Financial Analyst (CFA) program is one of the most widely recognized financial certifications globally. In this work, we test a variety of state-of-the-art large language models (LLMs) on mock CFA exams to provide an overview of their financial analysis capabilities using the same evaluation standards applied for human professionals. We benchmark five leading proprietary models and eight open-source models on all three levels of the CFA through challenging multiple-choice and essay questions. We find that flagship proprietary models perform relatively well and can solidly pass levels I and II exams, but fail at level III due to essay questions. Open-source models generally fall short of estimated passing scores, but still show strong performance considering their size, cost, and availability advantages. We also find that using textbook data helps bridge the gap between open-source and proprietary models to a certain extent, despite reduced gains in CFA levels II and III. By understanding the current financial analysis abilities of LLMs, we aim to guide practitioners on which models are best suited for enhancing automation in the financial industry.

## introduction with over 190,000 charterholders in 160 markets, the chartered financial analyst (cfa) program (cfa institute, 2024a) is amongst the most soughtafter credentials for investment professionals, requiring over a thousand hours of preparation on average. cfa charterholders achieve one of the highest distinctions in investment management, possessing in-depth training in the core skills of investment strategy and high-level money management @xcite . studies have shown that cfa training enhances job performance and productivity for financial analysts in financial firms @xcite . correspondence to mahmoud.a.mahfouz@jpmchase.com given the rapid advancement of large language models (llms) @xcite @xcite and their potential for automation, it has become fundamental to ensure such models meet the necessary standards for professional application and decision-making in finance. in this regard, benchmarking the capabilities of llms on cfa exams constitutes a crucial foray. this paper provides the most comprehensive study to date on the performance of state-of-the-art llms, both open-source and proprietary, on cfa exams -aiming to give an overview of the landscape of the financial analysis capabilities of llms. we share our observations on advantages and limitations of their application. our contributions are summarized as follows: ## background earning the cfa certification requires a bachelor's degree, three years of qualified work experience, and passing all cfa exam levels (cfa institute, 2024a). the examination process is structured into three levels (i, ii, iii; see in summary, from level i to iii, llms must progress from answering questions based on concept memorization and simple calculations to understanding context and reasoning, and finally to organizing thoughts in essay writing. each level presents increasingly challenging tasks for ai. ## experimental setup dataset. as official cfa exams are not public, we use cfa mock exams purchased from analyst-prep (analystprep, 2024) in this study, covering all three levels of the cfa program. the dataset includes both mcqs and essay questions, each accompanied with corresponding answers, explanations, grading details, as well as metadata such as the cfa topic each question belongs to. we use the set of mock exams of the year 2023, which corresponds to the 2023 cfa curriculum. given that the mock exam data is secured behind a paywall, the risk of data contamination is reduced for llms. the distribution of question topics is shown in ## conclusion in this paper, we benchmark the performance of 14 llms on the cfa exams, revealing that closedsource models like gpt-4o and claude 3 opus consistently outperform their open-source counterparts. these models not only demonstrated superior accuracy across all three cfa levels, but also highlighted the importance of model architecture and training data quality over sheer size. our detailed analysis of topic-wise performance and error modes underscores the complexities llms face in financial tasks, particularly in math-heavy sections. this research advances our understanding of llm capabilities in high-stakes financial environments and identifies areas for improvement in their application to domain-specific challenges. we hope this work will serve as a point of reference for the evaluation of future models as steps forward are made, and hope the insights will inform future work developing financial domain-specific models.