Exploring the Readiness of Prominent Small Language Models for the Democratization of Financial Literacy (2024)

Abstract: The use of small language models (SLMs), herein defined as models with less than three billion parameters, is increasing across various domains and applications. Due to their ability to run on more accessible hardware and preserve user privacy, SLMs possess the potential to democratize access to language models for individuals of different socioeconomic status and with different privacy preferences. This study assesses several state-of-the-art SLMs (e.g., Apple’s OpenELM, Microsoft’s Phi, Google’s Gemma, and the Tinyllama project) for use in the financial domain to support the development of financial literacy LMs. Democratizing access to quality financial information for those who are financially under educated is greatly needed in society, particularly as new financial markets and products emerge and participation in financial markets increases due to ease of access. We are the first to examine the use of open-source SLMs to democratize access to financial question answering capabilities for individuals and students. To this end, we provide an analysis of the memory usage, inference time, similarity comparisons to ground-truth answers, and output readability of prominent SLMs to determine which models are most accessible and capable of supporting access to financial information. We analyze zero-shot and few-shot learning variants of the models. The results suggest that some off-the-shelf SLMs merit further exploration and fine-tuning to prepare them for individual use, while others may have limits to their democratization. Code to replicate our experiments is shared.

## introduction recent advances in natural language processing (nlp) @xcite @xcite @xcite @xcite @xcite have helped push artificial intelligence (ai) as a field into the public sphere of awareness. a language model (lm) can be defined as a statistical model which predicts the conditional probability of a word given some context @xcite . lms present an opportunity within the financial sector, where financial knowledge resides primarily with financial professionals. with the advent of new financial technology (fintech) applications, such as online brokerage apps and tax preparation software, the general population has greater access to manage their own finances. fintech applications, new financial markets (e.g., crypto and nfts), and social media have facilitated a spike in participation in financial markets @xcite . increased access to financial markets, the proliferation of new markets, and the constant evolution of financial regulations and codes has brought about unwise use of markets and money. for example, the investment behavior of many individual investors resembles gambling @xcite , draws on simple heuristics such as mimicking the behavior of social influencers @xcite , or even stems from a fear of missing out on social investing trends @xcite . although access to financial markets has improved, financial literacy is still limited by the scattered and sometimes costly nature of financial data. the current democratization of financial technology is leading to uneducated and potentially risky behavior by individuals @xcite . language models are used in various financial sector activities, such as predictive modeling (stock prediction), portfolio management, financial text mining, providing financial advice and customer service, picking stocks, and generating meaningful narratives from unstructured financial data @xcite @xcite @xcite @xcite @xcite . although large language models (llms) have shown promise in various financial sector activities descried above, these models require access to substantial computing resources or expensive services provided by third parties. for example, bloomberg gpt is a costly llm currently utilized by financial professionals that is offered through the bloomberg terminal @xcite . these larger models are not particularly accessible to individuals with lower socioeconomic status or low technological capability. small language models may provide a solution to truly democratize access to language models to support financial literacy in a low-cost and privacy-preserving manner. slms capable of answering financial questions may also be beneficial for finance, accounting, and economics students. although some finance programs in the u.s. invest in the bloomberg terminal, not all universities in the u.s. or globally can afford such tools. slms that can run on student computers, lab computers, or even small, cost-effective university servers could provide students with access to slms for developing financial literacy and exploring the use of lms within finance, accounting, and economics. although it may be difficult or impossible to create an slm with the same capabilities as an llm @xcite , slms may still provide value in democratizing access to information in a low-cost and private manner. the term small language model (slm) is not yet well defined in the literature. there is not a consensus on what qualifies as an slm, yet the topic continues to be of interest to researchers @xcite @xcite . herein, we define slms as language models with three billion or fewer parameters. this is an arbitrary threshold, but represents an approximate parameter count that can be executable with reasonable inference times on consumer grade technologies, such as personal computers, laptops, and even mobile phones. due to the smaller number of parameters, these models are less resource intensive than their larger counterparts and can be run privately on an individual's computing device, making them an excellent candidate for democratizing lms for financial literacy. based on the potential of slms for democratization, this study seeks to examine the following research question: are state-of-the-art, off-the-shelf slms capable of answering financial questions for individuals with only zero-shot or few-shot learning? the rest of this paper is organized as follows. section 2 covers related work including a review of existing financial lms and prominent slms. section 3 describes the selection of slms, model parameters, ground-truth data, and comparison criteria for the study. section 4 presents the results of the analysis of nine slm models based on criteria related to memory usage, inference time, similarity measures, and a readability test. section 5 summarizes our work, identifies which slm performed the best on the selected criteria, and makes suggestions for future efforts. in addition to the core content of our paper, a section for limitations and an ethics statement are included alongside acknowledgements at the end of the paper. ## related work our work explores the application of slms in the support of financial literacy, which branches across research on financial language models and on slms. ## method to evaluate whether state-of-the-art slms are prepared to answer financial questions, we: 1) identified a set of state-of-the-art open source slms, 2) selected important criteria to evaluate the model output to ensure the outputs were accessible to individuals from different socioeconomic groups and education levels, 3) identified a set of open-source question/answer pairs to evaluate the models, and 4) conducted a study to determine how well each model performed on the selected criteria. ## results as outlined in the following sections, the models possess different strengths and limitations that affect their ability to be utilized by individuals on consumer-grade electronics to develop financial literacy. we assess each model on a variety of aspects, including memory usage, inference time, capability of answering financial questions, and the readability of model outputs. ## conclusions the results of the study suggest that certain slms are better candidate models for future fine-tuning and development to support the democratization of financial literacy lms than others. see appendix b for further comparisons. several of apple's openelm models show great promise for future study of the democratization of financial literacy lms. the low memory requirements and inference times, and higher readability scores exhibited by these models make them ideal for democratization. of these models, the openelm-270m model provides the greatest accessibility in terms of gpu requirements, inference times, and readable outputs. the openelm-270m zero-shot and few-shot models also scored in the top four performers on at least one of the similarity score metrics, which larger counterparts (e.g., the openelm-450m and openelm-3b models) did not achieve. the openelm-1.1b few-shot model showed very promising scores across the similarity comparison metrics, scoring in the top four models for all of the similarity metrics. however, the higher gpu requirements and inference time limits its accessibility and usefulness. the microsoft phi models also show some promise for future study in this domain, but likely only for college educated individuals. although the phi-1b model exhibited some of the worst comparability metric scores of all of the models and the lowest zero-shot readability score, the phi-1.5b and phi-2b few-shot models both appeared in the the top four performers on two similarity comparison metrics. of course, these models also exhibited low readability scores, which changed only slightly after few-shot learning. the other major limiting factor of the phi models is their higher gpu requirements. although the phi models show promise for future financial literacy models, they may be better suited for college students or graduates than by those with a lower reading level. google's gemma model also shows promise for moderately powerful consumer-grade technology. it had faster inference times than most of the models. the zero-shot model also had the best readability score, making it the most accessible in terms of reading level. the gemma-2b model also exhibited some of the best similarity comparison scores, scoring in the top four for all metrics, though only after few-shot learning. this model could be ideal for individuals with access to respectable consumer-grade computing devices. like gemma-2b, the tinyllama-1.1b model had good inference speeds and reasonable readability scores. the few-shot model also appeared in the top four performers on similarity comparison metrics twice. also like gemma-2b, the tinyllama-1.1b model suffers from higher gpu memory requirements. overall, this study suggests that apple's new openelm-270m model deserves further research attention from the lens of democratizing language models for the greatest number of individuals. this model has a small memory profile, fast inference times, produces reasonable results as compared to ground-truth finance responses, and produces reasonable readability scores. however, in some cases, such as for individuals with more powerful computing devices, the openelm-1.1b and gemma-2b models share similar gpu requirements, inference times, and high readability and similarity comparison scores. the phi-1.5b and phi-2b models may also be useful for college educated individuals. society is on the cusp of democratizing financial investment information, which has long been limited to the financial elite. we encourage researchers to continue to explore slms and further fine-tune models like openelm to support the development of financial literacy for the greatest population possible. we provide a starting point for future research by identifying the most promising models that meet criteria for truly democratized models.