LLM s Cannot (Yet) Match the Specificity and Simplicity of Online Communities in Long Form Question Answering (2024)

Abstract: Retail investing is on the rise, and a growing number of users is relying on online finance communities to educate themselves.However, recent years have positioned Large Language Models (LLMs) as powerful question answering (QA) tools, shifting users away from interacting in communities towards discourse with AI-driven conversational interfaces.These AI tools are currently limited by the availability of labelled data containing domain-specific financial knowledge.Therefore, in this work, we curate a QA preference dataset SocialFinanceQA for fine-tuning and aligning LLMs, extracted from more than 7.4 million submissions and 82 million comments from 2008 to 2022 in Reddit’s 15 largest finance communities. Additionally, we propose a novel framework called SocialQA-Eval as a generally-applicable method to evaluate generated QA responses.We evaluate various LLMs fine-tuned on this dataset, using traditional metrics, LLM-based evaluation, and human annotation. Our results demonstrate the value of high-quality Reddit data, with even state-of-the-art LLMs improving on producing simpler and more specific responses.

## introduction recent years have brought remarkable growth of retail investment activity @xcite , which has been accompanied and potentially fuelled by the rise of trading apps such as robinhood @xcite . retail investors, i.e., nonprofessional traders, often lack formal financial training and tend to educate themselves via the internet @xcite . the reddit platform contains the largest domain-specific communities on the internet, has over 73 million daily active unique users 1 in its subreddits (i.e., topic-specific communities), and almost 74 million users in total subscribed to subreddits focused on financial topics. these communities showcase the combined financial knowledge of the (western) internet through millions of posts and comments, with responses rated by the respective community using reddit's upvote score mechanism. however, the information acquisition process has started shifting away from online sources due to the emergence of an accessible and powerful alternative with llms such as openai's chatgpt in 2022 @xcite ) -e.g., studies show that web traffic on sites like stack overflow is declining, while usage of ai tools increases, which may be caused by a migration of users @xcite . furthermore, early studies suggest that ai tools are affecting academic learning @xcite as well as school education @xcite . smaller, instruction-tuned llms like llama-2-7b-chat @xcite and zephyr @xcite already provide a remarkable quality of texts while being small enough to run on consumergrade hardware. despite their capabilities, llms face challenges in finance due to domain-specific jargon, acronyms, and context-dependent nuances. in addition, the dynamic nature of financial markets requires adaptability and domain expertise for accurate analysis. our goal is to bridge the gap between the financial knowledge available on reddit and the capabilities of llms to provide accurate, personalized insights into financial markets. this raises the following research questions: how can the domain knowledge of reddit users be leveraged to improve the performance of llms in this field? what degree of improvement can we measure? to this end, we introduce a high-quality question-answering (qa) dataset extracted from 7.4 million posts and 82 million comments from the 15 largest financerelated subreddits. we use this dataset to fine-tune various llms for financial qa tasks using supervised fine-tuning (sft; @xcite and direct preference optimization (dpo; @xcite . the evaluation process involves a comprehensive evaluation framework that assesses the quality aspects of question answering, utilizing lexical metrics, human annotation, and llm-based evaluation. 2 our main contributions are: 1. we curate a high-quality reddit finance dataset called socialfinanceqa, the first social-media preference dataset for fine-tuning and alignment of llms for the finance domain. additionally, we provide insights into the nuanced differences between the source subreddits in our analysis. 2. we demonstrate that fine-tuning llms with so-cialfinanceqa is beneficial for the questionanswering capabilities and can help to modify stateof-the-art instruction-tuned llms to generate more specific and simpler responses. ## dataset the raw dataset has been collected from the pushshift api 3 and is a collection of submissions (i.e., posts) and comments made to the top 15 largest finance-related subreddits ranging back from january 2008 until the end of 2022. unfortunately, we cannot work with more recent data, as reddit changed its terms in mid-2023 to restrict access. all 15 subreddits combined make up over seven million submissions and more than 82 million comments ranging from january 2008 until december 2022. the size of the considered subreddits varies drastically, with wallstreetbets, explainlikeimfive, and personalfinance contributing a considerable portion of the overall dataset. characteristic words & topic modeling: understanding the topics discussed in financial subreddits is crucial to identify the interests and focus of each community and to provide an overview of the contents of our dataset. for a nuanced analysis, we extract important keywords as well as topics. tf-idf scores are calculated for all submission titles, treating each subreddit as a separate "document" @xcite . subsequently, bertopic is leveraged for topic modelling on a sample of 50,000 submission titles from each subreddit separately. we identify the following three major areas of knowledge. personal finances, budgeting, and real estate: includes topics around credit cards, debt, retirement, and real estate investments. key subreddits are personalfinance, financialindependence, finan-cialplanning, realestate, and realestateinvesting. stocks, trading, and market movements: consists of topics on the analysis and trading of specific stocks and investment funds, options, and share prices, often on a daily or weekly basis. the main subreddits are investing, stocks, stockmarket, options, pennystocks, wallstreetbets, wallstreetbetsnew. theoretical economic concepts and trends: includes topics on the economy, inflation, prices, and their effects on people. salient subreddits include economics, askeconomics, explainlikeimfive. ## methodology 4.1 models we use the pre-trained llms llama-2-7b @xcite and mistral-7b-v0.1 @xcite , as well as instruction-tuned models llama-2-7b-chat @xcite and zephyr β @xcite , to evaluate the effect of fine-tuning our dataset on variants of different llm families. we leverage different prompt templates based on the prior training of the considered models (details in section c.1 in the appendix). mistral-7b enhances efficiency by introducing changes such as sliding window attention, rolling buffer cache as well as pre-fill and chunking to their training compared to llama @xcite . zephyr-7b was created from mistral-7b using sft with the ultrachat dataset @xcite and dpo with the ultrafeedback dataset @xcite . we refer to these models as llama-2, mistral, llama-2-chat, and zephyr for simplicity. ## discussion our results demonstrate that fine-tuning llms with the socialfinanceqa dataset leads to responses with improved conciseness and simpler language, with reduced sentence count and length, reduced reading time, and a lower flesch-kincaid readability score (ranging between 4.7 -8.2, i.e., the text is comprehensible for 5 th to 8 th graders). our evaluation using socialqa-eval shows that the fine-tuned models are perceived as more specific and simple by both llm and human evaluators, benefitting the llama-2 and mistral models significantly. llama-2-chat and zephyr improve on specificity and simplicity, but the scores of the other three evaluation criteria decrease slightly. this is likely due to reddit responses being more the quantitative evaluation of long-form responses remains challenging and requires careful development and iteration to minimize randomness and subjectivity. despite the significant overlap between llm-based and human evaluation, llmbased evaluation still has limitations and is affected by the following biases. preference of llm-generated texts: our llmbased evaluation rates llama-2 and mistral much higher than human evaluators do, despite the texts often containing long, illogical lists of unrelated words. in addition, the llm judge rates the original reddit answers consistently lower than the human evaluators. this issue can only be mitigated by continuing to conduct additional human evaluation but may improve with larger, more powerful judge llms in the future. more homogeneous scores: the llm judge penalizes poor-quality texts less and gives the highest score more rarely. this leads to more homogeneous results that restrict the score range, making it more difficult to identify differences in text quality when comparing scores. besides conducting human evaluation, a potential mitigation is to normalize or recalibrate the llm scores across the full score range after the experiments. preference of longer answers: our analysis confirms previous observations that the llm judge favours longer, verbose responses @xcite . we mitigated this by introducing evaluation criteria that address specificity (coverage of only those topics that are relevant to answering the question) and simplicity (usage of short and easy-to-understand explanations). ## conclusion this paper proposes a novel qa preference dataset, socialfinanceqa, based on finance discussions on social media, and investigates the impact of using it to fine-tune and align llms. we conduct a multifaceted evaluation including our novel evaluation framework socialqa-eval and reveal reddit's strengths in specificity and simplicity of responses, benefitting both baseline and advanced, instruction-tuned llms. our findings can be beneficial for researchers and practitioners who (1) aim to create domainspecific datasets of high quality, (2) modify existing state-of-the-art llms for their specific context, or (3) evaluate the qa performance of their llms for social media text or similar. the socialfi-nanceqa preference dataset offers benefits for qa in social media contexts, but the evaluation criteria of socialqa-eval can be generalized to text generation capabilities of llms in other domains, providing a better approximation of human author responses to questions.