D e F ine: Decision-Making with Analogical Reasoning over Factor Profiles (2025)

Abstract: LLMs are ideal for decision-making thanks to their ability to reason over long contexts. However, challenges arise when processing speech transcripts that describe complex scenarios, as they are verbose and include repetition, hedging, and vagueness. E.g., during a company’s earnings call, an executive might project a positive revenue outlook to reassure investors, despite uncertainty regarding future earnings. It is crucial for LLMs to incorporate this uncertainty systematically when making decisions. In this paper, we introduce DeFine, a modular framework that constructs probabilistic factor profiles from complex scenarios. It then integrates these profiles with analogical reasoning, leveraging insights from similar past experiences to guide LLMs in making critical decisions in new situations. Our framework separates the tasks of quantifying uncertainty and incorporating it into LLM decision-making. This approach is particularly useful in areas such as consulting and financial deliberation, where making decisions under uncertainty is vital.

## introduction large language models are increasingly relied on for decision-making, thanks to their powerful reasoning capabilities @xcite @xcite . while research has advanced rapidly in areas such as math, coding, and logical reasoning @xcite @xcite @xcite , there is growing interest in exploring how llms reason through complex, real-world environments to make high-stakes decisions, such as financial investments @xcite . the challenges are compounded when processing long contexts @xcite . core issues may include recency bias, hallucinations, numerical inconsistencies, and more @xcite @xcite . while these models are designed to produce reasoning traces, their explanations can remain vague and even unfaithful @xcite . critically, llms still lack precise, quantitative insight into the key factors that lead to their final decisions, as well as mechanisms for incorporating uncertainty into their decision-making. in this paper, we present define, a new framework for constructing probabilistic factor profiles from speech transcripts that describe complex scenarios, and leveraging these profiles to enhance decision-making. for example, during an earnings call, a company executive might project strong revenue growth to boost investor confidence, despite substantial uncertainty surrounding these projections @xcite . as illustrated in figure 1 , define generates a structured factor profile from each transcript, capturing not only what is explicitly stated, but also the implications of what is left unsaid. it then uses the bt model (bradley and @xcite to identify dominant factors and evaluate how these factors collectively impact decisionmaking. our research integrates probabilistic factor profiles with analogical reasoning, a type of reasoning that identifies connections between similar situations to facilitate knowledge transfer from a familiar context to a new situation @xcite . instead of relying on text matching, we use factor profiles to retrieve analogous examples, which are historical cases with similar levels of uncertainty across key dimensions. analogical reasoning further sets our work apart from traditional bayesian inference models used in decision-making @xcite @xcite , which usually lack direct connections to historical cases. our key contributions are as follows. ## the define framework we investigate how llms make critical decisions about post-earnings stock movements using earnings call transcripts @xcite . an earnings call is a teleconference in which company executives discuss financial results with analysts and investors for a given quarter or fiscal year. as shown in figure 5 in the appendix, these transcripts typically consist of two parts: prepared remarks from the company's executives and a subsequent q&a session. during these calls, the executives provide a deep dive into the company's financials, discuss key performance indicators, and share strategic plans for the future. however, discussions of financials, e.g., revenue, expenses, and profit margins, can be overwhelming. a factor profile seeks to distills these discussions into multiple variables, allowing decision-makers to focus on the most impactful factors @xcite . notably, if critical elements such as debt levels are not addressed by company executives, they can be marked as "unknown or uncertain." this approach contrasts with traditional textual summaries of the transcript @xcite @xcite , which may be biased toward the topics emphasized by executives and discussed during the q&a session. ## bayesian decision-making in bayesian decision-making, utility functions play a crucial role in navigating uncertainty @xcite @xcite . a bayesian framework updates beliefs about possible outcomes. decisions are then made by evaluating the expected utility for each possible action, which involves calculating the utility across the updated beliefs. this method ensures that choices are made to maximize expected utility, so decisions are aligned with the decision-maker's preferences and risk tolerance. concretely, to compute p (o ij |x), we construct a probabilistic factor profile from a given earnings call transcript, where o ij represents the j-th outcome of the i-th factor. the likelihood p (y |o ij ), which estimates how the j-th outcome influences stock investment decisions, is calculated using the bradley-terry model. this model provides a framework for quantifying the impact each factor outcome has on the decision-making process. using these probabilities, the bayesian decision-making formula integrates over all factors and their potential outcomes to determine the optimal action. the overall decision is derived by: the parameters calculated by the bradley-terry model for p (y |o ij ) help us determine how each factor influences stock movements. during our testing phase, transcripts are assigned to one of five decision categories based on their computed scores. for example, if the ground truth indicates there are k 'strong buy' recommendations, the top k scoring transcripts are classified correspondingly as 'strong buy'. this approach uses probabilistic factor profiles in conjunction with bradley-terry modeling to identify influential factors, providing a transparent method for understanding decision-driving elements. moving forward, we extend beyond individual factors by examining analogous cases that directly influence decisions. ## analogical reasoning analogical reasoning, which involves drawing parallels between similar situations @xcite @xcite @xcite , is an effective method for decision-making. this approach is particularly useful when analyzing how stocks react to earnings announcements by referencing past, similar events. for example, in the tech sector, stocks often show high volatility after earnings calls that introduce significant technological updates, even if the revenue and eps meet expectations. if a tech company is rumored to discuss a new technology trend in its upcoming earnings announcement, using this method, we can infer that this company's stock might also experience increased volatility. investors might use this analysis to make investment decisions or hedge against potential volatility. accurately identifying analogous examples from earnings call transcripts is crucial. we propose a method that utilizes probabilistic factor profiles, denoted as p (o ij |x), where o ij represents the j-th outcome of the i-th factor. to measure the similarity between profiles, we calculate the kullback-leibler (kl) divergence, which quantifies the information loss when one probability distribution approximates another. the kl divergence is computed as follows: here, p represents the factor profile for the target transcript, and q denotes the profile for a comparative transcript x c from our training set. transcripts with lower kl divergence values are considered more analogous, and therefore more likely to influence investor decisions similarly. data statistics num. of transcripts 11,950 num. of companies 869 avg. #tokens per transcript 10,187 avg. #qa pairs per transcript 10 avg. #trans per company 14 avg. #speakers per transcript 12 year range 2017-2024 during testing, we identify the top-k profiles that show the least divergence from a test instance's profile and present these as analogical examples for the llm to consider when reasoning about stock movements. the llm is asked to select the most analogous example from the top-k and carefully evaluates the current test instance to make its prediction. this approach ensures that the alignment between profiles is contextually appropriate, thereby drawing meaningful comparisons across different transcripts. by focusing on factor profiles rather than full transcripts or their summaries, we emphasize key market-moving information, avoiding unnecessary details. for example, google and broadcom could have analogous profiles even though their discussions in earnings calls might vary widely. using factor profiles as analogous examples also requires significantly fewer tokens within the context window than full transcripts would. ## data collection our dataset contains 11,950 earnings call transcripts from s&p 500 and nasdaq 500 companies, gathered from the motley fool over the period of 2017-2024. the motley fool is a well-regarded financial service website that regularly publishes earnings call transcripts from u.s. companies. we make sure to follow their terms of use carefully during data collection. we do not use audio recordings or analyze acoustic or prosodic features. each transcript is formatted as a json object, including the company's stock ticker, the date of the earnings announcement, participant names and their affiliations, executive prepared remarks, and a series of question-answer pairs from the q&a session. we seek to make stock investment decisions by analyzing earnings call transcripts and focusing on performance over the 30-day period. we establish the ground truth decision on the 30th day following each earnings announcement @xcite : a stock drop exceeding 5% corresponds to a 'strong sell' decision, a decrease between 2% and 5% leads to a 'sell', fluctuation within -2% to +2% is labeled 'hold', an increase between 2% and 5% is labeled a 'buy', and an increase above 5% is a 'strong buy'. in our test set, the distribution of these labels is as follows: 'strong buy' at 34%, 'buy' at 15%, 'hold' at 21%, 'sell' at 9%, and 'strong sell' at 21%. this distribution is generally balanced, reflecting a slightly bullish market trend in 2024. ## experiments in this section, we evaluate the decision-making performance of various systems, analyze the key factors that influence stock movement predictions, and conduct an analysis of analogical reasoning. ## conclusion we propose define, a new framework for decision-making in complex scenarios, such as those encountered in corporate earnings calls. by combining probabilistic factor profiles with analogical reasoning, this framework not only captures the uncertainties embedded in earnings call transcripts but also allows the llm to apply previous insights to new challenges more efficiently. our approach surpasses strong baseline models and enhances the practical utility of llms by identifying analogous examples. the define framework offers a promising avenue for navigating complex data and supporting decision-making processes. to acknowledge that its performance might vary in the complexity of real-world settings due to factors such as data quality and specific contextual nuances. we encourage users to consider these variables when adapting the define approach to ensure its optimal application and to mitigate potential discrepancies between expected and actual results.