Automating Alternative Generation in Decision-Making (2025)

Abstract: In decision making, generating alternative solutions is crucial for solving a problem. However, cognitive biases can impede this process by constraining individual decision makers’ creativity. To address this issue, we introduce a new task for automatically generating alternatives, inspired by the process of human “brainstorming”. We define alternative options based on atomic action components and present a dataset of 106 annotated Reddit r/Advice posts containing unique alternative options extracted from users’ replies. We also introduce new metrics to assess the quality of generated components, including distinctiveness, creativity, upvote-weighted, crowd intersection, and final commit intersection scores. As a baseline, we evaluated the large language models (LLMs) LLaMa3:8b, LLaMa3.1:8b, and Gemma 2:9b on the alternative component generation task. On the one hand, models demonstrated high creativity (ability to generate options beyond what Reddit users suggested) and performed well at proposing distinct alternatives. A subset of generated components was manually evaluated and found overall useful. This indicates that LLMs might be used to extend lists of alternative options, helping decision makers consider a problem from different perspectives. On the other hand, LLMs’ outputs often failed to align with human suggestions, implying that they still tend to miss important components.

## introduction decision-making is the process of choosing any course of action that aims to solve a problem in the best possible way. every aspect of human life 1 https://github.com/arg-tech/decisionmaking_ alternative_components 2 https://huggingface.co/datasets/arg-tech/ decisionmaking_alternative_components involves decision making to some degree, from selecting what to wear based on the weather to deliberating how to resolve a large-scale conflict. yet, often human creativity is limited and constrained by biases when it comes to imagining different alternative actions leading to the best possible outcome. although different theories disagree on the structure of the decision making process @xcite , most theoretical frameworks consider decision making to be the process of selecting a preferred option from a set of alternative options, frequently without specifying from where this set of alternatives originates. it has been highlighted in multiple studies that this process is a very important yet challenging step @xcite . as it relies on memory retrieval of relevant information @xcite , the process of identifying possible actions poses human challenges. for example, information may not be organized in a coherent structure useful for the problem @xcite , interference from previous knowledge can make it more difficult to restructure information to see problems from different perspectives @xcite , and the use of heuristics based on prototypical problems may introduce further human biases such as a tendency to learn towards highly representative options, and struggling to retrieve or creatively generate high-quality solutions @xcite . there are multiple theoretical frameworks developed to help decision makers overcome these challenges @xcite . "brainstorming" is one of them @xcite . this method relies on aggregated judgment from multiple people, trying to utilize their cognitive efforts and mitigate their individual biases by providing different opinions and perspectives on the problem at hand. overcom-ing humans' limited cognitive flexibility, "brainstorming" is also particularly well-suited for computational automation @xcite . developing an automatic algorithm can help overcome human biases by generating wider set of possible alternatives. such algorithms could be used in any process or task requiring decision making, such as operational planning, conflict resolution, and so on. in this paper, we introduce a novel task in which the algorithm needs to generate lists of possible atomic options for solving human decision making problems. as a baseline, we evaluate different large language models (llms) in few-and zeroshot settings. to evaluate the performance of the models, a new dataset based on reddit comments was manually labeled, approximating the "brainstorm" technique by incorporating advice replies from multiple users per question. the following contributions are made: for few-shot approaches, we used 5 and 10 examples, averaging results across three trials with different sets of examples. we conducted experiments with llama3:8b @xcite , llama3.1:8b @xcite , and gemma 2:9b @xcite . ## related work no datasets or evaluation metrics were available for the task at the time of writing. decision-making has been discussed in academic literature from various disciplines, yet there is a notable lack of materials when it comes the generation of alternatives from a computational perspective. early studies @xcite @xcite were focusing on the benefits and influence of alternative generation on the overall decision process. a comprehensive survey and overview of different techniques @xcite discussed in detail various different methods that a person could utilize to achieve the best possible set of alternative actions in any given scenario. mga (modeling for generating alternatives), a theoretical framework for alternative generation, has been presented and discussed in various studies @xcite @xcite @xcite . the proposed algorithm formalizes a decision making process. the method includes multi-objective optimization algorithms to explore the neighborhood of a possible solution in order to find the most optimal solution. it requires a distance function initialization that measures the differences between the solutions, as well as a strict definition of the importance of the model's objectives and constraints. recently, colorni and tsoukiàs (2020) formulated a general framework for formalizing alternatives, stressing how under-researched this area has been. the first attempts at systems practically aiding the decision making process were introduced in the late 90s @xcite @xcite figure 1 : pipeline overview of the framework. @xcite @xcite . recent studies address further theoretical subtleties, carefully structuring various techniques and methods that can aid decision makers @xcite . it is furthermore worth noting that research incorporating the use of artificial intelligence to assist idea generation generally @xcite is also thematically adjacent to our research, though it lacks the distinction between different competing alternatives which is so crucial for describing decision making. lastly, the proposed task also shares some similarity with question answering tasks (qa), for which there are different datasets available @xcite @xcite @xcite @xcite . however, qa tasks have a correct answer, whereas in the case of alternatives this is not necessarily clearly evident. ## definitions and tasks in this work, an alternative option (ao) refers to an action or a set of actions that could be taken in the context of a given problem. an action refers to the specific steps taken to implement the chosen alternative, aiming to resolve the problem and achieve the desired outcome. in the context of this work, a problem refers to some type of situation that requires a response and involves a choice between different aos to achieve or solve it, based on a definition by @xcite . in decision making, solutions are considered sideby-side and can therefore be termed "alternatives". each ao consists of smaller units of action that we call components, which are atomic, i.e., cannot be broken down further. more formally, components are the smallest actions that can be taken in solving the problem. they may include conditional elements, a certain order of components, concretize specific actors who should participate or perform the component, etc. components are characterized by the order or actions, semantic content, conditional parts, and the entities and participants included in the components. components c 1 and c 2 are considered identical if they fit the following component matching rules (cmrs), i.e., if c 1 and c 2 preserve the order of actions and overall semantics, maintain the same conditional parts, and refer to the same entities, participants, people, etc. consider the following example: problem: an office worker a accidentally took a cookie from a bowl in the office kitchen, assuming that it was a shared bowl. however, it turns out that the contents of the bowl were the private lunch of colleague b, who still has not noticed that one of the cookies is missing. what do you recommend worker a should do in this situation ? in this scenario, the components could be as follows: in the example, there are multiple suggested component actions that a can take. for example, components 2 and 4 are different, as there is an additional conditional part to component 4 that is not present in 2. components 2 and 8 include the same action, but different participants (component 2: b; component 8: b and hr). component 6 contains an order of actions. some of the components can be done together (e.g., components 1 and 5). if the components are mutually exclusive (e.g. components 1 and 2), they are referred to as competing components. each ao is a set of non-competing components, i.e., all of its actions can be performed without a conflict. competing alternatives are sets of aos whose components are competing with each other. this is similar to the concept of competing hypotheses @xcite . for instance, because of the competing components 1 and 2, alternative options ao 1 = {1, 3, 7} and ao 2 = {2, 3, 7} are competing alternatives, despite having overlapping components. a subset of the ao is an ao itself. the component generation (cg) task involves generating as many relevant components as possible to solve the problem, based on its title and description. this is framed as a text generation task. in this paper, we focus on the cg task and leave other potential tasks for future work. ## evaluation in the context of the component generation task, we propose the following metrics: distinctiveness, creativity, upvote-weighted intersection, crowd intersection, and final commit intersection scores. all the proposed metrics require an algorithm that determines whether the two components are identical. recall that the components c 1 and c 2 are considered identical if they fit the component matching rules (cmrs): c 1 and c 2 preserve the order of actions and overall semantics, have the same conditional parts, and refer to the same entities, participants, people, etc. the calculation is based on the components matching algorithm, which is detailed in the following subsection. ## experimental setup the experiments were conducted with the following llms: llama3:8b, llama3.1:8b @xcite , and gemma 2:9b @xcite . the models were instructed with an initial system prompt explaining the task and outlining that the output format should clearly separate components. the model was prompted to generate a special stop token once it had finished generating the components. the experiments were conducted with @xmath0, 5, and 10 examples from the dataset presented to the model. each example included title, post content (without the author's update), and expected list of components with the stop token at the end. then, the test title and post content were presented to the model. the model generated the components, and if the stop token appeared in the generated text, this was considered the final output. otherwise, the model was presented with an additional request to generate more of components and complete the previous conversation history. this process was run until the stop token appeared in the text, or when the maximum allowed number of follow-ups (20) was reached. for each of the few-shot experiments with @xmath1, 10} examples the model was run independently 3 times, selecting random n examples from the dataset. the final metrics were averaged over the experiments per n . as a preprocessing step, all the generated components were run through the matching algorithm to remove duplicate components. to evaluate a joint performance of the models, the generated results per n were aggregated. we set random seed equals 2, and set other generation parameters to defaults, including a temperature of 0.7. the constructed prompts and code are available in the project github repository. the overview of the pipeline is presented in figure 1 . ## results the results are presented in a random sample of the 327 generated components was manually evaluated on their usefulness and relevance to the problem. the annotation results are presented in figure 3 and in appendix c. the models mostly generated useful components for tackling the input problem, with annotators' agreement on useful/not useful labels reaching a cohen kappa score of 0.53. more details are provided in the appendix b. the correlation matrices of the scores are presented in appendix d. our results do not indicate a strong correlation between most metrics, except for the uwi and ci scores. this is expected, as ci is based on the intersection of llm-generated and target components, while uwi reflects the importance of posted suggestions to reddit users. based on the obtained results, all the models were able to output distinct sets of components when presented with examples. the distinctiveness scores in each run was 1.0 for @xmath2. however, when models were not presented with examples from the dataset, llama3.1 and gemma 2 obtained ds of 0.943 and 0.981 respectively. these scores are still high, but not as good as when presented with few-shot examples. in our experiments, the ds score was not highly discriminatory, as all evaluated models demonstrated similar levels of competence. the most creative model was llama3:8b, as it was able to outperform other considered models with zero-shot (with @xmath3.557 and std of 1.048) and with @xmath4.574 and std of 0.275). when provided with 5 examples, llama3.1 had the highest creativity score of 1.364. not only was this result the best on average in the group of @xmath5, but it also was the most consistent one with std of 0.041. for the final commit intersection score (fci), llama3.1 outperformed other models in all experiments with the scores of 0.061 (zero-shot), 0.065 (5-shot), and 0.06 @xcite . this score indicates an intersection with the "best" components -the ones that were selected by the original post author. however, in a lot of cases, this could also primarily represent a personal preference. often, more context is required to determine what might be considered the best option for any particular problem. in our experiments, we expected llama3.1 to outperform llama3 across the different experiments. however, llama3 demonstrated the better performance in the @xmath7. similar behavior has been observed before. based on the released results for these models, there are instances when llama3:8b showed better results than llama3.1:8b (for example, on gpqa @xcite dataset, llama3.1 obtained a score of 32.8 and llama3 obtained 34.2 @xcite ). ## conclusion our experiments showed that llms are capable of outputting distinct components for decision making. however, they still appear to be a far way from matching human judgement, even when presented with different examples of the expected alternative components. in our experiments, llms performed better when provided with more examples, as might be expected. in almost all the settings and experiments, the best performing models were llama3 and llama3.1. these models demonstrated the highest creativity, intersection with human judgement, and with which actions authors finally did or committed to doing. nevertheless, intersection scores were overall still low, indicating room for improvement. from a practical perspective, the creativity aspect is important in decision making as it provides a bigger picture for decision making. generating alternatives automatically might allow decision makers to go beyond cognitive limitations. we showed that the considered llms are able to produce high creativity score, outputting possible components that were not considered in reddit comments. therefore, llms could be helpful in creating and extending lists of options which might serve as a starting points for decision makers to consider a problem from different perspectives.