“Let’s Argue Both Sides”: Argument Generation Can Force Small Models to Utilize Previously Inaccessible Reasoning Capabilities (2024)

Abstract: Large Language Models (LLMs), despite achieving state-of-the-art results in a number of evaluation tasks, struggle to maintain their performance when logical reasoning is strictly required to correctly infer a prediction. In this work, we propose Argument Generation as a method of forcing models to utilize their reasoning capabilities when other approaches such as chain-of-thought reasoning prove insufficient. Our method involves the generation of arguments for each possible inference result, and asking the end model to rank the generated arguments. We show that Argument Generation can serve as an appropriate substitute for zero-shot prompting techniques without the requirement to add layers of complexity. Furthermore, we argue that knowledge-probing techniques such as chain-of-thought reasoning and Argument Generation are only useful when further reasoning is required to infer a prediction, making them auxiliary to more common zero-shot approaches. Finally, we demonstrate that our approach forces larger gains in smaller language models, showcasing a complex relationship between model size and prompting methods in foundation models.

## introduction large language models, including state-of-the-art models such as llama family of llms @xcite , mistral 7b @xcite , and phi-3 @xcite have shown to significantly outperform previous generation of models @xcite such as bert @xcite in several mainly classification tasks @xcite . however, despite their seemingly human-like auto-regressive behavior, large language models do not perform well when deep reasoning or analysis is required to effectively infer a prediction @xcite . in order to bolster the reasoning capabilities of large language models, the research community has done extensive recent work in the form of chain-of-thought reasoning @xcite , self-reflection @xcite , multi-agent debate @xcite , and socratic prompting @xcite , demonstrating that prompting the model to generate the reasoning behind its answer, or generating a step-by-step guide to reach its response can help predict better results. taking inspiration from chain-of-thought reasoning, and motivated by the need to develop better prompt techniques with the goal of increasing model performance in reasoning tasks, we introduce argument generation, a single-pass prompting technique that aims to utilize the reasoning and argumentation capabilities of large language models to generate better responses where deeper consideration of logic or reasoning is required to infer the correct result. argument generation involves a two-step process. we first prompt the model to generate possible reasoning for the truthfulness of each possible option, and then we ask the model to rank the generated arguments and map its ranking to a final output in accordance with the task expectations. we evaluate our method on a number of openly available state-of-the-art large language models using nine tasks of different natures. we find that argument generation at its weakest, does not perform significantly worse than chain-of-thought reasoning, and is able to outperform both zero-shot reasoning and chain-of-thought reasoning when a deeper understanding of the task options is required. furthermore, we note that in comparison to chain-of-thought reasoning, argument generation can be used as a stronger knowledge probing technique that is useful in instances where such probing is essential, or some level of prior knowledge regarding the task is present (such as possible response candidate). however, our method does not necessarily increase the model performance for inputs that observe acceptable results under more common methods. we make the following contributions: (1) we introduce argument generation, a novel prompting technique that aims to access the underlying reasoning capabilities of llms. (2) we show through a series of experiments that our method is able to effectively reason under conditions that fail chain-ofthought reasoning. (3) we show that our prompting method is more effective when used with smaller language models, eliciting further investigation into the relationship between prompting approaches and model capabilities. ## background and motivation argumentation is the cognitive capability of generating and evaluating "reasons" for deriving a conclusion @xcite . it is a central aspect of human intelligence and is omnipresent in natural human communication. it extends the conception of reasoning in llm-research @xcite ) by including the notion that conclusions drawn must be new. indeed, it has been suggested that human reasoning evolved for the purposes of enabling humans to persuade each other @xcite through arguments. we hypothesize that many day-to-day arguments are evaluated by humans in an intuitive (fast, system 1) manner, without deep thought or "epistemic vigilance" @xcite , unless they are from trusted sources and appear to contradict our own beliefs. thus, because llms were pretrained with human communicative interactions, we hy-pothesize that llms are capable of fast argumentative thinking. by triggering argumentative thought, we hypothesize that llms can effectively generate reasons and assess conclusions, as well as improve core reasoning capabilities across a variety of domains, including commonsense, logical, and social. ## related work general argumentation ability of llms have begun to be explored by researchers, with a focus on a number of computational argumentation subtasks such as argument mining, claim detection, evidence detection and type classification, argument generation, and summarization @xcite @xcite @xcite . research suggests that llms "exhibit commendable performance" @xcite in zero-shot and few-shot settings thereby supplying a foundation supporting our approach. delving deeper, we can explore two core aspects of argumentation. first, the ability to argue for/against all sides (thinking like a lawyer). second, the ability to generate implicit assumptions (necessary or sufficient warrants) needed to support the argument. arguing all sides is related to "backward reasoning" suggested in @xcite , where they discuss that it is "better to collect both supportive and opposing knowledge to compare the confidence of different conclusions for defeasible reasoning." additionally @xcite discuss the idea of allowing several different reasoning paths and choosing the "most consistent one". another approach is contrastive chain-of-thought @xcite where they consider both valid and invalid reasoning demonstrations alongside original prompt -a dual perspective approach. additionally, work in multiagent debate, for example @xcite uses a notion of a debate with multiple agents discussing and talking about the problem. however, none of these approaches attempt at rationalizing all sides of an argument. that is none of these offer up the best possible argument for/against each choice, and then evaluate the best argument (for example, anticipatory reflection of plans in @xcite ). extracting implicit information relates to work in "knowledge-enhanced" @xcite strate-gies in which an implicit model generates knowledge and rationales. also @xcite discusses leap-of-thought reasoning which uses implicit facts to answer questions. a related notion is that of decomposing implicit multi-hop questions down in connection with the general backward reasoning tactic of question-decomposition (see summary in @xcite ). work by @xcite suggests extracting implicit assumptions from premise-conclusion pairs, however, that work does not explore how such endeavor influences an llm's reasoning capability. although there is a growing body of work in question decomposition, it is unclear to what extent they take implicit assumptions into account. general llm reasoning capabilities have been improving over the past several years with numerous datasets targeting different types of reasoning -logical, mathematical, commonsense, argumentation, and social reasoning @xcite @xcite @xcite . the methods have involved various techniques to evoke reasoning processes such as having the llm explicate its chain of thought @xcite , reflect on its own reasoning process @xcite , decompose complex reasoning processes into simpler problems that can be solved more easily @xcite , explore many different reasoning paths and decide on one that wins a majority vote @xcite , and others. these various methods have shown improvements in various reasoning tasks, but none have shown cross-domain effectiveness. moreover, their reasoning capabilities are limited when exposed to scenarios in which the model must resolve a disagreement @xcite , distinguish a correct phrase from an incorrect one @xcite , or assign a nondeterministic gender to a subject @xcite . overall, large language models have shown promising results in a variety of reasoning tasks while serious challenges and shortcomings still remain @xcite . what is missing is a cross-domain strategy to improve an llm's zero-shot reasoning capabilities, which we hypothesize to be enhanced by its latent capability for argumentative thinking. ## methodology we now provide details regarding our approach, including the proposed zero-shot approach and the reasoning behind our choice of argument generation as a prompting technique. argument generation involves two overall steps. given an initial input x with possible answers k 1 , k 2 , ...k n , we first prompt the model to generate arguments supporting and attacking each answer k i , creating arguments x ′ 1 , x ′ 2 , ...x ′ n for each possible answer. we then ask the model to choose the answer with the strongest argument as the final output. more concretely, the large language model is utilized as a proxy for an argument ranking function that chooses the most feasible options among arguments x ′ 1 , x ′ 2 , ...x ′ n . the rationale behind our approach is two-fold. first, it has been shown that large language models, when provided with a reasoning context towards the correct output, observe significantly improved performance @xcite , making the reasoning behind each choice an important contributor to model performance. second, large language models can act as effective rankers when provided with a list-wise input of possible options @xcite , indicating the feasibility of their possible utilization for the effective ranking of arguments. as a result, the proposed technique relies on the assumption that the correct answer k i to the query x should logically have the strongest argument supporting it, forcing the ranker model to choose the argument that is directly mapped to the correct answer. essentially, argument generation is similar to chain-of-thought reasoning because both focus on the generation of a token chain with the goal of increasing the probability of generating a viable final answer. however, chain-of-thought reasoning operates under the assumption that the generation of supporting steps is sufficient for the final true output. on the other hand, argument generation aims to take into consideration the possibility of the presence of a counterargument that is statistically more significant than the answer that is generated by pure chain-of-thought. as such, we hypothesize that chain-of-thought can sufficiently generate the most logically intuitive response to the user input, while argument generation might be better suited for cases where the correct answer is initially unintuitive but may increase in statistical significance as a valid counterargument is presented against the other answer candidates. ## evaluation to empirically evaluate the effectiveness of our proposed method, we have tested the performance of argument generation in nine datasets and across nine models. for the remainder of this section, we focus on describing our evaluation setting. ## evaluation results we now showcase our results as tested against the datasets mentioned in section 5. we additionally show that argument generation, when outperforming zero-shot chain-of-thought reasoning, demonstrates significantly higher performance gain, and suffers smaller losses in cases where it does not result in increased performance. we finally provide a model size analysis to better understand the relationship between prompting methods and the number of parameters present in a given large language model. ## discussion and future work prompting has been proposed as a method of improving model performance in either task-specific settings or broader, task-agnostic environments @xcite . despite the visible gains of employing prompting to yield better model results, the literature showcasing how, and when prompting works is limited @xcite . we observe that the proposed method is able to significantly boost the model performance in smaller models while gaining marginal improvements as the model size increases, which is contrary to the previous work showing that larger models have higher gains through prompting @xcite . this leads us to believe that the relationship between prompting and the nature of the model is complex, and might be affected both by the model size, and its relative task-specific knowledge and capabilities. further work is required to demonstrate the effects of prompting when models hold knowledge of varying degrees with respect to a task description. investigation of the learning resources used in model training can provide invaluable insight into the relationship between prompting and model reasoning. ## conclusion in this work, we have proposed argument generation as a novel, zero-shot prompting technique. through empirical evaluation using a number of datasets, we observe that our method is able to outperform both zero-shot prompting and zero-shot chain-of-thought reasoning in the majority of the conducted tests, making it a likely candidate when improving the model performance in a zero-shot setting is required. furthermore, we show that our approach yields larger gains in smaller models, both offering an effective method that can be used in small models and providing a possible future direction to better understand the relationship between model capabilities and prompting.