The Effect of Sampling Temperature on Problem Solving in Large Language Models (2024)

Abstract: In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used nine popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature

## introduction 1.background in recent years, large language models (llms) have revolutionized the field of artificial intelligence. the availability of open-source llms and pay-per-use apis has allowed engineers to incorporate llms in their ai systems. however, prompt engineering and hyperparameter tuning are required to work effectively with llms. prompt-engineering techniques help llms solve complex problems, avoid hallucinations, and provide more accurate responses. for example, we can use techniques like chain-of-thought, tree-ofthought, self-criticism, and self-consistency to improve llm performance @xcite . in addition, several inference hyperparameters can be adjusted to modify the llm's output at runtime. for example, hyperparameters like sampling temperature, top-k sampling, repetition penalty, and maximum token length all affect the llm's output and performance (openai, 2023d; llama-2-team, 2023; @xcite . despite significant interest in llms and progress in llm best practices, many open questions remain about optimal prompt-engineering techniques and inference hyperparameters for llms. to complicate matters, various local optima may exist for llms, prompt types, and problem domains @xcite . the prompt-engineering community has an abundance of opinions and anecdotal evidence regarding optimal prompt-engineering techniques and inference hyperparameter settings. however, we currently lack systematic studies and empirical evidence to support many of these claims. as a result, this paper aims to address the open question of the optimal llm sampling temperature for problem-solving tasks. in addition, we aim to provide a systematic study with empirical results to add to the growing body of knowledge used to create llm and prompt-engineering best practices.