A Thorough Examination of Decoding Methods in the Era of LLM s (2024)

Abstract: Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. Prior research on decoding methods, primarily focusing on task-specific models, may not extend to the current era of general-purpose large language models (LLMs). Moreover, the recent influx of decoding strategies has further complicated this landscape. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of LLMs, evaluating their performance, robustness to hyperparameter changes, and decoding speeds across a wide range of tasks, models, and deployment environments. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization. Intriguingly, sensitivity analysis exposes that certain methods achieve superior performance at the cost of extensive hyperparameter tuning, highlighting the trade-off between attaining optimal results and the practicality of implementation in varying contexts.

## introduction the advent of large language models (llms) @xcite @xcite has ushered in a new era of natural language processing (nlp). these models are trained to predict the next token on massive corpora, empowering them with extraordinary multitasking capabilities. this enables them to perform almost all nlp tasks through the lens of text generation, distinguishing them from traditional task-specific models. decoding methods, which are the bridge between next-token predictors and text generators, play an integral role in transforming llms into practical task solvers. recent studies have shown that the choice of decoding methods can substantially impact the performance of llms (o @xcite . however, these studies often focus on a narrow aspect (e.g., factuality @xcite ) and a limited set of similar tasks (e.g., math problem solving @xcite ). notably, @xcite ; @xcite provide a comparative analysis of various decoding methods using task-specific language models. they find that deterministic decoding methods (e.g., beam search) perform better than stochastic decoding methods (e.g., top-p sampling @xcite )) in closed-ended generation tasks such as machine translation, while the inverse is true for open-ended generation tasks such as story generation. however, their findings are confined to traditional task-specific models prior to the advent of llms. it is uncertain whether their conclusions still hold for general-purpose llms. in addition, a plethora of new decoding methods @xcite @xcite @xcite have been proposed afterward, each claiming to outperform the previous state-of-the-art in particular tasks. nevertheless, today's most performant llms such as @xcite only provide apis for temperature and topp sampling, seemingly overlooking the potential benefits of other advanced decoding methods. the above observations raise a natural question: what is the best practice for choosing decoding methods in the era of llms? a thorough analysis of decoding methods is essential for researchers and practitioners to understand the strengths and weaknesses of different decoding methods and to choose the one that best fits their needs. our work fills this gap by providing a comprehensive study of the performance, robustness, and speed of various decoding methods across a wide range of different tasks, models, and deployment environments. we also provide in-depth analyses to uncover the un-derlying reasons for the observed results. our key findings include the following: • overall the optimal decoding method depends on the task, the model, and the priority (e.g., performance vs. robustness vs. speed) in hand. there is no short guideline. the complexity of our results calls for more comprehensive evaluations in future research on decoding methods and careful consideration for practitioners. ## decoding methods modern llms typically generate text in a left-toright, token-by-token fashion. for each prefix, the model computes a probability distribution of the next token over a fixed vocabulary. a decoding method defines how the generated token sequence is derived from these probability estimations. we consider decoding methods ranging from deterministic to stochastic. each method is briefly reviewed below, with detailed descriptions in appendix a. the hyperparameter search range of each method is guided by recommendations from relevant literature and common practices. ## experimental results we perform a thorough evaluation of various decoding methods, assessing them from three critical dimensions. initially, our analysis centers on the efficacy of these methods across a diverse range of tasks and models. then, we delve into hyperparameter sensitivity and decoding efficiency. ## conclusion this study offered a comprehensive analysis of diverse traditional and contemporary decoding methods in the context of llms. our experiments shed light on the efficacy, robustness, efficiency, and universality of these decoding methods across a range of tasks, models, and settings. one primary finding is that the choice of decoding methods remains crucial and different decoding methods manifests different advantages in different scenarios. however we still provide some practical guidelines in appendix i. we hope this investigation provides valuable insights and guidance for practitioners and researchers in selecting and advancing decoding methods for llms.