How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code (2025)

Abstract: Language models (LMs) have exhibited impressive abilities in generating code from natural language requirements. In this work, we highlight the diversity of code generated by LMs as a critical criterion for evaluating their code generation capabilities. There is a lack of studies focused on assessing the diversity of generated code, which overlooks its importance in code LMs. Therefore, we propose a systematic approach to evaluate code diversity, introducing various metrics with inter-code similarity. Specifically, we introduce code clustering methods that leverages LMs’ capabilities in code understanding and reasoning, resulting in a set of metrics that represent the number of algorithms in model-generated solutions. We extensively investigate the property of model-generated solutions by contrasting them with human-written ones and quantifying the impact of various factors on code diversity: model size, temperature, instruction tuning, and problem complexity. Our analysis demonstrates that model-generated solutions exhibit low algorithmic diversity, which was neglected by the research community. Moreover, we explore methods to increase code diversity by combining solutions from different models and increasing sampling temperatures. Our findings highlight that code diversity can be enhanced with the help of heterogeneous models and setting temperature beyond 1.0 that has not been fully explored due to the functional correctness degradation. To facilitate our research direction, we publicly share our code and datasets through open-source repositories.

## introduction there's more than one way to do it. ## algorithmic diversity evaluation in this section, we propose an evaluation process for algorithmic diversity. ## experiments we analyze the algorithmic diversity encoded in model-generated solutions and discover their potential in code generation task. ## conclusion throughout this work, we comprehensively analyze the code diversity of model-generated solutions. we develop clustering-based metrics, namely da@k, ea, and nauadc, leveraging llms' capabilities to effectively contrast code semantics across different codes. these metrics yield meaningful indicators, i.e., the mean algorithmic count within solution sets, while accounting for the distribution of generated solutions. our extensive analysis unveils previously unexplored findings about code diversity. first, we identify the algorithmic redundancy inside model-generated solutions. especially, our algorithmic diversity curve reveals that the model struggles to devise novel algorithms, despite generating numerous correct solutions. finally, we investigate multiple methods to improve algorithmic diversity through solution merging and adjusting sampling temperature, showing that combining solutions from heterogeneous models and generating with extremely high sampling temperature effectively amplifies algorithmic variety. we hope that our findings facilitate further research aimed at enhancing code diversity to generate more diverse and correct codes.