Evaluating LLM s with Multiple Problems at once (2025)

Abstract: This paper shows the benefits and fruitfulness of evaluating LLMs with multiple problems at once, a paradigm we call multi-problem evaluation (MPE). Unlike conventional single-problem evaluation, where a prompt presents a single problem and expects one specific answer, MPE places multiple problems together in a single prompt and assesses how well an LLM answers all these problems in a single output. Leveraging 6 classification and 12 reasoning benchmarks that already exist, we introduce a new benchmark called ZeMPE (Zero-shot Multi-Problem Evaluation), comprising 53,100 zero-shot multi-problem prompts. We experiment with a total of 13 LLMs from 5 model families on ZeMPE to present a comprehensive and systematic MPE. Our results show that LLMs are capable of handling multiple problems from a single data source as well as handling them separately, but there are conditions this multiple problem handling capability falls short. In addition, we perform in-depth further analyses and explore model-level factors that may enable multiple problem handling capabilities in LLMs. We release our corpus and code to facilitate future research.

## introduction thanks to the advances in both gpu hardware and algorithms @xcite @xcite @xcite , large language models (llms) have been developed with increasingly larger context windows (e.g., 8k, 128k, 2m). to leverage the extended context windows, recent studies @xcite @xcite have proposed various prompting strategies that place multiple problems in a single prompt, which we collectively call multi-problem prompting (mpp). the basic idea of mpp is to place multiple problems after a shared context c (e.g., task instruction and/or exemplars) to avoid repeating c for each problem as in standard single-problem prompting (spp), which improves input token utilization and reduces llm inference costs per problem. in this study, we evaluate a wide range of llms with multiple problems at once through mpp, a paradigm we call multi-problem evaluation (mpe) @xcite . 2 while the main goal of mpp is to improve the cost-efficiency of llm inference, we view mpe primarily as a valuable evaluation paradigm for probing llm capabilities, rather than merely a cost-saving engineering strategy. unlike conventional single-problem evaluation that assesses an llm's ability to answer a single problem through spp, mpe assesses an llm's ability to concurrently handle multiple problems at once or in a single output. understanding the multiple problem handling capabilities of llms is an important research question because it gives us a foundational insight into how llms operate over multi-problem inputs that can be sufficiently long and use information from individual problems contained within each multi-problem input. to enable a comprehensive and systematic mpe, we introduce zempe (zero-shot multi-problem evaluation), a new benchmark comprising 53,1000 zero-shot multi-problem prompts. zempe is synthetically generated by leveraging 6 classification and 12 reasoning benchmarks that already exist and are widely used. moreover, zempe includes various types of evaluation tasks to allow for deep and nuanced analyses, taking into account how multiple problems are presented in the prompt and whether these problems are sampled from the same data source or not. we do not mix classification and reasoning problems together due to the different natures of these two types of problems and to not make our experiments confounding. our main contributions are as follows: ## related work we note that current llm evaluation has predominantly focused on llm's performance on singleproblem prompts. each of such prompts presents a single problem and expects one specific answer to that problem, which may implicitly require multihop reasoning or multi-step task solving. recently, @xcite propose few-shot mpp named batch prompting that prompts llms with problems batched together from single sources following a few batches of equally sized exemplars. they find that few-shot mpp greatly increases llm inference efficiency while retaining downstream performance with a small batch size (e.g., <6). to ensure that batch prompting works with large batch sizes, @xcite introduce a sampling optimization method that takes a majority vote over repeated permutations of batch samples. instead of solving multiple separate problems, @xcite prompt llms with exemplars to solve multiple related tasks based on a shared problem setup by placing an explicit instruction for each task. they find that instructing llms to solve all the tasks at once outperforms solving the individual tasks one by one or in a batch. in addition to these few-shot studies, laskar et al. @xcite shows that instruction-tuned gpts can handle 5 short questions sampled from two open-domain qa benchmarks at once zero-shot, but the base gpt models can barely perform the task. to the best of our knowledge, @xcite present the first systematic evaluation of llms' zero-shot ability to tackle multiple homogeneous classification problems drawn from six standard benchmarks. they show that, while llms can usually solve several such classifications in a single prompt with accuracy comparable to handling them one-by-one, their performance deteriorates sharply when the prompt instead asks them to return the indices of texts belonging to each class-a shortfall that remains consistent across models, prompting conditions, and experimental settings. building on top of @xcite , this study examines a total of 13 llms on 18 existing benchmarks, including 12 reasoning benchmarks that are not part of @xcite 's evaluation. besides from reaffirming @xcite 's finding that llms are capable of handling multiple problems from a single data source as well as handling them separately, we perform in-depth further analyses to both validate such capabilities and expose their limitations. moreover, we explore model-level factors that may enable llm's strong multiple problem handling capabilities. ## multi-problem evaluation this section compares single-problem evaluation (spe) and multi-problem evaluation (mpe) and introduces zempe, a new mpe benchmark. ## experiments this section first describes the experimental setups and then reports and discusses the results. ## further analyses this section provide further analyses to better understand llms' multiple problem handling capabilities, their limitation, and what enables such capabilities. it starts with two error analyses for batch-clf, aiming to see whether models make similar prediction errors and positional errors under batch-clf compared to singleclf. we then investigate the reason why selectall appears to be harder than se-lectone, which seems counter-intuitive. lastly, we explore model-level factors that may enable llms to receive and handle multiple problems at once. ## conclusion in this study, we present a comprehensive and systematic mpe of llms. we evaluate various llms from 4 model families on single-source multi-problem prompts constructed from 6 classifi-cation and 12 reasoning benchmarks. in line with previous few-shot results, we confirm that llms are competent multi-problem solvers for classification and reasoning under zero-shot settings. moreover, we find multiple pieces of evidence that validate the strong innate multiple problem handling capabilities of llms, such as the similar classification errors llms make under spp and mpp, the lack of obvious positional biases, and the transferrability of zero-shot-cot under mpp. leveraging the strong multiple problem handling capabilities, we show that zero-shot mpp can be cost-efficient. two conditions are identified under which llms show consistent performance declines with mpp: (1) reformulating batch classification as index selection tasks; and (2) mixing reasoning problems from different sources in a multi-problem prompt. noticeably, these performance declines happen even when the number of problems included is rather small (e.g., ≤ 5), which may not be humanlike and indicates a lack of true understanding. in addition, we explore several model-level factors that may enable mpp and find instruction tuning to be an important factor that enhances mpp. overall, our experiment demonstrate surprisingly consistent observations across different llms and across multi-problem prompts constructed from various benchmarks. this consistency indicates the reliability and fruitfulness of mpe as an evaluation paradigm. as a result of our study, we create a new benchmark comprising 53,100 zero-shot multi-problem prompts. we call it zempe, which stands for zero-shot multi-problem evaluation. we release zempe to aid future mpe research.