Exploring Limitations of LLM Capabilities with Multi-Problem Evaluation (2025)

Abstract: We propose using prompts made up of multiple problems to evaluate LLM capabilities, an approach we call multi-problem evaluation. We examine 7 LLMs on 4 related task types constructed from 6 existing classification benchmarks. We find that while LLMs can generally perform multiple homogeneous classifications at once (Batch Classification) as well as when they do so separately, they perform significantly worse on two selection tasks that are conceptually equivalent to Batch Classification and involve selecting indices of text falling into each class label, either independently or altogether. We show that such a significant performance drop is due to LLMs’ inability to adequately combine index selection with text classification. Such a drop is surprisingly observed across all LLMs attested, under zero-shot, few-shot, and CoT settings, and even with a novel synthetic dataset, potentially reflecting an inherent capability limitation with modern LLMs.

## introduction in recent years, large language models (llms) have demonstrated remarkable natural language understanding and reasoning capabilities measured by a wide range of benchmarks @xcite @xcite . however, given their internet-scale training data, there is growing concern over whether llms' often superhuman benchmark performance is achieved due to data contamination @xcite . several studies @xcite have demonstrated the limitations of llms' reasoning capabilities by showing that their performance significantly drops when given the same reasoning tasks but with different assumptions or conditions. these studies are often done through synthetic data generation. in this study, we explore the limitations of llm capabilities through multi-problem evaluation, a simple evaluation method that leverages existing benchmarks to construct prompts made up of multiple problems. as illustrated in fig 2 , unlike conventional single-problem evaluation that prompts an llm to solve a single problem at a time, multiproblem evaluation prompts an llm to solve multiple problems at once in a single input prompt. in this study, we leverage 6 existing classification benchmarks to construct prompts made up of multiple homogeneous problems to form 4 contrastive task types and examine various llms on these task types to explore their limitations. because of the combinatory nature of constructing prompts from multiple problems, it is less likely for llms to encounter exact long multi-problem prompts during pre-training, which makes our evaluation less susceptible to data contamination. we find that while llms can typically handle multiple classifications simultaneously (batch classification) as well as when performed separately, they exhibit a significant drop in performance on two selection tasks that are conceptually equivalent to batch classification and involve selecting indices of text falling into each class label, either independently or altogether. we show that this drop results from llms' inability to adequately combine index selection with text classification, which persists across all tested llms, under zero-shot, few-shot, or chain-of-thought (cot, @xcite settings, and with a novel synthetic dataset. we use the test splits wherever possible, except for cola, for which we use the dev split, since the test split is not publicly available. for agnews and snli, we randomly sample 1,000 examples from the test splits. ## experimental setup this section describes the data, tasks, llms, and performance metric used for our experiments. ## results fig 3 shows the average accuracy on the 4 task types across task sizes for each benchmark with the related full results for each llm provided in llms perform significantly worse on the selection tasks. despite the impressive batchclf performance, llms nearly always perform much worse in selectone and selectall than batchclf, even when the task size is just 3 or 5. the overall discrepancy in accuracy between batchclf and the two tasks is large and statistically significant (32.0% for selectone and 12.1% for selectall, see ## follow-up experiments we perform a series of follow-up experiments to further understand and validate our main findings. more details about these experiments are presented in appendix b. ## conclusion we propose multi-problem evaluation and present a comprehensive multi-problem evaluation of llms, leveraging 6 existing classification benchmarks and 4 related task types constructed from those benchmarks. our results provide new insights into the multiple problem handling capabilities of llms: llms are competent multi-problem solvers for multiple homogeneous classification problems (batch classification), but they perform significantly worse on two selection tasks that are conceptually equivalent to batch classification and involve selecting indices of text falling into each class label, either independently or altogether. this is due to their inability to adequately combine index selection with text classification. the surprisingly consistent performance drop on the two selection tasks observed across 7 llms and a wide range of evaluation settings potentially indicates an inherent limitation with modern llm capabilities. this also showcases the potential of multi-problem evaluation as a useful and effective in discovering limitations of llm capabilities. there are several directions worth future explorations. for example, to better understand how well how llms can handle multiple problems in general, it is important to test llms with other types of problems (e.g., reasoning problems) and with multiple heterogeneous problems (e.g., mixing different benchmarks/tasks). it is also important to understand what causes llms to perform worse or better when prompted with multiple problems, such as benchmark, task size, and model's context length. in particular, model-level ablation studies are needed if we want to know how llms obtain the ability to handle multiple problems at once and how to improve their understanding capabilities.