Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models (2024)

Abstract: This paper explores the impact of extending input lengths on the capabilities of Large Language Models (LLMs). Despite LLMs advancements in recent times, their performance consistency across different input lengths is not well understood. We investigate this aspect by introducing a novel QA reasoning framework, specifically designed to assess the impact of input length. We isolate the effect of input length using multiple versions of the same sample, each being extended with padding of different lengths, types and locations. Our findings show a notable degradation in LLMs’ reasoning performance at much shorter input lengths than their technical maximum. We show that the degradation trend appears in every version of our dataset, although at different intensities.Additionally, our study reveals that the traditional metric of next word prediction correlates negatively with performance of LLMs’ on our reasoning dataset. We analyse our results and identify failure modes that can serve as useful guides for future research, potentially informing strategies to address the limitations observed in LLMs.

## introduction recent advancements in large language models (llms) show impressive performance across a range of tasks @xcite @xcite , including answering correctly complex questions requiring multiple reasoning steps @xcite . these models also claim to support increasingly longer inputs. this development underscores the need to examine their performance on the longer inputs they are now technically supporting. a reasonable assumption is that support for long inputs would transfer across tasks and enable a model adept at solving a task when presented in a * these authors contributed equally to this work. figure 1 : reasoning performance drops as input grows, across a variety of tasks. inputs are composed of text containing information relevant to the task (in red), and irrelevant text (grey) which is drawn from various sources and extended incrementally. two separate text spans are required to answer correctly, and are located randomly in the input. each point reflects the performance across 600 samples. in this work, we study the effect of increasing the input length on model performance, while keeping other factors as constant as possible. we employ a methodology to measure model performance trends as a function of input length, by isolating it as a variable, while keeping the underlying task intact ( §2). to that end, we introduce flexible length question answering dataset (flenqa) foot_0 , a qa dataset for text-based reasoning ( §3). for each sample, composed of a true/false question over two pieces of information required to answer it (the context), we create multiple versions of different lengths by embedding the context parts within longer, irrelevant texts. to ensure that models utilize their entire input, the dataset is composed of tasks for which both pieces of information must reasoned over together in order to correctly answer the question. at the same time, we keep the tasks simple enough such that models answer most of them correctly when the information pieces are presented on their own, with no additional padding. we show that llms quickly degrade in their reasoning capabilities, even on input length of 3000 tokens, which is much shorter than their technical maximum (on average over all tested models, a drop in accuracy from 0.92 to 0.68). additionally, we explore the effect of embedding the information pieces in various locations within the context, as well as with two kinds of contexts: similar to the information pieces, or dissimilar to them ( §4). we find that regardless of the experimental setting, there are similar trends of degradation. we also show that next-word prediction performance of models on long inputs is uncorrelated with their performance on downstream tasks of reasoning on long inputs ( §5). furthermore, we find that while chain-of-thought (cot) prompting @xcite increases performance in short inputs, in most models it does not mitigate the degradation of performance when inputs are longer: while cot prompting increases the accuracy over non-cot prompting, the amount of increase is roughly consistent across context lengths, and is far from closing the performance drop due to long context ( §6). the only exception to that is gpt4 foot_1 , in which the gap between cot and normal prompting increases as the input is longer. finally, we analyse our results and identify several failure modes in model responses ( §7). we find that with longer inputs models tend not to follow specific instructions in the input, either providing no answer, or -in the case of cot promptingpresenting the final answer before outlining the reasoning steps. we also observe a bias towards answering "false", as well as a decline in the models' ability to incorporate relevant information in their responses, as input length increases. ## desired data properties our goal is to understand how input length affects llms reasoning capabilities over text, given that the relevant information remains the same. we thus use question answering tasks that require models to reason over a given text. for the investigation to be applicable to both open and closed models, we chose a behavioral approach that relies on input intervention @xcite . we aim for our data to satisfy the following requirements: ensuring models reason over the input. to examine the performance of models on long inputs, we require that the task can only be solved correctly by drawing conclusions from evidence in the text @xcite . ## flenqa we introduce the flexible length question answering dataset (flenqa), which follows the requirements set in §2. flenqa is composed of three reasoning tasks: monotone relations (a new task), people in rooms (a new task) and a simplified version of ruletaker @xcite ( §3.2). each task consists of 100 base instances, from which we create variations of different lengths, different background texts, and different dispersion of facts within the background texts ( §3.3). each task is completely balanced in its label distribution ("true" and "false"), and we ensure that most base-instances within it will be solved correctly by the llms when presented in their unexpanded forms ( §3.4). we release the dataset and the code to generate it from scratch to support future studies of reasoning and long input performance. generating the dataset from scratch can be used to prevent data contamination in future evaluation. details and statistics of the tasks appear in appendix a. ## main experiments we report average accuracies over all three tasks, and maintain the same setup (prompt, temperature, etc.) over all input lengths. we evaluate five recent capable llms: gpt4, gpt3.5, gemini-pro, mistral medium and mixtral 8x7b. we consider an output where no answer was mentioned (e.g "i don't know") as incorrect. see appendix c for a detailed breakdown of our setup parameters. ## correlation with next word prediction perplexity is used as the main benchmark to show that models utilize their entire input @xcite @xcite . however, it was shown that performance on downstream tasks does not necessarily correlate with model perplexity @xcite @xcite . here, we will use the flexibility of our dataset to understand the correlation between perplexity and reasoning accuracy. in closed models we lack access to full vocabulary token probabilities so model perplexity cannot be measured, therefore we resort to measuring next word accuracy on our data. we prompt models to complete the next word in a given text, and the output is correct if it is an exact match to the true next word. we use the samples in our dataset (without the questions) as the text and compare the results to the reasoning performance on the same samples. our method finds similar trends on the next word prediction task to those shown in other works @xcite , namely accuracy increases as input is longer. however, as shown in figure 6 , next word accuracy correlates negatively with reasoning on flenqa 6 . this implies that measuring next word prediction and, similarly, perplexity, cannot substitute downstream task evaluation on long inputs. chain of thought (cot) prompting, introduced by @xcite ; @xcite , is a technique by which the llm is pushed to produce a text comprising of reasoning steps before concluding the correct answer for a question. @xcite found that a more specific and optimised instruction ("let's work this out in a step by step way to be sure we have the right answer."). the cot technique was shown to significantly improve the accuracy on many reasoning-based question-answering setups. will using it change the trend and allow the llms to perform effectively on longer inputs? we experiment with cot using the elicitation string of @xcite . the results show (figure 1 ) that cot has different effects on different llms, and overall does not mitigate the drop in performance due to length. in most cases (gpt4, mixtral 8x7b, mistral medium and gpt3.5) it improves performance, but only in gpt4 it has an increased effect as length increases, making it a limited mitigation technique. in the case of gemini-pro, we see that cot decrease performance as input length is increased, even though it increase performance on short length. the full results of the cot prompting over all tasks and setups can be found in appendix c. ## length-induced failure modes we find in the results four failure modes: foot_4 consistent patterns that correlate with incorrect responses. failure to answer all of the samples in the dataset can be answered with either "true" or "false", as instructed in our prompts (appendix b). however, some of llms responded with a refusal answer the question, often preceded by a sentence such as "there is not enough information in the text". this tendency grows as the input length increases, indicating a failure to comply to the instruction that specified a clear choice between "true" and "false". the trend is demonstrated in figure 7 , and results over all models in appendix c. label bias as discussed in §3, our dataset is completely balanced in the label distribution. we find that certain llms tend to favour one of the labels, typically "false", as the input length grows. results for all models are in appendix c. answer first, reason later when using chainof-thought prompting, some llms were much more likely to output the final true/false answer before the expected reasoning steps, as inputs grow longer. in recent work, @xcite found that when models are elicited to provide the reasoning steps after the answer their performance does not increase (as expected when using autoregressive models that only attend to earlier tokens). this can be viewed as a case of failing to follow prompt instructions (see prompt instructions in appendix b) as the length increases. in testing, we found that incorrect responses are statistically dependent on the occurrence of answers before the reasoning steps foot_5 . figure 8 : most of the models tend to generate an answer before the reasoning steps, in a zero-shot cot prompt setting, and do so more as input length increases. ## related work the evaluation of llms on long inputs has followed two distinct pathways: benchmarks of downstream tasks and next word prediction. in the realm of benchmarks, studies proposed datasets of long input samples that can be used to evaluate models @xcite @xcite . those datasets are curated over inputs of different, but fixed, length. this approach, while straightforward, limits our ability to inputs of varying lengths, posing a challenge in understanding the true impact of input length on model performance. on the other hand, next word prediction evaluations do offer an insights into how models handle inputs of different lengths (like done in @xcite . however, the correlation of this task with downstream performance was found not consistent @xcite @xcite . in this paper we reproduce this finding with respect to extended length. this study builds upon prior research that examined different aspects through input intervention, studying the semantic content (theme) of a task @xcite , prompting strategies @xcite @xcite and various properties of the qa task @xcite . our investigation focuses on input length, isolating it, to reveal its impact on performance. ## discussion we study the effect of input length on reasoning performance of current large language models (llms). our findings reveal a significant drop in performance with longer inputs, occurring well before reaching the models' maximum input-length capacity. our experiments relied on flenqa, a dataset we constructed that allows to isolate the length factor, by adjusting the parts in the input that are irrelevant to the task. we show that regardless of how we adjust the samples, there is still a strong effect of length on reasoning performance. finally, we identified specific failure modes, including difficulties in following extended instructions and biases towards less relevant information. our analysis reveals specific failings, providing possible directions for future studies to address and rectify the weaknesses found in llms. in conclusion, our work indicates that evaluating a model's performance based on a single input length does not provide a full picture, and more nuanced evaluation is required. we argue that for a model to be considered capable at long range, it must maintain its performance at any length it technically supports.