Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration (2024)

Abstract: We identify two crucial limitations in the evaluation of recent parallel-integrated method Parallel Context Windows (PCW), which extends the maximum context lengths of language models, e.g., 2048 for LLaMA, by harnessing window-wise attention and positional embedding techniques. We first show that a simple yet strong baseline, weighted sum ensemble, is missing for the in-context few-shot classification. Moreover, on more challenging Chain-of-Thought (CoT) reasoning (e.g., HotpotQA), PCW would present unexpected deterioration regarding question miscomprehension and false inference. Based on our findings, we suggest that the existing PCW design may not guarantee sufficient improvement and practicality in handling lengthy documents in real-world applications. More community efforts on enabling language models’ long context understanding ability should be paid.

## introduction over the past few months, the field of large language models (llms) @xcite @xcite @xcite has undergone a remarkable resurgence, primarily gpt-4, which has proved reasoning abilities akin to human, spanning a variety of professional fields from law to mathematics and physics @xcite . llms experience a paradigm shift, from individual tasks such as machine translation @xcite , text summarization @xcite , and information extraction @xcite , and gravitate toward a unified solution where users engage and interact in dialogues with chatbots to query anything. still, a major challenge remains in llms -their abilities are constrained by their maximum context lengths. for example, @xcite mentions its few demonstration samples in in-context learning (icl) due to length limit. recent auto-gpt @xcite is also observed to suffer from lengthy histories induced by cot @xcite , which shepherds the lms to mirror human cognition through a step-bystep progression of thinking and reflection to solve challenging reasoning missions. hence it is vital to develop techniques to extend the context length of existing llms for reasoning. a recent related attempt is pcw @xcite , which brings the idea of parallel contexts to mitigate the length limitation problem in gpts. pcw segments the text sequence into windows, constraining the attention to be visible within each window while all windows share the same positional embeddings. it reports improvements in few-shot icl classification and generation tasks over the conventional sequential baseline, especially on fine-grained classification tasks with large label space such as banking77 @xcite and clinic150 @xcite . by introducing over-length number of demonstration samples in one sequence, lms can access more labels from context and thus outperform the sequential icl where fewer samples could be seen. however, in this work we identify limitations in pcw's evaluation, especially from two aspects: ## conclusion we raise concerns about the use of parallelintegrated methods to address context length restriction: (1) pcw is functionally equal with a simple weighted sum ensemble on label distribution among context windows; (2) pcw degrades the multi-step reasoning capabilities of llms in complex tasks requiring knowledge understanding. despite the fact that parallel-integrated methods show better classification performance when the label space is large, they merely brute-force ensemble each window's context, consequently weakening logical reasoning and knowledge comprehension.