Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens (2024)

Abstract: Large language models (LLMs) have shown promising efficacy across various tasks, becoming powerful tools in numerous aspects of human life. However, Transformer-based LLMs suffer a performance degradation when modeling long-term contexts due to they discard some information to reduce computational overhead. In this work, we propose a simple yet effective method to enable LLMs to take a deep breath, encouraging them to summarize information contained within discrete text chunks. Specifically, we segment the text into multiple chunks and insert special token <SR> at the end of each chunk. We then modify the attention mask to integrate the chunk’s information into the corresponding <SR> token. This facilitates LLMs to interpret information not only from historical individual tokens but also from the <SR> token, aggregating the chunk’s semantic information. Experiments on language modeling and out-of-domain downstream tasks validate the superiority of our approach.

## introduction in recent years, transformer-based large language models (llms) have become a focal point of research, leading to the emergence of numerous powerful models such as chatgpt @xcite , @xcite , llama @xcite and @xcite . however, in mainstream decoder-only models, subsequent tokens can only attend to preceding historical individual tokens without acquiring information from aggregated local contexts, thereby limiting the language modeling capability of llms. to tackle this problem, existing studies have explored various approaches to compress contexts, such as sentinel tokens @xcite , memory slots @xcite , and summary vectors @xcite . however, these methods may lose useful context information during the compression process, leading to performance degradation. besides, some of these compression and accumulation strategies still exhibit quadratic computing complexity of selfattention @xcite . in this work, we introduce a simple yet effective method that allows llms to take a deep breath, enabling them to gather information not only from preceding historical individual tokens but also from special tokens that encapsulate the holistic information of chunks. specifically, we propose a strategy to insert new tokens denoted @xmath0. during training, @xmath1. that is to say, after processing each chunk, the llms are prompted to summarize key information within this chunk. consequently, subsequent tokens can acquire information not only from the preceding individual tokens as in the original approach, but also from the sentinel @xmath2. as a result, when generating the next token, tokens can harness both local information (individual tokens within a chunk) and relatively holistic information (@xmath3. by prompting the llms to take a deep breath, we enable them to obtain richer semantic information during decoding, thereby enhancing its language modeling capability. we conducted experiments on the wikitext-2 language modeling benchmark using models ranging from 1.3b to 13b in size, employing diverse positional encoding strategies @xcite . the experimental results demonstrate that the introduction of sentinel tokens enhances the language modeling capabilities of llms. additionally, we further demonstrated the effectiveness of our method on out-of-domain downstream tasks. ## related work attention mask a number of architectural modifications have been proposed to constrain and sparsify the attention window @xcite @xcite . @xcite introduced conditional computation. @xcite and @xcite introduced some sparse attention mechanisms to reduce computational complexity. however, most of these architectures require expensive training from scratch. our approach modifies the attention mask while requiring only a small amount of fine-tuning. ## approach in this section, we will present our approach which integrating the comprehensive information of each chunk into the @xmath4. this is achieved by strategically placing sentinel @xmath5.