N ext L evel BERT : Masked Language Modeling with Higher-Level Representations for Long Documents (2024)

Abstract: While (large) language models have significantly improved over the last years, they still struggle to sensibly process long sequences found, e.g., in books, due to the quadratic scaling of the underlying attention mechanism. To address this, we propose NextLevelBERT, a Masked Language Model operating not on tokens, but on higher-level semantic representations in the form of text embeddings. We pretrain NextLevelBERT to predict the vector representation of entire masked text chunks and evaluate the effectiveness of the resulting document vectors on three types of tasks: 1) Semantic Textual Similarity via zero-shot document embeddings, 2) Long document classification, 3) Multiple-choice question answering. We find that next-level Masked Language Modeling is an effective technique to tackle long-document use cases and can outperform much larger embedding models as long as the required level of detail of semantic information is not too fine. Our models and code are publicly available online.

## introduction while the transformer model is the basis for most large language models, its attention mechanism scales quadratically with respect to the length of the input sequence @xcite . this makes it prohibitive to process longer input sequences such as books or movie scripts. to address this limitation, the most common strategy has been to sparsify or approximate the attention operation @xcite @xcite , or to extend the positional information during inference @xcite . other approaches to process long documents include switching to a retrieval augmentation approach @xcite , modeling long-range dependencies via a graph structure @xcite , as well as compressing the input either via an explicit memory module @xcite or via hierarchical processing @xcite . further related work is discussed in section 5. from a cognitive perspective, language is widely regarded as a hierarchical process @xcite @xcite , enabling connections between distant pieces of information. when investigating the differences between how human brains and typical large language models process language, @xcite found that humans seem to use a predictive coding hierarchy @xcite to anticipate words at multiple time spans. beyond word-level predictions, the original bert architecture @xcite also includes next sentence prediction as one of the pretraining tasks, but this still is based on individual tokens as input. masked language modeling has previously been explored at the level of short spans of tokens @xcite , but again assuming tokens as inputs. we present nextlevelbert, a masked language model operating entirely at larger timescales, with inputs that are not individual tokens but rather generic chunks of text (e.g., sentences or larger spans of tokens). for each chunk, we obtain a representation using a pretrained encoding model. some of these chunks are then masked and the respective representations need to be predicted based on the context. our results show that applying masked language modeling to higher-level semantic representations results in meaningful document vectors. nextlevelbert uses only a third of the number of parameters of its strongest contender, but showcases stronger performance than other commonly used approaches during downstream testing as long as the task does not require a high level of semantic detail at the token level. our contributions are as follows: we apply masked language modeling in a novel way on text in a hierarchical setup, the frozen sentence encoder first generates independent chunk vectors from the input document. the chunk vectors v 2 and v n are masked and need to be predicted from the surrounding context. through its higher-level masked language pretraining objective, nextlevelbert learns to produce contextualized versions of the chunk vectors. ## next-level masked language modeling our goal is to obtain a masked language model that operates on higher-level semantic units. instead of predicting masked tokens based on other tokens in the context, we predict masked text chunk representations based on other text chunk representations. ## conclusion successfully modeling long-range dependencies is an important step towards building language models with general language understanding. inspired by the idea of modeling language at a coarser resolution than that of tokens, we have proposed nextlevelbert, and transferred masked language modeling from the token level to general text representation vectors. we pretrained nextlevelbert on a large collection of books and evaluated the resulting text representations on three diverse downstream tasks in a long document setting. our results show that next-level masked language modeling leads to sensibly contextualized text chunk representations and generates useful long-document vectors, especially for tasks that do not require specific token-level information. additionally, nextlevel-bert is very parameter-efficient, using only a third of the parameters of the nomic embed model while maintaining similar performance. in the future, combining the predictions of a nextlevel model with a token-level component based on the required granularity could be a way of preserving token-level details. furthermore, pretraining nextlevelbert on data from more diverse domains could improve its performance in general and extend its usefulness to a wider range of tasks. lastly, while not straightforward, adapting decoderonly models to the next level could translate similar gains as in the mlm setup to generative language models.