Autoregressive Pre-Training on Pixels and Texts (2024)

Abstract: The integration of visual and textual information represents a promising direction in the advancement of language models. In this paper, we explore the dual modality of language—both visual and textual—within an autoregressive framework, pre-trained on both document images and texts. Our method employs a multimodal training strategy, utilizing visual data through next patch prediction with a regression head and/or textual data through next token prediction with a classification head. We focus on understanding the interaction between these two modalities and their combined impact on model performance. Our extensive evaluation across a wide range of benchmarks shows that incorporating both visual and textual data significantly improves the performance of pixel-based language models. Remarkably, we find that a unidirectional pixel-based model trained solely on visual data can achieve comparable results to state-of-the-art bidirectional models on several language understanding tasks. This work uncovers the untapped potential of integrating visual and textual modalities for more effective language modeling. We release our code, data, and model checkpoints at https://github.com/ernie-research/pixelgpt.

## introduction recent advancements in large language models (llms) have pushed the boundaries of their capabilities in diverse applications, including language assistant @xcite , code generation @xcite , and multimodal comprehension @xcite . llms typically tokenize input text into sequences of discrete subword units, allowing for a wide array of applications. however, tokenization-based approaches struggle with visually complex textual content, such as pdfs, where * work done during ql and jx's internship at baidu. converting visual data into plain text often results in significant information loss. traditional solutions rely on optical character recognition (ocr) models for extracting text from images, but these methods are inherently limited by the accuracy of text extraction and the fidelity of the original document structure. to address these challenges, recent work has introduced a new paradigm: pixel-based language modeling. this approach learns directly from the visual representation of text (as images) rather than relying solely on tokenized text. models such as pixel @xcite exemplify this shift, offering solutions that circumvent the limitations of traditional tokenization by treating text as image data. pixel-based modeling also addresses the vocabulary bottleneck-a trade-off between input encoding granularity and the computational costs associated with vocabulary estimation in conventional language models @xcite . in the previous literature, the development of pixel-based language models has been bifurcated into encoder-based @xcite or encoder-decoder architectures @xcite , encompassing models that either employ bidirectional mechanisms akin to mae @xcite or utilize an encoderdecoder framework, where a pixel-based model serves as the encoder, paired with a unidirectional language decoder. despite these advancements, the exploration of pixel-based models employing a decoder-centric approach remains in its infancy. moreover, current research often processes visual text as 8-bit grayscale @xcite or 2bit binary images @xcite . this approach constrain the richness of the visual input, especially when processing content with color information, such as emojis or highlighted text. this limitation suggests that processing real-valued rgb images could offer a more detailed representation of visual text. however, the potential of pre-training autoregressive language models on raw rgb images, which more closely mirror the natural visual characteristics of documents, has not been fully explored. this research addresses two distinct challenges in language modeling: (1) the feasibility of tokenization-free autoregressive pre-training using pixelgpt, and (2) the synergistic benefits of multimodal pre-training with dualgpt. first, we focus on the performance of pixelgpt, a tokenization-free model that processes raw visual text images. we investigate whether training an autoregressive model directly on real-valued pixels can achieve competitive results without tokenization, particularly in multilingual contexts. this exploration assesses whether pixelgpt can overcome the vocabulary bottleneck in multilingual tasks by generalizing linguistic features across diverse languages, thus bypassing the constraints of predefined vocabularies typically encountered in traditional text-based models. second, we evaluate dualgpt, which integrates both visual and textual modalities during pretraining. by leveraging pixel-based and text-based pre-training together, dualgpt is designed to harness the interaction between these two modalities. we explore how this multimodal strategy improves model performance on language understanding tasks and cross-lingual generalization, offering advantages over models that rely on a single modality. contribution to conclude, our main contributions are as follows: ## conclusion and future work in this paper, we have investigated the potential of pixel-based autoregressive pre-training using visual text images. our results demonstrate that incorporating visual orthographic features significantly enhances language understanding and mul-tilingual capabilities. additionally, our empirical findings suggest that using pixel-text paired data effectively reduces modality competition during training, thereby improving model performance. looking forward, scaling this approach to larger model sizes holds considerable promise for advancing the field of multimodal language processing.