PIXAR : Auto-Regressive Language Modeling in Pixel Space (2024)

Abstract: Recent work showed the possibility of building open-vocabulary large language models (LLMs) that directly operate on pixel representations. These models are implemented as autoencoders that reconstruct masked patches of rendered text.However, these pixel-based LLMs are limited to discriminative tasks (e.g., classification) and, similar to BERT, cannot be used to generate text.Therefore, they cannot be used for generative tasks such as free-form question answering. In this work, we introduce PIXAR, the first pixel-based autoregressive LLM that performs text generation. Consisting of only a decoder, PIXAR can perform free-form generative tasks while keeping the number of parameters on par with previous encoder-decoder models.Furthermore, we highlight the challenges of generating text as non-noisy images and show this is due to using a maximum likelihood objective. To overcome this problem, we propose an adversarial pretraining stage that improves the readability and accuracy of PIXAR by 8.1 on LAMBADA and 8.5 on bAbI— making it comparable to GPT-2 on text generation tasks.This paves the way to build open-vocabulary LLMs that operate on perceptual input only and calls into question the necessity of the usual symbolic input representation, i.e., text as (sub)tokens.

## introduction in natural language processing (nlp) pipelines, tokenizers are an essential ingredient used to divide the raw text into a sequence of sub-units, such as sub-words @xcite , characters @xcite , sentence pieces @xcite , and bytes @xcite . normally, nlp models represent these sub-units as symbols within an id-based categorical vocabulary. this categorical representation highlights the following weaknesses of traditional nlp systems. first, to encode each item in the vocabulary, nlp systems have to allocate an embedding matrix that grows linearly with vocabulary size. modern large language models (llms) allocate millions of parameters just for this. 1 additionally, fixing a vocabulary a priori can lead to performance degradation due to unseen out-of-vocabulary (oov) words @xcite . tokenizers with smaller granularities, such as characters and bytes, can alleviate the oov issue but are still brittle as they can suffer from orthographic attacks @xcite . on the other hand, humans are incredibly robust to a variety of text permutations @xcite because they leverage the graphical information in text @xcite . to tackle these problems, @xcite proposed pixel, a pixel-based llm that treats text as images. pixel-based embeddings remove the need for a finite vocabulary and keep the visual information of text, questioning whether we need symbolic representations of text as input at all, or if a pixelbased llm can learn symbols implicitly. pixel achieved comparable performance with bert @xcite in a range of downstream classification and regression nlp tasks while being robust to character-level visual attacks @xcite . however, because of its close architectural similarities with bert, pixel cannot deal with free-form generative tasks, such as generative question answering @xcite . to fill this gap, we present pixar, the first pixelbased autoregressive llm that can generate short sequences of text as images. pixar is to gpt-like architectures as pixel is to bert-like architectures: it consists of a transformer decoder @xcite that autoregressively generates text image patches as output. generating new text as pixels starting from pixels only is, however, more challenging than selecting symbolic tokens from a vocabulary (as gpt-like models) or reconstructing masked image patches (as in pixel). this is because the model has to learn to generate longer sequences of pixels. to this end, we introduce a two-stage pretraining strategy for pixar. first, following previous work on autoregressive llms @xcite and image generation models @xcite , pixar is trained by reconstructing the next patch of pixels derived from a large-scale corpus of rendered text using teacherforcing. this maximum-likelihood approach, however, can generate image patches containing noisy text. to mitigate this problem, we proposed a second pretraining stage, where pixar is trained with an additional adversarial loss. our experiments in section 4 show that 200 steps of stage 2 pretraining improve the readability of generated text significantly and achieve comparable performance with gpt-2 @xcite on open-answer short generative tasks such as babi @xcite and lambada @xcite . additionally, pixar achieves better performance than pixel on the glue benchmark @xcite while using a computational budget and a number of model parameters equivalent to the encoder part of pixel. ## beyond token-based llms the idea of using pixel-based representations of text has been applied in various nlp tasks. for instance, @xcite used a cnn-based block to extract character-level visual representations of chinese writing and similarly, @xcite for text classification. graphical features of chinese characters are a good example where canonical symbolic tokenizers miss important information, as highlighted in @xcite exploiting the tianzige feature of chinese characters or @xcite and chinesebert @xcite integrating character-level visual features into embedding vectors for bert-like models. sub-word tokenization as in character-level or byte-level tokens is possible and partially solves some issues. 2 however, these approaches still struggle with a medium that is inherently visual. this is the case if we consider certain writing systems that are (partially) logographic such as in chinese, or when text contains emojis. pixel-based representations mitigate these issues. @xcite built machine translation models using visual information as input. however, their output layers still rely on embeddings over a fixed vocabulary. inspired by @xcite , pixel @xcite pretrained a masked au-toencoder (mae) @xcite with a large corpus of rendered text using a masked reconstruction objective. this can be considered the first purely pixel-based llm that can be applied to a variety of downstream tasks such as pos tagging and extractive question answering. this approach was extended by @xcite to deal with multiple languages as well. moreover, visual text representation can be utilized in multimodal models to build a unified representation for both image and text modalities. specifically, @xcite replaces the id-based text encoder of clip @xcite with a single pixelbased encoder to process both regular images and rendered text for visual qa. due to the encoderonly architecture of pixel and clippo, they still cannot be used to generate text. glyphdiffusion @xcite made attempts to use a diffusion model to generate new texts as images from noise and conditioned on encoded text features. however, its encoder still relies on symbolic embeddings. on top of the benefits of purely pixel-based llms, we are interested in understanding if a deep learning model can learn symbolic representation from perceptual information only, a very challenging task, as discussed next. ## pixar in contrast with the aforementioned approaches, we design pixar as a pixel-based autoregressive llm whose inputs and outputs are only pixels representing textual information. pixar abandons the mae architecture of pixel and instead applies the design choices of other generative llms like gpt-2 and llama-2 @xcite . pixar is pretrained to predict the next patches of pixels, conditioning only on the previous patches of rendered text. while this allows pixar to tackle free-form generative tasks, it presents several challenges. we start by reviewing its architecture. ## experiments we aim to answer the following research questions: rq1) how does pixar compare w.r.t., pixel on discriminative tasks? rq2) how does it perform as a generative model? rq3) how robust is generated text to orthographic attacks? and rq4) what does pixar attend to in a prompt if no symbolic tokens are provided to it as in classical llms? ## conclusion we introduced pixar, an autoregressive pixelbased llm that inherits the benefits of the first purely pixel-based llm pixel but, differently from it, can support generative language tasks. we showcased the capabilities of pixar on discriminative tasks such as glue where it achieves comparable (or superior in some cases) performance w.r.t. pixel, and on generative free-text qa tasks, such as lambada and babi, where it gets closer to gpt-2 while being more robust to orthographic attacks. while these results are encouraging, pixar performance is still behind more sophisticated llms that operate on symbolic tokens. this is expected, as reading and generating text as pure pixels is a much more challenging task, and we expect to improve on pixar's architecture and performance in future work, e.g., by training larger variants of pixar, on more text data rendered as images, and with more language variants. furthermore, we highlighted how an adversarial loss can greatly improve the quality and accuracy of generated text. we consider pixel-based language models as a promising avenue for several reasons: 1) they represent a more naturalistic way of "accessing" language data which can facilitate fusion with other visual inputs such as images; 2) when developing personal assistant for real-world use cases such as vqa systems for visually impaired users @xcite , it is essential to be able to understand language that is embedded in the visual input (e.g., reading labels on cans or medicines). overall, our work shed light on the possibility of treating texts as image-domain data paving the way towards more expressive language models that can efficiently generalise across languages and cultures @xcite and derive language representations in a fully multimodal way @xcite .