Text Rendering Strategies for Pixel Language Models (2023)

Abstract: Pixel-based language models process text rendered as images, which allows them to handle any script, making them a promising approach to open vocabulary language modelling. However, recent approaches use text renderers that produce a large set of almost-equivalent input patches, which may prove sub-optimal for downstream tasks, due to redundancy in the input representations. In this paper, we investigate four approaches to rendering text in the PIXEL model (Rust et al., 2023), and find that simple character bigram rendering brings improved performance on sentence-level tasks without compromising performance on token-level or multilingual tasks. This new rendering strategy also makes it possible to train a more compact model with only 22M parameters that performs on par with the original 86M parameter model. Our analyses show that character bigram rendering leads to a consistently better model but with an anisotropic patch embedding space, driven by a patch frequency bias, highlighting the connections between image patch- and tokenization-based language models.

## introduction there is a growing movement in nlp towards tokenization-free methods @xcite @xcite including pixel-based representations of text @xcite @xcite . it has been shown that these tokenization-free methods can readily handle unseen languages and that they are more robust to noise attacks than tokenizationbased models. in addition, pixel-based approaches can effectively exploit visual similarities between characters and scripts because they allow for complete parameter sharing across all inputs, making them a promising direction for multilingual nlp. previous work on pixel-based models segments the rendered text into either consecutive patches @xcite or with a sliding window @xcite as in speech processing. although the proposed approaches have the appealing properties of yielding compact and transferable representations, they also result in a very large input space because there is no unique way to represent lexical units. as a consequence, pixel-based models could observe a new set of image representations with every new sentence, which adds redundancy in the input space and is sub-optimal for developing contextual language representations. we refer to these unstructured rendering strategies as continuous and illustrate the point qualitatively in figure 1 and figure 2 , and quantitatively in figure 3 . in this work, we ask whether structuring the input, which leads to more frequent parameter updates through now-unique word representations, would enable pixel-based models to develop a deeper understanding of context and semantics. we then propose rendering strategies structured around providing the model with a compressed input space. we demonstrate how enforcing a bigramsstructured rendering strategy leads to both a more capable and data-efficient model: when evaluated on semantic sentence-level tasks, we find that a 22m parameters model performs 10155 competitively with the unstructured original at 86m parameters, and that scaling back up to 86m parameters narrows the performance gap to bert @xcite trained on the same data. in subsequent analyses, we find that the added input structure provokes a clear visual token frequency bias in the learned embedding space. while also found in bert, frequency biases have been shown to degrade the quality of embedding spaces when word representations are not only determined by semantic relations but also by the number of model updates @xcite @xcite . we show that frequent words have more context-specific representations than infrequent words, especially in the upper layers. finally, we show that pixel models acquire a non-trivial semantic understanding during pretraining, but that their sentence representations are easily influenced by this frequency bias. we release all models 1 and code 2 for pretraining and finetuning. ## background: modelling text as images we build upon the general-purpose language encoder framework presented in @xcite : pixel is a text autoencoder which builds on the masked autoencoding vision transformer (vit-mae; @xcite and is similarly pretrained with a masked reconstruction objective. however, instead of patches from natural images of objects @xcite , the patches now contain images of text. to go from text to images of text, pixel relies on a rendering library (pangocairo) 3 to produce a sequence-level image which is sliced into image patches of size 16 × 16 pixels. the sequence-length maximum of 529 patches approximately equals the memory requirements of bert, the closest benchmark for pixel. by using the google noto font family which supports the majority of unicode codepoints, 4 the renderer supports all languages that can currently be typeset. before the first layer of the pixel model, image patches are linearly projected to obtain a sequence of patch 'embeddings'. during pretraining, 25% of embeddings are masked in spans of up to 6 patches and only the unmasked patches with a prepended cls embedding are passed through the encoder. after replacing the masked embeddings amidst the encoder outputs, relying on fixed sinusoidal position embeddings for ordering information, the decoder predicts the pixel values of solely the masked patches. to later finetune the encoder on a classification task, the decoder can be replaced with a task-specific head and the masking ratio set to 0%. ## structured rendering previously proposed approaches to rendering text as images render full sequences of text and segment into either consecutive patches @xcite or with a sliding window @xcite . these continuous strategies result in a significant number of uniquely-valued patches, many of which may be observed only once during training. we depict this redundancy in figure 2 and quantify it in figure 3 , showing how similar text inputs result in unique visual representations. we compare four rendering strategies: the original unstructured (continuous), and three structured (words, mono, bigrams), as depicted in figure 1 . to render words we separate seg-ments with additional whitespace foot_1 such that new segments begin at the beginning of the next image patch, regulating possible spatial variation. bi-grams, rendering two characters per image patch, is chosen to be widely applicable, without knowledge of word or morphemic segmentation @xcite . more specificallyconsider the word pairs ⟨"grow", "growing"⟩ and ⟨"growing", "walking"⟩-the bigrams renderer will produce an overlap of image patches (underlined) for both pairs while the same extent is not guaranteed with words-level rendering as it is regulated by character width. the choice of character (@xmath0. mono instead applies monospaced fonts where each character is a fixed width; depending on font size, this may result in character bigram patches without breaks within characters, but this is not guaranteed. the main difference between bigrams and mono is that mono simply slides across the sentence, two characters at the time, yielding two ways to represent a word whereas bigrams renders the words and then pads with whitespace, ensuring unique inputs. 6 as seen in figure 3 , the structured rendering strategies result in a greatly compressed input space as measured by the number of unique image patches processed by the model, but figure 1 reveals that it comes at the cost of longer sequence lengths. while the rendering strategies we propose were not specifically designed for english, they may not equally generalise to other languages or scripts. we further discuss the representational efficiencies of these strategies in § a.1 and limitations to generalisability under limitations. ## model scale variants recall from figure 3 that continuous rendering produces a significantly larger set of unique image patches compared to other approaches. a consequence of this is that models must learn to encode many almost-identical visual representations, which may be wasteful, both in terms of parameters and training efficiency. therefore, we hypothesise that pixel models that operate over fewer unique image patches can be scaled down without sacrificing performance. while "base" models and larger ones are widely used for their strong performance, proven scaling laws @xcite enable greater experimentation and model development at smaller scale @xcite , which is both more environmentally friendly @xcite @xcite and facilitates contributions with limited computational resources. with this in mind, we propose two smaller architectures which we will compare across downstream tasks in § 5. our base model architecture is directly adopted from vit @xcite and pixel, and we add two more compact small and tiny model variants, as described in ## experiments we pretrain small models with the proposed rendering strategies. the models are then evaluated on dependency parsing (udp) with data from universal dependencies v2.10 treebanks @xcite and glue @xcite , exploring the models' capabilities at syntactic processing on the word level and semantic processing on the sentence level. ## related work recent work on pixel-based language modelling has demonstrated how visual language understanding can be achieved through pixels only @xcite , observed that the visual similarity of languages plays an important role in cross-lingual transfer @xcite , and shown how unifying the modalities for text and images allow a single encoder to perform multimodal tasks @xcite . by relying on bytes directly, the unification of modalities can be taken even further @xcite @xcite . the work most closely 16 mean hidden state output across all tokens in a sentence, excluding the cls token and black end-of-sequence token. our addition of bigrams structure resembles the addition of optional but hugely beneficial (@xmath1. while character-level n-gram models @xcite have been succeeded by transformerbased language models, character-level features remain valuable as they are less sparse and more robust to misspellings than word n-grams, and remain useful for especially morphologically rich languages @xcite . previous work have hypothesised that character-level models would be more suitable than subword-based for modelling morphologically-rich languages @xcite , but a semantically capable design has proven non-obvious @xcite @xcite . we see potential for future work with pixel-based language models exploring appropriate strategies for learning morphological patterns @xcite @xcite . ## conclusion we evaluate four text rendering strategies to address the problem of redundancy in the input space of pixel-based language models. consequently, more frequent parameter updates lead to better contextualised language representations. we find that rendering two characters per image patch (bigrams) is a good trade-off between efficiency and generalisability, resulting in substantial improvements on downstream semantic and sentencelevel tasks; contributing to open-vocabulary nlp with limited computational resources. further analyses reveal how the added rendering structure provokes clear representational similarities to what has been found in bert. we see potential in future work generalising improvements found for tokenization-based masked language models to pixel-based masked language models. furthermore, considering that the vision transformer has also been applied to speech modelling @xcite , and that patch representation has been suggested to be a critical component for the success of vits @xcite , we see potential for image patches as the basis for unifying modalities.