Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models (2024)

Abstract: Pixel-based language models have emerged as a compelling alternative to subword-based language modelling, particularly because they can represent virtually any script. PIXEL, a canonical example of such a model, is a vision transformer that has been pre-trained on rendered text. While PIXEL has shown promising cross-script transfer abilities and robustness to orthographic perturbations, it falls short of outperforming monolingual subword counterparts like BERT in most other contexts. This discrepancy raises questions about the amount of linguistic knowledge learnt by these models and whether their performance in language tasks stems more from their visual capabilities than their linguistic ones. To explore this, we probe PIXEL using a variety of linguistic and visual tasks to assess its position on the vision-to-language spectrum. Our findings reveal a substantial gap between the model’s visual and linguistic understanding. The lower layers of PIXEL predominantly capture superficial visual features, whereas the higher layers gradually learn more syntactic and semantic abstractions. Additionally, we examine variants of PIXEL trained with different text rendering strategies, discovering that introducing certain orthographic constraints at the input level can facilitate earlier learning of surface-level features. With this study, we hope to provide insights that aid the further development of pixel-based language models.

## introduction subwords are currently the standard units of processing in language modelling @xcite . while they have been shown to work well in monolingual models @xcite , in a multilingual context they can lead to an inevitable vocabulary bottleneck with each language competing for space in a finite vocabulary @xcite . characters and byte-based models have been proposed as alternatives to subwords, but they lead to longer input sequences @xcite @xcite . another proposed solution is pixel-based models where patches of pixels are the main unit of representation. a canonical example of this is the pixel (pixel-based encoder of language) model @xcite , where text is rendered as a sequence of fixed-sized patches and passed as input to a vision transformer (vit) @xcite . this approach allows the model to represent virtually any script. although current versions of the pixel-based language models do not outperform their monolingual subword-based counterparts on most downstream tasks @xcite , they are a promising approach to multilingual modelling and offer a unique opportunity to explore modelling language through images. pixel is a juxtaposition of a vision and language model: even though it receives image patches as input, the content of those patches is rendered text, making it a visual model of language. with this study, we aim to understand where pixel stands on the vision-tolanguage spectrum. to this end, we probe pixel on various visual and language tasks and compare performance with bert @xcite -the language model it is most comparable to -and vit-mae @xcite -the vision model it is most comparable to. we conduct a comprehensive analysis of the linguistic and visual capabilities of pixel that can be used to aid further development of pixel-based language models. concretely: rq1: how much linguistic knowledge is encoded in pixel? rq2: how much visual capability does pixel have? 2 we find that pixel learns surface-level linguistic information in the lower layers, resulting in higher-level syntactic and semantic abstractions appearing in higher layers than bert ( §5.1). when comparing to vit-mae, pixel underperforms on image tasks, with visual probing accuracy decreasing in the higher layers ( §5.2). thus, the surface-level information is diluted as it acquires linguistic knowledge in the higher layers. @xcite trained newer pixel-based language models that add some orthographic constraints to the input that can potentially augment linguistic learning in the lower layers. in this context, we ask the following question: rq3: does adding orthographic constraints to the input enhance the linguistic capabilities in pixel? we find that a rendering strategy that makes word boundaries more explicit in the input enables pixel to learn surface-level linguistic features earlier in the model, thereby aiding semantic understanding ( §5.3). overall, we take inspiration from bertology, the study of the linguistic capabilities in bert @xcite , and aim for this work to foster future explorations and advancements for pixel. ## probing tasks we now introduce the probing tasks used in our experiments. we probe pixel on two levels: linguistic and visual. for linguistic probing, we rely on the senteval framework. vits have more direct access to surface-level information than subwordbased models since their input is segmented into units of fixed visual size (as opposed to variablesized tokens) and shown to the model after a continuous linear projection (as opposed to a lookup). thus, we also employ tasks that are designed to verify whether orthographic information is more easily identifiable throughout pixel. ## conclusion this study is a first step towards understanding the language modelling capabilities of pixel-based models. although these models exhibit substantial linguistic understanding, the nature of image-text representations leads to a gap in visual and linguistic understanding. pixel-based models need to learn the discrete representations that subwordbased models already have access to at the input level. adding orthographic constraints to the input can help bridge this gap, but further architectural modifications could improve these models more, which is a promising direction for future work.