Pixel-Level BPE for Auto-Regressive Image Generation (2022)

Abstract: Pixel-level autoregression with Transformer models (Image GPT or iGPT) is one of the recent approaches to image generation that has not received massive attention and elaboration due to quadratic complexity of attention as it imposes huge memory requirements and thus restricts the resolution of the generated images. In this paper, we propose to tackle this problem by adopting Byte-Pair-Encoding (BPE) originally proposed for text processing to the image domain to drastically reduce the length of the modeled sequence. The obtained results demonstrate that it is possible to decrease the amount of computation required to generate images pixel-by-pixel while preserving their quality and the expressiveness of the features extracted from the model. Our results show that there is room for improvement for iGPT-like models with more thorough research on the way to the optimal sequence encoding techniques for images.

## introduction modern deep learning includes a broad scope of problems with varying difficulty. to solve these tasks a paradigm of pre-training is widely used in some domains, to the greatest extent in computer vision (cv) and natural language processing (nlp). whilst unsupervised or self-supervised pre-training is more dominant in the nlp domain, cv models are mainly trained using large amounts of labeled data. authors of igpt @xcite have attempted to prove that given appropriate conditions (namely flexible architecture and significant amount of computation) it is possible to pre-train a model that will reach state-of-the-art performance on several cv downstream tasks even with unlabeled data. they have achieved it using an autoregressive pixel-level image generation as an unsupervised training objective for training a transformer @xcite model. the approach of pixel-by-pixel generation exploited in the igpt paper simply models an image as a continuous sequence of pixels and models the probability distribution of the next pixel conditioned on all previous ones. flattening images results in sequences of an enormous length, for example, such representation of a 128x128 rgb image will require 49152 tokens, which is infeasible for rnns as well as for transformer models where complexity is quadratic with respect to the sequence length. despite there being numerous ways of optimizing attention operation in transformer authors of the igpt model have deliberately chosen dense attention due to it being domain agnostic and not imposing any additional biases on the data. in our work, we continue research in this direction concentrating on the optimization of the image-tosequence representation mechanism rather than the attention mechanism or the transformer architecture itself. in the presented paper we try to adopt a tokenization approach widely used in the nlp domain: byte-pair encoding (bpe) to the image domain to mitigate the main issue of the original igpt paper. these methods allow to significantly squeeze input sequences thus reducing the amount of computation required for training and inference. following the methodology of the original paper, we also test the ability of the transformer model pre-trained on image generation to be used as a feature extractor that competitively performs on downstream tasks, namely, image classification on cifar datasets foot_0 . the main contributions of this paper are as follows: ## related work autoregressive approaches have proven to be very efficient in the nlp domain both in a pretraining and a variety of natural-language generation tasks ( radford et al. (2019), raffel et al. (2020) ). however, in the cv domain, it has been quite a challenge due to the high dimensionality of the data. one of the effective ways to tractably model a joint distribution of pixels in an image is to cast it as a product of conditional distributions. it was adopted in several models such as fully visible sigmoid belief networks @xcite or nade @xcite . recurrent neural networks (rnn) are powerful models that offer a compact, shared parametrization of a series of conditional distributions. authors of @xcite have applied this architecture to an image domain. the authors suggested two types of convolutional lstm layers to compute all the states along one of the spatial dimensions (rows or diagonals of the image). moreover, instead of lstm blocks a convolutional layer with a mask to avoid seeing the future context was used. this method was called pixelcnn and got further development such as pixelcnn++ @xcite . a small receptive field was an obvious disadvantage of these approaches that was overcome with the emergence of transformers. transformer-based @xcite models are extremely successful in natural language generation and understanding fields. @xcite demonstrated human-level performance in text generating and zero-shot tasks via prompt engineering. there were numerous attempts to use gpt architecture for image generation, which can be divided into two groups: discrete feature-based regression (e.g. @xcite ) or pixel-level regression (igpt chen et al. ( 2020 )). the latter type of model is not fairly popular, as processing the 1d-sequence of flattened rgb-image pixels is too memory-expensive due to the length of the context and attention mechanism. to deal with this problem authors resize images to a low resolution (like 32 2 × 3, 48 2 × 3, 96 2 × 3 or 192 2 × 3) with further clustering (r, g, b) pixel values using k-means with @xmath0. however, the igpt model demonstrated decent results in lowresolution image generation and downstream tasks over contextualized features. to measure model performance linear probe method was used. the method consists of training multi-class logistic regression on embeddings from a model with frozen weights on an image classification task. during pretraining on imagenet authors also used vqvae as a downsampler instead of rgb-clustering to keep the context of 48 2 length. on the other side, there are numerous methods for sequence length compression in the nlp domain -different tokenization techniques, which exploit the pre-computed merge dictionaries for optimal encoding of words or byte groups. one of the most efficient methods is byte-pair-encoding @xcite . the idea of this algorithm is to find the most frequent pair of consecutive twocharacter codes in the text and then substitute an unused code for the occurrences of the pair. this method has become a good trade-off between vocabulary size and the length of the sequence fed to the model. in gpt models special modification of this algorithm is used which works at bytelevel @xcite ) -this is one more step towards optimal sequence squeezing. ## igpt with bpe image tokenization our bpe-enabled igpt model relies on the gpt-2 model originally designed for text processing. more specifically we use embedding size @xmath1, number of layers @xmath2. due to limited computational resources, we have not conducted experiments with the bert pre-training objective and used only linear probing as an evaluation approach. in our experiments, we provide results for prompted image generation and linear probe on ci-far10 and cifar100 datasets with pre-training on imagenet @xcite dataset. also we demonstrate unconditional image generation on celeba dataset @xcite aligned with mtcnn framework @xcite . ## conclusion in this paper, we explored the use of the bpe technique originally proposed for textual data in the image domain. it allows significantly squeeze the tokenized image sequence length mitigating the limitations of the original igpt model. we quantitatively show that this method reduces the amount of required computation by an order of magnitude and qualitatively verify that it does not affect the quality of generated images. moreover, applying bpe tokenization improves the representative ability of the models trained on unlabeled data. our results suggest that the potential of image-to-sequence squeezing is not fully unleashed yet and that there is room for improvement of igpt-like models.