Scalable Pre-training of Large Autoregressive Image Models
Abstract
This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling properties. Specifically, we highlight two key findings: (1) the performance of the visual features scale with both the model capacity and the quantity of data, (2) the value of the objective function correlates with the performance of the model on downstream tasks. We illustrate the practical implication of these findings by pre-training a 7 billion parameter AIM on 2 billion images, that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at this scale, we observe no sign of saturation in performance, suggesting that AIM potentially represents a new frontier for training large-scale vision models. The pre-training of AIM is similar to the pre-training of LLMs, and does not require any image-specific strategy to stabilize the training at scale.
Community
Is this an image classification model?
Never liked the super specific self supervised training for images with so many hacks and complexities only working under very specific conditions. Contrastive learning is the worst.
Is this an image classification model?
Like GPT, it is a next token prediction model that operates in the pixel domain. It is trained to predict the next patch based on a sequence of prefix patches. Similar to the self-supervised ViT models like DINO, you can use intermediate representations of the trained model for linear probing or kNN classification, as shown in Figure 10, or you can come up with an original few-shot prompting/finetuning technique, as in the language modeling GPT.
Is this an image classification model?
Like GPT, it is a next token prediction model that operates in the pixel domain. It is trained to predict the next patch based on a sequence of prefix patches. Similar to the self-supervised ViT models like DINO, you can use intermediate representations of the trained model for linear probing or kNN classification, as shown in Figure 10, or you can come up with an original few-shot prompting/finetuning technique, as in the language modeling GPT.
Interesting, thank you
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Rejuvenating image-GPT as Strong Visual Representation Learners (2023)
- Improve Supervised Representation Learning with Masked Image Modeling (2023)
- 4M: Massively Multimodal Masked Modeling (2023)
- Semantic-Aware Autoregressive Image Modeling for Visual Representation Learning (2023)
- A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
Models citing this paper 4
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper