|
--- |
|
library_name: diffusers |
|
license: mit |
|
pipeline_tag: unconditional-image-generation |
|
--- |
|
|
|
# Autoregressive Image Generation without Vector Quantization |
|
|
|
## About |
|
This model (MAR) introduces a novel approach to autoregressive image generation by eliminating the need for vector quantization. |
|
Instead of relying on discrete tokens, the model operates in a continuous-valued space using a diffusion process to model the per-token probability distribution. |
|
By employing a Diffusion Loss function, the model achieves efficient and high-quality image generation while benefiting from the speed advantages of autoregressive sequence modeling. |
|
This approach simplifies the generation process, making it applicable to broader continuous-valued domains beyond just image synthesis. |
|
It is based on [this paper](https://arxiv.org/abs/2406.11838) |
|
|
|
## Usage: |
|
You can easily load it through the Hugging Face `DiffusionPipeline` and optionally customize various parameters such as the model type, number of steps, and class labels. |
|
|
|
```python |
|
from diffusers import DiffusionPipeline |
|
|
|
# load the pretrained model |
|
pipeline = DiffusionPipeline.from_pretrained("jadechoghari/mar", trust_remote_code=True, custom_pipeline="jadechoghari/mar") |
|
|
|
# generate an image with the model |
|
generated_image = pipeline( |
|
model_type="mar_huge", # choose from 'mar_base', 'mar_large', or 'mar_huge' |
|
seed=42, # set a seed for reproducibility |
|
num_ar_steps=64, # number of autoregressive steps |
|
class_labels=[207, 360, 388], # provide valid ImageNet class labels |
|
cfg_scale=4, # classifier-free guidance scale |
|
output_dir="./images", # directory to save generated images |
|
cfg_schedule = "constant", # choose between 'constant' (suggested) and 'linear' |
|
) |
|
|
|
# display the generated image |
|
generated_image.show() |
|
``` |
|
|
|
<p align="center"> |
|
<img src="https://github.com/LTH14/mar/raw/main/demo/visual.png" width="500"> |
|
</p> |
|
|
|
This code loads the model, configures it for image generation, and saves the output to a specified directory. |
|
|
|
We offer three pre-trained MAR models in `safetensors` format: |
|
- `mar-base.safetensors` |
|
- `mar-large.safetensors` |
|
- `mar-huge.safetensors` |
|
|
|
|
|
<!-- <p align="center"> |
|
<img src="https://github.com/LTH14/mar/raw/main/demo/visual.png" width="720"> |
|
</p> --> |
|
|
|
This is a Hugging Face Diffusers/GPU implementation of the paper [Autoregressive Image Generation without Vector Quantization](https://arxiv.org/abs/2406.11838) |
|
|
|
The Official PyTorch Implementation is released in [this repository](https://github.com/LTH14/mar) |
|
|
|
``` |
|
@article{li2024autoregressive, |
|
title={Autoregressive Image Generation without Vector Quantization}, |
|
author={Li, Tianhong and Tian, Yonglong and Li, He and Deng, Mingyang and He, Kaiming}, |
|
journal={arXiv preprint arXiv:2406.11838}, |
|
year={2024} |
|
} |
|
``` |
|
|
|
## Acknowledgements |
|
We thank Congyue Deng and Xinlei Chen for helpful discussion. We thank |
|
Google TPU Research Cloud (TRC) for granting us access to TPUs, and Google Cloud Platform for |
|
supporting GPU resources. |
|
|
|
A large portion of codes in this repo is based on [MAE](https://github.com/facebookresearch/mae), [MAGE](https://github.com/LTH14/mage) and [DiT](https://github.com/facebookresearch/DiT). |
|
|
|
## Contact |
|
|
|
If you have any questions, feel free to contact me through email (tianhong@mit.edu). Enjoy! |
