  Autoregressive models are widely used for tasks such as image and audio
generation. The sampling process of these models, however, does not allow
interruptions and cannot adapt to real-time computational resources. This
challenge impedes the deployment of powerful autoregressive models, which
involve a slow sampling process that is sequential in nature and typically
scales linearly with respect to the data dimension. To address this difficulty,
we propose a new family of autoregressive models that enables anytime sampling.
Inspired by Principal Component Analysis, we learn a structured representation
space where dimensions are ordered based on their importance with respect to
reconstruction. Using an autoregressive model in this latent space, we trade
off sample quality for computational efficiency by truncating the generation
process before decoding into the original data space. Experimentally, we
demonstrate in several image and audio generation tasks that sample quality
degrades gracefully as we reduce the computational budget for sampling. The
approach suffers almost no loss in sample quality (measured by FID) using only
60\% to 80\% of all latent dimensions for image data. Code is available at
https://github.com/Newbeeer/Anytime-Auto-Regressive-Model .
