  Anytime inference requires a model to make a progression of predictions which
might be halted at any time. Prior research on anytime visual recognition has
mostly focused on image classification. We propose the first unified and
end-to-end model approach for anytime pixel-level recognition. A cascade of
"exits" is attached to the model to make multiple predictions and direct
further computation. We redesign the exits to account for the depth and spatial
resolution of the features for each exit. To reduce total computation, and make
full use of prior predictions, we develop a novel spatially adaptive approach
to avoid further computation on regions where early predictions are already
sufficiently confident. Our full model with redesigned exit architecture and
spatial adaptivity enables anytime inference, achieves the same level of final
accuracy, and even significantly reduces total computation. We evaluate our
approach on semantic segmentation and human pose estimation. On Cityscapes
semantic segmentation and MPII human pose estimation, our approach enables
anytime inference while also reducing the total FLOPs of its base models by
44.4% and 59.1% without sacrificing accuracy. As a new anytime baseline, we
measure the anytime capability of deep equilibrium networks, a recent class of
model that is intrinsically iterative, and we show that the
accuracy-computation curve of our architecture strictly dominates it.
