  The importance of hierarchical image organization has been witnessed by a
wide spectrum of applications in computer vision and graphics. Different from
image segmentation with the spatial whole-part consideration, this work designs
a modern framework for disassembling an image into a family of derived signals
from a scale-space perspective. Specifically, we first offer a formal
definition of image disassembly. Then, by concerning desired properties, such
as peeling hierarchy and structure preservation, we convert the original
complex problem into a series of two-component separation sub-problems,
significantly reducing the complexity. The proposed framework is flexible to
both supervised and unsupervised settings. A compact recurrent network, namely
hierarchical image peeling net, is customized to efficiently and effectively
fulfill the task, which is about 3.5Mb in size, and can handle 1080p images in
more than 60 fps per recurrence on a GTX 2080Ti GPU, making it attractive for
practical use. Both theoretical findings and experimental results are provided
to demonstrate the efficacy of the proposed framework, reveal its superiority
over other state-of-the-art alternatives, and show its potential to various
applicable scenarios. Our code is available at
\url{https://github.com/ForawardStar/HIPe}.
