Self–Similarity Priors: Neural Collages as Differentiable Fractal Representations
Abstract
Many patterns in nature exhibit self-similarity : they can be compactly described via self-referential transformations.
Said patterns commonly appear in natural and artificial objects, such as molecules, shorelines, galaxies and even images.
In this work, we investigate the role of learning in the automated discovery of self-similarity and in its utilization for downstream tasks.
To this end, we design a novel class of implicit neural operators, Neural Collages , which (1) represent data as the parameters of a self-referential, structured transformation, and (2) employ hypernetworks to amortize the cost of finding these parameters to a single forward pass.
We detail how to leverage the representations produced by Neural Collages in various tasks, including data compression and generation.
Neural Collage image compressors are orders of magnitude faster than other self–similarity–based algorithms during encoding and offer compression rates competitive with implicit methods.
Finally, we showcase applications of Neural Collages in fractal art and as deep generative models.
Introduction
Given a specified image, can one come up with a dynamical system with it as its attractor?
(Welstead, 1999)
Scientific fields are underpinned by a search for structure.
Geometry, sparsity and invariances, when appropriately introduced in a mechanistic model, allow us to concisely describe phenomena.
To this end, machine learning has been introduced as a means to fix partial priors in a model and discover the rest through data (Rackauckas et al., 2020; Dao et al., 2020;
Bronstein et al., 2021).
In general, the notion of structure is also essential for compression: through a suitable choice of language, one can explain phenomena in fewer symbols, yielding shorter representations of the observables (Tishby et al., 2000; Lee et al., 2007).
Objects exhibiting self-similarity structure are composed of patterns that appear similar to themselves at multiple scales, as shown in Figure 1.
This type of structure frequently appears in nature, at different degrees: shorelines, molecules, plants, turbulent flows and basins of attraction of dynamical systems all display elements of self-similarity (Mandelbrot and
Mandelbrot, 1982; Song et al., 2005; Vulpiani et al., 2009; Barnsley, 2014).
In this work, we explore the role of learning in the automatic discovery of self-similarity structure in data, and how it can serve as an inductive bias in machine learning models.
The mathematical embodiment of this idea is found in fractal patterns, which often arise by characterizing limit sets of nonlinear maps e.g. iterations of complex numbers for Julia and Mandelbrot sets (Julia, 1918; Mandelbrot, 1980).
Fractals are scale-invariant: they can be "zoomed in" by increasing the resolution of the limit sets, and manifest arbitrarily similar patterns at different scales.
Despite their apparent infinite complexity, a fractal can be uniquely and compactly described by its generating nonlinear map.
A method to discover self-similar structure in data (not necessarily of fractal nature) can then be formalized as an optimization problem: after choosing an appropriate class of contractive, parameterized maps, one searches for parameters such that a given data point can be (approximately) recovered as the (unique) fixed-point of the chosen map.
This approach, pioneered in (Barnsley et al., 1986), paved the way for one of the most successful algorithmic applications of self-similarity, fractal image compression (Jacquin et al., 1992; Jacquin, 1993;
Barnsley et al., 1996; Welstead, 1999; Fisher, 2012).
First, an encoding step carries out a search to solve the inverse problem of data to operator parameters via extensive search, or by restricting the class of operators such that a closed-form solution may be found.
Then, given the parameters, the decoding step solves for the fixed-point of the operator, corresponding to a corrupted version of the original data.
The quality of decoded images i.e. the loss of the fractal compression method is directly tied with the expressivity of the class of operators considered, which are often designed to seek self-similarities in pixel space.
Yet, larger classes induce more challenging optimization problems, leading to long encoding times.
Here, we propose a novel learning-based technique to extract and utilize self-similar representations of data.
We develop Neural Collages , a family of differentiable, parametrized operators structured to capture self-similarity between partitions of data.
The inverse problem of Neural Collages is solved with a single forward pass of a hypernetwork (Ha et al., 2016) trained to generate a set of parameters as the fractal code.
This amortized approach is orders of magnitude faster than traditional search based methods.
Collage operators are composable with neural network architectures and have wide applicability beyond compression, as they can be optimized end-to-end for a variety of tasks including generative modeling.
In data compression, Neural Collages preserve advantages of fractal compression methods, with up to 10 × (accounting for training time) and 100 × (at test time) speedups during encoding.
Further, we investigate deep generative models based on Neural Collages , where Collage parameters assume the role of latent variables of a hierarchical variational autoencoder (VAE) (Kingma and
Welling, 2013).
Collage VAEs can sample at resolutions unseen during training by decoding at higher resolutions through their Collage operator, revealing additional detail over upsampling via interpolation.
Finally, we showcase applications for fractal art, where an image can be "fractalized" i.e. reconstructed as a collage of smaller copies of itself at different scales (see Figure 1, top).
Problem Setting
We start with an example related to a core idea behind fractal compression and Neural Collages: representing data as the parameters of a function that converges to it.
2.1 Representing Data as a Fixed-Point of an Iterated Function
Consider a value x that happens to be of interest for a downstream task, for example regression.
Rather than performing computation on it directly, we can use the parameters ω of a contractive map with x as its fixed-point, for example a simple affine function with parameters ω :
= ( a, b ) , a ∈ R , b ∈ R :
[Equation]
Given ω , the value x can be recovered by repeatedly applying the map starting from any other initial condition, that is
[Equation]
Parameters ω can then replace x as an alternative representation, using two values ( a, b ) instead of x .
Other classes of functions, if chosen appropriately, can represent data with less parameters than its dimensionality.
In general, encoding through a function is a lossy process: the original data is recovered up to a degree of accuracy.
The insight behind fractal compression and Neural Collages is to view data as sets , and define iterated functions as ensembles of simple scalar linear functions acting on different subsets.
Below, we introduce background required to make these notions precise.
2.2 Fractal Compression Background
We proceed by formalizing fractal data encoding as an optimization problem (Barnsley and Demko, 1985; Fisher, 2012; Barnsley, 2014).
Let ( X , d ) be a complete metric space and ( H ( X ) , d H ) its corresponding Haussdorff metric space, i.e. H ( X ) =
{ A ⊂ X : A is compact } .
We represent a data point as a set S ∈ H ( X ) .
This choice of space supports an data modality-agnostic treatment of fractal data encoding.
A concrete realization is discussed for image domains in Section 3.
Results on metric spaces are provided as reference in
Appendix A.
Example 1.
Consider binary images on a square domain.
Then, X is a finite compact subset of R. An image S is the finite set of coordinates of either black or white pixels.
Further, each image corresponds to a point in H ( X ) .
Let { f 1 , f 2 , . . .
, f K } be a collection of maps on X , f k : X → X .
This is colloquially referred to as iterated function system (IFS) (Barnsley, 2014).
We can then define a map F : H ( X ) → H ( X ) by
[Equation]
where f k ( A ) is intended as f k ( A )
= { f k ( a ) : a ∈ A } .
An interpretation is the following: F produces as output a composition, or collage , of transformations applied to a subset A of X .
2.3 The Inverse Problem: Data to IFS
Given data S , can we find a map F with S as its fixed point?
This can be achieved by identifying a collection of maps f k : X → X such that the following conditions hold
[Equation]
ii.
S
[Equation]
Note that, F is contractive w.r.t the Hausdorff metric with Lipsichitz constant L < 1 iff all the maps f k are contractive w.r.t d with constant ℓ k < 1 .
In such a case it holds L = max k { ℓ k } and F admits a unique fixed point.
A classical result provides an indirect method to find such F .
Theorem 1 (Collage Theorem (CT) (Barnsley and Demko, 1985)) .
Let ( X , d ) be a complete metric space and let f : X → X be a ℓ -Lipschitz contractive map with fixed point x ∗ ∈ X .
Then,
[Equation]
By applying the CT directly to F using the Hausdorff metric d H , we have d H ( S , A ∗ ) ≤ 1 1 − L d H (cid:16) S , (cid:83)
Kk =1 f k ( S ) (cid:17) .
This means that we can upper bound the distance d H ( S , A ∗ ) between data S and attractor A ∗ of F via d ( S , F ( S )) , which requires a single application of F and is thus cheaper to evaluate.
Even if it is not possible to stitch together transformed copies f k ( S ) to perfectly reconstruct the data S ,
i.e. d H (cid:16) S , (cid:83) Kk =1 f k ( S ) (cid:17) ̸ = 0 ( ⇔ S ̸ = F ( S )) , a smaller IFS Lipschitz constant L of the IFS implies a lower distance between the data S and the attractor A ∗ of F , given a mismatch d H ( S , F ( S )) .
This, in turn, implicitly promotes the use of “very contractive” maps f k (i.e. with low ℓ k ).
We refer to the procedure of searching for an F that minimizes the r.h.s of the CT bound as the fractal data encoding problem.
A learning perspective of fractal data encoding Fractal data encoding problem can be translated into finding a parametric representation f k ( · ; w k ) , w ∈ R n w for functions f k ( · ) (e.g. neural networks with parameters w k ) where w = ( w 1 , . . .
, w K ) ∈ W are optimized to minimize a Hausdorff metric loss function d H ( S , F ( S ; w )) naturally induced by the CT,
[Equation]
To decode the data encoded in parameters w of F encodes in one samples any initial condition A 0 , and iterates A t
+1 =
F ( A t ) until convergence to A ∗ ≈ S .
IFS, PIFS and Beyond Partitioned iterated function systems (PIFS) (Jacquin et al., 1992) are a generalization of IFSs that can capture localized self-similarity by allowing each domain of a contraction map f k to be a different subset A k ⊂ S .
his introduces a significant challenge in 2.2: the optimization problem need now determine optimal (as measured by d H ) domains A k for each f k by searching across all possible subsets of S , yielding an exploding combinatorial problem.
Solving for affine IFS
As with traditional approximation problems, there is a tension in the objective of fractal data encoding between the "expressiveness" of the class of functions, and the tractability of the optimization problem.
The solution w of (2.2) is an equivalent representation for S (up to
d H ( S , A ∗ ) ).
Existing methods based on the idea of (Barnsley and Demko, 1985) resolve this tension by considering (a) compression as a task, such that w should be encodeable in the least number of bits possible and (b) affine functions f k ( x ; w k )
= a k x + b k .
With these choices, a solution to (2.2) can be found in closed-form (Fisher, 2012), and the parametrization w results compact enough to be a valid compression code - only two floats for each f k in F , i.e. w k = ( a k , b k ) ∈ R 2 .
There are a number of limitations we aim to address:
• Fractal data encoding is only considered as an intermediate step towards compression.
We develop a learning-based approach to the solution of (2.2) for tasks beyond compression.
• Solving (2.2) on a collection of data as per (Fisher, 2012) is computationally expensive.
We directly solve a collection of fractal data encoding problems in parallel via hypernetworks (Ha et al., 2016), effectively amortizing the cost.
• As noted by (Welstead, 1999; Fisher, 2012), for an (affine) IFS to provide a satisfactory solution to (2.2), the self-similarity property has to be global across the set S .
That is, the entire set S is made up of smaller copies of itself, or a part of itself, property that is rather rare in natural data: indeed, most images are only self-similar to a degree.
To alleviate these restrictions, we develop Collage operators, a generalization of IFSs which can be broadly categorized as a soft- partitioned iterated function system (PIFS) (Jacquin et al., 1992).
Neural Collages
Moving forward, we treat Neural Collages algebraically.
This allows us to discuss in detail the properties of a Neural Collage , including algorithmic differences with other iterated function systems.
To do so we consider, instead of generic sets, data that can be expressed as simple ordered sets: in other words, as vectors.
Images will be our recurring example, with the understanding that the entire discussion can readily be adapted to other modalities e.g. sequences.
From generic sets to vectors Following the PIFS treatment of Øien and Lepsøy (1995), we focus our analysis on Collage composed of affine maps, operating on the space of discrete images of a given resolution with a total number m of pixels each taking values in R .
Pixels of different channels are treated without loss of generality as different elements.
This allows us to collect all m pixel values in an ordered 2 vector z ∈ R m .
A type of subsets on images involves the formation of square patches.
Let us assume that each image is partitioned into (1) K non-overlapping range cells and (
2) N possibly-overlapping domain cells .
A range cell R k is then of size n r,k × n r,k , such that m = (cid:80)
Kk =1 n 2 r,k , and domain cells D n are of size n d,n × n d,n .
With such coordinatization, the (affine) fixed-point map F reduces to a structured affine contraction on R m .
Definition 1 (Neural Collage Operator) .
Consider a m -pixel image represented by the ordered vector z ∈ R m .
Then, a Collage Operator is defined as the parametric linear map:
[Equation]
• S n ∈ R n 2 d,n × m selects a domain cell D n of n d,n × n d,n pixels.
• P k,n ∈ R n 2 r,k × n 2 d,n is a pooling operator that shrinks the domain cell D k into the size of the corresponding range cell R k ,
i
e. from n d,n × n d,n to n r,k × n r,k pixels; • T k ∈ R m × n 2 r,k positions the pooled domain cell in the correct range cell location and zeroes out the rest; • a k,n , b k,n ∈ R scales and translates the value in each pixel of the pooled domain cell, respectively.
• γ k,n ∈ R ; convex combination of affine outputs produced from all N domains.
The parameters w is the collection of all a k,n , b k,n and the mixing weights γ k,n .
Collage operators represent each range cell R k as a convex combination of pooled and scaled versions of all domain cells D n translated block-wise by b n .
On individual range cells comprising the output, a symbolic representation can be given as
[Equation]
In the generic set formulation, these maps correspond to functions f k of F .
Differently from a PIFS, each f k does not act on a different subset (domain cell) to produce R k .
Instead, all f k aggregate affine transformations – parametrized by a k,n , b k,n – on domains via γ k,n .
However, each f k is equipped with different mixing weights and different affine maps.
Hence, a Collage in this form can be seen as a soft-PIFS.
Introducing auxiliary domains
A Collage step maps mixtures of all domains to each range, and assembles the ranges into its output.
As Neural Collages are often optimized on datasets, rather than single data points, we posit that improvements in the expressiveness can be readily achieved by mixing additional dataset-level information through auxiliary domains U v .
Definition 2 ( Collage operator with auxiliary domains) .
[Equation]
In coordinates this translates to
[Equation]
where F w ( z ) is (3.1) and R k is (3.2) , and the parameters w include the coefficients γ k,v , a k,v of the auxiliary domains U v , as well as γ k,n , a k,n , b k,n .
We consider different variants of U v , including: deterministic transformations of domain cells D n e.g. rotations as per (Jacquin et al., 1992), learned cells directly parametrized and optimized for an objective, similar to feature maps in (Jaegle et al., 2021), and U v produced by a neural network encoder.
Specifics are provided in Section 4 .
A schematic of a single step of Collage is given in
Fig.
3.1 The Forward Problem: Collage to Data
Given a parametrization w for the Collage operator F w and an initial image z 0 , the attractor z ∗ : z ∗ =
F w ( z ∗ ) can be recovered by iterating the fixed-point map
[Equation]
assuming F to be a contraction w.r.t.
the standard Euclidean metric on R m .
This can be ensured by an appropriate choice of the coefficients a k,n , γ k,n .
In particular, if all the mixing weights are such that (cid:80) Nn =1 γ k,n = 1 and | a k,n | < 1 , then contractivity of the collage operator follows as in standard
PIFS (see e.g. Fisher (2012)).
Note that the attractor of a Collage can be also computed in closed-form as z ∗ =
[ I − A ( w )] − 1 b ( w ) .
A similar discussion follows for a general Collage with auxiliary domains.
Note that auxiliary domains U across iterations t are to be chosen such that the sequence { U t } ∞ t =0 converges e.g. constant functions.
Pseudocode for a single step of a Collage is shown below.
Figure 2 provides a visualization of the convergence of a Collage to its fixed-point (after repeated application of the operator).
[Table]
return z
Decoding at higher resolutions A Collage can be applied, without change, to images of different resolutions.
Consider scaling the resolution by a factor s > 1 , ( s ∈ N ) .
Then the magnified image representation is made up of s 2 images z i ∈ R m .
The Collage operator can then be thought to act singularly on each z i obtaining a forward fixed-point iteration
[Equation]
When solving by unrolling the fixed-point iteration, the second term can be precomputed as it does not depend on z t .
3.2 Amortized Solution of the Inverse Problem
[Figure]
Although the CT suggests a constructive procedure via (2.2) to find a valid fractal representation w , there are no guidelines in case other objectives are of interest.
Further, the class of
PIFS – without modifications – does not lend itself well to numerical optimization as it involves the combinatorial problem of matching domains to ranges.
With their soft aggregation, Neural
Collages can instead be used for task-based optimization.
Given an input image x ∈ R m , we can optimize the parameters w (and pixel values u of the auxiliary domains) of the Neural Collage operator to minimize an objective J ω ( x, z ∗ ( w, u )) by solving a nonlinear program min w,u J w ( x, z ∗ ( w, u )) with z ∗ ( w, u ) obtained by the fixed point iteration on ˆ F ( z, u ; w ) .
Choosing
J w to be a reconstruction objective yields a problem similar to fractal data encoding (2.2).
Suppose instead to be given an image dataset whose distribution p is known only through i.i.d.
samples x ∈ R m , x ∼ p ( x ) .
In this context, the fractal data encoding problem in standard form needs to be solved for each sample x .
Instead, we introduce an hypernetwork (Ha et al., 2016)
E with weights θ , generating a fractal code w given an input image x , i.e. parametrizing a map x
(cid:55)→ w θ ( x ) ,
[Equation]
∼ p
[Equation]
The hypernetwork is trained to solve the following empirical risk minimization problem, effectively amortizing the cost over the full dataset:
[Equation]
Collages in Learning Tasks
x
[Equation]
decoder
Collage
[Equation]
u
We showcase applications of Neural Collages as decoders for deep generative models, as neural compressors for images and as a method to generate fractal art.
All variants of the model share a common structure, outlined in Figure 3.
The code is available at github.com/ermongroup/ self-similarity-prior .
4.1 Generative Neural Collages
We investigate application of Neural Collages as deep generative models for distributions p ( x ) of images.
In this context, w assume the role of latent variables.
In particular, we consider a hierarchical variational autoencoder (VDVAE) (
Magnifying samples via Collage VAEs VAE models seek to data x into a latent representation w such that the following lower bound ( ELBO ) on data log-likelihood be maximized
[Equation]
where approximate posterior q ϕ ( w | x ) , prior p ϕ ( w ) , and generator p θ ( x | w ) are implemented as neural networks.
A multiplicative hyperparameter β is often introduce to control the relative weight between
VDVAE
Collage
[Figure]
28 × 28)
[Figure]
rate and distortion (Higgins et al., 2016).
In a Collage VAE, q ϕ ( w | x ) and p ϕ ( w ) are parameterized as per (Child, 2020), except the approximate posterior q ϕ ( w | x ) need not produce large feature maps but scalar Collage codes w .
Moreover, p θ ( x | w ) is a Collage such that by leveraging the consideration of Section 3.1, samples can be decoded at any resolution.
Results To investigate the properties of Collage VAEs, including quality of magnified samples, we compare VDVAEs and Collage VAEs on dynamically binarized MNIST.
We report both test rate and distortion, following the analysis of (Alemi et al., 2018).
This supports a more detailed evaluation of model behavior at different β weights for the rate.
We train several Collage VAEs and
VDVAEs with β ranging in [0 . 5 , 0 . 7 , 1 . 0 , 1 . 2 , 1 . 5] , and report the rate-distortion curve in Figure(right).
Collage VAEs are only marginally Pareto suboptimal, but are shown to be less sensitive to training under different β , which is a common strategy employed to stabilize VAE training ("KL warmup").
Furthermore, Collage VAEs can generate samples at resolution unseen during training, as showcased in Figure 4 (left).
Direct samples at a magnification factor of 40 × reveal additional details over VDVAE samples that are magnified via bicubic interpolation.
We note that as discussed in Section 5.1, the detail introduced by magnification is entirely dependent on the Collage class; future designs may be developed to introduce types of detail expected in a given dataset.
4.2 Neural Collage Compressors
[Table]
baselines (baseline fractal implicit, spectral) and Neural Collages compressors.
Neural Collages introduces less visible artifacts than other self-similarity or implicit compression schemes, narrowing the gap with spectral compression methods such as JPEG.
Next, we apply Neural Collage to image compression.
We store images as the parameters w of an affine Collage produced by a neural encoder.
After training, the encoder can be used to compress additional images in parallel with a single forward pass by producing the corresponding w .
Figure 3 provides an overview of the computation done by a Neural Collage compressor.
Collage compressors employ learned feature maps as auxiliary domains.
For compression, the main desideratum is visual fidelity: regular domains D n allow Neural Collages to capture intra-image self-similarity, whereas auxiliary domains U optimized for image quality complement them by focusing on inter-image patterns.
Compression of high-resolution aerial images We consider compressing images obtained from the DOTA large-scale aerial images dataset (Xia et al., 2018).
From the DOTA training set, we produce 80000 random 40 × 40 crops as our training dataset.
We optimize convolutional encoder parameters θ and auxiliary sources on the reconstruction objective J ( x, z ∗ ( θ, u ))
= (cid:80) mi =1 ( x i − z ∗
i ( θ, u )) 2 + ∥ w ∥[-
-]
The model is trained on the 40 × 40 crops and evaluated on 10 held-out 1200 × 1200 images.
This is possible as an image of any resolution can be first broken up into blocks of appropriate size, in this case the training resolution, 40 × 40 , passed through the encoder to obtain the corresponding parameters, then concatenated to construct a valid code for the entire image.
This operation can be performed in parallel by treating each block as an element of a pseudo-batch.
Neural Collage compressor can thus be used to compress images of any resolution at test time, without retraining.
Results We compare peak signal-to-noise ratio (PSNR), in addition encoding and decoding measurements for a variety of compression baselines.
We contextualize our results with comparisons to both non-neural as well as another implicit neural compressor.
In particular, we evaluate the performance of a standard PIFS-based fractal compression as per (Jacquin, 1993), implemented to exploit parallelization on GPU, and COIN (Dupont et al., 2021).
Fractal compression baselines and Collage both use non-adaptive tiling partitioning schemes.
We evaluate two variants of fractal compression, one where domain cells are augmented via rotations and color flips (Welstead, 1999), and one without augmentations.
We further compare with block-DCT, the spectral lossy compression backbone of most JPEG codecs.
All compression methods are evaluated at low and medium bpps, with metrics provided in Table 1.
Figure 5 provides a visual comparison of images obtained by decoding the lossy code of each method.
Neural Collages show less noticeable artifacts and improved color retention.
Block
Fractal
[Figure]
Wall-clock time (s) measurements of compression methods applied to the held-out set of
DOTA. For neural compressors, we report encoding times including training, as well as encoding times on unseen images (after training).
At test time, encoding for Neural Collage compressors is 100 × faster than fractal compression.
Finally, Table 1 provides wall-clock time measurements of all methods during the respective per image encoding and decoding procedures.
Results for COIN and Neural Collage measure encoding times (including the training procedure), and encoding times after training.
Neural
Collages are orders of magnitude faster than fractal compression and, at test-time, of COIN.
Although spectral lossy compressors common in state-of-the-art codecs perform with best PSNR in medium and high bpp settings, Collages narrows the gap in terms of reconstruction quality as well as encoding speed.
Code length of a Collage The total bits-perpixel (bpp) cost of the code w corresponding to the parameters of a Collage depends on the coding scheme used to store each numeric entry.
We exploit bounds on a, b, γ , enforced via tanh and softmax, as well as regularization to reduce the total cost.
The L 2 regularization term on w is introduced to shrink the range of values assumed by elements of w , ensuring that less bits can be used for storage.
We do not use lossless coding schemes to store parameters of Collages and other baselines.
Further details are provided in the Appendix.
4.3 Neural Collages for Fractal Art
When data is represented through the parameters w of a Collage , it can be arbitrarily magnified by decoding at any resolution.
The type of patterns revealed through magnification need not be corresponding to real detail missing from the image.
In particular, the patterns found depend on how one generates domains, auxiliary domains and class of operator F w .
Similar phenomena have been observed in the literature of fractal compression (Mitra et al., 2000).
As an example, consider Figure 7, where the fractal pattern of snowflakes within snowflakes does not correspond to reality.
We call this type of globally self-referential magnification as fractalization of an image.
Here, we use Neural
Collage fractalizers to generate fractal art.
Experimental details We solve the inverse problem of a Collage , namely image x to w , via a convolutional architecture parametrized by θ .
The objective of the inverse problem (3.3) is a reconstruction objective J ( x,
z ∗ ( θ, u ))
= (cid:80) mi =1 ( x i − z ∗
i ( θ, u )) 2 where z ∗ is the fixed-point of the
Neural Collage with parameters w computed through the encoder w =
E θ ( x ) .
We choose the single domain D to be the entire image.
Before applying F w , we augment the domain via rotations of itself, utilizing those as auxiliary domains.
Figure 1 and 7 show example fractalizations possible with
Neural Collages , on greyscale and RGB images.
The images can be magnified to any resolution (up to memory limits), revealing multiple fractal levels.
Related Work and Conclusion
Implicit representations and models Repre sentation of data implicitly through functions is extensively used in simulation (Osher et al., 2004).
(Sitzmann et al., 2020; Mildenhall et al., 2020; Dupont et al., 2021) parametrize the implicit functions via neural networks for use in downstream tasks such as compression.
Implicit models, on the other hand, solve optimization problems within their forward pass (Poli et al., 2020).
Neural Collages belong to both classes of methods: a fixed–point iteration with parameters defining data implicitly.
In particular, Neural Collages can be framed as a compactly– parametrized operator variant of deep equilibrium networks (DEQs) (Bai et al., 2019), with parameters produced by a hypernetwork (Ha et al., 2016).
Fractal compression The idea of representing images through iterated function systems (IFS) dates back to (Barnsley and Demko, 1985; Barnsley, 1986).
Jacquin et al.
(1992) introduces more flexible fractal compression schemes for images based on partitioned iterated function systems (PIFSs).
Since then, alternative partitioning schemes i.e. adaptive quadtrees have been proposed.
We refer to (Fisher, 2012) for an overview of the main variants.
Additional references are provided in the Appendix.
Conclusion This work builds a framework for a learning–based approach to automated discovery of self–similarity.
We introduce Neural Collages , implicit models designed to represent and manipulate data as the parameters of a structured fixed–point iteration, and showcase their application to compression and deep generative modeling of images.
We envisage future use of Neural Collage with other data modalities with naturally occurring self–similarity, such as audio, sequences, or turbulent flows.
Acknowledgments
This work is supported by NSF (1651565), AFOSR (FA95501910024), ARO (W911NF-21-1-0125),
ONR, DOE, CZ Biohub and Sloan Fellowship.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix B. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Self
Supplementary Material
A Background and Extended Formulation A.1 Metric Spaces
Lemma 1 (Useful results on bounded and closed sets) .
The following hold:
i. Let X =
(cid:83) ni
=1 A i such that A i is a closed subset of R n for all i = 1 , . . . , n .
Then, X is closed .
ii.
Let X =
(cid:83) ni
=1 A i such that A
i is a bounded subset of R n for all i = 1 , . . .
, n .
Then, X is bounded .
iii.
Let f : R n → R n to be a continuous function and let A to be a closed, bounded subset of R n .
Then f ( A ) =
{ f ( x ) : x ∈ A } is also closed and bounded.
Definition 3 (Metric space) .
A metric space is a pair ( X , d ) where X is a set and d : X × X
→ R is a map such that for all x, y, z ∈ X the following conditions hold:
i. d ( x, y ) ≥ 0; iii.
d ( x, y ) =
d ( y, x ); ii.
d ( x, y ) = 0; ⇔ x = y iv.
d ( x, y ) ≤ d ( x, z ) + d ( z, y ) .
The function d is called “metric”.
Note that a metric space is called compact if X is closed and bounded.
Definition 4 (Convergent sequence) .
Given a metric space ( X , d ) , a sequence { x t } ∞ t =0 is said to converge to some x ∗ ∈
X ∀ ϵ >
∃ t ∗ : ∀ t > t ∗ d ( x ∗ , x t ) <
ϵ.
Definition 5 (Cauchy sequence) .
A sequence { x t } ∞ t =0 is X is a Cauchy sequence if
[Equation]
Definition 6 (Complete metric space) .
A metric space ( X , d ) is complete if every Cauchy sequence in X is convergent in X .
Definition 7 (Hausdorff space) .
Let ( X , d ) be a complete metric space, and define H ( X ) as the set of all compact subsets of X :
[Equation]
compact
[Equation]
Definition 8 (Hausdorff metric) .
Let ( X , d ) be a metric space and let Y , Z ⊂ X .
The Hausdorff metric d H : H ( X ) × H ( X ) → R is then defined as
[Equation]
where d ( y, Z ) := inf z ∈ Z d
( y, z ) .
Theorem 2 (Completeness of Hausdorff metric space (Barnsley and Hurd, 1993)) .
Let ( X , d ) be a complete metric space.
Then ( H ( X ) , d H ) is a complete metric space.
(Barnsley and Hurd, 1993) calls ( H ( X ) , d H ) the space where fractals live .
It is the space where the mathematical foundations necessary to generate fractals via iterated function systems are developed.
A.2 Contraction mappings
Definition 9 (Lipschitz function) .
Let ( X , d ) be a metric space.
A map f : X → X is Lipschitz with constant ℓ if there exists ℓ > 0 such that
[Equation]
A Lipschitz function is then called contractive iff ℓ < 1 .
Moreover, if f : X → X is Lipschitz, then f is continuous.
Note: A map f is contractive if brings any elements of X close together .
The Lipschitz constant ℓ measures (bounds) how closer the points are brought together by one application of f .
Thus, intuitively, if we iterate a discrete dynamical system
[Equation]
starting from any point of x 0 within X , the sequence { x t } ∞ t =0 will converge to a unique fixed point x ∗ such that f ( x ∗ ) = x ∗ .
It is worth to notice that linear affine maps f : R n → R n (where we consider the standard
Euclidian distance to form a complete metric space on R n )
[Equation]
are contractive if and only if
[Equation]
If this is the case any sequence { x t } ∞ t =0 would converge to a fixed point
[Equation]
The above intuitions can be formalized in the following classic result
Theorem 3 (Banach fixed-point theorem) .
Let ( X , d ) be a (non-empty) complete metric space and let f : X → X be a contractive map.
Then f admits a unique fixed point x ∗ ∈
X such that any sequence { x t } ∞ t =0 defined by the iteration
[Equation]
converges to x ∗ for any starting point x 0 ∈ X , i.e.
[Equation]
Corollary 2 (Collage theorem) .
Under the assumptions of Theorem 3, it holds
[Equation]
A.3 Iterated Function Systems
With the aim of deriving fractal compression algorithms it is necessary to define functions on the
Hausdorff metric space ( H ( X ) , d H ) .
In particular, let { f 1 , f 2 , . . .
, f K } be a collection of maps on X , f k : X → X .
Then, we can define maps F : H ( X ) → H ( X ) by
[Equation]
where f k ( A ) is intended as f k ( A )
= { f k ( a ) : a ∈ A } .
Moreover, note that A ∈ H ( X ) is also a compact subset of X .
The following results shows that if all f k are contractive, then also F is.
Theorem 4 (Contractivity of maps on the Hausdorff metric space) .
If for all k = 1 , . . .
, K , the maps f k : X → X are contractive with Lipschitz constant ℓ k < 1 , then F : H ( X ) → H ( X );
A (cid:55)→ (cid:83)
Kk =1 f k ( A ) is contractive in the Hausdorff metric with Lipschitz constant L = max k { ℓ k } k .
Definition 10 (Iterated function system (IFS)) .
An iterated function system is a collection { f 1 , . . .
, f K } of contractive maps f k : X → X , represented by F : H ( X ) → H ( X ) .
Thanks to its contractivess (as established by Theorem 4), F define a unique fixed point ( attractor ) by the Banach fixed-point theorem (Theorem 3), i.e. the discrete iteration defined by
A t +1 =
F ( A t ) =
K (cid:91) k
= f k ( A t ) converges to A ∗ ∈ H ( X ) for t → ∞ .
Since the attractor A ∗ is unique, it is completely defined by the map F .
The data encoding problem can be then formulated as follows.
Problem: Fractal Data Encoding (Section 2.2)
If we are given some set S ∈ H ( X ) (our data ), can we find map F whose attractor is S ?
In other words, given data S ∈ H ( X ) , find the collection of maps f k : X → S such that the following conditions hold
[Equation]
f k ( A ) is contractive ; ii.
S is the fixed point of F, S = F ( S ) =
K (cid:91) k
= f k ( S );
Properties of data encoding
Note that condition ii.
suggests that, in order for the problem to admit a solution, data should be made up of transformed copies of itself.
Specifically, we are assuming that it is possible to take the data S , copy it K -times, apply to the copies some contractive transformations and finally stitch them together to reconstruct the initial data S .
The uniqueness of an attractor induced by the contractivity of F is fundamental to practically solve the fractal encoding problem because if we can find an F such that S = F ( S ) , then we will be sure that F is the unique solution of the encoding problem.
On the Collage Representation By applying the Collage Theorem (Corollary 2) to F using the
Hausdorff metric d H we have
[Equation]
This means that if we can’t stitch the transformed copies f k ( S ) together to perfectly reconstruct the data S , i.e. d H
[Equation]
then the lower the Lipschitz constant L of the IFS is, the lower the distance between the data S and the attractor A ∗ of S will be given a mismatch d H ( S , F ( S )) .
As mentioned in the main text, this implicitly promotes the use of “very contractive” maps f k (i.e. with low ℓ k ).
A learning perspective to the fractal data encoding In the language of machine learning practice, the fractal data encoding problem can be translated into finding a parametric representation f k ( · ; w k ) , w ∈ R n w for the functions f k ( · ) (e.g. Neural Networks with parameters w k ) where the parameters w = ( w 1 , . .
. , w K ) ∈ W are trained to minimize the Hausdorff metric loss function d H ( S , F ( S ; w )) naturally induced by the Collage Theorem , i.e. min w d H ( S , F ( S ; w )) subject to F ( S ; w ) =
K (cid:91) k
= f k ( S ; w k ) w ∈ W
Once optimal f k ( · ; w k ) are computed, it is easy to find the data that F encodes (i.e. the decoding process): after sampling any initial condition A 0 , the encoded data can be obtained by iterating
A t +1 =
F ( A t ) , until convergence to A ∗ ≈ S .
A.4 Partitioned Iterated Function Systems
Existance of solutions of the fractal encoding (inverse) problem requires data to be perfectly representable by an IFS.
Conversely, we can define self-similar data if it is the attractor of an IFS.
Definition 11 (Self-similar sets) .
A set S is called self-similar if and only if there exists a contractive map F : H ( X ) → H ( X ) whose attractor is S , i.e. S = F ( S ) .
While verifying the self-similarity of a specific data point is undoubtedly a NP-hard problem, natural data (e.g. in an image dataset) is unlikely to satisfy this strict property.
The challenge is that the self-similarity property has to be global across the set S .
That is, the entire set S has to be made up of smaller copies of itself, or parts of itself.
If one zooms in on it, it would display the same level of detail, regardless of the resolution scale (Welstead, 1999).
For this reason, it is necessary to extend the fractal encoding to more general, non globally self-similar sets.
This can be achieved by introducing the technology of partitioned function systems (PFS) where the domains of the contraction maps f k are restricted.
Definition 12 (Partitioned Function System) .
Let ( X , d ) be a complete metric space and let D k ⊂ X for k = 1 , . . .
, K .
A partitioned function system is a collection of contraction maps f k : D k → X .
Note that, according to Fisher (2012), it is not possible to extend Theorem 3 to PFSs in the general case to effectively ensure existance and uniqueness of fixed points.
Intuitively this is due to the fact that the domains of f k are restricted and the convergence of the decoding dynamics (i.e. the fixed-point iteration)
[Equation]
becomes dependent on the choice of the initialization A 0 .
In fact, even though if choose A 0 ⊂ (cid:84)
Kk =1 D k , after one step we may end up with an empty set.
Note that this is generally not a problem in practice when applying PFSs to or Collage working with common type of data such as images or audio signals.
A.5 Functional Representation of Data
In order to derive an implementation-oriented formulation of data encoding with partitioned functions systems in the general case, it can be convenient to rely on a functional description of data (see e.g.
(Welstead, 1999; Fisher, 2012)).
In example, images (of infinite resolution) can be represented as functions from the unit square to R .
Time series can also be thought as real continuous functions over a compact time domain.
Specifically we restrict our analysis to the space F = { ϕ : dom ( ϕ ) → R } of data defined the graphs ( z, ϕ ( z )) , z ∈ dom ( ϕ ) of (measurable) functions over the compact domain dom ( ϕ ) and values in R .
dom ( ϕ ) is assumed to be a compact subset of R n .
Partitioned fractal encoding a la Welstead (1999)
We choose F = L p ( X ; R ) with X a compact subset of R n
and we equip it with a metric d F induced by the Lebesgue measure
[Equation]
Then ( F , d F ) is a complete metric space and the Banach fixed-point (Theorem 3) holds.
Then, we specialize the partitioned function system on ( F , d F ) as comprised of the following collections of K elements: a. sub-
[Equation]
b
[Equation]
invertible contractive
[Equation]
k
[Equation]
k
[Equation]
c
[Equation]
f k : F → F defined
[Equation]
Note that we can define subsets R k as the range of v k operating on D k , i.e. R k
= v k
( D k ) .
The constants c k , d k realize an affine trasformation on ϕ by expanding/contracting and shifting the range of ϕ .
The contractive maps v k are the “spatial part” of the PFS and map the domains D k to their respective ranges R k .
v k are often chosen to be affine maps
[Equation]
Note that it is possible to choose A k and c k so that f k is contractive.
In particular, it is sufficient to require | c k || det A k | 1 /p <
Definition 13 (Tiling partition) .
A collection of ranges R k ⊂ X is said to tile X iff X
=
(cid:83) Kk =1 R k and ∀ i ̸
=
j R i ∩ R j = ∅ .
If the ranges R k tile X , we can define the operator F : F → F by
F ( ϕ )( x ) =
f k ( ϕ )( x ) for x ∈ R k , i.e.
[Equation]
Since the ranges R k tile X , F is defined for all x ∈ X , so F ( ϕ ) is a function of the same class of ϕ .
If ϕ is an image on the unit square tiled by the ranges R k , then F ( ϕ ) will also be an image on the unit square.
Assuming all the maps f k to be contractions on F , F satisfies the Banach fixed point theorem and has unique fixed point ϕ ∗ ∈ F such that
[Equation]
Collage and PIFS for digital images Similarly to the main text, we can define PIFS by restricting our analysis to affine maps, operating on the space of discrete images of a given resolution with a total number m of pixels.
Note that pixels across channels can be treated effectively as different elements.
We assume the value of each pixel to range in R and to collect all the pixel values in an ordered 3 in a vector z ∈ R m .
Then, a partitioned function system (Jacquin, 1993; Welstead, 1999; Fisher, 2012) can be represented as the structured map
Definition 14 (Discrete PIFS) .
Consider a m -pixel image represented by the ordered vector z ∈ R m .
Then, a Discrete PIFS is defined as the parametric linear map F : R m → R m :
[Equation]
where T k , P k , S k are defined similarly to Definition 1.
Encoding
Decoding
Input: Image I , parametrized PIFS F ( · ; w ) .
Partition I into two collections of N domains D n and K ranges R k .
for k from 1 to K do
D ∗ n , w ∗ k = arg min D n ,w k d ( f k ( D n ; w k ) , R k ) matching end for
Store code := { w ∗ k , ( n, k ) }
Kk = fractal code and domain index
Initialize ˜ I 0 as any image with same size as I repeat ˜
I t +1 =
K (cid:91) k
= f k
( D ∗ n,t ; w ∗ k ) until convergence return ˜
I ∗ decoded data
The output of the collage operator for R k is thus a pooled and scaled version of D k translated block-wise by b k .
A symbolic formulation of the collage operator can be also given by
[Equation]
Since the collection of range cells R k tiles the whole image I , we can write ( a la IFS)
[Equation]
Remark 1 (Extensive search) .
Note that, in the classic setting of Jacquin (1993); Welstead (1999), for each range cell R k , the corresponding domain cell D k has to be found by extensive search through the set of all possible pooled domain cells.
We provide a compact algorithmic summary of the core steps in fractal compression as per Jacquin et al.
(1992); Jacquin (1993) in Figure 8.
Other variants of fractal compressions have historically been attempted, including ones with adaptive partitions and different algorithms to solve the combinatorial search during encoding.
B Additional Details
Code-length flexibility: Collage and PIFS A Collage operator is a generalization of PIFS operators of fractal compression algorithms (Jacquin et al., 1992; Jacquin, 1993; Fisher, 2012).
In particular, the Collage introduces additional flexbility in the choice of code length i.e. how many bits to allocate to the compression code.
For example, given non-adaptive (square) tiling domain and range cells 4 , the bits-per-dimension (bpp) cost of PIFS-based fractal compression of Jacquin et al.
(1992) is cost wk n 2 r,k , where cost w k is the cost of saving the parameters of single element in the f k of
F w .
Here, other than seeking further compression and reduction of cost w k by modifying the class of operator, the only degree-of-freedom is to reduce or increase the dimensions of tiling partitions.
Note that modifying the partition scheme has not only effect on the bpp cost but also on the type of self-similarity that can be captured.
Instead, Collage operators offer an additional design axis; indeed, the bpp budget – given a fixed partition – can be modified by increasing or decreasing the number of auxiliary domains introduced as learnable feature maps.
This number is independent on the number of original domains, whereas the number of additional domains generated as affine augmentations of fractal compression (see Welstead (1999); Fisher (2012) for details) is not.
On adaptive partitions Elaborated partitioning schemes have been developed for PIFS-based fractal compression methods (Fisher, 2012).
While the analysis and empirical comparisons of this work have been centered around the operators, rather than partition schemes, we remark that Collage are compatible with alternative and potentially adaptive schemes.
Much like for PIFS, this is a likely direction for further improvement of Neural Collage .
B.1 Collage Schematic Collage
[Figure]
3) F produces all range cells, each following (3.2) (4) T rearranges the output.
B.2 Extended related work
Attention operators and patches There exist superficial similarities between Collages and attention operators (Vaswani et al., 2017).
In particular, recent variants of vision transformers (Dosovitskiy et al., 2020) where attention acts on square patches, can be seen as a single step of a Collage , where source and target partition match and aggregation weights are found via similarity scores.
Collages differ from attention in that they are structured fixed-point iterations, are built to accommodate non-overlapping partitions, are resolution-invariant, and have a compact parametrization that can be used as a compression code.
It remains to be seen whether investigating attention operators through the lenses of Collages can yield improvements in theoretical understanding or performance.
Fractal compression Sun et al.
(2001) parametrize elements of the iterative map with small neural networks.
The proposed method still requires training on each image, with marginal improvements over standard variants.
(Guido et al., 2006) provide a preliminary exploration of fractal coding for audio.
Despite the extensive body of work, fractal methods for image compression are rarely used in place of other codecs due to slow encoding.
As discussed in the main text sections, Neural
Collages address this limitation via neural network amortization.
We note that Neural Collage remain compatible with adaptive partitioning schemes, which provides a likely avenue of further improvement.
We highlight a line of work on different probabilistic models of self-similarity (Zha et al., 2020) for tasks such as image restoration.
C Additional Experiment Details
Hardware and software The experiments have been performed on a workstation with 2 NVIDIA
G E F ORCE RTX 3090 GPUs.
We use JAX 5 for model implementation and distributed training.
C.1 Neural Collages for Fractal Art
We construct the encoder E for w by stacking 4 blocks composed of interleaved depthwise and pointwise convolutions.
We train for 5000 iterations on each image displayed in Figure 7 with
AdamW (Loshchilov and Hutter, 2017).
We produce U by augmenting at each step of the Collage with rotations of [90 , 180 , 270] degrees and flips, produced by multiplying all pixels values of a domain cell by − 1 .
We do not use any additional learned auxiliary domain, so that the patterns can be kept globally fractal.
Neural Collage Texturizers
We report additional results in D.3, where a Neural Collage is used to texturize images by optimizing transformations of a fixed U , provided as external "texture source".
To promote utilization of texture sources we introduce a coefficient to weigh U relative to D in the
Collage iteration.
We note that in this case the Neural Collage is not leveraging any self-similarity; rather, this should be intended as a display of the capability of a Neural Collage to aggregate both external as well as self-referential information to achieve a given task.
C.2 Neural Collages for Generation
We design the architecture of a Collage following (Child, 2020).
Table 3 describes the model structure.
We introduce 30 learned auxiliary domains, parametrized to be pixel patches of same size of range cells.
As the tiling partition, we choose a single domain cell of size 28 × 28 and size 4 range cells of 14 × 14 .
All models use a Bernoulli likelihood.
Note that in this case, the fixed-point of the generator p ( x | w ) , chosen as a Collage , is given by the collection of all Bernoulli parameters, one for each pixel.
This is thus an example of a Collage that is does not decode pixel-values of an image as its fixed-point, but rather parameters of their distributions.
We optimize the ELBO by sweeping the
KL weight β as discussed in Figure 4 for 2000 epochs.
Additional training details are provided in 4.
C.3 Neural Collages for Compression
We construct the encoder E for w by stacking 4 blocks composed of interleaved depthwise and pointwise convolutions.
We train for 5 epochs with AdamW Loshchilov and Hutter (2017) on a dataset of 8000 crops of size 40 × 40 obtained from the DOTA Xia et al.
(2018) aereal image training dataset.
The dataset is generated (statically) randomly by applying a random rotation, followed by a random crop.
We produce the 10 held-out images of size 1200 × 1200 with a similar procedure, applied to the test dataset.
We note that DOTA images are all of different resolutions, motivating the above procedure.
Speedup results are provided in Figure 12.
As baselines, we use the official COIN Dupont et al.
(2021) implementation.
We develop a GPUparallel version of fractal compression with PIFS Jacquin et al.
; Jacquin (1993); Welstead (1999); Fisher (2012) as a baseline.
We use the same partition strategy as for Neural Collages , namely tiling into domain and range cells.
Our evaluation includes a fractal compression variant which incorporate U by augmenting domains, D at each step, with rotations of [90 , 180 , 270] degrees and flips, produced by multiplying all pixels values of a domain cell by − 1 .
The matching problem of domains to ranges is solved via least-squares as per Welstead (1999).
We parallelize the least-square solving across domains.
C.4 Computation of bits-per-pixel
We report the per-image bits-per-pixel (bpp) cost of compression baselines and Neural Collage compressors.
[Table]
Neural
Computation of the bpp of fractal codes generated by a Neural Collage compressor requires the following considerations.
First, mixing weights γ k,n are premultiplied to both a k,n and b k,n .
The same holds for mixing weights of auxiliary domains.
We store a single b k for each R k , noting that the corresponding term can be precomputed as b k = (cid:80) Nn =1 γ k,n b k,n .
Further, we exploit the a priori knowledge that a, b ∈ [ − 1 , 1] , enforced via tanh , in combination with significant digit clipping.
Quantizing by clipping to a threshold ϵ of significant digits, in combination with the bounds enforced by tanh , allows bit-packing each into less than ⌈ log 2 10 ϵ ⌉ + 2 .
One of the 2 additional bits is for sign information.
This can be verified by noticing that by quantizing the range of values, after multiplying by 10 ϵ , is contained by the integer range [ − 10 ϵ , 10 ϵ − 1] .
In practice, the number of bits is less than ⌈ log 2 10 ϵ ⌉ + 2 since not all values in entire integer interval defined by ϵ -quantization are utilized by a n,k and b k for a given image.
In particular, we consider maximum absolute values of the quantized values, and restrict the interval accordingly.
Neural Collage do not require storing of domain cell addresses, since each map of the Collage corresponding to a range cell (see 3.2) always transforms all domains.
The specification of patchsizes, especially when they are the same across all domains, and across all ranges, as well as type of pooling operators can be considered part of the codec, adding a negligible amount of bits.
Further considerations are necessary in one wishes to employ more elaborate partition schemes (Fisher, 2012).
The overall cost is given by
[Equation]
where N is the number of domains, V is the number of learned auxiliary cells, and K the number of ranges.
We indicate with bpp u the cost of saving auxiliary learned patches.
This cost is amortized across each image of the held-out set, as the patches are the same for a given Neural Collage .
For V auxiliary patches of size h × w × c , the (non-amortized) bit cost is 32 · V · h · w · c bits.
We provide some example calculations in 2.
COIN We use the official implementation of (Dupont et al., 2021), where the bpp is computed by serializing the weights of the network into a bytes.
Fractal compression baseline We quantize fractal compression affine maps f k into half-floats,bits for each a k and b k .
As the address of the domain cell associated to a given range R k , we store the index with cost ⌈ log 2 ( N + V ) ⌉ -bit, where N is the number of source domains D and V the number of auxiliary domains U .
block-DCT We apply a forward, two-dimensional discrete cosine transform (DCT) to patches of sizes 12 (high bpp) and 16 (medium bpp) and filter all but the lowest coefficient.
The total cost is thus 32 · n patches .
The image is decoded by applying an inverse DCT.
A RCHITECTURE
[Table]
EARNING
[Table]
[Table]
D Additional Results D.1 Super-Resolution of
[Figure]
D.2 Compression
[Figure] [Figure]