G ENERALIZED RECTIFIER WAVELET COVARIANCE MODELS FOR TEXTURE SYNTHESIS
A BSTRACT
State-of-the-art maximum entropy models for texture synthesis are built from statistics relying on image representations deﬁned by convolutional neural networks (CNN).
Such representations capture rich structures in texture images, outperforming wavelet-based representations in this regard.
However, conversely to neural networks, wavelets offer meaningful representations, as they are known to detect structures at multiple scales (e g. edges) in images.
In this work, we propose a family of statistics built upon non-linear wavelet based representations, that can be viewed as a particular instance of a one-layer CNN, using a generalized rectiﬁer non-linearity.
These statistics signiﬁcantly improve the visual quality of previous classical wavelet-based models and allow one to produce syntheses of similar quality to state-of-the-art models, on both gray-scale and color textures.
We further provide insights on memorization effects in these models
I NTRODUCTION
Textures ares spatially homogeneous images, consisting of similar patterns forming a coherent ensemble.
In texture modeling, one of the standard approaches to synthesize textures relies on deﬁning a maximum entropy model (Jaynes, 1957) using a single observed image (Raad et al., 2018).
It consists of computing a set of prescribed statistics from the observed texture image, and then generating synthetic textures producing the same statistics as the observation.
If the statistics correctly describe the structures present in the observation, then any new image with the same statistics should appear similar to the observation.
A major challenge of such methods resides in ﬁnding a suitable set of statistics, that can generate both high-quality and diverse synthetic samples.
This problem is fundamental as it is at the heart of many texture related problems.
For example, in patch re-arrangement methods for texture modeling, these statistics are used to compute high-level similarities of image patches (Li & Wand, 2016;
Raad et al., 2018).
Such models are also used for visual perception (Freeman & Simoncelli, 2011; Wallis et al., 2019; Vacher et al., 2020), style transfer (Gatys et al., 2016; Deza et al., 2019) and image inpainting (Laube et al., 2018).
A key question along this line of research is to ﬁnd what it takes to generate natural textures.
This problem was originally posed in Julesz (1962), in which the author looks for a statistical characterization of textures.
In the classical work of Portilla & Simoncelli (2000) (noted PS in this work), the authors presented a model whose statistics are built on the wavelet transform of an input texture image.
These statistics were carefully chosen, by showing that each of them captured a speciﬁc aspect of the structure of the image.
This model produces satisfying results for a wide range of textures, but fails to reproduce complex geometric structures present in some natural texture images.
Figure 1 presents a typical example composed of radishes, and synthetic images from three state-of-the-art models developed over the last few decades.
To address this problem, the work of Gatys et al.
(2015) proposes to use statistics built on the correlations between the feature maps of a deep CNN, pre-trained on the ImageNet classiﬁcation problem (Deng et al., 2009; Simonyan & Zisserman, 2014).
While this model produces visually appealing images, these statistics are hard to interpret.
The work of Ustyuzhaninov et al.
(2017) made a signiﬁcant simpliﬁcation of such statistics, by using the feature maps of a one-layer rectiﬁer CNN with random ﬁlters (without learning).
A crucial aspect of this simpliﬁcation relies on using multi-scale ﬁlters, which are naturally connected to the wavelet transform.
In this paper, we propose a wavelet-based model, more interpretable than CNN-based models (with learned or random ﬁlters), to synthesize textures with complex geometric structures.
It allows to bridge the gap between the classical work of Portilla & Simoncelli (2000), and state-of-the-art models.
Observation
PS VGG RF This model is built on the recent development of the phase harmonics for image representations and non-Gaussian stationary process modeling (Mallat et al., 2020; Zhang & Mallat, 2021).
The phase harmonics are non-linear transformations that adjust the phase of a complex number.
In Portilla & Simoncelli (2000); Zhang & Mallat (2021), the authors illustrate that the phase dependencies between wavelet coefﬁcients across scales contain important information about the geometric structures in textures and turbulent ﬂows, and that they can be captured by applying the phase harmonics to complex wavelet coefﬁcients.
Remarkably, Mallat et al.
(2020) show that the phase harmonics admit a dual representation, closely related to the rectiﬁer non-linearity in CNNs.
Our main contributions are:
• We develop a family of texture models based on the wavelet transform and a generalized rectiﬁer non-linearity, that signiﬁcantly improves the visual quality of the classical waveletbased model of Portilla & Simoncelli (2000) on a wide range of textures.
It relies on introducing spatial shift statistics across scales to capture geometric structures in textures.
• By changing the number of statistics in our models, we show explicitly the trade-off on the quality and diversity of the synthesis.
When there are too many statistics, our model tends to memorize image patches.
We further investigate such memorization effects on non-stationary images and ﬁnd that it sometimes relies on what statistics are chosen, rather than on how many.
• Through the modeling of geometric structures in gray-scale textures, our model indicates the possibility of reducing signiﬁcantly the number of statistics in the works of Gatys et al.
and
Ustyuzhaninov et al.
(2017), to achieve a similar visual quality.
The rest of the paper is organized as follows: Section 2 reviews the framework of microcanonical maximum-entropy models, build upon a general family of covariance statistics.
We then present our model for both gray-scale and color textures in Section 3.
Section 4 shows synthesis results of our model, compared with state-of-the-art models.
Finally, in Section 5, we discuss possible improvements of our model 2 .
Notations Throughout the paper, N denotes a positive integer.
A gray-scale image x is an element of R N × N , i.e. x = x ( u ) , u ∈ Ω N , with Ω N := { 0 , · · · , N − 1 } 2 .
A color image x = { x c } c =1 , 2 , 3 is an element of R 3 × N × N , or equivalently, each x c ∈ R N × N .
We shall denote ¯ x the observed texture (observation), which is assumed to be a realisation a random vector X .
For any complex number z ∈ C , z ∗ is the complex conjugate of z , Real ( z ) its real part, | z | its modulus, and ϕ ( z ) its phase.
M ICROCANONICAL COVARIANCE MODELS
We brieﬂy review the standard framework of micro-canonical maximum-entropy models for textures.
To reliably estimate the statistics in these models, we assume that a texture is a realization of a stationary and ergodic process X (restricted to Ω N ).
We then review a special family of statistics that are used in the state-of-the-art texture models (mentioned in Figure 1), based on covariance statistics of an image representation.
2. F RAMEWORK
Given a observation texture ¯ x , we aim at generating new texture images, similar but different from ¯ x .
To that end, a classical method is to deﬁne a set of statistics C ¯ x , computed on the observation, and try to sample from the microcanonical set
[Equation]
where (cid:107) · (cid:107) denotes the L 2 norm.
Under the stationary and ergodic assumption of X , one can construct Cx as a statistical estimator of E ( CX ) , from a complex-valued representation Rx .
The set of covariance statistics Cx of a model can then be constructed by computing an averaging over the spatial variable u , i.e.
[Equation]
for ( γ, γ (cid:48) , τ ) ∈ Υ ⊆ Γ × Γ × Ω N .
The statistics Cx ( γ, γ (cid:48) , τ ) can be interpreted as estimating the covariance (resp.
correlations) between RX ( γ, u ) and RX ( γ (cid:48) , u − τ ) for zero-mean RX (resp.
nonzero mean RX ).
The ergodicity assumption ensures that when N is large enough, the approximation
C ¯ x (cid:39) E ( CX ) over Υ should hold with high probability.
Under these conditions, it makes sense to sample the microcanonical set in order to generate new texture samples.
This framework encompasses a wide range of state-of-the-art texture models 4 .
In particular, the PS model takes inspiration from the human early visual system to deﬁne a multi-scale representation based on the wavelet transform of the image (Heeger & Bergen, 1995).
We next review a family of covariance model which generalizes the statistics in the PS model.
We write C M the statistics for a speciﬁc model M that uses the representation R M .
2. W AVELET PHASE HARMONIC COVARIANCE MODELS
We review a family of microcanonical covariance models deﬁned by a representation built upon the wavelet transform and phase harmonics.
It deﬁnes a class of covariance statistics that capture dependencies between wavelet coefﬁcients across scales.
2.2. W AVELET TRANSFORM
The wavelet transform is a powerful tool in image processing to analyze signal structures, by deﬁning a sparse representation (Mallat, 2001).
For texture modeling, we consider oriented wavelets to model geometric structures in images at multiple scales.
They include the Morlet wavelets and steerable wavelets, proposed in Goupillaud et al.
; Simoncelli & Freeman (1995); Unser &
Chenouard (2013).
In particular, the Simoncelli steerable wavelets have been used to model a diverse variety of textures in Portilla & Simoncelli (2000).
Oriented wavelets are deﬁned by the dilation and rotation of a complex function ψ : R 2 (cid:55)→ C on a plane.
Let r θ denote the rotation by angle θ in R 2 .
They are derived from ψ with dilations by factors 2 j , for j ∈ { 0 , 1 , · · · , J − 1 } , and rotations r θ over angles θ =
(cid:96)π/L for 0 ≤ (cid:96) <
L , where L is the number of angles in [0 , π ) .
The wavelet at scale j and angle θ is deﬁned by
[Equation]
Scales equal or larger than J are carried by a low-pass ﬁlter φ J .
The wavelet transform of an image x ∈ R N × N is a family of functions obtained by the convolution of x with discrete wavelets.
5 Let Λ := { 0 , · · · , J − 1 } × πL { 0 , · · · , L − 1 } be an index set.
The wavelet coefﬁcients are
[Equation]
The low-pass coefﬁcients x (cid:63) φ J are deﬁned similarly.
[Equation] W AVELET PHASE HARMONICS AND THE PS MODEL
To model natural textures, it has been shown (Portilla & Simoncelli, 2000; Zhang & Mallat, 2021) that it is crucial to capture statistical dependencies between wavelet coefﬁcients across scales.
This can be achieved by using a wavelet phase harmonic representation, which is deﬁned by the composition of a linear wavelet transform of x , and a non-linear phase harmonic transform.
In Mallat et al.
(2020), the authors introduce the phase harmonics to adjust the phase of a complex number z ∈ C .
More precisely, the phase harmonics { [ z ] k } k ∈ Z of a complex number z ∈ C are deﬁned by multiplying its phase ϕ
( z ) of z by integers k , while keeping the modulus constant, i.e.
∀ k ∈ Z , [ z ] k :=
| z | e ikϕ ( z ) .
The wavelet phase harmonic representation (WPH) is then deﬁned by
[Equation]
where µ γ is deﬁned as the spatial average of [¯ x (cid:63) ψ j,θ ] k .
It is shown in Zhang & Mallat (2021) that the PS model can be regarded as a low-order wavelet phase harmonics covariance model, which considers only a restricted number of pairs ( k, k (cid:48) )
(see
Appendix A for more details).
In the next section, we shall use a dual representation of the phase harmonic operator to deﬁne a covariance model to capture high-order phase harmonics.
G ENERALIZED RECTIFIER WAVELET COVARIANCE MODEL
In the previous section, we presented a class of models, built from the wavelet phase harmonic representation.
A dual representation of the phase harmonic operator [ · ] k can be deﬁned via a generalized rectiﬁed linear unit, that we review in Section 3.1.
We then discuss in Section 3.2 how to deﬁne an appropriate index set of Γ for gray-scale textures.
Section 3.3 extends the model to color textures.
3. F ROM PHASE HARMONICS TO THE GENERALIZED RECTIFIER
The generalized rectiﬁed linear unit of a complex number z , with a phase shifted by α ∈ [0 , 2 π ] , is deﬁned by
[Equation]
where ρ is a rectiﬁed linear unit, i.e. for any t ∈ R , ρ ( t ) :
= max(0 , t ) .
In Mallat et al.
(2020), it is shown that applying a Fourier transform on ρ α ( z ) along the variable α results in the phase harmonics of z (up to some normalization constant).
This suggests an alternative model, deﬁned by coefﬁcients of the form
[Equation]
for γ ∈ Γ
= Λ × [0 , 2 π ] , and µ γ is deﬁned as the spatial average of ρ α (¯ x (cid:63) ψ j,θ ( u )) over u ∈ Ω N .
Relation with high-order phase harmonics Based on the duality between the phase harmonics k ∈ Z and the phase shift variable α ∈ [0 , 2 π ] , we now present the relation between C ALPHA and the high-order phase harmonics in C WPH , ﬁrst proved in Mallat et al.
Proposition 1 There exists a complex-valued sequence { c k } k ∈ Z such that for all ( j, θ, α ) ∈ Γ , ( j (cid:48) , θ (cid:48) , α (cid:48) )
∈ Γ , and all τ ∈ Ω N ,
[Equation]
The proof is given in
Appendix B. We remark that the sequence { c k } k ∈ Z is uniquely determined by the rectiﬁer non-linearity ρ , and they are non-zero if k is even (Mallat et al., 2020).
This result shows that for a suitable choice of ( α, α (cid:48) ) , the covariance statistics C ALPHA x can implicitly capture
C WPH x with a wide range of k and k (cid:48) .
Relation with second order statistics Using a simple decomposition of wavelet coefﬁcients into their positive, negative, real and imaginary parts, we can further show that the covariance statistics
C ALPHA x capture the classical second order statistics of wavelet coefﬁcients, also used in the PS model (with phase harmonic coefﬁcients
k = k (cid:48)
= 1 ).
Proposition 2 Let I = { 0 , π 2 , π, 3 π 2 } .
There exists a ﬁnite complex-valued sequence { w α,α
[Equation]
[Equation] ,α (cid:48) ) ∈
I 2 such that for all ( j, θ ) ∈ Λ , and all τ ∈ Ω N ,
[Equation]
The proof is given in Appendix C.
This shows that using only four α uniformly chosen between [0 , 2 π ] is sufﬁcient to capture second order statistics.
Because the wavelet transform is an invertible linear operator (on its range space), computing the r.h.s of eq.
(6) for all ( j, θ, τ ) , as well as the low-pass coefﬁcients carried out by Φ J , is equivalent to computing the correlation matrix of x .
Relation with the RF model Setting aside the subtraction by the spatial mean µ γ , the RF model can be viewed as a particular case of models deﬁned by eq.
Indeed, the statistics of the RF model take the form of eq.
(1), with
[Equation]
where { ψ f } being a family of multi-scale random ﬁlters.
By writing ρ α ( x (cid:63) ψ j,θ ( u − τ ))
=
ρ ( x (cid:63)
Real ( ψ τj,θ e iα )( u )) , with ψ τj,θ denoting the translation of ψ j,θ by τ , we see that the models are similar, the difference being that our models use wavelet-based ﬁlters instead of random ones.
3. D EFINING AN APPROPRIATE Υ
The choice of the covariance set Υ is of central importance in the deﬁnition of the model.
Intuitively, a too small set of indices will induce a model that could miss important structural information about the texture that we want to synthesize.
Conversely, if Υ contains too many indices, the syntheses can have good visual quality, but the statistics of the model may have a large variance, leading to the memorization of some patterns of the observation.
There is a trade-off between these two aspects: one must capture enough information to get syntheses of good visual quality, but not much, so as not to reproduce parts of the original image.
To illustrate this point, we shall study the model ALPHA deﬁned with three different sets Υ : A smaller model ALPHA S with a limited amount of elements in Υ , an intermediate model ALPHA I , and a larger model ALPHA L .
To precisely deﬁne these models, let us note J := { 0 , · · · , J − 1 } , Θ := πL { 0 , · · · , L − 1 } , and A A =πA { 0 , · · · , A − 1 } .
Let us also deﬁne the set T := { 0 }∪{ 2 j (cos( θ ) , sin( θ )) } 0 ≤ j<J, θ ∈ πL { 0 , ··· , 2 L − 1 } , from which the spatial shift shall be selected.
Table 1 summarizes the conditions that all parameters have to satisfy to be contained in these sets.
Additionally, these models include large scale information through the covariance of a low-pass ﬁlter, i.e. the spatial average of x (cid:63) φ J ( · ) x (cid:63) φ J ( · − τ ) , for τ ∈ T .
To count the size of Υ without redundancies, Appendix A.3 provides an upper bound on the non-redundant statistics in our models.
This upper bound is used to count the number of statistics in our models.
To keep this number from being too large, instead of taking all shifts in a square box, such as in Portilla & Simoncelli (2000), we choose to select only shifts of dyadic moduli, and with the same orientations as the wavelets.
ALPHA S vs. ALPHA I
The small model ALPHA S is inspired from the PS model, as it only takes into account of the interactions between nearby scales (i.e. | j (cid:48)
− j | ≤ 1 ), and the spatial shift correlations are only considered for ( j, θ ) =
( j (cid:48) , θ (cid:48) ) .
There are two notable differences in the statistics included in the ALPHA S and ALPHA I models.
The ﬁrst one is the range of scales being correlated.
It has been shown in Zhang & Mallat (2021) that constraining correlation between a wider [Table] range of scales induces a better model for non-Gaussian stationary processes, and a better estimation of cosmological parameters from observed data (Allys et al., 2020).
The second difference, which has a signiﬁcant impact on the number of statistics (it increases the model size by a factor ∼ 10), is the number of spatial shifts in the correlations.
In the ALPHA I model, spatially shifted correlations are computed for all pairs of coefﬁcient ( γ, γ (cid:48) ) .
For both stationary textures and non-stationary images in gray-scale, shape and contours of salient structures and objects are better reproduced with
ALPHA I , as illustrated in Figure 2.
More examples are given in Appendix D.
ALPHA I vs. ALPHA L
As we observe in Figure 2, the ALPHA I model, containing 4 times less coefﬁcients than the ALPHA L , suffers less from memorization effects, while still capturing most of the geometric information in the images.
This small loss of information can be partially explained by the frequency transposition property of the phase harmonics operator (Mallat et al., 2020), for compactly supported wavelets in the frequency domain, as detailed in Appendix E.
In order to avoid this memorization effect, we shall, in the rest of the paper, consider only the intermediate model.
3. M ODELLING COLOR INTERACTIONS
In order to generate color textures, the covariance model ALPHA I deﬁned in Section 3.2 could be directly applied to each R, G and B color channel independently.
However, it would not take into account the color coherence in the structures of the observation.
To capture color interactions in the observation image, we shall impose the covariance between the coefﬁcients of eq.
(5) for all indices in Υ and all color channels.
More precisely, let x = { x c } c =1 , 2 ,be a color image, with the parameter c representing the color channel.
The ALPHA C color model is deﬁned by correlations between coefﬁcients of the form:
[Equation]
The set of indices is deﬁned as Υ ALPHA C
[Equation]
Υ ALPHA I , ( c, c (cid:48) )
∈ { 1 , 2 , 3 } 2 } .
Reduced ALPHA C
The model ALPHA C has a large size as it computes correlations between all coefﬁcients for all color channels.
This size can be signiﬁcantly reduced by computing spatially shifted coefﬁcients only for the same color channels (to capture their geometries).
This reduced model contains three times less coefﬁcients ( ∼ 113k), without signiﬁcant reduction of the visual quality of syntheses, as detailed in Appendix F.
N UMERICAL RESULTS
In this section, we compare our intermediate model to the state-of-art models (PS, RF and VGG) on both gray-scale and color textures.
We ﬁrst specify the experimental setup.
We then present the synthesis results of various examples, and discuss their quality through visual inspection.
A quantitative evaluation of the quality of the syntheses, based on the synthesis loss of the VGG model, and proposed in Ustyuzhaninov et al.
(2017), is discussed in Appendix G.
[Figure]
4. E XPERIMENTAL S ETUP
For our experiments, we choose gray-scale and color textures with various visual structures.
In the gray-scale examples, we also include a stationary turbulent ﬁeld (vorticity), which is simulated from
Navier-Stokes equations in two dimensions (Schneider et al., 2006).
These examples all contain complex geometric structures that are hard to model by the classical PS model.
The texture images presented in this work have a size of N = 256 , giving a number of pixels of ∼ 65k.
For all the ALPHA models, we use Morlet wavelets with a maximal scale J = 5 and number of orientations L = 4 .
This choice differs from the PS model, which uses Simoncelli steerable wavelets.
In Appendix H, we discuss the impact of these wavelets on our model.
To draw the samples, we follow gradient-based sampling algorithms, suitable in high-dimensions (Bruna &
Mallat, 2018), to minimize the objective function (cid:107)
Cx − C ¯ x (cid:107) 2 , starting from an initial sample from a normal distribution.
Similarly to Gatys et al.
; Ustyuzhaninov et al.
(2017), we use the LBFGS algorithm (Nocedal, 1980) for the optimization of the objective.
As in the VGG model (Gatys et al., 2015), we further apply a histogram matching procedure as post-processing after optimization.
The details of the PS, RF and VGG models, as well as detailed speciﬁcations of our models, are given in Appendix A. More synthesis examples can be found in Appendix I.
4. R ESULTS
In Figure 3, we present examples of syntheses from the ALPHA I (or ALPHA C ), PS, RF and VGG models, for both gray-scale and color textures, as well as for non-stationary images.
We observe that our model ALPHA I produces texture syntheses of similar visual quality to the RF and VGG models.
It also signiﬁcantly outperforms the PS model in terms of the visual quality, without introducing visible memorization effects.
As the model PS uses the statistics closer to ALPHA S compared to
ALPHA I , the performance of PS is somehow expected.
Note that for the tiles example (the ﬁfth row) in Figure 3, the VGG model produces less convincing textures, because the long-range correlations present in the image (aligned tiles) are not reproduced.
To remedy this issue, it has been proposed in Berger & Memisevic (2017) to add spatial shifts to the correlations of the network feature maps.
These shift statistics are similar to the parameter τ in our model.
We also observe that, in the case of the sixth row example (ﬂowers), all models fail to reproduce complex structures at object-level.
Possible improvements of such models is further discussed in Section 5.
For non-stationary images, we ﬁnd that certain image patches can be more or less memorized by the RF, VGG and ALPHA I models, as illustrated in the seventh row example.
Understanding such memorization effect of non-stationary images is a subtle topic, as we ﬁnd that in some binary images ( ¯ x ( u ) ∈ { 0 , 1 } ), only the PS model can reproduce the observation, even though it has a much smaller number of statistics (the last row example).
We ﬁnd that this is related to the spatial correlation statistics in PS (non-zero τ ).
By removing this constraint, ¯ x is no longer always reproduced.
7 The non-stationary nature also appear in some logo near the boundary of some textures (e.g. bottom left in the observation of the ﬁrst and fourth rows).
Although this logo is reproduced by RF, VGG,
ALPHA I and ALPHA C , it is a very local phenomenon, as we do not ﬁnd visible copies of the textures when there is no logo, and it is likely related to the way one addresses the boundary effect (see more in Appendix A.5).
D ISCUSSION
In this work, we presented a new wavelet-based model for natural textures.
This model incorporates a wide range of statistics , by computing covariances between rectiﬁed wavelet coefﬁcients, at different scales, phases and positions.
We showed that this model is able to capture and reproduce complex geometric structures, present in natural textures or physical processes, producing syntheses of similar quality to state-of-the-art models that use CNN-based representations.
Being deﬁned with a wavelet family instead of multi-scale random ﬁlters, the proposed model uses less statistics than the RF model.
For the gray-scale textures, our model has about 15 times less statistics, as it focuses on capturing the geometric structures present in images.
Although our color model has a slightly larger number of statistics than VGG, the reduced color model, presented in
Section 3.3 is three times smaller than ALPHA C , while achieving similar visual quality.
It shows the potential to further reduce the size of the color model.
In the PS model, a PCA on the color channels is performed (Vacher & Briand, 2021).
The same idea could also be applied to our model.
Furthermore, there are examples where all the models may all fail to produce some geometric structures at object-level, as illustrated in Figure 3 (sixth row).
In this situation, we still need to ﬁnd more informative statistics.
One may for example consider to incorporate a second layer of wavelet transform as in the wavelet scattering transform (Leonarduzzi et al., 2019; He & Hirn, 2021).
Another line of research is to introduce other kinds of losses (such as to encourage image smoothness) in order to improve the VGG model (Liu et al., 2016; Sendik & Cohen-Or, 2017).
These losses are complementary, and could thus also be added to our models.
Integrating these models with learning-based approaches is another promising direction (Zhou et al., 2018; Zhu et al., 1997; Xie et al., 2016; 2018).
Finding a minimal set of statistics to deﬁne a texture model remains important because a large number of statistics can result in a high variance of the estimators, and the associated model may suffer from memorization effects.
This is a problem because the aim of the model is to approximate the underlying distribution of the observation, and therefore produce diverse textures.
In this regard, the mere visual evaluation of the synthetic textures can fail to take this aspect of the model into account.
Deﬁning a quantitative evaluation of quality and diversity, coherent with visual perception, remains an open problem (Ustyuzhaninov et al., 2017; Yu et al., 2019).
Observation
PS (3.2k/17k)
RF (525k)
Ours (35k/320k)
VGG (177k)
Acknowledgments We thank all the reviewers for their insightful feedback.
This work was partially supported by a grant from the PRAIRIE 3IA Institute of the French ANR-19-P3IA- 0001 pro- gram.
Sixin Zhang acknowledges the support provided by 3IA Artiﬁcial and Natural Intelligence
Toulouse Institute, French ”Investing for the Future - PIA3” program under the Grant agreement
ANR-19-PI3A-0004, and by Toulouse INP ETI and AAP unique CNRS 2022 under the Project
LRGMD. Reproducibility Statement In order to reproduce the experiments presented in this work, the main parameters to deﬁne the model can be found in Section 4.1. More details are given in Appendix A.
Additionally, the code is made publicly available.
Ethics Statement The authors acknowledge that no potential conﬂicts of interest, discrimination, bias, fairness concerns or research integrity issues have been raised during the completion of this work.
A M ODEL AND ALGORITHMIC SPECIFICATION
We provide additional information needed to reproduce the numerical results of the models (for both gray-scale and color textures) considered in this paper.
First, we detail the models of PS, RF, VGG.
We then give algorithmic parameters to obtain the synthesis images.
For natural textures, we also propose a strategy to synthesize non-periodic images in our models.
An image is non-periodic if a periodic extension of the image to the domain outside Ω N create discontinuities at the contour of Ω N .
Sources of textures Our natural texture examples were obtained from the following three sources:
CNS NYU 8 , Textures.com 9 , Describable Textures Dataset model 10 and the Github page of Berger & Memisevic (2017) 11 .
A. M ODEL PARAMETERS
We specify the model parameters to synthesize both gray-scale and color textures.
We also discuss how to extend the RF and the VGG model, originally designed for color textures, to model gray-scale textures.
• PS: For both gray-scale and color models, we set the number of scales J = 5 , and the number of orientations L = 8 for the Simoncelli steerable wavelets.
The spatial shift τ = ( τ 1 , τ 2 ) ∈
Ω N is chosen to be in the range of max ( τ 1 , τ 2 ) ≤ ∆ = 4 .
Note that this is different to the model parameters reported for the default PS model (which is J = 4 , L = 4 , ∆ = 3 ) as we ﬁnd that it results in a larger set of statistics and better visual quality.
The synthesis results of this model can be reproduced by a Matlab software.• RF: For gray-scale textures, we consider J × L random convolutional ﬁlters ( ψ f ) 1 ≤ f ≤ JL .
Let f = ( j, (cid:96) ) , the index j representing the scale of each ﬁlter, whose size is ( W j , W j ) for 1 ≤ j ≤ J .
For Γ = {
f
= ( j, (cid:96) )
| j ≤ J, (cid:96) ≤ L } , the representation is
[Equation]
For a color image x = { x c } 1 ≤ c ≤ 3 , we use 3 × J × L random convolutional ﬁlters
The representation of x is
[Equation]
The correlations C RF x are deﬁned for all pairs ( γ, γ (cid:48) )
∈ Γ × Γ .
In both the gray and color cases, it results in a correlation matrix C RF with J 2 L 2 statistics.
Following the default setting of the RF model, we set J = 8 and L = 128 for ﬁlters whose sizes are W 1 = 3 , W 2 = 5 , W 3 = 7 , W 4 = 11 , W 5 = 15 , W 6 = 23 , W 7 = 37 , W 8 = 55 .
Each ﬁlter ψ j,(cid:96) or ψ c,j,(cid:96) is generated randomly according to GlorotUniform in the software Lasagne.• VGG.
For a color image x = { x c } , the VGG model computes a correlation matrix C VGG x between the features maps within different layers of a pre-trained CNN network.
To adapt this model to gray-scale textures, we shall add one input layer which converts a gray-scale image y into a color image, by setting x c
= y for each color channel c .
This allows one to use the same C VGG x to compute the gradient of the VGG loss with respect to y , and therefore to synthesise a gray-scale texture.
For both gray-scale and color textures, we use only ﬁve layers ’conv1 1’, ’pool1’, ’pool2’, ’pool3’, ’pool4’, as proposed in the original work.
• ALPHA .
See the main text.
A. A LGORITHMIC PARAMETERS
We specify the optimization parameters used to synthesize both gray-scale and color textures.
• PS: It utilizes iterative projections onto constraint sets to generate textures.
We set the number of iterations to 200.
• RF: It uses the L-BFGS procedure 14 with a memory size 20, and with a maximal number of iterations 2000.
The initialization for each pixel value of a gray-image is Uniform between [ − 1 , 1] .
For the color image case, each RGB channel is initialized independently with a Uniform distribution between [ − 1 , 1] .
To address non-zero mean textures (i.e. E ( X ( u )) (cid:54) = 0 ), the empirical mean of ¯ x is subtracted from the input x to compute the representation.
It is added back to the output of the optimization to produce a synthesis.
• VGG: It uses the L-BFGS procedure 15 with a memory size of 20, and with a maximal number of iterations of 2000.For both gray-scale and color images, bounds constraints are used for the optimization.
The initialization of each pixel value is the standard normal distribution (zero mean, unit variance).
To address non-zero mean textures (i.e. E ( X ( u )) (cid:54) = 0 ), the VGG mean is subtracted from the input x of the representation 16 .
It is added back to the output after the optimization to produce a synthesis (with an additional histogram matching post-processing).
• ALPHA :
For all the models in ALPHA , we use the L-BFGS optimization algorithm with restarts.
Starting from the standard normal distribution, with mean and standard deviation estimated from the observation, we use the L-BFGS procedure implemented in Pytorch.
It runs for 500 iterations and then it is restarted with an initialization obtained from the previous L-BFGS result.
This is repeated 10 times to obtain the synthesis (with an additional histogram matching post-processing).
A. N UMBER OF STATISTICS IN THE ALPHA MODELS
In this section, we detail the number of statistics in the different ALPHA models.
We begin by giving the formula for each model (note that we do not include the low-pass statistics, which numbers are negligible).
To (partially) avoid redundancy in the coefﬁcients, for all models, we compute only the correlations for indices in Υ such that j 2 ≥ j 1 .
This gives us the following formulas for the number of statistics:
• #(ALPHA S ) =
(2 J − 1) | Θ 4 | 2 |A 4 | + J | Θ 4 ||A 4 || T | • #(ALPHA I ) = 1 2 J ( J + 1) | Θ 4 | 2 |A 4 || T |
io
• #(ALPHA L ) = 12 J ( J + 1) | Θ 4 | 2 |A 4 | 2 | T | • #(ALPHA C )
= 92 J ( J + 1) | Θ 4 | 2 |A 4 || T |
Note that, for all ALPHA models, we also compute ﬁrst order statistics µ γ , i.e. the spatial averages of R ALPHA
¯ x ( γ, u ) .
There are J | Θ 4 ||A 4 | statistics of this sort in every model, which is negligible with respect to the total number of second order statistics.
Note also that there are still some redundancies in these statistics, as for j = j (cid:48) , all correlations for θ, θ (cid:48) , α, α (cid:48) , τ are counted twice.
The number of such statistics is 17 , for all model, superior to the number of ﬁrst order statistics, which shows that our formula is in fact an upper bound for the exact number of statistics.
A. N UMBER OF STATISTICS IN THE PS MODELS
The PS model can be interpreted as a particular case of the WPH covariance model as it contains the following three key categories of statistics:
• Raw coefﬁcient correlations ( k = k (cid:48)
= 1 ): they capture 2nd order statistics of a stationary process X ,
i
e. the correlations between X ( u ) and X ( u (cid:48) ) for ( u, u (cid:48) )
∈
Ω 2 N .
• Coefﬁcient magnitude statistics ( k = k (cid:48)
= 0 ): they capture information of X beyond the 2nd order statistics.
Very often, nearby scales and angles are considered in the model, such as j (cid:48)
≈ j and θ (cid:48)
≈ θ
. • Cross-scale phase statistics ( k = 1 , k (cid:48)
= 2 ): they capture local phase alignments of the wavelet coefﬁcients at nearby scales j (cid:48) =
j + 1 , which are complementary to the magnitude correlations.
The major issue to count the number of statistics of this model is to avoid double counting the statistics which are the same.
This is mostly due to the symmetries of the covariance matrices.
We next detail how obtain the number of statistics for both the gray-scale and color model, by following the work of Portilla & Simoncelli (2000) and Vacher & Briand (2021).
As before, we assume the number of wavelet scales is J , the number of wavelet orientations is L , and the spatial shift range is within a square of size (2∆+1) × (2∆+1) .
Let us denote N a = 2∆+1 .
We next describe in detail the number of statistics of each category counted in our paper,
A.4. PS IN GRAY - SCALE
• Marginal statistics of x : 6 .
They include mean, variance, skewness, etc.
• Marginal statistics of wavelet coefﬁcients: 2( J + 1) + 1 . • Auto-correlation of wavelet coefﬁcients (raw coefﬁcient correlations): ( J + 1)( N 2 a
+ 1) / 2 .
• Auto-correlation of magnitude of wavelet coefﬁcients (coefﬁcient magnitude statistics): JL ( N 2 a + 1) / 2 + JL ( L − 1) /
+ ( J − 1) L 2 .
• Mean of magnitude of wavelet coefﬁcients: JL
+ 2 .
This is not counted in the paper of Portilla & Simoncelli (2000), but it used in the Matlab software.
• Cross-correlation of phase of wavelet coefﬁcients (cross-scale phase statistics): 2( J − 1)
L 2 + JL 2 .
The extra JL 2 coefﬁcients are cross-correlation of real sub-band cousin coefﬁcients, which are not counted in the paper, but used in the Matlab software.
To compare with the model size of the original work of Portilla & Simoncelli (2000), one can check that the sum of these number is 792 when J = 4 , L = 4 , ∆ = 3 ( N a = 7) .
If we do not count the coefﬁcients which are only counted in the Matlab software ( JL + 2 + JL 2 = 82 ), it results inas reported in the original paper.
A.4. PS IN COLOR
• Marginal statistics of x and PCA transform of x : 6 × 3 + 3 × 4 = 30 .
• Marginal statistics of wavelet coefﬁcients: 6( J + 1) + 9 . • Auto-correlation of wavelet coefﬁcients (raw coefﬁcient correlations): 3( J + 2)( N 2 a
+ 1) / 2 .
This is called Central autoCorr of the PCA bands in the Matalb software (which includes lowband).
• Auto-correlation of magnitude of wavelet coefﬁcients (coefﬁcient magnitude statistics): 3 JL ( N 2 a + 1) / 2 + J (3 L )(3 L − 1) /
+ ( J − 1)(3 L ) 2 .
• Mean of magnitude of wavelet coefﬁcients: 3( JL + 2) .
• Cross-correlation of phase of wavelet coefﬁcients (cross-scale phase statistics): J (3 L )
+ ( J − 1)(3 L )(6 L ) .
A. N ON PERIODIC BOUNDARIES IN NATURAL IMAGES
The convolution operation in the wavelet transform (equation 2) is performed using the Fast Fourier
Transform.
Additionally, recall from Section 2.1 that spatial shifts are deﬁned with periodic boundary conditions.
This implies periodicity of the input image x .
However, natural texture images are not periodic, so one needs to adapt the computation of coefﬁcients to take into account possible border effects.
To that end, instead of averaging over all u ∈ Ω N as in eq.
(1), each correlation coefﬁcient is averaged over a sub-window inside Ω N , which size depends on the scales of the coefﬁcients being correlated.
More precisely, let γ =
( j, θ, α ) and γ (cid:48) =
( j (cid:48) , θ (cid:48) , α (cid:48) ) .
Note j m :=
max( j, j (cid:48) ) .
We deﬁne Ω j m := { u = ( u 1 , u 2 )
∈
Ω N : 2 j m ≤ u i < N − 2 j m , i = 1 , 2 } .
Then, for non periodic images, we compute
[Equation]
where the spatial shifts are deﬁned periodically.
Note that the spatial averages µ γ and µ γ (cid:48) are also performed on Ω j m .
B P ROOF OF P ROPOSITION
Using the fact that
[Equation]
and computing the Fourier coefﬁcients of the 2 π -periodic function
ρ α ( z ) in the variable α , we obtain
[Equation]
where c k is the Fourier transform of h ( . )
= ρ (cos( . ))
at the frequency k .
The function α
(cid:55)→ ρ α ( z ) being periodic in α , we have its decomposition in Fourier series
[Equation]
We can then write, for any z, z (cid:48) ∈ C , and α, α (cid:48) ∈
[0 , 2 π ] ,
[Equation]
α ( z ) ρ α (cid:48)
( z (cid:48) ) ∗ = (cid:88)
[Equation]
Replacing z and z (cid:48) by any two wavelet coefﬁcients x (cid:63) ψ j,θ ( u ) and x (cid:63) ψ j (cid:48) ,θ (cid:48) ( u − τ ) , we thus obtain the relation in Proposition 1.
C P ROOF OF P ROPOSITION
Let z ∈ C , and recall from eq.
(4), that ρ α ( z )
=
ρ ( Real ( ze iα )) .
Note that we have the following relation
[Equation]
We can then write
[Equation]
with I = { 0 , π 2 , π, 3 π 2 } .
Replacing z with x (cid:63) ψ j,θ ( u ) , z (cid:48) with x (cid:63) ψ j (cid:48) ,θ (cid:48) ( u − τ ) ,, and injecting this relation in eq.
(1) gives us the desired result, with w α,α (cid:48) = Ω N w (cid:48) α,α (cid:48)
[Equation]
D S UPPLEMENTARY RESULTS FOR THE GRAY - SCALE ALPHA MODELS
Here, we present further visual comparison between the different ALPHA models for gray-scale images, to illustrate the trade-off between quality and diversity.
E R ELATION BETWEEN ALPHA L AND ALPHA
I
Here, we informally explain why, under some conditions on the wavelet family, setting α (cid:48) ∈ { 0 } in the ALPHA models should not lose too much (but still some) information captured by the statistics.
First, remark that the simple linear relation
[Equation]
tells us that computing all correlations for α
∈
A 4 and α (cid:48)
=
gives us at least the information contained in the r.h.s.
of eq.
Furthermore, if α ∈ A 4 = { 0 , · · · , 3 π 4 } , we make the following approximation
[Equation]
Recall also from the proof of Proposition 2, that F ( ρ α ( z ))
( k ) =
[ z ] k c k , where the Fourier transform is taken along the variable α .
Therefore,
[Equation]
Therefore, (cid:0) (cid:88)
[Equation]
[Figure]
Then, replacing z and z (cid:48) with wavelet coefﬁcients, we get (cid:88)
[Figure]
[Equation]
Using Plancherel’s theorem, we can write that
[Equation]
where the Fourier transform of an image x is deﬁned by F ( x )( ω ) :
= (cid:80) u ∈ Ω N x ( u ) e − iωu , and t τ denotes the translation by τ , i.e. t τ f
( · ) = f
( · − τ ) .
Now, suppose that the wavelets ψ α have disjoint compact frequency support, in balls B λ (2 − j C (cid:48) ) , where λ = 2 − j r − θ ξ 0 , and ξ 0 is the central frequency of the mother wavelet ψ (cf. ?).
Suppose also that frequency transposition property of the phase harmonics operator (cf.
Mallat et al.
(2020)) is such that [ x (cid:63) ψ λ ] k has (approximately) frequency support in B kλ ( k 2 − j C (cid:48) ) .
Then, for all λ, λ (cid:48) , and all k ∈ Z , there exists only one k ∗ such that the frequency supports of [ x (cid:63) ψ λ ] k and [ x (cid:63) ψ λ (cid:48) ]
− k ∗ are not disjoint, i.e. such that C WPH x (( j, θ, k ) , ( j (cid:48) , θ (cid:48) , k (cid:48) ) , τ ) (cid:54) = 0 .
This tells us that
[Equation]
Thus, computing all correlations for α ∈ A 4 , and α (cid:48)
=
gives us (approximately) all the information contained in WPH coefﬁcients for any pair k, k (cid:48) .
This result lies on several approximations, and strong assumptions about the wavelets, which are not fully met in practice.
For this reason, setting α (cid:48)
=
instead of α (cid:48)
∈
A 4 effectively reduced the amount of information captured by the statistics, and therefore increases the diversity of the model.
However, as we observe in Section 3.2, there is not too much information lost, and the resulting model still captures most of the important geometric structures in texture images.
F R EDUCED COLOR MODEL
One can reduce the number of statistics in the color model by selection the spatial shift parameter τ to be non-zero only for correlations between the same color channels.
More precisely, it is deﬁned by the following index set: Υ := { ( γ, γ (cid:48) , τ ) : (( j, θ, α ) , ( j (cid:48) , θ (cid:48) , α (cid:48) ) ,
τ ) ∈ Υ ALPHA
I , c
= c (cid:48) ∈ { 1 , 2 , 3 }} ∪ { ( γ, γ (cid:48) , 0) :
(( j, θ, α ) , ( j (cid:48) , θ (cid:48) , α (cid:48) ) ,
) ∈
Υ ALPHA I , ( c, c (cid:48) )
∈ { 1 , 2 , 3 } 2 } .
This gives a model of size ∼ 113 k, with little degradation of the visual quality, as shown in Figure 5.
G VGG SCORE
In Ustyuzhaninov et al.
(2017), the authors proposed to use the synthesis loss of the VGG model to evaluate the quality of syntheses from any model.
The goal is to deﬁne a quantitative, and more objective evaluation method than mere visual inspection.
Since the VGG model produces syntheses almost indistinguishable from real textures, it is natural to consider its loss to asses the quality of a synthesis.
We computed this loss for the ﬁrst two examples of Figure 3 (radishes and cherries), and the frist two examples of Figure 9 (gravel and Turbulence ﬂow).
Note however that this loss is not exactly the same as the one used in Ustyuzhaninov et al.
(2017), as the layers selected to compute the loss are different.
In this work, we chose to use the layers suggested in Gatys et al.
(2015), (i.e.
’conv1 1’, ’pool1’, ’pool2’, ’pool3’, and ’pool4’ of the VGG-19 network (Simonyan & Zisserman, 2014)), and compute the relative VGG loss 18 .
We notice that this score is not always consistent with visual inspection, as there are texture examples and models for which the syntheses do not look much like the observation image, yet produce a small
VGG loss (see e.g. the ﬁrst and last rows of Figure 3, the RF model syntheses have the smallest loss).
It should also be noted that the VGG loss reported on the VGG syntheses is not the synthesis loss after optimization, as a histogram matching (HM) procedure is performed as post-processing after optimization.
We observed that the VGG loss of the syntheses from the VGG model after HM was considerably higher than the one for syntheses before it, while being visually very similar as illustrated in Figure 6.
These observations suggest that the VGG score suffers from instabilities after reaching a certain level (that is, if the VGG loss is small enough, small perturbations of the values of the image pixels might have a strong impact on the loss).
H I NFLUENCE OF THE CHOICE OF THE WAVELET TRANSFORM H. I NFLUENCE OF THE WAVELET FAMILY
In Section 3.2, we illustrated the importance of the set of indices Υ that deﬁne the wavelet coefﬁcients being correlated.
Another important role is played by the choice of the wavelets used in [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Table]
[Figure] equation 2. As illustrated in Figure 7, this choice can have a visible impact on the quality of the textures. We observe that, while on the ﬁrst example, the coherence of the structures appear similar for the three wavelet families, the second example shows that the wavelets used in Portilla & Simoncelli (2000) are less efﬁcient in reproducing the contours of the objects (pebbles).
While in our experiments, we chose to use the classical Morlet wavelets, an optimal choice for the wavelet family remains an open problem.
Observation
Simoncelli
Bump
Morlet
H. I NFLUENCE OF SCALE PARAMETER
Recall from Section 2.2.1, the wavelet transform of an image x is deﬁned by
[Equation]
The maximal scale parameter J also plays an important role in the deﬁnition of the wavelet transform.
It determines the scales of the structures being captured by the transform.
If this parameter is too small, large structures in the observation image might not be captured and reproduced in the model syntheses.
Conversely, if J is too large, then the large scale statistics may have a high variance, inducing a memorization effect in the syntheses.
Figure 8 illustrates this point on two examples from Section 4.2.
By setting J = 4 (i.e. the maximal range of structures captured by the wavelets is of size 2 4 = 16 ), we observe on the ﬁrst example that the larger structures (bubbles) are not well reproduced.
When J is set to 6, the observation is almost identically reproduced by the synthesis.
Similarly on the second examples, several parts of the synthesis appear very similar to ones in the observation.
We found that a suitable trade-off consists in setting J = 5 for images of size N = 256 .
Observation
J = J = J =
I S UPPLEMENTARY RESULTS OF ALPHA I / ALPHA C VS . PS, RF, VGG
In Figure 9 we present additional syntheses on various examples, from the PS, ALPHA C , RF and
VGG models.
These examples can be viewed as random (Turbulence ﬂow, tree bark, porous stone), structured (gravel, paisley pattern, tree leaf, school text), or inhomogeneous (crafted pattern of third row, but also the porous stone).
We see that ALPHA I again signiﬁcantly improves the visual quality of the PS model on gray-scale textures such as the gravel and turbulence ﬂow.
The visual quality of RF and VGG seems also worse on Turbulence ﬂow compared to ALPHA I .
In some examples such as tree leaf, we ﬁnd the synthesis of all the models are similar.
On the inhomogeneous porous stone, non of the models give satisfying visual results.
Similarly, in Figure 10 are presented supplementary syntheses form the color models, for structured images (radishes, bubbles, ﬂowers), quasi-periodic images (scales, honeycomb, bricks), and nonstationary images (feathers).
As previously observed, for highly structured quasi-periodic images such as the bricks example, the VGG model fails to capture long-range correlation, which can be solved using the method of Berger & Memisevic (2017).
Syntheses of non-stationary images exhibit memorization effects, as previously observed.
Observation
[Figure]
2k
[Figure]
Observation
[Figure]