G AUSSIAN M IXTURE C ONVOLUTION N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes a novel method for deep learning based on the analytical convolution of multidimensional Gaussian mixtures.
In contrast to tensors, these do not suffer from the curse of dimensionality and allow for a compact representation, as data is only stored where details exist.
Convolution kernels and data are Gaussian mixtures with unconstrained weights, positions, and covariance matrices.
Similar to discrete convolutional networks, each convolution step produces several feature channels, represented by independent Gaussian mixtures.
Since traditional transfer functions like ReLUs do not produce Gaussian mixtures, we propose using a ﬁtting of these functions instead.
This ﬁtting step also acts as a pooling layer if the number of Gaussian components is reduced appropriately.
We demonstrate that networks based on this architecture reach competitive accuracy on Gaussian mixtures ﬁtted to the MNIST and ModelNet datasets.
I NTRODUCTION
Convolutional neural networks (CNNs) have led to the widespread adoption of deep learning in many disciplines.
They operate on grids of discrete data samples, i.e., vectors, matrices, and tensors, which are structured representations of data.
While this is approach is sufﬁciently efﬁcient in one and two dimensions, it suffers the curse of dimensionality: The amount of data is O ( d k ) , where d is the resolution of data, and k is the dimensionality.
The exponential growth permeates all layers of a CNN, which must be scaled appropriately to learn a sufﬁcient number of k -dimensional features.
In three or more dimensions, the implied memory requirements thus quickly become intractable (Maturana & Scherer, 2015; Zhirong Wu et al., 2015), leading researchers to propose specialized
CNN architectures as a trade-off between mathematical exactness and performance (Riegler et al., 2017; Wang et al., 2017).
In this work, we propose a novel deep learning architecture based on the analytical convolution of Gaussian mixtures (GMs).
While maintaining the elegance of conventional
CNNs, our architecture does not suffer from the curse of dimensionality and is therefore well-suited for application to higher-dimensional problems.
GMs are inherently unstructured and sparse, in the sense that no storage is required to represent empty data regions.
In contrast to discretized k -dimensional volumes, this allows for a more compact representation of data across dimensions, preventing exponential memory requirements.
In this regard, they are similar to point clouds (Qi et al., 2017a).
However, GMs also encode the notion of a spatial extent.
The ﬁdelity of the representation directly depends on the number of Gaussians in the mixture, which means that, given an adequate ﬁtting algorithm and sufﬁcient resources, it can be arbitrarily adjusted.
Hence, GMs allow trading representation accuracy for memory requirements on a more ﬁne-grained level than voxel grids.
In this work, we present our novel architecture, the Gaussian mixture convolution network (GMCN), as an alternative to conventional CNNs.
An important distinguishing feature of GMCNs is that both, data and learnable kernels, are represented using mixtures of Gaussian functions.
We use
Gaussians with unconstrained weights, positions, and covariances, enabling the mixtures to adapt more accurately to the shape of data than a grid of discrete samples could.
Unlike probability models, our approach supports negative weights, which means that arbitrary data can be ﬁtted.
Especially for sparse data sets and problems, such a representation can lead to considerable memory savings.
At their core, CNNs are deep neural networks, where each pixel or voxel is a neuron with distinct connections to other neurons, with shared weights and biases.
On the other hand, GMCNs represent data as functions.
Hence there are no distinct neurons, and the concept of connection does not exist.
Our architecture could thus be more appropriately classiﬁed as a deep functional network .
Like conventional (discrete) CNNs, our proposed GM learning approach supports several convolution layers, an increasing number of feature channels, and pooling, in order to learn and detect aspects of the input.
For the convolution of two mixtures, a closed-form solution is used, which results in a new mixture.
Conventional transfer functions like ReLUs do not produce a GM.
However, a GM is required to feed the next convolution layer in a deep network.
Therefore, we propose to ﬁt a GM to the result of the transfer function.
The ﬁtting process can simultaneously act as a pooling layer if the number of ﬁtted components is reduced appropriately in successive convolution layers.
Unlike most point convolutional networks, which have per-point feature vectors, the feature channels of GMCNs are not restricted to share the same positions or covariance matrices.
With the presentation of GMCNs, we thus provide the following scientiﬁc contributions:
• A deep learning architecture based on the analytical convolution of Gaussian mixtures.
• A heuristic for ﬁtting a Gaussian mixture to the result of a transfer function.
• A ﬁtting method to both reduce the number of Gaussians and act as a pooling layer.
• A thorough evaluation of GMCNs, including ablation studies.
R ELATED WORK
CNNs have been successfully applied to images (e.g., Krizhevsky et al.
and small 3D voxel grids (e.g., Maturana & Scherer (2015), and Ji et al.
However, it is not trivial to apply discrete convolution to large 3- or more dimensional problems due to the dimensional explosion.
Riegler et al.
(2017) reduce memory consumption by storing data in an octree, but the underlying problem remains.
GMs have been used in 3D computer graphics, e.g., by Jakob et al.
(2011) for approximating volumetric data, and by Preiner et al.
(2014) and Eckart et al.
(2016a)
for a probabilistic model of point clouds.
In particular, the work by Eckart et al.
can be used to ﬁt GMs to point clouds, which can then be used as input to our network (see ablation study in Section 5.3).
We use the expectation-maximization (EM) algorithm proposed by Dempster et al.
(1977) to ﬁt input data, and
EM equations for ﬁtting small GMs to larger ones by Vasconcelos & Lippman (1999).
We refer to Ahmed et al.
(2018) for a general overview of deep learning on 3D data.
Like point clouds, GMs are unordered but with additional attributes (weight and covariance).
These additional attributes could be processed like feature vectors attached to points.
Therefore, prior work on point clouds is relevant, whereby Guo et al.
(2019) provide a good overview.
Several authors propose point convolutional networks, e.g., Li et al.
(2018b), Xu et al.
Hermosilla et al.
, Thomas et al.
(2019), Boulch (2019), and Atzmon et al.
The one by
Atzmon et al.
is most similar to ours.
They add a Gaussian with uniform variance for every point, creating a mixture that is a coarse ﬁtting of the point cloud.
After analytical convolution, the result is sampled at the original locations of the point cloud.
However, information away from these points is lost.
Effectively, such a restricted convolution creates a weighting function for neighboring points.
Radial Basis Function Networks also learn a GM to detect patterns on the input data (Broomhead &
Lowe, 1988).
However, the goal of such networks differs from our approach.
While their goal is, similar to MLPs, to detect patterns on a set of data points, our method uses radial basis functions to detect patterns using a convolution operator on another set of radial basis functions.
G AUSSIAN MIXTURES AND CONVOLUTION IN MULTIPLE DIMENSIONS
A multidimensional Gaussian function is deﬁned as
[Equation]
where k is the number of dimensions, a the weight, b the shift or centre point, and C the shape (covariance matrix) of the Gaussian.
A GM is the sum of multiple such Gaussians:
[Equation]
A GM is not a probabilistic model since the weights a can be negative and do not necessarily sum up to 1 .
The convolution of two Gaussians according to Vinga (2004) is given by
[Equation]
Due to the distributive property of convolution, we have:
[Equation]
where g i ( x ) is a short hand for g( x , a i , B i , C i ) .
Clearly, the convolution of one mixture with N terms with another mixture with M terms yields a mixture with N × M terms.
G AUSSIAN MIXTURE CONVOLUTION NETWORKS
Conventional CNNs typically consist of several feature channels, convolution layers, a transfer function, and pooling layers.
In this section, we will show that all these components can be implemented by GMCNs as well.
As a result, our overall architecture is similar to discrete CNNs, except that data and kernels are represented by Gaussian functions rather than pixels or voxels.
Convolution layer.
All batch elements and feature channels contain the same number of components, allowing for packing Gaussian data (i.e., weights, positions, and covariances) into a single tensor.
Like in conventional CNNs, each output channel ( F o ) in a GMCN is the sum of all F i input channels, convolved with individual kernels (Figure 1).
In practice, this means that multiple convolution results are concatenated in the convolution layer.
This results in a mixture with
N o =
F i N i N k components, where N i and N k are the number of Gaussians in the input and kernel mixtures, respectively.
Transfer function.
The ReLU is a commonly used transfer function, whose ramp-like shape is deﬁned by ϕ ( x ) = max(0 , x ) .
When applying this transfer function to a mixture gm , we get
[Equation]
We can use Equation 5 to evaluate the mixture at particular locations of x , as shown in Figure 2b.
However, any subsequent convolution layers in a GMCN will again require a GM as input.
Another problem inherent to the deﬁnition of GM convolution in Section 3, is that the number of intermediate output components (and thus implied memory requirements of hidden layers) is set to increase with each convolution.
We address both of these problems by introducing a dedicated ﬁtting step that produces a new, smaller mixture to represent the result of ϕ (gm( x )) and act as input for subsequent
GMCN layers.
4. G AUSSIAN MIXTURE FITTING
Fitting a GM to the output of the transfer function is not trivial.
In order to propagate the gradient to the kernels, the ﬁtting method must be differentiable with respect to the parameters of the Gaussian components in gm .
We decided to ﬁrst ﬁt a dense GM to the transfer function output (e.g., a ReLU ϕ (gm( x )) , shown in Figure 2b), and then reduce the number of Gaussians in a separate step.
Dense ﬁtting.
To reduce the degrees of freedom that we must optimize for during ﬁtting, we ﬁrst assume that some Gaussians in the output of the last convolution step are meaningfully placed.
Hence, we freeze their positions and adjust only their weights.
One possible solution to adjust the GM’s weights would be to set up an overdetermined linear system, deﬁned by a set of sample points in the mixture and the corresponding output values.
This set of equations could then be fed to a solver for determining the optimal GM weights to ﬁt the outputs according to least squares.
However, our experiments have shown that this approach quickly becomes intractable, as it requires too many samples to achieve a ﬁtting that is both stable wrt.
their selection and generalizes well to locations that lie between selected points (see Appendix C for details).
To avoid the implied restrictive memory overhead and severe bottleneck it would cause, we chose to pursue a less resource-heavy solution and propose the following heuristic instead:
First, a coarse approximation ( a (cid:48) ) of the new weights is computed according to
[Equation]
where a i are the weights of the old mixture, and i runs through all Gaussians.
The new weights a
[Equation]
are then computed using a correction based on the transfer function
[Equation]
where B contains the positions, C the covariance matrices, and ϕ represents the ReLU.
Effectively, the fraction computes a correction term for the coarse approximation a (cid:48) .
It is guaranteed, that the new weights a (cid:48)(cid:48) are positive, just like we would expect from a ReLU.
Examples of the coarse approximation ( a (cid:48) ) are given in Figure 2c, and the ﬁnal ﬁtting ( a (cid:48)(cid:48) ) in Figure 2d.
The ﬁtting is most precise at the Gaussian centers.
Problems arise at the discontinuity of the ReLU, where the result can be overly smooth, as seen in Figure 3a, or with an adverse conﬁguration of covariances, as is the case in Figure 3b.
[Figure]
-  +
Reduction step.
The next step in our ﬁtting performs expectation-maximization (EM) to reduce the number of Gaussians in the mixture.
We implemented two variants, a simpler modiﬁed EM variant based on the derivations from Vasconcelos & Lippman (1999), and our own tree-based hierarchical
EM method.
For the simpler, modiﬁed EM variant, we select the N p Gaussians with the largest integrals out of the N o available ones.
Then, one step of EM is performed, balancing accuracy with compute and memory resources, resulting in the ﬁtting.
However, this procedure implies O ( N o × N p ) in terms of memory and computation complexity, rendering it too expensive for larger mixtures.
To remedy this issue, we propose tree-based hierarchical expectation-maximization (TreeHEM).
A fundamental idea of TreeHEM is to merge Gaussians in close proximity preferentially.
To this end, the Gaussians are ﬁrst sorted into a tree hierarchy, based on the Morton code of their position, similar to Lauterbach et al.
This tree is traversed bottom-up, collecting up to 2 T Gaussians and performing a modiﬁed EM step, which reduces the number to T Gaussians.
The ﬁtted Gaussians are stored in tree nodes.
The process continues iteratively, collecting again up to 2 T Gaussians followed by ﬁtting, until we reach the root.
Finally, the tree is traversed top-down, greedily collecting the
Gaussians with most mass from the nodes until the desired number of ﬁtted Gaussians is reached.
Hence, TreeHEM yields an O ( N o log N o ) algorithm.
We found that a value of T = 2 works best in terms of computation time, memory consumption, and often ﬁtting error (Section 5.3).
TreeHEM was implemented in CUDA; gradients were derived manually.
Examples are shown in Figure 2e.
R EGULARIZATION AND MAINTAINING VALID KERNELS
In order to add support for regularization to the learning process, we can apply weight decay, which pushes weights towards zero and the covariance matrices towards identity:
[Equation]
where d i is the weight decay loss for kernel Gaussian i , and a , C , and I are the weights, covariances, and the identity matrix, respectively.
Training the GMCN includes updates to the covariance matrices of kernel Gaussians.
In order to prevent singular matrices, C l is learned, and then turned into a symmetrical non-singular matrix by
[Equation]
4. P OOLING
In conventional CNNs, pooling merges data elements and reduces the (discrete) domain size at the same time.
As ﬁlter kernels remain identical in size but are applied to a smaller domain, their receptive ﬁeld effectively increases.
For pooling data with GMCNs, we can set up the ﬁtting step (Section 4.1) such that the number of ﬁtted Gaussians is reduced by half on each convolution layer.
While this effectively pools the data
Gaussians by increasing their extent, it does not scale the domain.
Thus, using the same kernels on the so-ﬁtted data does not automatically lead to an increased receptive ﬁeld.
To achieve this, the kernel
Gaussians would have to increase in size to match the increase in the data Gaussians’ extent.
Instead, similar to conventional CNNs, we simply reduce the data mixtures’ domain by scaling positions and covariances.
We compute the scaling factor such that after each layer, the average trace of the ReLU + fit N p components GCL ( F i x F o / N p ) convolution F i x F o covariance norm G a u ss i a n s i n t e g r a t e G a u ss i a n s c onv e n ti on a l B
N c onv e n ti on a l l og s o f t m a x
G C L ( 1x / 64 )
G C L ( 8x / 32 )
G C L ( 16x / 16 )
G C L ( 32x / 8 )
G C L ( 64x / a ll ) covariance matrices becomes the number of dimensions ( 2 or 3 currently), leading to an average variance of 1 per dimension.
Effectively, this leads to Gaussian centers moving closer together and covariances becoming smaller.
Hence, the receptive ﬁeld of each kernel becomes larger in relation to the original data scale, even if the kernel size is ﬁxed.
E VALUATION
To evaluate our architecture, we use the proposed GMCN architecture to train classiﬁcation networks on a series of well-known tasks.
We used an NVIDIA GeForce 2080Ti for training.
For all our experiments, we used the same network architecture and training parameters shown in Figure 4.
First, we ﬁt a GM to each input 2D or 3D sample by using the k-means algorithm for deﬁning the center of the Gaussians and one step of the EM algorithm to compute the covariance matrices.
The mixture weights are then normalized such that the mixtures have the same activation across all training samples on average.
Covariance matrices and positions are normalized using the same procedure as for pooling (Section 4.3).
The input GM is then processed by four consecutive blocks of Gaussian convolution and ReLU ﬁtting layers, where each doubles the number of feature channels and halves the number of Gaussians.
The resulting numbers of channels per block are [8 , 16 , 32 , 64] .
Finally, an additional convolution and ReLU block outputs one feature channel per class.
For performance reasons, we do not reduce the number of Gaussians in this last block.
These feature channels are then integrated to generate a scalar value, which is further processed by a conventional batch-normalization layer.
The resulting values are converted to probabilities by using the Softmax operation, from which the negative-log-likelihood loss is computed.
The kernels of each convolutional layer are represented by ﬁve different Gaussians, with weights randomly initialized from a normal distribution with variance 1 . 0 and mean 0 .
One Gaussian is centered at the origin, while the centroids of the remaining ones are initialized along the positive and negative x- and y-axes at a distance of 2 .
5 units from the origin.
For the 3D case, the remaining
Gaussians are placed randomly at a distance of 2 .
5 units from the origin.
The covariance factor matrix of each Gaussian is initialized following ( I + R ∗ 0 . 05) ∗ 0 .
7 , where I is the identity, and R a matrix of random values in the range between − 1 and 1 .
The covariance matrix was then computed according to Equation 9.
We determined these values using a grid search.
Examples of kernels are shown in Appendix A.
All models are trained using the Adam optimizer, with an initial learning rate of 0 .
The learning rate is reduced by a scheduler once the accuracy plateaus.
Moreover, we apply weight decay scaled by 0 .
1 of the learning rate to avoid overﬁtting, as outlined in Section 4.2.
5. 2D DATA CLASSIFICATION (MNIST)
The MNIST data set (Lecun et al., 1998) consists of 70 K handwritten digits from 10 different classes.
In this task, models have to predict the digit each image represents, where performance is measured as overall accuracy.
We trained our GMCN architecture using the standard train/test splits by ﬁtting
Gaussians to each image and processing them with our model.
Figure 5 shows some examples of the generated GMs.
Our trained model achieved an accuracy of 99 . 4 % on this task, which is close to the 99 . 8 % currently reported by other state-of-the-art methods (Byerly et al., 2020; Mazzia et al., 2021).
However, when compared to a multi-layer perceptron (Simard et al., 2003) ( 98 . 4 % ) or vanilla CNN
[Table]
architectures (Lecun et al., 1998) ( 99 . 2 % ), which more closely reﬂect the GMCN design we used, our method performs better.
Compared to point-based methods, our network achieves competitive performance (PointNet (Qi et al., 2017a)
[Table]
2 % and PointNet++ (Qi et al., 2017b) 99 .
[Table]
5. 3D DATA CLASSIFICATION (M ODEL N ET 10)
ModelNet10 (Zhirong Wu et al., 2015) is a data set used to measure 3D classiﬁcation accuracy and consists of 3D models of different man-made objects.
The data set contains 4 .
8 K different models from 10 different categories.
We trained our network to classify ModelNet10 samples using the standard train/test splits.
In order to ﬁt the 3D data with a GM, we used the point clouds provided by Qi et al. (2017b), which were obtained by randomly sampling the surface of the 3D models.
We then ﬁt 128 Gaussians to these point clouds to generate the input for our network.
Our trained model achieved an accuracy of 93 .
3 % , which is similar to other state-of-the-art classiﬁers: 94 . 0 %
(Klokov & Lempitsky, 2017) or 95 .
7 % (Li et al., 2018a).
In Table 1, we again compare our method with PointNet (Qi et al., 2017a) and PointNet++ (Qi et al., 2017b), as they represent two well-established methods for point cloud processing.
To enable a fair comparison with each point-based technique, we ran two variants: The ﬁrst uses a point cloud as input, with a number of points equivalent to the memory footprint of our GM representation.
The second uses the same GM inputs as our network, where weight and covariance matrix are processed as additional features for each point.
Models are trained with various input sizes: 32 Gaussians (106 points), 64 Gaussians (213 points), and 128 Gaussians (426 points).
We found that our method outperforms PointNet on all experiments using an order of magnitude fewer parameters.
It also achieves similar performance to the point-based hierarchical architecture PointNet++ using ﬁve times fewer parameters.
Our method outperforms both architectures when processing GMs directly.
5. A BLATION STUDIES
In this section, we discuss additional experiments carried out to validate the individual elements of our network.
For these ablation studies, we use the ModelNet10 data set in all experiments.
Network depth.
First, we evaluate the network’s performance wrt.
the depth of our GMCN to determine the optimal number of layers.
Table 2 shows the classiﬁcation accuracy on the ModelNet data set for different models with varying numbers of convolution layers.
We conﬁrmed that GMCN can beneﬁt from additional layers; however, the accuracy stops increasing once we reach ﬁve layers.
[Figure]
[Table]
Gaussian mixture ﬁtting.
We evaluated the two proposed ﬁtting algorithms (Section 4.1) used after each convolution layer.
Three different reduction variants were tested: modiﬁed EM, TreeHEM with T = 2 and 4 Gaussians per node.
Table 3 shows the results of these methods in terms of ﬁtting error in the ﬁrst three layers and the accuracy of the model when using 64 and 128 Gaussians as input, respectively.
TreeHEM obtains a lower ﬁtting error in most of the layers when compared to the EM algorithm.
Moreover, the tree-based method also requires less memory during training compared to EM, which runs out of memory when trained on inputs represented by 128 Gaussians.
However,
EM obtains a slightly higher classiﬁcation accuracy for 64 input Gaussian.
Out of the two TreeHEM variants, the two-Gaussian approach performs better in ﬁtting error and classiﬁcation accuracy.
Input data ﬁtting.
Lastly, we evaluate how the model is affected by the different algorithms used to ﬁt the input data.
We compare four different methods: k-means initialization plus one single step of
EM ( k-means ), k-means initialization plus multiple EM steps ( k-means+EM ), random initialization plus multiple EM steps ( rand+EM ), and the method proposed by Eckart et al.
(2016b) ( Eckart ).
Figure 6 shows the resulting GMs of the four different algorithms for two different models from
ModelNet10.
We see that k-means produces a set of Gaussians with relatively uniform shapes, but overall does not ﬁt the data accurately.
Random init + EM and Eckart can ﬁt the model with high precision but also produces some blurred areas.
K-means init + EM achieves the best representation of the four methods.
The resulting accuracy of our network for the four types of input GMs is reported in Table 4.
K-means initialization achieves the best accuracy while Eckart performs worst.
We believe these results can be explained by the uneven Gaussian covariance matrices present on methods such as Eckart and rand+EM .
[Table]
EM
[Table]
[Table]
[Table]
92.2 % n/a 2.5GB n/a 6h n/a
Tree (4) 0.
91.3 % 92.9 % 1.6GB 6.8GB 5h 22h
Tree (2) 0.
92.0 % 93.4 % 1.3GB 5.6GB 4h 19h
[Table]
D ISCUSSION AND CONCLUSION
To the best of our knowledge, we are the ﬁrst to propose a deep learning method based on the analytical convolution of GMs with unconstrained positions and covariance matrices.
We have outlined a complete architecture for the task of general-purpose deep learning, including convolution layers, a transfer function, and pooling.
Our evaluation demonstrates that this architecture is capable of obtaining competitive results on well-known data sets.
Its ability to avoid the curse of dimensionality seems evident: even in comparison with sparse, point-based methods, it achieves comparable accuracy using between 5 × and 10 × fewer parameters.
Hence, we consider our architecture as a promising candidate for applications that require higher-dimensional input or compact networks.
Limitation and future work.
Currently, the most signiﬁcant bottleneck to limit the universal application of our work is the—unavoidable—ﬁtting process.
As reﬂected by the results in Table 2, the implied computation time limits the size of GMCNs that can be trained with consumer-grade hardware in an appropriate amount of time.
The simple GMCN architecture presented in Section 5 is insufﬁcient for achieving competitive results on more complex tasks, such as classiﬁcation on
ModelNet40 and FashionMNIST data sets (87% and 86% accuracy, respectively).
However, early experiments have shown that performance can be drastically improved by embedding a ﬁtting step in the convolution stage.
Speciﬁcally, we can use an on-the-ﬂy convolution of the positions to construct an auxiliary tree data structure, in a manner similar to Section 4.1.
The resulting tree is then used to perform a full convolution and ﬁtting in one stage.
We have identiﬁed highly parallel algorithms for the required steps and conﬁrmed that applying these optimizations to our methods can signiﬁcantly reduce memory trafﬁc and improve computation times by more than an order of magnitude.
We further conjecture that the achieved prediction quality with our architecture is currently limited by ﬁtting accuracy.
This could be addressed by sampling the GM and performing one or two EM steps on these samples.
Doing so should become feasible once the number of Gaussians is brought down using the convolution ﬁtting step described above.
Presumably, even a slight increase in accuracy would enable us to achieve state-of-the-art results on all tested use cases.
Additional extensions to raise the applicability of GMCNs would be the introduction of transposed convolution, as well as skip connections (Ronneberger et al., 2015).
We expect that these tasks could be solved with a reasonable amount of additional effort.
We hope to enable novel solutions for 3-dimensional problems with these extended GMCNs in future work, such as hole-ﬁlling for incomplete 3D scans or learning physical simulations for smoke and ﬂuids.
R EPRODUCIBILITY
The source code for a proof-of-concept implementation is provided in the supplemental material (Github after deanonymization).
A K ERNEL EXAMPLES
Figure 7 shows examples of training kernels for a 2d dataset.
B T HEORETICAL MINIMAL MEMORY FOOTPRINT ( AND NUMBER OF PARAMETERS )
In conventional CNNs, the convolution operation, that is “per-pixel kernel-sized window, elementwise multiplication, and sum”, are usually built into one compact function, or CUDA kernel.
Accordingly, intermediate results are never stored in memory.
The proof-of-concept implementation described in this paper is not compact in the same way.
The closed-form solution of the convolution with GMCNs yields a large number of intermediate output Gaussians, which are then greatly reduced in the ﬁtting step before being forwarded to the next layer.
The described prototype explicitly stores these intermediate results, making it easier to implement and explain.
However, as outlined in Chapter 6, the integration of a ﬁtting step directly into the convolution is possible with our approach, albeit more complex compared to conventional CNNs.
The purpose of this appendix is to provide formal equations and concrete examples to outline the theoretical minimum memory footprint of a GMCN and assess its overall potential for reducing memory consumption.
For this, we made the following assumptions:
• Fitting is integrated into the convolution layer and produces a mixture with a speciﬁed number of Gaussians N o .
No ﬁtting is performed if the speciﬁed number is equal to the number of Gaussians produced by the convolution.
None of the modules (convolution / reduction ﬁtting, transfer function ﬁtting, pooling) contain caches.
• Programming code is not counted.
• Minimal storage for a 2d Gaussian requires 6 ﬂoats, and for a 3d Gaussian 10 ﬂoats. [Table] [Table]
Convolution module A convolution module has F i input feature channels, F o output feature channels, batch size is B , incoming data mixtures contain N i Gaussians, kernels contain N k Gaussians, the integrated convolution-ﬁtting module is set to produce N o Gaussians, and the module contains a pooling layer, which reduces the number of Gaussians to N p .
Memory of the incoming data is accounted towards the previous module.
The minimal memory footprint in terms of Gaussians ( G ) is the sum of kernels ( K ), and data ( D ) times 2 (for the gradient):
[Equation]
K can be used to compute the number of parameters of the given module by multiplying with the representation used.
Model The factors and results for the model used for our evaluation (shown in Figure 4) are given in Tables 5 and 6.
Overall, if N o = N i is used, the model has a theoretical minimal memory footprint of 434,256 total 3D Gaussians, which is less than 17 megabytes. In comparison, the proof-of-concept implementation presented in this paper has a theoretical use of 781 megabytes. In practice, it requires several times more, since it uses convenience data structures that were not exhaustively optimized.
C C OMPARISON OF LEAST - SQUARES SOLUTION AND HEURISTIC FOR R E LU FITTING
In this section, we will compare a least-squares solution with our heuristic presented in Section 4.
for the ﬁtting of a transfer function.
The least-squares solution has to solve the following problem
[Equation]
(11) where t i = ϕ
(gm( D i )) are the target values at positions D , and A is a matrix:
[Equation]
[Equation]
[Equation]

[Equation] [Table]
with γ i ( x ) = [Equation]
[Table]
[Equation]
B i is the centroid, C i the covariance of Gaussian i , N is the number of Gaussians, and k the number of dimensions (e.g., 2 or 3).
In other words,
A contains the evaluations of the mixture Gaussians with the weight set to one, on a set of points contained in D .
We evaluated several choices for D :
• D = B , evaluate the mixture at the centre points of its
Gaussians • D = rand ( M, k ) , positions within the domain • D = concat ( B, rand ( M, k )) , mixture centres and random positions within the domain
Figure 8 shows part of our test input, along with the result of our heuristic.
After seeing the results in
Figure 9 and timings in Table 7, we refrained from computing the RMSE and selected the heuristic for our network.
The least-squares solution overshoots away from the sampling points, and the compute resources are several magnitudes higher compared to our heuristic.
The matrix A with choice D = B can be of rank ( A ) < N , and therefore (non-pseudo) inversion methods are not applicable.
D D ETAILS ON THE DENSE FITTING
(This section is still work in progress and will be updated before November 22.)
The purpose of this step is to ﬁt a transfer function, for example a ReLU.
Our ﬁtting method only computes new weights for the mixture. Position and covariance matrices of the Gaussians are not changed (see 4.1).
Accordingly, we give only equations for the new weights a (cid:48)(cid:48) , which are computed in 2 steps.
The ﬁrst step computes a coarse, or initial approximation to the target:
[Equation]
where a i are the weights of the old mixture, and i runs through all Gaussians.
This design has two reasons: We think that it is safer if the weights are positive only, which is guaranteed, since the second step will not introduce a negative sign.
Negative weights themselves are not a problem for the
GMCN architecture.
However, the ReLU gives a function that is positive through all of the domain, and so should the ﬁtting.
That still does not conﬂict with negative weights as long as the sum is greater zero throughout the domain.
But we look only at the evaluation at the Gaussian’s centroids, and negative weights could cause the sum to become negative away from these evaluation points.
Something similar happens when using a linear solver (Section C).
As a side effect, reduction ﬁtting becomes simpler if it does not have to deal with negative weights.
Secondly, we want to scale the
Gaussians (change the weight) as little as possible, therefore we do not initialize a (cid:48)
i simply to 1.
The rational here is, that we want to retain the shape of the mixture as much as possible without looking at the positions and covariance matrices.
As an example, take 2 Gaussians with the same centroid, one with a large variance and small weight, the other with a small variance and large weight.
This would give a large Gaussian-shaped arch with a strong bump in the middle.
Now, if we initialized the weights to one, the result at the centroid would be approximately correct, but the shape would change.
The new weights a (cid:48)(cid:48) are then computed using a correction based on the transfer function
[Equation]
[Equation] [Figure] [Figure] where B contains the positions, C the covariance matrices, and ϕ represents the ReLU.
The fraction computes a correction factor for each Gaussian by evaluating the target function and the current approximation at the Gaussians centroid.
This computation is O ( N 2 ) in the number of Gaussian components.
In order to avoid storage of intermediate results and a large gradient tensor, we implemented the evaluation in a custom CUDA kernel.
On the CPU it would be possible to trade accuracy for performance by ignoring contributions that are below a certain threshold, and using a tree based acceleration structure (e.g., a BVH).
However, our tests showed, that this does not scale well on the GPU.
The evaluation kernel is reasonably optimized, still, due to the large number of
Gaussians, which can easily go into thousands, times the number of feature channels, times the batch size, these two evaluations account for roughly half of the overall training/test runtime.
In order to improve performance decisively, either the number of Gaussians would have to be reduced before this ﬁtting step (e.g., during the convolution as outlined in the future work section), or a different algorithm used, or both.
E D ETAILS ON THE REDUCTION STEP
(This section is still work in progress and will be updated before November 22.)
The purpose of this step is to reduce the amount of data by ﬁtting the target with a smaller mixture.
It is applied after ReLU ﬁtting, hence all weights are positive, and the mixture can be turned into a probability distribution easily.
Therefore, probability based methods can be applied directly.
In particular, it would be possible to sample the mixture and then use a expectation maximization (EM) algorithm.
However, that would be prohibitively expensive for the given number of Gaussians.
In the following, we will ﬁrst explain the hierarchical EM algorithm by Vasconcelos & Lippman (1999) and then show, how that maps to our TreeHEM algorithm.
Hierarchical EM Hierarchical EM starts with one Gaussian per point, which is the ﬁrst level.
In the successive levels ( l + 1 ), a smaller mixture is ﬁtted to the previous level ( l ).
The new mixture needs to be initialized ﬁrst, for instance by taking a random subset of the mixture in level l .
In the E step, responsibilities are computed using (equations taken from Preiner et al.
[Equation]
16) where ˆ w i is the number of virtual samples.
In the original paper, N points were ﬁtted, so ˆ w i =
Nw i corresponded to the number of points that Gaussian was modeling.
We do not have points, so N becomes an implementation deﬁned constant.
Afterwords, the M step can be taken:
[Equation]
where w , µ , and Σ are the weight, centroid, and covariance matrix, respectively, i is the index of the (larger) target mixture, s the index of the new mixture.
In each level, only one iteration is taken (E and
M step), and the algorithm continues with the next level.
Preiner et al.
(2014) propose a geometrical regularization, in which r is is set to zero, if the and s is larger than a certain threshold.
We were not able to use that method for reasons including difﬁcult parallelization on the GPU, bad accuracy on our data, and bad performance, since every Gaussian needs to be tested against each other Gaussian.
TreeHEM We therefore developed the TreeHEM algorithm to improve on theses issues.
In short, we build a spatial tree of the (larger) target Gaussians, and use it to localize the ﬁtting, i.e., limit the ﬁtting to a ﬁxed-size subset of Gaussians in the vicinity.
The tree is built using the same algorithm, as is described by Lauterbach et al.
(2009) for building bounding volume hierarchies of triangles (LBVH), minus the bounding boxes.
In our case, we take the centroids, compute their Morton code, and use it to sort the mixture.
This puts the Gaussians on a
Z-order curve.
It is then possible to build a tree in parallel by looking at the bit-pattern of the sorted
Morton codes.
Please look at the excellent paper from Lauterbach et al.
for more details.
Storing the indices of children and parents in the tree allows us to traverse bottom-up and top-down.
The tree is traversed twice.
In the ﬁrst, bottom-up, pass we collect and ﬁt Gaussians, and then store them in a cache with up to T Gaussians in the tree nodes.
Bottom-up traversal is implemented in a parallel way, the ﬁrst thread entering simply exits, and the second does the work.
First, the Gaussians from the caches of the two children are collected, resulting in up to 2 T Gaussians.
If the children are leaves, or not yet full, then that number can be below T , in which case they are written into the cache.
Otherwise T new Gaussians are ﬁtted to the collected ones, stored in the cache, and traversal continues until reaching the root node.
For the ﬁtting, we ﬁrst select T initial Gaussians from the 2 T collected ones.
This is done by dividing the 2 T Gaussians into T clusters.
From each cluster, the Gaussian with largest weight is selected, giving T initial Gaussians.
The rational is to try selecting the most important Gaussians without overlap.
We then use Equations 15–20 to compute the ﬁtting, where we set l + 1 to the initial mixture explained just now, and l to the target mixture consisting of 2 T Gaussians.
At this point we have a tree, with T Gaussians in each node, that ﬁt their children.
In the second, top-down, pass we look to select N/T nodes that best describe the target mixture, where N is the number of Gaussians in the ﬁtting.
This was implemented as a greedy approach with a single thread per mixture (which is not a problem as the kernel only traverses the very top of the tree).
All selected nodes are stored in an vector, that only contains the root node at the beginning.
In each iteration, we search through the vector and select the node with largest mass (integral of Gaussians, i.e., sum of weights) and replace it with its two children.
The iteration terminates once we selected N/T nodes.
Finally, we go through these nodes, and copy the Gaussians from the cache into the destination tensor.
Gradient computation Since all of the above is implemented in a custom CUDA kernel, we do not have the usual automatic gradient computation.
Therefore we had to compute the gradient manually.
We implemented unit tests of our manual gradient against a gradient computed by the autodiff C++ library (Leal, 2018).
F A PPLICATION OF THE R E LU TO EACH G AUSSIAN INDIVIDUALLY
(This section is still work in progress and will be updated before November 22.)
Repeatedly the question came up, whether the ReLU ﬁtting could be simpliﬁed by applying the transfer function directly in parameter space, i.e., on the weight vector.
In other words, whether an approximation of the Gaussians by Diracs for the purpose of activation would be a possible solution.
Unfortunately that is not the case.
Currently, if a negative Gaussian is on top of a positive one, then the negative would reduce the contribution of the positive one.
The positive one could even disappear.
This would not be possible, if the ReLu would be applied directly.
All negative Gaussians would be cut away, while the positive ones would simply pass through.
Effectively, this would disable the nonlinearity, and make the network only capable of separating 2 classes using a hyperplane.
G C OMPLETE RESULTS
(This section is still work in progress and will be updated before November 22.)
If nothing else given, then the tables show [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
TreeHEM
[Table]
= [Table] [Table] =
[Table]
[Table]
[Table] [Table]
[Table]
TreeHEM (T=2)
[Table]
TreeHEM (T=4) 0.
[Table]
[Table]
MEM n/a n/a n/a
TreeHEM (T=2) n/a 0.
n/a
TreeHEM (T=4) n/a 0.
n/a
[Table]
[Table]
kmeans + em 82, 92, 92, random + em 80, 91, 92, eckart 80, 90, 91, kmeans 84, 93, 93, epochs:  batch kernels params time 1/128 - 8/64 - 16/32 - 32/16 - 64/8 - 128/477, 77,    4d 23h 17m 1/128 - 8/64 - 16/32 - 32/16 - 64/887, 87,    2d 14h 57m 1/128 - 8/64 - 16/32 - 32/16 - 64/886, 87,    2d 12h 7m