G EOMETRY AWARE CONVOLUTIONAL FILTERS FOR OMNIDIRECTIONAL IMAGES REPRESENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Due to their wide ﬁeld of view, omnidirectional cameras are frequently used by autonomous vehicles, drones and robots for navigation and other computer vision tasks.
The images captured by such cameras, are often analyzed and classiﬁed with techniques designed for planar images that unfortunately fail to properly handle the native geometry of such images.
That results in suboptimal performance, and lack of truly meaningful visual features.
In this paper we aim at improving popular deep convolutional neural networks so that they can properly take into account the speciﬁc properties of omnidirectional data.
In particular we propose an algorithm that adapts convolutional layers, which often serve as a core building block of a CNN, to the properties of omnidirectional images.
Thus, our ﬁlters have a shape and size that adapts with the location on the omnidirectional image.
We show that our method is not limited to spherical surfaces and is able to incorporate the knowledge about any kind of omnidirectional geometry inside the deep learning network. As depicted by our experiments, our method outperforms the existing deep neural network techniques for omnidirectional image classiﬁcation and compression tasks.
I NTRODUCTION
Drone vision, autonomous cars and robot navigation systems often use omnidirectional cameras, as they allow recording the scene with wide ﬁeld of view.
Despite their obvious advantages, images obtained by such cameras have different statistics compared to planar images.
Nevertheless, omnidirectional images are often processed with standard techniques, which are unfortunately poorly adapted to the speciﬁc geometry of such images.
In this paper we improve one of the most popular frameworks for image processing, namely convolutional neural network (CNN) for omnidirectional images.
CNNs prove to be effective, as they permit to achieve very good performance in many different tasks like image classiﬁcation, segmentation, generation and compression.
In the context of omnidirectional cameras, CNNs are typically applied directly to the unwrapped and distorted spherical images.
This approach, however, is suboptimal: due to speciﬁc geometry of these images, and, in particular, the change in the image statistics with the position in the image. The latter forces the network to learn different ﬁlters for different locations in the omnidirectional images (see Fig. 1).
To solve this issue, we replace ordinary convolutional ﬁlters with the graph-based ones that can adapt their size and shape depending on the position in the omnidirectional image thanks to the ﬂexible graph structure.
In order to overcome the common limitation of these graph-based ﬁlters being isotropic (i.e. invariant to the changes in the objects orientation in the image)
we suggest to use multiple directed graphs instead of a single undirected one, as used in (Defferrard et al., 2016; Kipf & Welling, 2017; Khasanova & Frossard, 2017a).
This permits our method to encode the orientation of the objects that appear in the images and, therefore, extract more meaningful features from them.
This together with the inherent ability of our ﬁlters to adapt to the image projective geometry allows our method to reach state-of-the-art performance when working not just with spherical projections, as done by Khasanova & Frossard (2017b);
Cohen et al.
(2018), but virtually with any type of image projective geometry that can be encoded by a graph.
In our experiments we show that apart from the state-of-the-art results on the regular omnidirectional images classiﬁcation task, our approach outperforms the existing classiﬁcation techniques when applied to the images projected on a randomly perturbed spherical surface or on a cube via the cube-map projection, which has recently become one of the popular ways to represent 360-images (Chen et al., 2018). Finally, we demonstrate that our method can be applied for orientation-dependent task such as compression and reduce artifacts comparing with a standard approaches.
R ELATED WORK
In this section we ﬁrst brieﬂy introduce the most recent approaches that combine the power of deep learning methods with graph signal processing techniques, as they are similar in spirit to our work. We then discuss in more details the recent trends in omnidirectional image processing. Geometric Deep learning.
In the recent years a number of deep learning methods have been introduced tailored for processing irregular data that is represented as a graph. One way to solve this problem is suggested by Monti et al. (2017), where the authors deﬁne a local system of ddimensional pseudo-coordinates for every node of the graph and learn both the ﬁlters and patch operators that work on these coordinates. A different direction is taken by Wang et al.
(2018), who propose using edge-based convolutional kernels and dynamically update the graph. While being ﬂexible and effective for general tasks, in the speciﬁc context of the omnidirectional images these methods do not directly take the advantage of the knowledge about the projective geometry, which we model using a speciﬁcally designed graph representation. While most of the existing methods work with undirected graphs, the recent work of (Monti et al., 2018) propose an approach for processing data deﬁned on a directed graph by exploiting local graph motifs that describe its connectivity patterns. The main differences of this method with our work are that ﬁrst, this method assumes that the directed graph is already given.
In our problem, building such a graph that is able to fully take advantage of the image projective geometry is one of the contributions. Second, the approach in (Monti et al., 2018) does not use the knowledge of the coordinate system associated with omnidirectional images, which we however use in our architecture, in order to deﬁne ﬁlter orientations. The list of the aforementioned works is by no means extensive, therefore we refer the interested reader to the survey (Bronstein et al., 2017), which summarizes many geometric deep learning methods in detail.
Omnidirectional image processing. The most typical way of dealing with images taken by omnidirectional cameras is to apply standard image processing techniques directly on the equirectangular projection images, which is one of the most common representations for the omnidirectional images (De Simone et al., 2016).
However, due to the strong distortion effects introduced by the process of unwrapping of the projection surface to a plane, standard techniques lose much of their efﬁciency, as the appearance of the same object may change depending on its location in the image. To overcome this problem the recent work of Khasanova & Frossard (2017b) suggests using graph-based special convolutional ﬁlers to adapt to the geometry of omnidirectional images. This method, however, relies on the convolutional operation deﬁned in the spectral domain, which leads to isotropic ﬁlters and may reduce the complexity of trained ﬁlters.
We, on the other hand, propose to build anisotropic graphs-based convolutional ﬁlters that do not have this limitation.
A different direction is taken by the authors of (Su & Grauman, 2017) who suggest adapting the size of the convolutional kernel to the elevation of the equirectangular image.
The main limitation of this technique, however, is that it requires a signiﬁcantly larger number of parameters than the competing techniques, as it does not have the weight sharing property of CNNs.
It rather requires learning different convolutional ﬁlters for different elevations of the equirectangular image.
Further, Jeon & Kim (2017) propose to learn the shape of the convolutional ﬁlter, by learning the sampling locations (position offsets), where the elements of the ﬁlter are evaluated.
The authors of (Dai et al., 2017) extend later this idea and suggest learning dynamic offsets of the ﬁlter elements depending on the image content, which allows the ﬁlters to adapt to different parts of the image.
This method is quite ﬂexible, however in the context of omnidirectional images requires an extensive training set, as the network needs to learn how it should react to various objects appearing at any possible elevation.
In our work we rather take advantage of the knowledge of the image projective geometry and use this knowledge in the design of our architecture to adapt the size and shape of convolutional ﬁlters.
A different approach is suggested by Cohen et al. (2018), who introduces a CNN that is designed for spherical shapes and deﬁne ﬁlters directly on its surface. This method, however, is speciﬁcally designed for processing spherical images, while our approach is easily adapted to different kind of shapes, which we show in our experiments. Further, the methods of (Coors et al., 2018; Tateno et al., 2018) suggest a different way of compensating for the distortion of omnidirectional image.
They suggest adapting the sampling locations of the convolutional ﬁlters to the geometry of the lens by projecting kernels to the sphere and using interpolated pixel values on the projected locations for implementing the convolutional ﬁlters.
While these works are the closest to in spirit to ours, we propose a more general architecture, which permits to adapt the shape and size of the convolutional kernel to the location of omnidirectional image, and therefore to use the information about all the pixels and not only of a subset of them.
Then, the authors in (Monroy et al., 2018; Ruder et al., 2018) suggest a completely different approach to tackle the distortion of omnidirectional images.
Instead of working with equirectangular images, they propose to project an omnidirectional image to a cube, where each of its faces represents an image that would have been seen through a regular perspective camera, with the optical center located in the center of the cube (Chen et al., 2018).
Representing an omnidirectional image in this way allows having less noticeable distortion effects as compared to equirectangular images.
This representation, however, suffers from another type of distortion that appears due to discontinuity effect on the borders between the faces of the cube.
To mitigate this issue, Monroy et al.
(2018) propose to apply a smoothing ﬁlter as a post-processing step and Ruder et al.
(2018) suggest an algorithm that enforces consistency between the neighboring facets of the cube.
Contrary to the mentioned approaches, as we model cube surface as a graph, our algorithm can easily handle the discontinuity problem and adapt to image distortions introduced by the cube-map projection.
G EOMETRY - AWARE CNN
In this section we describe our algorithm, which adapts convolutional ﬁlters to the distortion of omnidirectional images.
We start with the introduction of the equirectangular projection, as it is one of common ways to represent images from the omnidirecitonal cameras (De Simone et al., 2016;
Coors et al., 2018).
We then describe our graph-based representation learning framework.
3. E QUIRECTANGULAR PROJECTION
Omnidirectional visual content can be represented as a sphere with radius r , where the user is assumed to be located at the center.
Each 3D point can be projected to a point on the surface of this sphere, which is described by spherical coordinates, namely a longitude θ ∈ [ − π, π ] and a latitude φ ∈ [ − π 2 , π 2 ] .
Omnidirectional images are generally not processed directly in their native geometry, but they are ﬁrst projected to the 2D plane where classical image processing techniques can be activated.
One of the popular projections involves sampling the data on the sphere with equal steps ∆ θ and ∆ φ , which results in an equrectangular image on the 2D plane.
Thus, each point of equrectangular image is deﬁned by its spherical coordinates.
To describe this projection let us introduce the tangent plane
T , which is tangent to the sphere in the point ( θ 0 , φ 0 ) .
Thus, each point ( x, y ) on the tangent plane is projected to the sphere surface ( θ, φ ) as follows (Coors et al., 2018):
[Equation]
where ρ = (cid:112) ( x 2 + y 2 ) and ν = tan − 1 ρ .
In order to have similar ﬁlter response regardless of the position of the object we model distortion of the applied ﬁlter.
Thus, similarly to the works of (Khasanova & Frossard, 2017b) and (Coors et al., 2018), we deﬁne the ﬁlter kernel on this tangent plane T .
Fig.
1 illustrates a sample equirectangular image with various kernels corresponding to tangent plane at different positions on the sphere.
As we can see the projected area is different for various tangent plane locations.
This projected area deﬁnes the support of our geometry-aware features, as described in the next section.
3. G EOMETRY - ADAPTIVE FILTERS
In the context of omnidirectional cameras, the main drawback of the classical convolutional approach is that it applies the same rectangular ﬁlters to different image positions.
However, as we mentioned before, equirectangular images have different statistics at various elevations.
Therefore, we propose to adapt the size and shape of the ﬁlter to the elevation on the spherical image.
To do so we propose to use a graph-based approach, which has recently become popular.
The nodes of this graph v i ∈ G and signal y ( v i ) deﬁned on the nodes represent pixels and their intensity values respectively.
Based on this graph G we can then use the Laplacian polynomial ﬁlters F as proposed in (Defferrard et al., 2016; Khasanova & Frossard, 2017a), where normalized Laplacian operator is deﬁned as follows: [Equation]
where I is an identity matrix, D is a diagonal degree matrix and A is an adjacency matrix, which for each node v p of graph G deﬁnes its neighborhood N p .
Then, F has the following form:
[Equation]
where the α l are the trainable parameters, L is a Laplacian matrix and M is the degree of F .
Using these ﬁlters we can construct a Deep learning architecture and use it for various tasks, e.g., classiﬁcation of omnidirectional images.
The main advantage of our approach is that by appropriately constructing the graph G we make the Laplacian polynomial ﬁlters F react similarly to the same object seen at different elevation of the equirectangular image regardless of geometric distortion due to the spherical geometry.
Here we call those ﬁlters geometry-aware (GA).
In order to adapt the GA-ﬁlter F to the elevation we build a graph G in such a way that the neighbourhood of each node is different for different elevations of the omnidirectional image.
In the following section we describe two approaches that rely on undirected and directed graphs respectively, which consequently deﬁne isotropic and anisotropic ﬁlters.
Then we describe in more details the polynomial anisotropic ﬁlter F that is used for directed graphs..
Undirected graph construction for adaptive ﬁltering.
To adapt F to the elevation level we construct a graph G with nodes that have different neighborhoods depending on the elevation.
To do so we deﬁne a circular area on a tangent plane T , centered in the tangency point.
Then we move T such that it becomes tangent to the sphere S in different positions v p and project the circular area onto S .
For every point of S this creates a neighborhood N p , which changes its shape and size together with the elevation, as can be seen in Fig.
Based on this geometry adaptive neighborhood, we then construct the graph G in the following way.
We connect the node v p ∈ G , corresponding to a tangent point on the sphere, with the node v j ∈ N p .
The corresponding edge e pi has a weight w pi that is inversely proportional to the Euclidean distance between v p and v i , which are deﬁned on the sphere:
[Equation]
This allows us to vary the size of the neighbourhood according to the geometry of the omnidirectional image for each node in G , and weight the contribution of the nodes to the ﬁnal ﬁlter response according to their distances to v p .
Therefore, depending on the elevation the ﬁlter is changing its shape and size.
While effective, ﬁlter F does not have a deﬁned orientation in space as according to Eq.
(3) the ﬁlter applies the same weights α l to all nodes in the l-hoop neighborhood, with the contribution of each node being weighted by the distance to v p .
This results in F being isotropic, which leads to suboptimal representation, as the network is not able to encode the orientation of the object.
Directed graphs construction.
In order to overcome the limitations of isotropic ﬁlters, we propose to replace a single undirected graph G with multiple directed graphs G k , where each G k deﬁnes its own orientation.
Let us consider the case of a 3x3 classical convolutional ﬁlter.
In this case the ﬁlter has 9 distinct elements.
To mimic the same structure with our graph convolutional ﬁlters we employ the following algorithm.
First we deﬁne 9 non-overlapping areas s k , k = 1 .. 9 on the tangent plane, which together form a rectangular region that is centered in the tangency point v p of T as deﬁned in the Fig.
This rectangular region effectively deﬁnes the receptive ﬁeld of the ﬁlter on the tangent plane.
Then we build a set of nine directed graphs G k , k = 1 .. 9 in the similar way, as mentioned in the previous section for undirected graph.
In particular in order to build graph G k we do as follows.
For the area s k and for every node v p we move the tangent plane at point v p and then project s k from T onto the sphere.
This operation deﬁnes a speciﬁc neighborhood N k ( p ) on the sphere that consists of the points that belong to the projection of the region s k from the plane T .
We then connect v p with a directed edge to each of these points, where the weight of the edge is deﬁned in Eq.
Note that the direction of the edge is very important, because connecting v p and v i with an undirected edge forces v p to be part of the neighborhood N k
( i ) .
This, however, is not possible, as the neighborhood N k ( i ) is computed by projecting the area s k from the plane T that is tangent to the sphere at point v i and does not include v p .
This results in construction of the directed graph G k , which corresponds to the k th region of the ﬁlter, illustrated in Fig.
We repeat this operation for all the areas s k , k = 1 .. 9 of our ﬁlter, which leads to creation of 9 directed graphs G k , k = 1 .. 9 .
Given this set of graphs G k we deﬁne the resulting convolutional operation F as follows:
[Equation]
where F k is the ﬁltering operation deﬁned on the graph G k .
Note that this ﬁltering operation is slightly different from the operation that is used when working with undirected graphs and is discussed in more details in the following section.
To sum up, the introduced graph construction process allows having anisotropic ﬁlters F , deﬁned in
Eq.
(5) that are capable of capturing the orientation of the object and therefore learn more meaningful feature representation for an image compared to the isotropic graph-based ﬁlters.
It is important to note that in this paper we use the set of 9 non-overlapping rectangular areas deﬁned on the tangent plane, as shown by Fig.
2, due to their rough correspondence to the elements of a 3 × 3 convolutional ﬁlter.
However, our method can be easily extended to an arbitrary number of such areas with arbitrary shapes.
Geometry aware anisotropic ﬁlters.
For directed graphs G k Laplacian matrix is not deﬁned, therefore, we use the polynomial ﬁlters proposed in (Sakiyama et al., 2017).
Instead of the Laplacian matrix these ﬁlters rely on the normalized adjacency matrix A k , which is deﬁned as follows:
[Equation]
where A k and D k are the weighted adjacency and the diagonal degree matrices of graph G k respectively.
The elements of D k are computed as D k ( m, m )
=
(cid:80) n A k ( m, n ) .
Then, we deﬁne ﬁlters in the following way:
[Equation]
where α ( k ) 0 , α ( k ) 1 are the training parameters of our ﬁlter.
Here, we use polynomial ﬁlters of degree 1 , as they achieve good balance between speed and performance.
Network architecture.
The introduced approach focuses on the modiﬁcation of the convolutional layer to incorporate the knowledge about the image projective geometry inside the neural network.
Thus, it can be applied for a broad variety of tasks.
In this chapter we focus on the image classiﬁcation and compression problems. For the former one we use a relatively standard architecture that consists of a sequence of convolutional layers with the introduced graph-based ﬁlters, followed by a sequence of the fully connected layers.
For the compression task we use the architecture proposed in (Ball´e et al., 2017) and replace its convolutional ﬁlters with the proposed graph-based ones.
Discussion.
Our method can be seen as a generalization of different approaches that have been developed for omnidirectional images.
For example, if the node v p at elevation φ i has only one neighbor in each direction and the weight of the edges between nodes is always equal to one, it will be the standard CNN method (LeCun et al., 2001).
Further, if these neighbors correspond to the projected points it becomes the recently proposed algorithm of (Coors et al., 2018).
Finally, if we replace directed graphs with a single undirected one we get the same behavior of the polynomial ﬁlters as described in graph-based deep learning methods (Khasanova & Frossard, 2017a; Defferrard et al., 2016; Kipf & Welling, 2017; Khasanova & Frossard, 2017b).
R ESULTS
In this section we illustrate the performance of our approach. We start by evaluating our method with respect to competing approaches on the task of classiﬁcation images that are projected to different surfaces.
Finally, to show the generality of our approach and illustrate the effectiveness of the anisotropic graph-based ﬁlters, we evaluate our method on the image compression task.
4. I MAGE C LASSIFICATION
In this section we ﬁrst introduce the datasets that we used for the evaluation of our method.
We then discuss the baseline approaches and architectures, which we use in our experiments. Finally, we show the quantitative comparison of our method with the competing ones. Datasets. We evaluate our method on three different types of data, which feature different surface geometries, where the images are projected:
• Spherical dataset (S) consists of images projected on different locations on a sphere. The resulting spherical images are then unwrapped to for equirectangular images, as described in Section 3.1. • Mod-spherical dataset (MS) features image projections on more complicated surfaces that are depicted by Fig.
3 together with the representative examples of projected images. This dataset itself consists of three different versions: MS1, MS2, MS3 which correspond to the [Figure] [Figure]
surfaces which are getting further away from the spherical one. A more detailed discussion about the type of projection and the surface geometry used in these datasets can be found
in ApendixB. • Fish-eye dataset (F) consists of images projected on different locations on a sphere using stereographic projection Bettonvil (2005), which is frequently used in ﬁsh-eye cameras. • Cube-map dataset (CM) features projection of the images on the cube as shown by Fig. This type of projection has recently gained popularity for handling omnidirectional images, due to its ability to reduce distortion artifacts that appear due to the spherical geometry. In all these dataset we use MNIST images from (LeCun & Cortes, 2010) , which are divided into train, validation and test sets with 54k, 6k and 10k samples respectively.
Architecture. We compare our approach with standard ConvNets, the algorithm proposed in (Cohen et al., 2018) and other graph-based methods.
We present our result in the Table 1 1 .
For the graph-based methods we investigate 3 possible way of constructing G :
• Regular grid-graph with 8 neighbors and all equal weights w ij = 1 ; • Regular grid-graph with 8 neighbors and weights that depend on the Euclidean distance d ij between the nodes, as proposed in (Khasanova & Frossard, 2017b)
w ij = d − 1 ij ; • Irregular GA-graph w ij =
d − 1 ij (isotropic ﬁlters from Section 3.2).
For all of them we build a normalized Laplacian matrix (Khasanova & Frossard, 2017b) and use polynomial ﬁlters of degree 1 , which is equivalent to using 3 × 3 convolutional kernels.
Therefore, for the standard ConvNet we similarly rely on ﬁlters of size 3 × 3 . [Table]
All the competing approaches use the networks of roughly the same complexity.
For all the methods we use the architectures of similar structure and roughly the same number of parameters. For all the graph-based approaches we use the graph-based convolutions with stride two on each layer, which in turn requires building graph for each new layer according to its respective sampling.
The exact architecture of the classiﬁcation network is illustrated in Appendix A. For the method of (Cohen et al., 2018) we used the architecture proposed in the paper with roughly the same number of parameters as in competing approaches. Evaluation.
We present the result of comparison of our approach with the baseline methods in Table 1. Our method signiﬁcantly outperforms the standard ConvNets, as it is designed to use the geometry of the omnidirectional images.
Further it shows a much higher accuracy then other graphbased techniques, which rely on isotropic ﬁlters.
Further, our method achieves comparably accuracy with (Cohen et al., 2018) on spherical image representation, however, our method is more general, therefore, it outperforms (Cohen et al., 2018) on other datasets. Finally, we are able to run our approach on cub-map projection while the SphericalCNN by design is not applicable to such kind of images.
4. I MAGE C OMPRESSION
In all our previous experiment we have focused on evaluating our approach on the image classiﬁcation task. To show the generality of our method and better illustrate the effectiveness of anisotropic graph-based ﬁlters, we now evaluate their performance on an image compression problem.
For this task, we choose to modify the architecture introduced in (Ball´e et al., 2017) by replacing the ordinary convolutional layers with our own graph-based convolutions. In this section we ﬁrst introduce our approach and then compare the performance of the two graph-based methods, which rely on isotropic and anisotropic graph-based ﬁlters respectively.
Image compression framework.
The method introduced in (Ball´e et al., 2017), presents the process of image compression is an optimization of the tradeoff between having small distortion of the pixel intensity values and the small number of bits that are required for storing the compressed representation of these values.
As described in (Ball´e et al., 2016; Ball´e et al., 2017), this optimization can be represented as a variational autoencoder.
For more details we refer to Appendix C. In the context of omnidirectional images, we propose to modify the method proposed in (Ball´e et al., 2017) by using our geometry-aware ﬁlters instead of standard convolutional ones.
Evaluation.
We now evaluate the performance of our approach.
For this experiment we have implemented two versions of the system.
One with isotropic graph-based ﬁlters and the other one with the anisotropic ones.
We further evaluate the original method (Ball´e et al., 2017) for the sake of completeness. All three methods are trained and tested on the same splits of the modiﬁed version of the dataset (Xiao et al., 2012), which consists of omnidirectional images projected onto a cube.
From this dataset we use 3900 images for training and 1000 for testing of our approaches. We compare the methods in terms of the Peak Signal to Noise Ratio (PSNR) with respect to the average number of bits per pixel (bpp).
The results of the evaluation are presented in Fig.
As we can see, our method with anisotropic ﬁlters and (Ball´e et al., 2017) show similar PSNR values and [Figure]
C ONCLUSION
In this paper we have presented generic way of graph construction that allows incorporating the information about the image geometry inside the neural network.
Further, we have introduced the graph-based geometry aware convolutional ﬁlters that adapt their shape and size to the geometry of the projection surface.
In contrast to many existing graph-based ﬁlters, our ﬁlters are anisotropic, which allows to better adjust to the speciﬁc properties of the problem.
Our illustrative experiments show state-of-the-art performance of our approach applied to image classiﬁcation and compression tasks in the presence of various types of image distortions A PPENDIX A A RCHITECTURE OF CLASSIFICATION FRAMEWORK
Fig.
7 illustrates the architecture of our classiﬁcation network, which we use in Section 4.1.
B M ODIFIED SPHERICAL SURFACE .
In order to evaluate the performance of our method as a function of deformation of the spherical surface, we have created a set of datasets by projecting the MNIST images to random locations of the surfaces, which have shapes, shown in Fig. 3 (a,c) and unwrap these spherical images to equirectangular ones. The white color in Fig.
3 (a,c) denotes the areas of the generated surface that are the furthest from the spherical surface of the same radius. The Fig. 3 (b,d) illustrates sample images of digits projected onto the respective surfaces. Each of the aforementioned surfaces is the following modiﬁcation of a spherical one from Eq.
[Equation]
where ( x, y ) are the coordinates on the tangent plane and p ( φ, r, l ) is the perturbation function that can be written as
[Equation]
where φ is the elevation level; r is the parameter that regulates the perturbation magnitude and l deﬁnes frequency of the perturbation signal.
In our experiments we have set l = 10 .
Note that for a speciﬁc case of r =
we get the ordinary spherical surface.
We then use Eq.
8 to construct the graph G that allows our method to adapt to the surface geometry and evaluate our method on each of the generated datasets.
C C OMPRESSION
In this section we brieﬂy describe compression approach, which proposed by ? .
An input image x is encoded using a function g a ( x ; α ) , which results in the respective latent representation y .
Then, y is quantized into ˆ y , which can be losslessly compressed using entropy coding algorithms.
This ˆ y is then passed to the decoder g s (ˆ y ; β ) at the decompression step, which results in a decompressed image ˆ x .
Here, we denote by α and β the parameters of the encoding and decoding algorithms respectively.
While both encoder and decoder can be represented as a differentiable function, the process of quantization is non-differentiable.
Therefore the authors of (Ball´e et al., 2016) propose to replace quantization with an additive uniform noise at the training step as follows:
[Equation]
where ∆ y denotes additive i.i.d uniform noise.
This trick allows to perform the end-to-end optimization of both the encoder and decoder parameters using the following loss function:
[Equation]
where g s , g a are convolutional deep neural networks, d represents the distance between the images and λ is a weighting parameter.
Thus, during the training step, we add noise (according to Eq.
(10)) to be able to back propagate the error and at the inference time we apply quantization to the latent representation y .
The overall architecture that we use is similar to the one proposed in (Ball ´ e et al., 2017) and is summarized in Fig.
Further, the method of (Ball ´ e et al., 2017) relies on the standard convolutional layers, which are practical for ordinary images: they allow learning local image structures independently of their location in the image.
Additional visual results.
We run experiment with described compression architecture, where we compare three approaches: original and methods, where we replace convolutional ﬁlter from the architecture to graph-based isotropic and geometry-aware ﬁlters.
Fig.
9 further illustrates some visual comparison of the methods and we can see isotropic ﬁlters produce over-smoothed decompressed images, which do not look realistic and result in very low PSNR values.
On the other hand our method with anisotropic ﬁlters is able to produce sharp results.
[Figure]