

Convolution neural networks (CNNs) are brain-inspired architectures popular for their ability to train and relearn visually complex tasks. It is incremental and scalable; however, CNN is mostly treated as *black-box* and involves multiple trial & error runs. We observe that CNN constructs powerful internal representations that help achieve state-of-the-art performance. Here we propose three layer *glass-box* (analytical) CNN for two-class image classification problems. First is a representation layer that encompasses both the class information (group invariant) and symmetric transformations (group equivariant) of input images. It is then passed through dimension reduction layer (PCA). Finally the compact yet complete representation is provided to a classifier. Analytical machine learning classifiers and multilayer perceptrons are used to assess sensitivity. Proposed *glass-box* CNN is compared with equivariance of AlexNet (CNN) internal representation for better understanding and dissemination of results. In future, we would like to construct *glass-box* CNN for multiclass visually complex tasks.

Introduction {#intro}
============

Humans have an age-old deep rooted desire to understand functioning of the brain. Medical science approaches it through brain imaging, dissection and studying molecular structures @medical. On the other hand, engineers have chosen to build learning machines capable of mimicking the human brain in performing intelligent tasks @star. Machine learning approaches are large scale function estimation algorithms that can learn on their own and adapt to new data based on their past and future experiences @vapnik. It has wide applications in medical diagnosis @desh [@uday; @manu], autonomous cars @cars, facial recognition @nips, remote sensing @khilan, and in many other fields.

The learning machines can be broadly categorized into two ideologies @vapnik: i) the estimated function depends on the features of the data, e.g., k-nearest neighbors (kNN), Naive Bayes, support vector machine (SVM), and the regression; and ii) the estimated function consists superposition of linear indicators arranged to resemble the humans brain, say neural networks @NN and the decision trees @RF. Neural networks implement an extensive set of indicator functions which, in turn, are made of superposition of linear/non-linear function indicators @minsky. The origin of neural networks started from observations on the working of neurons, gained through studying the human brain @NN. Perceptrons are linear classifier functions that separate the input into $0$s and $1$s @perceptron. Multilayer perceptrons (MLP) @MLP1 are layers of fully connected Perceptrons capable of classifying complex data into multiple classes. Various categories of neural networks have evolved like convolution neural networks (CNNs), autoencoders, generative adversarial networks (GAN), and Boltzmann machines, to name a few. Meanwhile, decision trees are also developed wherein each leaf node represents an outcome for the test. Each internal node contains a condition regarding one of the features of the data. The split is generally decided based on the Gini index or entropy, @RF based on the model requirement. In particular, random forest (RF) @RF is an ensemble of multiple decision trees whose individual outputs are combined to give one prediction. However, the construction of RF is computationally taxing and it is limited to feature-based classification. Further, there is no closed-form solution available to decide the number of trees required for an optimal performance @RFC.

Unlike regular neural networks, all layers of CNNs are not fully connected, making them easier for scaling. This makes CNNs suitable for spatially structured data like images and audio signals. CNNs layers are of three types viz, convolution layers, pooling layers, and fully-connected layers. Classification by CNNs can be represented as a compositions of multiple function problem wherein each layer estimates an integral function and this is applied forward, i.e., $f(f(f(f(...))))$, where each $f(.)$ is a linear/non-linear function. Generalization is the potential to estimate properties of unseen (test) data based on the learning from the seen (training) data.

It is believed that CNN architecture facilitates rich internal representations of the data within the classifier, thereby, improving its generalization ability @rep. The main advantage of deep CNN is that it requires minimal preprocessing when compared to classical neural networks. On the other hand, numerous hyperparameters involved in designing a CNN architecture is the main challenge in various applications. Besides, the number of filters, sizes of filters, type of activation function(s), and size of the input image are all open questions. Till now, CNNs parameters are primarily adjusted based on the experience of the designer, and the final performance is based on trial-and-error. Hence, there is no guarantee that the optimal structure (CNN model) has been archived. A model *automatically* learns the *structures* present in the labeled data and employs this learning, i.e., learnt function, to classify unseen or test data. Note that, the user/designer has little to no control on the function learned making them *black-box* models. This is giving rise to unsubstantiated models which when implemented in sensitive fields like security and healthcare causes significant problems @new_paper. The need for interpretable models is eminent than theories that attempt to explain the created *black-box* models. In this work, we aim to study state-of-the-art *black-box* machine learning models i.e., convolution neural network (CNN), and implement the analytical inferences learned to move towards an interpretable *glass-box* CNN.

Related work
============

Since the introduction of CNNs, the main concern initially was that the CNNs were finding local minima instead of global minima due to backpropagation @deep. In @land, the landscape of function is theorized to have multiple saddle points similar to each other. Later research has been carried out on finding ideal hyperparameters for optimizing its structure, developing various versions of architectures and understanding its mathematical framework. Many significant statistical methods are introduced into CNNs like probability distributions in place of finite, fixed valued weights, and incorporating naive Bayes into deep learning @uncertainity to improve their performance @desh. Residual Network architecture (ResNet) @resnet uses residual blocks to decrease the training error. While in @inception, inception modules are implemented to increase performance through increasing depth, creating inception networks.

For understanding CNNs, inverted CNNs are used in @invert. In this, the last layer output, the image is reconstructed by traversing the CNN in reverse order. Such images are studied to understand the framework of CNNs. It is concluded that information about invariance in geometric and photometric data, is learned by the CNNs. Intra-class knowledge inside CNNs is studied in @intraclass. It is found that CNNs capture location variations and style variation in a class to better classify them.

In recent years, invariance @learn [@nips], and equivariance @ECCV1 [@GECN] in CNNs have been the focal point for analytical studies of deep CNNs. Many researchers have identified that the filters obtained through training a model are powerful equivariant to group transformations like translation, scaling, rotation, and small diffeomorphisms while non-expansive in nature @GECN. This discovery led to architectures being designed with the idea of built-in equivariance to transformations @GECN. Invariant scattering convolution networks (ISCN) are introduced in @ST. These networks employ wavelets of different degrees of rotations in order to obtain an invariant representation of the image to translation and rotation transformations. Note that the wavelets are in general invariant to small diffeomorphisms but are not invariant to translations @ST. Average pooling and modulus at the end of wavelets would build translation invariant transformations. The scaling can be linearized by separating the variations at different scales with wavelets. Unlike traditional CNNs, the ISCNs produce outputs at the end of each layer, and the filters are predefined wavelets and are not learned from the data. A limitation of ISCNs is that their performance is poor for data with high variability.

More recently, group equivariant convolutional networks (G-CNNs) @GECN implement rotated filters in the CNN architecture for equivariant representation to rotation transformations. The input images are also rotated inside the CNN for better generalization. G-CNNs exhibit increased expressive capacity when compared to conventional CNNs with the same number of parameters. The translation equivariance present in conventional CNNs is exploited to include rotation and reflection symmetry. In @PH, group equivariant non-expansive operators (GENEOs) that act as filters in CNN are introduced. The generated GENEOs are sampled based on topological data analysis tool. Note that they have suggested a CNN with fixed GENEOs isometric to Euclidean space to achieve equivariance in the architecture.

In this paper, we develop a novel glass-box CNN for two-class image classification tasks while addressing the representational aspect of data. Experiments on MNIST, fashion MNIST, CIFAR-10, plant pathology, skin lesions, cats vs dogs, and cancer detection datasets exhibit encouraging results.

Method
======

In this section, we first analytically study the properties of CNNs and utilize these insights to design a layer for extracting equivariant-invariant features from the images. This yields an effective representation of the data for better classification. It is then fed to a dimensional reduction layer to make the representation compact yet complete. It is presented to one of the classifiers as layers including the final kNN, SVM, RF and MLP. Finally we quantify equivariant ability of proposed representation.

<span>0.35</span>

![Function map of the problem set-up.\
<span data-label="fig:space"></span>](space.png)

<span>0.60</span>

Problem set-up {#sec:setup}
--------------

1.  $\{\mathbf{x} \in X\}$, the set of all data points

2.  $\{D\}$ set of all labels

3.  $\{\Phi\}$ set of internal representations of the model

4.  $f(\mathbf{x}) : \mathbf{x} \rightarrow D $, function that maps data point to class its label

5.  $\phi(x): X \rightarrow \Phi$, internal structures of a model

6.  $f'(x): \Phi \rightarrow D$, classifier that maps internal representation to class label

Referring to Fig \[fig:space\], we consider $\mathbf{x}$ as the data with dimension $d$, for example, an image i.e., $\mathbf{x} \in \mathbb{R}^{d \times d}$. The set of all the images is denoted by $\Omega$ which is considered as a subset of $\mathbb{R}^{n \times n}$. The function $f(\mathbf{x})$ maps an $\mathbf{x}$ to its class label and this is the function we estimate through any classification model in machine learning, $f(\mathbf{x}) : \mathbf{x} \rightarrow D$. $D \in \mathbb{R} $ is set of all class labels. $\Phi = \phi(x)$ as the internal representation of the data in a model. Note that the internal representation of a model promotes its generalization ability. Consider $g \in G$ is a group of symmetries acting on the function $f$, then $g(x)$ is the transformed image obtained by applying a group transformation $g$. In turn, $f'$ maps the internal representation $\phi(x)$ to the class label, i.e., $f'(x): \Phi(x) \rightarrow D$.

CNNs solve this problem by finding a wide range of representations of the image $x$ in their internal representation $\Phi$. This $\Phi$ in CNNs is *black-box* and is a very high dimensional representation. Here, we propose a $\Phi$ consisting purely of equivariant and invariant representations followed by dimension reduction. This proposed $\Phi$ is *glass-box* and is low dimensional representation. The equivariance ability of the proposed function $\Phi$ is also measured and compared with well-known AlexNet CNN @alex.

Some analytical insights into the CNNs {#sec:insight}
--------------------------------------

Let us begin with understanding the CNNs. They alternatively apply convolution (linear) and nonlinear operators on the data, layer-by-layer. A linear layer of CNNs contains filters of varying sizes that operate on the output of the previous layer while non-linear layers act as the activation function on the convolved output. Sigmoid, ReLu, and softmax are some of the popular activation functions used in the CNNs. One really crucial question is the depth of a CNN and how it impacts the performance. In @depth, experiments performed on CIFAR-10 @cifar and TIMIT acoustic-phonetic speech @timit proved that well-engineered shallow networks i.e., not a '*deep network*', are capable of performances that previously assumed to be impossible. This is done by designing a shallow network with the same number of hyperparameters as a deep network. Teacher networks are also implemented to mimic an ensemble of deep CNNs @timit. In this proposed work, we have also designed a shallow network framework that achieves the performances of deep CNNs for two-class problems.

Neural networks either perform channel combinations or not. Channel combination implies that the output of multiple filters of one layer is combined as input for the next layer. The behavior and properties of CNNs that do not combine multiple convolutions across channels are analyzed in @Mallat and these properties are theorized to extend to complex CNNs that do channel combination. Referring to our problem set-up shown in Fig \[fig:space\], to analyze high-dimensional data with smaller networks, it would be ideal if we can find a function $\phi$ that separates $f(\mathbf{x})$ such that $f(\mathbf{x})=f'(\phi(\mathbf{x}))$. The function $\phi$ should satisfy the condition if $f(x) \neq f(\mathbf{x}^\prime)$ then $\phi(\mathbf{x}) \neq \phi(\mathbf{x}^\prime)$ where $x,x' \in \mathbf{x}$. There exist two ways to find a function $\phi$: separation and linearization. In separation, the approximated $f(\mathbf{x})$ using a low dimensional $\phi(\mathbf{x})$ does not decrease the final dimension significantly in most cases. The alternate strategy is to linearize the variations of $f$ with change in variable $\phi(\mathbf{x}) = \{ \phi_k(\mathbf{x}) \}_{k \leq d^\prime}$ where $d^\prime \ggg d$, e.g. support vector machines (SVM). It should be noted that in most cases, only one dimension is decreased when separation is implemented. It is evident from the success of neural networks that linearization is the better alternative as it leads to better performance. Following this argument, we have employed linearization in the proposed glass-box network (see Sec \[sec:frame\]). The dimension of the data is linearly increased by producing invariant and equivariant representation (see Sec \[sec:equ\]) through convolution. Feature extraction (see Sec \[sec:pca\]) is later performed on this linearized data.

To linearize the function $f$, we need to find the direction in which the value $f(\mathbf{x})$ does not change. This can be achieved by considering level sets, i.e., $\Omega_t = \{\mathbf{x}:f(\mathbf{x})=t\}$ and using groups of symmetries. In this work, we address the (two-class) image classification problem hence we resort to translation, diffeomorphism, and scaling group actions to account for properties of CNN filters. Since global symmetries are difficult to find, we seek local symmetries $G$ that the function $f$ is locally invariant to :

$$\forall \mathbf{x} \in \Omega,\text{  } \forall g \in G, \text{  } \exists C_x > 0,\text{  } \forall |g|_G < C_x,\text{  } f(g(\mathbf{x}))=f(\mathbf{x}).$$

Here $C_x$ is the extent of transformation that does not impact the recognition of the image. That is, the equivariance capability of the representation.

$$\text{Translation group }G \in \mathbb{R} \text{  is   } g(\mathbf{x}(u))=\mathbf{x}(u-g(u)) \text{  with  } g \in \textbf{C}^1(\mathbb{R}^n),$$

where $u$ is the pixel position in the image. Diffeomorphism $diff(.)$ is a stronger constraint where the group transformations are of type diffeomorphism group, i.e.,

$$G=Diff(\mathbb{R}^n),$$

that transforms $\mathbf{x}(u)$ with differential wrapping of $u \in \mathbb{R}^n$. The properties of group symmetries in the data are combined for estimating the direction of $f(x)$ used while deriving equivariant and invariant representations in this work.

From this, one can see that the success of CNNs in visual recognition tasks can be attributed to their rich internal structures that are equivariant to various symmetric transformations. The process of obtaining different internal representations within the network is becoming computationally intensive for the state-of-the-art classifiers like CNN @xecp. Hence, in this paper, we address the representation aspects of data and present a more transparent CNN for two-class setting.

Proposed glass-box CNN {#sec:frame}
----------------------

In Fig. \[fig:model\] (a), we present our model for image classification with a proposed representation method that is both invariant as well as equivariant to a group of transformations acting on the data. The three layer architecture of the proposed *glass-box* CNN is detailed in this section. The first layer contains the image representation while the second layer contains the dimension reduction and feature extraction. Classification is finally performed in the third layer.

![Proposed *glass-box* CNNs: (a) *glass-box* CNN for image classification, (b) transparency of proposed *glass-box* CNN as in (a), and (c) scalable version of the proposed *glass-box* CNN as in (a).<span data-label="fig:model"></span>](final_model.png)

### Layer 1: Representation {#Sec:GIFE}

As shown in Fig. \[fig:model\] (a), the first layer of the proposed *glass-box* CNN is image representation. The input to this layer is an image and the invariant and equivariant representations are the outputs. Wavelets and group equivariant operators are implemented to obtain respective representations. Note that invariance representation is ideal for increasing generalization ability at the same time, equivariant representation helps control the extent of transformation influence and how one class differentiates from another.

Invariance to a transformation means that the representation for an image and its transformed image are equal. This is mathematically represented as follows. Let $\phi(x)$ be a representation, i.e., {$\phi(x): X \to \mathbb{R}^{n \times n}\}|\{x \in \mathbb{R}^{n \times n}\} \in X$ is a set of all images. Consider $G$ be the group of global symmetries acting on the $X$. Now, $\phi(x)$ is invariant to the action of $g \in G$, i.e.,

$$g.x=x_g$$

$$\phi(x)=\phi(x_g),$$

where, $x_g$ is transformed $x$ with the action of $g$. Let $\Phi_x$ be an invariant representation of $x$ under group $g$ be defined as,

$$\Phi_x = {\Phi_x}_g.
  \label{eq:group}$$

See that the group invariant representation equation (\[eq:group\]) captures the common features among all the data points belonging to a particular class. Nevertheless, it is a trade-off and one needs to be careful of possible loss of information. In proposed work, we obtain an invariant representation of the data by applying scattering wavelet transform (SWT) with a scattering order $1$. This yields a type of CNN with a specific architecture similar to @ST. Note that there are no weights to be learned in our framework as all the filters are wavelets assigned by the user. Unlike traditional CNNs where the last layer has the output, the scattering transform has an output at the end of each layer called $n^{th}$ order coefficients for $n^{th}$ layer. These are computed by convolving the input image with dilated and rotated wavelets $\psi$, followed by a scaling function $\phi$.

The initial layer contained in SWT is a scaling operator resulting in the output $x.\phi$. The $n^{th}$ order coefficients are $||x*\psi_1|*\psi_2|...|*\psi_n|.\phi$. We conducted experiments to determine the scattering order at which optimal representation is obtained. We have observed that at scattering order $1$, the representation has the highest invariance. So, the best representation is obtained when no cascade of wavelets is applied. SWT exploits the fact that the wavelet transformations are invariant to small diffeomorphisms. However, they are covariant to translations, and to obtain a translation invariant, a nonlinear operation is performed at the end of each layer like modulus. The scattering transform computes a translation invariant representation with deformation stability. The coefficients are used as invariant representations in the proposed model (Fig: \[fig:model\] (a)). In this proposed work, we employed Morlet wavelet as it is considered closest to human audio and visual perception @ST.

\[sec:equ\] Now referring to Fig. \[fig:model\] (a), the equivariant representation is also obtained in the layer $1$ in order to regulate the generalization in the features extracted to represent the image. Equivariant features contain information about the underlying symmetries in the data. Let an image $x$ be represented by function $x(i,j)$ and the equivariant representation after applying the operator is $\phi(i,j)$. A group equivariant operator acts as a function between the two spaces $(x,\phi)$ that act by the same symmetry groups. In @PH, a strategy is defined to generate, select, and sample IENEOs. Note that in @PH the sampled operators are used to initialize fixed filters in a CNN and their performance is observed. In contrast, in proposed work, we propose to preprocess the data to obtain the equivariant representation that is given as input to a classifier. The generation of IENEOs is done using @PH as,

$$\begin{aligned}
  H_p(i,j)=\sum_{m=1}^k a_mg_{\tau_m}\sqrt{(i^2+j^2)},
  \label{eq:1}\end{aligned}$$

where $g_{\tau_m}: \mathbb{R} \to \mathbb{R}$ is 1D Gaussian function with width $\sigma>0$ and centre $\tau \in \
\mathbb{R}$. $P=(a_1,\tau_1,a_2,\tau_2, ...,a_k,\tau_k)$ with condition $\sum_{m=1}^k {a_m}^2=\sum_{m=1}^k {\tau_m}^2 =1$ is a sub-manifold of $\mathbb{R}^{2k}$. $H_p :\mathbb{R} \to \mathbb{R}$ derived using $P$ and $g_{\tau_i}$. The IENEO is a convolution operator parametric with respect to $P$, mapping continuous function with compact support $x:\mathbb{R}^2 \to \mathbb{R}$ to a continuous compactly supported function $\phi:\mathbb{R}^2 \to \mathbb{R}$ defined in @PH as,

$$\begin{aligned}
  \phi (i,j) & = \int_{\mathbb{R}} x(\alpha,\beta)\frac{H_p(i-\alpha,j-\beta)}{{\|H_p\|}_{L^1}} d\alpha\text{ }d\beta \; ,
  \label{eq:2}\end{aligned}$$

where, $\alpha,\beta$ are coordinates of the image. See that it can be implemented to obtain an equivariant representation of the data in the Euclidean space.

We now need to select the operators that preserve the class information while decreasing variations in the data. To this end, we employ a topological data analysis tool called persistent homology @per. This can be used to measure the topological features and shapes of a function. This is used to compare the performance of randomly generated operators. The effect of their transformations on the data is studied and operators that increase the inter-class distance and decrease the intra-class distance are selected. While selecting the equivariant operator, we have introduced an extra step that builds a classifier using $70\% $ of training data and validated the model with the rest of the data. The operator with the best performance is selected to obtain equivariant representation. This leads to the selection of an equivariant operator best suitable for a classifier.

### Layer 2: Dimension reduction  {#sec:pca}

Once the invariant and equivariant representations are obtained in layer $1$, we perform principal component analysis (PCA) to further select the respective invariant and equivariant features in order to generate a compact yet almost complete representation. PCA selects features by calculating eigenvectors of the covariance matrix and the highest eigenvalues are considered as features. The eigenvalues are considered as features extracted from the image representations. Once the eigenvectors are calculated based on the training data, we create an instance of the model with a user defined percentage of variance retained. In our model, we found that $80\%$ variance retention has the best results. The eigenvectors of the training database are taken as a basis and the test data is represented using them.

### Layer 3: Classifier

Now that we have a set of compact features extracted from respective equivariant and invariant representations, we need to implement a learning model to classify images based on the compact features obtained after the dimension reduction. It is clear that these features encompass both the class information obtained through invariant representation and the information about the underlying symmetries of the data obtained through equivariant representation. Hence, they can now be used to better classify the data (image) without generating various representations within the network as generally done by the *black-box* CNNs.

As illustrated in Fig. \[fig:model\] (b), when an image of size $28 \times 28$ from MNIST database is given to the proposed *glass-box* CNN (Fig. \[fig:model\] (a)), a group invariant representation of the given image of size $9849 \times 1$ is obtained through SWT. Similarly, a group equivariant representation of size $784 \times 1$ is obtained by using IENEOs. Subsequently, features are extracted from the two representations using PCA @pca resulting in $25$ invariant features and $28$ equivariant features (Fig. \[fig:model\] (b)). These are finally given to a classifier model for the classification.

In our experiments, we have implemented kNN, SVM, and random forest models to classify the data. Through invariant representation, we obtain the common features of a class which help us identify the class label for an unknown data. On the other hand, through a non-expansive equivariant operator, we obtain features that are stable to transformations in the data. It is interesting to see that the features extracted from these two representations as an input help improve the accuracy of conventional classifiers, which do not have an abundant internal representation of the data, unlike neural network classifiers. Hence, this way we could convert a *black-box* CNN into a *glass-box* CNN. The proposed *glass-box* CNN helps to address the representational aspects of the data. In particular, SVM when implemented within the framework performs at-par with AlexNet. SVM and softmax are considered to be very similar to each other in practical application @softsvm. This indicates that while the representation part of the framework imitates the convolution layers in a CNN, the SVM as a classifier imitates the fully connected layers and softmax of the CNN. Hence, essentially the proposed framework acts as an *glass-box* architecture of the CNNs.

### A scalable version of the proposed *glass-box* CNN

In Fig: \[fig:model\] (a) we have derived the proposed representation for various conventional classifiers. A drawback of a conventional classifier when compared to the neural network architecture is that they lack the ability to update when they encounter new experiences. To address this, the model would have to be retrained with the old and the new instances. Hence, in order to incorporate scalability in the proposed *glass-box* CNN, we propose a multilayer perceptron (MLP) as layer 3 classifier. The architecture of the modified version of Fig: \[fig:model\] (a) is depicted in Fig: \[fig:model\] (c). A perceptron is a unit capable of calculating linear predicates @perceptron. These are modeled based on the neurons in the human brain. MLP as their name suggests, are networks composed of fully connected layers of perceptrons. MLP with one hidden layer are called "Vanilla" neural networks that are non-linear statistical models and behave very similar to a SVM @MLP. The numbers of perceptrons increase with increase in size of the input and this is the reason for decline in MLP usage. Nevertheless in the proposed *glass-box* CNN, the limitation is already resolved through effective representation of the data followed by dimension reduction. Hence, incorporating MLP in the layer 3 in lieu of classical machine learning models yields a scalable version of *glass-box* CNN.

Measuring equivariance ability of proposed representation in layer 2 (Fig: \[fig:model\]) {#sec:measure}
-----------------------------------------------------------------------------------------

Internal representation of the data plays a crucial role in the performance. The proposed features in layer 1 and 2 can be evaluated by measuring the equivariance capacity. This is done by estimating the equivariant map and evaluating the accuracy at which the impact of transformations is predicted by the map. An image representation $\phi(x)$ is considered equivariant with respect to a transformation $g$ if a map $M_g$ exists such that,

$$\begin{aligned}
  \phi(g(x)) = M_g(\phi(x)) \text{   } \forall x \in X
  \label{eq:mg}\end{aligned}$$

If the representation is invariant to a particular transformation then the equivariant map ($M_g$) will be identified. The existence of inverse of $\phi$ and closure with respect to transformations group $G$ are sufficient conditions for the equivariant map existence ($M_g$) @measure.

Equivariance of a representation to a transformation can be proved by calculating the equivariance map. In the proposed *glass-box* CNN model (Fig: \[fig:model\]), $M_g$ is calculated through linear regression, regularized regression and sparse regression. We found that regularized linear regression has the best estimate among the three. The approximation of the equivariant map is calculated through as,

$$\begin{aligned}
  \widehat{M_g}= \phi(g(x)){\phi(x)}^{-1},
  \label{eq:reg}\end{aligned}$$

where, $\widehat{M}_g$ is the approximation of $M_g$ calculated through the regularized linear regression. After calculating $\widehat{M}_g$, we evaluate the equivariance of the representation as follows,

$$\begin{aligned}
  \text{If } \phi(g(x) = \widehat{M_g}(\phi(x)) \text{ then,} \\
  \phi(x)=\widehat{M_g}(\phi(g^{-1}(x))) .\end{aligned}$$

$$\begin{aligned}
  E_q = \dfrac{1}{n} \sum_{\forall x} |\phi(x)-\widehat{M_g}(\phi(g^{-1}(x))) |.
  \label{eq:cal}\end{aligned}$$

Here $E_q$ indicates the equivariance capability of the function $\phi$. The lower the value of $E_q$, the higher the equivariant capacity of a representation. This is considered as an aid to evaluate the proposed interpretable framework.

Results {#sec:exe}
=======

This section demonstrates experimental results on various databases (Fig: \[fig:all\]) using *glass-box* CNN and performance is compared with AlexNet CNN along with conventional baseline classifiers including kNN, RF, and SVM.

Implementation details and parameter settings
---------------------------------------------

All the experiments are conducted on an Intel core i7-9th Gen 16GB RAM, kirsten56@example.com with Nvidia Geforce GTX 1660Ti 4GB graphics. The experiments are conducted with two classes in the dataset and each experiment is run three times. The average accuracy of the three runs is presented in the Sec: \[Sec:results\]. The accuracy is calculated using the confusion matrix as follows,

$$Accuracy = \frac{(TN+TP)}{(TP+TN+FN+FP)},\label{eq:5}$$

where TN : True negatives, TP : True positives, FP : False negatives, FN : False positives. The k value for kNN is empirically set to as 11 since odd numbers are preferred for classification with two classes @KNN. We also observed that the performance of kNN is stable to change in k value. RBF kernel is used for implementing the SVM algorithm. Besides, we have implemented linear and polynomial kernels and we found that RBF exhibits better performance among the three. In the case of random forest, 100 trees are implemented. We discerned that an increase in the number of trees has not improved the performance.

![Sample images from the databases used in our experiments: (a) MNIST @mnist with $70000$ images of size $28 \times 28$ pixels, (b) Fashion MNIST @fmnist with $70000$ images of size $28 \times 28$ pixels, (c) CIFAR-10 @cifar with $70000$ images of size $32 \times 32$ pixels, (d) Skin lesions @skin with $7000$ images of size $28 \times 28$ pixels, (e) $1138$ resized images of plant pathology @plant to size $100 \times 100$ pixels, (f) $10000$ images of dogs vs cats @dogs_cats of various breeds with size $80 \times 80$ pixels, and (g) Cancer detection images @cancer1 [@cancer2] with $7000$ each of size $96 \times 96$ pixels.<span data-label="fig:all"></span>](databases.png)

Result analysis {#Sec:results}
---------------

We observed from Table \[tab:quant\], that when the available data is small (plant pathology and cancer detection), our proposed *glass-box* CNN has a higher accuracy. This indicates that when the data availability is scarce, our proposed model is a better alternative, whereas the conventional CNNs (AlexNet) tend to overfit. On the other hand, we can also infer that when the images have natural backgrounds (CIFAR-10 and cats vs dogs), AlexNet performs better. Table \[tab:quant\] also details the improvement in the classification by implementing proposed *glass-box* representation in kNN, SVM, and random forest classifiers. We can observe that the SVM with RBF kernel has a higher improvement. The proposed representation not only improves the performance of conventional classifiers but also decreases the complexity. We interestingly notice that when the variance in the data is high, then the extracted features as input, has significantly improved the accuracy of a classifier. This can be observed by comparing CIFAR-10 results (Table: \[tab:quant\]).

Following this, we have implemented the algorithm to measure the equivariance of *glass-box* representation (layer 2 output in Fig: \[fig:model\]) by calculating equivariant map @measure using the equation (\[eq:cal\]). For comparison, we have measured the equivariance of CNN's internal representation. The output of the last layer before the softmax dense layer of AlexNet is considered as its internal representation. The values in Table \[tab:quant3\] indicate how predictable the impact of a transformation on the representation. The lower the value the higher equivariant the representation is to that transformation.

Table \[tab:quant3\] lists the equivariance capability of our proposed *glass-box* representation vs AlexNet internal representation. The $E_q$ values of each representation with respect to group actions i.e., translation, rotation, and reflection are calculated. The lower the value of $E_q$, the higher the equivariance capability. One can observe that our proposed representation is equivariant to symmetric transformations like translation, rotation, and reflection. This clearly demonstrates the better generalization ability of our proposed *glass-box* CNN.

<span>|c||c|c||c|c||c|c||c|c|</span> & & &\
& & &

  --------------------
          kNN
           in
   Fig: \[fig:model\]
          (a)
  --------------------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

&

  ----------
   Baseline
     kNN
  ----------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

&

  --------------------
           RF
           in
   Fig: \[fig:model\]
          (a)
  --------------------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

&

  ----------
   Baseline
      RF
  ----------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

&

  --------------------
          SVM
           in
   Fig: \[fig:model\]
          (a)
  --------------------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

&

  ----------
   Baseline
     SVM
  ----------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

\

  --------
   MNIST
   @mnist
  --------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

& **0.99** & **0.99** & **0.99** & 0.95 & **0.99** & **0.99** & **0.99** & **0.99**\

  ---------
   FMNIST
   @fmnist
  ---------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

& **0.98** & **0.98** & **0.98** & 0.94 & **0.98** & 0.96 & **0.98** & 0.96\

  ----------
   CIFAR-10
    @cifar
  ----------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

& 0.91 & **0.94** & **0.88** & 0.50 & **0.86** & 0.74 & **0.93** & 0.68\

  ---------
    Skin
   Lesions
    @skin
  ---------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

& **0.67** & 0.65 & **0.64** & 0.63 & **0.65** & 0.63 & **0.68** & 0.66\

  -----------
     Plant
   Pathology
    @plant
  -----------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

& **0.70** & 0.55 & **0.56** & 0.50 & **0.65** & 0.55 & **0.73** & 0.57\

  -----------------
       Cats Vs
   Dogs @dogs_cats
  -----------------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

& 0.69 & **0.81** & **0.62** & 0.59 & **0.64** & 0.60 & **0.71** & 0.62\

  ---------------------
         Cancer
        detection
   @cancer1 [@cancer2]
  ---------------------

  : Comparative accuracy analysis of proposed *glass-box* CNNs.<span data-label="tab:quant"></span>

& **0.70** & 0.55 & **0.67** & 0.62 & **0.67** & 0.66 & **0.71** & 0.65\

<span>|l||c||c|</span> Transformation &

  -------------------------------------
   Proposed *glass-box* representation
     (layer 2 of Fig: \[fig:model\])
  -------------------------------------

  : Equivariance of proposed *glass-box* representation (layer 2 of Fig: \[fig:model\]) Vs AlexNet (CNN) @alex.<span data-label="tab:quant3"></span>

& AlexNet (CNN) @alex\
MNIST Translation & **0.004** & 0.020\
MNIST Rotation & **0.012** & 0.070\
MNIST Reflection & **0.004** & 0.072\
Fashion MNIST Translation & **0.015** & 0.072\
Fashion MNIST Rotation & **0.020** & 0.11\
Fashion MNIST Reflection & **0.010** & 0.110\

Conclusion, discussion, and future work
=======================================

We have studied the structures and properties of a convolution neural network and learned that the generalization ability stems from its equivariant internal representation. From this, we propose a representation method that emulates equivariance and invariance to group transformations properties. The equivariant representation is obtained by implementing filters isometric to Euclidean space and invariant representation is obtained through wavelet scattering transform. Selected features (PCA) extracted from the equivariant and invariant representations are then employed for classification. The proposed *glass-box* CNN with MLP classifier has both the transparency of conventional machine learning algorithms as well as the generalization and scalability to update on new data like convolution neural networks. We observed from the experimental results that our proposed model performs better when training data is scarce.

Invariant representation captures the class information of the data while equivariant representation captures symmetries of the data. With this, our proposed representation is better suited to improve the accuracy of a classifier for the data with a high variance within the same class. We have also observed that the introduction of both invariance and equivariance representation reduces the solution space to be explored while constructing a classifier. This is reflected in results when classifiers that are sensitive to the local structure of the data like SVM are implemented in layer 3. It indicates that well-engineered shallow networks can achieve accuracies that are shown possible with deep neural networks like AlexNet. Finally, we have measured the equivariance to various transformations and observed that the proposed representation is robust to symmetric transformations of the input image.

Our future directions include to work on, equivariant representations and creating a network of such representation. This network could have the potential to surpass the really deep CNN architectures like ResNet-50. We sincerely hope that it will result in deep learning networks with a more complete understanding of the internal mathematical structures for multiclass visually complex tasks. Given that invariant representations and equivariant representations are independent, one may also implement a model that parallelly computes representations of various images to improve the speed of the classification.

Acknowledgment
==============

Our sincere gratitude to Prof. Ashish Ghosh, Indian Statistical Institute (ISI), Kolkata, India, for his insightful comments improving the scientific maturity of this manuscript.

{'source': 'arXiv_out/2101'}