nani1764/smai-mlp-model

Selected Models for SMAI Assignment 4 – Question 1

This repository contains the selected Multi-Layer Perceptron (MLP) and IOC-NN models evaluated on the CIFAR-10 dataset.

The MLP is implemented from scratch using NumPy, while the IOC-NN model is implemented using PyTorch.

1. Model Architectures

1.1 Multi-Layer Perceptron (MLP)

Property	Description
Implementation	NumPy (from scratch)
Data Type	float64
Architecture	3072 → 64 → 10
Hidden Activation	ReLU
Output Activation	Identity (Linear logits)
Number of Parameters	197,322
Validation Accuracy	48.67%
Test Accuracy	48.95%

Model File: selected_mlp_cifar10.npz

1.2 IOC-NN

Property	Description
Implementation	PyTorch (torch.nn)
Architecture	3072 → 64 → 10
Hidden Activation	ELU
Output Activation	Identity (Linear logits)
Number of Parameters	197,322
Test Accuracy (Clean, p = 0)	45.52%
Best Epoch Test Accuracy	50.66%

Model Files:

• selected_iocnn_cifar10.npz
• selected_iocnn_cifar10_state_dict.pth

Weight Constraints

For the IOC-NN model, the weights of layers with index $\ell \geq 2$ are constrained to be non-negative:

$$W^{(\mathrm{\ell })}\ge 0,\mathrm{\ell }\ge 2W^\{(\backslash ell)\}\; \backslash geq\; 0,\; \backslash qquad\; \backslash ell\; \backslash geq\; 2$$

The constraint is enforced using the following projection:

$$\boxed{{\displaystyle w\leftarrow e^{w-5}}}\backslash boxed\{w\; \backslash leftarrow\; e^\{w-5\}\}$$

2. Dataset and Preprocessing

Both models use the CIFAR-10 dataset, which consists of RGB images of dimensions $32 \times 32$.

2.1 Input Dimensions

Each image is flattened into a one-dimensional feature vector:

$$32\times 32\times 3=307232\; \backslash times\; 32\; \backslash times\; 3\; =\; 3072$$

Therefore, the input dimension of both models is:

$$\boxed{{\displaystyle d_{\text{input}}=3072}}\backslash boxed\{d\_\{\backslash text\{input\}\}\; =\; 3072\}$$

2.2 Pixel Scaling

The original pixel values are integers in the range $[0,255]$.

They are scaled to the range $[0,1]$ using:

$$\boxed{{\displaystyle x_{\text{scaled}}=\frac{x}{255.0}}}\backslash boxed\{x\_\{\backslash text\{scaled\}\}\; =\; \backslash frac\{x\}\{255.0\}\}$$

The resulting data is represented using float64.

2.3 Feature Standardization

After scaling, the input features are standardized column-wise using the mean and standard deviation calculated exclusively from the training split.

The standardization formula is:

$$\boxed{{\displaystyle x_{\text{norm}}=\frac{x_{\text{scaled}}-{\mu }_{\text{train}}}{{\sigma }_{\text{train}}}}}\backslash boxed\{\; x\_\{\backslash text\{norm\}\}\; =\; \backslash frac\{x\_\{\backslash text\{scaled\}\}-\backslash mu\_\{\backslash text\{train\}\}\}\; \{\backslash sigma\_\{\backslash text\{train\}\}\}\; \}$$

where:

• $x_{\text{scaled}}$ represents the scaled input feature.
• $\mu_{\text{train}}$ represents the training-set mean.
• $\sigma_{\text{train}}$ represents the training-set standard deviation.
• $x_{\text{norm}}$ represents the standardized feature.

Note: Validation and test data use the same training-set statistics for standardization. Their own means and standard deviations are not used, preventing data leakage.

3. Model Summary

Both selected models use the same architecture:

$$\boxed{{\displaystyle 3072\to 64\to 10}}\backslash boxed\{3072\; \backslash rightarrow\; 64\; \backslash rightarrow\; 10\}$$

The primary differences are their implementations and hidden-layer activation functions:

Feature	MLP	IOC-NN
Framework	NumPy	PyTorch
Hidden Activation	ReLU	ELU
Output Activation	Identity	Identity
Weight Constraints	Not specified	Non-negative weights for $\ell \geq 2$

Both models produce 10 output logits corresponding to the 10 CIFAR-10 classes.