YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
MNIST Classification & Adversarial Machine Learning
A multi-model machine learning project built around the MNIST handwritten digit dataset. The project explores image classification using PyTorch-based neural networks, CNNs, and Linear SVM, then extends the SVM experiment into Adversarial Machine Learning through evasion and data-poisoning attacks.
The project therefore combines three major areas:
Computer Vision → Deep Learning → Machine Learning Security
📌 Project Overview
MNIST is one of the most widely used benchmark datasets in computer vision and machine learning.
It consists of grayscale images of handwritten digits from 0 to 9, where each image has a resolution of:
28 × 28 pixels
The complete dataset contains:
70,000 images
60,000 → Training
10,000 → Testing
There are 10 classes:
0 1 2 3 4 5 6 7 8 9
This project uses MNIST to investigate how different machine-learning architectures perform on handwritten digit classification and how a classifier can behave under adversarial manipulation.
🎯 Objectives
The project has four primary objectives:
1. Build a Fully Connected Neural Network
Implement a Multi-Layer Perceptron (MLP) that classifies flattened MNIST images.
2. Build a Convolutional Neural Network
Use convolutional layers to learn spatial features directly from the original 28 × 28 image structure.
3. Compare Different Machine Learning Approaches
Evaluate different modeling approaches:
MLP
│
▼
CNN
│
▼
SVM
The MLP and CNN perform 10-class digit classification, while the SVM experiment focuses on a binary:
5 vs 9
classification problem.
4. Study Adversarial Machine Learning
Investigate the robustness of the SVM classifier against:
- Adversarial evasion attacks
- Training-data poisoning attacks
🗂️ Dataset
MNIST
Each MNIST sample is represented as a grayscale image.
| Property | Value |
|---|---|
| Image Size | 28 × 28 |
| Channels | 1 |
| Pixel Range | 0–255 |
| Classes | 10 |
| Training Images | 60,000 |
| Test Images | 10,000 |
| Total Images | 70,000 |
Normalization
For the OpenML/SecML experiments, pixel values are normalized using:
X = X.astype("float32")
X /= 255.0
This converts the pixel range from:
0 → 255
to:
0 → 1
For the PyTorch implementation, ToTensor() converts pixel values to approximately 0–1, followed by optional normalization using:
transforms.Normalize((0.5,), (0.5,))
which produces approximately:
-1 → 1
📊 Dataset Structure
MNIST
│
┌───────────┴───────────┐
│ │
Training Testing
60,000 10,000
│ │
└───────────┬───────────┘
│
10 Digit Classes
│
┌────┬────┬────┬────┬────┐
0 1 2 3 4 ...
5 6 7 8 9
🤖 Models
The project contains three main modeling approaches:
| Model | Task | Framework |
|---|---|---|
| MLP | 10-class classification | PyTorch |
| CNN | 10-class classification | PyTorch + Skorch |
| Linear SVM | Binary 5 vs 9 classification |
SecML |
The SVM experiment is then extended with adversarial attacks.
1️⃣ Multi-Layer Perceptron (MLP)
The first model is a simple fully connected neural network, also known as a Multi-Layer Perceptron.
Because a fully connected network expects a vector input, each MNIST image is flattened:
28 × 28
↓
784 features
Architecture
Input
784
│
▼
Linear
784 → 100
│
▼
ReLU
│
▼
Linear
100 → 50
│
▼
ReLU
│
▼
Output
50 → 10
│
▼
Digit Prediction
Layer Configuration
| Layer | Input | Output | Activation |
|---|---|---|---|
| Linear 1 | 784 | 100 | ReLU |
| Linear 2 | 100 | 50 | ReLU |
| Output | 50 | 10 | — |
The final layer produces 10 logits, corresponding to the ten MNIST classes.
Implementation
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.linear1 = nn.Linear(28 * 28, 100)
self.linear2 = nn.Linear(100, 50)
self.final = nn.Linear(50, 10)
self.relu = nn.ReLU()
def forward(self, img):
x = img.view(-1, 28 * 28)
x = self.relu(self.linear1(x))
x = self.relu(self.linear2(x))
x = self.final(x)
return x
⚙️ MLP Training
The MLP uses:
Loss Function: CrossEntropyLoss
Optimizer: Adam
Learning Rate: 0.001
Epochs: 10
Training Pipeline
MNIST Batch
│
▼
28 × 28 Image
│
▼
Flatten → 784
│
▼
Linear Layer
│
▼
ReLU
│
▼
Linear Layer
│
▼
ReLU
│
▼
Output Layer
│
▼
Cross Entropy Loss
│
▼
Backpropagation
│
▼
Adam Optimizer
2️⃣ Convolutional Neural Network (CNN)
The second approach uses a Convolutional Neural Network, which is more suitable for image data because it preserves the spatial structure of the input.
Instead of flattening the image immediately, the CNN receives:
1 × 28 × 28
Architecture
Input
1 × 28 × 28
│
▼
Conv2D
1 → 32
│
▼
ReLU
│
▼
Max Pooling
│
▼
Conv2D
32 → 64
│
▼
Dropout
│
▼
ReLU
│
▼
Max Pooling
│
▼
Flatten
│
▼
Fully Connected
1600 → 100
│
▼
Dropout
│
▼
Output
100 → 10
CNN Implementation
class Cnn(nn.Module):
def __init__(self, dropout=0.5):
super(Cnn, self).__init__()
self.conv1 = nn.Conv2d(
1, 32, kernel_size=3
)
self.conv2 = nn.Conv2d(
32, 64, kernel_size=3
)
self.conv2_drop = nn.Dropout2d(
p=dropout
)
self.fc1 = nn.Linear(1600, 100)
self.fc2 = nn.Linear(100, 10)
self.fc1_drop = nn.Dropout(
p=dropout
)
🧠 Why CNN?
A fully connected network treats the flattened pixels as individual input features.
A CNN can instead learn local spatial patterns.
Examples include:
Edges
↓
Curves
↓
Corners
↓
Stroke Patterns
↓
Digit Features
↓
Digit Class
This makes CNNs particularly effective for image classification.
⚙️ CNN Training
The CNN uses:
Optimizer: Adam
Learning Rate: 0.002
Epochs: 10
Dropout: 0.5
The model is trained using Skorch's NeuralNetClassifier, which provides a scikit-learn-compatible interface for PyTorch models.
3️⃣ Linear SVM
The third experiment uses a Support Vector Machine instead of a neural network.
Unlike the MLP and CNN experiments, the SVM experiment focuses specifically on two MNIST classes:
5 vs 9
This creates a binary classification problem.
The dataset is loaded using SecML:
loader = CDataLoaderMNIST()
Only samples belonging to digits 5 and 9 are selected.
🔢 Binary Classification
The experiment is configured around:
Digit 5
vs
Digit 9
The dataset configuration includes:
Training Samples: 100
Validation Samples: 500
Test Samples: 500
The images are normalized:
tr.X /= 255
val.X /= 255
ts.X /= 255
🤖 Linear SVM Architecture
A linear SVM is configured using:
CClassifierSVM(
C=10,
kernel="linear"
)
Workflow
MNIST
│
├── Digit 5
└── Digit 9
│
▼
Preprocessing
│
▼
Linear SVM
│
▼
Prediction
│
▼
5 / 9
🛡️ Adversarial Machine Learning
The SVM experiment is extended to study machine-learning security and robustness.
Two attack scenarios are investigated:
1. Evasion Attack
2. Data Poisoning Attack
These attacks target different stages of the machine-learning pipeline.
⚔️ 1. Evasion Attack
An evasion attack occurs after the model has already been trained.
Instead of modifying the model or its training data, the attacker modifies an input sample in an attempt to cause an incorrect prediction.
The project uses:
PGD-LS
Projected Gradient Descent with Line Search
through:
CAttackEvasionPGDLS
The attack uses an L2 perturbation constraint:
noise_type = "l2"
dmax = 2.5
🔄 Evasion Attack Pipeline
Original Image
│
▼
Trained SVM
│
▼
Correct Prediction
│
▼
Evasion Attack
│
▼
Perturbed Image
│
▼
SVM
│
▼
Potentially Incorrect Prediction
Conceptually:
Original
↓
5
+
Small Adversarial Perturbation
↓
Adversarial Example
↓
9
The purpose is to investigate how small changes to an input can affect the model's decision.
☠️ 2. Data Poisoning Attack
A poisoning attack occurs during the training phase.
Instead of modifying test samples, the attacker attempts to manipulate the training dataset.
The project uses:
CAttackPoisoningSVM
with:
Number of Poisoning Points = 15
🔄 Poisoning Attack Pipeline
Original Training Data
│
▼
Poisoning Attack
│
▼
Malicious Training Samples
│
▼
Add Samples
│
▼
Retrain SVM
│
▼
Modified Classifier
│
▼
Performance Evaluation
The objective is to measure how carefully crafted training samples can influence the learned decision boundary and classifier performance.
⚔️ Evasion vs Poisoning
| Attack | Target | Stage | Main Idea |
|---|---|---|---|
| Evasion | Input samples | Inference | Modify inputs to cause misclassification |
| Poisoning | Training data | Training | Manipulate training samples |
| Evasion | Model prediction | After training | Fool an already-trained model |
| Poisoning | Learned decision boundary | During training | Influence the model through malicious data |
The key difference is:
Evasion → attacks the input
Poisoning → attacks the training data
📊 Evaluation
The project evaluates model performance primarily using classification accuracy.
MLP
accuracy_score(
y_test,
y_pred
)
CNN
The CNN predictions are evaluated using the same general classification-metric approach.
SVM
SecML provides:
CMetricAccuracy()
for accuracy evaluation.
📈 Evaluation Strategy
MLP
MNIST
↓
MLP
↓
10-Class Prediction
↓
Accuracy
CNN
MNIST
↓
CNN
↓
10-Class Prediction
↓
Accuracy
SVM
5 vs 9
↓
Linear SVM
↓
Binary Prediction
↓
Accuracy
Adversarial Evaluation
Clean Model
│
▼
Baseline Accuracy
│
├───────────────┐
▼ ▼
Evasion Attack Poisoning Attack
│ │
▼ ▼
Adversarial Retrained
Evaluation Classifier
│ │
└───────┬───────┘
▼
Robustness Analysis
🖼️ Visualization
The project includes visualization of:
- MNIST handwritten digits
- Model predictions
- Incorrect predictions
- Adversarial examples
- Poisoning samples
- Perturbations
Example:
┌─────┬─────┬─────┬─────┬─────┐
│ 5 │ 9 │ 5 │ 9 │ 5 │
│ │ │ │ │ │
└─────┴─────┴─────┴─────┴─────┘
Incorrect predictions can be isolated using:
error_mask = y_pred != y_test
This makes it possible to inspect the samples where the classifier fails.
🧪 Experimental Comparison
The overall project can be summarized as:
MNIST
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
MLP CNN SVM
│ │ │
▼ ▼ ▼
10 Classes 10 Classes 5 vs 9
│
┌────────────┴────────────┐
│ │
▼ ▼
Evasion Poisoning
Attack Attack
│ │
└────────────┬────────────┘
▼
Robustness Analysis
This allows the project to compare both classification approaches and security behavior.
🛠️ Technologies
Deep Learning
- PyTorch
- Torchvision
- Torch Neural Networks
- Skorch
Machine Learning
- Scikit-learn
- Linear SVM
Adversarial Machine Learning
- SecML
Data Processing
- NumPy
- OpenML
- Torchvision Datasets
Visualization
- Matplotlib
📦 Installation
Install the main dependencies:
pip install torch torchvision
pip install scikit-learn
pip install numpy matplotlib
pip install skorch
For the adversarial machine-learning experiments:
pip install secml
The notebooks also contain installation steps suitable for Google Colab.
🚀 Getting Started
1. PyTorch MLP
Load MNIST using Torchvision:
from torchvision import datasets
import torchvision.transforms as transforms
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.5,), (0.5,))
])
train_dataset = datasets.MNIST(
root="./data",
train=True,
download=True,
transform=transform
)
test_dataset = datasets.MNIST(
root="./data",
train=False,
download=True,
transform=transform
)
Then create the DataLoaders, initialize the MLP, and train the model.
2. MLP with OpenML + Skorch
MNIST can also be retrieved through OpenML:
from sklearn.datasets import fetch_openml
mnist = fetch_openml(
"mnist_784",
as_frame=False,
cache=False
)
Normalize the data:
X = mnist.data.astype("float32")
X /= 255.0
3. CNN
Convert the flattened vectors back into image tensors:
XCnn = X.reshape(
-1, 1, 28, 28
)
The resulting shape is:
Number of Samples × 1 × 28 × 28
The CNN can then be trained using Skorch's NeuralNetClassifier.
4. SVM Security Experiment
Load the MNIST subset containing:
5
9
Train the Linear SVM and measure its baseline performance.
Then evaluate:
Baseline
↓
PGD-LS Evasion Attack
↓
SVM Poisoning Attack
↓
Robustness Analysis
📁 Suggested Repository Structure
MNIST-ML-Experiments/
│
├── README.md
├── requirements.txt
│
├── notebooks/
│ ├── mnist_mlp_pytorch.ipynb
│ ├── mnist_mlp_skorch.ipynb
│ ├── mnist_cnn.ipynb
│ └── mnist_adversarial_ml.ipynb
│
├── models/
│ ├── mlp/
│ ├── cnn/
│ └── svm/
│
├── results/
│ ├── accuracy/
│ ├── predictions/
│ ├── adversarial_examples/
│ └── poisoning_examples/
│
└── data/
└── MNIST/
🔍 Key Concepts Demonstrated
Neural Networks
- Fully connected layers
- Activation functions
- Forward propagation
- Backpropagation
- Gradient-based optimization
- Loss functions
- Adam optimizer
Computer Vision
- Image tensors
- Convolution
- Pooling
- Feature extraction
- Image classification
Classical Machine Learning
- Support Vector Machines
- Binary classification
- Train/test splitting
- Model evaluation
- Classification accuracy
Machine Learning Security
- Adversarial examples
- Evasion attacks
- PGD-based attacks
- Data poisoning
- Adversarial perturbations
- Model robustness
⚠️ Implementation Notes
1. Training vs Testing Data
One important issue in the original MLP implementation is the use of test_loader inside the training loop.
The training process should normally use:
for data in train_loader:
...
while testing should be performed separately:
for data in test_loader:
...
The correct workflow is:
Training Data
│
▼
Model Training
│
▼
Trained Model
│
▼
Test Data
│
▼
Final Evaluation
Using the training set for evaluation can produce an overly optimistic estimate of model performance.
2. CrossEntropyLoss and Softmax
Another important implementation consideration concerns Softmax.
When using:
nn.CrossEntropyLoss()
the model should normally return raw logits rather than applying Softmax inside forward().
For example:
def forward(self, x):
x = F.relu(self.hidden(x))
x = self.dropout(x)
x = self.output(x)
return x
CrossEntropyLoss internally applies the required log-softmax operation.
For inference, class probabilities can be obtained separately:
probabilities = torch.softmax(logits, dim=1)
This separation provides a cleaner and more standard PyTorch implementation.
📊 Recommended Metrics
The current project primarily uses accuracy, but future experiments can include:
- Accuracy
- Precision
- Recall
- F1-score
- Confusion Matrix
- ROC-AUC
- Clean accuracy
- Adversarial accuracy
- Attack success rate
- Perturbation magnitude
For adversarial experiments, comparing clean vs adversarial performance is particularly useful.
🚀 Future Improvements
Possible extensions include:
Model Improvements
- Compare MLP vs CNN accuracy
- Experiment with deeper CNN architectures
- Add Batch Normalization
- Tune learning rates
- Tune dropout
- Perform hyperparameter optimization
- Test additional SVM kernels
Dataset Improvements
- Increase the size of the
5 vs 9dataset - Apply data augmentation
- Experiment with different normalization strategies
Evaluation Improvements
- Add confusion matrices
- Add precision, recall, and F1-score
- Add ROC curves
- Compare training and test accuracy
- Track loss curves
Adversarial ML Improvements
- Test additional evasion attacks
- Compare L1, L2, and L∞ perturbations
- Evaluate multiple attack budgets
- Visualize perturbation magnitude
- Measure attack success rate
- Compare clean vs adversarial accuracy
- Experiment with adversarial training
- Investigate certified robustness methods
🧠 Learning Outcomes
This project provides hands-on experience with:
- MNIST image classification
- PyTorch fundamentals
- Neural network architecture
- CNN architecture
- Data preprocessing
- Data normalization
- PyTorch DataLoaders
- Model training
- Backpropagation
- Adam optimization
- Cross-entropy loss
- Support Vector Machines
- Scikit-learn
- Skorch
- SecML
- Adversarial examples
- Evasion attacks
- PGD-based attacks
- Data poisoning
- Model robustness
- Machine-learning security
🔑 Keywords
MNIST
Digit Classification
Handwritten Digit Recognition
PyTorch
Torchvision
Neural Network
MLP
CNN
Convolutional Neural Network
SVM
Support Vector Machine
Scikit-learn
Skorch
SecML
Adversarial Machine Learning
Adversarial Attacks
Evasion Attack
PGD
Data Poisoning
Machine Learning Security
Computer Vision
Deep Learning
Model Robustness
📜 Disclaimer
This project is intended for educational, research, and experimental purposes.
The adversarial machine-learning experiments are designed to study model robustness and understand potential vulnerabilities in machine-learning classifiers.
All security-related experiments should be conducted in controlled environments and against systems or datasets for which you have authorization.
👨💻 Project Summary
MNIST Classification & Adversarial Machine Learning is a multi-model computer-vision and machine-learning security project.
It begins with traditional neural-network classification using an MLP, progresses to spatial feature learning using a CNN, and then explores classical machine learning and adversarial robustness using a Linear SVM.
The complete workflow is:
MNIST
│
┌────────┴────────┐
│ │
▼ ▼
MLP CNN
│ │
└────────┬────────┘
│
▼
SVM
│
5 vs 9
│
┌────────┴────────┐
│ │
▼ ▼
Evasion Poisoning
Attack Attack
│ │
└────────┬────────┘
▼
Robustness Analysis
The project demonstrates not only how to build image classifiers, but also how different machine-learning approaches can be evaluated under adversarial conditions.
⭐ Project Highlights
✔ MNIST Handwritten Digit Classification
✔ PyTorch MLP
✔ Convolutional Neural Network
✔ Linear SVM
✔ Scikit-learn / Skorch Integration
✔ SecML Adversarial ML
✔ PGD-LS Evasion Attack
✔ SVM Data Poisoning
✔ Adversarial Example Visualization
✔ Model Robustness Analysis
✔ Clean vs Adversarial Evaluation
📌 Project Focus
Computer Vision
+
Deep Learning
+
Classical Machine Learning
+
Adversarial Machine Learning
=
End-to-End ML Experimentation