dfgfhdhhhghg/nn2-cifar10-project2-weights

Neural Network and Deep Learning Project-2

This repository contains a reproducible PyTorch empirical study for Project-2 of Neural Network and Deep Learning. The project has two required parts:

• Task 1: CIFAR-10 classification, including custom CNN architectures, model optimization, ablation studies, and model interpretation.
• Task 2: Batch Normalization analysis, comparing VGG-A with and without BatchNorm under multiple learning rates and visualizing loss variation bands.

The best CIFAR-10 result currently obtained is:

Best run	Test Acc	Test Error	Params	Train Time	Checkpoint
final_se_silu_adamw_warmup_cutmix	96.05%	3.95%	2,800,130	23m 11s	checkpoints/final_se_silu_adamw_warmup_cutmix/best.pt

Compared with the simple cnn_small_baseline at 75.66% test accuracy, the final model improves by +20.39 percentage points. Compared with the earlier strong residual baseline cnn_residual_aug_adamw_best at 94.93%, the final extended model improves by +1.12 percentage points.

Before final PDF submission, fill these in the report:

• Name: TODO
• Student ID: TODO
• Github code link: https://github.com/hyq-hyqhyq/nn2
• Dataset link: CIFAR-10 official dataset link or uploaded dataset link
• Model weights link: uploaded checkpoints/ link, for example Google Drive / Netdisk

Requirement Coverage

Project requirement	Implementation in this repo
Fully-connected layer	CIFARConvNet classifier MLP and VGGA classifier in src/project2/models.py
2D convolutional layer	All CNN, residual CNN, VGG-A, SE, and CBAM models use nn.Conv2d
2D pooling layer	MaxPool2d in CNN/VGG/residual stages; AdaptiveAvgPool2d for GAP
Activation function	ReLU, LeakyReLU, GELU, SiLU, Mish implemented through get_activation()
BatchNorm / Dropout / Residual / others	BatchNorm, Dropout, residual connections, GAP, SE, CBAM, GroupNorm, stochastic depth
Different filters / neurons	Small, medium, large, tiny, deep-medium, wide, deep-wide channel/depth sweeps
Different losses / regularization	CrossEntropy, weight decay, label smoothing, Focal Loss, MixUp, CutMix, RandomErasing/Cutout
Different activations	ReLU, LeakyReLU, GELU, SiLU, Mish
Optimizers	SGD, SGD + momentum, SGD + Nesterov, RMSprop, Adam, AdamW
Schedulers	Constant LR, StepLR, CosineAnnealingLR, OneCycleLR, Warmup + Cosine
Visualization	Loss/accuracy curves, comparison plots, confusion matrix, filters, misclassified examples, per-class accuracy, BN loss landscape
BatchNorm analysis	VGG-A and VGG-A-BN trained under 1e-4, 5e-4, 1e-3, 2e-3; loss bands use max/min curves

Code Structure

.
├── configs/                         # Single-run config examples
├── scripts/
│   ├── run_experiment.py            # Run one JSON-config experiment
│   ├── run_suite.py                 # Run predefined suites
│   ├── plot_results.py              # Generate all result figures
│   └── build_report_materials.py    # Generate report tables and REPORT_MATERIALS.md
├── src/project2/
│   ├── data.py                      # CIFAR-10 loaders and augmentations
│   ├── engine.py                    # Training loop, optimizers, schedulers, metrics, checkpoints
│   ├── models.py                    # CNN, residual CNN, VGG-A, SE, CBAM, etc.
│   └── utils.py                     # Reproducibility, metrics, IO helpers
├── results/
│   ├── summary.csv                  # Canonical full experiment summary, 95 rows
│   ├── summary.json                 # Same information in JSON
│   ├── REPORT_MATERIALS.md          # Auto-generated tables and suggested findings
│   ├── figures/                     # All saved plots
│   └── tables/                      # Report-ready CSV tables
└── checkpoints/                     # Best/final weights, not recommended for Github upload

The code intentionally uses custom CIFAR models instead of directly using torchvision ResNet18 as the final model.

Environment

Recommended server setup:

git clone https://github.com/hyq-hyqhyq/nn2.git
cd nn2

conda create -n nn2 python=3.10 -y
conda activate nn2

pip install --upgrade pip
pip install -r requirements.txt

For a machine whose driver reports CUDA 12.8, installing a PyTorch CUDA 12.6 wheel is normally fine because the NVIDIA driver is backward compatible with CUDA runtime wheels:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt

Quick check:

python - <<'PY'
import torch
print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else "cpu")
PY

Dataset and Preprocessing

Dataset: CIFAR-10, 60,000 RGB images of size 32 x 32, with 50,000 train images and 10,000 test images across 10 classes.

Implementation: src/project2/data.py

• Download method: torchvision.datasets.CIFAR10
• Normalization mean: (0.4914, 0.4822, 0.4465)
• Normalization std: (0.2470, 0.2435, 0.2616)
• none: tensor conversion + normalization only
• basic: random crop with padding 4 + random horizontal flip
• strong: basic augmentation + RandAugment + ColorJitter
• cutout / random_erasing: RandomErasing after normalization

Reproducing Experiments

Smoke test:

python scripts/run_suite.py --suite smoke --data-root data --download --device cuda

Original required experiments:

python scripts/run_suite.py \
  --suite all \
  --data-root data \
  --download \
  --device cuda \
  --epochs-main 100 \
  --epochs-ablation 50 \
  --epochs-bn 20 \
  --make-artifacts

Extended high-score experiments:

python scripts/run_suite.py \
  --suite extended_all \
  --data-root data \
  --download \
  --device cuda \
  --epochs-screening 30 \
  --epochs-focused 100 \
  --epochs-final 180 \
  --skip-existing \
  --make-artifacts

Regenerate all figures and report tables after training:

python scripts/plot_results.py \
  --results-dir results \
  --checkpoint-dir checkpoints \
  --data-root data \
  --device cuda \
  --download

python scripts/build_report_materials.py --results-dir results

Additional high-value analyses:

python scripts/analyze_bn_deep.py \
  --results-dir results \
  --checkpoint-dir checkpoints \
  --data-root data \
  --device cuda \
  --download \
  --run-diagnostics \
  --diagnostic-epochs 15 \
  --diagnostic-lrs 0.001

python scripts/generate_gradcam.py \
  --results-dir results \
  --checkpoint-dir checkpoints \
  --data-root data \
  --device cuda \
  --download \
  --experiment final_se_silu_adamw_warmup_cutmix

python scripts/build_report_materials.py --results-dir results

Run a single experiment from config:

python scripts/run_experiment.py --config configs/best_model.json

Each experiment saves:

• Config: results/runs/<experiment>/config.json
• Epoch metrics: results/runs/<experiment>/metrics.csv
• Step losses when enabled: results/runs/<experiment>/step_losses.csv
• Summary row: results/runs/<experiment>/summary_row.json
• Best checkpoint: checkpoints/<experiment>/best.pt
• Final checkpoint: checkpoints/<experiment>/final.pt

Main Models

CNN-Small Family

Implemented by CIFARConvNet in src/project2/models.py.

Default structure:

[Conv2d 3x3 -> optional norm -> activation -> optional Dropout2d -> optional attention -> MaxPool2d] x 3
Flatten or Global Average Pooling
Dropout
Linear hidden layer
Activation
Dropout
Linear classifier

This family is used for the simple baseline, BatchNorm/dropout experiments, filter-count ablations, activation ablations, and optimizer ablations.

Custom Residual CNN

Implemented by CIFARResidualNet in src/project2/models.py.

Default structure:

Stem: Conv2d 3x3 -> norm -> activation
Stage 1: residual blocks at 32x32
MaxPool2d
Stage 2: residual blocks at 16x16
MaxPool2d
Stage 3: residual blocks at 8x8
AdaptiveAvgPool2d
Dropout
Linear classifier

Each residual block contains:

Conv2d 3x3 -> norm -> activation -> optional Dropout2d
Conv2d 3x3 -> norm -> optional SE/CBAM -> optional stochastic depth
Skip connection
Activation

This is the main high-performance model family. It is a custom small residual network for CIFAR-10, not a public torchvision model.

VGG-A and VGG-A-BN

Implemented by VGGA in src/project2/models.py.

VGG-A CIFAR configuration:

64, MaxPool,
128, MaxPool,
256, 256, MaxPool,
512, 512, MaxPool,
512, 512, MaxPool,
Linear 512 -> Linear 512 -> Linear 10

vgg_a_bn adds BatchNorm after every convolution. These models are used only for Task 2 BatchNorm analysis.

Experiment Design

The experiments are organized in three layers:

1. Required coverage experiments: baseline CNNs, filter ablation, activation ablation, regularization/loss ablation, optimizer comparison, VGG-A BatchNorm analysis.
2. Extended single-factor ablations: GAP, SE, CBAM, GroupNorm, stochastic depth, capacity width/depth sweeps, MixUp/CutMix, optimizer-scheduler grid.
3. Final combined candidates: combine strong architecture, augmentation, optimizer, scheduler, activation, and regularization. These final models are not interpreted as single-factor ablations.

The full 95-row experiment summary is in:

• results/summary.csv
• results/summary.json
• results/tables/table1_overall.csv

Main CIFAR-10 Results

Accuracy and error below are test-set values.

Experiment	Main setting	Params	Train Time	Best Test Acc	Test Error
cnn_small_baseline	CNN, no BN, no dropout, Adam	620,362	3m 55s	75.66%	24.34%
cnn_bn	CNN + BatchNorm, Adam	620,586	3m 42s	77.75%	22.25%
cnn_bn_dropout	CNN + BN + dropout, AdamW, basic aug	620,586	3m 43s	85.61%	14.39%
cnn_residual	Custom residual CNN, Adam, basic aug	2,777,674	9m 33s	93.78%	6.22%
cnn_residual_aug_adamw_best	Residual + SiLU + strong aug + AdamW + label smoothing	2,777,674	9m 35s	94.93%	5.07%
final_se_silu_adamw_warmup_cutmix	Residual + SE + SiLU + warmup cosine + CutMix	2,800,130	23m 11s	96.05%	3.95%

Useful report interpretation:

• BatchNorm alone improves the small CNN from 75.66% to 77.75%.
• Adding dropout, AdamW, and basic augmentation gives a larger jump to 85.61%.
• The custom residual architecture is the largest single architecture gain, reaching 93.78%.
• Strong augmentation, SiLU, AdamW, label smoothing, and cosine scheduling improve the residual model to 94.93%.
• The best final model adds SE attention, stochastic depth, warmup cosine, CutMix, and longer focused training, reaching 96.05%.

Top Extended Results

Experiment	Group	Key setting	Params	Best Test Acc	Test Error
final_se_silu_adamw_warmup_cutmix	final	SE + SiLU + AdamW + warmup cosine + CutMix	2,800,130	96.05%	3.95%
focused_reg_cutmix	focused	CutMix focused retraining	2,800,130	95.34%	4.66%
focused_capacity_wide	focused	Wide residual [96,192,384], 2 blocks/stage	6,243,178	95.26%	4.74%
focused_capacity_deep_medium	focused	Deep-medium residual [64,128,256], 3 blocks/stage	4,327,754	95.22%	4.78%
focused_reg_mixup	focused	MixUp focused retraining	2,800,130	95.00%	5.00%
final_cbam_silu_adamw_warmup_mixup	final	CBAM + SiLU + AdamW + warmup cosine + MixUp	2,800,718	94.94%	5.06%
focused_optsched_sgd_nesterov_cosine	focused	SGD Nesterov + cosine	2,800,130	94.83%	5.17%
final_deepwide_se_mish_sgd_nesterov	final	Deep-wide SE + Mish + SGD Nesterov	9,804,232	94.81%	5.19%

Key conclusion: the 2.8M-parameter final SE model is better than the much larger 9.8M deep-wide final candidate, so the best result is not simply from adding parameters.

Architecture and Component Ablation

Full table: results/tables/component_ablation.csv Figure: results/figures/component_comparison.png

Experiment	Backbone	Component changed	Params	Best Test Acc	Test Error
component_cnn_fc	CNN-BN-Dropout	Fully-connected classifier	620,586	83.51%	16.49%
component_cnn_gap	CNN-BN-Dropout	GAP classifier	129,066	75.35%	24.65%
component_residual_gap	Residual	GAP baseline	2,777,674	92.78%	7.22%
component_residual_se	Residual	SE attention	2,800,130	91.80%	8.20%
component_residual_cbam	Residual	CBAM attention	2,800,718	91.09%	8.91%
component_residual_groupnorm	Residual	GroupNorm instead of BatchNorm	2,777,674	90.69%	9.31%
component_residual_stochdepth	Residual	Stochastic depth	2,777,674	92.43%	7.57%

Interpretation:

• Naive GAP greatly reduces parameters in the small CNN, but accuracy drops because the classifier capacity becomes too small.
• The residual model already uses GAP naturally and performs much better than the plain CNN.
• In short screening, SE/CBAM were not immediately better than the residual-GAP baseline, but SE became useful in focused/final training when combined with strong regularization and longer schedules.
• GroupNorm under this CIFAR-10 setting underperformed BatchNorm, supporting the usefulness of BN for this dataset and batch size.

Capacity Ablation

Full table: results/tables/capacity_ablation_extended.csv Figure: results/figures/capacity_accuracy_params_tradeoff.png

Variant	Channels	Blocks / Stage	Params	Best Test Acc	Test Error	Acc / M Params
tiny	[32,64,128]	1	308,650	87.39%	12.61%	2.8314
small	[48,96,192]	1	691,834	89.40%	10.60%	1.2922
medium	[64,128,256]	2	2,777,674	92.55%	7.45%	0.3332
deep-medium	[64,128,256]	3	4,327,754	92.68%	7.32%	0.2142
wide	[96,192,384]	2	6,243,178	92.87%	7.13%	0.1488
deep-wide	[96,192,384]	3	9,729,514	93.24%	6.76%	0.0958

Focused retraining improved the best capacity candidates:

• focused_capacity_wide: 95.26% test accuracy.
• focused_capacity_deep_medium: 95.22% test accuracy.

Interpretation:

• Increasing width/depth improves raw accuracy, but accuracy per million parameters decreases sharply.
• Tiny and small models are parameter-efficient but not competitive for the best final accuracy.
• Deep-wide has the best quick-screen raw accuracy but is much less efficient than the final 2.8M SE model.

Filter Number Ablation

Full table: results/tables/table2_filter_ablation.csv

Variant	Channels	Params	Best Test Acc	Test Error
small	[24,48,96]	448,866	82.43%	17.57%
medium	[32,64,128]	620,586	83.82%	16.18%
large	[64,128,256]	1,422,666	85.94%	14.06%

Interpretation: larger filter counts consistently improve the plain CNN family, but the gain is smaller than the gain from residual connections.

Activation Ablation

Original CNN activation table: results/tables/table3_activation_ablation.csv Extended residual activation table: results/tables/activation_ablation_extended.csv Figure: results/figures/activation_comparison.png

Original CNN-BN-Dropout results:

Activation	Optimizer	Best Test Acc	Test Error	Convergence
ReLU	AdamW	84.13%	15.87%	epoch 6
LeakyReLU	AdamW	86.88%	13.12%	epoch 4
GELU	AdamW	85.69%	14.31%	epoch 5
SiLU	AdamW	86.04%	13.96%	epoch 4

Extended residual results:

Activation	Optimizer	Scheduler	Best Test Acc	Test Error	Convergence
ReLU	AdamW	cosine	92.65%	7.35%	epoch 3
LeakyReLU	AdamW	cosine	91.96%	8.04%	epoch 2
GELU	AdamW	cosine	92.80%	7.20%	epoch 2
SiLU	AdamW	cosine	92.52%	7.48%	epoch 2
Mish	AdamW	cosine	92.71%	7.29%	epoch 2

Focused result:

• focused_activation_mish: 94.35% test accuracy.

Interpretation:

• In the smaller CNN, LeakyReLU and SiLU clearly beat ReLU.
• In the residual model, activation differences are smaller once BatchNorm/residual connections and cosine scheduling are used.
• GELU/Mish/SiLU are reasonable final candidates, but activation choice alone is not the biggest contributor.

Loss and Regularization Ablation

Original table: results/tables/table4_regularization_ablation.csv Extended table: results/tables/loss_regularization_extended.csv Figure: results/figures/regularization_comparison.png Gap figure: results/figures/train_test_gap_regularization.png

Extended single-factor results:

Variant	Loss / Regularization	Best Test Acc	Train-Test Gap
CrossEntropy	no weight decay	92.48%	0.0653
CrossEntropy + weight decay	weight decay 5e-4	92.70%	0.0635
CrossEntropy + label smoothing	smoothing 0.05	93.27%	0.0627
Focal Loss	gamma 2.0	92.08%	0.0638
MixUp	alpha 0.2	93.24%	-0.4072
CutMix	alpha 1.0	92.26%	-0.1965
RandomErasing / Cutout	random erasing augmentation	89.85%	0.0976

Focused retraining:

• focused_reg_cutmix: 95.34% test accuracy.
• focused_reg_mixup: 95.00% test accuracy.

Interpretation:

• Weight decay gives a small but consistent improvement.
• Label smoothing is the best simple regularizer in the quick single-factor setup.
• MixUp/CutMix change train accuracy interpretation because training labels/images are mixed; the train-test gap can become negative and should be interpreted carefully.
• CutMix is not best in short screening, but becomes the strongest focused/final regularizer with stronger training.
• RandomErasing/Cutout was not helpful in this setup and increased overfitting/instability.

Optimizer and Scheduler Ablation

Original optimizer table: results/tables/table5_optimizer_ablation.csv Extended optimizer-scheduler table: results/tables/optimizer_scheduler_ablation.csv Figures:

• results/figures/optimizer_comparison.png
• results/figures/optimizer_lr_heatmap.png

Original CNN optimizer results:

Optimizer	LR	Weight Decay	Best Test Acc	Stability
SGD	0.05	0.0005	82.25%	last-5 std 0.0005
SGD + momentum	0.08	0.0005	86.85%	last-5 std 0.0013
Adam	0.001	0.0005	85.17%	last-5 std 0.0009
AdamW	0.001	0.01	84.58%	last-5 std 0.0007

Best quick optimizer-scheduler results on the residual backbone:

Optimizer	Scheduler	LR	Best Test Acc	Stability
SGD	OneCycleLR	0.05	93.12%	last-5 std 0.0057
SGD + momentum	Warmup + Cosine	0.08	93.07%	last-5 std 0.0028
SGD + momentum	OneCycleLR	0.08	93.06%	last-5 std 0.0047
SGD + Nesterov	Warmup + Cosine	0.08	92.92%	last-5 std 0.0020
Adam	OneCycleLR	0.001	92.86%	last-5 std 0.0025
AdamW	Cosine	0.0008	92.60%	last-5 std 0.0006

Focused retraining:

• focused_optsched_sgd_nesterov_cosine: 94.83% test accuracy.
• focused_optsched_adamw_warmup: 94.56% test accuracy.

Interpretation:

• Learning-rate schedule matters as much as optimizer choice.
• Constant LR often underperforms scheduled LR, especially for Adam/AdamW/RMSprop.
• RMSprop is clearly weaker in this setup; rmsprop + constant only reaches 71.81%.
• SGD variants can match or exceed AdamW when tuned with OneCycle or cosine schedules.
• AdamW is still a strong and stable final choice when combined with warmup cosine, strong augmentation, label smoothing, and CutMix.

BatchNorm Analysis

Full table: results/tables/table6_bn_comparison.csv Figures:

• results/figures/vgga_bn_loss_curves.png
• results/figures/vgga_bn_loss_landscape.png

VGG-A and VGG-A-BN were trained under learning rates 1e-4, 5e-4, 1e-3, and 2e-3. For the loss landscape-style visualization, each step takes the maximum and minimum losses across learning rates:

max_curve[step] = max(loss_at_same_step_over_learning_rates)
min_curve[step] = min(loss_at_same_step_over_learning_rates)

The area between max_curve and min_curve is plotted with fill_between. A narrower band means smaller loss variation under different step sizes.

Model	LR	BN	Final Train Loss	Best Test Acc	Loss Variation Width
VGG-A-BN	0.0010	with BN	0.0452	83.21%	0.1966
VGG-A-BN	0.0020	with BN	0.0485	83.16%	0.1966
VGG-A-BN	0.0005	with BN	0.0415	81.99%	0.1966
VGG-A-BN	0.0001	with BN	0.0446	74.40%	0.1966
VGG-A	0.0005	without BN	0.0575	78.71%	0.4055
VGG-A	0.0010	without BN	0.0720	78.08%	0.4055
VGG-A	0.0001	without BN	0.0451	75.31%	0.4055
VGG-A	0.0020	without BN	0.2452	73.50%	0.4055

Interpretation:

• VGG-A-BN reaches higher best accuracy than VGG-A at comparable learning rates.
• The BN loss variation width is 0.1966, while the no-BN width is 0.4055.
• This supports the report claim that BatchNorm makes the effective optimization landscape smoother, reduces learning-rate sensitivity, and stabilizes training.
• At lr=0.002, the no-BN model has much larger final train loss, while BN remains stable.

Per-Class Accuracy

Full table: results/tables/per_class_accuracy.csv Figure: results/figures/per_class_accuracy.png

The best model per-class test accuracy:

Class	Accuracy
airplane	96.50%
automobile	98.50%
bird	94.40%
cat	90.40%
deer	97.10%
dog	92.80%
frog	99.00%
horse	96.80%
ship	98.20%
truck	96.80%

Interpretation:

• The hardest classes are cat and dog, which is consistent with CIFAR-10 semantic similarity and low resolution.
• Vehicle and frog classes are easier, with automobile/frog/ship above 98%.
• Use results/figures/confusion_matrix_best_model.png and results/figures/misclassified_examples.png to support this analysis visually.

Result Figures

All figures are saved under results/figures/.

Figure path	Use in report
results/figures/train_loss_curves.png	Training loss curves for selected main models
results/figures/test_accuracy_curves.png	Test accuracy curves for selected main models
results/figures/optimizer_comparison.png	Optimizer comparison for SGD, momentum, Adam, AdamW
results/figures/activation_comparison.png	Activation comparison
results/figures/regularization_comparison.png	Regularization/loss comparison
results/figures/confusion_matrix_best_model.png	Confusion matrix of the best model
results/figures/first_layer_filters.png	First convolution filters of the best model
results/figures/misclassified_examples.png	Examples misclassified by the best model
results/figures/vgga_bn_loss_curves.png	VGG-A vs VGG-A-BN loss/accuracy curves
results/figures/vgga_bn_loss_landscape.png	BN vs no-BN loss variation band
results/figures/bn_loss_landscape_extended.png	Quantitative BN loss-band comparison
results/figures/bn_gradient_norm_curves.png	Gradient norm stability diagnostic for VGG-A vs VGG-A-BN
results/figures/bn_activation_mean_std.png	Activation mean/std comparison from forward hooks
results/figures/bn_activation_histograms.png	Activation distribution histograms from forward hooks
results/figures/gradcam_correct_examples.png	Grad-CAM examples for correctly classified CIFAR-10 images
results/figures/gradcam_misclassified_examples.png	Grad-CAM examples for misclassified CIFAR-10 images
results/figures/component_comparison.png	GAP, SE, CBAM, GroupNorm, stochastic depth comparison
results/figures/capacity_accuracy_params_tradeoff.png	Accuracy vs parameter tradeoff
results/figures/train_test_gap_regularization.png	Train-test gap under regularization methods
results/figures/optimizer_lr_heatmap.png	Optimizer/scheduler heatmap
results/figures/per_class_accuracy.png	Per-class accuracy of the best model

Result Tables

All report-ready tables are saved under results/tables/.

Table path	Content
results/tables/table1_overall.csv	All 95 experiments: model, params, optimizer, activation, BN, dropout, residual, train time, accuracy, error
results/tables/table2_filter_ablation.csv	Original filter-number ablation
results/tables/table3_activation_ablation.csv	Original CNN activation ablation
results/tables/table4_regularization_ablation.csv	Original regularization ablation
results/tables/table5_optimizer_ablation.csv	Original optimizer comparison
results/tables/table6_bn_comparison.csv	VGG-A with/without BN under multiple learning rates
results/tables/bn_lr_robustness.csv	BN learning-rate robustness metrics
results/tables/bn_loss_band_metrics.csv	Quantitative BN loss-band width metrics
results/tables/bn_gradient_statistics.csv	Gradient norm mean/std/max/final statistics
results/tables/bn_activation_statistics.csv	Activation mean/std statistics from forward hooks
results/tables/gradcam_correct_examples.csv	Metadata for correct Grad-CAM examples
results/tables/gradcam_misclassified_examples.csv	Metadata for misclassified Grad-CAM examples
results/tables/component_ablation.csv	GAP, SE, CBAM, GroupNorm, stochastic depth component ablation
results/tables/capacity_ablation_extended.csv	Tiny/small/medium/deep/wide capacity ablation
results/tables/loss_regularization_extended.csv	CE, weight decay, label smoothing, Focal, MixUp, CutMix, RandomErasing/Cutout
results/tables/activation_ablation_extended.csv	ReLU, LeakyReLU, GELU, SiLU, Mish on residual model
results/tables/optimizer_scheduler_ablation.csv	Optimizer and scheduler grid
results/tables/per_class_accuracy.csv	Best model per-class accuracy

Auto-generated report notes:

• results/REPORT_MATERIALS.md

Key Findings for the Final Report

Use these as report bullet points, then support each claim with the corresponding tables and figures.

1. Residual connections are the largest architecture gain. The simple CNN baseline reaches 75.66%, while the custom residual CNN reaches 93.78%.
2. Regularized strong training matters. Strong augmentation, AdamW, SiLU, label smoothing, and cosine scheduling improve the residual model from 93.78% to 94.93%.
3. Final performance is not just parameter count. The best 2.8M-parameter SE model reaches 96.05%, beating the 9.8M deep-wide final candidate at 94.81%.
4. CutMix is the strongest final regularizer. It is not the best in short screening, but focused and final training show strong generalization.
5. Schedulers are crucial. OneCycle, cosine, and warmup cosine are much better than constant LR for most optimizers.
6. Activation choice helps, but is secondary. LeakyReLU/SiLU improve the small CNN; GELU/Mish/SiLU are close on the residual model.
7. Naive GAP can hurt small CNNs. It reduces parameters but also removes too much classifier capacity in the plain CNN setting.
8. BatchNorm improves optimization stability. VGG-A-BN has higher accuracy and a narrower loss variation band than VGG-A without BN.
9. Hard classes are semantically similar animals. Cat and dog have lower per-class accuracy than vehicles/frog, which is visible in the confusion matrix and misclassified examples.

Suggested Report Structure

1. Introduction and task summary.
2. Dataset and preprocessing.
3. Model architectures:
   • CNN-Small
   • CNN-BN
   • CNN-BN-Dropout
   • Custom Residual CNN
   • Final SE Residual CNN
   • VGG-A / VGG-A-BN
4. Experimental protocol:
   • optimizer, scheduler, batch size, epochs, augmentations
   • checkpoint and metric saving
5. Task 1 results:
   • main comparison
   • filter/capacity ablation
   • activation ablation
   • loss/regularization ablation
   • optimizer/scheduler ablation
   • visualization and interpretation
6. Task 2 BatchNorm analysis:
   • VGG-A vs VGG-A-BN accuracy/loss
   • loss variation band
   • explanation of smoother optimization landscape
7. Conclusion:
   • best test error
   • most useful components
   • methods that did not help much
   • remaining limitations

Notes on Checkpoints and Uploading

Checkpoints are generated under:

checkpoints/<experiment>/best.pt
checkpoints/<experiment>/final.pt

For the final report, upload at least the best checkpoint:

checkpoints/final_se_silu_adamw_warmup_cutmix/best.pt

Do not rely on Github for large weight files. Upload model weights to Google Drive, OneDrive, Baidu Netdisk, or another storage service, then put the link in the PDF report.