Simple MNIST recognition in Julia using Flux
Published: 2022-04-16 20:29:14

#Brief introduction
##Neural networks
Resources:
Paper:
* IBM Cloud Education - [What are Neural Networks?](https://www.ibm.com/cloud/learn/neural-networks)

Video:
* Harvard CS50 AI neural networks episode:
https://cs50.harvard.edu/ai/2020/weeks/5/

##Julia
[Julia](https://julialang.org/) is open source compiled, high level programming language. (more about Julia)
It can be used for example in IDE's like [VSCode](https://code.visualstudio.com/) with extension https://github.com/julia-vscode/julia-vscode or in [Juno](https://junolab.org/) (which is build on [Atom](https://atom.io/)).
##Flux
Flux is deep machine learning library written in Julia. More details can be readd here https://arxiv.org/pdf/1811.01457.pdf, at GitHub repository https://github.com/FluxML/Flux.jl or at home page https://fluxml.ai/.
##MNIST
[MNIST](http://yann.lecun.com/exdb/mnist/) is dataset which contains handwritten number images
#Program
##Import libraries
To donwload and install libraries in Julia you can use Pkg.add function
```julia
#Example
using Pkg
Pkg.add("Flux")
```
or use REPL and type `] add Flux`.
###Using libraries in project file
```julia
using Flux, MLDatasets, CUDA
using Flux: train!, onehotbatch
```
##Get data
[MLDatasets](https://github.com/JuliaML/MLDatasets.jl) library contains several vison datasets. Including [MNIST](https://en.wikipedia.org/wiki/MNIST_database).
`MLDatasets.MNIST.traindata()` returns array containing 60000 28x28 greyscale images and labels vector (numbers from 0 to 9).`MLDatasets.MNIST.testdata()` returns 10000 images and labels.
```julia
# Load training data (images, labels)
x_train, y_train = MLDatasets.MNIST.traindata()
# Load test data (images, labels)
x_test, y_test = MLDatasets.MNIST.testdata()
# Convert grayscale to float
x_train = Float32.(x_train)
# Create labels batch
y_train = Flux.onehotbatch(y_train, 0:9)
```
##Create model
###Model contains 3 layers, where:
* 1: layer containing 784 inputs 256 outputs
* 2: layer containing 256 inputs and 10 outputs
```julia
model = Chain(
    Dense(784, 256, relu),
    Dense(256, 10, relu), softmax
)
```
##Training
Loss function [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_loss_function_and_logistic_regression). In project were used `logitcrossentropy`, [as says Flux documentation](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitcrossentropy)
> "This is mathematically equivalent to crossentropy(softmax(ŷ), y), but is more numerically stable than using functions crossentropy and softmax separately."
> [Flux documentation](https://fluxml.ai/Flux.jl/stable/models/losses/#Flux.Losses.logitcrossentropy)

Docs: https://fluxml.ai/Flux.jl/stable/models/losses/
###Define loss function
```julia
loss(x, y) = Flux.Losses.logitcrossentropy(model(x), y)
```
Optimizer [Adam](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam) adaptative moment estimation, that combines loss function and parameters
Docs: https://fluxml.ai/Flux.jl/stable/training/optimisers
###Define optimizer
```julia
optimizer = ADAM(0.0001)
```
###Training
```julia
parameters = params(model)
# flatten() function converts array 28x28x60000 into 784x60000 (28*28x60000)
train_data = [(Flux.flatten(x_train), Flux.flatten(y_train))]
# Range in loop can be used smaller
for i in 1:400
    Flux.train!(loss, parameters, train_data, optimizer)
end
```
###Checking results
```julia
test_data = [(Flux.flatten(x_test), y_test)]
accuracy = 0
for i in 1:length(y_test)
    if findmax(model(test_data[1][1][:, i]))[2] - 1  == y_test[i]
        accuracy = accuracy + 1
    end
end
println(accuracy / length(y_test))
```
### Result of test 
`Accuracy: 0.91`
##Full code can be found here [GIST](https://gist.github.com/piotrek124-1/063738cff976cd6efb76648a16a97e0f)