#!/usr/bin/env python
# coding: utf-8

# # Training Deep Neural Networks on a GPU with PyTorch
#
# ### Part 4 of "Deep Learning with Pytorch: Zero to GANs"
#
# This tutorial series is a hands-on beginner-friendly introduction to deep learning using [PyTorch](https://pytorch.org), an open-source neural networks library. These tutorials take a practical and coding-focused approach. The best way to learn the material is to execute the code and experiment with it yourself. Check out the full series here:
#
# 1. [PyTorch Basics: Tensors & Gradients](https://jovian.ai/aakashns/01-pytorch-basics)
# 2. [Gradient Descent & Linear Regression](https://jovian.ai/aakashns/02-linear-regression)
# 3. [Working with Images & Logistic Regression](https://jovian.ai/aakashns/03-logistic-regression)
# 4. [Training Deep Neural Networks on a GPU](https://jovian.ai/aakashns/04-feedforward-nn)
# 5. [Image Classification using Convolutional Neural Networks](https://jovian.ai/aakashns/05-cifar10-cnn)
# 6. [Data Augmentation, Regularization and ResNets](https://jovian.ai/aakashns/05b-cifar10-resnet)
# 7. [Generating Images using Generative Adversarial Networks](https://jovian.ai/aakashns/06b-anime-dcgan/)
#

#  This tutorial covers the following topics:
#
#  * Creating a deep neural network with hidden layers
#  * Using a non-linear activation function
#  * Using a GPU (when available) to speed up training
#  * Experimenting with hyperparameters to improve the model

# ### How to run the code
#
# This tutorial is an executable [Jupyter notebook](https://jupyter.org) hosted on [Jovian](https://www.jovian.ai). You can _run_ this tutorial and experiment with the code examples in a couple of ways: *using free online resources* (recommended) or *on your computer*.
#
# #### Option 1: Running using free online resources (1-click, recommended)
#
# The easiest way to start executing the code is to click the **Run** button at the top of this page and select **Run on Colab**. [Google Colab](https://colab.research.google.com) is a free online platform for running Jupyter notebooks using Google's cloud infrastructure. You can also select "Run on Binder" or "Run on Kaggle" if you face issues running the notebook on Google Colab.
#
#
# #### Option 2: Running on your computer locally
#
# To run the code on your computer locally, you'll need to set up [Python](https://www.python.org), download the notebook and install the required libraries. We recommend using the [Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/) distribution of Python. Click the **Run** button at the top of this page, select the **Run Locally** option, and follow the instructions.
#
# >  **Jupyter Notebooks**: This tutorial is a [Jupyter notebook](https://jupyter.org) - a document made of _cells_. Each cell can contain code written in Python or explanations in plain English. You can execute code cells and view the results, e.g., numbers, messages, graphs, tables, files, etc., instantly within the notebook. Jupyter is a powerful platform for experimentation and analysis. Don't be afraid to mess around with the code & break things - you'll learn a lot by encountering and fixing errors. You can use the "Kernel > Restart & Clear Output" or "Edit > Clear Outputs" menu option to clear all outputs and start again from the top.

# ### Using a GPU for faster training
#
# You can use a [Graphics Processing Unit](https://en.wikipedia.org/wiki/Graphics_processing_unit) (GPU) to train your models faster if your execution platform is connected to a GPU manufactured by NVIDIA. Follow these instructions to use a GPU on the platform of your choice:
#
# * _Google Colab_: Use the menu option "Runtime > Change Runtime Type" and select "GPU" from the "Hardware Accelerator" dropdown.
# * _Kaggle_: In the "Settings" section of the sidebar, select "GPU" from the "Accelerator" dropdown. Use the button on the top-right to open the sidebar.
# * _Binder_: Notebooks running on Binder cannot use a GPU, as the machines powering Binder aren't connected to any GPUs.
# * _Linux_: If your laptop/desktop has an NVIDIA GPU (graphics card), make sure you have installed the [NVIDIA CUDA drivers](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html).
# * _Windows_: If your laptop/desktop has an NVIDIA GPU (graphics card), make sure you have installed the [NVIDIA CUDA drivers](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html).
# * _macOS_: macOS is not compatible with NVIDIA GPUs
#
#
# If you do not have access to a GPU or aren't sure what it is, don't worry, you can execute all the code in this tutorial just fine without a GPU.

# ## Preparing the Data
#
# In [the previous tutorial](https://jovian.ai/aakashns/03-logistic-regression), we trained a logistic regression model to identify handwritten digits from the MNIST dataset with an accuracy of around 86%. The dataset consists of 28px by 28px grayscale images of handwritten digits (0 to 9) and labels for each image indicating which digit it represents. Here are some sample images from the dataset:
#
# ![mnist-sample](https://i.imgur.com/CAYnuo1.jpg)
#
# We noticed that it's quite challenging to improve the accuracy of a logistic regression model beyond 87%, since the model assumes a linear relationship between pixel intensities and image labels. In this post, we'll try to improve upon it  using a *feed-forward neural network* which can capture non-linear relationships between inputs and targets.
#
# Let's begin by installing and importing the required modules and classes from `torch`, `torchvision`, `numpy`, and `matplotlib`.

# In[1]:

# Uncomment and run the appropriate command for your operating system, if required

# Linux / Binder
# !pip install numpy matplotlib torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

# MacOS
# !pip install numpy matplotlib torch torchvision torchaudio

# In[2]:

import torch
import torchvision
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import torch.nn as nn
import torch.nn.functional as F
from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor
from torchvision.utils import make_grid
from torch.utils.data.dataloader import DataLoader
from torch.utils.data import random_split

# Use a white background for matplotlib figures
matplotlib.rcParams['figure.facecolor'] = '#ffffff'

# We can download the data and create a PyTorch dataset using the `MNIST` class from `torchvision.datasets`.

# In[3]:

dataset = MNIST(root='data/', download=True, transform=ToTensor())

# Let's look at a couple of images from the dataset. The images are converted to PyTorch tensors with the shape `1x28x28` (the dimensions represent color channels, width and height). We can use `plt.imshow` to display the images. However, `plt.imshow` expects channels to be last dimension in an image tensor, so we use the `permute` method to reorder the dimensions of the image.

# In[4]:

image, label = dataset[0]
print('image.shape:', image.shape)
image_perm = image.permute(1, 2, 0)
print(image_perm.shape)
plt.imshow(image[0], cmap='gray')
print('Label:', label)

# In[5]:

image, label = dataset[0]
print('image.shape:', image.shape)
plt.imshow(image[0], cmap='gray')
print('Label:', label)

# Next, let's use the `random_split` helper function to set aside 10000 images for our validation set.

# In[6]:

val_size = 10000
train_size = len(dataset) - val_size

train_ds, val_ds = random_split(dataset, [train_size, val_size])
len(train_ds), len(val_ds)

# We can now create PyTorch data loaders for training and validation.

# In[7]:

batch_size=128

# In[8]:

train_loader = DataLoader(train_ds, batch_size, shuffle=True, num_workers=4, pin_memory=True)
val_loader = DataLoader(val_ds, batch_size*2, num_workers=4, pin_memory=True)

print(torch.cuda.device_count())

# Can you figure out the purpose of the arguments `num_workers` and `pin_memory`? Try looking into the documentation: https://pytorch.org/docs/stable/data.html .
#
# Let's visualize a batch of data in a grid using the `make_grid` function from `torchvision`. We'll also use the `.permute` method on the tensor to move the channels to the last dimension, as expected by `matplotlib`.

# In[9]:

for images, _ in train_loader:
    print('images.shape:', images.shape)
    plt.figure(figsize=(16,8))
    plt.axis('off')
    plt.imshow(make_grid(images, nrow=16).permute((1, 2, 0)))
    break

# ## Hidden Layers, Activation Functions and Non-Linearity
#
# We'll create a neural network with two layers: a _hidden layer_ and an _output layer_. Additionally, we'll use an _activation function_ between the two layers. Let's look at a step-by-step example to learn how hidden layers and activation functions can help capture non-linear relationships between inputs and outputs.
#
# First, let's create a batch of inputs tensors. We'll flatten the `1x28x28` images into vectors of size `784`, so they can be passed into an `nn.Linear` object.

# In[11]:

for images, labels in train_loader:
    print('images.shape:', images.shape)
    inputs = images.reshape(-1, 784)
    print('inputs.shape:', inputs.shape)
    break

# Next, let's create a `nn.Linear` object, which will serve as our _hidden_ layer. We'll set the size of the output from the hidden layer to 32. This number can be increased or decreased to change the _learning capacity_ of the model.

# In[59]:

input_size = inputs.shape[-1]
hidden_size = 256

# In[60]:

layer1 = nn.Linear(input_size, hidden_size)

# We can now compute intermediate outputs for the batch of images by passing `inputs` through `layer1`.

# In[61]:

inputs.shape

# In[62]:

layer1_outputs = layer1(inputs)
print('layer1_outputs.shape:', layer1_outputs.shape)

# The image vectors of size `784` are transformed into intermediate output vectors of length `32` by performing a matrix multiplication of `inputs` matrix with the transposed weights matrix of `layer1` and adding the bias. We can verify this using `torch.allclose`. For a more detailed explanation, review the tutorial on [linear regression](https://jovian.ai/aakashns/02-linear-regression).

# In[63]:

layer1_outputs_direct = inputs @ layer1.weight.t() + layer1.bias
layer1_outputs_direct.shape

# In[64]:

torch.allclose(layer1_outputs, layer1_outputs_direct, 1e-3)

# Thus, `layer1_outputs` and `inputs` have a linear relationship, i.e., each element of `layer_outputs` is a weighted sum of elements from `inputs`. Thus, even as we train the model and modify the weights, `layer1` can only capture linear relationships between `inputs` and `outputs`.
#
#
#

# Next, we'll use the Rectified Linear Unit (ReLU) function as the activation function for the outputs. It has the formula `relu(x) = max(0,x)` i.e. it simply replaces negative values in a given tensor with the value 0. ReLU is a non-linear function, as seen here visually:
#
#
#
# We can use the `F.relu` method to apply ReLU to the elements of a tensor.

# In[65]:

F.relu(torch.tensor([[1, -1, 0],
                     [-0.1, .2, 3]]))

# Let's apply the activation function to `layer1_outputs` and verify that negative values were replaced with 0.

# In[66]:

relu_outputs = F.relu(layer1_outputs)
print('min(layer1_outputs):', torch.min(layer1_outputs).item())
print('min(relu_outputs):', torch.min(relu_outputs).item())

# Now that we've applied a non-linear activation function, `relu_outputs` and `inputs` do not have a linear relationship. We refer to `ReLU` as the _activation function_, because for each input certain outputs are activated (those with non-zero values) while others turned off (those with zero values)
#
# Next, let's create an output layer to convert vectors of length `hidden_size` in `relu_outputs` into vectors of length 10, which is the desired output of our model (since there are 10 target labels).

# In[67]:

output_size = 10
layer2 = nn.Linear(hidden_size, output_size)

# In[68]:

layer2_outputs = layer2(relu_outputs)
print(layer2_outputs.shape)

# In[69]:

inputs.shape

# As expected, `layer2_outputs` contains a batch of vectors of size 10. We can now use this output to compute the loss using `F.cross_entropy` and adjust the weights of `layer1` and `layer2` using gradient descent.

# In[70]:

F.cross_entropy(layer2_outputs, labels)

# Thus, our model transforms `inputs` into `layer2_outputs` by applying a linear transformation (using `layer1`), followed by a non-linear activation (using `F.relu`), followed by another linear transformation (using `layer2`). Let's verify this by re-computing the output using basic matrix operations.

# In[71]:

# Expanded version of layer2(F.relu(layer1(inputs)))
outputs = (F.relu(inputs @ layer1.weight.t() + layer1.bias)) @ layer2.weight.t() + layer2.bias

# In[72]:

torch.allclose(outputs, layer2_outputs, 1e-3)

# Note that `outputs` and `inputs` do not have a linear relationship due to the non-linear activation function `F.relu`. As we train the model and adjust the weights of `layer1` and `layer2`, we can now capture non-linear relationships between the images and their labels. In other words, introducing non-linearity makes the model more powerful and versatile. Also, since `hidden_size` does not depend on the dimensions of the inputs or outputs, we vary it to increase the number of parameters within the model. We can also introduce new hidden layers and apply the same non-linear activation after each hidden layer.
#
# The model we just created is called a neural network. A _deep neural network_ is simply a neural network with one or more hidden layers. In fact, the [Universal Approximation Theorem](http://neuralnetworksanddeeplearning.com/chap4.html) states that a sufficiently large & deep neural network can compute any arbitrary function i.e. it can _learn_ rich and complex non-linear relationships between inputs and targets. Here are some examples:
#
# * Identifying if an image contains a cat or a dog (or [something else](https://machinelearningmastery.com/introduction-to-the-imagenet-large-scale-visual-recognition-challenge-ilsvrc/))
# * Identifying the genre of a song using a 10-second sample
# * Classifying movie reviews as positive or negative based on their content
# * Navigating self-driving cars using a video feed of the road
# * Translating sentences from English to French (and hundreds of other languages)
# * Converting a speech recording to text and vice versa
# * And many more...
#
# It's hard to imagine how the simple process of multiplying inputs with randomly initialized matrices, applying non-linear activations, and adjusting weights repeatedly using gradient descent can yield such astounding results. Deep learning models often contain millions of parameters, which can together capture far more complex relationships than the human brain can comprehend.
#
# If we hadn't included a non-linear activation between the two linear layers, the final relationship between inputs and outputs would still be linear. A simple refactoring of the computations illustrates this.

# In[73]:

# Same as layer2(layer1(inputs))
outputs2 = (inputs @ layer1.weight.t() + layer1.bias) @ layer2.weight.t() + layer2.bias

# In[74]:

# Create a single layer to replace the two linear layers
combined_layer = nn.Linear(input_size, output_size)

combined_layer.weight.data = layer2.weight @ layer1.weight
combined_layer.bias.data = layer1.bias @ layer2.weight.t() + layer2.bias

# In[75]:

# Same as combined_layer(inputs)
outputs3 = inputs @ combined_layer.weight.t() + combined_layer.bias

# In[76]:

torch.allclose(outputs2, outputs3, 1e-3)

# ### Save and upload your notebook
#
# Whether you're running this Jupyter notebook online or on your computer, it's essential to save your work from time to time. You can continue working on a saved notebook later or share it with friends and colleagues to let them execute your code. [Jovian](https://jovian.ai/platform-features) offers an easy way of saving and sharing your Jupyter notebooks online.

# In[39]:

# In[40]:

# In[41]:

# `jovian.commit` uploads the notebook to your Jovian account, captures the Python environment, and creates a shareable link for your notebook, as shown above. You can use this link to share your work and let anyone (including you) run your notebooks and reproduce your work.

# ## Model
#
# We are now ready to define our model. As discussed above, we'll create a neural network with one hidden layer. Here's what that means:
#
# * Instead of using a single `nn.Linear` object to transform a batch of inputs (pixel intensities) into outputs (class probabilities), we'll use two `nn.Linear` objects. Each of these is called a _layer_ in the network.
#
# * The first layer (also known as the hidden layer) will transform the input matrix of shape `batch_size x 784` into an intermediate output matrix of shape `batch_size x hidden_size`. The parameter `hidden_size` can be configured manually (e.g., 32 or 64).
#
# * We'll then apply a non-linear *activation function* to the intermediate outputs. The activation function transforms individual elements of the matrix.
#
# * The result of the activation function, which is also of size `batch_size x hidden_size`, is passed into the second layer (also known as the output layer).  The second layer transforms it into a matrix of size `batch_size x 10`. We can use this output to compute the loss and adjust weights using gradient descent.
#
#
# As discussed above, our model will contain one hidden layer. Here's what it looks like visually:
#
#
#
#
# Let's define the model by extending the `nn.Module` class from PyTorch.

# In[275]:

class MnistModel(nn.Module):
    """Feedfoward neural network with 1 hidden layer"""
    def __init__(self, in_size, hidden_size, out_size):
        super().__init__()

        self.hidden_count = 1
        self.hidden_layers = []

        # hidden layer
        self.linear1 = nn.Linear(in_size, hidden_size)
        # output layer

        for x in range(self.hidden_count) :
           self.hidden_layers.append(nn.Linear(hidden_size, hidden_size));

        self.linear4 = nn.Linear(hidden_size, out_size)

    def forward(self, xb):
        # Flatten the image tensors
        xb = xb.view(xb.size(0), -1)
        # Get intermediate outputs using hidden layer
        out = self.linear1(xb)
        # Apply activation function
        out = F.relu(out)

        #for layer in self.hidden_layers:
        #    out = layer(out)
        #    out = F.relu(out)

        # Get predictions using output layer
        out = self.linear4(out)
        return out

    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss, 'val_acc': acc}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))

# We also need to define an `accuracy` function which calculates the accuracy of the model's prediction on an batch of inputs. It's used in `validation_step` above.

# In[276]:

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))

# We'll create a model that contains a hidden layer with 32 activations.

# In[277]:

input_size = 784
hidden_size = 1024 # you can change this
num_classes = 10

# In[278]:

model = MnistModel(input_size, hidden_size=1024, out_size=num_classes)

# Let's take a look at the model's parameters. We expect to see one weight and bias matrix for each of the layers.

# In[279]:

for t in model.parameters():
    print(t.shape)

# Let's try and generate some outputs using our model. We'll take the first batch of 128 images from our dataset and pass them into our model.

# In[280]:

for images, labels in train_loader:
    outputs = model(images)
    loss = F.cross_entropy(outputs, labels)
    print('Loss:', loss.item())
    break

print('outputs.shape : ', outputs.shape)
print('Sample outputs :\n', outputs[:2].data)

# ## Using a GPU
#
# As the sizes of our models and datasets increase, we need to use GPUs to train our models within a reasonable amount of time. GPUs contain hundreds of cores optimized for performing expensive matrix operations on floating-point numbers quickly, making them ideal for training deep neural networks. You can use GPUs for free on [Google Colab](https://colab.research.google.com/) and [Kaggle](https://www.kaggle.com/kernels) or rent GPU-powered machines on services like [Google Cloud Platform](https://cloud.google.com/gpu/), [Amazon Web Services](https://docs.aws.amazon.com/dlami/latest/devguide/gpu.html), and [Paperspace](https://www.paperspace.com/).
#
# We can check if a GPU is available and the required NVIDIA CUDA drivers are installed using `torch.cuda.is_available`.

# In[281]:

torch.cuda.is_available()

# Let's define a helper function to ensure that our code uses the GPU if available and defaults to using the CPU if it isn't.

# In[282]:

def get_default_device():
    """Pick GPU if available, else CPU"""
    if torch.cuda.is_available():
        return torch.device('cuda')
    else:
        return torch.device('cpu')

# In[283]:

device = get_default_device()

# Next, let's define a function that can move data and model to a chosen device.

# In[284]:

def to_device(data, device):
    """Move tensor(s) to chosen device"""
    if isinstance(data, (list,tuple)):
        return [to_device(x, device) for x in data]
    return data.to(device, non_blocking=True)

# In[285]:

for images, labels in train_loader:
    print(images.shape)
    images = to_device(images, device)
    print(images.device)
    break

# Finally, we define a `DeviceDataLoader` class to wrap our existing data loaders and move batches of data to the selected device. Interestingly, we don't need to extend an existing class to create a PyTorch datal oader. All we need is an `__iter__` method to retrieve batches of data and an `__len__` method to get the number of batches.

# In[286]:

class DeviceDataLoader():
    """Wrap a dataloader to move data to a device"""
    def __init__(self, dl, device):
        self.dl = dl
        self.device = device

    def __iter__(self):
        """Yield a batch of data after moving it to device"""
        for b in self.dl:
            yield to_device(b, self.device)

    def __len__(self):
        """Number of batches"""
        return len(self.dl)

# The `yield` keyword in Python is used to create a generator function that can be used within a `for` loop, as illustrated below.

# In[287]:

def some_numbers():
    yield 10
    yield 20
    yield 30

for value in some_numbers():
    print(value)

# We can now wrap our data loaders using `DeviceDataLoader`.

# In[288]:

train_loader = DeviceDataLoader(train_loader, device)
val_loader = DeviceDataLoader(val_loader, device)

# Tensors moved to the GPU have a `device` property which includes that word `cuda`. Let's verify this by looking at a batch of data from `valid_dl`.

# In[289]:

for xb, yb in val_loader:
    print('xb.device:', xb.device)
    print('yb:', yb)
    break

# ## Training the Model
#
# We'll define two functions: `fit` and `evaluate` to train the model using gradient descent and evaluate its performance on the validation set. For a detailed walkthrough of these functions, check out the [previous tutorial](https://jovian.ai/aakashns/03-logistic-regression).

# In[290]:

def evaluate(model, val_loader):
    """Evaluate the model's performance on the validation set"""
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    """Train the model using gradient descent"""
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result)
        history.append(result)
    return history

# Before we train the model, we need to ensure that the data and the model's parameters (weights and biases) are on the same device (CPU or GPU). We can reuse the `to_device` function to move the model's parameters to the right device.

# In[291]:

# Model (on GPU)
model = MnistModel(input_size, hidden_size=hidden_size, out_size=num_classes)
to_device(model, device)

# Let's see how the model performs on the validation set with the initial set of weights and biases.

# In[292]:

history = [evaluate(model, val_loader)]

# The initial accuracy is around 10%, as one might expect from a randomly initialized model (since it has a 1 in 10 chance of getting a label right by guessing randomly).
#
# Let's train the model for five epochs and look at the results. We can use a relatively high learning rate of 0.5.

# In[293]:

history += fit(5, 0.5, model, train_loader, val_loader)

# 96% is pretty good! Let's train the model for five more epochs at a lower learning rate of 0.1 to improve the accuracy further.

# In[179]:

history += fit(10, 0.001, model, train_loader, val_loader)

# We can now plot the losses & accuracies to study how the model improves over time.

# In[180]:

losses = [x['val_loss'] for x in history]
plt.plot(losses, '-x')
plt.xlabel('epoch')
plt.ylabel('loss')
plt.title('Loss vs. No. of epochs');

# In[181]:

accuracies = [x['val_acc'] for x in history]
plt.plot(accuracies, '-x')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.title('Accuracy vs. No. of epochs');

# Our current model outperforms the logistic regression model (which could only achieve around 86% accuracy) by a considerable margin! It quickly reaches an accuracy of 97% but doesn't improve much beyond this. To improve accuracy further, we need to make the model more powerful by increasing the hidden layer's size or adding more hidden layers with activations. I encourage you to try out both these approaches and see which one works better.

# As a final step, we can save and commit our work using the `jovian` library.

# ## Testing with individual images
#
# While we have been tracking the overall accuracy of a model so far, it's also a good idea to look at model's results on some sample images. Let's test out our model with some images from the predefined test dataset of 10000 images. We begin by recreating the test dataset with the `ToTensor` transform.

# In[69]:

# Define test dataset
test_dataset = MNIST(root='data/',
                     train=False,
                     transform=ToTensor())

# Let's define a helper function `predict_image`, which returns the predicted label for a single image tensor.

# In[70]:

def predict_image(img, model):
    xb = to_device(img.unsqueeze(0), device)
    yb = model(xb)
    _, preds  = torch.max(yb, dim=1)
    return preds[0].item()

# Let's try it out with a few images.

# In[71]:

img, label = test_dataset[0]
plt.imshow(img[0], cmap='gray')
print('Label:', label, ', Predicted:', predict_image(img, model))

# In[72]:

img, label = test_dataset[1839]
plt.imshow(img[0], cmap='gray')
print('Label:', label, ', Predicted:', predict_image(img, model))

# In[73]:

img, label = test_dataset[193]
plt.imshow(img[0], cmap='gray')
print('Label:', label, ', Predicted:', predict_image(img, model))

# Identifying where our model performs poorly can help us improve the model, by collecting more training data, increasing/decreasing the complexity of the model, and changing the hypeparameters.
#
# As a final step, let's also look at the overall loss and accuracy of the model on the test set.

# In[74]:

test_loader = DeviceDataLoader(DataLoader(test_dataset, batch_size=256), device)
result = evaluate(model, test_loader)
result

# We expect this to be similar to the accuracy/loss on the validation set. If not, we might need a better validation set that has similar data and distribution as the test set (which often comes from real world data).

# Let's save the model's weights and attach it to the notebook using `jovian.commit`. We will also record the model's performance on the test dataset using `jovian.log_metrics`.

# In[75]:

# In[76]:

torch.save(model.state_dict(), 'mnist-feedforward.pth')

# In[ ]:

# ## Exercises
#
# Try out the following exercises to apply the concepts and techniques you have learned so far:
#
# * Coding exercises on end-to-end model training: https://jovian.ai/aakashns/03-cifar10-feedforward
# * Starter notebook for deep learning models:  https://jovian.ai/aakashns/fashion-feedforward-minimal
#
# Training great machine learning models reliably takes practice and experience. Try experimenting with different datasets, models and hyperparameters, it's the best way to acquire this skill.

# ## Summary and Further Reading
#
# Here is a summary of the topics covered in this tutorial:
#
# * We created a neural network with one hidden layer to improve upon the logistic regression model from the previous tutorial. We also used the ReLU activation function to introduce non-linearity into the model, allowing it to learn more complex relationships between the inputs (pixel densities) and outputs (class probabilities).
#
# * We defined some utilities like `get_default_device`, `to_device` and `DeviceDataLoader` to leverage a GPU if available, by moving the input data and model parameters to the appropriate device.
#
# * We were able to use the exact same training loop: the `fit` function we had define earlier to train out model and evaluate it using the validation dataset.
#
# There's a lot of scope to experiment here, and I encourage you to use the interactive nature of Jupyter to play around with the various parameters. Here are a few ideas:
#
# * Try changing the size of the hidden layer, or add more hidden layers and see if you can achieve a higher accuracy.
#
# * Try changing the batch size and learning rate to see if you can achieve the same accuracy in fewer epochs.
#
# * Compare the training times on a CPU vs. GPU. Do you see a significant difference. How does it vary with the size of the dataset and the size of the model (no. of weights and parameters)?
#
# * Try building a model for a different dataset, such as the [CIFAR10 or CIFAR100 datasets](https://www.cs.toronto.edu/~kriz/cifar.html).
#
# Here are some references for further reading:
#
# * [A visual proof that neural networks can compute any function](http://neuralnetworksanddeeplearning.com/chap4.html), also known as the Universal Approximation Theorem.
#
# * [But what *is* a neural network?](https://www.youtube.com/watch?v=aircAruvnKk) - A visual and intuitive introduction to what neural networks are and what the intermediate layers represent
#
# * [Stanford CS229 Lecture notes on Backpropagation](http://cs229.stanford.edu/notes/cs229-notes-backprop.pdf) - for a more mathematical treatment of how gradients are calculated and weights are updated for neural networks with multiple layers.
#
#
# You are now ready to move on to the next tutorial: [Image Classification using Convolutional Neural Networks](https://jovian.ai/aakashns/05-cifar10-cnn).


#!/usr/bin/env python
# coding: utf-8

# # Team13: Capstone project of Python Bootcamp
#
# This is [the Capstone project for Team 13 of the Python Data Analysis Bootcamp](https://github.com/pyladiesams/Bootcamp-Data-Analysis-beginner-apr-may2020/blob/master/Capstone/README.md).
# We are trying, more or less, to follow the structure of [jupytemplate](https://github.com/xtreamsrl/jupytemplate/blob/master/jupytemplate/jupytemplate/template.ipynb).
#
# ## Purpose
#
# State the purpose of the notebook.
#
# ## Methology
#
# Quickly describe assumptions and processing steps.
#
# ## TODO / Improvements
#
# * [x] Find a dataset that has at least 2 CSV files
# * [ ] Come up with 5 questions that you want to answer while exploring the dataset
# * [ ] Perform EDA (Exploratoty Data Analysis) on your dataset with basic visualisations
#
# ## Results

# ## Setup

# In[1]:

# install system dependencies
import sys
import os

get_ipython().system('conda install -c conda-forge --yes --prefix {sys.prefix} pandas jupyterthemes seaborn jupyter_contrib_nbextensions pandoc')

# ### Library Import

# In[2]:

# load libraries and setup environment
# mandatory
import pandas as pd

get_ipython().run_line_magic('matplotlib', 'inline')
import matplotlib.pyplot as plt

# optional
import numpy as np
import seaborn as sns
from jupyterthemes import jtplot
from IPython.core.display import HTML
jtplot.style(theme='monokai', context='notebook', ticks=True, grid=False)

# ## Parameter definition
#
# We set all relevant parameters for our notebook. By convention, parameters are uppercase, while all the other variables follow Python's guidelines.

# In[3]:

COAST_COLUMN = "Coastline (coast/area ratio)"
COUNTRY_COLUMN = 'Country'
SATISFACTION_COLUMN = "People with highest life satisfaction [%]"

# ## Data import
# We retrieve all the required data for the analysis.

# In[4]:

cost_of_living = pd.read_csv('../data/andytran11996_cost-of-living/datasets_73059_162758_cost-of-living-2018.csv')

# we are droppping the Rank column because it's entirely empty
cost_of_living = cost_of_living.drop(columns = 'Rank')

life_satisfaction = pd.read_csv('../data/roshansharma_europe-datasets/datasets_231225_493692_life_satisfaction_2013.csv')
life_satisfaction = life_satisfaction.rename(columns = { "prct_life_satis_high": "People with highest life satisfaction [%]", "country": "Country" })
life_satisfaction['Country'] = life_satisfaction['Country'].astype(str)
life_satisfaction['Country'] = life_satisfaction['Country'].str.strip()

generic_country_data = pd.read_csv('../data/fernandol_countries-of-the-world/datasets_23752_30346_countries of the world.csv', decimal=',')
generic_country_data['Country'] = generic_country_data['Country'].astype(str)
generic_country_data['Country'] = generic_country_data['Country'].str.strip()
generic_european_country_data = generic_country_data[generic_country_data['Region'].str.contains('EUROPE', case = False)]

print('successfully imported the datasets.')

# ## Data processing
#
# ### 1. What are the five cities with the highest/lowest cost of living (incl. rent)?

# In[5]:

caption_column = 'City'
index_column = 'Cost of Living Plus Rent Index'

def display_cost_of_living(costs, title):
    filtered_costs = costs[[caption_column, index_column]].sort_values(index_column, ascending = True)
    filtered_costs.plot.barh(title = title, x = caption_column, y = index_column)
    plt.show();
    display(filtered_costs.sort_values(index_column, ascending = False).style.hide_index())

# print the ten most expensive cities in the database in 2018
display_cost_of_living(cost_of_living.nlargest(5, index_column), 'Largest Rent Index')
display_cost_of_living(cost_of_living.nsmallest(5, index_column), 'Smallest Rent Index')

# ## 2. What are the five happiest countries in Europe?

# In[6]:

index_column = "People with highest life satisfaction [%]"
caption_column = 'Country'

top_countries_life_satisfaction = life_satisfaction[[caption_column, index_column]]
top_countries_life_satisfaction = top_countries_life_satisfaction.nlargest(5, index_column)
top_countries_life_satisfaction = top_countries_life_satisfaction.sort_values(index_column, ascending = True)
top_countries_life_satisfaction.plot.barh(title = 'Percentage of satisfied people', x = caption_column, y = index_column);
plt.show();
display(top_countries_life_satisfaction.sort_values(index_column, ascending = False).style.hide_index())

# ## 3. What are the European countries with the most coast line in relation to their area?

# In[7]:

index_column = "Coastline (coast/area ratio)"
caption_column = 'Country'

coastline_data = generic_european_country_data[[caption_column, index_column]]

coastline_data = coastline_data.nlargest(5, index_column)
coastline_data = coastline_data.sort_values(index_column, ascending = True)
coastline_data.plot.barh(title = 'Countries with the most coast line in relation to their area', x = caption_column, y = index_column);
plt.show();
display(coastline_data.sort_values(index_column, ascending = False).style.hide_index())

# ## 4. Is there a correlation between happiness and access to a coastline?

# In[9]:

merged = pd.merge(generic_european_country_data, life_satisfaction, on = COUNTRY_COLUMN, how = 'inner')

coastline_data = generic_european_country_data[[COAST_COLUMN, COUNTRY_COLUMN]]

# sort by coast
merged = merged.sort_values(COAST_COLUMN, ascending = True, ignore_index = True)

ax = plt.gca()
merged.plot(kind = 'line', y = COAST_COLUMN, x = COUNTRY_COLUMN ,ax=ax)
merged.plot(kind = 'line', y = SATISFACTION_COLUMN, x = COUNTRY_COLUMN ,ax=ax)

plt.grid(b = True, color = 'aqua', alpha = 0.1, linestyle = 'dashdot')
plt.show();

# ## References
#
# * [data for the cost of living](https://www.kaggle.com/andytran11996/cost-of-living/)
# * [base data for countries of the world](https://www.kaggle.com/fernandol/countries-of-the-world)
# * [data for life expectancy from the WHO](https://www.kaggle.com/kumarajarshi/life-expectancy-who)
# * [roshansharma_europe-datasets](https://www.kaggle.com/roshansharma/europe-datasets)


![Callysto.ca Banner](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-top.jpg?raw=true)

To run this notebook press the >> Button and confirm "Restart and Run all".

![](./images/RunB.png)

from IPython.display import HTML

HTML('''
The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click here.''')

%%html

# Modules

import string
import numpy as np
import pandas as pd
import qgrid as q
import matplotlib.pyplot as plt

# Widgets & Display modules, etc..

from ipywidgets import widgets as w
from ipywidgets import Button, Layout
from IPython.display import display, Javascript, Markdown

# grid features for interactive grids

grid_features = { 'fullWidthRows': True,
                  'syncColumnCellResize': True,
                  'forceFitColumns': True,
                  'rowHeight': 40,
                  'enableColumnReorder': True,
                  'enableTextSelectionOnCells': True,
                  'editable': True,
                  'filterable': False,
                  'sortable': False,
                  'highlightSelectedRow': True}

from ipywidgets import Button , Layout , interact,widgets
from IPython.display import Javascript, display

# Function: executes previous cell on button widget click event and hides achievement indicators message

def run_current(ev):

    display(Javascript('IPython.notebook.execute_cell_range(IPython.notebook.get_selected_index()+0,IPython.notebook.get_selected_index()+1)'))

# Counter for toggling achievement indicator on/off

button_ctr = 0

# Achievement Indicators

line_1 = "#### Achievement Indicators"
line_2 = "**General Outcome:**"
line_3 = "* The general outcome of this notebook is to use experimental or theoretical probabilities to represent and solve problems involving uncertainty."
line_4 = "**Specific Outcome 4:**"
line_5 = "* Express probabilities as ratios, fractions and percents."
line_6 = "**Specific Outcome 5:**"
line_7 = "* Identify the sample space (where the combined sample space has 36 or fewer elements) for a probability experiment involving two independent events.*"
line_8 = "**Specific Outcome 6:**"
line_9 = "* Conduct a probability experiment to compare the theoretical probability (determined using a tree diagram, table or other graphic organizer) and experimental probability of two independent events*"

# Use to print lines, then save in lines_list

def print_lines(n):

    lines_str = ""

    for i in range(1,n+1):
        lines_str = lines_str + "line_"+str(i)+","

    lines_str = lines_str[:-1]

    print(lines_str)

lines_list = [line_1,line_2,line_3,line_4,line_5,line_6,line_7,line_8,line_9]

# Show/Hide buttons

ai_button_show = widgets.Button(button_style='info',description="Show Achievement Indicators", layout=Layout(width='25%', height='30px') )
ai_button_hide = widgets.Button(button_style='info',description="Hide Achievement Indicators", layout=Layout(width='25%', height='30px') )

display(Markdown("For instructors:"))

button_ctr += 1

if(button_ctr % 2 == 0):

    for line in lines_list:
        display(Markdown(line))

    display(ai_button_hide)
    ai_button_hide.on_click( run_current )

else:

    display(ai_button_show)
    ai_button_show.on_click( run_current )

# Statistics and Probability

## Chance and Uncertainty

#### Grade 11  Math

Overview

In this notebook we will explore basic notions and properties about expressing and manipulating probabilities.
The general outcome of this notebook is to use experimental or theoretical probabilities to represent and solve problems involving uncertainty.

Probabilities as ratios, fractions and percents

A natural question to ask is: how do we measure the probability associated to an event of a given probability experiment, i.e. for a given event, what is the probability it occurs?  We define this now, illustrate with a simple example involving dice, and we will then provide an interactive exercise.

Definition. The probability of an event is the ratio between the size of the event (as a collection of outcomes) and the size of the sample space.

The sample space of rolling a single dice is given by $\lbrace$1,2,3,4,5,6 $\rbrace$ and had sample size 6.  If we assume each face is equally likely to occur (i.e. the dice is unbiased), then the probability of getting each face is the ratio or fraction $\dfrac{1}{6}$.

We will denote the probability of getting each number as $P(i)$, where $i$ can either be 1,2,3,4,5,6.
Then, the probability of getting event 1, denoted $P(1)$ is equal to $\dfrac{1}{6}$; more precisely: $P(1)=P(2)=P(3)=P(4)=P(5)=P(6)=\dfrac{1}{6}$.

This is equivalent to stating that the probability of getting any given face is 1 in 6, or $1:6$. Using ratios we have $P(i) = 1:6$, where $i = 1,2,3,4,5,6$. We can also express probabilities using percents. The total number of outcomes is considered 100%. Since there are 6 possible outcomes and assuming equal probability, $P(i) = 100 / 6  = 16.67 \%$.

We summarize in the table below.

|Event  $i$                  |1 |2 |3 |4 |5 |6 |
|----------------------------|--|--|--|--|--|--|
|Probability $P(i)$ as a fraction|$\dfrac{1}{6}$|$\dfrac{1}{6}$|$\dfrac{1}{6}$|$\dfrac{1}{6}$|$\dfrac{1}{6}$|$\dfrac{1}{6}$|
|Probability $P(i)$ as a ratio            |1:6|1:6|1:6|1:6|1:6|1:6|
|Probability $P(i)$ as a percent|16.67%|16.67%|16.67%|16.67%|16.67%|16.67%|

import numpy as np
import matplotlib
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import random
import matplotlib.gridspec as gridspec
import ipywidgets
from ipywidgets import interact,interact_manual,widgets
%matplotlib inline

### Interactive Example: Probabilities as Fractions, Ratios and Percents

The widget below illustrates a probability experiment using roulettes of various sizes. The basic experiment consists in spinning  a roulette, divided in a given number of compartments, each of the same size and associated with a unique number that identifies it. The outcome of the experiment is the compartment of the roulette whose number shows in red. Basically, this is an experiment similar to rolling a dice, but where we control the number of faces (number of compartments in the roulette). We assume the roulette is unbiased: each compartment has the same chance to appear in red whenspinning the roulette.

On the upper side of the widget you find a drop down menu indicating the size of the sample space, which is the number of compartments of the roulette. In this case, we are considering roulettes whose outcomes are integers from 1 to the size of the sample space. We consider roulettes with sample spaces of size 2, 4, 6 and 8.

Below the drop down menu you will find find a red button. Click it to play.

On the left hand side of the widget you will see a roulette with numbers in black and one number in red. The number in red corresponds to the outcome of the experiment.

On the right hand side you will find a printed message explaining what the probability of the event associated to the obtained outcome is.

Play multiple times to simulate what would happen if you spun the roulette. Change the size of the sample space to learn what the different probabilities associated to each event in the roulette are.

###
def roulette(number_parititions,value):
    if value==True or value==False:
        lucky_number_one = random.choice(np.arange(1,2*number_parititions+1))

        axalpha = 0.05
        figcolor = 'white'
        dpi = 80
        fig = plt.figure(figsize=(15,10), dpi=dpi,facecolor='black')
        plt.subplot(211)

        plt.subplot(212)

        fig.patch.set_edgecolor(figcolor)
        fig.patch.set_facecolor(figcolor)
        ax = plt.subplot(121,projection='polar',facecolor="red")

        ax.patch.set_alpha(axalpha)
        ax.set_axisbelow(True)

        ax1 = plt.subplot(122)
        ax1.grid(False)
        ax1.set_xlim(0.1,0.9)
        ax1.set_ylim(0.4,0.8)

        ax1.set_xticklabels([])
        ax1.set_yticklabels([])
        ax1.axis("off")

        arc = 2. * np.pi
        N = number_parititions
        theta = np.arange(0.0, arc, arc/N)

        if number_parititions == 4:
            radii_ar = [1.0 for i in range(number_parititions)]
            width_ar = [1.0 for i in range(number_parititions)]

            radii = 10 * np.array(radii_ar)
            width = np.pi/4  * np.array(width_ar)

            bars = ax.bar(theta, radii, width=width, bottom=0.0)
            for r, bar in zip(radii, bars):
                bar.set_facecolor("pink")
                bar.set_alpha(0.6)

            ax.text(0, 7, "1", fontsize=20,transform=ax.transData._b,)
            ax.text(5, 5, "2", fontsize=20,transform=ax.transData._b)
            ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b)
            ax.text(4, -5.5, "4", fontsize=20,transform=ax.transData._b)
            ax.text(0, -7, "5", fontsize=20,transform=ax.transData._b)
            ax.text(-5, -5, "6", fontsize=20,transform=ax.transData._b)
            ax.text(-5, 5, "8", fontsize=20,transform=ax.transData._b)
            ax.text(-7, 0, "7", fontsize=20,transform=ax.transData._b)

            if lucky_number_one==1:
                ax.text(0, 7, "1", fontsize=20,transform=ax.transData._b,color="red")
            elif lucky_number_one==2:
                ax.text(5, 5, "2", fontsize=20,transform=ax.transData._b,color="red")
            elif lucky_number_one==3:
                ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==4:
                ax.text(4, -5.5, "4", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==5:
                ax.text(0, -7, "5", fontsize=20,transform=ax.transData._b,color="red")
            elif lucky_number_one==6:
                ax.text(-5, -5, "6", fontsize=20,transform=ax.transData._b,color="red")
            elif lucky_number_one==7:
                ax.text(-7, 0, "7", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==8:
                ax.text(-5, 5, "8", fontsize=20,transform=ax.transData._b,color='red')
            ax1.text(0.1,0.77,"Probability of each event",fontsize=25)
            ax1.text(0.1,0.5,"P(1) = 1/8 = 1:8 = 12.5%\n\nP(2) = 1/8 = 1:8 = 12.5%\n\nP(3) = 1/8 = 1:8 = 12.5%\n\nP(4) = 1/8 = 1:8 = 12.5%\n\nP(5) = 1/8 = 1:8 = 12.5%\n\nP(6) = 1/8 = 1:8 = 12.5%\n\nP(7) = 1/8 = 1:8 = 12.5%\n\nP(8) = 1/8 = 1:8 = 12.5%"\
                      ,fontsize=20)
            ax1.text(0.1,0.45,"Probability Outcome",fontsize=25)
            ax1.text(0.1,0.42,"The result after spinning is " + str(lucky_number_one),fontsize=20)

        elif number_parititions == 3:

            radii_ar = [1.0 for i in range(number_parititions)]
            width_ar = [1.3 for i in range(number_parititions)]

            radii = 10 * np.array(radii_ar)
            width = np.pi/4  * np.array(width_ar)

            bars = ax.bar(theta, radii, width=width, bottom=0.0)
            for r, bar in zip(radii, bars):
                bar.set_facecolor("pink")
                bar.set_alpha(0.6)
            ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b)
            ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b)
            ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b)
            ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b)
            ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b)
            ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b)

            if lucky_number_one==1:
                ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==2:
                ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==3:
                ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==4:
                ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==5:
                ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==6:
                ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b,color='red')

            ax1.text(0.1,0.74,"Probability of each event",fontsize=25)
            ax1.text(0.1,0.55,"P(1) = 1/6 = 1:6 = 16.67%\n\nP(2) = 1/6 = 1:6 = 16.67%\n\nP(3) = 1/6 = 1:6 = 16.67\n\nP(4) = 1/6 = 1:6 = 16.67%\n\nP(5) = 1/6 = 1:6 = 16.67%\n\nP(6) = 1/6 = 1:6 = 16.67%"\
                      ,fontsize=20)
            ax1.text(0.1,0.5,"Probability Outcome",fontsize=25)
            ax1.text(0.1,0.48,"The result after spinning is " + str(lucky_number_one),fontsize=20)

        elif number_parititions == 2:
            radii_ar = [1.0 for i in range(number_parititions)]
            width_ar = [1.7 for i in range(number_parititions)]

            radii = 10 * np.array(radii_ar)
            width = np.pi/4  * np.array(width_ar)

            bars = ax.bar(theta, radii, width=width, bottom=0.0)
            for r, bar in zip(radii, bars):
                bar.set_facecolor("pink")
                bar.set_alpha(0.6)
            ax.text(0, 8, "1", fontsize=20,transform=ax.transData._b)
            ax.text(7, 0, "2", fontsize=20,transform=ax.transData._b)
            ax.text(0, -8, "3", fontsize=20,transform=ax.transData._b)
            ax.text(-8, 0, "4", fontsize=20,transform=ax.transData._b)

            if lucky_number_one==1:
                ax.text(0, 8, "1", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==2:
                ax.text(7, 0, "2", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==3:
                ax.text(0, -8, "3", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==4:
                ax.text(-8, 0, "4", fontsize=20,transform=ax.transData._b,color='red')

            ax1.text(0.1,0.74,"Probability of each event",fontsize=25)
            ax1.text(0.1,0.6,"P(1) = 1/4 = 1:4 = 25%\n\nP(2) = 1/4 = 1:4 =25%\n\nP(3) = 1/4 = 1:4 =25%\n\nP(4) = 1/4 = 1:4 =25%",fontsize=20)
            #ax1.text(0.1,0.5,"This is equivalent to 50%.",fontsize=20)
            ax1.text(0.1,0.54,"Probability Outcome",fontsize=25)
            ax1.text(0.1,0.52,"The result after spinning is " + str(lucky_number_one),fontsize=20)

        elif number_parititions == 1:
            radii_ar = [1.0 for i in range(number_parititions)]
            width_ar = [4 for i in range(number_parititions)]

            radii = 10 * np.array(radii_ar)
            width = np.pi/4  * np.array(width_ar)

            bars = ax.bar(theta, radii, width=width, bottom=0.0)
            for r, bar in zip(radii, bars):
                bar.set_facecolor("pink")
                bar.set_alpha(0.6)
            ax.text(8, 0, "1", fontsize=20,transform=ax.transData._b)
            ax.text(-8, 0, "2", fontsize=20,transform=ax.transData._b)

            if lucky_number_one==1:
                ax.text(8, 0, "1", fontsize=20,transform=ax.transData._b,color='red')
            elif lucky_number_one==2:
                ax.text(-8, 0, "2", fontsize=20,transform=ax.transData._b,color='red')

            ax1.text(0.1,0.7,"Probability of each event",fontsize=25)
            ax1.text(0.1,0.6,"P(1) = 1/2 = 1:2 = 50%\n\nP(2) = 1/2 = 1:2 =50%",fontsize=20)
            ax1.text(0.1,0.54,"Probability Outcome",fontsize=25)
            ax1.text(0.1,0.52,"The result after spinning is " + str(lucky_number_one),fontsize=20)

        ax.tick_params(labelbottom=False, labeltop=False,
                   labelleft=False, labelright=False)

        ax.grid(False)
        ax.set_yticks(np.arange(1, 9, 2))
        ax1.set_yticks([])
        ax1.set_xticks([])

        ax1.set_title("Probabilities as fractions, ratios and percents",fontsize=30)
        plt.show()

style = {'description_width': 'initial'}
lucky = interact(roulette,number_parititions = widgets.Dropdown(
    options={'Two': 1, 'Four': 2, 'Six': 3,'Eight':4},
    value=4,
    description='Size of sample space:',
    style=style
),
    value = widgets.ToggleButton(
        value=True,
        description='Click to Play!',
        disabled=False,
        button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
        tooltip='Description',
        icon='check'
    ))

### Question 1

Using fractions, what is the probability of the event 1, denoted $P(1)$, if the sample size of the roulette is 4?

from ipywidgets import interact_manual,widgets

s = {'description_width': 'initial'}
@interact(answer =widgets.Select(
                    options=["Select option","1/2",\
                            "4","1/3",\
                             "1/4"],
                    value='Select option',
                    description="Probability as fraction",
                    disabled=False,
                    style=s
))

def reflective_angle_question(answer):
    if answer=="Select option":
        print("Click on the correct probability expressed as a fraction.")

    elif answer=="1/4":
        print("Correct!\nWith a sample space of size 4, each with equal likelihood, P(1)=1/4.")
    elif answer != "1/4" or answer != "Select Option":
        print("Hint: What is P(1) if P(i) = 1/4 for all i = 1,2,3,4?")

### Question 2

What is P(1) if the size of the sample space is 4, but this time expressed as percent?

from ipywidgets import interact_manual,widgets

s = {'description_width': 'initial'}
@interact(answer =widgets.Select(
                    options=["Select option","25%",\
                            "100%","40%",\
                             "10%"],
                    value='Select option',
                    description="Probability as percent",
                    disabled=False,
                    style=s
))

def reflective_angle_question(answer):
    if answer=="Select option":
        print("Click on the correct probability expressed in percent.")

    elif answer=="25%":
        print("Correct!\nWith four probability events, each with equal likelihood, P(1) = 25%.")
    elif answer != "25%" or answer != "Select Option":
        print("Hint: The total number of outcomes is 4, which corresponds to 100%. What is 100/4?")

### Question 3

Using the widget created above, change the Size of sample space to 8. What is $P(7)$ as a ratio?

from ipywidgets import interact_manual,widgets

s = {'description_width': 'initial'}
@interact(answer =widgets.Select(
                    options=["Select option","1:6",\
                            "1:4","1:8",\
                             "1:7"],
                    value='Select option',
                    description="Probability as ratio",
                    disabled=False,
                    style=s
))

def reflective_angle_question(answer):
    if answer=="Select option":
        print("Click on the correct probability expressed as a percentage.")

    elif answer=="1:8":
        print("Correct!\nWith eight probability events, each with equal likelihood, \nP(7) =  1:8.")
    elif answer != "1:8" or answer != "Select Option":
        print("Hint: 1 in every 8 corresponds to the event 7. What is P(7) in ratio?")

Independent Events & Sample Space

In this section, we define the concept of **independent probability experiments** as well as the corresponding sample space, and provide a game where we can experiment with the two concepts.

Definition. We say that two probability experiments are independent if the outcome of one does not affect the outcome of the other.

For example, if we spin two roulettes at the same time, then the two experiments are independent since the outcome of each does not affect the other.

Try spinning the two roulettes below via using the red button. As before, in each the number in red denotes the outcome of the experiment. These two roulettes are not linked and as you can see from a few spins, their outcomes are not related at all: spinning them are independent experiments.

def spin(value):
    if value==True or value==False:

        lucky_number_one = random.choice([1,2,3,4,5,6])

        lucky_number_one_c = random.choice([1,2,3,4,5,6])

        x_t,y_t = [i/10 for i in range(10)],[i/10 for i in range(10)]

        axalpha = 0.05
        figcolor = 'white'
        dpi = 80

        gs = gridspec.GridSpec(2, 3)
        fig = plt.figure(figsize=(15,8), dpi=dpi,facecolor='black')

        fig.patch.set_edgecolor(figcolor)
        fig.patch.set_facecolor(figcolor)

        ax1 = plt.subplot(gs[0, 0])
        ax1.grid(False)
        ax1.axis("Off")
        ax2 = plt.subplot(gs[1, 0])
        ax2.grid(False)
        ax2.axis("Off")
        ax5 = plt.subplot(gs[0, 2])
        ax5.grid(False)
        ax5.axis("Off")
        ax6 = plt.subplot(gs[1, 2])
        ax6.grid(False)
        ax6.axis("Off")
        #ax.axis("Off")

        ax = plt.subplot(gs[0,0],projection='polar',facecolor="red") # row 0, col 0
        plt.plot([0,1])

        ax.plot(x_t,y_t,transform=ax.transData._b,color="#FEE9FF",linewidth=5)

        ax.patch.set_alpha(axalpha)
        ax.set_axisbelow(True)

        number_parititions = 3
        arc = 2. * np.pi
        N = number_parititions
        theta = np.arange(0.0, arc, arc/N)

        radii_ar = [1.0 for i in range(number_parititions)]
        width_ar = [1.3 for i in range(number_parititions)]

        radii = 10 * np.array(radii_ar)
        width = np.pi/4  * np.array(width_ar)

        bars = ax.bar(theta, radii, width=width, bottom=0.0)
        for r, bar in zip(radii, bars):
            bar.set_facecolor("pink")
            bar.set_alpha(0.6)
        ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b)
        ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b)
        ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b)
        ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b)
        ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b)
        ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b)

        if lucky_number_one==1:
            #bar3.set_facecolor("#000000")
            ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b,color='red')
        elif lucky_number_one==2:
            #bar2.set_facecolor("#000000")
            ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==3:
            #bar1.set_facecolor("#000000")
            ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==4:
            #bar6.set_facecolor("#000000")
            ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==5:
            #bar5.set_facecolor("#000000")
            ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==6:
            #bar4.set_facecolor("#000000")
            ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b,color='red')

        ax.tick_params(labelbottom=False, labeltop=False,
                   labelleft=False, labelright=False)

        ax.grid(False)
        ax.axis("On")
        ax.set_title("Top Roulette",fontsize=20)

        ax4 = plt.subplot(gs[1, 0],projection='polar',facecolor="red") # row 0, col 0
        plt.plot([0,1])

        ax4.plot(x_t,y_t,transform=ax4.transData._b,color="#FEE9FF",linewidth=5)
        ax4.patch.set_alpha(axalpha)
        ax4.set_axisbelow(True)

        bars = ax4.bar(theta, radii, width=width, bottom=0.0)
        for r, bar in zip(radii, bars):
            bar.set_facecolor("pink")
            bar.set_alpha(0.6)
        ax4.text(-4, 7, "1", fontsize=20,transform=ax4.transData._b)
        ax4.text(3, 6.5, "2", fontsize=20,transform=ax4.transData._b)
        ax4.text(7, 0, "3", fontsize=20,transform=ax4.transData._b)
        ax4.text(3, -6.5, "4", fontsize=20,transform=ax4.transData._b)
        ax4.text(-4, -7, "5", fontsize=20,transform=ax4.transData._b)
        ax4.text(-8, 0, "6", fontsize=20,transform=ax4.transData._b)

        if lucky_number_one_c==1:
            #bar3.set_facecolor("#000000")
            ax4.text(-4, 7, "1", fontsize=20,transform=ax4.transData._b,color='red')
        elif lucky_number_one_c==2:
            #bar2.set_facecolor("#000000")
            ax4.text(3, 6.5, "2", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==3:
            #bar1.set_facecolor("#000000")
            ax4.text(7, 0, "3", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==4:
            #bar6.set_facecolor("#000000")
            ax4.text(3, -6.5, "4", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==5:
            #bar5.set_facecolor("#000000")
            ax4.text(-4, -7, "5", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==6:
            #bar4.set_facecolor("#000000")
            ax4.text(-8, 0, "6", fontsize=20,transform=ax4.transData._b,color='red')

        ax4.tick_params(labelbottom=False, labeltop=False,
                   labelleft=False, labelright=False)

        ax4.grid(False)
        ax4.axis("On")
        ax4.set_title("Bottom Roulette",fontsize=20)

        x,y = np.array([i/10 for i in range(11)]),np.array([i/10 for i in range(11)])

        ax2 = plt.subplot(gs[0,1:]) # row 0, col 0
        plt.plot([0,1])

        ax2.grid(False)
        ax2.axis("Off")
        #ax2.set_title("Top Roulette Outcome",fontsize=20)
        ax2.plot(x,y,color='white',linewidth=4)
        ax2.text(0.1,0.5,"Top Roulette Outcome: " +str(lucky_number_one),fontsize=20)

        ax5 = plt.subplot(gs[1,1:]) # row 0, col 0
        plt.plot([0,1])

        ax5.grid(False)
        ax5.axis("Off")
        #ax5.set_title("Bottom Roulette Outcome",fontsize=20)
        ax5.plot(x,y,color='white',linewidth=4)
        ax5.text(0.1,0.5,"Bottom Roulette Outcome: " + str(lucky_number_one_c),fontsize=20)

lucky = interact(spin,value = widgets.ToggleButton(
        value=True,
        description="Spin",
        disabled=False,
        button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
        tooltip='Description',
        icon='check'
    ))

Recall that

    Definition. The sample space of an experiment is the set of all possible outcomes of that experiment.

If, for example, we take the two roulettes above, we can define the sample space of spinning both of then at the same time as the set of all ordered pairs $(n_t,n_b)$, where $n_t$ denotes the outcome of spinning the top roulette and $n_b$ denotes the outcome of spinning the bottom roulette. This sample space is given by the table below, which is a 6 by 6 table where each entry is a pair $(n_t,n_b)$.

def spin_sample_space(value):
    if value==True or value==False:

        lucky_number_one = random.choice([1,2,3,4,5,6])

        lucky_number_one_c = random.choice([1,2,3,4,5,6])

        x_t,y_t = [i/10 for i in range(10)],[i/10 for i in range(10)]

        axalpha = 0.05
        figcolor = 'white'
        dpi = 80

        gs = gridspec.GridSpec(2, 2)
        fig = plt.figure(figsize=(15,8), dpi=dpi,facecolor='black')

        fig.patch.set_edgecolor(figcolor)
        fig.patch.set_facecolor(figcolor)
        ax = plt.subplot(gs[0, 0],projection='polar',facecolor="red")
        plt.plot([0,1])

        ax.patch.set_alpha(axalpha)
        ax.set_axisbelow(True)

        ax.plot(x_t,y_t,transform=ax.transData._b,color="#FEE9FF",linewidth=5)

        number_parititions = 3
        arc = 2. * np.pi
        N = number_parititions
        theta = np.arange(0.0, arc, arc/N)

        radii_ar = [1.0 for i in range(number_parititions)]
        width_ar = [1.3 for i in range(number_parititions)]

        radii = 10 * np.array(radii_ar)
        width = np.pi/4  * np.array(width_ar)

        bars = ax.bar(theta, radii, width=width, bottom=0.0)
        for r, bar in zip(radii, bars):
            bar.set_facecolor("pink")
            bar.set_alpha(0.6)
        ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b)
        ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b)
        ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b)
        ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b)
        ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b)
        ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b)

        if lucky_number_one==1:
            #bar3.set_facecolor("#000000")
            ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b,color='red')
        elif lucky_number_one==2:
            #bar2.set_facecolor("#000000")
            ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==3:
            #bar1.set_facecolor("#000000")
            ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==4:
            #bar6.set_facecolor("#000000")
            ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==5:
            #bar5.set_facecolor("#000000")
            ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==6:
            #bar4.set_facecolor("#000000")
            ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b,color='red')

        ax.tick_params(labelbottom=False, labeltop=False,
                   labelleft=False, labelright=False)

        ax.grid(False)
        ax.axis("On")
        ax.set_title("Top Roulette",fontsize=20)

        ax4 = plt.subplot(gs[1, 0],projection='polar',facecolor="red") # row 0, col 0
        plt.plot([0,1])

        ax4.patch.set_alpha(axalpha)
        ax4.set_axisbelow(True)

        ax4.plot(x_t,y_t,transform=ax4.transData._b,color="#FEE9FF",linewidth=5)
        bars = ax4.bar(theta, radii, width=width, bottom=0.0)
        for r, bar in zip(radii, bars):
            bar.set_facecolor("pink")
            bar.set_alpha(0.6)
        ax4.text(-4, 7, "1", fontsize=20,transform=ax4.transData._b)
        ax4.text(3, 6.5, "2", fontsize=20,transform=ax4.transData._b)
        ax4.text(7, 0, "3", fontsize=20,transform=ax4.transData._b)
        ax4.text(3, -6.5, "4", fontsize=20,transform=ax4.transData._b)
        ax4.text(-4, -7, "5", fontsize=20,transform=ax4.transData._b)
        ax4.text(-8, 0, "6", fontsize=20,transform=ax4.transData._b)

        if lucky_number_one_c==1:
            #bar3.set_facecolor("#000000")
            ax4.text(-4, 7, "1", fontsize=20,transform=ax4.transData._b,color='red')
        elif lucky_number_one_c==2:
            #bar2.set_facecolor("#000000")
            ax4.text(3, 6.5, "2", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==3:
            #bar1.set_facecolor("#000000")
            ax4.text(7, 0, "3", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==4:
            #bar6.set_facecolor("#000000")
            ax4.text(3, -6.5, "4", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==5:
            #bar5.set_facecolor("#000000")
            ax4.text(-4, -7, "5", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==6:
            #bar4.set_facecolor("#000000")
            ax4.text(-8, 0, "6", fontsize=20,transform=ax4.transData._b,color='red')

        ax4.tick_params(labelbottom=False, labeltop=False,
                   labelleft=False, labelright=False)

        ax4.grid(False)
        ax4.axis("On")
        ax4.set_title("Bottom Roulette",fontsize=20)

        ax1 = plt.subplot(gs[:, 1:],facecolor='#0475A8') # row 1, span all columns
        plt.plot([0,1])
        ax1.set_axisbelow(True)
        ax1.grid(color='black', linestyle='-', linewidth=2)

        ax1.set_xlim(0.1,0.7)
        ax1.set_ylim(0.1,0.7)

        rec = Rectangle([lucky_number_one/10,lucky_number_one_c/10],0.1,0.1,facecolor="black")
        ax1.add_patch(rec)
        x,y = [lucky_number_one/10,lucky_number_one/10+ 0.1], [lucky_number_one_c/10,lucky_number_one_c/10 + 0.1]
        ax1.plot(x,y,color='black',linewidth=4)
        for i in range(1,7):
            for j in range(1,7):
                ax1.text(i/10 + 0.02,j/10 + 0.055,"(" + str(i)+","+ str(j)+")",fontsize = 15,color='white')
        ax1.set_xticklabels([" ",1,2,3,4,5,6])
        ax1.set_yticklabels([" ",1,2,3,4,5,6])
        ax1.set_xlabel("Top Roulette Outcome",fontsize = 20)
        ax1.set_ylabel("Bottom Roulette Outcome",fontsize = 20)
        ax1.xaxis.tick_top()
        ax1.invert_yaxis()

        plt.show()

lucky = interact(spin_sample_space,value = widgets.ToggleButton(
        value=True,
        description="Spin",
        disabled=False,
        button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
        tooltip='Description',
        icon='check'
    ))

In this case, since the events are independent and each event has six possible outcomes, the sample space contains

$$6 \times 6 = 36$$

possible outcomes, where each outcome is a pair of the form $(n_t,n_b)$.

The probability of obtaining any given pair $(n_t,n_b)$, is given by the probability of obtaining $n_t$ with the top roulette (first experiment) multiplied by the probability of obtaining $n_b$ with the bottom roulette (second experiment). If we assume that each event is equally likely to occur

$$P(n_t,n_b) = \dfrac{1}{6} \times \dfrac{1}{6} = \dfrac{1}{36}$$

Note that we can multiply
* the sizes of the two sample spaces (for each roulette) to obtain the sample space size of the combined experiments
* the probabilities of obtaining $n_t$ and $n_b$ to obtain the probability of the event $(n_t,n_b)$
**because** the two experiments are **independent**. This is an important property of joint probability experiments.

Property. Consider two independent probability experiments $E_1$ and $E_2$ of respective sample space sizes $N_1$ and $N_2$. The size of the sample space of the joint experiment $(E_1,E_2)$ is $N_1\times N_2$. The probability of the event $(n_1,n_2)$ is $P(n_1)\times P(n_2)$.

## Question 4

What is the probability assigned to the event (1,1)?

from ipywidgets import interact_manual,widgets

s = {'description_width': 'initial'}
@interact(answer =widgets.Select(
                    options=["Select option","2/36",\
                            "1/6","36",\
                             "1/36"],
                    value='Select option',
                    description="Probability as fraction",
                    disabled=False,
                    style=s
))

def reflective_angle_question(answer):
    if answer=="Select option":
        print("Click on the correct probability expressed as a fraction.")

    elif answer=="1/36":
        print("Correct!")
    elif answer != "1/36" or answer != "Select Option":
        print("Hint: There are 36 events, each with equal likelihood of occurrence. \nYou also know that each pair is unique.")

A second example of a sample space

Let's take the two roulettes as before but this time let's define the sample space of spinning both of them at the same time as the as the **parity** of the sum $$n_t + n_b$$ where as before $n_t$ denotes the outcome of the Top Roulette and $n_b$ denotes the outcome of the Bottom Roulette.

Then the sample space is given by the set $\lbrace \text{even},\text{odd} \rbrace$

This sample space is given by the table below, which is a 6 by 6 table where each entry is a pair contains the **parity** of the sum $$n_t + n_b$$

def fair(value):
    if value==True or value==False:

        lucky_number_one = random.choice([1,2,3,4,5,6])
        lucky_number_one_c = random.choice([1,2,3,4,5,6])

        x_t,y_t = [i/10 for i in range(10)],[i/10 for i in range(10)]

        axalpha = 0.05
        figcolor = 'white'
        dpi = 80

        gs = gridspec.GridSpec(2, 2)
        fig = plt.figure(figsize=(15,8), dpi=dpi,facecolor='black')

        fig.patch.set_edgecolor(figcolor)
        fig.patch.set_facecolor(figcolor)
        ax = plt.subplot(gs[0, 0],projection='polar',facecolor="red") # row 0, col 0
        plt.plot([0,1])

        ax.patch.set_alpha(axalpha)
        ax.set_axisbelow(True)
        ax.plot(x_t,y_t,transform=ax.transData._b,color="#FEE9FF",linewidth=5)
        number_parititions = 3
        arc = 2. * np.pi
        N = number_parititions
        theta = np.arange(0.0, arc, arc/N)

        radii_ar = [1.0 for i in range(number_parititions)]
        width_ar = [1.3 for i in range(number_parititions)]

        radii = 10 * np.array(radii_ar)
        width = np.pi/4  * np.array(width_ar)

        bars = ax.bar(theta, radii, width=width, bottom=0.0)
        for r, bar in zip(radii, bars):
            bar.set_facecolor("pink")
            bar.set_alpha(0.6)
        ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b)
        ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b)
        ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b)
        ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b)
        ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b)
        ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b)

        if lucky_number_one==1:
            #bar3.set_facecolor("#000000")
            ax.text(-4, 7, "1", fontsize=20,transform=ax.transData._b,color='red')
        elif lucky_number_one==2:
            #bar2.set_facecolor("#000000")
            ax.text(3, 6.5, "2", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==3:
            #bar1.set_facecolor("#000000")
            ax.text(7, 0, "3", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==4:
            #bar6.set_facecolor("#000000")
            ax.text(3, -6.5, "4", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==5:
            #bar5.set_facecolor("#000000")
            ax.text(-4, -7, "5", fontsize=20,transform=ax.transData._b,color='red')

        elif lucky_number_one==6:
            #bar4.set_facecolor("#000000")
            ax.text(-8, 0, "6", fontsize=20,transform=ax.transData._b,color='red')

        ax.tick_params(labelbottom=False, labeltop=False,
                   labelleft=False, labelright=False)

        ax.grid(False)
        ax.axis("On")
        ax.set_title("Top Roulette",fontsize=20)

        ax1 = plt.subplot(gs[:, 1:],facecolor='#0475A8') # row 1, span all columns
        plt.plot([0,1])
        ax1.set_axisbelow(True)
        ax1.grid(color='black', linestyle='-', linewidth=2)

        ax1.set_xlim(0.1,0.7)
        ax1.set_ylim(0.1,0.7)
        ax1.set_xlabel("Top Roulette Outcome",fontsize=18)
        ax1.set_ylabel("Bottom Roulette Outcome",fontsize=18)
        ax1.xaxis.tick_top()
        ax1.invert_yaxis()
        #
        for i in range(1,7):
            for j in range(1,7):
                if (i+j)%2==0:
                    ax1.text(i/10 + 0.02,j/10 + 0.055,"even" ,fontsize = 15,color='white')
                else:
                    ax1.text(i/10 + 0.02,j/10 + 0.055,"odd" ,fontsize = 15,color='white')

        rec = Rectangle([lucky_number_one/10,lucky_number_one_c/10],0.1,0.1,facecolor="black")
        x,y = [lucky_number_one/10,lucky_number_one/10+ 0.1], [lucky_number_one_c/10,lucky_number_one_c/10 + 0.1]
        ax1.plot(x,y,color='black',linewidth=4)
        #rec_c = Rectangle([lucky_number_one_c/10,lucky_number_two_c/10],0.1,0.1,facecolor="#6b6e72")
        ax1.add_patch(rec)
        #ax1.add_patch(rec_c)
        ax1.set_xticklabels([" ",1,2,3,4,5,6])
        ax1.set_yticklabels([" ",1,2,3,4,5,6])

        ax4 = plt.subplot(gs[1, 0],projection='polar',facecolor="red") # row 0, col 0
        plt.plot([0,1])

        ax4.patch.set_alpha(axalpha)
        ax4.set_axisbelow(True)

        ax4.plot(x_t,y_t,transform=ax4.transData._b,color="#FEE9FF",linewidth=5)

        bars = ax4.bar(theta, radii, width=width, bottom=0.0)
        for r, bar in zip(radii, bars):
            bar.set_facecolor("pink")
            bar.set_alpha(0.6)
        ax4.text(-4, 7, "1", fontsize=20,transform=ax4.transData._b)
        ax4.text(3, 6.5, "2", fontsize=20,transform=ax4.transData._b)
        ax4.text(7, 0, "3", fontsize=20,transform=ax4.transData._b)
        ax4.text(3, -6.5, "4", fontsize=20,transform=ax4.transData._b)
        ax4.text(-4, -7, "5", fontsize=20,transform=ax4.transData._b)
        ax4.text(-8, 0, "6", fontsize=20,transform=ax4.transData._b)

        if lucky_number_one_c==1:
            #bar3.set_facecolor("#000000")
            ax4.text(-4, 7, "1", fontsize=20,transform=ax4.transData._b,color='red')
        elif lucky_number_one_c==2:
            #bar2.set_facecolor("#000000")
            ax4.text(3, 6.5, "2", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==3:
            #bar1.set_facecolor("#000000")
            ax4.text(7, 0, "3", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==4:
            #bar6.set_facecolor("#000000")
            ax4.text(3, -6.5, "4", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==5:
            #bar5.set_facecolor("#000000")
            ax4.text(-4, -7, "5", fontsize=20,transform=ax4.transData._b,color='red')

        elif lucky_number_one_c==6:
            #bar4.set_facecolor("#000000")
            ax4.text(-8, 0, "6", fontsize=20,transform=ax4.transData._b,color='red')

        ax4.tick_params(labelbottom=False, labeltop=False,
                   labelleft=False, labelright=False)

        ax4.grid(False)
        ax4.axis("On")
        ax4.set_title("Bottom Roulette",fontsize=20)

        sum_n = lucky_number_one + lucky_number_one_c

        #print("Top Roulette Outcome + Bottom Roulette Outcome")
        print(str(lucky_number_one) + " + " + str(lucky_number_one_c) + " = " + str(sum_n))
        if sum_n %2==0:
            print("OUTCOME: even" )
        else:
            print("OUTCOME: odd" )

        plt.show()

lucky = interact(fair,value = widgets.ToggleButton(
        value=True,
        description="Let's Play",
        disabled=False,
        button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
        tooltip='Description',
        icon='check'
    ))

### Question 5

We claim that this game is fair, but how can we verify it?

Recall that a probability experiment is fair if every outcome is equally likely to occur. In order for this experiment to be fair, the probability of the event **even**  *must* be equal to the probability of event **odd**.

What is the probability that the sum of the numbers in the top and bottom roulettes is even?

from ipywidgets import interact_manual,widgets

s = {'description_width': 'initial'}
@interact(answer =widgets.Select(
                    options=["Select option","1/2",\
                            "4/36","1/3",\
                             "18/36"],
                    value='Select option',
                    description="Probability sum is even",
                    disabled=False,
                    style=s
))

def fair_game(answer):
    if answer=="Select option":
        print("Click on the correct probability expressed as a fraction.")

    elif answer=="1/2" or answer=="18/36":
        print("Correct!\nThere are a total of 36 possible outcomes. 18 out of 36 are even numbers.\nThus the probability P(even) = 18/36 or 1/2. ")
    elif answer != "1/2" or answer != "Select Option" or answer!="18/36":
        print("Hint: There are 36 entries in our sample space, each with equal likelihood of occurrence.\nHow many of the 36 correspond to even numbers")

### Question 6
What is the probability that the sum of the numbers in the top and bottom roulettes is odd? In other words, what is the probability that Bob will win?

from ipywidgets import interact_manual,widgets

s = {'description_width': 'initial'}
@interact(answer =widgets.Select(
                    options=["Select option","19/36",\
                            "17/36","18/36",\
                             "1/2"],
                    value='Select option',
                    description="Probability sum is odd",
                    disabled=False,
                    style=s
))

def fair_game(answer):
    if answer=="Select option":
        print("Click on the correct probability expressed as a fraction.")

    elif answer=="1/2" or answer=="18/36":
        print("Correct!\nThere are a total of 36 possible outcomes. 18 out of 36 are odd numbers.\nThus the P(odd) = 18/36 or 1/2. ")
    elif answer != "1/2" or answer != "Select Option" or answer!="18/36":
        print("Hint: There are 36 entries in our sample space, each with equal likelihood of occurrence.\nHow many of the 36 correspond to odd numbers?")

In the section above we learned that the there are 18 out of 36 possible outcomes where the sum

$$n_t + n_b$$

is an even number. Thus

$$P(even) = \dfrac{18}{36} = \dfrac{1}{2}$$

Similarly, there are 18 out of 36 possible outcomes where the sum

$$n_t + n_b$$

is an odd number. Thus

$$P(odd) = \dfrac{18}{36} = \dfrac{1}{2}$$

Then $P(odd) = P(even)$. With this we verify that indeed the experiment is fair.

Theoretical vs Experimental Probability

We begin by stating a few definitions.

    Definition. The Theoretical Probability of an event $A$, denoted $P_T(A)$, is the ratio of the number of outcomes corresponding to this event to the number of possible outcomes.

$$P_T(A) = \dfrac{\text{Total Number of Instances of event A in the Sample Space}}{\text{Total Number of Possible Outcomes}}$$

If we take our fair experiment with two roulettes and sample space parity of outcome sum $\lbrace \text{even}, \text{odd} \rbrace$, the theoretical probability of an even event is

$$P_T(\text{even}) = \dfrac{18}{36}$$

Definition. The Experimental Probability of an event $A$, denoted $P_E(A)$, is computed over running the probability experiment a number of times and computing the observed ratio between the number of time the event occured  and the number of trials of the experiment.

$$P_E(A) = \dfrac{\text{Number of Times Event A Actually Occurred}}{\text{Number of trials}}$$

In order to determine $P_E(\text{even})$, we first need to spin the roulettes a few times and compare.

Use the widget below to simulate spinning the two roulettes. Use the slider to set a number of trials. In this interactive exercise, you can set number of trials as an integer between 1 and 100. Press `Run Interact` to run an experiment for the given number of trials.

On the right hand side you will find a printed message outlining the experimental probability of each event: Sum is Even and Sum is Odd from the number of trial specified using the widget.

On the left hand side you can find a graph comparing both.

Press the `Run Interact` button several times.

%matplotlib inline

def die(number):
    count_A,count_C = 0,0

    for i in range(number):
        lucky_number_one = random.choice([1,2,3,4,5,6])
        lucky_number_two = random.choice([1,2,3,4,5,6])
        if lucky_number_one - lucky_number_two >=0:
            count_A +=1
        else:
            count_C +=1

    return [count_A,count_C]

def even_sum(number):
    count_A,count_C = 0,0

    for i in range(number):
        lucky_number_one = random.choice([1,2,3,4,5,6])
        lucky_number_two = random.choice([1,2,3,4,5,6])

        sum_n = lucky_number_one + lucky_number_two
        if  sum_n%2 == 0:
            count_A +=1
        else:
            count_C +=1

    return [count_A,count_C]

def experimental_prob(number):
    [varoi_1,varoi_2] = even_sum(number)
    fig,(ax1,ax2,ax3) = plt.subplots(1,3,sharey=True,figsize=(15,4))

    ax2.axis("Off")
    ax3.axis("Off")
    axalpha = 0.05

    even = varoi_1/number
    odd = varoi_2/number

    labels = ['', '',  'Even Sum', '','Odd Sum']

    ax1 = plt.subplot(131,facecolor="white")
    ax1.set_title("Experimental Probability",fontsize=20)
    ax1.set_ylabel("Probability",fontsize=15)
    ax1.set_xlabel("Outcomes",fontsize=15)
    ax1.set_xlim([0,3])

    ax1.set_xticklabels(labels)

    x = np.arange(1,3)
    f1,f2= ax1.bar(x,[even,odd])
    f1.set_facecolor("#8642f4")
    f2.set_facecolor("#518900")

    ax1.grid(which='both')
    ax1.grid(b=True,which='minor',alpha=0.2,linestyle='--',color='black')
    ax1.grid(which='major', alpha=0.2,linestyle='--',color='black')

    ax3 = plt.subplot(132)
    ax3.axis("Off")

    #ax3.set_title( "Positive vs Negative\nLuck Roulette: Outcome",fontsize=20)
    rec1 = Rectangle((0.1,0.8),0.3,0.1,facecolor="#8642f4")
    ax3.add_patch(rec1)
    ax3.text(0.5,0.83,"Experimental Probability: Sum is Even",fontsize=20)
    rec2 = Rectangle((0.1,0.7),0.3,0.1,facecolor="#518900")
    ax3.add_patch(rec2)
    ax3.text(0.5,0.73,"Experimental Probability: Sum is Odd",fontsize=20)

    ax2 = plt.subplot(133,facecolor="white")
    ax2.axis("Off")

    ax2.text(0.9,0.83,str(varoi_1) + "/" + str(number),fontsize=20)
    ax2.text(0.9,0.73,str(varoi_2) + "/" + str(number),fontsize=20)

    ax3.set_title("                   Even or Odd Sum Experiment:\n",fontsize=20)

    plt.show()

def run_fair_exp(number):
    experimental_prob(number)
    #experimental_prob(number,"Negative")
interact_manual(experimental_prob,number=widgets.IntSlider(
            value=10,
            min=1,
            max=100,
            step=1,
            description='Total Number of Trials',
            disabled=False,
            continuous_update=False,
            orientation='horizontal',
            readout=True,
            readout_format='d',
            style =style
));

### Question 7

Use the box below to enter your observations.

How do the experimental probabilities of each event change as you increase the number of trials?

from ipywidgets import widgets as w
from ipywidgets import Button, Layout
from IPython.display import display, Javascript, Markdown

def rerun_cell( b ):
    display(Javascript('IPython.notebook.execute_cell_range(IPython.notebook.get_selected_index()+1,IPython.notebook.get_selected_index()+2)'))

style = {'description_width': 'initial'}

question7_text = w.Textarea( value='', placeholder='Write your answer here. Press Record Answer when you finish.', description='', disabled=False , layout=Layout(width='100%', height='75px') )
question7_button = w.Button(button_style='info',description="Record Answer", layout=Layout(width='15%', height='30px'))

display(question7_text)
display(question7_button)

question7_button.on_click( rerun_cell )

question7_input = question7_text.value

if(question7_input != ''):
    question7_text.close()
    question7_button.close()
    display(Markdown("### Your answer for Question 7: Conclusions"))
    display(Markdown(question7_input))

**Remarks**

We observe that every time we press the `Run Interact` button on the interactive above, the experimental probability of each event varies. However, it seems like as we increase the number of trials, the experimental probabilities of each event approach $1/2$.

Let us explore what happens if we increase the number of trials to, say 10,000.

In the widget below you can find a slider that allows you to control the number of trials. In this interactive exercise, you can set number of trials as an integer between 1 and 10,000.

On the left hand side you can find a plot like the one we explored above. On the right hand side you can find the theoretical probability of events Sum is Even and Sum is Odd.

Press the `Run Interact` button. Increase the Total Number of Trials and press the `Run Interact` button multiple times.

def toss(number):
    store_head = []
    store_tail = []
    other = []
    for i in range(number):
        toss_coin= random.choice(np.arange(2))

        if toss_coin==0:
            store_head.append(toss_coin)
        elif toss_coin==1:
            store_tail.append(toss_coin)

    return [store_head,store_tail]

def plot_coin_experiment(number):
    varoi = toss(number)
    fig,ax = plt.subplots(figsize=(5,5))

    ax.set_title("Distribution of Experimental Coin Flipping",fontsize=35)
    plt.ylabel("Frequency",fontsize=25)
    plt.xlabel("Heads or Tails",fontsize=25)
    plt.xticks(np.arange(2), ('Total Number of Heads', 'Total Number of Tails'))
    plt.hist(varoi[0])
    plt.hist(varoi[1])
    plt.grid(which='both')
    plt.grid(b=True,which='minor',alpha=0.2,linestyle='--',color='black')
    plt.grid(which='major', alpha=0.2,linestyle='--',color='black')

def plot_die_experiment(number):
    varoi = die(number)
    theor_A = 21/36
    theor_C = 15/36
    #print(theor)
    fig,(ax1,ax2) = plt.subplots(1,2,sharey=True,figsize=(15,8))

    # Experimental Probability
    ax1.set_title("Experimental Distribution",fontsize=25)
    ax1.set_ylabel("Frequency",fontsize=25)
    ax1.set_xlabel("Outcomes",fontsize=25)
    ax1.set_xlim([0,3])

    ax1.set_xticks([])

    x = np.arange(1,3)
    dice = [varoi[0]/number,varoi[1]/number]
    f1,f2 = ax1.bar(x,dice)
    f1.set_facecolor("#8642f4")
    f2.set_facecolor("#518900")
    ax1.grid(which='both')
    ax1.grid(b=True,which='minor',alpha=0.2,linestyle='--',color='black')
    ax1.grid(which='major', alpha=0.2,linestyle='--',color='black')

    # Theoretical Probability
    ax2.set_title("Theoretical Distribution",fontsize=25)
    ax2.set_ylabel("Frequency",fontsize=25)
    ax2.set_xlabel("Outcomes",fontsize=25)
    x = np.arange(1,3)
    dice_exp = [theor_A,theor_C]
    f11,f21 = ax2.bar(x,dice_exp)
    f11.set_facecolor("#8642f4")
    f21.set_facecolor("#518900")

    ax2.set_xlim([0,3])

    ax2.set_xticks(["Even","Odd"])

    ax2.grid(which='both')
    ax2.grid(b=True,which='minor',alpha=0.2,linestyle='--',color='black')
    ax2.grid(b=True,which='major', alpha=0.2,linestyle='--',color='black')

    plt.ylim(0,number)

    plt.show()

def plot_fair_experiment(number):
    [varoi_1,varoi_2] = even_sum(number)
    theor_A = 18/36
    theor_C = 18/36

    even = varoi_1/number
    odd= varoi_2/number
    x = np.arange(1,3)

    labels = ['', '',  'Even Sum', '','Odd Sum']

    fig,(ax1,ax2) = plt.subplots(1,2,sharey=True,figsize=(15,8))

    # Experimental Probability
    ax1.set_title("Even or Odd Sum Probability Experiment:\nExperimental Probability",fontsize=20)
    ax1.set_ylabel("Probability",fontsize=25)
    ax1.set_xlabel("Events",fontsize=25)
    ax1.set_xlim([0,3])
    ax1.set_xticklabels(labels)
    ax1.grid(which='major', alpha=0.2,linestyle='--',color='black')

    f1,f2 = ax1.bar(x,[even,odd])
    f1.set_facecolor("#8642f4")
    f2.set_facecolor("#518900")

    # Theoretical Probability
    ax2.set_title("Even or Odd Sum Probability Experiment:\nTheoretical Probability",fontsize=20)
    ax2.set_ylabel("Probability",fontsize=25)
    ax2.set_xlabel("Events",fontsize=25)
    ax2.set_xlim([0,3])
    ax2.set_xticklabels(labels)
    ax2.grid(b=True,which='major', alpha=0.2,linestyle='--',color='black')

    dice_exp = [theor_A,theor_C]
    f11,f21 = ax2.bar(x,dice_exp)
    f11.set_facecolor("#8642f4")
    f21.set_facecolor("#518900")

    plt.ylim(0,1)
    plt.show()

interact_manual(plot_fair_experiment,number=widgets.IntSlider(
            value=5,
            min=1,
            max=10000,
            step=1,
            description='Total Number of Trials',
            disabled=False,
            continuous_update=False,
            orientation='horizontal',
            readout=True,
            readout_format='d',
            style =style
));

### Question 8

How does the experimental probability of each event change as we increase the number of trials? Use the textbox below to record your answers.

from ipywidgets import widgets as w
from ipywidgets import Button, Layout
from IPython.display import display, Javascript, Markdown

def rerun_cell( b ):

    display(Javascript('IPython.notebook.execute_cell_range(IPython.notebook.get_selected_index()+1,IPython.notebook.get_selected_index()+2)'))
style = {'description_width': 'initial'}

question8_text = w.Textarea( value='', placeholder='Write your answer here. Press Record Answer when you finish.', description='', disabled=False , layout=Layout(width='100%', height='75px') )
question8_button = w.Button(button_style='info',description="Record Answer", layout=Layout(width='15%', height='30px'))

display(question8_text)
display(question8_button)

question8_button.on_click( rerun_cell )

question8_input = question8_text.value

if(question8_input != ''):

    question8_text.close()
    question8_button.close()
    display(Markdown("### Your answer for Question 8: Conclusions"))
    display(Markdown(question8_input))

As the number of trials increases, we observe that the experimental probability of each event approaches the corresponding theoretical probability. This is known as the "Law of Large Numbers".

Conclusion

In this notebook we learned what a probability experiment is, what the sample space, events and outcome associated to a given probability experiment are. We learned how to express probabilities as ratios, fractions and percents.

We learned what independent probability events are and introduced the concept of a fair game.

We learned what theoretical and experimental probability are and with the help of an interactive exercise, we learned that as the number of trials increases, experimental probability approaches theoretical probability.

[![Callysto.ca License](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-bottom.jpg?raw=true)](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)