Review of Lesson 3 + SGD

Welcome back and here is lesson 4 which is where we get deep into the weeds of exactly what is going on when we are training a neural network and we started looking at this in the previous lesson. We were looking at the stochastic gradient descent and so to remind you, we were looking at what Arthur Samuel said. “Suppose we arrange for some automatic means of testing the effectiveness of any current weight assignment ( or we would call it parameter) in terms of actual performance and provide a mechanism for altering the weight assignment so as to maximize that performance. So we could make that entirely automatic and a machine so programmed would learn from its experience” and that was it. So our initial attempt on the MNIST data set was not really based on that. We didn't really have any parameters. So then , last week we tried to figure out how we could parameterize it, how we could create a function that had parameters. And, what we thought we could do would be to have something where that say the probability of being some particular number was expressed in terms of the pixels of that number and some weights, and then we would just multiply them together and add them up. So we looked at how stochastic gradient descent works last week and the basic idea is that we start out by initializing the parameters randomly. We use them to make a prediction using a function, such as this one. We then see how good that prediction is, by measuring using a loss function, we then calculate the gradient which is how much with the loss change if I change one parameter by a little bit, we then use that to make a small step to change each of the parameters by a little bit, and by multiplying the learning rate by the gradient to get a new set of predictions and so we went round and round and round a few times until eventually we decided to stop and so these are the basic seven steps. Then we went through and so we did that for simple quadratic equation, and we had something which looked like this and so by the end we had this nice curve getting closer and closer and closer. So I have a little summary at the start of this section summarizing gradient descent that Silvyan and I have in the notebooks, in the book, of what we just did, so you can review that, and make sure it makes sense to you.

MNIST Loss Function

So now let's use this to create our MNIST “3” vs. “7” model and so to create a model, we're going to need to create something that we can pass into a function like, let's see where it was, passing to a function like this one. So we need just some pixels that are all lined up and some parameters that are all lined up, and then we're going to sum them up. So our axis are going to be pixels and so in this case because we're just going to multiply each pixel by a parameter and add them up, the effect that they're laid out in a grid is not important so let's reshape those grids and turn them into vectors. The way we reshape things in Pytorch is by using the “.view” method. The view method you can pass to it how large you want each dimension to be and so in this case we want the number of columns to be equal to the total number of pixels in each picture., which is 28 x 28. because they're 28 by 28 images. And then the number of rows will be however many rows there are in the data, and so if you just use minus one when you call view, that means, you know, as many as there are in the data, so this will create something of the same, with the same total number of elements that we had before. So we can grab all our 3 we can concatenate them with torch.cat with all of our 7 and then reshape that into a Matrix where each row is one Image with all of the rows and columns of the image all lined up in a single vector. Then we're going to need labels, so that's our X, so we're going to need labels, our labels will be a 1 for each of the 3s and a 0 for each of the 7s so basically, we're creating “is 3 model”. So that's going to create a vector, we actually need it to be a matrix in Pytorch so .unsqueeze will add an additional unit dimension to wherever I've asked for so here in position one, so in other words, this is going to turn up from something which is a vector of 12396 long into a matrix with 12396 rows and one column. That's just what Pytorch expects to see.

What is a Dataset in PyTorch?

So now we're going to turn our X, Y into a Dataset and a Dataset is a very specific concept in Pytorch. It's something which we can index into, using square brackets, and when we do so, it's expected to return a tuple. So here if we look at how we're going to create this Dataset and when we index into it, it's going to return a tuple containing our independent variable and a dependent variable, for each particular row, and so to do that we can use the Python zip function, which takes one element of the first thing and combines it with concatenates it with one element of the second thing and then it does that again and again and again and so then if we create a list of those it gives us a, It is as a Dataset!. It gives us a list which when we index into it, it's going to contain one image and one label, and so here you can see why there's my label and my image I won't print out the whole thing, but it's a 784 long vector. So that's a really important concept, a Dataset is something that you can index into, and get back at Tuple. And here I am, this is called destructuring the tuple: which means I'm taking the two parts of the tuple and putting the first part in one variable in the second part in the other variable, which is something we do a lot in Python, it's pretty handy, a lot of other languages support that as well. Repeat the same three steps for a validation set. So we've now got a training Dataset and a validation Dataset.

Initializing our parameters

Right! So now we need to initialize our parameters and so to do that, as we've discussed, we just do it randomly. So here's a function. They're given some size, some, some shape if you like. We'll randomly initialize, using a normal random number distribution in PyTorch that's what .Randn does and we can hit shift tab to see how that works, okay!? And it says here that it's going to have a variance of one. So, I probably should NOT call this standard deviation, I probably should have called this: variance actually. So multiply it by the variance - to change its variance to whatever is requested, which will default to one. And then as we talked about, when it comes to calculating our gradients we have to tell PyTorch which things we want gradients for and the way we do that is requires_grad_ . Remember this underscore at the end is a special magic symbol which tells PyTorch that we want this function to actually change the thing that it's referring to. This will change this tensor, Such that it requires radiance. So here's some weights, so our weights are going to need to be 28 by 28 by 1 shape 28 by 28 because every pixel is going to need a weight and then 1 because we're going to need it again , so we're going to need to have that unit access to make it into a column. So that's what PyTorch expects. So there's our weights. Now just weights by pixels actually isn't going to be enough because weights by pixels were always equal zero. When the pixels are equal to zero, it has a zero intercept. So we really want something where it's like W * X + B, a line. So the B is we call the bias and So that's just going to be a single number. So let's grab a single number for our bias. So, remember I told you, there's a difference between the parameters and weights. So, actually speaking, so here the weights are the W in this equation, the bias is B in this equation, and the weights and bias together is the parameters of the function they're all the things that we're going to change, they're all the things that have gradients that we're going to update. So there's an important bit of jargon for you: the weights and biases of the model are the parameters. We can... yes question! R: “What's the difference between gradient descent and stochastic gradient descent? J: So far we've only done gradient descent and will be doing stochastic gradient descent in a few minutes.

Predicting images with matrix multiplication

We can now create and calculate predictions for one image so we can take an image such as the first one and multiply by the weights only to transpose them to make them line up in terms of the rows and columns and add it up, and add the bias and there is a prediction. We want to do that for every image we could do that with a for loop and that would be really really slow. It wouldn't run on the GPU and it wouldn't run in optimize and see code. So we actually want to use always to do kind of like looping over pixels, looping over images. You always need to try to make sure you're doing that without a Python for loop, in this case doing this calculation for lots of rows and columns is a mathematical operation called matrix multiply, so if you've forgotten your matrix multiplication or maybe never quite got around to it at a high school. It would be a good idea to have a look at Khan Academy or something to learn about what it is, but it's actually I'll give you the quick answer. This is from Wikipedia, if these are two matrices A and B then this element here, 1, 2 in the output is going to be equal to the first bit here times the first bit here, plus the second bit here, times the second bit here. So it's going to be A1,2 * A1,1 + B 2,2 * A 1,2 that's you can see the orange matches the orange. Ditto for over here. This would be equal to B1,3 * A 3,1 + B2,3 * A 3,2 and so forth for every. Here's a great picture of that in action if you look at matrixmultiplication.XYZ another way to think of it is we can kind of flip the second bit over on top and then multiply each bit together and add them up, multiplied each bit together and add them up and you can see always the second one here and ends up in the second spot and the first one ends up in the first spot. And that's what matrix multiplication is. So we can do our, multiply and Add them up by using matrix multiplication and in Python and therefore PyTorch matrix multiplication is the @ sign operator. So when you see @, that means matrix multiply so here is our 20.2336 if I do a matrix multiply of our training set by our weights and then we add the bias and here is our 20.2336 for the first one. And you can see through. It's doing every single one, okay!?. So that's really important is that matrix multiplication gives us an optimized way to do these simple linear functions, whereas we want as many kinds of rows and columns as we want. So this is one of the two fundamental equations of any neural network. Some rows, of data rows and columns of data much use multiply some weights add some bias and the second one which was here in a moment is an activation function.

Why you shouldn't use accuracy loss function to update parameters

So that is some predictions from our randomly initialized model, so we can check how good our model is and so to do that we can decide that anything greater than 0 we will call a 3 and anything less than 0 we will call a 7. So preds greater than 0.0 tells us whether or not something is predicted to be a 3 or not. Then turn that into a float, so rather than true and false, make it one in zero because it's what our training set contains and then check with our thresholded predictions are equal to our training set and this will return true every time a row is correctly predicted and false otherwise. So if we take all those trues and falses and turn them into floats, that'll be ones and zeroes and then take their mean: It's 0.49, so not surprisingly our randomly initialized model is right about half the time at predicting threes from sevens. I Added one more method here, which is .item() without item This would return a tensor, It's a rank zero tensor. It has no rows. It has no columns it just it's just a number on its own, but I actually wanted to unwrap it to create a normal Python scalar mainly just because I wanted to see them easily see the full set of decimal places and the reason for that is I want to show you how we're going to calculate the derivative on the accuracy. By changing a parameter a tiny bit, so let's take one parameter which will be weights[0] and Multiply it by 1.0001 and so that's going to make it a little bit bigger and then if I calculate how the the accuracy changes based on the change in that weight that will be the gradient of the accuracy with respect to that parameter so I can do that by calculating my new set of predictions and then I can threshold them and then I can check whether they're equal to the training set and then take the meanAnd I get back: exactly the same number so remember that Gradient is equal to Rise over run if you remember back to your calculus or if you'd forgotten your calculus. Hopefully you've reviewed it on Khan Academy so The change in the Y so y_new - y_old which is 0.4912 etc minus 0.4912 etc, which is 0 divided by This change will give us 0 so at this point we have a problem our derivative is 0 so we have 0 gradients: Which means our step will be zero which means our prediction will be unchanged.Okay So we have a problem and our problem is that our gradient is zero and with a gradient ofZero we can't take a step and we can't get better predictions. And so Intuitively speaking the reason that our gradient is zero is because when we change a single pixel by a tiny bit we might not ever in any way change an actual prediction to change from a three predicting a three to a seven or Vice versa because we have this we have this threshold. And so in other words our Our accuracy Loss function here is very bumpy. It's like a flat step flat step flat step. So it's got this Zero gradient all over the place.

Creating a good loss function

So what we need to do is use something other than accuracy as our loss function So, let's try and create a new function and what this new function is going to do is it's going to Give us a better value. Kind of in much the same way that Accuracy gives a better value. So this is the loss member of small losses better so to give us a lower loss when the accuracy is better, but it won't have a zero gradient. It means that a slightly better prediction needs to have a slightly better lossSo, let's have a look at an example. Let's say our Targets, our labels of like that are three. Oh There's just three rows three images here one zero one, okayAnd we've made some predictions from a neural net and those predictions gave us a point. [0.9, 0.4, 0.2] so now consider: This loss function a loss function: we're going to use torch.where() which is basically the same as This list comprehension it’s basically an if statement so it's going to say for for where target equals one We're going to return 1 minus predictions. So here target is one so it'll be 1 minus 0.9 and Where target is not 1 it'll just be predictions so well these Examples here. The first one target equals 1 will be 1 - 0.9 = 0.1.The next one is target equals 0 so to speak the prediction is just 0.4 And then for the third one, it's a 1 for target So it'll be 1 - prediction, which is 0.8. And so you can see here when the Prediction is correct. Correct. In other words, it's a number, you know It's a high number when the target is 1 and a low number when the target is 0, these numbers are going to be smaller. So the worst one is when we predicted 0.2. So we're pretty we really thought that was actually a zero But it's actually a 1 so we ended up with a 0.8 here because this is 1 minus prediction 1 - 0.2 = 0.8, So we can then take the mean of all of these to calculate a loss. So if you think about it this loss will be the smallest if The predictions are exactly right. So if we did predictions is actually identical to the targets this will be [0., 0., 0.] okay, where else if they were exactly wrong say they were one then it's [1., 1., 1.]. So it's going to be the loss will be better ie smaller when the predictions are closer to the targets. And so here we can now take the mean and when we do we get here 0.433. Let's say we change this last bad one in accurate prediction from 0.2 to 0.8 and the loss gets better from 0.43 to 0.23 but this is just this function is torch.where(). So this is actually pretty good. This is actually a loss function which pretty closely tracks accuracy was the accuracies better, The loss will be smaller But also it doesn't have these zero gradients because every time we change the prediction the loss changes Because the prediction is literally harder the loss that's pretty neat, isn't it? One problem is this is only going to work. Well, as long as the predictions are between 0 & 1 Otherwise, this one - prediction thing is going to look a bit funny. We should try and find a way to ensure that the predictions are always between zero and one and that's also going to just make a lot more intuitive sense because you know we like to be able to kind of think of these as if they're like probabilities or at least nicely scaled numbers so we need some function that can take our numbers: Have a look, something which can take these big numbers and turn them all into numbers between zero and one and it so happens that we have exactly the right function. It's called the sigmoid functions of the sigmoid function looks like this if you pass in a really small number you get a number very close to zeroIf you pass in a big number you get a number very close to 1 it never gets past one and it never goes smaller than zero and then it's kind of like this smooth curve between and in the middle It looks a lot like the y = x line. This is the definition of the sigmoid function It's 1 over 1 + e to minus x What is exp? exp is just e to the power of something so if we look at e: It's just a number like pi so simply, it's just a number that has a particular value right? So if we go e Squared and we look at It's going to be a tensor, use PyTorch, make it a float:There we go and you can see that these are the same number so that's what torch.exp means. Okay, so you know for me when I see these kinds of interesting functions, I don't worry too much about The definition what I care about is the shape alright So you can have a play around with graphing calculators or whatever to kind of see Why it is that you end up with this shape from this particular equation but for me, I just never think about that it never Really matters to me what's important is this sigmoid shape, which is what we want. It's something that squashes every number to be between naught and 1 So we can change em nest loss to be exactly the same as it was before but first we can make everything into sigmoid First and then use torch,where() so that is a loss function that has all the properties we want. It'sAt something which is going to be have not have any of those nasty Zero gradients and we've ensured that the input to the where() between naught and one SoThe reason we did this is because our our accuracy Was Kind of what we really care about is a good accuracy. We can't use it to get our gradients. Just just to create our steps to improve our parameters So we can change Our our accuracy to another function that is similar in terms of it It's better when the accuracy is better, but it also does not have these zero gradients And so you can see now where why we have a metric and a loss the metric is the thing we actually care about the loss is the thing that's similar to what we care about but has a nicely behaved gradient.

Updating parameters with mini-batches and DataLoader

Sometimes the thing you care about your metric does have a nicely defined gradient and you can use it directly as a lossFor example, we often use means squared error but for classification unfortunately notSo we need to now use this toTo update the parameters And so there's a couple of ways we could do this one would be to loop through every image, Calculate a prediction for that image and then calculate a loss and then do a step andThen step the other parameters and then do that again for the next image in the next image in the next image. That's going to be really slow. Because we're we're doing a single step for a single image. So that would mean an epoch would take quite a while. We could go much faster, By doing every single image in the data set so a big matrix multiplication It can all be paralyzed on the GPU and then so then we can We could then do a step based on the gradients looking at the entire dataset but now that's going to be like a lot of work to just update the weights once remember sometimes our datasets have Millions or tens of millions of items. So that's probably a bad idea too. So why not compromise? Let's grab a few data items at a timeTo calculate our loss and our step now if we grab a few data items at a time those two data items are called a mini batch and a mini batch just means a few pieces of dataAnd so the size of your mini batch is called not surprisingly the batch size, right? so the bigger the batch size the closer you get to the full size of your data set the longer it's going to take to Calculate a singleSet of losses a single step But the more accurate it's going to be, it's going to be like, the gradients are going to be much closer to the true data set gradients. And then the smaller the batch size the faster each step we'll be able to do, but those steps will represent a smaller number of items and so they won't be such an accurate approximation of the real gradient of the whole dataset. Is there a reason the mean of the loss is calculated over, say, doing a median, since the median is less prone to getting influenced by outliers? In the example you gave, if the third point which was wrongly predicted as an outlier, then the derivative would push the function away while doing SGD, and a median could be better in that case. Honestly, I've never tried using a median. The problem with a median is, it ends up really only caring about one number, which is the number in the middle. So it could end up really pretty much ignoring all of the things at each end, and all it really cares about is the order of things. So my guess is that you would end up with something that is only good at predicting one thing in the middle. But I haven't tried it. It’d be interesting to see. Well, I guess the other thing that would happen with a median is, you would have a lot of zero gradients, I think. Because it's picking the thing in the middle, and you could, you know change your values, and the thing in the middle, well wouldn't be zero gradients, but bumpy gradients. I think in the middle would suddenly jump to being a different item. So it might not behave very well. That's my guess. You should try it. Okay, so how do we ask for a few items at a time? It turns out that Pytorch and Fastai provide something to do that for you. You can pass in any dataset to this class called DataLoader and it will grab a few items from that dataset at a time. You can ask for how many by asking for a batch size, and then, as you can see, it will grab a few items at a time until it's grabbed all of them. So here, I'm saying let's create a collection that just contains all the numbers from nought to 14. Let's pass that into a DataLoader with a batch size of 5, And then, that's going to be something called an iterator, in Python. It's something that you can ask for one more thing from an iterator. If you pass an iterator to list in Python, it returns all of the things from the iterator. So here are my three mini-batches, and you'll see here all the numbers from nought to 15 appear. They appear in a random order, and they appear five at a time. They appear in random order, because shuffle = True. So normally in the training set we ask for things to be shuffled, so it gives us a little bit more randomization. More randomization is good, because it makes it harder for it to learn what the dataset looks like. So that's how our DataLoader is created. Now, remember though, that our datasets actually return tuples, and here I've just got single ints. So let's actually create a tuple. So if we enumerate all the letters of English, then that means that returns (0, ‘a’), (1, ‘b’), (2, ’c’) etc. Let's make that our dataset. So, if we pass that, to a DataLoader with a batch size of 6, and as you can see, it returns tuples containing 6 of the first things, and the associated 6 of the second things. So this is like our independent variable and this is like our dependent variable. Okay, and so at the end, you know that with the batch size weren't necessarily exactly divided nicely into the full size of the Dataset, you might end up with a smaller batch.

Putting it all together

So basically then, we already have a Dataset remember, and so we could pass it to a DataLoader and then we can basically say this. An iterator in Python is something that you can actually loop through. So when we say for in DataLoader, it's going to return a tuple. We can de-structure it into the first bit and the second bit, and so that's going to be our x and y. We can calculate our predictions, we can calculate our loss from the predictions and the targets, we can ask it to calculate our gradients and then we can update our parameters just like we did in our toy SGD example for the quadratic equation. So that's re-initialize our weights and bias with the same two lines of code before, let's create the data loader this time from our actual MNIST dataset and create a nice big batch size, so we do plenty of work each time, and just to take a look. Let's just grab the first thing from the ‘DataLoader’. ‘first’ is a fast AI function, which just grabs the first thing from an iterator. It's just, it’s useful to look at, you know, kind of an arbitrary mini batch. So here is the shape we're going to have. The first mini batch is 256 rows of 784 long, that's 28 by 28. So 256 flattened out images, and 256 labels that are 1. Well, because there's just the number 0 or the number 1, depending on whether as a 3 or a 7. We do the same for the validation set. So here's a validation DataLoader... And so let's grab a batch here, testing, pass it into, well, why do we do that? We should... What... Look... Yeah, I guess, yeah, actually for our testing, I'm going to just manually grab the first four things just so that we can make sure everything lines up. So... So let's grab just the first four things. We'll call that a batch. Pass it into that linear function, we created earlier. Remember linear, was just, x batch at weights matrix multiply, plus bias. And so that's going to give us four results. That's a prediction of each of those four images. And so then we can calculate the loss, using that loss function we just used, and let's just grab the first four items of the training set, and there's the loss. Okay. And so now we can calculate the gradients, and so the gradients are 784 by 1, so in other words it's a column where every weight as a gradient. It's what's the change in loss for a small change in that parameter, and then the bias as a gradient it’s a single number, because the bias is just a single number. So we can take those three steps and put it in a function. If you pass... If you... This is ‘calculate gradient’. You pass it an X batch, a Y batch, and some model, then it's going to calculate the predictions, calculate the loss, and do the backward step. And here we see ‘calculate gradient’, and so we can get the, just to take a look, the mean of the weights gradient, and the bias gradient. And there it is. If I call it a second time, and look. Notice I have not done any step here. This is exactly the same parameters. I get a different value. That’s a concern. You would expect to get the same gradient every time you called it with the same data. Why have the gradients changed? That's because, ‘loss.backward’ does not just calculate the gradients. It calculates the gradients, and adds them to the existing gradients. The things in the ‘.grad’ attribute. The reasons for that we'll come to later, but for now the thing to know is just it does that. So actually what we need to do is to call ‘grad.dot.zero_’. So ‘dot.zero’ returns a tensor containing zeros, and remember ‘_’ does it in place so that updates the ‘weights.grad’ attribute, which is a tensor, to contain zeros. So now if I do that, and call it again, I will get exactly the same number. So here is how you train one epoch with SGD. Loop through the DataLoader, grabbing the X batch, and the Y batch, calculate the gradient, prediction, loss backward. Go through each of the parameters. We're going to be passing those in. So there's going to be the 768 weights, and the one bias, and then, for each of those, update the parameter, to go minus equals gradient times learning rate. That's our Gradient Descent step. And then zero it out for the next time around the loop. I'm not just saying p minus equals. I'm saying ‘p.data’ minus equals, and the reason for that is that, remember, PyTorch keeps track of all of the calculations we do, so that it can calculate the gradient. Well, I don't want to calculate the gradient of my Gradient Descent step. That's like not part of the model, right? So dot data is a special attribute in Pytorch, where if you write to it, it tells Pytorch not to update the gradients using that calculation. So this is your most basic, standard SGD, stochastic gradient descent loop. So now we can answer that earlier question. The difference between stochastic gradient descent and gradient descent, is that gradient descent does not have this here that loops through each mini-batch. For gradient descent it does it on the whole dataset each time around. So train epoch or gradient descent, would simply not have the for loop at all, but instead it would calculate the gradient for the whole dataset and update the parameters based on the whole dataset, which we never really do in practice. We always use mini-batches of various sizes. Okay, so we can take the function we had before where we compare the predictions to whether that, well that, we used to be comparing the predictions to whether they were greater or less than zero, right? But now that we're doing the sigmoid, remember the sigmoid will squish everything between naught and one. So now we should compare the predictions to whether they're greater than 0.5 or not. If they're greater than 0.5 ,just to look back at our sigmoid function. So zero, what used to be zero, is now, on the sigmoid is 0.5. Okay, so we need to, just to make that slight change to our measure of accuracy. So to calculate the accuracy for some X-batch and some Y-batch, oh this is actually assumed this is actually the predictionsThen we take the sigmoid of the predictions. We compare them to 0.5 to tell us whether it's a 3 or not, we check what the actual target was, to see which ones are correct, and then we take the mean of those, after converting the booleans to floats. So we can check that accuracy. Let's take our batch, put it through our simple linear model, compare it to the four items of the training set, and there's the accuracy. So if we do that for every batch in the validation set, then we can loop through with a list comprehension, every batch in the validation set, get the accuracy based on some model, stack those all up together, so that this is a list, right? So, if we want to turn that list into a tensor, where the items of the list, of the tensor, are the items of the list. That's what stack does. So we can stack up all those, take the mean, convert it to a standard Python scalar, we're calling that item, round it to four decimal places just for display, and so here is our validation set accuracy. As you would expect, it's about 50% because it's random. So we can now train for one epoch. So we can say, remember “train_epoch” needed the parameters? So, our parameters in this case are the weights tensor and the bias tensor. So train one epoch using the “linear1” model with the learner, with a learning rate of one, with these two parameters, and then validate, and look at that! Our accuracy is now 68.8%. So we've trained an epoch. So let's just repeat that many times. Train and validate, and you can see the accuracy goes up and up and up and up and up to about 97%. So that’s coole! Ae've built an SGD optimizer of a simple linear function that is getting about 97% on our simplified MNIST where there's just the threes in the sevens.

Refactoring and Creating an optimizer

So a lot of steps there, let's simplify this through some refactoring. So the kind of simple refactoring we're going to do, we're going to do a couple, but the basic idea is, we're going to create something called an optimizer class. The first thing we'll do is, we'll get rid of the “linear1” function. But remember the “linear1” function does ‘x’ ‘@’ ‘w’ plus ‘b’. There's actually a class in Pytorch that does that equation for us, so we may as well use it. It's called nn.linear, and nn.linear does two things, it does that function for us, and it also initializes the parameters for us, so we don't have to do weights and bias init_params anymore. We just create an nn.linear class and that's going to create a matrix of size (28, 28, 1) and a bias of size 1. It will set requires_grad=True for us. It's all going to be encapsulated in this class, and then when I call that as a function, it's going to do my X hat W + B. So to see the parameters in it, we would expect it to contain 784 weights and 1 bias, we can just call that parameters and we can destructure it to w, b and see, yep! It is 784 and 1 for the weights and bias. So that's cool. So this is just, you could, you know, it could be an interesting exercise for you to create this class yourself, from scratch. You should be able to, at this point. So that you can confirm that you can recreate something that behaves exactly like nn.linear. So, now that we've got this object which contains our parameters in a parameters method, we could now create an optimizer. So if your optimizer we're going to pass it the parameters to optimize and a learning rate, we’ll store them away and we'll have something called step which goes through each parameter and does that thing we just saw: p.data -= p.grad times learning rate. And it's also going to have something called zero grad, which goes through each parameter and zeroes it out, or we could even just set it to None. So that's the thing we're going to call Basic Optimizer. So those are exactly the same lines of code we've already seen wrapped up into a class. So we can now create an optimizer, passing in the parameters of the linear model, these, and our learning rate, and so now our training loop is: look through each mini batch in the data loader, calculate the gradient, opt.step, opt.zero_grad, that's it! Validation function doesn't have to change, and so let’s put our training loop into a function, that's going to loop through a bunch of epochs, call an epoch, print validate_epoch and then run it. And it's the same! We're getting a slightly different result here, but it’s much the same idea. Okay, so that's cool, right, we've now refactoring using, you know, create our own optimizer and using PyTorch built-in nn.linear class. And you know, by the way, we don't actually need to use our own BasicOptim. Not surprisingly, PyTorch comes with something which does exactly this, and not surprisingly it's called SGD. So, and actually this SGD is provided by fastai: fastai and PyTorch provide some overlapping functionality. Then it works much the same way, so you can pass to SGD your parameters and your learning rate, just like BasicOptim, okay? And train it, and get the same result. So, as you can see, these classes that are in fastai and PyTorch, are not mysterious, they're just pretty, you know, thin wrappers around functionality that we've now written ourselves. So there's quite a few steps there, and if you haven't done gradient descent before, then there's a lot of unpacking. So, this lesson is kind of the key lesson. It's the one where, you know, like we should, you know, really take us, stop and a deep breath at this point, and make sure you're comfortable. What's the data set? What's the data loader? What's nn.linear? What's SGD? And if, you know, if one, any or all of those don't make sense, go back to where we defined it from scratch using Python code. Well the data loader we didn't define from scratch, but it, you know, the functionality is not particularly interesting. You can certainly create your own from scratch if you wanted to--that would be another pretty good exercise!

The DataLoaders class

Let's refactor some more. fastai has a ‘dataloaders’ class, which as we've mentioned before is a tiny class, that just you pass it a bunch of dataloaders and it just stores them away as a .train and a .valid. Even though it's a tiny class, it's super handy, because with that we now have a single object that knows all the data we have: and so it can make sure that your training dataloader is shuffled and your validation loader isn't shuffled, you know, make sure everything works properly. So that's what the dataloaders class is: you can pass in the training and valid dataloader.

The Learner class

And then the next thing we have in fastai is the learner class. And the learner class is something where we're going to pass in our dataloaders, we're going to pass in our model, we're going to pass in our optimization function, we're going to pass in our loss function, we're going to pass in our metrics. So all the stuff we've just done manually--that's all learner does! It's just going to do that for us. So it's just going to call this train_model and this train_epoch. It's just you know, it's inside learner. So now if we go learn.fit(), you can see again, it's doing the same thing, getting the same result. And it's got some nice functionality. It's printing it out into a pretty table for us, and it's showing us the losses and the accuracy and how long it takes. But there's nothing magic, right? You've been able to do exactly the same thing by hand using Python and PyTorch. Okay, so these abstractions are here to let you write less code and to save some time and to save some cognitive overhead, but they're not doing anything you can't do yourself. And that's important, right? Because if they're doing things you can't do yourself, then you can't customize them, you can't debug them, you know, you can't profile them. So we want to make sure that the stuff we're using is stuff that we understand what it's doing.

Adding a non-linearity to create a neural network

So this is just a linear function, it’s not great: we want a neural network. So, how do we turn this into a neural network? Remember this is a linear function x@w+B. To turn it into a neural network, we have two linear functions, exactly the same but with different weights and different biases and in between, this magic line of code, which takes the result of our first linear function and then does a max() between that and 0. So a max() of res and 0 is going to take any negative numbers and turn them into 0’s. So we're going to do a linear function, we're going to replace the negatives with 0 and then we're going to take that and put it through another linear function. That (believe it or not) is a neural net! So, w1 and w2 were weight tensors b1 and b2 are bias tensors (just like before) so we can initialize them (just like before) and we can now call exactly the same training code that we did before to roll these. So res.max(0) is called a rectified linear unit. Which you will always see referred to as ReLU. In PyTorch it already has this function--it's called f.relu(). And so if we plot it you can see it's as you'd expect, it's 0 for all negative numbers and then it's y=x for positive numbers. Here's some jargon “rectified linear unit” sounds scary, sounds complicated, but it's actually this incredibly tiny line of code, this incredibly simple function. And this happens a lot in deep learning. Things that sound complicated and sophisticated and impressive turnout to be normally super simple, frankly... At least once, you know what it is... So: Why do we do: Linear layer ReLu Linear Layer: Well if we got rid of the middle If we got rid of the middle ReLu and just went linear layer linear layer then you could rewrite that as a single linear layer when your multiply things and add and then multiply things and add andYou can just change the coefficients and make it into a single multiply and then addSo no matter how many linear layers we stack on top of each other we can never make anything moreAnd of effective than a simple linear modelBut if you put a non-linearity between the linear layersThen actually you have the opposite. This is now where something called the universal approximation theorem holds which is that if the size of the weight and bias matrices are big enoughThis can actually approximate any arbitrary function including the function of how do I recognize threes from sevens or Or whateverSo that's kind of amazing, right this tiny thing is actually a universal function approximator as long as you have W1 B1 W2 and B2Have the right numbers and we know how to make them the right numbers we use SGDCould take a very long time. It could take a lot of memoryBut the basic idea is that there is some solution to any computable problem andThis is one of the biggest challengesA lot of beginners have to deep learning is that there's nothing else to it like that? There's often this likeOkay, how do I make a neural net?Oh, that is the neural net. Well, how do I, do deep learning training with SGDthere's things to like Make a train a bit faster. There's you know things to mean you need a few less parameters but everything from here is justPerformance tweaks honestly, rightSo this is you know, this is the key understanding of of training a neural networkOkay, we can simplify things a bit more We already know that we can use nn.linear to replaceTheir weight and bias, so let's do that for both of the linear layers, and then since we're simply takingThe result of one function and passing it into the nextThe result of that function passive to the next and so forth and then returned the end this is called function composition function composition is when you justTake the result of one function pass it to a new one take a result of one function. Pass it to a new one and so every pretty much neural network is just doing function composition of linear layers and these are called activation functions or nonlinearities So PyTorch provides something to do function composition for us and it's called nn.sequential so it's gonna do a linear layer pass the result to a ReLu you pass the result to a linear layerYou'll see here. I'm not using F.ReLU. I'm using nn.ReLU This is identical returns exactly the same thing, but this is a classRather than a function. Yes, Rachel “By using the non-linearity Won't using a function that makes all negative output zero make many of the gradients in the network zero and stop the learning process dueto many zero gradients?” Well, that's a fantastic question and the answer is yes, it doesBut they won't be zero for every image and remember the mini-batches a shuffled soEven if it's zero for every image in one mini batch, it won't be for the next mini batchAnd it won't be the next time around we go for another epoch. So Yes, it can create zeros and ifThe neural net ends up with a set of parametersThat's that lots and lots of inputs end up as zeros. You can end up with whole mini batches that are zero andYou can end up in a situation where some of the neurons remain In active inactive means their zero and they're basically dead units And this is a huge problemIt basically means you're wasting computationSo there's a few tricks to avoid that which we'll be learning about a lot one. Simple trick is toNot make this thing flat here, but just make it a less steepMine that's called a leakyReLU. Well, you leaky rectified linear unit andIt that they helped a bitAs well learn though even better is to make sure that we just kind of initialize to sensible initial values that are not too big and not too small and step by sensible initial values that are particularly not too big and generally if we do that we can keep things in the zone where they're positive most of the time but we are going to learn about how to actually analyze inside a network and find out how many dead units we have how many of these zeros we have becauseAs is as you point out they are they are bad news. They don't do any work and they will Continue to not do any work if enough of the inputs end up being zeroOkay, so now that we've got a neural netWe can use exactly the same learner we had before but this time we pass in the simple netInstead of the linear one. Everything else is the same and we can call fit just like before andGenerally as your models get deeper. So here we've gone from one layer-twoAnd I'm only counting the parameterised layers as layers. You could say it's three. I was going to call it two. There's twoTrainable layers. So I've gone from one layer to I've checked dropped my learning rate from 1 to 0.1because the deeper models all tend to be kind of bumpier less nicely behaved so often you need to use lower learning ratesAnd so we trained it for awhile okay, andCan actually find out what that training looks like by looking inside our learner and there's an attribute we create for recorder andThat's going to recordWell everything that appears in this tableBasically, well these three things the training loss the validation loss and the accuracy or any metricsso recorded values contains that kind of table of results and so item number two ofEach row will be the accuracy and so the capital L class, which I'm using here as a nice little method called itemgot that will will getThe second item from every row and then I can plot thatHow the training went and I can get the final accuracyBy grabbing the last row of the table and grabbing the second It's indexed to 0 1 2 then my final accuracy.

Looking at what the NN is learning by looking at the parameters

Not bad98.3%So this is pretty amazing, we now have a function that can solve any problemTo any level of accuracy if we can find the right parameters and we have a way to findHopefully the best or at least a very good that our parameters for any functionSo this is kind of the magic yes, RachelHow could we use what we're learning here to get an idea of what the network is learning along the wayLike Zieler and Fergus did more or lessWe will look at that laterNot in the full detail of their paper but basically you can look in the dot parameters to see the values of those parametersand at this point. Well, I mean, why don't you try it yourself? Right? You've actually got nowThe parameters, so if you want to grab the model you can actually see the learned.modelSo we can we can look inside learn.model to see the actual model that we just trained and you can see it's got the three things in it. They're linear then ReLU than linear, and you know, what I kind of like to do is to put that into a variable, make it a Bit easy to work with, you can grab one layer by indexing in parameters and that just gives me something called a generator. It's something that will give me a list of the parameters when I ask for them. I could just go weight comma bias equals to de-structure them and so the weight Here's 30 by 784: because that's what I asked for. So one of the things to note here is that to create a Neural Net so something that's more than one layer. I actually have 30 outputs not just one right so I'm kind of generating lot so if you can think of as generating lots of featuresSo it's kind of like 30 different linear of linear models here and then I combined those 30 back into one. So you could look at one of those by having a look at here, so there's the numbers in the first row, we could reshape that into the the original shape of the images and we could even have a look and there it is right? So you can see this is something So this cool right we can actually see here we've got something which isWhich is kind of learning to find things at the top and the bottom and the middle And so we could look at the second one. Okay, no idea what that's showing and so some of them are kind of you know, I probably got far more than I need which is why they're not that obvious. But you can see yeah, here's another thing that's looking pretty similar kind of looking for this little bit in the middle, so yeah, this is the basic idea to understand the features that are not the first layer but later layers, you have to be a bit more sophisticated but yeah to see the first layer ones you can you can just plot themOkay, so then, you know just to compare we could use the full fastai toolkit so grab our data loaders by using data loaders from folder as we've done before and create a cnn_ learner and a ResNet and fit it for a single epoch and, WOAH, 99.7!

Comparing the results with the fastai toolkit

All right, so we did 40 epochs and got 98.3 as I said using all the tricks you can really speed things up and make things a lot better and so by the end of this course or at least both parts of this course, you'll be able to from scratch at this 99.7 in a single epoch, all right, so Jargon!

Jargon review

So jargon: just remind us ReLU function that returns zero for negatives mini-batch a few inputs and labels, which optionally are randomly selected the forward pass is the bit where we calculate the predictions the loss is the function that we're going to take the derivative of and then the gradient is the derivative of the loss with respect to each parameter the backward pass is when we calculate those gradients gradient descent is that full thing of taking a step in the direction opposite to the gradients by capital after calculating the loss andThen the learning rate is the size of the step that we take Other things to know, perhaps the two most important pieces of jargon are all of the numbers that are in a neural network the numbers that we're learning are called parameters and then the numbers that we're calculating so every value that's calculated every matrix multiplication element that's calculated: They're called activations so activations and parameters are all of the numbers in the neural net and so be very careful when I say from here on in in these lessons activations or parameters. You've got to make sure you know what those mean because that's that's the entire basically almost the entire set of numbers that exist inside a neural net so activations are calculated, Parameters are learned. We're doing this stuff with Tensors and Tensors are just regularly shaped to arrays, rank zero tensors, we call scalars, rank 1 tensor:. we call vectors rank two tensors we call matrices and we continue on to rank 3 tensors rank 4 tensors and so forth and rank five tensors are very common in deep learning. So don't be scared of going up to higher numbers of dimensions.

Is there a rule of thumb for which non-linearity to choose?

Okay, so let's have a break oh we got a question, okay R: “Is there a rule of thumb for what non-linearity to choose given that there are many?” Yeah, there are many non-linearities to choose from and it doesn't generally matter very much which you choose so just use ReLU or Leaky ReLU or yeah, whatever any anyone should work fine later on we'll we'll look at the minor differences between between them but it's not so much something that you pick on a per problem it's more like some take a little bit longer and a little bit more accurate and some over it faster and a little bit less accurate. That's a good question, okay. So before you move on it's really important that you finish the questionnaire for this chapter because there's a whole lot of concepts that we've just done so, you know try to go through the questionnaire go back and relook at the notebook and please run the code through the cat experiments and make sure it makes senseAll right. Let's have a seven minute break see you back here in seven minutes time.

Pet breeds image classification

Okay, welcome back, so now that we know how to create and train a Neural Net. Let's cycle back and look deeper at some applications. And so we're going to try to kind of interpolate in from one end we've done they're kind of from scratch version at the other end we've done the kind of four lines of code version and we're going to gradually nibble at each end until we find ourselves in the middle and we've we've we've touched on all of itso let's go back up to the kind of the four lines of code version and and delve a little deeper. So, let's go back to PETs and let's think though about like: How do you actually, start with a new dataset and figure out how to use it so, you know the the data sets we provide it's easy enough to untar them you to say untar that will download it and untar it. If it's a data set that you're getting you can just use the terminal or [? ]a Python or whatever, so, let's assume we have a path that's pointing at something so initially you don't you don't know what that something is, so we can start by doing LS to have a look and see what's inside there. So the PETs data set that we saw in Lesson one contains three things annotations images and models and you'll see we have this little trick here where we say path.BASE_ path equals and then the path to our data and that just does a little simple thin:. Where when we print it out, it just doesn't show us. It just shows us relative to this path, which is a bit convenient. So, go and have a look at the readme for the original PETs dataset, it tells you what these images and annotations folders are and not surprisingly the images path, so if we go path slash images, that's how we use PathLib to grab the sub directory and then LS we can see here are the names that the paths through the images. As it mentions here most functions and methods in fastai which returned a collection don't return a Python list that they returned a capital L and a capital L as we briefly mentioned is basically an enhanced list. One of the enhancements is the way it prints the representation of it starts by showing you. How many items there are in the list in the collection: so there's 7349 images and, It it if there's more than ten things it truncates it and just says dot dot to avoid filling up your screen, so there's a couple of little conveniences there, and so we can see from this output that a file name as we mentioned in lesson 1 if the first letter is a capital it means it's a Cat and if the first letter is lowercase it means it's a dog, but this time we've got to do something a bit more complex a lot more complex which is figure out what breed it is and so you can see the breed is kind of everything up to after the in the file name: It's everything up to the the last underscore and before this number is the breed. So we want to label everything with its breed, so we're going to take advantage of this structure, so the way I would do this is to use a regular expression. A regular expression is something that looks at a string and basically lets you kind of pull it apart into its pieces in very flexible way. It is kind of simple little language for doing that. Um, if you haven't used regular expressions before um, please Google regular expression tutorial now and look it's going to be like one of the most useful tools you'll come across in your life. I use them almost every day .I will go to details about how to use them since there's so many great tutorials. And there's also a lot of great like exercises, you know, there's regex regex is short for regular expression. There's regex crosswords, There's reges Q&A all kinds of core regex things a lot of people like me love this tool in order to, there's also a regex lesson in the fastAI NLP course, maybe even to regex lessons. Oh, yeah, I'm sorry for forgetting about the first day. I know because, what an excellent resource that is! So, RegularExpressions are how to get right the first time. So the best thing to do is to get a sample string. So good - good way to do that would be to just grab one of the file names. So let's pop it in Fname and then you can experiment with reg expressions. So re is the regular expression module in Python and find all will just grab all the parts of a regular expression that have parentheses around them. So this regular expression and R is a special kind of string in Python which basically says don't treat backslash as special because normally in Python like backslash n means a newline. So here's us a string, which I'm going to capture. Any letter one or more times followed by an underscore followed by a digit one or more times, followed by anything I probably don’t have to use backslash t for this but that’s fine followed by the letters jpg followed by the end of the string and so if I call that regular expression against my file names name, Oh! Looks good, right so we kind of check it out! So, now that seems to work we can create a data block where the independent variables are images the dependent variables are categories just like before get items is going to be get image files we're going to spit it randomly as per usual and then we're going to get the label by calling regex labeler, which is a just a handy little fastai class which labels things with a regular expression.

Presizing

We can't call the regular expression this particular regular expression directly on the path lib path object we actually want to call it on the name attribute and fast AI has a nice little function called using attr using attribute which takes this function and changes it to a function which will be passed this attribute that's going to be using regex labeler on the name attribute and then from that data block we can create the data loaders as usual there's two interesting lines here resize and aug_transforms() aug_transforms() we have seen before in notebook 2, in the section core data augmentation and so aug_transforms() was the thing which can zoom in and zoom out, and warp, and rotate and change contrast and change brightness and so forth and flip, to kind of give us almost, It's like giving us more data being generated synthetically from the data. We already have and we also learned about random resize crop: which is a kind of a really cool way of getting, ensuring you get square images at the same time that you're augmenting the data here, we have a resize to a really large image but you know by deep learning standards 460x460 is a really large image and then we're using aug_transforms() with a size. So that's actually going to use random resize crop to a smaller size Why are we doing that? This particular combination of two steps does something which I think is unique to Fastai which we call pre-sizing. And the best way is, I will show you this beautiful example of Powerpoint wizardry that I'm so excited about, to show how pre-sizing works. What pre-sizing does, is that the first step where we say resize to 460 by 460 is, it grabs a square, and it grabs it randomly. If it's a kind of landscape orientation photo, it'll grab it randomly. So it'll take the whole height and randomly grab somewhere from along the side. If it's a portrait orientation, then it will grab it, you know, take the full width and grab a random bit from top to bottom. So then we take this area here, and here it is, right? And so that's what the first resize does. And then the second aug_transforms bit, will grab a random warped crop, possibly rotated, and will turn that into a square. So there's two steps, it’s first of all resize to a square that's big, and then the second step, is to a kind of rotation and warping and zooming stage to something smaller, in this case 224 by 224. Because this first step creates something that's square, and always is the same size, the second step can happen on the GPU. Normally, things like rotating and image warping are actually pretty slow. Also, normally doing a zoom and rotate and a warp actually is really destructive to the image because each one of those things requires an interpolation step. Which it's not just slow, it actually makes the image really low quality. So we do it in a very special way in Fastai. I think it's unique, where we do all of the all of these kind of coordinate transforms like rotations and warps and zooms and so forth, not on the actual pixels, but instead we kind of keep track of the changing coordinate values in a in a non-lossy way, so the full floating-point value, and then once at the very end, we then do the interpolation. The results are quite striking. Here is what the difference looks like. Hopefully you can see this on the video. On the left is our pre-sizing approach, and on the right is the standard approach that other libraries use. And you can see that the one on the right is a lot less nicely focused, and it also has weird things like this should be grass here, but it's actually got its bum sticking way out. This has a little bit of weird distortions, this has got loads of weird distortions. So you can see the pre-sized version really ends up way way better and I think we have a question, Rachel. Are the blocks in the DataBlock an ordered list? Do they specify the input and output structures respectively? Are there always two blocks or can there be more than two? For example, if you wanted a segmentation model, would the second block be something about segmentation? So, yeah, this is an ordered list. So the first item says I want to create an image, and then the second item says I want to create a category. So that's my independent and dependent variable. You can have one thing here, you can have three things here, you can have any amount of things here you want. Obviously the vast majority of the time it'll be two only: there's an independent variable and a dependent variable. We'll be seeing this in more detail later, although if you go back to the earlier lesson when we introduced DataBlocks, I do have a picture, kind of, showing how these pieces get together.

Checking and debugging a DataBlock

So, after you've put together DataBlock, created your DataLoaders, you want to make sure it's working correctly. So the obvious thing to do for a computer vision DataBlock is show_batch and show_batch will show you the items, and you can kind of just make sure they look sensible, that looks like the labels are reasonable. If you add a unique=True, then it's going to show you the same image with all the different augmentations. This is a good way to make sure your augmentations work. If you make a mistake in your DataBlock, in this example, there's no resize, so different images are going to be different sizes or be impossible to collate them into a batch. So if you call ‘.summary’, this is a really neat thing, which will go through and tell you everything that's happening. So I... Collecting the items. How many did I find? What happened when I split them? What are the different variables, independent, dependent variables I’m creating. Let's try and create one of these. Here’s each step. Create my image. Create categorize. Here’s what the first thing gave me. An American Bulldog. Here’s the final sample. Is this image, this size, this category. And then eventually it says oh, oh, it's not possible to collate your items. I tried to collate these zero index members of your tuples. So in other words, that's the independent variable and I got, this was size 500 by 375, this was 375 by 500. Oh, I can't collate these into a tensor because they're different sizes. So this is a super great debugging tool for debugging your DataBlocks.

Presizing (question)

We have a question. How does the item transforms presize work if the resize is smaller than the image? Is a whole width or height still taken, or is it just a random crop with the revised value? So if you remember back to Lesson 2, we looked at the different ways of creating these things you can use squish, you can use pad, or you can use crop. So if your image is smaller than the precise value, then squish will really be zoom, so it will just small stretch. It'll stretch it, and then pattern crop will do much the same thing. And so you'll just end up with a, you know, the same. This looks like these, but it'll be a, kind of, lower, more pixelated, lower resolution because it's having to zoom in a little bit.

Training model to clean your data

Okay, so a lot of people say that you should do a hell of a lot of data cleaning before you model. We don't. We say model as soon as you can, because remember what we found in, in Notebook 2. Your, your model can teach you about the problems in your data. So as soon as I've got to a point where I have a DataBlock, that's working, and I have DataLoaders, I'm going to build a model. And so here I'm, you know, it also tells me how I'm going. So, I'm getting 7% error. Wow, that's actually really good for a pets model. And so at this point now that I have a model I can do that stuff we learned about earlier, in 02, the Notebook 02, where we trained our model, and used it to clean the data. So we can look at the classification, a confusion matrix, top losses, the image cleaner widget, you know, so forth.

How fastai chooses a loss function

Okay, now one thing interesting here is in Notebook 4 we included a loss function, when we created a Learner, and here we don't pass in our loss function. Why is that? That's because fastAI will try to automatically pick a somewhat sensible loss function for you. And so for a image classification task it knows what loss function is the normal one to pick, and it's done it for you, but let's have a look and see what it actually did pick. So we could have a look at ‘learn.loss_func’ and we will see it is cross-entropy loss. What on earth is cross-entropy loss. I'm glad you asked. Let's find out. Cross entropy loss is really much the same as the MNIST loss we created with that, with that, sigmoid and the one minus predictions and predictions, but it's, it's a, kind of, extended version of that. And the extended version of that, is that, that ‘torch.where’ that we looked at in Notebook 4, only works when you have a binary outcome. In that case it was: ‘Is it a three or not?’ But in this case we've got which of the thirty-seven pet breeds is it? So, we want to, kind of, create something just like that sigmoid and ‘torch.where’, that which also works nicely for more than two categories. So, let's see how we can do that, so first of all, let's grab a batch. There is a... Yes? There is a question. Why do we want to build a model before cleaning the data? I would think a clean dataset would help in training. Yeah, absolutely a current clean dataset helps in training, but remember as we saw in notebook 02, an initial model helps you clean the dataset. So remember how ‘plot_top_losses’ helped us identify mislabeled images, and the confusion matrix helps us recognize which things we were getting confused, and might need, you know, fixing and the ‘ImageClassifierCleaner’ actually let us find things like, an image that contained two bears, rather than one bear, and cleaned it up. So a model is just a fantastic way to help you zoom in on the data that matters, which things seem to have the problems, which things are most important. Stuff like that. So you would go through and you clean it, with the model helping you, and then you go back and train it again, with the clean data. Thanks for the great question. Okay, so in order to understand cross-entropy loss let's grab a batch of data, which we can use ‘dls.one_batch’, and that's going to grab a batch from the training set. We could also go first(dls.train) and that's going to do exactly the same thing. And so then we can destructure that into the independent, dependent variable, and so the dependent variable shows us we've got a batch size of 64. So it shows us the 64 categories. And remember those numbers simply refer to the index of, into the vocab. So for example 16 is a boxer. And so that all happens for you automatically, when we say ‘show_batch’, it shows us those strings. So here’s a first mini-batch, and so now we can view the predictions, that is the activations of the final layer of the network, by calling ‘get_prieds’. And you can pass in a DataLoader, and a DataLoader can really be anything that's going to return a sequence of mini batches. So we can just pass in a list, containing our mini batch, as a DataLoader, and so that's going to get the predictions for one mini batch. So here's some predictions. Okay, so the actual predictions, if we go ‘preds[0].sum’, to grab the predictions for the first image, and add them all up, they add up to one. And there are 37 of them. So that makes sense. Right? It's like the very first thing is, what is the probability that that is a ‘dls.vocab’, the first thing is what's the probability it's an Abyssinian cat. It's ten to the negative six. You see? And so forth. So it's basically like it's not this, it's not this, it's not this, and you can look through and, oh here this one here, you know, obviously what it thinks it is. So how did it? You know, so we... We obviously want the probabilities to sum to one, because it would be pretty weird if, if they didn't. It would say, you know, that the, the probability of being one of these things is more than 1 or less than 1, which would be extremely odd. So how do we go about creating these predictions, where each one is between zero and one, and they all add up to 1. To do that we use something called softmax. Softmax is basically an extension of sigmoid, to handle more than two levels, two categories. So remember the sigmoid function looked like this. We used that for our 3s vs. 7s model. So what if we want 37 categories, rather than two categories. We need one activation for every category. So actually the threes and sevens model, rather than thinking of that as an ‘is-3’ model, we could actually say: ‘Oh that has two categories, so let's actually create two activations. One representing how three like something is, and one representing how seven like something is.’ So let's say, you know, let's just say that we have 6 MNIST digits and these were the... Can I do this? And this first column were the activations of my model for, for one activation, and the second column was for a second activation. So my final layer actually has two activations now. So this is like how much like a 3 is it? And this is how much like a 7 is it? But this one is not at all like a 3, and it's slightly not like a seven. This is very much like a three, and not much like a seven, and so forth. So we can take that model, and rather having, rather than having one activation for like, is three, we can have two activations for how much like a three, how much like a seven. So if we take the sigmoid of that, we get two numbers between naught and one, but they don't add up to one. So that doesn't make any sense. It can't be 0.66 chances of three , and 0.56 chances of seven, because every digit in that data set is only one, or the other. So that's not going to work, but what we could do is we could take the difference between this value, and this value and, say that's how likely it is to be a three. So in other words this one here, with a high number here, and a low number here, is very likely to be a three. So we could basically say in the binary case, these activations, that what really matters is their relative confidence of being a three versus a seven. So we could calculate the difference between column one and column two, or column index zero and column index one, right? And here's the difference between the two columns, there's that big difference, and we could take the sigmoid of that. Right? And so this is now giving us a single number between naught and one, and so then, since we wanted two columns, we could make column index zero, the sigmoid, and column index one, could be one minus that, and now look these all add up to one. So here's probability of three, probability of seven, but the second one, probably three, probability of seven, and so forth. So like that's a way that we could go from having two activations for every image, to creating two probabilities, each of which is between naught and one, and each pair of which adds to one. Great. How do we extend that to more than two columns? To extend it to more than two columns we use this function, which is called softmax. Softmax is equal to e to the x, divided by the sum of e to the x. Just to show you if I go softmax on my activations, I get 0.6025, 0.3975, 0.6025, 0.3975, I get exactly the same thing. Right? So softmax in the binary case, is identical to the sigmoid that we just looked at. But in the multi category case, we basically end up with something like this. Let's say we were doing the teddy bear, grizzly bear, brown bear, and for that, remember, our neural net is going to have the final layer, will have three activations. So let's say it was 0.02, -2.49, 1.25. So to calculate softmax I first go e to the power of each of these three things, so here's e to the power of .02, e to the power of -2.49, e to the power of 3.4, e to the power of 1.25. Ok, then I add them up so there's the sum of the exps and then softmax will simply be 1.02 divided by 4.6 and then this one will be 0.08 divided by 4.6. And this one will be 3.49 divided by 4.6 so since each one of these represents each number divided by the sum, that means that the total is one. Okay, and because all of these are positive and each one is an item divided by the sum it means all of these must be between naught and one. So this shows you that softmax always gives you numbers between naught and 1 and they always add up to 1. That in practice you can just call torch dot softmax. And it will give you this result of this, this function. So you should experiment with this in your own time, you know, write this out by hand and try putting in these numbers, right, and, and see how that you get back the numbers I claim you're going to get back and make sure this makes sense to you. So one of the interesting points about softmax is, remember I told you that exp is e to the power of something, and now what that means is that e to the power of something grows very very fast. Right? So like exp of 4 is 54, exp of 8 is 29, 2980, right. It grows super fast and what that means is that if you have one activation that's just a bit bigger than the others, its softmax will be a lot bigger than the others. So intuitively the softmax function really wants to pick one class among the others. Which is generally what you want, right, when you're trying to train a classifier to say which breed is it. You kind of want it to to pick one and kind of go for it, right? And so that's what softmax does. That's not what you always want. So sometimes at inference time you want it to be a bit cautious. And so you kind of got to remember that softmax isn't always the perfect approach but it's the default. It's what we use most of the time and it works well on a lot of situations. So that is softmax. Now in the binary case for the MNIST three versus sevens, this was how we calculated the MNIST loss, we took the sigmoid and then we did either one minus that or that as our loss function. Which is fine as you saw it, it worked, right? And so we could do this exactly the same thing. We can't use torch dot where anymore because targets aren't just zero or one, targets could be any number from naught to 36. So we could do that by replacing the torch dot where with indexing. So here's an example for the binary case. Let's say these are our targets 0 1 0 1 1 0 and these are our softmax activations which we calculated before, they’re just from some random numbers, just for a toy example. So one way to do instead of doing torch dot where, we could instead, have a look at this, I could say I could grab all the numbers from naught to 5 and if I index into here With all the numbers from 0 to 5 and then my targets, 0 1 0 1 0 1 1 0 then what that's going to do is it's going to pick a row 0 it'll pick 0.6. And then for row 1 it'll pick 1, a 0.49. for row 2, it'll pick 0, a .13, for row 4 it'll pick 1, a .003 and so forth. So this is a super nifty indexing expression which you should definitely play with, right, and it's basically this trick of passing multiple things to the pytorch indexer. The first thing says, which rows should you return; and the second thing says, for each of those rows, which column should you return? So this is returning all the rows and these columns, for each one and so this is actually identical to torch dot where. Or isn't that tricky? And so the nice thing is we can now use that for more than just two values. And so here's, here's the fully worked out thing, so I've got my threes column, I've got my sevens column, here's that target, here’s the indexes from naught one, two, three, four five. And so here's 0, 0, .6; 1, 1, .49; 0, 2, .13, and so forth. So yeah this works just as well with more than two columns. So we can add, you know, for doing a full MNIST, you know, so all the digits from naught to nine. We could have ten columns and we would just be indexing into the ten. So this thing we're doing where we're going minus our activations matrix, all of the numbers from naught to N and then our targets, is exactly the same as something that already exists in pytorch called F dot nll_loss as you can see. Exactly the same. And so again, we're kind of seeing that these things inside pytorch and fastAI are just little shortcuts for stuff we can write ourselves. Nll_loss stands for negative log likelihood, again, it sounds complex, but actually it's just this indexing expression. Rather confusingly, there's no log in it. We'll see why in a moment. So let's talk about logs. So this loss, this loss function works quite well as we saw in the notebook 04. It's basically this, it is exactly the same as we learned in notebook 04, just a different way of expressing it, but we can actually make it better. Because remember the probabilities we're looking at are between naught and one so they can't be smaller than zero. They can't be greater than one, which means that if our model is trying to decide whether to predict .990 or .999 , it's going to think that those numbers are very very close together, but won't really care. But actually if you think about the error, you know if there's like a hundred things, a thousand things, then this would like be ten things are wrong and this would be like one thing is wrong. But this is really like ten times better than this so really, what we'd like to do is to transform the numbers between zero and one to instead between, be between negative infinity and infinity. And there's a function that does exactly that which is called logarithm. Okay, so, as the, so the numbers we could have can be between zero and one and as we get closer and closer to zero it goes down to infinity and then at one, it's going to be zero and we can't go above zero because our loss function we want to be negative. So, this logarithm, in case you forgot, is, hopefully you vaguely remember what logarithm is from high school, but that basically the definition is this: if you have some number that is y that is b to the power of a Then logarithm is defined such that a equals the logarithm of y comma b. In other words it tells you b to the power of what equals y. Which is not that interesting of itself but one of the really interesting things about logarithms is this very cool relationship, which is that log of a times b equals log of a plus log of b. And we use that all the time in deep learning and machine learning because this number here a times b can get very very big or very very small. If you multiply things, a lot of small things together, you'll get a tiny number, if you multiply a lot of big things together, you'll get a huge number. It can get so big or so small that the kind of the precision in your computer's floating point gets really bad, whereas this thing here adding is not going to get out of control. So we really love using logarithms like particularly in a deep neural net where there's lots of layers, we're kind of multiplying and adding many times, though, this kind of tends to come out quite nicely. So when we take the probabilities that we saw before, the things that came out of this function, and we take their logs and we take the mean, that is called negative log likelihood, and so this ends up being kind of a really nicely behaved number because of this property of the log that we described. So if you take the softmax and then take the log, then pass that to an nll_loss because remember that we didn't actually take the log at all despite the name, that gives you cross entropy loss. So that leaves an obvious question of why doesn't nll_loss actually take the log and the reason for that is that it's more convenient computationally to actually take the log back at the softmax step. So pytorch has a function called log_softmax so since it's actually easier to do the log at the softmax stage, it's just faster and more accurate. Pytorch assumes that you use soft log max and then pass that to nll_loss. so nll_loss does not do the log. It assumes that you've done the log beforehand. So log_softmax followed by nll_loss is the definition of cross-entropy loss in pytorch. So that's our loss function and so you can pass that some activations and some targets and get back a number and pretty much everything in pytorch every every one of these kinds of functions, you can either use the NN version as a class like this and then call that object as if it's a function, or you can just use F dot with the camelcase name as a function directly and as you can see, they're exactly the same number. People normally use the class version in the documentation in pytorch, you'll see it normally uses a class version so we tend to use the class version as well. You'll see that it's returning a single number and that's because it takes the mean because a loss needs to be as we've discussed the mean but if you want to see the underlying numbers before taking the mean you can just pass in reduction=none and that shows you the individual cross-entropy losses before taking the mean. Okay, great, so this is a good place to stop with our discussion of loss functions and such things. Rach, were there any questions about this? Why does the loss function need to be negative? Well, okay, I mean I guess it doesn't but it's we want something that the lower it is, the better, and we kind of need it to cut off somewhere. I have to think about this more during the week because I'm, it's a bit tough, I’m a bit tired. Yeah, so let me let me refresh my memory when I'm awake Okay.

Data Ethics and Efficacy of Masks for COVID-19

Now next week ... well, nope not for the video. Next week actually happened last week so it's the thing I'm about to say is actually your. So next week we're going to be talking about data ethics, and I wanted to kind of segue into that by talking about how my week’s gone, because a week or two ago I did as part of a lesson I actually talked about the efficacy of masks. I mean specifically wearing masks in public and I pointed out that the efficacy of masks seemed like it could be really high and maybe everybody should be wearing them. And somehow I found myself as the face of a global advocacy campaign. And so if you go to masks4all.co, you’ll find a website talking about masks. And I've been on, you know, TV shows in South Africa and the US and England and Australia and on radio and blah blah blah talking about masks. Why is this? Well, it's because as a data scientist, you know, I noticed that the data around masks seemed to be getting misunderstood and it seemed that that misunderstanding was costing possibly hundreds of thousands of lives. You know, literally in the places that were using masks it seemed to be associated with orders of magnitude fewer deaths and one of the things to talk about next week is like, you know, what's your role as a data scientist. And, and you know, I strongly believe that it's to understand the data and then do something about it. And so nobody was talking about this So I ended up writing an article that appeared in The Washington Post that basically called on people to really consider wearing masks (which is this article). And, you know I was, I was lucky, I managed to kind of get a huge team of brilliant (not, not a huge, but a pretty decent-sized team of brilliant) volunteers who helped, you know, kind of build this website and kind of some PR folks and stuff like that. But what became clear was, and I was talking to politicians, you know, senators, and staffers, and what was becoming clear is that people weren't convinced by the science, which is fair enough because it's, it's hard to. You know when the WHO and the CDC is saying you don't need to wear a mask and some random data scientist is saying but doesn't seem to be what the data is showing. You know, you've got half a brain you would pick the WHO and the CDC not the random data scientist. So I really felt like I, if I was going to be an effective advocate, I needed to sort the science out. And you, know credentialism is strong. And so it wouldn't be enough for me to say it. I needed to find other people to say it. So I put together a team of 19 scientists, Including you know a professor of sociology, a professor of aerosol dynamics, the founder of an African movement that's that kind of studied preventive methods for methods for tuberculosis, a Stanford professor who studies mask disposal and cleaning methods, a bunch of Chinese scientists who study epidemiology modeling A UCLA professor, who is one of the top Infectious disease epidemiologist experts, and so forth. So like this kind of all-star team of people from all around the world, and I had never met any of these people before so (well, no not quite true, I knew Austin a little bit and I knew Zeynep a little bit, and Lex a little bit). But on the whole you know (and well Reshama, we all know she's awesome. So it's great to actually have a fast.ai community person there too. And, so, but yeah, I kind of tried to pull together people from you know, as many geographies as possible and as many areas of expertise as possible. And you know the kind of the global community helped me find papers about, about everything. About, you know, how different materials work, about how droplets form, about epidemiology, about case studies of people infecting with and without masks, blah blah blah. And we ended up in the last week; basically we wrote this paper. It contains 84 citations. And you know, we basically worked around the clock on it as a team, and it's out. And it's been sent to a number of, some of the earlier versions 3 or 4 days ago we sent to some governments. So one of the things is in this team. I try to look for people who were working closely with government leaders, not just that they're scientists. And so this, this went out to a number of government ministers. And in the last few days, I've heard that it was a very significant part of decisions by governments to change their, to change their guidelines around masks. And you know the fight’s not over by any means, and in particular the UK is a bit of a holdout. But I'm going to be on ITV tomorrow and then BBC the next day. You know, it's it's kind of required stepping out to be a lot more than just a data scientist. So I've had to pull together, you know politicians and staffers. I've had to, you know, you know , hassle with the media to try and get you know coverage. And you know today I'm now starting to do a lot of work with unions to try to get unions to understand this You know, it's really a case of like saying - okay as a data scientist, and in conjunction with real scientists, we've built this really strong understanding that masks, you know this simple, but incredibly powerful tool. That doesn't do anything unless I can effectively communicate this to decision-makers. So today I was you know on the phone to, you know, one of the top union leaders in the country, explaining what this means. Basically it turns out that in buses in America, the kind of the air conditioning is set up so that it blows from the back to the front. And there's actually case studies in the medical literature of how people that are seated downwind of an air conditioning unit in a restaurant ended up all getting sick with Covid 19. And so we can see why like bus drivers are dying. Because they're like, they're right in the wrong spot here and their passengers aren't wearing masks. So I tried to unexplained this science to union leaders so that they understand that to keep the workers safe it's not enough just for the driver to wear a mask, but all the people on the bus needed to be wearing masks as well. So, you know all this is basically to say ,,, you know as data scientists, I think we have a responsibility to study the data and then do something about it. It's not just a research exercise, it's not just a computation exercise, you know. What, what's the point of doing things if it doesn't lead to anything? So, yeah, so, next week. We'll be talking about this a lot more, but I think you know - this is a really to me kind of interesting example of how digging into the data can lead to really amazing things happening. And, and in this case, I strongly believe, and a lot of people are telling me they strongly believe that this kind of advocacy work that's come out of this data analysis is, is already saving lives. And so I hope this might help inspire you to, to take your data analysis and to take it to places that it really makes a difference. So thank you very much, and I'll see you next week.