I did my doctorate in the Met Office.

Weather forecasting is two separate problems. The first of these is physics - given the state of the atmosphere right now, what will it do. And this is hard, because there are so many different effects, combined with the fact that our computational models have a limited resolution. There's a huge amount of work that goes into making the simulation behave like a real atmosphere does, and a lot of that is faking what is going on at a smaller scale than the model grid.

The second part is to work out what the current state of the atmosphere is. This is what takes vast amounts of computing power. We don't have an observation station at every grid point and at every altitude in the atmospheric model, so we need to find some other way to infer what the atmospheric state is from the observations that we can from it. Many of these observations are limited in locality, like weather stations, or are a complex function of the atmospheric state, like satellite imagery. The light reaching a satellite has been affected by all the layers of the atmosphere it passes through, and sometimes in a highly nonlinear way. In order to calculate the atmospheric state, we need to take the previous forecast of the current atmospheric state, compare it to the observations, then find the first derivative (as in calculus) of the observation function so that we can adjust the atmospheric state estimate to the new best estimate. This is then complicated by the fact that the observations were not all taken at a single time snapshot - for instance polar orbiting satellites will be taking observations spread out in time. So, we need to use the physics model to wind the atmospheric state back in time to when the observation was taken, find the first derivative of that too, and use it to reconcile the observations with the atmospheric state.

It's a massive minimisation/optimisation problem with millions of free variables, and in some cases we need the second derivative of all these functions too in order to make the whole thing converge correctly and within a reasonable amount of time. It takes a reasonable number of iterations of the minimisation algorithm to get it settle on a solution. The problem is that these minimisation methods often assume that the function being minimised is reasonably linear, which certain atmospheric phenomena are not (such as clouds), so certain observations have to be left out of the analysis to avoid the whole thing blowing up.

My doctorate was looking to see if the nonlinearity involved in a cloud forming as air was moving upwards could be used to translate a time-series of satellite infra-red observations into a measurement of vertical air velocity. The answer was that this single form of nonlinearity made the whole minimisation process fairly dire. I implemented a fairly simple not-quite-machine-learning approach, and it was able to find a solution that was almost as accurate but much more reliable than the traditional minimisation method.

Also, to answer the dead sibling comment asking whether weather is really a chaotic system - yes it is. The definition of a chaotic system is that a small change in current state results in a very large change in outcome, and that's definitely the case. The improvements in weather forecasting over the last few decades have been due to improvements in solving both of the above problems - the physics has been pinned down better, but we're also better as working out the current atmospheric state fairly accurately, and that has added something like a day of forecasting accuracy each decade we have been working on it.