Daniel Mminele: Forecasts, errors and what we can learn from them Keynote address by Mr Daniel Mminele, Deputy Governor of the South African Reserve Bank, at the Reuters Economist of the Year Award, Johannesburg, 5 August 2011. * 1. * * Introduction Good morning ladies and gentlemen. Thank you to Thomson Reuters for inviting me to address you this morning and to celebrate with you your achievements in economic forecasting. May I take this opportunity to extend my sincere congratulations to the winner of this prestigious award to be handed out this morning, and also to congratulate all of the nominees present here today. I was very tempted to start off my remarks this morning by telling a little joke about economists, but decided against it, as I have been warned that an economist is someone who did not have enough personality to become an accountant. I promise to keep my remarks this morning quite brief. 2. Economic forecasting tested during the crisis Thomas Kida1 noted, “...the amount of knowledge we have in a certain area will not help us predict what will happen if the events are inherently unpredictable.” The recent global financial crisis and recession has proved to be a humbling experience for economists, financial market participants, policy makers, regulators and many others that derive their income from economic forecasting, understanding the world economy and that which governs it. The crisis was not foreseen by most forecasters, although there was a minority who had been warning of a pending disaster for some time. The failure to forecast the “Great Recession” caused the Queen of England herself to ask why it had not been foreseen. The turmoil of recent years has prompted critics to maintain that many economic models, and particularly macroeconomic models based on rational expectations, have been proven to be fundamentally flawed. Much of the criticism directed towards forecasting and macroeconomic analysts has probably been misplaced. It has never been so much about economic models being wrong, as it is about understanding that all models are limited – by definition – and that models that limit certain types of relationships will eventually fail when those relationships in the real world change. In other words, the right criticism is to say that economic models constructed in particular ways need to be used and understood in similarly particular ways. A model that does not include household balance sheets and what drives them will not tell you if you will have a household debt crisis. By the same token, a model that says that future wage inflation will be a perfect function of expected future inflation will always be at odds with reality. Criticising economists for thinking that their models replace what actually happens in the real world may be satisfying, but it is also off-key. By the same token, placing complete faith in an economic forecast or model is equally unhelpful, as we know that no model can fully capture the complex relationships we find in the real world. Therefore models should never be used mechanistically. Forecast errors Thomas Kida is a professor at the Isenberg School of Management at the University of Massachusetts. BIS central bankers’ speeches cannot tell us anything useful about financial and economic panic and crashes unless the models on which they are based account for the build up of imbalances that become unsustainable. Clearly, the conditions for sustainability also need to be included. For example, the ability of Greece to manage its current debt load differs depending on how fast the Greek economy can grow. The mistake was to allow so much of the global community, especially the financial markets and policy makers, to rely on economic analysis that placed little emphasis on structural factors and balance sheets and which ignored the herd behaviour occurring in financial markets. It was not a model of real estate prices that failed to see the absurdity of ever-rising prices in the US housing market, but the assessment of the bull market in property by many, often famous, economists. So how might we think about dealing with this problem of economic models and forecasting that will be by definition insufficient and, in fact, wrong, and the need for robust and useful economic analysis both of what models tell us and what they don’t? Certainly part of the answer lies in making progress with developing models that are useful in the appropriate contexts. For the Reserve Bank, a major step forward in modelling came with the implementation of the inflation targeting monetary policy framework in 2000. More accurate modelling and forecasting activities were (and are) required for inflation targeting, and the Research Department of the Bank set up a so-called “suite of models”. This ranges from single equation, high frequency data models to longer-term structural multivariate models, and also varies according to the combination of economic theory and statistical techniques applied.2 It is useful to supplement the forecasts from structural models with information obtained from higher frequency models to establish the starting point for the forecast trajectory. This approach allows the forecasters to optimise the positive characteristics of a particular model type and use the different approaches to improve the trajectory of the forecast and minimise errors. The suite of models approach can be useful to obtain different forecasts for the same variable which can then be pooled together to obtain the final forecast. Models differ in their focus and consequently in their characteristics and transmission mechanisms. Different outcomes are useful in gaining new perspectives on what is driving particular outcomes. There are of course limits to this approach. Too many different model properties make the assessment of the transmission mechanisms ambiguous and complicates the interpretation of the results. The success of any modelling and forecasting endeavour should be measured not by the sophistication or complexity of the techniques used or the accuracy of the data employed, but rather the degree of forecast error minimisation and the quality of the analysis that it supports. More and better information is critical to model building, forecasting and analysis. And here, the Reuters Survey provides a valuable service to the modelling activities of the Reserve Bank, and of course other forecasters. It is useful to compare forecasts to shed light on how other participants view the future and how they quantify the most important transmission channels of the economy. The analysis of surveys over time, moreover, tells us something about the nature of shocks hitting the economy, how the economy adjusts to the shocks, and the extent of policy reaction expected by the markets. For example, time series models help us forecast high frequency data for a horizon up to 2 quarters. However by definition, these models rely solely on past data patterns and are unable to predict turning points. Structural models are better suited to forecasting over longer horizons. BIS central bankers’ speeches In our recent discussions of the economy, considerable attention was devoted to understanding the combination of inflation, growth, and monetary policy reaction functions exhibited in private sector forecasts. Greater uncertainty in the global economic environment and newer data from international and domestic sources made it critically important to gather more views of conditions in the world and understand how they were being interpreted by the markets. A levelling-off of commodity prices and weaker economic growth in many economies presented significant differences from how we understood the world even two months ago. There have been other changes too, including on the domestic front, such as signs of weaker second quarter growth rates. Our task is to try to peer through these various changes, in part to ask ourselves hard questions about our forecasts and test them against potential scenarios. Scenarios, in turn, imply probabilities of events actually transpiring, and here our analysis has had to cross the boundaries into other fields. We noted for instance in the Monetary Policy Committee Statement our concerns about the debt crises in advanced economies and considered in our discussions the possibilities of how this crisis might be resolved. This general increase in uncertainty has become a characteristic of the post-recession world, and one that appears especially hard to cast off. Uncertainty demands a greater application of judgement across a broader range of perspectives than in times when variables move in clear directions and risks are low. 3. Conclusion These thoughts prompt us to recognise that flexibility is a key ingredient of useful macroeconomic forecasting exercises, even as flexibility may seem to violate the very idea of a model. This tension is in fact quite important. The inflexibility of a model’s fixed relationships help us to assess whether today’s actual outcomes will continue in the future or adjust in line with the historical relationships reflected in the model. As real world outcomes change, we in turn ask ourselves what is wrong with our model relationships or what other conditions have changed to generate outcomes beyond what the model suggests. In short, the framework used to produce forecasts must be dynamic and able to change should circumstances require. One area for future work for us in the Reserve Bank is to incorporate the lessons of the financial crisis and recession into how we use our macro econometric and other models. The financial crisis that has occurred was so unexpected that very few, if any, macroeconometric models captured the inter-linkages between the real and the financial spheres. Beyond interest rates, macro econometric models generally do not include financial transmission mechanisms. Clearly this needs to change, but we also need to recognise that even much more extensive coverage will not compensate for intensive regulatory focus on financial and asset markets and the balance sheets of institutions operating in them. And from a policy-making perspective, we need to recognise that whatever progress we make in refining and calibrating our models, and achieving a higher level of their sophistication, they will never be able to absolve us from our responsibility to ultimately exercise judgement in reaching any policy decisions. Thank you. BIS central bankers’ speeches