To begin, how do we assess errors from models intended to capture performance under stressed conditions when those conditions have not yet been realized and might not be in the historical data? How can we assess the uncertainty or margin of error around loss and revenue projections derived from models that can be quite complex, often involving multiple estimation steps? Further, since the final stress test calculations are derived from the combination of projections from many different (often multi-step) models, how do we assess the error around the ultimate calculation of stressed capital ratios? How do we measure correlation in model errors in a tractable and practical way? How much model risk owes to the decision to develop complex models for many individual pieces of the net income and regulatory capital ratio calculations instead of using simpler, “top down” estimation approaches? A final tactical research area involves this last point – the role of simpler, easier to estimate models of net income and its key components. Federal Reserve modeling teams have already developed a set of these “benchmark” models that produce loss and revenue estimates as a comparison to the projections from the more sophisticated and complex production models. Creating more of these models – including multiple benchmarks for an individual loss, expense or revenue component, benchmarks that cover the full range of detailed net income elements, and benchmarks at higher levels of aggregation – would enable more robust assessments of supervisory results both across institutions and over time.

Modeling socalled pre-provision net revenue (PPNR) was one of the innovations introduced in the SCAP and represented one of the key challenges in that exercise – and, I will argue, is a continuing challenge today. Given the choice to model net income and regulatory capital ratios, another set of design choices concerns the horizon over which net income is calculated and how cumulative changes in net income over this horizon are incorporated into regulatory capital ratios. The SCAP assumed a two-year forward horizon, while the subsequent CCAR and DFAST stress tests have assumed a 9-quarter forward horizon.6 Other supervisory stress test regimes have made different assumptions – for instance, the European stress tests use a three-year forward horizon.7 An 2/8  BIS central bankers' speeches  alternative choice would be to project losses and income over the full lifetime of a bank’s loans and other assets. Estimating lifetime losses on particular fixed portfolios of loans and securities was arguably the prevalent approach during the financial crisis period.8 But in a stress testing context, the lifetime loss approach presents practical difficulties given the varying maturities of different types of assets, as well as the greater uncertainty inherent in more distant future points. At the time of the SCAP, we judged that a two-year horizon was long enough to capture significant losses while remaining within what could reasonably be projected conditional on the macroeconomic scenario.9 Closely related to the scenario horizon question is how cumulative losses and revenues are incorporated to calculate post-stress capital ratios.