Model risk can arise in two ways: from fundamental flaws in models and from misuse of models. Turning first to model flaws, the risk is not that the model is imperfect – that is inevitable. Rather it is that errors are so fundamental that the model produces inaccurate or undesirable outputs when viewed against design objectives and intended business uses. For example, an insurer’s internal model might produce a capital requirement that is significantly too low; or a pricing model might include analytical methods that lead to outcomes that are outside the insurer’s risk appetite or, in the extreme, in breach of regulations or illegal. The Model Risk Working Party of the Institute and Faculty of Actuaries have published two good papers on model risk.5 The second one6 relates the case of the loss of the Mars climate orbiter in 1998 as an example  4  Supervisory Guidance on Model Risk Management, SR Letter 11-7, Federal Reserve Board and Office of Comptroller of the Currency, April 4 2011 5 https://www.actuaries.org.uk/practice-areas/risk-management/disbanded-research-working-parties/model-risk 6  All speeches are available online at www.bankofengland.co.uk/publications/Pages/speeches/default.aspx  6  of flawed model design. The trajectory of the spacecraft had been incorrectly calculated, which meant that the spacecraft had actually been orbiting much closer to Mars than had been targeted causing the space probe to disintegrate in the planet’s atmosphere. The problem was as basic as the use of incorrect units in part of the navigation software. An external contractor had produced software to provide Thruster performance data in English units of pound-seconds.

Model Drift  Model drift is the risk that the capital requirements calculated using an internal model may gradually weaken over time such that they no longer reflect the risks to which the firm is exposed. No measure of model drift is perfect. The PRA tracks movements in internal model capital compared to the standard formula and the net best estimate of liabilities.2 The first benchmarks the tailored risk calibrations in an insurer’s internal model against the ‘off the peg’ measures of risk specified in Solvency II. The second compares them to the size of the insurer’s balance sheet – a sort of insurance ‘leverage ratio’. Supervisors supplement these aggregate measures with ‘bottom up’ analysis to compare the outputs of internal models, looking at calibrations risk by risk, over time and across insurers.3  (see Chart 2)  In 2016 and 2017, the aggregate capital requirements of UK general insurers with internal models increased more or less in line with the standard formula and best estimate of liabilities. That is the outcome I would expect, other things being equal. No generalised model drift was apparent. For life insurers in aggregate, by contrast, both standard formula capital and best estimates of liabilities rose considerably more than internal model capital. At face value, this looks like considerable model drift. For a couple of reasons, though, we are not yet ringing the alarm bells. First, it is only two years of data. We will need a longer period to draw strong conclusions about trends.