Transparency in numbers: the dangers of statistical illiteracy
Authors: Imogen Evans, Hazel Thornton
Journal: Journal of the Royal Society of Medicine (2009)

There can be few pervasive and potentially serious conditions that regularly capture the headlines but are generally overlooked. Yet we turn a blind eye at our peril to the consequences of statistical illiteracy, especially where our health is concerned. Or more specifically, as Gigerenzer and colleagues have lately highlighted,1 to the societal problem they call collective statistical illiteracy. Gigerenzer's team of psychologists and physicians are not bemoaning public ignorance of Bayesian analysis and the like but simply the widespread inability to understand the meaning of numbers commonly used to express health statistics. They use the term ‘collective’ because the malaise affects doctors, patients and journalists – one could add politicians, lawyers and ethics committees too – and because these groups influence one another. At a time when patients are rightly being encouraged to participate in discussions concerning their health and to weigh up the benefits, risks and potential harms of various diagnostic or treatment options, this shared decision-making will only work if both doctor and patient are clear about what the numbers actually mean.

The root of the misunderstanding can often be traced back to the way health statistics are expressed in both the professional and lay press. For example, the probabilities, relative risks and survival rates that dominate patient information leaflets, press releases, newspaper reports and articles in medical journals cause considerable alarm and confusion. By contrast, natural frequencies (i.e. simple number counts) are not only simple and more intuitive but also contain more information. However, they also sound rather puny. Consider this comparison from Goldacre,2 another champion of natural frequencies: ‘patients taking ibuprofen for arthritis face a 24% increased risk of suffering a heart attack’ versus ‘in a study of 1005 people taking ibuprofen for arthritis, there was one extra heart attack’. The former was used to justify the headline ‘How pills for your headache could kill’; the latter would hardly warrant a passing mention.

Focusing on just two diseases – prostate cancer and breast cancer – shows the seriousness of the issue. Rudy Giuliani, former mayor of New York, used his experience of prostate cancer to political effect in a 2007 campaign advertisement: ‘I had prostate cancer five, six years ago. My chance of surviving prostate cancer in the United States? Eighty-two percent. My chance of surviving prostate cancer in England? Only 44 percent under socialized medicine.’1 That certainly grabbed the headlines – it was also unquestionably wrong. The risk of death from prostate cancer, as Gigerenzer et al. point out, is actually about the same in American and British men – 26 and 27 prostate cancer deaths per 100,000, respectively.

Giuliani was citing 5-year survival rates, not mortality rates. Moreover he was not comparing like with like, i.e. there were differences in the people being compared.3 American men tend to be diagnosed by prostate specific antigen (PSA) screening and therefore earlier than men in Britain, where PSA screening is not nearly so prevalent. But that does not automatically make them live longer – this is the phenomenon of lead-time bias. By way of illustrating its importance, Gigerenzer et al. ask us to imagine a group of prostate cancer patients diagnosed at the age of 67 years, all of whom die at 70, giving a 5-year survival rate of 0%, versus a group diagnosed by PSA screening at the earlier age of 60 years, who also die at 70, giving a 5-year survival rate of 100%. The survival rate has changed but the time of death is exactly the same. The Giuliani example also illustrates another source of bias – over-diagnosis bias, whereby screening detects abnormalities that conform to a pathological diagnosis but will never become symptomatic during a patient's lifetime. Owing to these two sorts of bias, changes in 5-year survival rates are not reliably related to changes in mortality.

The results of PSA screening per se are likewise headline fodder.4 ‘Prostate cancer screening could cut deaths by 20%’ (The Guardian) and ‘Better cancer screening is every man's right’(Scotsman) were just two responses in the British press to a reported study of 160,000 men aged 55–69 years randomly assigned to PSA screening or no screening. There were indeed 20% fewer deaths in the screening group – but in real numbers this means that 1410 men would need to be screened to prevent one death. Put another way, for each death prevented, 48 men would need to be treated, some unnecessarily and at the expense of distressing side-effects such as incontinence and impotence.

By contrast, breast cancer screening is well-established in Britain, with the NHS Breast Screening Programme (NHS BSP) this year celebrating its 20th anniversary. Nevertheless, the 2 million healthy and symptomless women invited annually for screening (as part of a 3-year cycle) are being misled by an information leaflet that oversells the benefits and glosses over the harms. Recent criticism of the content is the latest challenge to its misleading inadequacy.5 The title alone, ‘Breast Screening: The Facts’, oozes trustworthiness.6 And true, the leaflet uses concrete numbers. The trouble is that, on the few occasions it does use them, they bear no resemblance to those readily available from robust sources such as Cochrane reviews.

So, on the benefits side the leaflet declares that screening saves ‘an estimated 1400 lives each year in this country’. Unfortunately there is no reliable evidence that screening reduces mortality. Contrast this with an evidence-based leaflet by Gotzsche et al.5 based on the lay summary from the latest Cochrane review:7 ‘If 2000 women are screened regularly for 10 years, one will benefit from the screening, as she will avoid dying from breast cancer. At the same time, 10 healthy women will, as a consequence, become cancer patients and will be treated unnecessarily … Furthermore, about 200 healthy women will experience a false alarm.’ Harms receive scant attention in the NHS BSP leaflet. Above all, as Gotzsche et al. point out, the major harm of screening – unnecessary treatment of harmless lesions – is totally ignored.5

The director of NHS Cancer Screening Programmes, responding to a call by international experts for the leaflet to be rewritten,8 said that ‘putting too much numerical information meant women just put the leaflet down’.9 Her opinion flies in the face of findings by Gigerenzer et al. that the public should be given more [concrete] numbers since there is evidence that patients perceive risk more accurately when it is also given numerically, particularly if accompanied by simple tables and other decision aids. Accurate, comprehensible information matters, not least to the one in five screened women in the NHS BSP who will be diagnosed with cancer and then learn that they have ductal carcinoma in situ.10 None of them will have been forewarned what that means in terms of difficult decision-making, anxiety, over-diagnosis and over-treatment.

Is there a cure for collective statistical illiteracy? Transparent use of health information, framed in a way that everyone most readily comprehends, has to be the initial step. As Gigerenzer et al. point out, this is a prerequisite for understanding risks and asking critical questions, which should ultimately mean that people's hopes and anxieties are less easily manipulated. And for health professionals what better place to start than with medical journals. Will JRSM rise to the challenge?