Observational evidence for determining drug safety
Authors: Nick Freemantle, Alar Irs
Journal: BMJ (2008)

Drug regulators are constantly criticised for their handling of drug safety. Comment on the activities of the US Food and Drug Administration with regard to drug safety has become particularly loud and damning. A recent article that proposed sweeping changes to the system criticised the FDA for lacking relevant expertise, being too close to the drug industry, and failing to ensure that companies deliver on their safety commitments with new drugs.1 The main practical recommendation was that systems of monitoring and reporting adverse events should be enhanced, highlighting a perceived role for observational studies. Drug safety is also an important concern in Europe, and the European Commission Enterprise and Industry Directorate General is consulting with stakeholders on changes of legislation to strengthen and rationalise pharmacovigilance in Europe.2 It describes a strategy that also focuses on observational studies of drug safety. But can observational studies alone improve judgments on drug safety?

Using adverse event reporting and observational studies to examine safety might seem attractive, but for several reasons observational studies alone cannot provide reliable estimates of treatment effects. For example, safety outcomes can be confounded with patients’ prognosis, drugs may be allocated according to clinician’s choices rather than by a quasi-random process, and clinicians may be influenced by trying to improve outcomes in people they think are at high risk.

In a properly randomised controlled trial, experimental conditions differ according to chance and the intended treatment, which enables robust inference. In observational studies on drug safety, the way that clinicians select patients can have important effects. Patients treated with an agent of interest may be at increased risk as described by known prognostic factors. This was the case in a recent highly publicised observational study of aprotinin for the reduction of bleeding and risk of transfusion in cardiac surgery,3 4 which suggested a 48% increase in the risk of death in the long term associated with the use of aprotinin.4 Multivariable methods may help but inclusion of such prognostic factors in a multivariable model does not fully account for their effects. Identified risk factors and their inter-relations are often complex and statistical models deal with these only crudely and partially. Thus a statistical model may include hypertension as just a “yes or no” dichotomy, rather than consider the level of raised blood pressure. In addition, the effects of raised blood pressure may differ in people who do, or do not, have other risk factors in ways that cannot be readily dealt with in analysis. In any case, the multivariable analysis simply estimates the effects of known risk factors from the available data, and the best estimate available from the data may not be a very good one.

Observational studies are further compromised if clinicians make a judgment on whether or not to treat a patient with the drug of interest. In part, this may be haphazard or essentially quasi-random, or it may be based at least on the known prognostic factors and a range of other factors that we cannot fully understand from the summary data collected. Where this clinical judgment accurately predicts outcome for a patient beyond the effects of known risk factors, it represents a latency that is confounded with drug exposure. This will lead to a negative bias in the results of multivariable analyses.

A further problem associated with observational studies is that they can affect our judgment on more reliable measurements. Would the BART trial comparing aprotinin with the antifibrinolytic agents tranexamic acid or epsilon-aminocaproic acid in patients undergoing cardiac surgery have been stopped and aprotinin withdrawn had it not been for the previous observational study indicating risk associated with aprotinin?5 This happened even though a recent systematic review of randomised trials found no increased risks associated with treatment.6

Information on the size of the increased risk seen in the BART trial is currently limited, but an FDA statement indicates that the adverse signal had not reached conventional levels of statistical significance at the interim analysis of the trial.5 It is also not clear whether this conventional level of statistical significance had accounted for multiple testing as is appropriate in interim safety monitoring. Given the lack of a placebo group, the observed difference between the groups may have been caused by superiority of the comparator agents rather than increased risk with aprotinin, although chance also seems a plausible explanation, and stopping the trial early rules out more conclusive evidence on this important question.

So what is the answer to concerns about drug safety? Only properly randomised trials can provide truly reliable evidence on adverse events, just as these are the only source of convincing data on drug efficacy. Observational studies may provide some limited reassurance that a drug is safe, or they may provide an early indication of a problem, but by design they cannot provide reliable evidence on questions of drug safety. Regulators can, and do, require large scale randomised trials to be conducted after launch that look at important emerging safety concerns, and their power and determination to ensure this happens should be enhanced. For example, they could make continued marketing authorisation conditional on the provision of new data from trials which meet pre-agreed standards. If these trials are properly designed they can provide important answers not just about safety concerns, but crucially about the balance of risks and benefits associated with the use of a particular agent.