Medicine and books: When and How to Assess Fast-changing Technologies: A Comparative Study of Medical Applications of Four Generic Technologies
Authors: JP Bunker
Journal: BMJ (1998)

G Mowatt, D J Bower, J A Brebner, J A Cairns, A M Grant, L McKee 

Health Technology Assessment, £52 (free to NHS and learned bodies), pp 149 

ISSN 1366 5278 

Every encounter with a patient is an experiment, according to the old saying. Certainly, every encounter during which a new treatment is first used should be. What that experiment should be, however, is not self evident. Whether it should be a randomised clinical trial, as some have urged, is highly debatable and cannot be easily resolved. 

It may not ever be possible or desirable to randomise the first patient for a new operation or invasive diagnostic procedure, for much the same reason that the first clinical use of a new drug should not be randomised until preliminary assessments of toxicity have been completed. The introduction of new drugs offers a useful model, as one of my American colleagues suggested in a commentary entitled “Why not an FDA for surgery?” 

In phase I of the standard procedure for introducing a new drug its toxicity is first assessed in animals. Similarly, a new surgical technique can be developed in animals. Norman Shumway spent 10 years perfecting cardiac transplantation in the animal laboratory. Anti-vivisectionists may argue that animal research is not necessary, as they did in a Baltimore court many years ago, claiming that Mr R C (later Lord) Brock had not needed animals to introduce the Blalock operation for the treatment of tetralogy of Fallot into England. But Brock happened to be in the gallery of the court and was able to tell the court that Blalock’s prior experimental work in dogs had been essential to his innovative operation in children 

Phase I may not always be feasible or necessary for surgery or for other invasive procedures. Phase II of drug innovation, on the other hand, is a close parallel to the “learning curve” during which a surgeon or physician learns how to do a new procedure, with phase III consisting of randomised trials if possible and appropriate. The importance of post-marketing surveillance in drug innovation—so called phase IV—is increasingly recognised, and this is equally important for new procedures, to which the authors of the report call attention. Just how important it is to assess the efficacy of a new procedure once it has been accepted for general use may not be widely appreciated. We are, in fact, much better at devising and assessing new treatments than we are at adopting them for general practice. 

The authors of this report engaged in a search for a blueprint, a “formula for the timing of health technology assessment,” with the hope of developing criteria “for guiding the timing and nature of assessment, and for deciding when [innovations] should become routine services.” Towards this end they undertook a series of case studies, a literature search, and interviews with “key individuals associated with the reviewed health technologies.” At the end of their search they came to realise that there could be no single, all encompassing formula or protocol, that there are “problems associated with all methods, and hence, a varied, empirical, and iterative approach [gives] the most reliable results.” 

In the process of looking, they have produced a sophisticated and comprehensive review of the problems encountered in assessing the technologies of medical care. Addressing the long debated issue of whether medical technologies should be randomised at the outset, for example, they encountered the “catch 22” situation that besets so much of evaluative research: the need to postpone assessment “until clinicians have reached an appropriate point on the learning curve, [which] can usually only be recognised retrospectively, by which time clinicians may no longer be prepared to randomise patients.” 


They also encountered the insoluble ethical dilemma that, in a randomised controlled trial, a patient randomised to the experimental treatment may be exposed to an unknown risk while a patient randomised to control is deprived of the opportunity offered by the new and potentially improved treatment. They summarise other equally daunting ethical problems, such as that randomised controlled trials “might be terminated for ‘ethical reasons’ once sufficient clinical data [has] been collected to convince the investigator (of the usefulness or otherwise of the health technology), even though further trials might be needed to convince others.” 


April bestsellers BMJ Bookshop
1   BMA Complete Family Health Encyclopaedia Dr T Smith Dorling Kindersley £29.95, ISBN 0751301612 

2   Notes for the MRCGP, 3rd (updated for the new modular MRCGP exam) K T Palmer Blackwell Science £19.95, ISBN 086542 777 1 

3   BNF No 35, March 1998 BMA/Royal Pharmaceutical Society £14.95, ISBN 0853694117 

4   Winning the Publications Game T Albert Radcliffe Medical Press £15.00, ISBN 1857751833 

5   Oxford Handbook of Clinical Medicine R A Hope, J M Longmore, T J Hodgetts OUP, £13.95, ISBN 0192621157 

6   How to Read a Paper: the basis of EBM T Greenhalgh BMJ Books, £14.95, ISBN 07279 11392 

7   Therapeutic Uses of Cannabis BMA/Board of Science Harwood Academic Publishers £11.99, ISBN 90 5702 3180 

8   Advanced Paediatric Life Support, 2nd ed Advanced Life Support Group BMJ Books, £22.95, ISBN 07279 1069 8 

9   Pocket Guide to Critical Appraisal Ian K Crombie BMJ Books, £9.95, ISBN 07279 1099 X 

10  Guide to the Internet: an introduction for healthcare professionals M Pallen BMJ Books, £4.95, ISBN 0 7279 1255 0 




The question asked in this report is not a very useful one. My quibble is with the report’s stated purpose: “to try to identify the optimal time at which to start assessing new and fast-evolving health technologies.” At the end the authors came to recognise that how a new procedure should most effectively be assessed is apt to vary with every such new treatment. 

But, as a review of the problems and methods of medical innovation, it is very successful. It seems to have been written primarily for policy decision makers and “purchasers,” and it should be useful in introducing them to the complexities and subtleties of clinical research. It should also be useful to the uninitiated doctor intending to embark on clinical investigation. It will be less useful to the mature investigator.