When and how to evaluate health information systems?
Aims: Evaluating large scale health information systems (HIS) such as hospital systems can be difficult. This article discusses the reasons we need to evaluate these systems and a range of appropriate methods to carry out evaluations. It is written in non-technical language to assist health policy makers and others commissioning or implementing such systems, with references and a web site containing information for those wishing more detail (http://www.ucl.ac.uk/kmc/evaluation/index.html). Methods: A variety of questions relevant to HIS and qualitative and quantitative methods ranging from simple beforeafter to controlled beforeafter and fully randomised designs, are discussed. A running exampleevaluating the impact of an order communications system on lab requestsis used to illustrate the potential problems, and how they can be resolved. Results: The main types of biases affecting impact studies and methods to reduce them are described. The article then discusses some trade-offs between the low cost, easily conducted beforeafter study with its unreliable results versus the more complex, expensive but much more rigorous randomised trial. Conclusions: As would be expected, the correct methods to evaluate depend not on what technology is being evaluatedwhether an information system or a drugbut on the questions the study is designed to answer, and how reliable the answers must be. Only those commissioning an evaluation study can decide these.
health information systems evaluation methods technology assistant