Clinical predictors and prediction rules to estimate initial patient risk for infective endocarditis in Staphylococcus aureus bacteraemia: attention must be paid to the reference standard
Authors: G.S. Heriot, A.C. Cheng, S.Y.C. Tong, D. Liew
Journal: Clinical Microbiology and Infection (2017)

We read with interest the recent systematic review of clinical prediction rules for endocarditis in patients with Staphylococcus aureus bacteraemia (SAB) performed by Bai et al. [[1]Bai A.D. Agarwal A. Steinberg M. Showler A. Burry L. Tomlinson G.A. et al.Clinical predictors and clinical prediction rules to estimate initial patient risk for infective endocarditis in Staphylococcus aureus bacteraemia: a systematic review and meta-analysis.Clin Microbiol Infect. 2017; 23: 900-906Abstract Full Text Full Text PDF PubMed Scopus (45) Google Scholar]. While we agree with their overall conclusion that this literature is a meaningful step towards a risk-stratification approach to the use of echocardiography in SAB, we wish to raise an important issue affecting these studies not discussed by the authors. In their review, Bai et al. present the diagnostic performance of six prediction rules measured against the modified Duke criteria as the reference standard for the diagnosis of endocarditis. Of these six studies, only one was restricted to patients undergoing transoesophageal echocardiography (TOE); in the other five studies TOE rates ranged from 16% to 71%. The sensitivity of the modified Duke criteria as a reference standard in SAB is highly dependent on the use of TOE, as up to half of patients with definite endocarditis do not manifest multiple minor criteria [[2]Incani A. Hair C. Purnell P. O'Brien D.P. Cheng A.C. Appelbe A. et al.Staphylococcus aureus bacteraemia: evaluation of the role of transoesophageal echocardiography in identifying clinically unsuspected endocarditis.Eur J Clin Microbiol Infect Dis. 2013; 32: 1003-1008Crossref PubMed Scopus (47) Google Scholar], and transthoracic echocardiography has inferior diagnostic performance compared with TOE [[3]Bai A.D. Steinberg M. Showler A. Burry L. Bhatia R.S. Tomlinson G.A. et al.Diagnostic accuracy of transthoracic echocardiography for infective endocarditis findings using transesophageal echocardiography as the reference standard: a meta-analysis.J Am Soc Echocardiogr. 2017; 30: 639-646Abstract Full Text Full Text PDF PubMed Scopus (72) Google Scholar]. While the authors of some of these studies suggested that all cases of unrecognized endocarditis would manifest clinically during the follow-up period, there are no data to support this view, particularly in the context of the prolonged parenteral antibiotic therapy given to patients without diagnosed endocarditis in these studies (e.g. a median of 28 days in the study by Palraj et al. [[4]Palraj B.R. Baddour L.M. Hess E.P. Steckelberg J.M. Wilson W.R. Lahr B.D. et al.Predicting Risk of Endocarditis Using a Clinical Tool (PREDICT): scoring system to guide use of echocardiography in the management of Staphylococcus aureus bacteremia.Clin Infect Dis. 2015; 61: 18-28Crossref PubMed Scopus (73) Google Scholar]). The under-diagnosis of endocarditis because of incomplete TOE coverage is evident within the studies discussed by Bai et al. Ten of the 30 studies of SAB retrieved by the authors reported the prevalence of endocarditis as defined by the modified Duke criteria among unselected cases of SAB; seven of these provided sufficient data to calculate the proportion of cases undergoing TOE. Fig. 1 presents a weighted univariate generalized linear model with binomial error structure and logit link function relating the natural log of the proportion of patients undergoing TOE (the horizontal axis) to the apparent prevalence of endocarditis (the vertical axis) for these seven studies. Also included on these axes (although not in the regression model) is the apparent endocarditis prevalence in the three studies retrieved by Bai et al. reporting the prevalence of modified Duke criteria endocarditis in series restricted to patients investigated with TOE. As can be seen in Fig. 1, there is a significant association between TOE coverage and apparent endocarditis prevalence in these studies (p <0.001), which appears to explain much of the variation in apparent endocarditis prevalence noted by Bai et al. While this ecological relationship could also be interpreted as showing that clinicians employ TOE less often in settings where endocarditis is less common among patients with SAB, evidence from interventional studies of infectious diseases specialist consultation would suggest that the direction of causation is as we suggest [[5]Paulsen J. Solligard E. Damas J.K. DeWan A. Asvold B.O. Bracken M.B. The impact of infectious disease specialist consultation for Staphylococcus aureus bloodstream infections: a systematic review.Open Forum Infect Dis. 2016; 3: ofw048Crossref PubMed Scopus (53) Google Scholar]. This phenomenon has two important effects on the results of the five clinical prediction rules mentioned above. First, misclassification errors in the reference standard introduce bias into the apparent diagnostic performance of the clinical prediction rules. As all of the clinical prediction rules incorporate at least one minor criterion from the modified Duke criteria, and generally describe the same biological phenomenon (manifestations of complicated SAB), it is reasonable to expect significant covariance between the rules and the reference standard, and therefore overestimation of their diagnostic performance. Quantification of this bias can be attempted using the results of the model presented in Fig. 1 to estimate the sensitivity of the modified Duke criteria at different TOE rates (assuming that patients meeting the modified Duke criteria without TOE would also meet the criteria with TOE). As an example, the negative likelihood ratio point estimate of the clinical prediction rule reported by Palraj et al. [[4]Palraj B.R. Baddour L.M. Hess E.P. Steckelberg J.M. Wilson W.R. Lahr B.D. et al.Predicting Risk of Endocarditis Using a Clinical Tool (PREDICT): scoring system to guide use of echocardiography in the management of Staphylococcus aureus bacteremia.Clin Infect Dis. 2015; 61: 18-28Crossref PubMed Scopus (73) Google Scholar] (0.15) could actually be anywhere between 0.14 (assuming conditional independence) and 0.85 (assuming maximal conditional dependence) [[6]Vacek P.M. The effect of conditional dependence on the evaluation of diagnostic tests.Biometrics. 1985; 41: 959-968Crossref PubMed Scopus (229) Google Scholar]. Second, the application of TOE in these series is not random, and is almost certainly related to clinicians' assessment of endocarditis risk [[7]Young H. Knepper B.C. Price C.S. Heard S. Jenkins T.C. Clinical reasoning of infectious diseases physicians behind the use or nonuse of transesophageal echocardiography in Staphylococcus aureus bacteremia.Open Forum Infect Dis. 2016; 3: ofw204Crossref PubMed Scopus (9) Google Scholar], which is itself based on the clinical risk factors incorporated into the clinical prediction rules. This selection results in lower TOE usage in the low-risk groups [[8]Gow N. Lowe B.S. Freeman J. Roberts S. The role of echocardiography in Staphylococcus aureus bacteraemia at Auckland City Hospital.N Z Med J. 2015; 128: 28-35PubMed Google Scholar], biasing the results of these studies in favour of the clinical prediction rules. These two effects may explain the disappointing performance of the Kaasch criteria [[9]Kaasch A.J. Fowler V.G. Rieg S. Peyerl-Hoffmann G. Birkholz H. Hellmich M. et al.Use of a simple criteria set for guiding echocardiography in nosocomial Staphylococcus aureus bacteremia.Clin Infect Dis. 2011; 53: 1-9Crossref PubMed Scopus (103) Google Scholar] when assessed in patients uniformly investigated with TOE [[10]Khatib R. Sharma M. Echocardiography is dispensable in uncomplicated Staphylococcus aureus bacteremia.Medicine. 2013; 92: 182-188Crossref PubMed Scopus (49) Google Scholar]. Given that previous work has suggested that patients with a probability of endocarditis as low as 2% might benefit from TOE [[11]Rosen A.B. Fowler Jr., V.G. Corey G.R. Downs S.M. Biddle A.K. Li J. et al.Cost-effectiveness of transesophageal echocardiography to determine the duration of therapy for intravascular catheter-associated Staphylococcus aureus bacteremia.Ann Int Med. 1999; 130: 810-820Crossref PubMed Scopus (154) Google Scholar], clinical prediction rules used to direct echocardiography decisions need to reliably place a patient at a very low risk of endocarditis. The insensitive reference standard applied in studies relying on the modified Duke criteria despite incomplete echocardiography coverage undermines their ability to achieve this goal. GSH is the primary author of one of the studies mentioned in the review by Bai et al. GSH is supported by an Australian Government Research and Training Program stipend and a faculty award from the Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Australia.