Response to Rhodes and Yarnold regarding “limitations of classification and regression tree analysis in vancomycin exposure-response relationship studies: insights from data simulation” (CLM-21-22045)
Authors: Bruce R. Dalton, Anish Krishnan, Jackson J. Stewart, Sarah C.J. Jorgensen
Journal: Clinical Microbiology and Infection (2021)

We thank Rhodes and Yarnold [[1]Rhodes N.J. Yarnold P.R. Re: 'Limitations of Classification and Regression Tree Analysis in Vancomycin Exposure-Response Relationship Studies' by Dalton et al.Clin Microbiol Infect. 2021; https://doi.org/10.1016/j.cmi.2021.08.013Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar] for their comments on our analyses [[2]Dalton B.R. Krishnan A. Stewart J.J. Jorgensen S.C.J. Limitations of classification and regression tree analysis in vancomycin exposure-response relationship studies: insights from data simulation.Clin Microbiol Infect. 2021; https://doi.org/10.1016/j.cmi.2021.07.028Abstract Full Text Full Text PDF Scopus (1) Google Scholar]. Herein we address what they describe as serious mathematical and methodological flaws in our work. First, we emphasize that the objective of our analysis was to use simulations to mimic the methods used in studies exploring exposure–response or toxicity relationships employing classification and regression tree (CART) analysis using vancomycin studies as an example. Thus methodological flaws observed in our work may be representative of many other studies, some of which we outlined in our report. With regards to the transparency of our methods, we direct readers to the Supplementary Material. We encourage the readers to contact us for any additional details. We disagree with Rhodes and Yarnold's unreferenced assertion that type 1 errors are not increased by CART analysis because a complete analysis represents a single hypothesis test. The literature is replete with statisticians' expressions of concern about inflated type 1 error rates with CART [3Venkatasubramaniam A. Wolfson J. Mitchell N. Barnes T. JaKa M. French S. Decision trees in epidemiological research.Emerg Themes Epidemiol. 2017; 14: 11Crossref PubMed Scopus (42) Google Scholar, 4Jung S.H. Chen Y. Ahn H. Type 1 error control for tree classification.Cancer Inform. 2014; 13: 11-18PubMed Google Scholar]. A decision tree analyst does not generate a specific hypothesis: e.g. "Does an AUC >373 predict survival better that ≤373?" Rather, they explore the data for a cutpoint of maximal heterogeneity by multiple simultaneous comparisons using all possible cutpoints (within the specifications of the analysis), each comparison representing a specific hypothesis. An alternative decision tree method, χ2 automatic interaction detection (CHAID), was created with intrinsic significance testing with Bonferroni adjustment (BA) for multiple comparisons. To further illustrate the importance of type 1 error control in decision tree analysis, we have rerun our decision tree analyses on the 30 simulated datasets using CHAID with and without BA (see Supplementary Material for methods and SPSS code). Without type 1 error control using BA, 10/30 (33%) of the CHAID p values were <0.05. In contrast, with BA, 2/30 simulations (6.7%) produced p values that were <0.05 and 17/30 (57%) had a value of 1.0 (Fig. 1), which is closer to what one would expect with randomly generated data. Therefore CHAID with BA may be a more conservative method of decision tree analysis and perhaps would have lower rates of type 1 error. In a distinct but related point, the correspondents assert that we should have controlled for type 1 error using BA since we ran 30 different simulations. Each simulation is an independent dataset, representing an independent hypothetical study. Applying BA in the manner proposed would be akin to applying adjustment to meta-analytic methods or replication studies, such that analysis of a larger number of studies use adjusted (lowered) p values. This would be illogical since a greater number of studies should increase the confidence of the results not reduce it. As discussed above, it would be appropriate to apply BA within the analysis of each dataset. This was not done as we were replicating the methods of vancomycin exposure studies. In regard to our sample sizes, a recently published systematic review of studies of efficacy outcomes of patients treated with vancomycin, many of which used CART analysis, included 30 studies ranging in size from 20 to 320 patients [[5]Dalton B.R. Rajakumar I. Langevin A. Ondro C. Sabuda D. Griener T.P. et al.Vancomycin area under the curve to minimum inhibitory concentration ratio predicting clinical outcome: a systematic review and meta-analysis with pooled sensitivity and specificity.Clin Microbiol Infect. 2020; 26: 436-446Abstract Full Text Full Text PDF PubMed Scopus (19) Google Scholar]. None discussed an a priori sample size determination. Therefore, our simulations with samples of 50, 100 and 200 cases adequately, although perhaps not perfectly, represent studies previously published on this topic, and precedent has been set for not performing a priori sample size calculation. Our analyses recapitulated the flawed methods of previous studies to illustrate their limitations. Similarly, a further point of criticism is that we did not perform internal validation measures such as hold-out sampling. Indeed, since we replicated the methods of vancomycin exposure studies which did not describe any of these measures, we omitted this step as well. We agree on the importance of internal validation, and in fact this is the very point of our analysis: CART results require validation (internal and external) since CART may find random associations in data. Our findings demonstrate what can go wrong when researchers fail to incorporate these important steps. We do agree with the correspondents' final point that clinician scientists must continue to focus on how to maximize the safety and efficacy of this agent. It is only by acknowledgement of the problems of previous analyses that improvement in the rigour of future studies is possible. BRD, AK, JJS and SCJJ contributed to the study design, presentation of results, and writing of the manuscript. BRD performed the statistical analyses. SCJJ has received a speaker's honorarium from Sunovion. The other authors declare that they have no conflicts of interest. No specific funding has been provided for this project. The authors gratefully acknowledge the support and advice of Peter Faris and M. Sarah Rose, Health Services Statistical and Analytic Methods, Alberta Health Services, Calgary, Canada. The following is the Supplementary data to this article: Download .docx (.02 MB) Help with docx files Multimedia component 1 Re: 'Limitations of classification and regression tree analysis in vancomycin exposure – response relationship studies' by Dalton et al.Clinical Microbiology and InfectionVol. 27Issue 12PreviewDalton et al. recently wrote that classification and regression tree (CART) models increase the type 1 error rate [1]. They question the validity of studies cited within the updated vancomycin therapeutic monitoring guidelines [2], which report that greater exposures (vis-à-vis area under the curve) are associated with different clinical outcomes, and they suggest that classification rules derived using CART models are invalid [1]. Using simulation to demonstrate the risk of type 1 error with CART analysis, they generated 30 data sets of randomly sampled areas under the curve and randomly assigned outcome classes (e.g. Full-Text PDF