Repeated transarterial chemoembolization: An overfitting effort?
Authors: Antonio Facciorusso, Sherrie Bhoori, Carlo Sposito, Vincenzo Mazzaferro
Journal: Journal of Hepatology (2015)

Retreatment with TACE: The ABCR SCORE, an aid to the decision-making processJournal of HepatologyVol. 62Issue 4PreviewTransarterial chemoembolization (TACE) is the standard of care for intermediate stage hepatocellular carcinoma (HCC) and it is the most commonly used treatment for HCC worldwide. However, no prognostic indices, designed to select appropriate candidates for repeat conventional TACE, have been incorporated in the guidelines. Full-Text PDF Reply to: ''Repeated transarterial chemoembolization: An overfitting effort?''Journal of HepatologyVol. 62Issue 6PreviewWe would like to thank Facciorusso et al. for their commentaries on our article and for applying our ABCR score to their important cohort of patients. Full-Text PDF Open Access We read with great interest the article by Adhoute et al. published in Journal of Hepatology [[1]Adhoute X. Penaranda G. Naude S. Raoul J.L. Perrier H. Bayle O. et al.Retreatment with TACE: the ABCR score, an aid to the decision-making process.J Hepatol. 2014; (pii: S0168-8278(14)00858-7)Google Scholar]. The authors developed a point score system, the ABCR [standing for Alpha-fetoprotein (AFP), Barcelona Clinic Liver Cancer (BCLC), Child-Pugh and Response], to assist in the decision making on whether to retreat hepatocellular carcinoma (HCC) patients with multiple transarterial chemoembolization (TACE) sessions. The study population consisted of HCC patients treated with repeated consecutive TACE sessions and the resulting significant parameters from regression analysis were used to build the score. In this way, the authors differentiated three groups with different survival. The score was consistently validated in training and confirmatory cohorts and a higher ABCR score after the first TACE course was found to be associated with patients at poorer prognosis who may not benefit from further TACE sessions. Current guidelines do not specify the criteria for repeating TACE and the correct number of repeated procedures to undertake, hence the paper by Adhoute et al. is certainly of interest. However, it should be noted that patients in more advanced BCLC stage and with higher baseline AFP levels, namely those requiring further treatment repetitions are considered less likely to benefit from repeated TACE courses by the final score. This discrepancy could be explained by the fact that the accuracy of ABCR score is mostly hampered by the impact of baseline tumoral parameters (BCLC stage and AFP) on survival. Moreover, many of the features determining the score are not commonly observed in conventional intermediate patients undergoing TACE. In our cohort of 189 consecutive HCC patients treated with repeated TACEs on demand, only 4/189 (2.1%) patients were in BCLC C and only 30 (15.8%) presented with an AFP level ⩾200 IU/ml. In addition, less than 2% of patients were retreated within 6–8 weeks from original TACE in presence of Child-Pugh deterioration ⩾2 points. On these premises, and in the absence of consideration of patient's performance status, variations in the ABCR will be largely influenced by only two covariates: the early treatment response and the BCLC A/B status, with consequent loss in discriminatory abilities. In a way, the reduced weight assigned to very common clinical conditions observed in TACE could make the proposed score scarcely applicable in different contexts, particularly when current guidelines for TACE indication are more rigorously followed. At least, this was the case when the ABCR score was applied to our series (Fig. 1). In recent years a number of scores and nomograms have been proposed to properly guide the therapeutic decision of TACE repetition in HCC patients but none of them unequivocally confirmed in clinical practice. For instance the Assessment for Repetition of TACE (ART) score seemed to properly address the need of a reliable objective tool easily applicable at bedside [[2]Sieghart W. Hucke F. Pinter M. Graziadei I. Vogel W. Müller C. et al.The ART of decision making: retreatment with transarterial chemoembolization in patients with hepatocellular carcinoma.Hepatology. 2013; 57: 2261-2273Crossref PubMed Scopus (262) Google Scholar] but, despite the initial enthusiasm and the correct methodological approach, it has not been validated and to the best of our knowledge, has yet to enter clinical practice. The same can be said of other numeric scores proposed by other reputed groups [3Kadalayil L. Benini R. Pallan L. O'Beirne J. Marelli L. Yu D. et al.A simple prognostic scoring system for patients receiving transarterial embolisation for hepatocellular cancer.Ann Oncol. 2013; 24: 2565-2570Crossref PubMed Scopus (232) Google Scholar, 4Pinato D.J. Stebbing J. Ishizuka M. Khan S.A. Wasan H.S. North B.V. et al.A novel and validated prognostic index in hepatocellular carcinoma: the inflammation based index (IBI).J Hepatol. 2012; 57: 1013-1020Abstract Full Text Full Text PDF PubMed Scopus (151) Google Scholar]. All these efforts, although properly conducted, suffer from overfitting: a phenomenon occurring when a model maximizes its performance on some set of data but its predictive performance is not confirmed elsewhere due to random fluctuations of patients' characteristics in different clinical and demographical backgrounds. The very fact that so many different scores keep on being proposed confirms the excessive importance given to external validation in preventing overfitting. As in the case of the study by Adhoute et al., the score is tested in a different but "plausibly related" cohort and that is called external validation [1Adhoute X. Penaranda G. Naude S. Raoul J.L. Perrier H. Bayle O. et al.Retreatment with TACE: the ABCR score, an aid to the decision-making process.J Hepatol. 2014; (pii: S0168-8278(14)00858-7)Google Scholar, 2Sieghart W. Hucke F. Pinter M. Graziadei I. Vogel W. Müller C. et al.The ART of decision making: retreatment with transarterial chemoembolization in patients with hepatocellular carcinoma.Hepatology. 2013; 57: 2261-2273Crossref PubMed Scopus (262) Google Scholar, 3Kadalayil L. Benini R. Pallan L. O'Beirne J. Marelli L. Yu D. et al.A simple prognostic scoring system for patients receiving transarterial embolisation for hepatocellular cancer.Ann Oncol. 2013; 24: 2565-2570Crossref PubMed Scopus (232) Google Scholar], although it is known that external validation shows sufficient power to detect clinically important changes in performance only when substantial sample sizes are available [[5]Steyerberg E.W. Bleeker S.E. Moll H.A. Grobbee D.E. Moons K.G. Internal and external validation of predictive models: a simulation study of bias and precision in small samples.J Clin Epidemiol. 2003; 56: 441-447Abstract Full Text Full Text PDF PubMed Scopus (400) Google Scholar]. With smaller series, as in this study from two French groups, the sole external validation may have led to an overestimation of the performance of the model. A possible often neglected alternative is the internal validation performed by means of bootstrap sampling, aimed at obtaining a large number of samples randomly drawn with replacement from the original population [[6]Altman D.G. Andersen P.K. Bootstrap investigation of the stability of a Cox regression model.Stat Med. 1989; 8: 771-783Crossref PubMed Scopus (249) Google Scholar]. This way, the model developed in the bootstrap sample is validated in the original sample and the procedure is repeated, usually at least 100 times [[6]Altman D.G. Andersen P.K. Bootstrap investigation of the stability of a Cox regression model.Stat Med. 1989; 8: 771-783Crossref PubMed Scopus (249) Google Scholar]. Recent simulation studies recommend, in the absence of substantially sized external cohorts, internal validation with bootstrapping, because this statistical procedure is able to simultaneously validate both the model building process and its performance in a broad range of random samples [[5]Steyerberg E.W. Bleeker S.E. Moll H.A. Grobbee D.E. Moons K.G. Internal and external validation of predictive models: a simulation study of bias and precision in small samples.J Clin Epidemiol. 2003; 56: 441-447Abstract Full Text Full Text PDF PubMed Scopus (400) Google Scholar]. We think that prognostication of TACE and indication to retreatment should rely mostly on well-known clinical and tumor factors that have to be weighed properly with sounded estimation methods. On this ground, refinements of the current point scores computation seems advisable. This is a study supported by National Cancer Institute institutional grants (Hepato-Oncology Project, 5x1000) and the Italian Association for Cancer Research (AIRC). The authors who have taken part in this study declared that they do not have anything to disclose regarding funding or conflict of interest with respect to this manuscript. Antonio Facciorusso wrote the letter and performed the analysis; Antonio Facciorusso, Sherrie Bhoori and Carlo Sposito collected the data; Vincenzo Mazzaferro revised the manuscript. All the authors approved the final version of the manuscript.