ESMO-MCBS v1.1: statistical and patient-relevant shortcomings
Authors: R. Emprechtinger, N. Grössmann, C. Wild
Journal: Annals of Oncology (2018)

We appreciate the publication of the statistical simulation of the ESMO-Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules [1.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar]. Dafni et al. [1.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar] were able to show that the instrument has, under ideal scenarios (e.g. random censoring), the desired characteristics. However, the study also shows that important points of criticism [2.Wild C. Grossmann N. Bonanno P.V. et al.Utilisation of the ESMO-MCBS in practice of HTA.Ann Oncol. 2016; 27: 2134-2136Abstract Full Text Full Text PDF PubMed Scopus (34) Google Scholar] were justified. Not all of these flaws are addressed sufficiently in the new ESMO-MCBS version 1.1 [3.Cherny N.I. Dafni U. Bogaerts J. et al.ESMO-Magnitude of Clinical Benefit Scale version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (325) Google Scholar]. We wish to respond to these issues to promote additional improvements for an already useful tool. One important shortcoming is that the simulation study is on a weak basis due to the misinterpretation of the confidence interval (CI). The relative benefit (RB) rule uses the lower limit of the 95% CI of the hazard ratio (HR) and compares it to specific thresholds. The authors stated, ‘a magnitude of RB included in the interval of HR is a plausible true value for the treatment benefit and should be recognised as such’ [1.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar]. This assumption is a common misconception of the use of the CI in frequentist statistics. For instance, Hoekstra et al. [4.Hoekstra R. Morey R.D. Rouder J.N. Wagenmakers E.-J. Robust misinterpretation of confidence intervals.Psychon Bull Rev. 2014; 21: 1157-1164Crossref PubMed Scopus (218) Google Scholar] explicitly mention in their article named ‘Robust misinterpretation of confidence intervals’ that a common failure is to assume that the CI is a set of plausible values. The CI is in fact a statement about the precision of an instrument. If an experiment is repeated 100 times, we would expect that 95 of our 95% CI cover the true value. Further, the CI asks for the probability of a result under the null hypothesis (H0) but it is not possible to make any probability statements on the true values of a parameter based on a CI. Consequently, the CI offers no information on the strength of an effect nor is it a range of ‘plausible true values for the treatment benefit’ [4.Hoekstra R. Morey R.D. Rouder J.N. Wagenmakers E.-J. Robust misinterpretation of confidence intervals.Psychon Bull Rev. 2014; 21: 1157-1164Crossref PubMed Scopus (218) Google Scholar]. The misinterpretation of the CI affects the simulation study and the interpretation of the results. Drawing conclusions of single studies is generally risky, especially if they have a small sample size [5.IntHout J. Ioannidis J.P.A. Borm G.F. Obtaining evidence by a single well-powered trial or several modestly powered trials.Stat Methods Med Res. 2016; 25: 538-552Crossref PubMed Scopus (46) Google Scholar]. However, the simulation study proofs that the ESMO-MCBS unfairly favours smaller studies with over-optimistic expectations: Figure 5 in Dafni et al. [1.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar] shows that a drug with a true HR of 0.9 and a targeted HR of 0.75 has an approximately 20% chance to reach statistical significance. The majority of these statistical significant studies achieves the acceptance for maximal relative benefit. Even more problematic, studies with 90% power are treated worse than studies with 80%. Lower test power has a reduced chance to cover the true effect. Consequently, the chance that such a study reflects the true effect size is reduced. Additionally, the chance that adverse events and increased toxicity are detected decreases with lower test power due to a smaller sample size. Aggravating this situation, the ESMO-MCBS requires statistically significant results of these outcomes to lead to a change of the final score. Again, this is associated to an incorrect use of frequentist statistical methods. The fact that the P-value is not below 0.05 is not an evidence for no effect as implicitly assumed by this procedure [6.Altman D.G. Bland J.M. Statistics notes: absence of evidence is not evidence of absence.BMJ. 1995; 311: 485.Crossref PubMed Scopus (1180) Google Scholar]. Dafni et al. [1.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar] mentioned that the ESMO-MCBS v1.1 will incorporate a disclaimer for small-sized studies that have a higher risk for false-positive inclusions. However, the disclaimer is only intended for phase II studies. Further, we question, if the sole introduction of a disclaimer is sufficient at all. Other adaptions for important sources of bias are missing, such as how to deal with informative censoring or control arms not reflecting the best available treatment. Unfortunately, the ESMO-MCBS v1.1 introduces new problems. For instance, the new adjustment for progression-free survival (PFS) studies: ‘Upgrade 1 level if study had early crossover because of early stopping or crossover based on detection of survival advantage at interim analysis’. The authors stated that this upgrade should not apply to situations in which crossover was caused due to early stopping based on PFS advantage at interim analysis [3.Cherny N.I. Dafni U. Bogaerts J. et al.ESMO-Magnitude of Clinical Benefit Scale version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (325) Google Scholar]. However, since the two conditions are conjunct by an ‘or’ we would expect that a crossover due to early stopping by whatever reason (e.g. improved PFS) is sufficient for an upgrade. Additionally, the adjustment is inconclusive since form 2a of the ESMO-MCBS has to be used for studies that have shown improved survival. There are further issues that are still not sufficiently tackled in v1.1 [3.Cherny N.I. Dafni U. Bogaerts J. et al.ESMO-Magnitude of Clinical Benefit Scale version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (325) Google Scholar]. For example, the assessment of PFS is prone to bias: Toxicity is positively associated with increased PFS [7.Abola M.V. Prasad V. Jena A.B. Association between treatment toxicity and outcomes in oncology clinical trials.Ann Oncol. 2014; 25: 2284-2289Abstract Full Text Full Text PDF PubMed Scopus (29) Google Scholar], indicating that PFS is not necessarily an indicator for increased quality of life (QoL). Since the reporting of outcomes such as toxicity is regularly biased, especially if the primary outcome showed an effect in the intended direction [8.Vera-Badillo F.E. Shapiro R. Ocana A. et al.Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer.Ann Oncol. 2013; 24: 1238-1244Abstract Full Text Full Text PDF PubMed Scopus (92) Google Scholar], the risk that the assessment of PFS might be misleading, is high. The downgrade in the ESMO-MCBS v1.1 when the drug only leads to improved PFS without showing improved QoL partially tackles this issue. Inconsequently, there is no downgrade applied if QoL is not reported at all. This is a highly important issue, since studies that had a crossover because of early stopping and no assessment of QoL can now reach a score of 4. Another issue with the assessment of PFS is that negative results in overall survival (OS) are still ignored in the ESMO-MCBS v1.1. OS is considered as the gold standard for clinical benefit and should receive more weight than PFS, which is just a surrogate. An improved PFS alongside a worsened OS is a strong hint for a drug that causes more harm than good. This needs to be taken into account when rating the clinical benefit. Thus, we recommend a critical revaluation of the assessment of PFS in the ESMO-MCBS v1.1. Finally, we criticise the scoring of single-arm studies investigating orphan diseases. The conclusions that might be drawn of studies with such a low quality are very limited. Although the threshold for meaningful clinical benefit cannot be reached by these studies a rating of the clinical benefit based on such unreliable evidence is likely proven to be false in future trials. None declared.