Reply to the letter to the editor ‘ESMO-MCBS v1.1: statistical and patient relevant shortcomings’ by Emprechtinger et al.
Authors: N.I. Cherny, U. Dafni, M. Piccart, N.J. Latino, J.-Y. Douillard, J. Bogaerts, D. Karlis, P. Zygoura, G. Pentheroudakis, J. Tabernero, C. Zielinski, E.G.E. de Vries
Journal: Annals of Oncology (2018)

First of all we would like to thank Emprechtinger et al. [1.Emprechtinger R. Grossmann N. Wild C. ESMO-MCBS v1.1: statistical and patient-relevant shortcomings.Ann Oncol. 2018; 29: 1070-1071Abstract Full Text Full Text PDF PubMed Scopus (2) Google Scholar] for their ongoing interest in the ESMO-Magnitude of Clinical Benefit Scale. In their letter, they present issues relating to the validation paper published in ESMO Open [2.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (13) Google Scholar] and to the newly published ESMO-MCBS v1.1 [3.Cherny N. Dafni U. Bogaerts J. et al.ESMO-Magnitude of Clinical Benefit Scale Version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (220) Google Scholar]. We will address each of these in turn: We appreciate their statement that, by the statistical simulations, the instrument was shown to have the desired characteristics. The authors raise four concerns that require redress:1.They imply that the simulations use ideal (non-representative) scenarios highlighting the use of random censoring: The normal basis for simulations across a range of scenarios is to apply the random censoring assumption. ESMO-MCBS evaluates the benefit estimated from the primary analysis of a randomised, controlled trial, which is based on the ‘intention-to-treat’ principle, as widely recommended [4.Gupta S.K. Intention-to-treat concept: a review.Perspect Clin Res. 2011; 2: 109-112Crossref PubMed Google Scholar, 5.Moher D. Hopewell S. Schulz K.F. et al.CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.BMJ. 2010; 340: c869Crossref PubMed Scopus (2952) Google Scholar, 6.Schulz K.F. Altman D.G. Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials.Ann Intern Med. 2010; 152: 726-732Crossref PubMed Scopus (2021) Google Scholar]. This analysis reflects the practical clinical scenario, ignoring noncompliance, protocol deviations, withdrawal, and anything that happens after randomisation [4.Gupta S.K. Intention-to-treat concept: a review.Perspect Clin Res. 2011; 2: 109-112Crossref PubMed Google Scholar]. The simulations in Dafni et al. correspond to general trial scenarios where informative ‘missingness’ is not taken into account. It is a very rare occasion, if any, that informative censoring is accounted for, in the primary analysis of a randomised, controlled trial. Thus, we maintain that the simulation scenarios are representative pragmatic scenarios of evaluated randomised, controlled trials.2.The authors claim that the ESMO-MCBS misinterprets confidence intervals and this weakens the conclusions derived from the simulation studies: We fully agree with the elementary principles of confidence intervals outlined in the letter, and we emphasise that these well-known principles are fully respected by our approach. Confidence interval estimation encompasses hypothesis testing, and their use in presenting research results is widely recommended in guidelines for the reporting of clinical research, e.g. the CONSORT statement [5.Moher D. Hopewell S. Schulz K.F. et al.CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.BMJ. 2010; 340: c869Crossref PubMed Scopus (2952) Google Scholar, 6.Schulz K.F. Altman D.G. Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials.Ann Intern Med. 2010; 152: 726-732Crossref PubMed Scopus (2021) Google Scholar, 7.Zou G. Donner A. Construction of confidence limits about effect measures: a general approach.Stat Med. 2008; 27: 1693-1702Crossref PubMed Scopus (133) Google Scholar]. The relative benefit rule of the ESMO-MCBS uses the ‘duality property between confidence intervals and hypothesis tests’ [8.Neyman J. Outline of a theory of statistical estimation based on the classical theory of probability.Philos Trans R Soc Lond A. 1937; 236: 333-380Crossref Google Scholar]. In the development of the ESMO-MCBS the experts felt that for accreditation of maximal relative benefit an Hazard Ratio (HR) of ≤0.65 (and in other settings, of ≤0.70) was appropriate. Thus, the true parameter of interest was not random but of a very specific value, HR = 0.65, and the question was on whether it could be rejected or not. Two possible results could stem from this rule:A.If the value of 0.65 is lower than the lower limit of the interval, and thus it is rejected, the treatment benefit is not considered maximal. Thus, a decision is made, only when a positive result is achieved, i.e. the conclusion is to reject HR ≤ 0.65, (similar to a one-sided test at alpha = 0.025).B.If the value of 0.65 is included in the interval and thus it cannot be rejected, lenience leads to not penalising this study, by adhering to the statement: ‘absence of evidence is not evidence of absence’ [9.Altman DG. Why we need confidence intervals.World J Surg. 2005; 29: 554-556Crossref PubMed Scopus (53) Google Scholar, 10.Altman D. Bland J.M. Confidence intervals illuminate absence of evidence.BMJ. 2004; 328: 1016-1017Crossref PubMed Google Scholar, 11.Altman D.G. Bland J.M. Statistics notes: absence of evidence is not evidence of absence.BMJ. 1995; 311: 485.Crossref PubMed Scopus (1048) Google Scholar]. So, if the hypothesis of interest (HR ≤ 0.65) could not be rejected, a study was not penalised based on this criterion and subsequently the only decisive criterion for the dual rule was the magnitude of absolute benefit. The statement ‘A magnitude of relative benefit included in the interval of HR is a plausible true value for the treatment benefit and should be recognised as such.’, refers to the fact that ‘absence of evidence’ that HR ≤ 0.65 is not ‘evidence of absence’ of a relative benefit of this magnitude, since this value is not rejected [9.Altman DG. Why we need confidence intervals.World J Surg. 2005; 29: 554-556Crossref PubMed Scopus (53) Google Scholar, 10.Altman D. Bland J.M. Confidence intervals illuminate absence of evidence.BMJ. 2004; 328: 1016-1017Crossref PubMed Google Scholar, 11.Altman D.G. Bland J.M. Statistics notes: absence of evidence is not evidence of absence.BMJ. 1995; 311: 485.Crossref PubMed Scopus (1048) Google Scholar]. We do not talk about a range of plausible values, but on whether the specific value of interest 0.65 (or 0.70) is compatible with the observed data, after failing to reject it! The answer is obviously yes! This approach is consistent with the CONSORT statement ‘The confidence interval may be interpreted as the range of values for the treatment effect that is compatible with the observed data.’ [5.Moher D. Hopewell S. Schulz K.F. et al.CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.BMJ. 2010; 340: c869Crossref PubMed Scopus (2952) Google Scholar].3.Regarding conclusions of single studies and especially in ‘Small single studies’: Nowhere does the ESMO-MCBS support or endorse inadequately powered clinical studies or small studies. On the contrary, we have demonstrated that the characteristics of inclusiveness and discernment are more robust in more highly powered studies. However, referring to the duality property above, it is unreasonable to suggest that no statistical test can be carried out, or appropriately interpreted, in a single study. Furthermore, where multiple studies have been aggregated into well-designed and reported meta-analyses, the ESMO-MCBS is again applicable and it may identify or refute benefits identified or not among the contributing previously graded smaller contributing studies. This will be well illustrated in our forthcoming evaluation of the ESMO-MCBS in early stage breast cancer studies.4.Regarding the simulations in Figure 5 of the statistical validation paper. The authors of this letter ignore the full range of scenarios evaluated (as well as the other simulation data) and choose to narrow their discussion to a single scenario which piques their concern regarding discernment. The upper row of Figure 5a and 5b of the simulation paper includes panels illustrating trials studying treatments with a true HR of 0.9, over a range of targeted HR ranging from 0.65 to 0.9. When the targeted HR equals 0.75 (the focus of the authors concern), the figures illustrate that among those studies achieving statistical significance (approximately 20%), a smaller overall proportion (approximately 4.5% total) will reach the relative benefit threshold in studies with 90% power, than in studies with 80% power (approximately 12% total). This section of the figure illustrates the potential for false-positive results in ascribing high benefit based on relative benefit criteria alone in relatively small studies. This potential is acknowledged and is present also when using a point estimate threshold. Indeed, as has been previously stated on many occasions now, this is the reason that the ESMO-MCBS incorporates absolute benefit thresholds as a part of the dual rule in the ESMO-MCBS, to reduce the likelihood of false-positive results. Throughout this article, we recognise and illustrate that there will be some false-positive and false-negative results in the grading system of the ESMO-MCBS since ‘no set of rules can accommodate every possible situation, acknowledging that no approach will be perfect’ [2.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (13) Google Scholar]. Extensive simulations, however, have shown that the behaviour of the ESMO-MCBS rules is preferable compared with all other possible rules that have been hitherto suggested. Isolated criticism that focuses on acknowledged shortcomings, without considering the full range of evaluations undertaken and without an evidence-based counterproposal does not advance the issue. A proposal of something more robust would be appreciated. Indeed we invite scientists to propose and test alternatives that provide more robust solutions and, if verified by testing, to present them with the supporting evidence as we have done with our published validation experience. We believe that such proposals, based on data-driven evidence, would propel the field forward. The authors raise six concerns that require redress:1.Regarding toxicity scoring: The authors describe that the chance that adverse events and increased toxicity are detected decreases with (i) small sample size and (ii) due to the ESMO-MCBS requirement of significant results at a = 0.05, since if a P-value is not below 0.05, this is not evidence for no effect. Issues related to toxicity evaluation in the ESMO-MCBS are currently under review, and we welcome their criticism which will be given due consideration in the ongoing review process.2.Regarding small studies: The authors remark that we mentioned that the ESMO-MCBS v1.1 will incorporate a disclaimer for small-sized phase II studies [2.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (13) Google Scholar]. The disclaimer addresses only phase II studies, since we did not expect and have not yet encountered this issue in adequately powered phase III studies. We agree that the limited predictive reliability of phase II studies when subsequently tested in a phase III setting is a concern. It is for this reason that we highlight data derived from randomised phase II studies (see supplementary files) [3.Cherny N. Dafni U. Bogaerts J. et al.ESMO-Magnitude of Clinical Benefit Scale Version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (220) Google Scholar] and that the grading for single-arm phase II studies, using form 3, is capitated. Evidence of benefit derived from randomised phase II studies is not currently penalised, but rather it is highlighted for cautious consideration by end-users of the scale. This issue will be subject to further deliberation by the ESMO-MCBS Working Group.3.Regarding adaptions for sources of bias: Here, the authors express concerns regarding informative censoring and control arms. With regard to informative censoring, we acknowledge that the missing data that occur when patients do not complete the study (and drop-out without further measurements) are commonplace. These may be random or non-random. When missing data are not random, this constitutes informative censoring, and this compromises the credibility of generated findings. While there are applicable methods for handling informative censoring, these are not universally accepted or used in the primary analysis of the randomised, controlled trials [6.Schulz K.F. Altman D.G. Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials.Ann Intern Med. 2010; 152: 726-732Crossref PubMed Scopus (2021) Google Scholar, 12.Piaggio G. Elbourne D.R. Pocock S.J. et al.Reporting of noninferiority and equivalence randomized trials: extension of the CONSORT 2010 statement.JAMA. 2012; 308: 2594-2604Crossref PubMed Scopus (671) Google Scholar]. This underscores the importance of careful scrutiny of data and information regarding patients in the ITT groups with premature withdrawal from participation. With regard to the remarks regarding control arms not reflecting the best available treatment, it is important to emphasise that the Declaration of Helsinki [13.World Medical Association. World Medical Association Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects. In 64th WMA General Assembly, Fortaleza, Brazil, October 2013.Google Scholar] addresses this matter directly where it states ‘The benefits, risks, burdens and effectiveness of a new intervention must be tested against those of the best proven intervention(s), except in the following circumstances: 1) Where no proven intervention exists, the use of placebo, or no intervention, is acceptable; 2) Where for compelling and scientifically sound methodological reasons the use of any intervention less effective than the best proven one, the use of placebo, or no intervention is necessary to determine the efficacy or safety of an intervention and the patients who receive any intervention less effective than the best proven one, placebo, or no intervention will not be subject to additional risks of serious or irreversible harm as a result of not receiving the best proven intervention. Indeed, extreme care must be taken to avoid abuse of the best proven intervention as control arm’. This consideration is addressed in our text describing factors that may skew scoring [14.Cherny N.I. Sullivan R. Dafni U. et al.A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).Ann Oncol. 2015; 26: 1547-1573Abstract Full Text Full Text PDF PubMed Scopus (443) Google Scholar] and it is taken into account in the ESMO-MCBS scoring process. At the first step of the evaluation process, before proceeding to scoring the outcome, the strength of the randomised evidence is established. This obviously includes a critical evaluation of the control arm [2.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (13) Google Scholar, 14.Cherny N.I. Sullivan R. Dafni U. et al.A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).Ann Oncol. 2015; 26: 1547-1573Abstract Full Text Full Text PDF PubMed Scopus (443) Google Scholar]. It is important to emphasise that ESMO-MCBS grades the added benefit that is observed as compared with a contemporaneous best standard control arm and the results generated are NOT for a medication but rather they are specific to a particular trial. Consequently, we have observed that the added value of an add-on treatment may diminish as the standard control arm improves. This is well illustrated by the diminishing added benefit of bevacizumab in metastatic colorectal cancer when used with irinotecan, leucovorin, 5-flurouracil (ILF, where it was large) as compared with leucovorin, 5-fluorouracil, oxaliplatin (where it was very limited).4.Regarding an upgrade in progression free survival scores when there has been early stopping and crossover: We apologise if we have not been adequately clear regarding this issue and we will re check our use of language. To be unambiguous, this upgrade, in case of crossover, is only applicable where there has been an interim analysis that has demonstrated an overall survival (OS) advantage that is sufficient to justify early stopping of the trial. Since this decision is based on an interim analysis of OS, without final data including median survival, it precludes grading with form 2a.5.Regarding the scoring of studies based on progression-free survival: The authors have a number of remarks with regards to the assessment of progression-free survival (PFS) and modification of the preliminary score. We are, of course, acutely aware of the many issues rendering PFS a limited surrogate for either OS or improved quality of life and for this reason maximal preliminary scores are capitated at a maximal grade of 3 and toxicity penalties are more stringent (in form 2b) than they are for OS studies (in form 2a) [3.Cherny N. Dafni U. Bogaerts J. et al.ESMO-Magnitude of Clinical Benefit Scale Version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (220) Google Scholar]. Indeed, one can reasonably interpret a PFS score of 3 as one of ‘indeterminate benefit’, which may evolve to substantial or low depending on what further evidence is produced by subsequent OS or quality of life data. This reflects the tension in trying to optimise the balance between being inclusive (high sensitivity: true positive crediting true big benefit) and discerning (high specificity: avoiding false-positive over accrediting studies with true small benefit, as studies with large benefit). Failure to confirm surrogacy of PFS for OS does not preclude the possibility of quality of life surrogacy. When surrogacy of PFS is disproven for both OS and quality of life, attainable scores are penalised and are appropriately very low.6.Regarding the scoring of single-arm studies: Finally, the authors are extremely critical about our proposal to score single arm studies. Single-arm data are currently used for the FDA and EMA registration of new medication. This has forced the need for an approach to grading as doctors and patients have to know how to interpret this data even though the interpretation of single-arm data are challenging. The limited predictive reliability of phase II studies when subsequently tested in a phase III setting is acknowledged and is a concern. For this reason, the grading for single-arm phase II studies, using form 3, is capitated with a maximal grade of 3. Indeed, as with PFS studies, one can reasonably interpret a single-arm score of 3 as one of ‘indeterminate benefit’, which may evolve to substantial or low depending on what further evidence is produced by confirmatory phase III studies. None declared.