Breast-conservation therapy involves a number of physician decisions not required for mastectomy, including the selection of patients for breast conservation, the amount of tissue to be removed from the area surrounding the tumor, the details of administering radiation, and so forth. (See Sacks and Baum, 1993; Winchester and Cox, 1992; Harris et al., 1990; NIH, 1991.) And since breast-conservation therapy involves radiation, its implementation would logically vary depending upon the availability of appropriate radiation equipment and expertise in operating that equipment. Breast-conservation therapy also requires “careful long-term breast monitoring” in order to identify and treat local recurrences in the breast that was subjected to lumpectomy (NIH, 1991). All these treatment-implementation factors can potentially affect breast-conservation patients’ survival—and may not be the same in randomized studies and in medical practice. At least, the typical treatments given in day-to-day medical practice could fall short of the presumably consistent and high-quality treatments provided by a single prestigious research center, such as the National Cancer Institute (NCI). Some randomized studies are conducted at single centers, while others are conducted at diverse sites (that is, multiple centers). To more closely approximate day-to-day medical practice, multicenter studies have, in some instances, intentionally involved “community surgeons.” For this reason—and also because the treatments given in multicenter studies may vary from one center to another—multicenter studies’ results may more closely approximate results in medical practice than the results of single-center studies at prestigious institutions. But unlike medical practice, both single-center and multicenter studies stipulate that participating physicians follow a set of prespecified procedures. The question remains, then, as to whether or not breast conservation therapy has produced results similar to mastectomy in day-to-day medical practice. Randomized clinical studies are the “gold standard” of medical research. Random assignment essentially equates patients in the two treatment groups. Because the two groups should not differ on variables related to cancer survival, their outcomes can be directly compared, and any difference in survival can be attributed to the difference in treatment. In contrast, the statistical analysis of cases from a medical practice database represents a potential “window” on how well breast-conservation therapy has, in fact, worked in community medical practice. But the results of such analyses may be less conclusive because of their vulnerability to hidden selection bias. (See Byar, 1980; Office of Technology Assessment, 1994.) Briefly, in day-to-day medical practice, patients and physicians freely choose between treatments; a database analyst must, therefore, attempt to control for the potentially differing characteristics of patients who received breast-conservation therapy and those who received mastectomy. In this report, we have made all possible efforts to minimize the impact of selection bias, as described below. The analyses presented here are based on a unique combination of meta-analysis (to summarize randomized studies’ results), statistical analysis of records from a medical practice database, and cross design comparison of results. To our knowledge, this is the first time such an approach has been used in the area of breast cancer treatment. In all analyses presented here, breast-conservation therapy is defined as including lumpectomy, nodal dissection, and radiation. With respect to time frame, the randomized studies enrolled and treated patients from 1972 to 1989, and the medical practice cases selected for this analysis were diagnosed from 1983 to 1985. Because of limitations in the medical practice database (discussed in appendix I), all our analyses use the outcome criterion of 5-year survival and examine node-negative patients only. The medical practice data were drawn from the National Cancer Institute’s Surveillance, Epidemiology, and End Results (SEER) database. SEER archives records for almost all cancer patients residing in five states— Connecticut, Hawaii, Iowa, New Mexico, and Utah—and four metropolitan areas—Atlanta, Detroit, San Francisco-Oakland, and Seattle -Puget Sound. (See Hankey et al., 1992.) Our analysis consisted of three major steps. In step 1, we performed a meta-analysis to summarize randomized studies’ results and obtain summary figures that can be compared to medical practice results. We conducted meta-analyses separately for the single-center studies and for the more generalizable multicenter studies to determine if similarity of survival following breast-conservation therapy and mastectomy holds for both kinds of studies. In step 2, we obtained information on the survival of breast-conservation and mastectomy patients in day-to-day medical practice. Specifically, from the SEER database, we drew records for a relatively homogeneous set of patients who, on the basis of several characteristics, were comparable to those enrolled in randomized studies. For this group of SEER patients, we conducted an analysis of survival following breast-conservation therapy and mastectomy. SEER results were adjusted for tumor size and several other variables so that patients who had received breast-conservation therapy would be “matched” to those who had received mastectomy. The matching was intended to minimize the effects of differing characteristics of patients who received breast-conservation therapy and mastectomy. In addition, a sensitivity analysis was performed to check for selection bias on life-threatening factors unrelated to cancer (such as heart-disease). In step 3, we compared (1) the summary results for the single-center and multicenter randomized studies to (2) the results of our analysis of cases selected from the SEER medical practice data. We also considered the logic of our analyses and, in particular, whether the resulting evidence was sufficient to conclude that—in day-to-day medical practice— breast- conservation therapy has been followed by survival similar to that observed for mastectomy. Throughout step 3, we drew on the principles of “cross design synthesis.” In this report, we use the term “similar” when the observed difference between the survival rates (1) is not statistically significant and (2) has an absolute value of less than 1.5 percentage points. Conversely, when a comparison of survival rates shows a difference of 1.5 percentage points or larger—and that difference is also statistically significant—we state that one rate is higher (or lower) than the other. Step 1 (the analysis of randomized studies) began with the identification of relevant single-center and multicenter studies through bibliographic searches and a survey of U.S. breast cancer researchers. Our inclusion criteria were as follows: randomization of enrolled patients to alternative treatments—breast-conservation therapy or mastectomy; breast-conservation therapy that included lumpectomy, nodal dissection, and radiation; no confounding treatments (such as the administration of an additional therapy to one treatment group); availability of 5-year survival rates by treatment group among node-negative patients (either previously published in a scholarly research journal or provided at our request); and published in English (if a non-U.S. study). Six studies—three single-center and three multicenter studies—met these criteria. (See table 1.) Almost 2,500 node-negative breast cancer patients were enrolled and treated in these randomized studies. The treatment effect—that is, the effect of breast-conservation therapy relative to mastectomy—is represented by a comparison of survival following breast-conservation therapy to survival following mastectomy. (See table 2.) by subtracting the 5-year survival rate for mastectomy patients from the 5-year survival rate for breast-conservation patients to determine the difference between the rates; and by calculating the odds ratio (dividing the odds of surviving with breast-conservation therapy by the odds of surviving with mastectomy). As indicated in table 2, the breast-conservation and mastectomy treatment groups experienced similar survival rates in each of the studies, and the odds ratios are close to 1 (the point of equivalence). The confidence intervals for the odds ratios all overlap 1, indicating no statistically significant difference in survival odds for the two treatments. However, the confidence intervals surrounding these estimates are quite broad, indicating a lack of precision in the individual-study estimates. (The U.S.-NSABP figures in table 2 are taken from recalculations published by an NCI contractor in March 1994. The recalculations were published following charges of fraudulent data collection at one U.S.-NSABP center; they exclude the data from that center.) A meta-analysis combining the results for node-negative patients across studies gives more precise estimates of the treatment effect. Table 3 shows meta-analysis results summarizing the treatment effect for single-center studies, multicenter studies, and both types of studies taken together. In addition, table 3 shows meta-analysis results calculated in two ways: (1) including the U.S.-NSABP recalculations published in March 1994 and (2) omitting U.S.-NSABP results entirely. Similar rates of 5-year survival characterized the breast-conservation therapy and mastectomy groups—not only in single-center studies (93.7 percent for breast-conservation patients and 93.7 percent for mastectomy patients), but also in multicenter studies, which may more closely approximate medical practice (88 percent for breast conservation and 88 percent for mastectomy, omitting U.S.-NSABP). Again, the odds ratios are close to 1, and the confidence intervals all overlap 1, indicating no statistically significant difference for any group of randomized studies. Finally, referring again to tables 2 and 3, the 5-year survival rates appear to be higher in single-center studies than in multicenter studies. This could be because of more effective treatments in single-center studies, varying tumor-size limits across the studies, or hidden cross-study differences in patient prognoses prior to treatment. (Step 3 presents more precise comparisons of combined-treatments survival rates.) Because the purpose of this report is to determine whether the treatment effect in day-to-day medical practice corresponds to the treatment effects observed in the single-center and multicenter studies, we would ideally “compare like with like.” Therefore, step 2 (analysis of the medical practice data) began with the selection of SEER patients who, on the basis of their characteristics, would have been covered by randomized studies. Table 4 shows the specific criteria we used in selecting SEER cases; the resulting SEER dataset included 5,326 cases that we believe are at least roughly comparable to the participants in randomized studies. (Appendix I assesses the kinds of patients who participated in randomized studies and discusses SEER cases lost to follow-up.) As described below, our statistical analysis of the selected SEER cases used “propensity-score” adjustments (Rosenbaum and Rubin, 1984) that essentially “matched” the kinds of patients who received breast-conservation therapy and mastectomy on demographic characteristics and tumor size. Using these adjustments, we found that, on average, similar patient survival followed the two treatments. To achieve matched groups of patients for the two treatments, the 5,326 SEER cases were first divided into five quintiles, as shown in table 5. Patients were assigned to these quintiles based on their propensity scores, which were calculated to indicate each patient’s likelihood of receiving breast-conservation therapy. Patients in the first quintile shown in table 5 have very low propensity scores; that is, they are the kinds of patients who were quite unlikely to receive breast-conservation therapy. (An example of a patient with an extremely low propensity score would be a woman in her sixties, living in Iowa, diagnosed in 1983—the earliest year examined here—with a tumor sized 3 to 4 cm.) By contrast, patients assigned to each successive quintile were more likely to receive breast-conservation therapy. (An example of a patient with a relatively high propensity score would be under 40 years old, non-Asian, living in the San Francisco- Oakland or the Seattle-Puget Sound area and diagnosed in 1985—the most recent year examined—with a very small tumor.) In table 5, 5-year survival estimates are shown separately for breast- conservation patients and for mastectomy patients in each quintile. Within each quintile, patients are homogeneous, and the survival rates for the two treatments represent an estimate of the treatment effect for that quintile. The bottom rows of table 5 show the overall survival rates used to calculate the treatment effect for all selected SEER cases taken together. These summary rates, which are termed “adjusted across quintiles,” are clearly similar to each other: 86.3 percent for breast-conservation therapy and 86.9 percent for mastectomy. The adjusted breast-conservation rate (86.3 percent) was calculated by combining the five separate quintile survival rates for breast-conservation patients—giving each of the five rates an equal weight of one-fifth. The adjusted mastectomy rate (86.9 percent) was calculated using analogous procedures. Thus, the adjusted survival rates are based on “matched” treatment groups; that is, the kinds of patients who were unlikely to receive breast-conservation therapy contribute equally to the breast- conservation and the mastectomy survival estimates—as do the kinds of patients who were much more likely to receive breast- conservation therapy. In this way, selection bias on measured variables was minimized. As shown in table 6, the difference between the adjusted 5-year survival estimates for breast-conservation and mastectomy patients is just six-tenths of a percentage point, the odds ratio is relatively close to 1, and the confidence interval overlaps 1, indicating no statistically significant difference. Thus, on average, the two treatments appear to produce similar results in day-to-day medical practice. However, referring again to table 5, the results shown for quintile 3 do not meet our criteria for use of the term “similar” because the observed (nonsignificant) difference between the survival rates is greater than 1.5 percentage points. According to our criteria, this nonsignificant pattern should be regarded as inconclusive. The propensity-score adjustments were intended to minimize selection bias on measured variables, such as tumor size and demographic characteristics. However, noncancer-related life-threatening illnesses or conditions, such as serious heart disease, were not measured in the SEER data and therefore could not be included in the propensity score. Such illnesses or conditions might at once influence treatment selection and limit 5-year survival—and could represent a form of selection bias not accounted for by the propensity scores. SEER data does, however, include codes for cause of death. Therefore, it was possible to check for selection bias on illnesses and conditions not related to cancer in the following way: We performed a sensitivity analysis in which we reproduced table 5 omitting patients who were coded as having died of illnesses and conditions unrelated to cancer within the 5-year interval. As indicated in table 7, with those patients omitted, the difference in survival following breast-conservation therapy and mastectomy is, on average, again within 1.5 percentage points of zero, and it is not statistically significant. At the same time, however, the breast-conservation and mastectomy survival rates within each of the first three quintiles fall short of our criteria for similarity; specifically, although the differences between the breast-conservation and mastectomy survival rates for these quintiles are not statistically significant, each is slightly larger than 1.5 percentage points. According to our criteria, the separate results for quintiles 1 through 3 are inconclusive. Yet when results for these quintiles are considered together—and compared to the results for quintiles 4 and 5—there are two potential implications: (1) breast-conservation therapy may not have been quite as effective as mastectomy for some of the patients who were less likely to receive it—such as those who resided in “low-lumpectomy” areas (in which breast-conservation therapy was relatively uncommon); and (2) breast conservation has been at least as effective as mastectomy for those who were most likely to receive it. There are various possible explanations for this nonsignificant pattern, based on the different components of the propensity score. (See appendix I.) However, at the present time, exploratory analyses would be difficult, at best, because within the rather homogeneous group of patients examined in this report, there is a relatively small number of breast-conservation patients (1,072) and only about one-third of them (340) fall into quintiles 1 through 3. Step 3 consists of cross design comparisons and a consideration of the evidence. An informal comparison of the summary results for step 1 and step 2 suggests that the average treatment effect estimated in the statistical analysis of selected SEER cases is similar to the effects observed in the single-center and multicenter randomized studies. The more precise comparisons in tables 8 and 9 show that, quantitatively, this is indeed the case. But do these data constitute sufficient evidence to conclude that the effectiveness of breast-conservation therapy in day-to-day medical practice really is, at least on average, similar to its effectiveness in randomized studies? To address this issue, we considered (1) the potential differences distinguishing the SEER analysis from single-center and multicenter randomized studies (including the potential for hidden selection bias in the SEER analysis) and (2) the impact that these potential differences might have on the treatment effects we observed. We then used an additional type of cross design comparison as a validity check. Three potential cross design differences could affect comparisons of the treatment effect estimated for the SEER medical practice data to the treatment effects observed in single-center and multicenter randomized studies. These are potential differences in actual treatment effectiveness (SEER versus single-center and multicenter studies), potential differences in patients (again, SEER versus single-center and multicenter studies), which might be related to differences in treatment effectiveness, and lack of randomization in the SEER data versus randomization in the single-center and multicenter studies—which could lead to differences in the estimates of treatment effectiveness. Each of these potential differences could affect the comparison of treatment effects (SEER versus single-center and multicenter studies) in the following ways: If there are real differences in treatment effectiveness (for example, if breast-conservation therapy is less effective than mastectomy in day-to-day medical practice), this would affect the comparison of effects—SEER versus randomized studies. (This is, in fact, the hypothesis we have sought to test.) If there are differences in patients—again SEER cases versus randomized studies—this also could affect the comparison of effects, but only if breast-conservation therapy is, in fact, more or less effective for the particular kinds of patients who were included in the SEER analysis than for the kinds of patients included in the randomized studies. And, the lack of randomization in the SEER data could affect our estimate of the treatment effect in day-to-day medical practice if (1) the kinds of SEER patients who were selected for one treatment had better prognoses than those selected for the other treatment—and (2) this was not corrected as part of our analysis. In the foregoing analyses, our intent was to test for whether the effectiveness of breast-conservation therapy relative to mastectomy was indeed the same in day-to-day medical practice as in single-center and multicenter randomized studies. In comparing the effect of nominally identical treatments across designs, our goal was to identify the first type of difference listed above. We therefore attempted to minimize the influence of each of the other two potential differences. With respect to differences in patients, we selected SEER patients that were at least roughly comparable to those treated in the randomized studies. With respect to selection bias, the fact that we began with a homogeneous group of SEER patients (node-negative, tumors 4 cm or less, age 70 or younger) argues against substantial amounts of bias. We used the propensity-score method to minimize bias on tumor size and on other measured variables. We also conducted a sensitivity analysis to check for selection bias on life-threatening diseases or conditions other than cancer (for example, heart disease)—and found none. Nevertheless, we realize that despite such efforts, some patient differences or some degree of hidden selection bias can persist. The similarity of the average treatment effect observed for the SEER medical practice data and the effects observed for the randomized studies (that is, the results shown in tables 8 and 9) argue that none of the potential differences listed above had a major impact. The most parsimonious interpretation of the data presented in tables 8 and 9 is that breast-conservation therapy is, on average, similarly effective to mastectomy in day-to-day medical practice. Logically, however, it is also possible that if two of three potential cross design differences occurred simultaneously, they could “balance each other out” to produce a false impression of similar treatment effects across designs. Of particular relevance is the possibility that hidden selection bias in the SEER data analysis (specifically, a hidden bias toward selecting better-prognosis patients for breast conservation) could “counterbalance” treatment differences (specifically, less effective breast-conservation therapy in medical practice)—and thus create an impression of similar treatment effects across study designs. We reasoned that an additional indication of the relative effectiveness of treatments across designs would be afforded by a comparison of (1) the combined-treatments survival rate for the SEER analysis to (2) the corresponding rates for single-center and multicenter studies. Logically, the SEER combined-treatments survival rate is not affected by internal selection bias. Thus, if the SEER rate proved to be similar to the corresponding rates in single-center and multicenter studies, this would point to minimal differences both in patients and in treatment effectiveness across the designs. In short, similar combined-treatments survival rates for the selected SEER cases and for a set of randomized studies would support the conclusion of similar overall effectiveness of breast conservation in day-to-day medical practice and in the randomized studies. In contrast, if the SEER combined-treatments survival rate proved to be different from the corresponding rates for randomized studies, a number of interpretations would be possible—including a difference in patients as well as a difference in the general quality of treatments being given. In comparing combined-treatments survival rates across studies, it is necessary to take account of any differences in tumor size—specifically, any differences between the tumor sizes of the selected SEER patients and the patients in the single-center and multicenter randomized studies. This is because tumor size is related to patient survival. As previously noted, four of the six randomized studies had a tumor-size limit of 4 cm, whereas two studies had a limit of 2 cm; the roughly comparable set of SEER patients had a tumor-size limit of 4 cm. The comparison of combined-treatments survival rates is easiest to make for SEER data versus multicenter studies. This is because all multicenter studies had, in effect, the same tumor-size limit (4 cm) and the SEER cases selected for our analyses were also subjected to the 4-cm limit. Therefore, in this section, we separately discuss (1) the comparison of the SEER combined-treatments survival rate to the combined-treatments survival rate for multicenter studies and (2) the corresponding comparison for SEER and single-center studies. Table 10 (first row) shows the combined-treatments 5-year survival rate for the full set of SEER cases used in the foregoing analyses; this rate—86.9 percent (or 87 percent, rounded)—is appropriate for comparison to the multicenter studies. As shown in the bottom row of table 11, the difference in rates is only 1 percentage point and is not significant. The most parsimonious explanation of this result is that, at least on average and with respect to 5-year survival, there are (1) no substantial differences between the patients in our SEER analysis and the patients in the multicenter studies and (2) no large difference between the effectiveness of breast-conservation therapy or mastectomy across the two types of analyses. The comparisons are more complex for SEER versus the single-center studies because two of the three studies had a 2-cm limit. The SEER weighted composite estimate in the last row of table 10 (89.4 percent) combines (1) the survival estimate for the full set of selected SEER cases (4-cm limit) with (2) the survival estimate for the subset of cases defined with a 2-cm limit. (See table 10, note c.) This survival estimate is appropriate for comparison to the single-center studies’ estimate (93.7 percent). From table 11, it is clear that with breast-conservation and mastectomy patients taken together, the 5-year survival rate for patients in single-center randomized studies is higher than the rate for the corresponding SEER estimate—by a difference of 4.3 percentage points, which is statistically significant. The meaning of this finding is unclear. It could be explained by the argument that implementations of treatments in single-center studies are generally better than implementations in multicenter studies or day-to-day medical practice (which seems to be logical). But it could also be explained by hidden selection of patients with better prognoses for the single-center studies. In this report, we examined the relative effectiveness of breast- conservation therapy and mastectomy for patients treated in three contexts: single-center randomized studies, multicenter randomized studies, and day-to-day medical practice. In each context, the summary data indicated that 5-year survival was similar following the two alternative treatments. The best outcomes for both treatments occurred in the single-center studies; however, outcomes for the SEER medical practice patients were comparable to outcomes in the multicenter studies. We recognize that database analyses are vulnerable to hidden selection bias. But we believe such bias is likely to be minimal in the SEER analyses presented here because (1) a homogeneous group of patients was examined, (2) careful adjustments were made for differences in tumor size and demographic characteristics (using the propensity-score method), and (3) a check for possible selection bias on life-threatening factors unrelated to cancer (such as heart disease) reaffirmed our initial conclusion. In addition, the fact that the combined-treatments survival rate was similar in multicenter studies and in the SEER data points to similar levels of treatment effectiveness across these two designs. We caution that this analysis does not prove the absence of selection bias in the SEER analysis—and that these results are limited to the patient population, treatments, and outcome that we were able to examine empirically. Nevertheless, virtually all the evidence that we were able to examine pointed toward the similarity of patient survival following breast-conservation and mastectomy—in day-to-day medical practice as well as in the randomized studies. Only one caveat was suggested by the results of our analyses: A minority of breast-conservation patients—the kinds of patients for whom breast-conservation therapy was relatively unlikely to be used (based on factors such as residence in areas where breast-conservation is relatively uncommon) but who nevertheless did receive it—may have achieved slightly better results with mastectomy. The observed difference, however, was not statistically significant. This report does not examine agency programs; thus, we did not request agency comments. However, we obtained reviewer comments from staff at the National Cancer Institute and the Agency for Health Care Policy and Research; from a number of university-based researchers with expertise in statistics, research methods, or breast cancer; and from investigators in charge of each of the randomized studies. (See appendix II.) We will be sending copies of this report to the Director of the National Cancer Institute and to other interested parties. We will also make copies available upon request. If you have any questions, please call me at (202) 512-2900, or call Robert L. York, Director of Program Evaluation in Human Services Areas, at (202) 512-5885 or Judith A. Droitcour, Assistant Director, at (202) 512-5885. Major contributors to this report are listed in appendix III. Some of the tables in this report present 95-percent confidence intervals in addition to point estimates. These intervals reflect the fact that estimates of the parameter in question (for example, the odds ratio) might fluctuate because of random variation in the data. If the 95-percent confidence interval for an odds ratio includes 1 (the point of equivalent odds), there is no statistically significant difference (at the .05 level) between the odds of survival following breast-conservation therapy and the odds of survival following mastectomy. Similarly, if the 95-percent confidence interval for a difference in percentages includes 0 (the point of equivalence), there is no statistically significant difference between the percentages being compared (at the .05 level). A statistically significant difference is one that is not likely to have occurred by chance alone. The utility of confidence intervals and significance tests is not limited to randomly selected samples. (See Winch and Campbell, 1969.) In comparing patient survival rates—for example, in comparing the 5-year survival rate for breast-conservation patients to the corresponding rate for mastectomy patients—we termed the two rates “similar” when the observed difference between rates was less than 1.5 percentage points (absolute value), and that difference in rates was not statistically significant. (See table I.1.) Nonsignificant pattern (inconclusive owing to a lack of significance) When a comparison of survival rates showed a difference of 1.5 percentage points or larger—and that difference was statistically significant—we used the terms “higher” and “lower.” When survival rates differed by 1.5 percentage points or more—but statistical significance was not attained—we termed the result a nonsignificant pattern. (A nonsignificant pattern is considered inconclusive because of the lack of statistical significance. See table I.1.) This approach recognizes that a high degree of statistical power is required to detect significant differences as small as 1.5 percentage points.Without a high degree of statistical power, we believe it would be inappropriate to term results “similar” merely because of a failure to find a significant difference. With respect to the remaining possibility depicted in table I.1—a difference of less than 1.5 percentage points that is statistically significant—we note that this would not occur except where extremely large samples allowed very precise estimates. Were any findings to fall into this category, the conclusion would be that a real, although relatively small, difference does exist—and has been estimated very precisely. A size-of-difference criterion (cutting point) was used because of the relative imprecision of the estimates, given the existing studies and data. We wished to choose a cutting point that, in our judgment, would represent a difference in survival rates that could reasonably be considered “similar.” Thus, we rejected potential cutting points that seemed too high (such as 5 percentage points) because we believed most patients would not consider survival rates that differed by that amount to be similar. In this context, a criterion of 1 percentage point or less versus a larger difference initially seemed reasonable. We chose 1.5 as the specific cutting point (that is, a difference of less than 1.5 percentage points versus 1.5 or greater) because it was possible to obtain most, though not all, survival estimates rounded to the nearest tenth of a percent. patients in each treatment group. In the area of breast-conservation therapy and mastectomy, the samples in the randomized studies—and in the database analysis presented here—fall short of this number. recognize that it is, to some extent, arbitrary. We do not mean to imply that this figure represents the point at which a particular physician or patient would distinguish between a “meaningful difference” and an irrelevant one. We are also cognizant of the fact that, for every 10,000 patients who receive a treatment characterized by even a 1-percentage-point lower survival rate than an available alternative treatment, there would be 100 deaths that could have been avoided by choosing the other treatment—provided that the observed 1-percentage-point difference is, in fact, a real difference and not merely the result of random variation. In this report, we have avoided use of the term “equivalent” to describe the survival rates observed for breast-conservation and mastectomy patients. A technical reason for this is that to claim “equivalent” survival following the two treatments would require the confidence interval surrounding the difference to be so small that it could be entirely enclosed by a prespecified interval—specifically, one defined such that all values within it would be justifiable as clinical equivalence. (See Fleiss, 1992.) That is, not only would we have to justify a difference of 1.5 percentage points as clinically equivalent, but both the upper and lower bounds of the confidence interval surrounding our estimate of the difference would have to be within 1.5 percentage points of zero. This degree of precision would only be possible with very large samples. This section (1) describes our methods of combining randomized studies’ results, including the use of “effective n’s” and rounding rules; (2) describes the patients included in the six randomized studies that met our criteria; and (3) briefly discusses the two English studies that were omitted from our analyses because they did not meet our treatment criteria. We conducted the meta-analysis of six randomized studies primarily to produce information that could be compared to the separate statistical analysis of selected cases from the SEER database. We began our work for the meta-analysis of randomized studies’ results by calculating for each randomized study an odds ratio for 5-year survival (because the outcome criterion for the SEER analysis was 5-year survival). Then we tested for the homogeneity of the odds ratios and, because no significant heterogeneity was found, combined them in a common odds ratio. Specifically, we used the Mantel-Haenszel (1959) method and the STAT XACT program produced by Cytel Software of Cambridge, Massachusetts. STAT XACT uses the Breslow-Day (1980) method of testing for homogeneity of odds ratios. The confidence intervals surrounding the odds ratios were also calculated using the STAT XACT program and are based on the variance estimation method of Robins, Breslow, and Greenland (1986). Three of the six randomized studies—the Milan study, the French study conducted at the Institut Gustave-Roussy, and the U.S.-NSABP—had both (1) started long enough ago that, except for patients lost to follow-up, all had been followed for 5 years and (2) calculated recent estimates of 5-year survival for node-negative patients. Thus, for these three studies, estimates of 5-year survival were based on 5 or more years of follow-up for all or almost all patients. For the other three studies (U.S.-NCI, Danish, and EORTC), the 5-year survival estimates were actuarial and included a more substantial number of patients who had not been followed for 5 years. To treat these actuarial estimates appropriately in our meta-analyses, we developed the following approach: obtain the standard errors of the actuarial estimates (that is, standard errors that take account of how long each patient has been followed up); calculate the “effective n” associated with each actuarial estimate, according to the formula shown by Cutler and Ederer (1958); multiply the actuarial estimate of 5-year survival by the effective n—thus obtaining the effective number who survived (and, by subtraction, died) in each treatment group of each study; and use these “effective n’s” in calculating the common odds ratio for the meta-analysis. Effective n’s for the three studies were calculated as shown in table I.2. The most precise estimates available were used. The combined-studies (and combined-treatments) survival estimates were, where possible, rounded to the nearest tenth of a percentage point. Because estimates for one multicenter randomized study (EORTC) were only available to the nearest full percentage point, summary figures involving that study were rounded to the nearest full percentage point—to avoid implying greater precision than was possible. Odds ratios were calculated using the most precise figures possible; however, in preparing the data from each randomized study, the number of patients who died within 5 years and the number who survived were calculated from reported percentages and then rounded to the nearest patient (whole number). Odds ratios, which were based on the rounded numbers of patients, were themselves rounded at the second decimal place. Differences between reported survival rates were calculated using the most precise figures possible—and then rounded for presentation in tables. Slight differences in results may have occurred because of rounding procedures and the use of “effective n’s” (described above). This description of the characteristics of patients who participated in randomized studies is based on published eligibility requirements as well as informal requirements identified through data on the kinds of patients that were actually included (which we obtained, as needed, by calling investigators). Briefly, all patients in the six randomized studies had invasive breast cancer. As shown in table I.3, almost all patients were age 70 or younger and had tumors of 4 cm or less. Two of the three single-center studies admitted only patients with tumors of 2 cm or less. Most randomized studies had numerous eligibility requirements in addition to the age and tumor-size limits. For the U.S. studies, these were as follows: U.S.-NCI. Tumor confined to breast and axillary nodes, no advanced local disease, no inflammatory carcinoma, no multiple masses or bilateral cancer, no Paget’s disease, no prior cancer. U.S.-NSABP. No fixation to underlying muscle or chest wall, no clinical evidence of skin involvement or distant metastases, no multiple masses (unless all but one proved benign), no prior cancer. With respect to type of breast cancer (histology), the U.S.-NCI randomized study further noted that almost all patients had infiltrating duct carcinoma. The Milan study also reported that a majority of patients had this type of cancer. Two English studies (Atkins et al., 1972; Hayward, 1981; Hayward and Caleffi, 1987) did not meet our treatment criteria because they did not include nodal dissection as part of the breast-conservation therapy that they provided. The two studies are unique in several ways and are therefore briefly discussed in this appendix. First, treatments given in the two English studies differed from treatments given in other randomized studies. As mentioned above, the 1961 and 1971 English studies did not perform nodal dissection on breast-conservation patients. In addition, they have been criticized for providing inadequate radiation (Harris et al., 1983). Second, patient survival rates appeared to be considerably lower than in the six studies that met our criteria. This suggests that patients in the English studies may have had poorer prognoses or been subjected to poorer treatment implementations, or both. Third, the two English studies were conducted earlier than the other studies. They began in 1961 and 1971, and the 1971 study used the same procedures as the 1961 study. The six studies in our analysis were begun between 1972 and 1983. Fourth, in the two English studies, the overall pattern indicated that lumpectomy was less effective than mastectomy. In the first English study, it was clear—early on—that clinically node-positive patients who received lumpectomy showed lower survival rates than those who received mastectomy. Therefore, only clinically node-negative patients were included in the second English study; however, the clinically node-negative breast-conservation patients in the second English study showed lower 5-year survival than corresponding mastectomy patients. And when the 10-year follow-up was completed for the first study, the clinically node-negative patients in that study also showed a pattern of higher survival with mastectomy than with lumpectomy. Although the English studies did not qualify for our analyses, we believe they are noteworthy in that they caution that there is at least some question as to whether breast-conservation therapy and mastectomy produce comparable survival results when treatment implementations are poorer or when patients have poorer prognoses. SEER began recording the type of surgery that breast cancer patients received for the cohort diagnosed in 1983. At the time we performed the analyses reported here, SEER follow-up was available through 1990. We therefore selected patients diagnosed from 1983 through 1985—all of whom could be followed for 5 years. The number of positive nodes was not recorded for these diagnostic cohorts. Because the number of positive nodes is a key prognostic factor for early-stage node-positive patients—and may also be associated with selection of surgery—we believe it is necessary for a statistical analysis aimed at minimizing selection bias among node-positive patients. Data on longer term survival and on node-positive patients are provided by randomized studies. As more SEER data become available, SEER analyses that cover node-positive patients and longer term survival will be possible. The SEER analyses presented in this report are based on 5,326 breast cancer patients. This dataset was formed by accessing the SEER database for 1983 to 1985 diagnoses and selecting patients who met the following criteria: no previous diagnosis with another cancer; type of treatment, disease-related, and demographic characteristics known; patient followed for 5-years or longer; node-negative invasive breast cancer that had not spread beyond the breast (no chest wall involvement, no skin involvement, no attachment to the pectoral muscle); tumor 4 cm or smaller; type of cancer: infiltrating duct carcinoma or adenocarcinoma (NOS); type of treatment: if breast-conservation therapy, lumpectomy with nodal dissection plus radiation; if mastectomy, no “outlier” treatments (that is, no subcutaneous mastectomy, no mastectomy without nodal dissection, no radical mastectomy, no mastectomy plus radiation); and age 70 or younger. In the resulting dataset, which included 5,326 patients, about 20 percent of patients received breast-conservation therapy; the remaining 80 percent received mastectomy. Preliminary analyses on a broader set of SEER patients included those that had been lost to follow-up before the requisite 5 years had elapsed following diagnosis (6.2 percent had been lost to follow-up). In these analyses, the patients who were followed for at least 5 years and those who were not, proved to be virtually identical with respect to both tumor size (the main prognostic factor for node-negative patients) and type of surgery. Specifically, Patients not followed had an average tumor size of 2 cm, as did those followed for all 5 years. Seventeen percent of the followed patients received breast-conservation therapy (as opposed to mastectomy), as did 17 percent of those lost to follow-up. To derive the propensity scores, we entered patient characteristics into a logistic regression model predicting selection for breast-conservation therapy. The six patient characteristics entered were year in which patient was diagnosed (time), geographic area of residence (place), size of the patient’s tumor, patient’s age at diagnosis, marital status, and race or ethnicity. Because the ultimate objective of the propensity-score analysis was to enhance equivalence of the two SEER treatment groups on all measured variables, all six variables were included in the final model. Five of the six variables did prove to significantly affect a patient’s probability of receiving breast-conservation therapy. The model also included one significant interaction term—the interaction of geographic area with diagnostic year. (See table I.4.) As expected, patients with smaller tumors were more likely to receive breast-conservation therapy than patients with larger tumors. However, the other patient characteristics determining selection for breast- conservation therapy argued against a unidimensional selection process in which patients with better prognoses are consistently selected for breast-conservation therapy. Notably, Patients under 40 had relatively high odds of receiving breast-conservation therapy, although there is some evidence that they may have less favorable prognoses than middle-aged patients (de la Rochefordiere et al., 1993). Asian women had lower odds than others of receiving breast-conservation therapy, although they may have somewhat better prognoses than other breast cancer patients. The propensity scores (probabilities of breast-conservation therapy obtained using the model in table I.4) for the SEER patients examined here ranged from .01 to .69. The propensity scores were used to create five quintiles, as suggested by Rosenbaum and Rubin (1984). The first quintile consists of patients who were least likely to receive breast-conservation therapy, whereas the fifth quintile consists of those who were most likely to receive it. Tumor size (cm) Interaction: DODY and registry (continued) –2 log likelihood = 4794.498 Chi-square = 646.975 with 27 df Selection for breast-conservation therapy (versus mastectomy) is predicted using date of diagnostic year (DODY, 1983 to 1985), SEER registry (geographic location), and patient characteristics. Breast-conservation therapy was coded 1 and mastectomy was coded 0. As intended, the propensity-score quintiles differentiated between patient subgroups; that is, major differences across the quintiles were apparent. Notably, half (51 percent) of quintile 1 patients (low probability of breast-conservation therapy) had tumors larger than 2 cm; whereas only 14 percent of quintile 5 had tumors of that size. With respect to geographic area, 70 percent of quintile 1 patients were from Iowa, metropolitan Detroit, or metropolitan Atlanta; by contrast, 73 percent of quintile 5 patients were from the San Francisco-Oakland or the Seattle-Puget Sound registries. Only 6 percent of quintile 1 patients were diagnosed in 1985, compared to 66 percent of quintile 5. Within each propensity-score quintile, we checked the breast-conservation therapy and mastectomy groups for equivalence on all six variables. No major differences were found; two relatively minor differences were adjusted for, as follows: First, with respect to tumor size, within four of five quintiles, a slightly higher proportion of mastectomy patients than breast-conservation patients had tumors larger than 2 cm. For example, within quintile 5, 15 percent of mastectomy patients had tumors larger than 2 cm, as compared to 12 percent of breast-conservation patients. We therefore adjusted results within each quintile so that the patients with larger tumors would contribute equally to the mastectomy survival estimate and to the breast-conservation survival estimate for that quintile. Second, with respect to year of diagnosis, within quintile 5 there was a significant difference between mastectomy patients and breast- conservation patients: 64 percent of the mastectomy patients in quintile 5 had been diagnosed in 1985 as compared to 70 percent of breast-conservation patients. Although year of diagnosis is not generally associated with differences in patient survival, we took the precaution of adjusting results for quintile 5 so that patients diagnosed in 1985 would contribute equally to that quintile’s breast-conservation survival estimate and its mastectomy survival estimate (as would patients diagnosed in 1984 and 1983). Using the quintiles together with the additional adjustments ensures that the comparison between survival rates following breast-conservation therapy and mastectomy is based on patient groups that were adjusted to be as “equivalent” as possible on all relevant measured variables. The experts listed here commented on one or more drafts of the report or advised us on the methods used in our analyses, or both. We are grateful for the gracious contributions of all these individuals. Altman, Roberta, and Michael Sarg. The Cancer Dictionary. New York: Facts on File, 1992. Andersen, Knud West. Personal communication. Danish Breast Cancer Cooperative Group, Aug. 10, 1993. Atkins, Sir Hedley, et al. “Treatment of Early Breast Cancer: A Report after Ten Years of a Clinical Trial,” British Medical Journal, 2:423-29, 1972. Blichert-Toft, M., et al. “Danish Randomized Trial Comparing Breast Conservation Therapy with Mastectomy: Six Years of Life-Table Analysis,” Journal of the National Cancer Institute Monographs, 11:19-25, 1992. Blichert-Toft, M., et al. “A Danish Randomized Trial Comparing Breast-Preserving Therapy with Mastectomy in Mammary Carcinoma: Preliminary Results,” Acta Oncologica, 27(Fasc. 6a):671-77, 1988. Breslow, N.E., and N.E. Day. Statistical Methods in Cancer Research: Volume 1 - The Analysis of Case Control Studies. Lyon, France: International Agency for Research on Cancer (Pub. No. 32), 1980. Byar, David P. “Why Data Bases Should Not Replace Randomized Clinical Trials,” Biometrics, 36:337-42, 1980. Cutler, Sidney J., and Fred Ederer. “Maximum Utilization of the Life Table Method in Analyzing Survival,” Journal of Chronic Diseases, 8:699-712, 1958. de la Rochefordiere, Anne, et al. “Age as a Prognostic Factor in Premenopausal Breast Carcinoma,” Lancet, 341:1039-43, 1993. Dickersin, K., and J. Berlin. “Meta-Analysis: State-of-the-Science,” Epidemiologic Reviews, 14:154-76, 1992. Droitcour, Judith A., George Silberman, and Eleanor Chelimsky. “Cross Design Synthesis: A New Form of Meta-analysis for Combining Results from Randomized Clinical Trials and Medical-Practice Databases,” International Journal of Technology Assessment in Health Care, 9(3):440-49, 1993. Ellenberg, Susan S. “Meta-Analysis: The Quantitative Approach to Research Review,” Seminars in Oncology, 15:472-81, 1988. Fleiss, Joseph L. “General Design Issues in Efficacy, Equivalency, and Superiority Trials,” Journal of Periodontal Research (special issue), 27:306-13, 1992. Freidlin, Boris. Personal communication. EMMES Corp., June 23, 1994. GAO. (See U.S. General Accounting Office.) Hankey, Benjamin F., et al. “Overview.” In Barry A. Miller et al., Cancer Statistics Review, 1973-1989 (NIH Pub. No. 92-2789). Bethesda, Md.: National Institutes of Health, 1992. Harris, Jay R., et al. “Conservative Surgery and Radiotherapy for Early Breast Cancer,” Cancer, 66 (Sept. 15 Supp.):1427-38, 1990. Harris, Jay R., Samuel Hellman, and William Silen (eds.). Conservative Management of Breast Cancer: New Surgical and Radiotherapeutic Techniques. Philadelphia: J.B. Lippincott, 1983. Hayward, John L. “The Guy’s Hospital Trials on Breast Conservation.” In Jay R. Harris, Samuel Hellman, and William Silen (eds.), Conservative Management of Breast Cancer: New Surgical and Radiotherapeutic Techniques. Philadelphia: J.B. Lippincott, 1983, pp. 77-90. Hayward, John, and Maira Caleffi. “The Significance of Local Control in the Primary Treatment of Breast Cancer,” Archives of Surgery, 122:1244-47, 1987. Kahn, Harold A., and Christopher T. Sempos. Statistical Methods in Epidemiology. New York: Oxford University Press, 1989. Lee-Feldstein, Anna, Hoda Anton-Culver, and Paul J. Feldstein. “Treatment Differences and Other Prognostic Factors Related to Breast Cancer Survival: Delivery Systems and Medical Outcomes,” Journal of the American Medical Association, 271(15):1163-68, 1994. Lichter, Allen S., et al. “Mastectomy Versus Breast-Conserving Therapy in the Treatment of Stage I and II Carcinoma of the Breast: A Randomized Trial at the National Cancer Institute,” Journal of Clinical Oncology, 10(6):976-83, 1992. Louis, Thomas A., Harvey V. Fineberg, and Frederick Mosteller. “Findings for Public Health From Meta-Analyses,” Annual Review of Public Health, 6:1-20, 1985. Mignolet, Francoise. Personal communication. Brussels: EORTC Data Center, Oct. 12, 1994. Mosteller, Frederick, and John W. Tukey. Data Analysis and Regression. Reading, Mass.: Addison-Wesley, 1977. National Institutes of Health. “Early-Stage Breast Cancer—NIH Consensus Conference,” Journal of the American Medical Association, 265(3):391-95, 1991. Office of Technology Assessment. (See U.S. Congress, Office of Technology Assessment.) Percy, Constance, Valerie Van Holten, and Calum Muir. International Classification of Diseases for Oncology, 2nd ed. Geneva: World Health Organization, 1990. Robins, J., N. Breslow, and S. Greenland. “Estimators of the Mantel- Haenszel Variance Consistent in Both Sparse Data and Large-Strata Limiting Models,” Biometrics, 42:311-23, 1986. Rosenbaum, Paul R., and Donald B. Rubin. “Reducing Bias in Observational Studies Using Subclassification on the Propensity Score,” Journal of the American Statistical Association, 79(387):516-24, 1984. Rubin, Donald B., and N. Thomas. “Characterizing the Effect of Matching Using Linear Propensity Score Methods with Normal Distributions,” Biometrika, 79(4):797-809, 1992. Sacks, Nigel P.M., and M. Baum. “Primary Management of Carcinoma of the Breast,” Lancet, 342:1402-08, 1993. Sarrazin, Daniele. Personal communication. Institut Gustave-Roussy, Aug. 6, 1993. Sarrazin, Daniele, et al. “Ten-year Results of a Randomized Trial Comparing a Conservative Treatment to Mastectomy in Early Breast Cancer,” Radiotherapy and Oncology, 14:177-84, 1989. Sarrazin, Daniele, et al. “Conservative Treatment Versus Mastectomy in Breast Cancer Tumors with Macroscopic Diameter of 20 Millimeters or Less: The Experience of the Institut Gustave-Roussy,” Cancer, 53:1209-13, 1984. Sarrazin, Daniele, et al. “Conservative Treatment Versus Mastectomy in T1 or Small T2 Breast Cancer—A Randomized Clinical Trial.” In Jay R. Harris, Samuel Hellman, and William Silen (eds.), Conservative Management of Breast Cancer: New Surgical and Radiotherapeutic Techniques. Philadelphia: J.B. Lippincott, 1983, pp. 101-11. Stablein, D.M. Personal communication. EMMES Corp., June 10, 1994a. Stablein, D.M. A Reanalysis of NSABP Protocol B06: Final Report. Potomac, Md.: EMMES Corp., 1994b. Steinberg, Seth M. Personal communication. National Cancer Institute, July 13, 1993. Straus, K., et al. “Results of the National Cancer Institute Early Breast Cancer Trial,” Journal of the National Cancer Institute Monographs, 11:27-32, 1992. Swanson, G. Marie, et al. “Trends in Conserving Treatment of Invasive Carcinoma of the Breast in Females,” Surgery, Gynecology & Obstetrics, 171:465-71, 1990. U.S. Congress, Office of Technology Assessment. Identifying Health Technologies That Work: Searching for Evidence, OTA-H-608. Washington, D.C.: U.S. Government Printing Office, 1994. U.S. General Accounting Office. Cross Design Synthesis: A New Strategy for Medical Effectiveness Research (GAO/PEMD-92-18). Washington, D.C.: U.S. General Accounting Office, 1992. van Dongen, J.A., et al. “Factors Influencing Local Relapse and Survival and Results of Salvage Treatment after Breast-Conserving Therapy in Operable Breast Cancer: EORTC Trial 10801, Breast Conservation Compared With Mastectomy in TNM Stage I and II Breast Cancer,” European Journal of Cancer, 28A(4-5):801-05, 1992a. van Dongen, J.A., et al. “Randomized Clinical Trial to Assess the Value of Breast-Conserving Therapy in Stage I and II Breast Cancer, EORTC 10801 Trial,” Journal of the National Cancer Institute Monographs, 11:15-18, 1992b. Veronesi, Umberto. Personal communication. Istituto Europeo di Oncologia, June 21, 1994. Veronesi, Umberto. “Local Control and Survival in Early Breast Cancer: The Milan Trial,” International Journal of Radiation: Oncology— Biology—Physics, 12:717-20, 1986a. Veronesi, Umberto, et al. “Comparison of Halsted Mastectomy with Quadrantectomy, Axillary Dissection, and Radiotherapy in Early Breast Cancer: Long-Term Results,” European Journal of Cancer and Clinical Oncology, 22(9):1085-89, 1986b. Veronesi, Umberto, et al. “Comparing Radical Mastectomy with Quadrantectomy, Axillary Dissection, and Radiotherapy in Patients with Small Cancers of the Breast,” New England Journal of Medicine, 305(1):6-11, 1981. Winch, Robert, and Donald Campbell. “Proof? No. Evidence? Yes. The Significance of Significance Tests,” American Sociologist, May 1969:140-43. Winchester, David P., and James D. Cox. “Standards for Breast-Conservation Treatment,” CA-A Cancer Journal for Clinicians, 42(3):134-59, 1992. Woolf, B. “On Estimating the Relation Between Blood Group and Disease,” Annals of Human Genetics, 19:251-53, 1955. The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Summary: Pursuant to a congressional request, GAO reviewed the survival rates of patients receiving breast-conservation therapy versus mastectomy therapy, focusing on whether patients' survivability is affected by having treatment in single-center studies rather than multi-center studies. GAO found that: (1) the 5-year survival rates for breast cancer patients treated with breast-conservation therapy were similar to that of mastectomy in community medical practices; (2) patients that were treated at single-center facilities usually had slightly higher survivability rates than patients who received treatment at multi-center facilities; (3) on average, breast cancer patients appear to be at no appreciable risk by selecting breast-conservation therapy rather than mastectomy; (4) although patient survivability data were vulnerable to hidden selection bias, such bias was unlikely since the group of patients reviewed were homogeneous and adjustments were made for patients' health and demographic characteristics; and (5) although a minority of patients who voluntarily chose breast-conservation therapy over recommended mastectomy therapy could have achieved slightly better results with mastectomy, the differences were not statistically significant.