Violations of SUTVA. Our analysis concerns how estimates would change if observations were replaced with hypothetical or counterfactual cases. But estimates could also change if observed outcomes changed when others’ assignments were altered. For example, the achievement of the students who were actually retained could change if some promoted students were retained. This could occur if one child’s retention reduced the stigma of another, or competed with the resources, such as the attention of the teacher (Shepard & Smith, 1989). Changes in observed outcomes as a result of others’ assignments to treatment conditions constitute a violation of SUTVA, an assumption made for most inferences. As such, there are some (recent) techniques to inform the implications of violations to SUTVA. For example, one can use agent based models to explore the dynamic implications of changes in treatment assignments (e.g., Maroulis et al., 2010). Measurement Error. Error in measurement, especially of the outcomes, could bias estimates. Generally, measurement error will lead to conservative inferences because it reduces precision of estimates. This might be preferred if one wanted to be cautious in implementing new programs. But measurement error could hide a substantively important negative effect (an example of a type II error, failure to reject the null when in fact the null hypothesis is false). Measurement error is not a large concern for the two focal examples in the article in which the outcomes were measured with high reliability. The reliability of the Terra Nova comprehensive reading test used by Borman et al was very high (.84 to .93 across grades 3-6 2001), and would likely be higher at the classroom level given fairly large sample sizes per classroom (see Brennan, 1995, for a discussion). Similarly, the test used by Hong and Raudenbush had a reliability of .95 (based on the variance of repeated estimates of overall ability --see ECLS-K user guide, section 3.1.6). Furthermore, both Borman et al., and
Table 1 Quantifying the Robustness of Inferences from Randomized Experiments Study (author, year) Treatment vs Control Blocking Population Outcome Estimated effect, standard error, source Effect size (correlation) % bias to make the inference invalid A multi-site cluster randomized field trial of Open Court Reading (Borman et al., 2008) Open Court curriculum versus business as usual Within grade and school 917 students in 49 classrooms Terra Nova comprehensive reading score 7.95 (1.83) Table 4, results for reading composite score .16 (.54) 47% Answers and questions about class size: A statewide experiment (Finn & Achilles, 1990) Small classes versus all others By school 6500 students in 328 classrooms Stanford Achievement Test, reading 13.14 (2.34) Table 5 mean for other classes is based on the regular and aide classes combined proportional to their sample sizes. .23 (.30) 64% Experimental Evaluation of the Effects of a Preschool Mathematics Curriculum (Clements & Sarama, 2008) Building Blocks research to practice curriculum versus alternate math intensive curriculum By program type 276 children within 35 classrooms randomly sampled from volunteers within income strata Change in Early Mathematics Childhood Assessment IRT scale score (mean=50, std=10) 3.55 (1.16) Building Blocks vs comparison group Table 6 (df of 19 used based on footnote b) .5 (.60) 31%
Table 2 Quantifying the Robustness of Inferences from Observational Studies Study (author, year) Predictor of interest Condition on pretest Population Outcome Estimated effect, standard error, source Effect size (correlation) % bias to make inference invalid Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics (Hong & Raudenbush, 2005) Kindergarten retention versus promotion Multiple 7639 kindergarteners in 1080 retention schools in ECLS-K ECLS-K Reading IRT scale score 9 (.68) Table 11, model based estimate .67 (.14) 85% Counterfactuals, causal effect heterogeneity, and the Catholic school effect on learning, (Morgan, 2001) Catholic versus public school Single 10835 high school students nested within 973 schools in NELS NELS Math IRT scale score .99 (.33) Table 1, (model with pretest + family background) .23 (.10) 34% Effects of teachers’ mathematical knowledge for teaching on student achievement (Hill et al., 2005) Content knowledge for teacher mathematics Gain score 1773 third graders nested within 365 teachers Terra Nova math scale score 2.28 (.75) Table 7, model 1, (third graders) NA (.16) 36%
Table B1 Quantifying the Robustness of Inferences from Randomized Experiments in EEPA posted on-line July 24-Nov 15 2012 Study (1st author) Predictor of interest Population (df) Outcome source r % bias to make inference invalid Multiplier to make estimate significant Incentive Pay Programs Do Not Affect Teacher Motivation or Reported Practices (Yuan) Pay-for performance 5th-8th grade teachers in Nashville (143) Test Table 3, POINT program .250 33% Effects of Two Scientific Inquiry Professional Development Interventions on Teaching Practice (Grigg) Professional development for inquiry in science 4th-5th grade Classrooms in Los Angeles (70 schools) Inquiry used Table 5 Model 1 .277 15% A Randomized Controlled Trial Evaluation of Time to Read, a Volunteer Tutoring Program for 8- to 9-Year-Olds (Miller) Time to Read (tutoring) 8-9 year old children in Northern Ireland (734) Future aspiration Table 5, column 1 .078 7% Incentive Pay Programs Do Not Affect Teacher Motivation or Reported Practices (Yuan) Pay-for performance 5th-8th grade teachers in Nashville (143) Extra hours worked/ week Table 3, POINT program .077 -154% 2.53 a Degrees of freedom for calculating threshold value of r are defined at the level of the predictor of interest, subtracting the number of parameters estimated at that level. Calculations can be conducted from spreadsheet at . Studies listed in order of % robustness.
Table B2 Quantifying the Robustness of Inferences from Observational Studies in EEPA posted on-line July 24-Nov 15 2012 Study (1st author) Random sample Predictor of interest Condition on pretest (df)a Outcome source r % bias to make inference invalid Multiplier to make estimate significant Academic Effects of Summer Instruction and Retention in New York City (Mariano) Summer instruction and retention RD 5th graders in New York (29,987) English & Language Arts achievement 6th grade Table 2, Panel 1, column 1 .029 60% Unintended Consequences of an Algebra-for-All Policy on High-Skill Students (Nomi) schools that increased algebra enrollment for students by more than 5%) Time Series 9th graders in Chicago (17,987) Peer ability levels for high skill students (1999 difference for affected schools) Table 2, column 1 .030 50% Impact of Dual Enrollment on College Degree Attainment (An) yes Dual enrollment NELS (8,754) Any degree obtained Table 1, Panel A, column 1 .043 50% Life After Vouchers (Carlson) Student transferred to public school No 3rd-10th grade in public schools Math Table 5, all students .092 36%
Figure Captions Figure 1. Estimated Treatment Effects in Hypothetical Studies A and B Relative to a Threshold for Inference Figure 2. Example Replacement of Cases from Non-Volunteer Schools to Invalidate Inference of an Effect of the Open Court Curriculum Figure 3. Effect of Retention on Reading Achievement in Retention Schools (from Hong and Raudenbush, 2005, Figure 2, p. 218). Figure 4. Example Replacement of Cases with Counterfactual Data to Invalidate Inference of an Effect of Kindergarten Retention Figure 2. Example Replacement of Cases from Non-Volunteer Schools to Invalidate Inference of an Effect of the Open Court Curriculum Figure 3. Effect of Retention on Reading Achievement in Retention Schools (from Hong and Raudenbush, 2005, Figure 2, p. 218).