B.1. Data used to address implementation research questions (example data included in italics for expository purposes) Implementation element of data used to assess whether the element of the intervention was implemented as intended Frequency/sampling of data collection Party responsible for data collection Adherence: How often were sessions offered? How many were offered? e.g., All sessions offered are captured in MIS Length (number of minutes) of program sessions captured in MIS e.g., All sessions delivered are captured in MIS Session length sampled once a week e.g., Program staff Program staff Adherence: What and how much was received? e.g., Daily attendance records e.g., Student attendance at all sessions is captured in MIS e.g., Program staff Adherence: What content was delivered to youth? e.g., Number of topics covered captured on observation spreadsheeta e.g., Classroom observations occurred twice a year e.g., Evaluation staff Adherence: Who delivered material to youth? e.g., List of staff members hired and trained to implement program Background qualifications of staff members from staff applications e.g., Data on all staff members are available to program staff e.g., Program staff Quality: Quality of staff-participant interactions e.g., Observations of interaction quality using protocol developed by evaluators e.g., Convenience sample of 10% of classroom sessions were selected for observation e.g., Evaluation staff Quality: Quality of youth engagement with program e.g., Observations of engagement using the YPQA e.g., Random sample of 5% of all sessions were selected for observation e.g., Evaluation Counterfactual: Experiences of comparison condition e.g., Survey items on baseline and follow-up assessments Focus groups of comparison group members e.g., Pre- and post-intervention Convenience sample of comparison group participants (once) e.g., Evaluation Context: TPP programming available or offered to study participants (both intervention and comparison) e.g., District website listing all TPP programming Interview with school district curriculum director e.g., Ad hoc Once per year e.g., Evaluation staff Evaluation staff
Youth: Contributed a follow-up survey Immediately 9 (=9a + 9b) 9a 9b =9/6 =9a/6a =9b/6b Youth: Contributed a follow-up survey 6-months 10 (=10a + 10b) 10a 10b =10/6 =10a/6a =10b/6b Youth: Contributed a follow-up survey 12-months 11 (=11a + 11b) 11a 11b =11/6 =11a/6a =11b/6b Table C.1b. Youth sample sizes by intervention status – individual-level assignment designs Number of youth Time Period Total sample size Intervention sample size Comparison sample size response rate Intervention response rate Comparison response rate Assigned to condition 1 (=1a + 1b) 1a 1b N/A NA N/A Contributed a baseline survey 2 (=2a + 2b) 2a 2b =2/1 =2a/1a =2b/1b Contributed a follow-up survey Immediately 3 (=3a + 3b) 3a 3b =3/1 =3a/1a =3b/1b Contributed a follow-up survey 6-months 4 (=4a + 4b) 4a 4b =4/1 =4a/1a =4b/1b Contributed a follow-up survey 12-months 5 (=5a + 5b) 5a 5b =5/1 =5a/1a =5b/1b Table D.1. Methods used to address implementation research questions (example data included in italics for expository purposes) Implementation element Methods used to address each implementation element programming programming programming programming