# **IV. Methodology and Data**

## **a. Sample Description and Sampling Technique**

Our primary source of data comes from a survey covering three consecutive cohorts of EF trainees (from 2010 to 2012), with two rounds of data collection for each cohort.<sup>24</sup> Figure 1 depicts the impact evaluation timeline.

We sample at the training event and the applicant level. The main sampling frame for data used in this study consisted of all EF training courses sponsored in a given year. The number of training events comprising the sample frame ranges from 598 (in 2010) to 711 (in 2012). Table 1 reports the number of events and participants by year.

Sampling into this study included a combination of stratified, random and convenience sampling. First, we selected a subset of training events occurring between the months of January through April.<sup>25,26,27</sup> Second, from the universe of training events offered during these four months, we randomly selected up to 15 districts. Third, from that list of training events occurring in these districts, we randomly selected 20 percent of the training events. Finally, a survey team visited each sampled training event on the day when applicant selection took place. Each event’s ranking sheet listed the shortlisted applicants from the top-scorer to the bottom and indicated the threshold, or minimum score needed to gain admission to the course. From this ranking sheet, the survey team selected applicants whose scores were within 20 percent of the threshold for admission to training events. The sampled applicants above the threshold comprise this study’s treatment group, while those below the threshold make up the control group. Immediately following the sampling of applicants, a baseline survey was conducted on the treatment and control groups, before the results of the selection process were announced.

Table 2 shows the resultant sample of events for the three cohorts. The 2010 event sample comprised 64 events across 30 districts. The 2011 sample comprised 182 events, of which 113 events were dropped from the baseline survey, either because the survey team could not reach the event on the day of applicant selection or because the event was not “oversubscribed”.<sup>28</sup> The remaining 69 events in 34 districts were included in the 2011 baseline sample. The sampling process was much improved in 2012, with 85 out of 112 sampled events successfully included in the final evaluation sample.

This non-random (in 2010) and partially-random (in 2011 and 2012) sampling of training events and applicants introduces bias in two potentially important ways. First, the training events in our study may not be representative of all EF-sponsored trainings. In all three cohorts, training events that enter our sample

> 24 For the 2010 cohort, a second follow-up was conducted on half of the cohort. Future analysis will examine the longer-term outcomes for this group.

> 25 Because of the AGEI focus of this study, we prioritize AGEI training events (identified by the T&E as we described earlier). Because the selection into the study population was based on an individuals’ proximity to the threshold score, it was not possible to stratify on AGEI status. However, events that were likely to include more AGEI candidates were purposely oversampled in 2011 and 2012 so as to increase the number of AGEI candidates in the study population.

> 26 Eighty percent of EF training events occurred during these four months.

> 27 In 2010, because a complete event listing was not available in advance, the events were chosen by convenience, based on scheduling and accessibility.

> 28 The survey team was instructed to drop the event from the sample if there were not at least 3 rejected candidates that fell within 20% of the threshold score. In other words, if there were not at least 3 people who could be sampled for the control group, the event was dropped from the sample.

8