# **4. Overview of experimental studies designs**

## **4.1 Experimental designs**

Table 2 summarizes the experimental design and sample for the 11 studies included in this review. Of the 11 studies, five use individual or household-level randomized designs (Djibouti, Côte d’Ivoire, Lao PDR, and the two studies in DRC), two use cluster-level randomized designs with randomization at the community level (Sierra Leone and Ethiopia<sup>11</sup> ), and four use a two-stage cluster randomized designs (Comoros,<sup>12</sup> both studies in Egypt, and Tunisia).<sup>13</sup> In the two-stage cluster RCTs, communities were first randomized into treatment (receives the public works programs) and control (does not receive the public works programs); in the second stage, within the treated communities, individuals/households were further randomized into treatment (eligible to work in the public works program) and control (ineligible to work in the public works program). The goal of the two-stage designs was to assess spillover effects, which we do not discuss here.<sup>14</sup> Two of the public works experiments included multiple treatment arms to isolate the impact of the work component from complementary interventions that were designed to facilitate post-program transition to other forms of employment that generate more sustained impacts. In Urban DRC, <mark>these complementary features included incentivized savings and skills training (</mark> Brandily-Snyers et al., 2021). In Côte d’Ivoire, random subsets of participants were offered <mark>either additional entrepreneurship training to facilitate selfemployment or training in job search skills to facilitate wage employment (B</mark> ertrand et al., 2021). Since only these two studies contain multiple treatments, we do not attempt to draw more general conclusions on complementary program design features in this paper.

# **4.2 Evaluation sample**

The median study sample size is 2,126. Individual-level randomized designs have a sample size ranging from 952 to 6,014. Cluster randomized designs have larger samples ranging from 1,017 to 19,442 with at least 30 clusters per treatment arm across studies. The median sample size per cluster is 31, comparable to the median sample size per cluster of 26 noted in Muralidharan and Niehaus (2017).<sup>15</sup>

> 11  In Ethiopia, clusters were randomized into treatment and control, and, within treatment clusters, the participants were selected by local _Ketena_ committees. 12  In Comoros the design varied the treatment intensity, 40 percent of eligible households received the <mark>intervention in treatment clusters and 20 percent of eligible households received the intervention</mark> in control clusters.

> 13  A few studies also offer public works opportunities to the control group in the post-evaluation period in a randomized phase-in design, namely, Egypt infrastructure, Sierra Leone, Ethiopia, and Djibouti. 14 In Comoros, a third level of randomization was carried out: in every treatment household, the gender of the participant was further randomly chosen within the household.

> 15 All individual level randomized experiments can detect at least a $1.50 improvement in consumption expenditure and earnings, assuming a standard deviation of $8 with 5 percent type I error rate and 80 percent power. Among the cluster RCTs, there is an improvement in consumption and earnings––the Egypt studies can detect at least a $1.40 improvement, Comoros $2.10, Ethiopia $1.70, Sierra Leone $1.00, and Tunisia $1.90. These are computed using the country specific cluster sizes noted in Table 2 and assuming a standard deviation of $8, intra-cluster correlation of 0.10, 5 percent type I error rate and 80 percent power. All these calculations also assume 100 percent take-up. In practice, however most programs did not have 100 percent take-up. Particularly low take-up rates in Urban DRC (56 percent) and Egypt infrastructure (40 percent) will reduce the power to detect significant effects substantially.

11