**Appendix 1: How would leading field experiment experts approach randomization for the same intervention?**

# Our survey of experts presented researchers with the following question:

_Consider the following (hypothetical) pilot experiment being carried out. An intervention is being carried out with the goal of raising the incomes of day laborers by helping them gain new interview and employment skills. The pilot group consists of 100 men and 100 women aged 20 to 45, all in the same geographic area. Baseline data include age, current income, current weekly hours of work, education, age, marital status, and household size. A colleague asks for your advice on how to assign 100 of these 200 individuals to the treatment. Follow-ups will be at 6 months and 1 year. Please describe how you would recommend that they carry out this randomization. If your answer depends on other information not provided here, state the conditions under which you would do one method vs another. Please be specific in terms of what variables if any they should stratify on, what you recommend they should do to check for balance, and whether (and how) they should take multiple random draws if you recommend doing this to ensure balance on particular variables._ The responses to these questions were as follows:

- I would just randomize. Stratifying on such a small sample will cause weird things in the data.

- Random assignment with multiple draws; pick the one where the R-squared of the baseline data has the least explanatory power. My thinking here is that 200 units seems "small", so the potential for covariate imbalance in a single random draw large

- I would recommend that they choose the characteristics most likely to be correlated with the outcome of interest and then stratify based on those characteristics. Afterwards, they can check the balancedness of the sample if they have the available data.

- I would stratify on gender, education and current income. Then take a random draw after stratification and then check balance.

- Given the relatively small sample size, I would want to make sure to stratify on variables that might interact with the treatment. Thus, I'd certainly want to stratify on age categories, education categories and possibly gender. If there is reason to believe, as a result of previous work on the efficacy of interview and employment skills training (I am not familiar with this literature), that there are other strong factors that would lead to heterogenous treatment effects, then I would add those factors as well (that's why gender is a maybe for me here). After stratifying, I would make multiple random draws, checking for which draw yields the best balance across all the variables I have.

- I would stratify on gender, current income and current weekly hours of work, since these would all have 1st-order effects on the estimated program effect. I would do this by generating variables for an individual's place in the income and weekly hours of work distributions (say by quartile) and then stratifying by these variables. After performing the randomization, I would check differences between the treatment and control groups in the other variables, (as well as in income level and hours worked since the levels might be slightly off if randomization is by quartile). If these differences were very large, I would re-run the randomization. If I had a bigger sample, I would stratify by more variables. The baseline variables listed in the example are all "important," so I would use all or a large subset of them in the randomization. But with only 200 people, I would keep to the variables listed above and check the rest.

- I would definitely create matched pairs and then randomly select treatment/control among each of the matched pairs. The single-most important variable to match upon is the lagged dependent variable (current income), but the other variables are important too. I would perform an "optimal matching" algorithm per Biostatistics (2004), 5, 2, pp. 263–275.

- Think about the covariates that are likely to affect the treatment effect, and use them to stratify. For example, if in the context you study you have good reasons to think that the treatment effect will be different for women than for men, or if you have no prior on gender differences but you want to be able to check for heterogeneity in the treatment effect across gender groups, then you should really stratify by gender. My guess is that, given the intervention you describe, you should stratify by gender and education. Also by current income if there is a lot of heterogeneity in income at baseline, although if the intervention is well targeted I'd expect only little variation in current income and so there would be no