Integrating an ethnography into the design of an impact evaluation can provide a robust way of examining processes and mechanisms. In 2007, I worked with colleagues to test an intervention to improve the quality of citizen engagement in Indian village governments (Rao, Ananthpur and Malik 2017). The intervention provided two weeks of training to village residents on participatory planning, village government regulations, budgets and administrative systems, which was followed by monthly follow-up visits by facilitators over a two-year period to mobilize citizens and translate their demands and needs into actions. We picked a hundred villages at random, and then randomly assigned 50 of them to the intervention. Quantitative data from surveys of villages, households and key informants were conducted prior to the intervention in 2007 and after the two-year period in 2009. A 10% sub-sample of the quantitative sample, five treatment and five control villages, were selected for the ethnography, which is a large sample for participant-observation. We developed an “embedded reporter” approach where five trained ethnographers were hired to live in these villages (each reporter was assigned to one treatmentcontrol pair) to conduct participant-observations. They prepared monthly reports of everything they observed – including economic changes, social changes, and political events. These 240 monthly reports constituted our qualitative data. Early rounds of the qualitative data helped inform the development of the quantitative survey instruments.

The quantitative data showed considerable improvement over time across a variety of indicators, but the difference in outcomes between treatment and control villages were not significantly different from each other. The ethnography allowed us to unpack the reasons why the intervention “failed” highlighting the role of variations in the quality of facilitation, lack of top-down support, and difficulties in confronting the stubborn challenge of persistent inequality. We found that while surveys have the advantage of being able to measure predictable outcomes with measurable effects, major events sometimes occur that are best investigated on the ground, during the moment they happen, by participant observers. Important shifts occur during points of conflict, or during periods of mobilization and coordination. These are not quotidian events and are very hard to predictably measure. Moreover, qualitative investigators can build strong relationships within communities that allow them to “see” differently and thus capture insights that a survey interviewer is unable to do.

These three studies, all of which integrate qualitative and quantitative methods into their design at the outset and not as an afterthought, clearly show how much these methods complement each other. Ethnographic work not only helps design better survey instruments, it can also help design better interventions and provide deep insights into the processes that underly the changes measured by quantitative data. This obvious point has been made several times, but ethnographic methods still have not become a routine part of the economist’s tool kit, though recently a small number of impact evaluation papers have used open-ended interviews and focus groups to understand mechanisms (e.g., Muralidharan and Singh, 2020).

# E. <u>Participation: Respondent as Analyst</u>

If cognitive empathy is largely about reducing the social and political distance between the researcher and the researched, participation, or co-production, takes that a step further by actively involving communities in the research process. Robert Chambers and his colleagues at the Institute for Development Studies at Sussex have for several decades now advocated methods broadly labeled as “Participatory Rural Appraisal,” that teach elementary graphic and visualization tools to communities so that can analyze and improve their own lives (Chambers, 1994) which have had a sizable impact in the practice of communitydriven development.

16