that are not directly measurable on the ad platform itself. Malaria prevention campaigns offer an example. Malaria is a disease which disproportionately affects rural people in India. Behaviorally informed ads could provide information and persuade individuals to take precautionary measures. However, while digital advertising provides the ability to micro-target messages to specific subgroups of the population, certain segments remain harder/more expensive to reach than others. At the same time, measuring the impact of such ads on offline behaviors like using bed nets is challenging, and requires availability of individual survey responses that could be connected with ad exposure data.

In this paper, we first introduce a new methodology and a novel toolkit for conducting surveys and experiments on social media. We then use this tool to evaluate a Facebook malaria-prevention ad campaign designed and run by Malaria No More, a global public health non-profit with a strong presence in India. The campaign’s goals were the promotion of behaviors to prevent malaria infection and to seek timely treatment in case of symptoms. It consisted of more than 50 creatives targeted to different personas. With an average frequency of 3.6 impressions per user, the campaign had a total reach of about 130 million people on Facebook and Instagram over 6 months. In the evaluation, we consider the question “Did the campaign work?” along with the question “Did it work for those most at risk?”.

We recruited two groups of respondents (a longitudinal and a cross-sectional sample) on Facebook and Instagram offering mobile credit as an incentive to fill out a series of chat surveys conducted on Messenger. We stratified the recruitment across 80 districts in three high-burden states, and used real-time optimization to guarantee representativeness of individuals living in non-solid dwellings, where malaria risk is higher. We then worked with the non-profit advertising team to exclude 40 randomly-picked districts from the delivery of the campaign. Relying on a set of selfreported attitudes and behavioral outcomes, we found evidence that the ad campaign increased the use of bed nets among individuals living in solid houses (by 11% compared to control), while it was ineffective for households living in non-solid dwellings. Moreover, self-reported household malaria incidence decreased by 44-53% among those living in solid houses compared to control, but not among non-solid dwellers. The reduction in malaria incidence was predominantly driven by bed net owners. Consistently, the analysis of administrative health facility data showed that monthly malaria incidence in urban areas decreased by 6.2 cases per million people. This is about 30% of the overall monthly malaria incidence rate in the pre-treatment period and implies an upper bound cost per averted _reported_ urban case of USD 3.4. By contrast, rural incidence was not significantly affected.

Was the ad content ineffective for households at greater malaria risk or did it fail to sufficiently reach them? To answer this question, in a second trial we experimentally varied exposure to the same ads at the individual level, using the re-marketing tools of the ad platform. We found that bed net use and timely treatment seeking increased for both types of households, suggesting that social media campaigns need to invest in improved targeting strategies to reach their development objectives.

35