# **1 Introduction**

With over half of the world’s population engaged in social media (We Are Social, 2023), governments and organizations increasingly leverage these platforms in their social marketing campaigns to influence individuals’ beliefs, attitudes and real-world behaviors.<sup>1</sup> Typically, the effectiveness of these campaigns is assessed using online engagement and direct response metrics, such as likes, clicks, or website activity, along with “Brand Lift” surveys that measure short-term effects (Athey et al. 2023; Shawky et al. 2019). However, while these methods may be adequate to evaluate ads’ impact on purchase intent and sales (e.g., Bart et al. 2014; Johnson et al. 2017), they fall short when assessing the longer-term objectives of social marketing campaigns, such as those promoting public health. These campaigns require outcome data that go beyond what can be measured on ad platforms and may need offline data, as their goal is to influence offline behaviors. Moreover, even when ad content is proven effective, campaigns may fail to reach the intended audience due to the ad delivery optimization algorithms used by social media platforms (Lambrecht and Tucker 2019; Sapiezynski et al. 2022). Additionally, recent changes in privacy policies by regulators and tech companies, which restrict access to third-party data that facilitates targeting, can further hinder both the success and assessment of these advertising efforts (Goldfarb and Tucker 2011 _b_ ; Gordon et al. 2021; Johnson et al. 2020).

This paper addresses these measurement and targeting challenges by employing a newly-developed experimental methodology and toolkit (Rao et al. 2020) to evaluate the impact of a nationwide malaria prevention ad campaign and its heterogeneous treatment effects across sub-populations in India. By linking survey responses and administrative records with (potential) ad exposure data, our design enables us to assess the causal impact of online public health messages on offline behaviors and real-world outcomes, such as consistent bed net usage, timely treatment-seeking, and malaria incidence. Additionally, we investigate whether the ad content reached its intended audience by explicitly targeting and oversampling respondents at higher risk of contracting the disease. Our findings show that the ad campaign was effective at improving health outcomes and reducing malaria incidence only among individuals who were ex-ante less likely to contract the disease, underscoring the need for improved measurement and targeting in public health campaigns.

Malaria is transmitted by the bite of infected mosquitoes and disproportionately affects poor and rural people in India (Dev et al., 2004). The ad campaign was designed by the public health NGO Malaria No More (MNM) and used behaviorally-informed ads to provide information and encourage individuals to take preventive measures (e.g., sleeping under mosquito nets) and response actions (e.g., timely seeking treatment).<sup>2</sup> It was run on Meta platforms and aimed to achieve its public health objectives by maximizing engagement with the ads and MNM’s pages. While it succeeded in achieving these engagement metrics (e.g., the ads reached 130 million people and garnered 46

> 1Social marketing entails applying commercial marketing techniques to influence the behavior of target audiences to enhance personal welfare and societal well-being (Andreasen 1994; Kotler and Zaltman 1971).

> 2For example, 75% of households in our baseline survey have mosquito nets, but only 70% of them use them regularly, implying that almost 50% of households remain unprotected to some degree.

2